Skip to content

perf: Use Memory-Mapped Files (mmap) for Zero-Copy Hashing & Fast Scanning #125

Description

@kavix

🎯 Goal

Replace standard os.ReadFile / io.ReadAll with memory-mapped file access (mmap) during file hashing and diff generation for files larger than 1MB.

💡 Why

Standard file reading allocates Go heap memory buffers and copies bytes from OS kernel space to user space.
With mmap: the file is mapped directly into process memory space, allowing the CPU to hash bytes directly from OS page cache with zero heap allocations and zero memory copies.

🛠️ Implementation Steps

  1. Add golang.org/x/sys/unix / golang.org/x/exp/mmap wrapper in internal/util/mmap.go.
  2. Use mmap in internal/objects/store.go and internal/api/diff.go for files exceeding 1MB.
  3. Ensure proper memory unmapping (munmap) on deferred cleanup.
  4. Benchmark heap allocations (allocs/op) and CPU cycles during snapshot saves.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions