🎯 Goal
Replace standard os.ReadFile / io.ReadAll with memory-mapped file access (mmap) during file hashing and diff generation for files larger than 1MB.
💡 Why
Standard file reading allocates Go heap memory buffers and copies bytes from OS kernel space to user space.
With mmap: the file is mapped directly into process memory space, allowing the CPU to hash bytes directly from OS page cache with zero heap allocations and zero memory copies.
🛠️ Implementation Steps
- Add
golang.org/x/sys/unix / golang.org/x/exp/mmap wrapper in internal/util/mmap.go.
- Use mmap in
internal/objects/store.go and internal/api/diff.go for files exceeding 1MB.
- Ensure proper memory unmapping (
munmap) on deferred cleanup.
- Benchmark heap allocations (
allocs/op) and CPU cycles during snapshot saves.
🎯 Goal
Replace standard
os.ReadFile/io.ReadAllwith memory-mapped file access (mmap) during file hashing and diff generation for files larger than 1MB.💡 Why
Standard file reading allocates Go heap memory buffers and copies bytes from OS kernel space to user space.
With
mmap: the file is mapped directly into process memory space, allowing the CPU to hash bytes directly from OS page cache with zero heap allocations and zero memory copies.🛠️ Implementation Steps
golang.org/x/sys/unix/golang.org/x/exp/mmapwrapper ininternal/util/mmap.go.internal/objects/store.goandinternal/api/diff.gofor files exceeding 1MB.munmap) on deferred cleanup.allocs/op) and CPU cycles during snapshot saves.