[tamirms]

Why

A record is the unit of a read: reading any item in it reads all of it. Records closed only on their item count, so a few large items made a record large and every read of a small neighbour paid for them. In index.pack a rare term's bitmap shares a 128-item record with popular terms' bitmaps of up to a megabyte, so a lookup of the rare term read whatever its neighbours weighed: the worst small-term read was 1,062 KB on a real pubnet chunk and 5,757 KB on one folded to seven times its density. In events.pack one contract's big events inflate the reads of other contracts' events. And index.pack's new layout (a later PR in this stack) needs records bounded by bytes alone, so that every index read is one small EBS I/O.

The design

Records get a byte limit, WriterOptions.MaxRecordBytes, which closes a record before an item that would take its raw bytes past the limit; only a record holding a single item can exceed it. The item limit stays, because events.pack's 128 events come to about 32 KB raw, so its records keep closing on count and store no counts; a byte-only budget there would grow the index read per open by 55% on the largest chunk (1.25 MB instead of 0.82 MB). ItemsPerRecord 0 now means no item limit and requires a byte limit.

A record closed early holds fewer items than the limit, so the index has to say how many. A count column per group, a compressed list of the group's record counts marked by the directory's flag bit, costs 0.9 KB on a real pubnet chunk and 6.0 KB on the largest against 2.0 and 13.1 KB for a flat per-record list, and in the worst case of ten million 16 KiB events it is 206 KB decoded per open instead of 10 MB. Counts in every group would add 13 to 74 KB to every open, and a pack-wide "counts present" flag is redundant with the directory's per-group bit. The directory's first items become running totals, so lookups keep their binary search. Packs whose records all fill carry no counts and are byte-identical to before, and a reader without counts rejects a pack with them.

How it works

Create now rejects options with no limit at all, and a byte limit with one item per record; every production writer already sets an item limit. Open checks each group's flag and item span, and a read checks a group's counts when it decodes it. A record closed early keeps the shape its pack's limit sets, so it still carries its item size index and CRC. The code that assumed a positive item limit is gone with the assumption.

Measured

Over six interleaved rounds with the file in the page cache, BenchmarkReader stays within 2.4% of before and BenchmarkOpen within 2.3%. On the same records closed at 128 items by a byte limit under an item limit of 256, so that every group has counts, a ReadItem costs 0.3 µs more for the count column's decode, and ReadItems and ReadRange under 1% more. Bit-flip sweeps of 9,920 and 18,312 single-bit mutations over packs with counts found one gap, a last group that carried no counts when its predecessor did, which is fixed.

🤖 Generated with Claude Code