Skip to content

Stream index creation through byte arrays - #15

Merged
platypii merged 1 commit into
masterfrom
byte-array-index-pipeline
Aug 1, 2026
Merged

Stream index creation through byte arrays#15
platypii merged 1 commit into
masterfrom
byte-array-index-pipeline

Conversation

@platypii

@platypii platypii commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • consume Hyparquet column chunks directly instead of materializing row objects
  • preserve UTF-8 source values as byte arrays during the numeric-token indexing path
  • extract tokens directly from UTF-8 and write shared token byte arrays without decode/re-encode cycles
  • require hyparquet-writer 0.16.4 for byte-array page statistics
  • report peak RSS in the WildChat build benchmark

WildChat benchmark

  • before byte-array output: 37m48.8s, 21.14 GB peak RSS
  • after: 28m29.2s, 17.16 GB peak RSS
  • improvement: 24.7% less time and 18.8% less peak memory
  • output is byte-for-byte identical to the prior index

The previous JavaScript baseline was 36 minutes; this build completes about 20.9% faster.

Testing

  • npm test (155 tests)
  • npm run lint
  • npx tsc --noEmit
  • git diff --check

@platypii
platypii merged commit e8108c4 into master Aug 1, 2026
6 checks passed
@platypii
platypii deleted the byte-array-index-pipeline branch August 1, 2026 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant