feat: Optimize BPE tokenization: sharded cache, packed merge keys, FxHash (10-15% speedup)#1967
Open
michaelfeil wants to merge 15 commits intohuggingface:mainfrom
Open
feat: Optimize BPE tokenization: sharded cache, packed merge keys, FxHash (10-15% speedup)#1967michaelfeil wants to merge 15 commits intohuggingface:mainfrom
michaelfeil wants to merge 15 commits intohuggingface:mainfrom
Commits
Commits on Mar 19, 2026
- committed
- committed
- committed
- committed
Commits on Mar 24, 2026
- committed
Commits on Mar 25, 2026
- andauthored
- andauthored
Commits on Apr 10, 2026
Commits on Apr 12, 2026
- committed
- committed
- committed
- committed
- committed
- committed
- authored