Skip to content

Pull requests: huggingface/tokenizers

New pull request New
46 Open 195 Closed
46 Open 195 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

perf(unigram): pre-size token map and replace per-node HashMap with Vec
#2039 opened Apr 26, 2026 by taeyun16 Loading…
1
feat(ByteLevel): skip per-byte transform for printable-ASCII tokens
#2038 opened Apr 26, 2026 by KimBioInfoStudio Loading…
2 of 3 tasks
3
feat(NFC): skip Unicode pass for all-ASCII inputs
#2037 opened Apr 26, 2026 by KimBioInfoStudio Loading…
2 of 3 tasks
1
feat: SIMD ASCII fast path for Lowercase normalizer (~30-49x)
#2036 opened Apr 26, 2026 by KimBioInfoStudio Loading…
5 of 6 tasks
1
V0.23 release
#2032 opened Apr 24, 2026 by ArthurZucker Collaborator Loading…
1
perf(byte_level): port GPT-2 split regex to logos FSM (−22% on GPT-2 encode)
#2031 opened Apr 23, 2026 by ArthurZucker Collaborator Loading…
2
Real-world (batch × input_length) tokenizer benchmark + cross-library leaderboard
#2030 opened Apr 23, 2026 by ArthurZucker Collaborator Loading…
4 tasks
1
Batch encode: lock-free work queue with dynamic window sizing
#2029 opened Apr 23, 2026 by sebpop Contributor Loading…
5
perf: skip alignment tracking in encode_fast normalization
#2022 opened Apr 10, 2026 by ArthurZucker Collaborator Loading…
5
feat: Normalizer::normalize_str — skip NormalizedString allocation
#2020 opened Apr 10, 2026 by ArthurZucker Collaborator Loading…
1
Reduce crate size
#2015 opened Apr 9, 2026 by ArthurZucker Collaborator Loading…
6
feat: open-addressing merge table with cache-line-local linear probing for BPE
#2012 opened Apr 8, 2026 by ArthurZucker Collaborator Loading…
4
feat: compact vocabulary — single-allocation id→token store for BPE
#2011 opened Apr 8, 2026 by ArthurZucker Collaborator Loading…
5
node: bump version to 0.22.2 for release
#2009 opened Apr 4, 2026 by MayCXC Contributor Loading…
2
feat(pattern): parallel regex find_matches for large inputs
#2003 opened Mar 31, 2026 by McPatate Member Loading…
3
fix: skip serializing ByteLevel fields at their default value
#2001 opened Mar 30, 2026 by ArthurZucker Collaborator Loading…
1
1
Regex split parity
#1991 opened Mar 27, 2026 by ArthurZucker Collaborator Loading…
1
feat: add new faster whitespace split pretok
#1985 opened Mar 26, 2026 by McPatate Member Loading…
1
6
Implementing Parity-aware BPE
#1974 opened Mar 21, 2026 by cimeister Loading…
9
feat: performance, adding pcre2 backend + regex-shards (5-15% speedup)
#1968 opened Mar 19, 2026 by michaelfeil Contributor Loading…
14
feat: Optimize BPE tokenization: sharded cache, packed merge keys, FxHash (10-15% speedup)
#1967 opened Mar 19, 2026 by michaelfeil Contributor Loading…
29
Fix type_ids not applied to overflow encodings
#1965 opened Mar 17, 2026 by joaquinhuigomez Loading…
1
feat: add pcre2 as optional feature
#1959 opened Mar 2, 2026 by wheynelau Contributor Loading…
6
Add get_special_tokens and is_special_token methods
#1945 opened Feb 5, 2026 by ArthurZucker Collaborator Loading…
2 tasks done
1
Add post_process_tokens and post_process_ids methods
#1944 opened Feb 5, 2026 by ArthurZucker Collaborator Loading…
3 tasks done
1
ProTip! no:milestone will show everything without a milestone.