chore(deps): update dependency sentencepiece to v0.2.2

[dreadnode-renovate-bot]

This PR contains the following updates:

Package	Change	Age	Confidence
sentencepiece	==0.2.1 → ==0.2.2

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

google/sentencepiece (sentencepiece)

v0.2.2

Compare Source

Release Notes v0.2.2

This release contains major updates, including the migration of the Python wrapper to pybind11, native NumPy support, enhanced offset mapping, performance optimizations, security hardening, and internal code modernization.

1. Python Wrapper (pybind11 Migration & Features)

• SWIG to pybind11 Migration: Fully migrated the Python wrapper from SWIG to pybind11, enabling better performance, safer memory management, and cleaner code integration (#​1266, Commit 528dc9e).
• Native NumPy Support:
  • direct encode/decoder from/to numpy object
• Enhanced Offset Mapping:
  • Added new offset mapping features supporting both Unicode character offsets and raw byte offsets.
  • Added bounds checking to prevent OOB reads in offset mapping (Commit c5b0ce0).
  • Fixed crash issues and OOB spans in alignment conversions (#​1231, #​1211).
• Type Stubs (PEP 561): Added inline type stubs (.pyi files) and packaged PEP 561 marker to provide auto-completion and static type safety in IDEs (#​1258, Commit 8b7f97b).
• Parallel Encoding API: Introduced parallel_encode and ThreadPool APIs, allowing Python users to tokenize single large documents across multiple CPU cores in parallel (#​1254, Commit 3c37c39).
• Direct Proto Loading: Added support for loading processors and normalizers directly from in-memory Proto instances (Commit 12788ba).
• Concurrency Enhancements: Added GIL release during C++ execution to improve multi-threaded Python performance (#​1251, Commit 0a044ae).
• BOS/EOS Behavior Hardening: Aligned BOS/EOS adding behavior with C++ constraints, raising explicit ValueError instead of silent failures or crashes (Commit 780618a).

2. Performance Optimizations

• BPE Training Optimization: Ported O(log n) lazy priority queue update optimization to BPE trainer to speed up training (Commit 8db9878).
• Encode/Decode Optimizations: Decoding/Encoding optimization for integer/string input case by skipping SentencePieceText conversion, resulting in ~2x speedup (Commit 4b9231b).
• Unigram Pruning: Improved unigram pruning using token-normalized frequency calculations (#​1135, Commit 1cd0c59).

3. Security & Hardening

• Trie Validation at Load: Added verification of darts-clone trie offsets when loading models to prevent segfaults on corrupted or malicious model files (#​1224, Commit 6b2f05d).
• Bounds Checking in Maps: Added bounds checks to piece accessors and validated value offsets in DecompileCharsMap (#​1265, Commit 82b7c6d).
• Crash Prevention: Replaced internal CHECK macros with DCHECK or returned errors to prevent host application crashes on invalid inputs (Commit 66b2587).
• Null Character Disallowment: Explicitly disallow vocabulary pieces containing null characters (Commit eb3ba49).
• NBest Timeout: Implemented a global timeout for NBest A* search to prevent hangs on pathological inputs (Commit ba57749).

4. C++ API & Internals Modernization

• Abseil Migration: Extensively migrated from custom utilities to Abseil primitives ([#​1270]
• Normalizer Maps: Supported loading normalizer from raw maps in C++ (#​1273, Commit e476e5e).

5. Build & Packaging

• Deprecate Win32 Wheels: Stopped support and distribution of 32-bit Windows wheels (win32) (Commit 24fd00c).
• Modernized Python builds: Switched from calling setup.py directly to the standard pypa/build PEP 517 frontend (#​1167, Commit f300c7d).
• PEP 639 License: Added PEP 639 license metadata to pyproject.toml (#​1173).

6. Documentation

• API Cheat Sheet: Added comparison tables, zero-copy, and sampling comparison guides.
• Refactored Options: Fully reorganized options.md and special_symbols.md with verified code examples.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR has been generated by Mend Renovate.

[GangGreenTemperTatum]

Approving renovate dependency update for merge.