Repository navigation
Conversation
- Added S3TokenizerV3, AudioEncoderV3, MultiHeadAttentionV3, ResidualAttentionBlockV3 in model_v3.py - Implemented onnx2torch_v3 with bias detection for correct V3 conversion - Updated LayerNorm to handle bfloat16 mixed precision on CPU - Fixed apply_rotary_emb to avoid upcasting bfloat16 - Updated test_onnx.py to include v3 model tests with appropriate thresholds - Updated README with V3 support info
WalkthroughAdds a V3 tokenizer implementation (S3TokenizerV3), V3 ONNX→PyTorch converter, registry/checksum entries and docs/tests for Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Client
participant S3TokenizerV3
participant AudioEncoderV3
participant ResidualBlockV3
participant FSQVectorQuantization
participant MixedBatchHandler
Client->>S3TokenizerV3: quantize(mel, mel_len)
S3TokenizerV3->>S3TokenizerV3: check mel_len > max_frames?
alt normal path
S3TokenizerV3->>AudioEncoderV3: forward(mel, mel_len)
loop per residual block
AudioEncoderV3->>ResidualBlockV3: forward(fragment, mask, freqs_cis)
ResidualBlockV3-->>AudioEncoderV3: processed fragment
end
AudioEncoderV3->>FSQVectorQuantization: quantize(encoder_out)
FSQVectorQuantization-->>S3TokenizerV3: tokens, vq_loss
else mixed-batch path
S3TokenizerV3->>MixedBatchHandler: _quantize_mixed_batch(mel, mel_len)
MixedBatchHandler->>MixedBatchHandler: segment inputs & create batches
MixedBatchHandler->>AudioEncoderV3: forward(batched_segments)
AudioEncoderV3-->>MixedBatchHandler: encoded_segments
MixedBatchHandler->>FSQVectorQuantization: quantize(batched_encoded)
MixedBatchHandler->>MixedBatchHandler: merge tokens (overlap handling)
MixedBatchHandler-->>S3TokenizerV3: merged tokens, merged_vq_loss
end
S3TokenizerV3-->>Client: return tokens, vq_loss
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes
Possibly related PRs
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (5)
s3tokenizer/utils.py (2)
177-179: Consider using explicitOptional[str]for type hint.Static analysis flags that PEP 484 prohibits implicit
Optional. While this works at runtime, explicit typing improves clarity.+from typing import Optional + -def onnx2torch_v3(onnx_path: str, - torch_path: str = None, - verbose: bool = False): +def onnx2torch_v3(onnx_path: str, + torch_path: Optional[str] = None, + verbose: bool = False):
305-312: FSMN weight handling approach is clear but could use a comment.The
passstatement may appear incomplete. Consider adding a brief comment explaining that FSMN weights are handled via the initializer scan below rather than through node processing.# 3. FSMN weights if 'fsmn_block/Conv' in name: - pass + pass # FSMN weights are extracted via initializer_map scan below # Handle explicit FSMN weights and Quantizer weights that might not be caught aboves3tokenizer/model_v3.py (3)
160-160: Avoid mutable default argument in function signature.While
ModelConfigV3contains only immutable primitives, using a class instance as a default argument is still considered bad practice. The same instance would be shared across all calls that don't provide a config.- def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()): + def __init__(self, name: str, config: Optional[ModelConfigV3] = None): super().__init__() self.name = name - self.config = config + self.config = config if config is not None else ModelConfigV3()
339-342: Unnecessary override:load_state_dictjust calls super.This override adds no functionality beyond the docstring comment. Consider removing it or adding actual logic if needed.
- def load_state_dict(self, state_dict, strict=True): - # Allow loading state dict with missing keys (like LN bias) if strictly necessary - # But for now we try to stick to standard behavior - return super().load_state_dict(state_dict, strict=strict)
330-333: Consider logging whenstrict=Falsecauses missing/unexpected keys.Using
strict=Falseininit_from_onnxwill silently ignore missing or unexpected keys, which could mask conversion issues. Consider capturing and logging the result.def init_from_onnx(self, onnx_path: str): ckpt = onnx2torch_v3(onnx_path, None, False) # Set verbose back to False - self.load_state_dict(ckpt, strict=False) + missing, unexpected = super().load_state_dict(ckpt, strict=False) + if missing or unexpected: + import warnings + if missing: + warnings.warn(f"Missing keys in ONNX checkpoint: {missing}") + if unexpected: + warnings.warn(f"Unexpected keys in ONNX checkpoint: {unexpected}")
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
README.md(4 hunks)s3tokenizer/__init__.py(4 hunks)s3tokenizer/model.py(1 hunks)s3tokenizer/model_v2.py(1 hunks)s3tokenizer/model_v3.py(1 hunks)s3tokenizer/utils.py(2 hunks)test/test_onnx.py(4 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
s3tokenizer/__init__.py (2)
s3tokenizer/model_v3.py (1)
S3TokenizerV3(154-346)s3tokenizer/utils.py (4)
merge_tokenized_segments(532-556)onnx2torch(68-174)onnx2torch_v3(177-339)padding(511-529)
s3tokenizer/model_v3.py (1)
s3tokenizer/model_v2.py (9)
FSMNMultiHeadAttention(152-249)FSQVectorQuantization(120-149)forward(237-249)forward(275-287)forward(320-354)forward(384-386)quantize(389-417)encode(97-112)encode(142-143)
🪛 Ruff (0.14.8)
s3tokenizer/utils.py
178-178: PEP 484 prohibits implicit Optional
Convert to T | None
(RUF013)
s3tokenizer/model_v3.py
160-160: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
201-201: Unused method argument: max_frames
(ARG002)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (14)
s3tokenizer/model_v2.py (1)
58-59: LGTM! Dtype consistency fix for bfloat16 support.Casting
cosandsintoxq.dtypeensures proper dtype alignment when using bfloat16 mixed precision, avoiding implicit upcasting during the rotary embedding computation.s3tokenizer/utils.py (1)
551-552: Minor formatting change - no functional impact.README.md (2)
24-24: LGTM! Documentation correctly reflects V3 model addition.
38-38: Usage examples updated consistently.The comment now includes
speech_tokenizer_v3_25hzas an available option alongside other model variants.s3tokenizer/model.py (1)
45-52: LGTM! LayerNorm updated for bfloat16 mixed-precision support.Computing layer normalization in float32 and casting back to the input dtype is the correct approach for CPU bfloat16 compatibility where native mixed-precision support may be limited.
test/test_onnx.py (3)
313-315: Verify the 4x higher miss rate threshold for V3 is intentional.The V3 model uses a 2.0% threshold compared to 0.5% for other models. This is a significant difference. Ensure this aligns with expected V3 model behavior and isn't masking conversion issues.
274-274: LGTM! V3 model correctly added to test parameters.
369-371: Threshold consistency maintained between short and long audio tests.The same 2.0%/0.5% threshold logic is applied to long audio tests, which is consistent with the short audio test approach.
s3tokenizer/__init__.py (3)
28-28: LGTM! S3TokenizerV3 import added for V3 model support.
49-51: V3 model URL and registry entry added correctly.The model URL points to the Fun-CosyVoice3-0.5B-2512 repository on ModelScope.
154-159: Model version routing logic is correct and follows existing pattern.The routing checks for 'v3' first, then 'v2', with S3Tokenizer as the fallback for v1 models. This is consistent with the established pattern.
s3tokenizer/model_v3.py (3)
28-37: LGTM!The dataclass configuration with immutable primitive defaults is clean and appropriate.
40-52: LGTM!Overriding the linear projections to enable bias for V3 is a valid approach. The parent class creates these layers first, but they're immediately replaced here, which is functionally correct.
55-83: LGTM!Clean residual attention block implementation with pre-norm architecture and proper residual connections.
Corrected the epsilon value for LayerNorm layers in S3TokenizerV3 (model_v3.py) from 1e-6 to 1e-5 to precisely match the value used in the exported ONNX model. This change significantly reduces numerical discrepancies and improves the token mismatch rate between the PyTorch and ONNX implementations of the V3 model.
The max_frames parameter was being passed to _quantize_mixed_batch but was not being used, as the function internally calculated frames_per_window. This commit removes the unused parameter from both the function signature and its call site, improving code clarity.
The computation of 'cos' and 'sin' tensors from 'freqs_cis' in AudioEncoderV3's forward method was identified as dead code, as these computed tensors were never subsequently used. Only the original 'freqs_cis' was passed to the attention blocks. This commit removes the unused computation for improved code clarity and maintainability.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
s3tokenizer/model_v3.py (2)
152-152: Consider avoiding mutable default argument.While dataclasses are generally safe here since they're not mutated, using
Noneas default and creating the instance inside the function is the conventional pattern.- def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()): + def __init__(self, name: str, config: Optional[ModelConfigV3] = None): super().__init__() self.name = name - self.config = config + self.config = config if config is not None else ModelConfigV3()
329-332: Unnecessary wrapper method.This
load_state_dictoverride just callssuper()with the same arguments. The comment hints at future plans, but as-is it adds no functionality.Either remove the override entirely, or if you anticipate needing custom logic, add a TODO comment:
def load_state_dict(self, state_dict, strict=True): - # Allow loading state dict with missing keys (like LN bias) if strictly necessary - # But for now we try to stick to standard behavior + # TODO: Add custom handling for missing keys (e.g., LN bias) if needed return super().load_state_dict(state_dict, strict=strict)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
s3tokenizer/model_v3.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
s3tokenizer/model_v3.py (3)
s3tokenizer/model.py (9)
Conv1d(65-70)LayerNorm(43-52)Linear(55-62)forward(45-52)forward(57-62)forward(96-106)forward(158-165)forward(199-222)forward(329-330)s3tokenizer/model_v2.py (6)
FSMNMultiHeadAttention(152-249)FSQVectorQuantization(120-149)forward(237-249)forward(275-287)forward(320-354)forward(384-386)s3tokenizer/utils.py (4)
make_non_pad_mask(435-472)mask_to_bias(475-508)merge_tokenized_segments(532-556)onnx2torch_v3(177-339)
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py
152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (5)
s3tokenizer/model_v3.py (5)
1-26: LGTM!Clean imports and well-organized dependencies. Good approach to re-use V2 components where applicable.
28-37: LGTM!Configuration dataclass is well-structured with sensible defaults. The
n_codebook_size = 3**8is a nice way to express the FSQ codebook dimensionality.
40-83: LGTM!Clean implementation of V3-specific attention and residual blocks. The explicit bias=True override correctly differentiates V3 from the parent class, and the residual connections are properly structured.
188-314: LGTM!The mixed-batch quantization logic correctly handles both short and long audio segments with proper overlap merging. The sliding window approach with configurable overlap is well-implemented.
320-327: LGTM!The ONNX and checkpoint loading utilities are well-implemented. The
mmap=Trueparameter for efficient memory-mapped loading is a nice touch for large checkpoints.
Corrected the calculation of x_slen (sequence length for padding mask) after the second convolutional layer (conv2) in AudioEncoderV3. The previous calculation incorrectly used self.stride (conv1's stride) instead of conv2's hardcoded stride of 2. This fix ensures accurate mask generation for downstream blocks.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
s3tokenizer/model_v3.py (1)
152-152: PreferNoneas default and instantiate inside the function.Using a function call
ModelConfigV3()as a default argument is evaluated once at function definition time, which can lead to unexpected behavior if the config object is mutated. While this is safe for immutable dataclass defaults, it's better practice to use the standard pattern.As per static analysis, this is flagged by Ruff B008.
Apply this diff:
- def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()): + def __init__(self, name: str, config: Optional[ModelConfigV3] = None): super().__init__() self.name = name - self.config = config + self.config = config if config is not None else ModelConfigV3()
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
s3tokenizer/model_v3.py(1 hunks)test/test_batch_efficiency.py(5 hunks)
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py
152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (9)
test/test_batch_efficiency.py (2)
8-11: LGTM!The import reordering has no functional impact and is acceptable.
53-53: LGTM!Adding the V3 model variant to all test parametrizations ensures comprehensive test coverage for the new model. The changes are consistent and aligned with the PR objectives.
Also applies to: 168-168, 218-218, 234-234
s3tokenizer/model_v3.py (7)
28-38: LGTM!The ModelConfigV3 dataclass provides sensible defaults for the V3 architecture with appropriate documentation.
40-53: LGTM!The V3 attention module correctly overrides the linear layers to enable bias (PyTorch's default), which is the intended difference from V2.
55-84: LGTM!The residual attention block follows standard transformer architecture with proper residual connections, masking support, and rotary position embeddings.
86-144: LGTM!The audio encoder properly implements strided convolutions with correct length calculations, masking, and applies the transformer blocks with rotary position embeddings. Past review concerns about stride consistency have been addressed.
173-186: LGTM!The quantize method correctly routes long audio (>3000 frames) to the mixed-batch handler and processes standard audio through the encoder directly.
188-314: LGTM!The mixed-batch quantization method correctly segments long audio into overlapping windows, processes them efficiently in a unified batch, and merges the results. Past review concerns about the unused
max_framesparameter have been addressed.
316-336: LGTM!The utility methods provide appropriate functionality for device introspection, weight loading from different formats (ONNX with flexible matching, PyTorch with strict matching), and parameter freezing.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
s3tokenizer/model_v3.py (2)
152-167: Avoid mutable default argument in function signature.The static analysis correctly flags
ModelConfigV3()as a function call in argument defaults. While dataclasses are relatively safe here, the idiomatic pattern is to useNoneand create the default inside the function.- def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()): + def __init__(self, name: str, config: Optional[ModelConfigV3] = None): super().__init__() self.name = name - self.config = config + self.config = config if config is not None else ModelConfigV3()
292-295: Consider parameterizingtoken_ratefor future model variants.The
token_rateis hardcoded to 25, which is correct for V3 (25hz). However, if future V3 variants use different rates, this would need modification.+ # V3 uses 25hz token rate token_rate = 25A comment clarifying this assumption would improve maintainability.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
s3tokenizer/model_v3.py(1 hunks)test/test_onnx.py(3 hunks)
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py
152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (12)
test/test_onnx.py (3)
7-7: LGTM!Import reordering is purely cosmetic with no semantic change.
272-275: LGTM!Adding the V3 model variant to the short audio test coverage is appropriate. The token_rate logic at lines 177-180 correctly defaults to 25 for non-v1 models, which aligns with the "25hz" in the model name.
320-323: LGTM!V3 model added to long audio test coverage. The existing test infrastructure handles the V3 variant correctly.
s3tokenizer/model_v3.py (9)
1-26: LGTM!License header and imports are well-organized. The reuse of V2 components where appropriate follows good DRY principles.
28-38: LGTM!Configuration dataclass with sensible defaults for the V3 model architecture.
40-52: LGTM!V3 attention correctly configures bias for Linear layers, with
keyhavingbias=Falseand others using the defaultbias=True. This aligns with the PR objective of bias detection for correct V3 conversion.
55-83: LGTM!Pre-norm residual attention block with proper residual connections. The MLP uses standard 4x expansion factor.
86-143: LGTM!Audio encoder with two-stage convolutions and residual attention blocks. The stride calculations at lines 132-133 correctly use the hardcoded stride of 2 matching
conv2's configuration.
169-186: LGTM!The quantization flow correctly separates short and long audio processing paths with the 3000-frame (30s) threshold.
316-318: LGTM!Standard pattern for getting the device from module parameters.
320-323: Note:strict=Falsemay mask loading issues.Using
strict=Falsewhen loading the ONNX-converted state dict could silently ignore missing or unexpected keys. Consider logging mismatches for debugging purposes.def init_from_onnx(self, onnx_path: str): ckpt = onnx2torch_v3(onnx_path, None, False) # Set verbose back to False - self.load_state_dict(ckpt, strict=False) + missing, unexpected = super().load_state_dict(ckpt, strict=False) + if missing or unexpected: + import logging + logging.warning(f"ONNX load: missing={missing}, unexpected={unexpected}")
329-336: LGTM!The
freezemethod correctly disables gradient computation for all parameters. Theload_state_dictoverride is passthrough but documents the intent for potential future customization.
| def init_from_pt(self, ckpt_path: str): | ||
| ckpt = torch.load(ckpt_path, map_location="cpu", mmap=True) | ||
| self.load_state_dict(ckpt, strict=True) |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
torch.load weights_only parameter PyTorch 2.0 security
💡 Result:
Short answer: In April 2025 a critical RCE vulnerability (CVE-2025-32434) was reported in torch.load when called with weights_only=True for affected PyTorch releases (<= 2.5.1). The bug allowed crafted legacy-format checkpoint files to bypass the weights-only restrictions and execute arbitrary code. The issue was fixed in PyTorch 2.6.0; recommended mitigations are to upgrade to a patched PyTorch, never load untrusted .pt/.pth files, prefer saving/loading state_dicts or using safetensors, and run model-loading inside an isolated environment when needed. [1][2][3][4]
Sources
- PyTorch GitHub advisory (GHSA) / fix and affected/patched versions. [1]
- CVE/technical writeups describing legacy-format bypass and RCE (CVE-2025-32434). [2][3]
- PyTorch serialization docs and guidance to use weights_only, safetensors, or allowlisting. [4]
References
[1] PyTorch advisory (GHSA) for torch.load weights_only RCE. (github.com)
[2] CVE summary / vulnerability databases reporting CVE-2025-32434. (cve.news)
[3] Press/analysis writeups on the RCE and exploit vector. (cyberpress.org)
[4] PyTorch serialization docs and safe-loading guidance (weights_only, safe_globals, safetensors recommendation). (docs.pytorch.org)
If you want, I can (a) check the exact PyTorch wheel version you have installed and tell you whether it's vulnerable, or (b) give example commands to upgrade, convert checkpoints to safetensors, or inspect a checkpoint safely.
Citations:
- 1: https://github.com/advisories/GHSA-53q9-r3pm-6pq6?utm_source=openai
- 2: https://www.cve.news/cve-2025-32434/?utm_source=openai
- 3: https://cyberpress.org/critical-pytorch-vulnerability/?utm_source=openai
- 4: https://docs.pytorch.org/docs/stable/notes/serialization?utm_source=openai
🏁 Script executed:
# Check the actual code in the repository
find . -name "model_v3.py" -type fRepository: xingchensong/S3Tokenizer
Length of output: 93
🏁 Script executed:
# Read the actual code at the specified lines
sed -n '320,335p' ./s3tokenizer/model_v3.py
# Check for PyTorch version requirements
find . -name "requirements.txt" -o -name "setup.py" -o -name "pyproject.toml" -o -name "environment.yml" | head -5Repository: xingchensong/S3Tokenizer
Length of output: 808
🏁 Script executed:
# Check PyTorch version requirements
cat ./requirements.txt ./setup.py 2>/dev/null | grep -i torch
# Also check pyproject.toml if it exists
if [ -f ./pyproject.toml ]; then cat ./pyproject.toml | grep -i torch; fiRepository: xingchensong/S3Tokenizer
Length of output: 84
Security: torch.load requires comprehensive protection beyond weights_only.
torch.load() without protections can execute arbitrary code. While weights_only=True is recommended in PyTorch documentation, a critical RCE vulnerability (CVE-2025-32434) exists in PyTorch versions ≤ 2.5.1 that bypasses weights_only restrictions using legacy-format checkpoints. Since your requirements do not pin PyTorch version, this project is at risk if any version ≤ 2.5.1 is installed.
Recommended actions: Either (1) upgrade PyTorch to 2.6.0+ and add weights_only=True, (2) load only state_dict() files instead of full checkpoints, (3) adopt safetensors format, or (4) pin PyTorch version to 2.6.0+.
🤖 Prompt for AI Agents
In s3tokenizer/model_v3.py around lines 325-327, using torch.load(ckpt_path,
map_location="cpu", mmap=True) is unsafe for PyTorch ≤2.5.1 because
legacy-format checkpoints can execute code; change to a safe loading strategy:
either (A) require and assert torch.__version__ >= "2.6.0" and call
torch.load(ckpt_path, map_location="cpu", weights_only=True) (and fail loudly if
version is older), or (B) accept only state_dict files and load via
torch.load(..., weights_only=True) into a dict then call
self.load_state_dict(state_dict, strict=True), or (C) switch to safetensors
format and use its safe loader, and also update/pin project dependencies to
PyTorch >=2.6.0 in requirements to ensure the protection is present.
|
============================= test session starts ============================== S3Tokenizer/test/test_onnx.py speech_tokenizer_v1 Short Audio Summary: speech_tokenizer_v1_25hz Short Audio Summary: speech_tokenizer_v2_25hz Short Audio Summary: speech_tokenizer_v3_25hz Short Audio Summary: speech_tokenizer_v1 Long Audio Results: speech_tokenizer_v1_25hz Long Audio Results: speech_tokenizer_v2_25hz Long Audio Results: speech_tokenizer_v3_25hz Long Audio Results: |
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
s3tokenizer/model_v3.py (1)
320-327: Hardeninit_from_ptagainst unsafetorch.loaddeserialization (duplicate of prior concern).
torch.load(ckpt_path, map_location="cpu", mmap=True)will execute arbitrary code embedded in a crafted checkpoint. Even withweights_only=True, PyTorch ≤ 2.5.1 has a known RCE issue; safest is to both (a) pin torch ≥ 2.6.0 in your dependencies and (b) load weights‑only state dicts.Within this method, you can at least switch to
weights_only=Trueand make it clear that a state_dict is expected:def init_from_pt(self, ckpt_path: str): - ckpt = torch.load(ckpt_path, map_location="cpu", mmap=True) - self.load_state_dict(ckpt, strict=True) + # Expect a state_dict checkpoint and load weights only. + ckpt = torch.load( + ckpt_path, + map_location="cpu", + weights_only=True, + ) + self.load_state_dict(ckpt, strict=True)And separately, update your
requirements.txt/pyproject.tomlto requiretorch>=2.6.0soweights_onlyactually provides the intended protection.To double‑check current PyTorch guidance and affected versions, please run a quick web search for the latest
torch.load weights_only vulnerabilityand ensure your pinned torch version is in the patched range.
🧹 Nitpick comments (2)
s3tokenizer/model_v3.py (2)
152-163: Avoid callingModelConfigV3()in the default argument (Ruff B008).Using a constructed dataclass instance as a default value means a single shared
ModelConfigV3is reused across all instances, which is brittle if the config is ever mutated. It also triggers Ruff B008.Consider deferring the construction into the body:
- def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()): - super().__init__() - self.name = name - self.config = config + def __init__(self, name: str, config: Optional[ModelConfigV3] = None): + super().__init__() + self.name = name + self.config = config or ModelConfigV3()
173-314: Mixed‑batch quantization logic is sound; consider normalizing length types for clarity.The windowing/overlap logic and use of
merge_tokenized_segmentslook correct and consistent with a 30s / 25 Hz setup. To make the control flow more robust and remove subtle int/Tensor mixing, you can convert per‑sample lengths to Python ints once:- for batch_idx in range(batch_size): - audio_mel = mel[batch_idx] - audio_mel_len = mel_len[batch_idx] - is_long_audio = long_audio_mask[batch_idx].item() + for batch_idx in range(batch_size): + audio_mel = mel[batch_idx] + audio_mel_len = int(mel_len[batch_idx].item()) + is_long_audio = bool(long_audio_mask[batch_idx].item()) @@ - if not is_long_audio: - segment = audio_mel[:, :audio_mel_len] - seg_len = audio_mel_len.item() + if not is_long_audio: + segment = audio_mel[:, :audio_mel_len] + seg_len = audio_mel_len @@ - else: - start = 0 - segment_idx = 0 - while start < audio_mel_len: - end = min(start + frames_per_window, audio_mel_len) + else: + start = 0 + segment_idx = 0 + while start < audio_mel_len: + end = min(start + frames_per_window, audio_mel_len)This keeps all indexing and loop conditions in plain Python ints, which is easier to reason about and avoids any future surprises if
mel_len’s dtype or device handling changes.Optionally, you might also derive
max_framesinquantizefrom the same formula used forframes_per_windowto make their coupling explicit (30 s @ 100 fps → 3000 frames).
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
s3tokenizer/model_v3.py(1 hunks)test/test_onnx.py(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- test/test_onnx.py
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py
152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
🔇 Additional comments (1)
s3tokenizer/model_v3.py (1)
40-83: Attention + residual block wiring looks consistent with V2/V3 shape expectations.Bias patterns on Q/K/V/Out and the LayerNorm+MLP residual structure match the usual transformer block layout and should align well with the V3 weights. No issues spotted here.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
s3tokenizer/model_v3.py (1)
325-327: Prior security concern abouttorch.loadremains unaddressed.A previous review flagged the security risk of using
torch.loadwithoutweights_only=Truedue to CVE-2025-32434 affecting PyTorch ≤2.5.1. This allows arbitrary code execution when loading untrusted checkpoints. The issue has not been resolved.
🧹 Nitpick comments (1)
s3tokenizer/model_v3.py (1)
152-152: Avoid mutable default argument to prevent shared state.Using
ModelConfigV3()as a default argument creates a single shared instance across all calls. If any code mutates the config object, it affects all instances. UseNoneas the default and instantiate inside the method.Apply this diff:
- def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()): + def __init__(self, name: str, config: Optional[ModelConfigV3] = None): super().__init__() self.name = name - self.config = config + self.config = config if config is not None else ModelConfigV3()Don't forget to import
Optionalfromtypingat the top if not already present.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
s3tokenizer/model_v2.py(3 hunks)s3tokenizer/model_v3.py(1 hunks)test/test_onnx.py(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- test/test_onnx.py
🧰 Additional context used
🧬 Code graph analysis (2)
s3tokenizer/model_v3.py (3)
s3tokenizer/model.py (13)
Conv1d(65-70)LayerNorm(43-52)Linear(55-62)forward(45-52)forward(57-62)forward(96-106)forward(158-165)forward(199-222)forward(329-330)quantize(244-249)quantize(333-360)encode(261-269)encode(295-298)s3tokenizer/model_v2.py (9)
FSMNMultiHeadAttention(152-250)FSQVectorQuantization(120-149)forward(238-250)forward(276-288)forward(321-355)forward(385-387)quantize(390-418)encode(97-112)encode(142-143)s3tokenizer/utils.py (5)
make_non_pad_mask(435-472)mask_to_bias(475-508)merge_tokenized_segments(532-556)onnx2torch_v3(177-339)padding(511-529)
s3tokenizer/model_v2.py (1)
s3tokenizer/model.py (2)
Linear(55-62)LayerNorm(43-52)
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py
152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
🔇 Additional comments (9)
s3tokenizer/model_v2.py (3)
58-59: LGTM! Dtype consistency improved.The explicit cast to
xq.dtypeensures rotary embeddings work correctly with mixed precision (e.g., bfloat16), preventing dtype mismatches during computation.
176-176: Remove redundantself.keydefinition; parentMultiHeadAttentionalready defines it with identical settings.The parent
MultiHeadAttentionclass definesself.key = Linear(n_state, n_state, bias=False)at line 90 ofmodel.py. SinceFSMNMultiHeadAttentioninherits from it and does not change the bias setting, this override at line 176 is redundant and can be removed.
268-268: Verify LayerNorm epsilon value and its compatibility with existing V2 checkpoints.The
attn_lnLayerNorm useseps=1e-5whilemlp_lnuses the default. Confirm this difference is intentional and validate compatibility with any existing V2 checkpoint versions, particularly if they were created with different epsilon values.s3tokenizer/model_v3.py (6)
28-38: LGTM!The V3 configuration dataclass is well-structured with appropriate defaults for the 12-layer V3 architecture.
55-84: LGTM!The residual attention block is correctly structured with proper pre-normalization and residual connections.
86-144: LGTM!The encoder correctly applies the two-stage CNN front-end with proper length tracking and passes the appropriate rotary frequency components to the attention blocks.
146-187: LGTM!The quantization method correctly handles both short and long audio with an appropriate 30-second threshold (3000 frames).
188-314: LGTM!The mixed-batch quantization correctly implements sliding-window processing for long audio with appropriate overlap handling and token merging at 25Hz rate.
334-336: LGTM!The freeze method correctly disables gradient computation for all parameters.
|
============================= test session starts ============================== S3Tokenizer/test/test_onnx.py speech_tokenizer_v1 Short Audio Summary: speech_tokenizer_v1_25hz Short Audio Summary: speech_tokenizer_v2_25hz Short Audio Summary: speech_tokenizer_v3_25hz Short Audio Summary: speech_tokenizer_v1 Long Audio Results: speech_tokenizer_v1_25hz Long Audio Results: speech_tokenizer_v2_25hz Long Audio Results: speech_tokenizer_v3_25hz Long Audio Results: ======================== 8 passed in 149.84s (0:02:29) ========================= |
…d align Eps Detailed analysis revealed that the Key projection layer in the ONNX model (both V2 and V3) does not have a bias term (unlike Query and Value). This commit sets 'bias=False' for the Key Linear layer in both model_v2.py and model_v3.py, correcting a significant source of numerical divergence. Additionally, aligned LayerNorm epsilon to 1e-5 for both V2 and V3 models to match ONNX constants. Findings & Precautions: 1. **Key Bias**: The ONNX model structure for 'speech_tokenizer_v3' and 'v2' has an asymmetric bias configuration. Query and Value projections have bias terms, but the Key projection does NOT. PyTorch's default 'Linear' includes bias, which caused substantial output mismatch. 2. **LayerNorm Epsilon**: The V2/V3 ONNX models use 'eps=1e-5', differing from the default '1e-6'. This was corrected. 3. **BF16 Support**: Fixed 'LayerNorm' and 'RotaryEmbedding' to support BFloat16 inference on CPU without mixed-precision errors. Final Test Results (PyTorch vs ONNX, 0.5% Threshold): - speech_tokenizer_v3_25hz: - Short Audio (All 4): ✅ 0.00% Miss Rate - Long Audio (All 3): ✅ 0.00% Miss Rate - speech_tokenizer_v2_25hz: - Short Audio (All 4): ✅ 0.00% Miss Rate (Improved from 0.27%) - Long Audio (All 3): ✅ 0.00% Miss Rate All tests passed with strict alignment.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
s3tokenizer/model_v3.py (1)
325-327: Security:torch.loadrequires protection (see existing comment).This issue was previously flagged with detailed CVE information and mitigation strategies. Please address the security concern by either adding
weights_only=True(requires PyTorch ≥2.6.0), switching to state_dict loading, or using safetensors format.
🧹 Nitpick comments (5)
s3tokenizer/model_v3.py (5)
152-152: Consider moving default instantiation inside the function.While safe here (dataclasses create new instances), calling
ModelConfigV3()in the default argument is evaluated at function definition time, which can be surprising and is flagged by linters.Apply this diff:
- def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()): + def __init__(self, name: str, config: Optional[ModelConfigV3] = None): super().__init__() self.name = name - self.config = config + self.config = config if config is not None else ModelConfigV3()
178-178: Consider makingmax_framesthreshold configurable.The 3000-frame threshold for long audio detection is hardcoded. Consider adding it to
ModelConfigV3for easier tuning across different use cases.
195-198: Hardcoded audio parameters should match input assumptions.The constants
sample_rate=16000andhop_length=160must match how the mel spectrogram was computed upstream. Consider either:
- Adding these to
ModelConfigV3and validating they match the input- Adding documentation specifying the expected mel spectrogram parameters
277-277: Consider optimizing the CPU conversion for performance.The
.cpu().numpy().tolist()conversion moves data off-device and creates Python lists. Ifmerge_tokenized_segmentscan be refactored to accept tensors directly, this would avoid the device transfer overhead.
292-292: Document the hardcoded token rate.The
token_rate=25corresponds to the 25Hz output mentioned in the PR title. Consider adding a comment or moving this to a named constant/config for clarity.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
s3tokenizer/model_v2.py(3 hunks)s3tokenizer/model_v3.py(1 hunks)test/test_onnx.py(3 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- test/test_onnx.py
- s3tokenizer/model_v2.py
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py
152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
🔇 Additional comments (4)
s3tokenizer/model_v3.py (4)
40-53: LGTM: V3 attention correctly configured with bias overrides.The implementation properly inherits from FSMNMultiHeadAttention and overrides the linear projections with the correct bias settings for V3 (query/value/out with bias, key without).
55-84: LGTM: Residual attention block follows standard transformer architecture.The implementation correctly applies pre-normalization (LayerNorm before attention/MLP) with residual connections, following modern transformer best practices.
86-144: LGTM: Encoder correctly implements strided convolutions and attention blocks.The forward pass properly tracks sequence lengths through both convolution layers and applies masking consistently throughout the attention blocks.
316-318: LGTM: Utility methods follow standard patterns.The device property, load_state_dict wrapper, and freeze method all follow PyTorch conventions correctly.
Also applies to: 329-336
|
🐮🍺 |
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests
✏️ Tip: You can customize this high-level summary in your review settings.