Skip to content

feat: support speech_tokenizer_v3_25hz - #47

Merged
Mddct merged 6 commits into
mainfrom
dh-cosy3
Dec 18, 2025
Merged

Mddct merged 6 commits into
mainfrom
dh-cosy3

Conversation

@Mddct

@Mddct Mddct commented Dec 15, 2025 •

Copy link
Copy Markdown
Collaborator
  • Added S3TokenizerV3, AudioEncoderV3, MultiHeadAttentionV3, ResidualAttentionBlockV3 in model_v3.py
  • Implemented onnx2torch_v3 with bias detection for correct V3 conversion
  • Updated LayerNorm to handle bfloat16 mixed precision on CPU
  • Fixed apply_rotary_emb to avoid upcasting bfloat16
  • Updated test_onnx.py to include v3 model tests with appropriate thresholds
  • Updated README with V3 support info

Summary by CodeRabbit

  • New Features

    • Added support for S3Tokenizer V3 (25 Hz): new V3 tokenizer pipeline, long-audio mixed-batch handling, and a V3 ONNX→PyTorch conversion utility.
  • Bug Fixes

    • Improved numeric robustness in rotary embeddings and layer normalization for more stable behavior.
  • Documentation

    • README and usage examples updated to list the new model variant.
  • Tests

    • Test suite updated to include the V3 model variant.

✏️ Tip: You can customize this high-level summary in your review settings.

- Added S3TokenizerV3, AudioEncoderV3, MultiHeadAttentionV3, ResidualAttentionBlockV3 in model_v3.py
- Implemented onnx2torch_v3 with bias detection for correct V3 conversion
- Updated LayerNorm to handle bfloat16 mixed precision on CPU
- Fixed apply_rotary_emb to avoid upcasting bfloat16
- Updated test_onnx.py to include v3 model tests with appropriate thresholds
- Updated README with V3 support info
@coderabbitai

coderabbitai Bot commented Dec 15, 2025 •

Copy link
Copy Markdown

Walkthrough

Adds a V3 tokenizer implementation (S3TokenizerV3), V3 ONNX→PyTorch converter, registry/checksum entries and docs/tests for speech_tokenizer_v3_25hz; updates LayerNorm and rotary-dtype handling and a small attention attribute in v2; no external API removals.

Changes

Cohort / File(s) Summary
Documentation
README.md
Added speech_tokenizer_v3_25hz to supported models and updated usage examples/comments to list the V3 variant.
Package entry & registry
s3tokenizer/__init__.py
Exported S3TokenizerV3 and onnx2torch_v3; added _MODELS and _SHA256S entries for speech_tokenizer_v3_25hz; load_model now selects V3 when name contains 'v3'.
LayerNorm change
s3tokenizer/model.py
Replaced super().forward(...) with explicit F.layer_norm(...) call and cast result back to input dtype.
v2 attention & rotary fixes
s3tokenizer/model_v2.py
Cast cos/sin to query dtype for rotary embeddings; added self.key Linear (no bias) to FSMNMultiHeadAttention; adjusted LayerNorm eps in ResidualAttentionBlock.
New V3 model
s3tokenizer/model_v3.py
New V3 implementation: ModelConfigV3, MultiHeadAttentionV3, ResidualAttentionBlockV3, AudioEncoderV3, S3TokenizerV3; CNN+attention encoder, FSQ quantizer integration, mixed-batch long-audio segmentation/merge, device/load helpers, and forward/quantize flows.
ONNX→PyTorch converter (V3)
s3tokenizer/utils.py
Added onnx2torch_v3 to map V3 ONNX initializers/nodes into a PyTorch state dict (Conv, attention, MLP, LN, quantizer mappings); minor formatting tweak in merge_tokenized_segments.
Tests
test/test_onnx.py, test/test_batch_efficiency.py
Added speech_tokenizer_v3_25hz to parametrized tests and adjusted import ordering; tests now exercise the V3 variant.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    actor Client
    participant S3TokenizerV3
    participant AudioEncoderV3
    participant ResidualBlockV3
    participant FSQVectorQuantization
    participant MixedBatchHandler

    Client->>S3TokenizerV3: quantize(mel, mel_len)
    S3TokenizerV3->>S3TokenizerV3: check mel_len > max_frames?
    alt normal path
        S3TokenizerV3->>AudioEncoderV3: forward(mel, mel_len)
        loop per residual block
            AudioEncoderV3->>ResidualBlockV3: forward(fragment, mask, freqs_cis)
            ResidualBlockV3-->>AudioEncoderV3: processed fragment
        end
        AudioEncoderV3->>FSQVectorQuantization: quantize(encoder_out)
        FSQVectorQuantization-->>S3TokenizerV3: tokens, vq_loss
    else mixed-batch path
        S3TokenizerV3->>MixedBatchHandler: _quantize_mixed_batch(mel, mel_len)
        MixedBatchHandler->>MixedBatchHandler: segment inputs & create batches
        MixedBatchHandler->>AudioEncoderV3: forward(batched_segments)
        AudioEncoderV3-->>MixedBatchHandler: encoded_segments
        MixedBatchHandler->>FSQVectorQuantization: quantize(batched_encoded)
        MixedBatchHandler->>MixedBatchHandler: merge tokens (overlap handling)
        MixedBatchHandler-->>S3TokenizerV3: merged tokens, merged_vq_loss
    end
    S3TokenizerV3-->>Client: return tokens, vq_loss
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

  • Pay extra attention to:
    • s3tokenizer/model_v3.py — attention math, masking, mixed-batch segmentation/merge correctness, dtype/device consistency, and load helpers.
    • s3tokenizer/utils.py (onnx2torch_v3) — mapping of ONNX initializers/nodes to PyTorch names/shapes.
    • s3tokenizer/__init__.py — registry entries and load_model branching.
    • s3tokenizer/model.py & model_v2.py — numerical parity after LayerNorm and rotary dtype changes.

Possibly related PRs

  • [feat] support long audio #35 — Implements similar long-audio segmentation and overlap-aware token merging; strongly related to _quantize_mixed_batch and merging logic.

Poem

🐰 I nibble mels at morning light,

I slice long tracks and stitch them tight.
V3 hops in with attentive ears,
Tokens hop out, avoiding tears. 🎶

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 42.42% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The pull request title clearly and concisely summarizes the main change: adding support for the speech_tokenizer_v3_25hz model variant across the codebase.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch dh-cosy3

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (5)
s3tokenizer/utils.py (2)

177-179: Consider using explicit Optional[str] for type hint.

Static analysis flags that PEP 484 prohibits implicit Optional. While this works at runtime, explicit typing improves clarity.

+from typing import Optional
+
-def onnx2torch_v3(onnx_path: str,
-                  torch_path: str = None,
-                  verbose: bool = False):
+def onnx2torch_v3(onnx_path: str,
+                  torch_path: Optional[str] = None,
+                  verbose: bool = False):

305-312: FSMN weight handling approach is clear but could use a comment.

The pass statement may appear incomplete. Consider adding a brief comment explaining that FSMN weights are handled via the initializer scan below rather than through node processing.

         # 3. FSMN weights
         if 'fsmn_block/Conv' in name:
-            pass
+            pass  # FSMN weights are extracted via initializer_map scan below

     # Handle explicit FSMN weights and Quantizer weights that might not be caught above
s3tokenizer/model_v3.py (3)

160-160: Avoid mutable default argument in function signature.

While ModelConfigV3 contains only immutable primitives, using a class instance as a default argument is still considered bad practice. The same instance would be shared across all calls that don't provide a config.

-    def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()):
+    def __init__(self, name: str, config: Optional[ModelConfigV3] = None):
         super().__init__()
         self.name = name
-        self.config = config
+        self.config = config if config is not None else ModelConfigV3()

339-342: Unnecessary override: load_state_dict just calls super.

This override adds no functionality beyond the docstring comment. Consider removing it or adding actual logic if needed.

-    def load_state_dict(self, state_dict, strict=True):
-        # Allow loading state dict with missing keys (like LN bias) if strictly necessary
-        # But for now we try to stick to standard behavior
-        return super().load_state_dict(state_dict, strict=strict)

330-333: Consider logging when strict=False causes missing/unexpected keys.

Using strict=False in init_from_onnx will silently ignore missing or unexpected keys, which could mask conversion issues. Consider capturing and logging the result.

     def init_from_onnx(self, onnx_path: str):
         ckpt = onnx2torch_v3(onnx_path, None,
                              False)  # Set verbose back to False
-        self.load_state_dict(ckpt, strict=False)
+        missing, unexpected = super().load_state_dict(ckpt, strict=False)
+        if missing or unexpected:
+            import warnings
+            if missing:
+                warnings.warn(f"Missing keys in ONNX checkpoint: {missing}")
+            if unexpected:
+                warnings.warn(f"Unexpected keys in ONNX checkpoint: {unexpected}")
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2abe1c5 and 68390f8.

📒 Files selected for processing (7)
  • README.md (4 hunks)
  • s3tokenizer/__init__.py (4 hunks)
  • s3tokenizer/model.py (1 hunks)
  • s3tokenizer/model_v2.py (1 hunks)
  • s3tokenizer/model_v3.py (1 hunks)
  • s3tokenizer/utils.py (2 hunks)
  • test/test_onnx.py (4 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
s3tokenizer/__init__.py (2)
s3tokenizer/model_v3.py (1)
  • S3TokenizerV3 (154-346)
s3tokenizer/utils.py (4)
  • merge_tokenized_segments (532-556)
  • onnx2torch (68-174)
  • onnx2torch_v3 (177-339)
  • padding (511-529)
s3tokenizer/model_v3.py (1)
s3tokenizer/model_v2.py (9)
  • FSMNMultiHeadAttention (152-249)
  • FSQVectorQuantization (120-149)
  • forward (237-249)
  • forward (275-287)
  • forward (320-354)
  • forward (384-386)
  • quantize (389-417)
  • encode (97-112)
  • encode (142-143)
🪛 Ruff (0.14.8)
s3tokenizer/utils.py

178-178: PEP 484 prohibits implicit Optional

Convert to T | None

(RUF013)

s3tokenizer/model_v3.py

160-160: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)


201-201: Unused method argument: max_frames

(ARG002)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (14)
s3tokenizer/model_v2.py (1)

58-59: LGTM! Dtype consistency fix for bfloat16 support.

Casting cos and sin to xq.dtype ensures proper dtype alignment when using bfloat16 mixed precision, avoiding implicit upcasting during the rotary embedding computation.

s3tokenizer/utils.py (1)

551-552: Minor formatting change - no functional impact.

README.md (2)

24-24: LGTM! Documentation correctly reflects V3 model addition.


38-38: Usage examples updated consistently.

The comment now includes speech_tokenizer_v3_25hz as an available option alongside other model variants.

s3tokenizer/model.py (1)

45-52: LGTM! LayerNorm updated for bfloat16 mixed-precision support.

Computing layer normalization in float32 and casting back to the input dtype is the correct approach for CPU bfloat16 compatibility where native mixed-precision support may be limited.

test/test_onnx.py (3)

313-315: Verify the 4x higher miss rate threshold for V3 is intentional.

The V3 model uses a 2.0% threshold compared to 0.5% for other models. This is a significant difference. Ensure this aligns with expected V3 model behavior and isn't masking conversion issues.


274-274: LGTM! V3 model correctly added to test parameters.


369-371: Threshold consistency maintained between short and long audio tests.

The same 2.0%/0.5% threshold logic is applied to long audio tests, which is consistent with the short audio test approach.

s3tokenizer/__init__.py (3)

28-28: LGTM! S3TokenizerV3 import added for V3 model support.


49-51: V3 model URL and registry entry added correctly.

The model URL points to the Fun-CosyVoice3-0.5B-2512 repository on ModelScope.


154-159: Model version routing logic is correct and follows existing pattern.

The routing checks for 'v3' first, then 'v2', with S3Tokenizer as the fallback for v1 models. This is consistent with the established pattern.

s3tokenizer/model_v3.py (3)

28-37: LGTM!

The dataclass configuration with immutable primitive defaults is clean and appropriate.


40-52: LGTM!

Overriding the linear projections to enable bias for V3 is a valid approach. The parent class creates these layers first, but they're immediately replaced here, which is functionally correct.


55-83: LGTM!

Clean residual attention block implementation with pre-norm architecture and proper residual connections.

Comment thread s3tokenizer/model_v3.py Resolved
Comment thread s3tokenizer/model_v3.py Resolved
Corrected the epsilon value for LayerNorm layers in S3TokenizerV3 (model_v3.py) from 1e-6 to 1e-5 to precisely match the value used in the exported ONNX model. This change significantly reduces numerical discrepancies and improves the token mismatch rate between the PyTorch and ONNX implementations of the V3 model.
The max_frames parameter was being passed to _quantize_mixed_batch but was
not being used, as the function internally calculated frames_per_window.
This commit removes the unused parameter from both the function signature
and its call site, improving code clarity.
The computation of 'cos' and 'sin' tensors from 'freqs_cis' in AudioEncoderV3's forward method was identified as dead code, as these computed tensors were never subsequently used. Only the original 'freqs_cis' was passed to the attention blocks. This commit removes the unused computation for improved code clarity and maintainability.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
s3tokenizer/model_v3.py (2)

152-152: Consider avoiding mutable default argument.

While dataclasses are generally safe here since they're not mutated, using None as default and creating the instance inside the function is the conventional pattern.

-    def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()):
+    def __init__(self, name: str, config: Optional[ModelConfigV3] = None):
         super().__init__()
         self.name = name
-        self.config = config
+        self.config = config if config is not None else ModelConfigV3()

329-332: Unnecessary wrapper method.

This load_state_dict override just calls super() with the same arguments. The comment hints at future plans, but as-is it adds no functionality.

Either remove the override entirely, or if you anticipate needing custom logic, add a TODO comment:

     def load_state_dict(self, state_dict, strict=True):
-        # Allow loading state dict with missing keys (like LN bias) if strictly necessary
-        # But for now we try to stick to standard behavior
+        # TODO: Add custom handling for missing keys (e.g., LN bias) if needed
         return super().load_state_dict(state_dict, strict=strict)
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 68390f8 and ed67972.

📒 Files selected for processing (1)
  • s3tokenizer/model_v3.py (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
s3tokenizer/model_v3.py (3)
s3tokenizer/model.py (9)
  • Conv1d (65-70)
  • LayerNorm (43-52)
  • Linear (55-62)
  • forward (45-52)
  • forward (57-62)
  • forward (96-106)
  • forward (158-165)
  • forward (199-222)
  • forward (329-330)
s3tokenizer/model_v2.py (6)
  • FSMNMultiHeadAttention (152-249)
  • FSQVectorQuantization (120-149)
  • forward (237-249)
  • forward (275-287)
  • forward (320-354)
  • forward (384-386)
s3tokenizer/utils.py (4)
  • make_non_pad_mask (435-472)
  • mask_to_bias (475-508)
  • merge_tokenized_segments (532-556)
  • onnx2torch_v3 (177-339)
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py

152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (5)
s3tokenizer/model_v3.py (5)

1-26: LGTM!

Clean imports and well-organized dependencies. Good approach to re-use V2 components where applicable.


28-37: LGTM!

Configuration dataclass is well-structured with sensible defaults. The n_codebook_size = 3**8 is a nice way to express the FSQ codebook dimensionality.


40-83: LGTM!

Clean implementation of V3-specific attention and residual blocks. The explicit bias=True override correctly differentiates V3 from the parent class, and the residual connections are properly structured.


188-314: LGTM!

The mixed-batch quantization logic correctly handles both short and long audio segments with proper overlap merging. The sliding window approach with configurable overlap is well-implemented.


320-327: LGTM!

The ONNX and checkpoint loading utilities are well-implemented. The mmap=True parameter for efficient memory-mapped loading is a nice touch for large checkpoints.

Comment thread s3tokenizer/model_v3.py Resolved
Corrected the calculation of x_slen (sequence length for padding mask) after the second convolutional layer (conv2) in AudioEncoderV3. The previous calculation incorrectly used self.stride (conv1's stride) instead of conv2's hardcoded stride of 2. This fix ensures accurate mask generation for downstream blocks.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
s3tokenizer/model_v3.py (1)

152-152: Prefer None as default and instantiate inside the function.

Using a function call ModelConfigV3() as a default argument is evaluated once at function definition time, which can lead to unexpected behavior if the config object is mutated. While this is safe for immutable dataclass defaults, it's better practice to use the standard pattern.

As per static analysis, this is flagged by Ruff B008.

Apply this diff:

-    def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()):
+    def __init__(self, name: str, config: Optional[ModelConfigV3] = None):
         super().__init__()
         self.name = name
-        self.config = config
+        self.config = config if config is not None else ModelConfigV3()
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ed67972 and 88b491d.

📒 Files selected for processing (2)
  • s3tokenizer/model_v3.py (1 hunks)
  • test/test_batch_efficiency.py (5 hunks)
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py

152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (9)
test/test_batch_efficiency.py (2)

8-11: LGTM!

The import reordering has no functional impact and is acceptable.


53-53: LGTM!

Adding the V3 model variant to all test parametrizations ensures comprehensive test coverage for the new model. The changes are consistent and aligned with the PR objectives.

Also applies to: 168-168, 218-218, 234-234

s3tokenizer/model_v3.py (7)

28-38: LGTM!

The ModelConfigV3 dataclass provides sensible defaults for the V3 architecture with appropriate documentation.


40-53: LGTM!

The V3 attention module correctly overrides the linear layers to enable bias (PyTorch's default), which is the intended difference from V2.


55-84: LGTM!

The residual attention block follows standard transformer architecture with proper residual connections, masking support, and rotary position embeddings.


86-144: LGTM!

The audio encoder properly implements strided convolutions with correct length calculations, masking, and applies the transformer blocks with rotary position embeddings. Past review concerns about stride consistency have been addressed.


173-186: LGTM!

The quantize method correctly routes long audio (>3000 frames) to the mixed-batch handler and processes standard audio through the encoder directly.


188-314: LGTM!

The mixed-batch quantization method correctly segments long audio into overlapping windows, processes them efficiently in a unified batch, and merges the results. Past review concerns about the unused max_frames parameter have been addressed.


316-336: LGTM!

The utility methods provide appropriate functionality for device introspection, weight loading from different formats (ONNX with flexible matching, PyTorch with strict matching), and parameter freezing.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
s3tokenizer/model_v3.py (2)

152-167: Avoid mutable default argument in function signature.

The static analysis correctly flags ModelConfigV3() as a function call in argument defaults. While dataclasses are relatively safe here, the idiomatic pattern is to use None and create the default inside the function.

-    def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()):
+    def __init__(self, name: str, config: Optional[ModelConfigV3] = None):
         super().__init__()
         self.name = name
-        self.config = config
+        self.config = config if config is not None else ModelConfigV3()

292-295: Consider parameterizing token_rate for future model variants.

The token_rate is hardcoded to 25, which is correct for V3 (25hz). However, if future V3 variants use different rates, this would need modification.

+                # V3 uses 25hz token rate
                 token_rate = 25

A comment clarifying this assumption would improve maintainability.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 88b491d and f36dc21.

📒 Files selected for processing (2)
  • s3tokenizer/model_v3.py (1 hunks)
  • test/test_onnx.py (3 hunks)
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py

152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: unit-test (ubuntu-22.04, 3.10.16)
🔇 Additional comments (12)
test/test_onnx.py (3)

7-7: LGTM!

Import reordering is purely cosmetic with no semantic change.


272-275: LGTM!

Adding the V3 model variant to the short audio test coverage is appropriate. The token_rate logic at lines 177-180 correctly defaults to 25 for non-v1 models, which aligns with the "25hz" in the model name.


320-323: LGTM!

V3 model added to long audio test coverage. The existing test infrastructure handles the V3 variant correctly.

s3tokenizer/model_v3.py (9)

1-26: LGTM!

License header and imports are well-organized. The reuse of V2 components where appropriate follows good DRY principles.


28-38: LGTM!

Configuration dataclass with sensible defaults for the V3 model architecture.


40-52: LGTM!

V3 attention correctly configures bias for Linear layers, with key having bias=False and others using the default bias=True. This aligns with the PR objective of bias detection for correct V3 conversion.


55-83: LGTM!

Pre-norm residual attention block with proper residual connections. The MLP uses standard 4x expansion factor.


86-143: LGTM!

Audio encoder with two-stage convolutions and residual attention blocks. The stride calculations at lines 132-133 correctly use the hardcoded stride of 2 matching conv2's configuration.


169-186: LGTM!

The quantization flow correctly separates short and long audio processing paths with the 3000-frame (30s) threshold.


316-318: LGTM!

Standard pattern for getting the device from module parameters.


320-323: Note: strict=False may mask loading issues.

Using strict=False when loading the ONNX-converted state dict could silently ignore missing or unexpected keys. Consider logging mismatches for debugging purposes.

     def init_from_onnx(self, onnx_path: str):
         ckpt = onnx2torch_v3(onnx_path, None,
                              False)  # Set verbose back to False
-        self.load_state_dict(ckpt, strict=False)
+        missing, unexpected = super().load_state_dict(ckpt, strict=False)
+        if missing or unexpected:
+            import logging
+            logging.warning(f"ONNX load: missing={missing}, unexpected={unexpected}")

329-336: LGTM!

The freeze method correctly disables gradient computation for all parameters. The load_state_dict override is passthrough but documents the intent for potential future customization.

Comment thread s3tokenizer/model_v3.py
Comment on lines +325 to +327
def init_from_pt(self, ckpt_path: str):
ckpt = torch.load(ckpt_path, map_location="cpu", mmap=True)
self.load_state_dict(ckpt, strict=True)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

torch.load weights_only parameter PyTorch 2.0 security

💡 Result:

Short answer: In April 2025 a critical RCE vulnerability (CVE-2025-32434) was reported in torch.load when called with weights_only=True for affected PyTorch releases (<= 2.5.1). The bug allowed crafted legacy-format checkpoint files to bypass the weights-only restrictions and execute arbitrary code. The issue was fixed in PyTorch 2.6.0; recommended mitigations are to upgrade to a patched PyTorch, never load untrusted .pt/.pth files, prefer saving/loading state_dicts or using safetensors, and run model-loading inside an isolated environment when needed. [1][2][3][4]

Sources

  • PyTorch GitHub advisory (GHSA) / fix and affected/patched versions. [1]
  • CVE/technical writeups describing legacy-format bypass and RCE (CVE-2025-32434). [2][3]
  • PyTorch serialization docs and guidance to use weights_only, safetensors, or allowlisting. [4]

References
[1] PyTorch advisory (GHSA) for torch.load weights_only RCE. (github.com)
[2] CVE summary / vulnerability databases reporting CVE-2025-32434. (cve.news)
[3] Press/analysis writeups on the RCE and exploit vector. (cyberpress.org)
[4] PyTorch serialization docs and safe-loading guidance (weights_only, safe_globals, safetensors recommendation). (docs.pytorch.org)

If you want, I can (a) check the exact PyTorch wheel version you have installed and tell you whether it's vulnerable, or (b) give example commands to upgrade, convert checkpoints to safetensors, or inspect a checkpoint safely.

Citations:


🏁 Script executed:

# Check the actual code in the repository
find . -name "model_v3.py" -type f

Repository: xingchensong/S3Tokenizer

Length of output: 93


🏁 Script executed:

# Read the actual code at the specified lines
sed -n '320,335p' ./s3tokenizer/model_v3.py

# Check for PyTorch version requirements
find . -name "requirements.txt" -o -name "setup.py" -o -name "pyproject.toml" -o -name "environment.yml" | head -5

Repository: xingchensong/S3Tokenizer

Length of output: 808


🏁 Script executed:

# Check PyTorch version requirements
cat ./requirements.txt ./setup.py 2>/dev/null | grep -i torch

# Also check pyproject.toml if it exists
if [ -f ./pyproject.toml ]; then cat ./pyproject.toml | grep -i torch; fi

Repository: xingchensong/S3Tokenizer

Length of output: 84


Security: torch.load requires comprehensive protection beyond weights_only.

torch.load() without protections can execute arbitrary code. While weights_only=True is recommended in PyTorch documentation, a critical RCE vulnerability (CVE-2025-32434) exists in PyTorch versions ≤ 2.5.1 that bypasses weights_only restrictions using legacy-format checkpoints. Since your requirements do not pin PyTorch version, this project is at risk if any version ≤ 2.5.1 is installed.

Recommended actions: Either (1) upgrade PyTorch to 2.6.0+ and add weights_only=True, (2) load only state_dict() files instead of full checkpoints, (3) adopt safetensors format, or (4) pin PyTorch version to 2.6.0+.

🤖 Prompt for AI Agents
In s3tokenizer/model_v3.py around lines 325-327, using torch.load(ckpt_path,
map_location="cpu", mmap=True) is unsafe for PyTorch ≤2.5.1 because
legacy-format checkpoints can execute code; change to a safe loading strategy:
either (A) require and assert torch.__version__ >= "2.6.0" and call
torch.load(ckpt_path, map_location="cpu", weights_only=True) (and fail loudly if
version is older), or (B) accept only state_dict files and load via
torch.load(..., weights_only=True) into a dict then call
self.load_state_dict(state_dict, strict=True), or (C) switch to safetensors
format and use its safe loader, and also update/pin project dependencies to
PyTorch >=2.6.0 in requirements to ensure the protection is present.

@Mddct

Mddct commented Dec 15, 2025

Copy link
Copy Markdown
Collaborator Author

============================= test session starts ==============================
platform darwin -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/mddct/Downloads/ckpt/cosyvoice3/S3Tokenizer
plugins: anyio-4.11.0, hydra-core-1.3.2
collected 8 items

S3Tokenizer/test/test_onnx.py
=== Testing speech_tokenizer_v1 on Short Audio ===
✅ short_audio_1: 5.0s → torch:250, onnx:250
✅ short_audio_2: 15.0s → torch:750, onnx:750
✅ medium_audio: 25.0s → torch:1250, onnx:1250
✅ medium_audio_2: 30.0s → torch:1500, onnx:1500

speech_tokenizer_v1 Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v1_25hz on Short Audio ===
✅ short_audio_1: 5.0s → torch:125, onnx:125
✅ short_audio_2: 15.0s → torch:375, onnx:375
✅ medium_audio: 25.0s → torch:625, onnx:625
✅ medium_audio_2: 30.0s → torch:750, onnx:750

speech_tokenizer_v1_25hz Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v2_25hz on Short Audio ===
✅ short_audio_1: 5.0s → torch:125, onnx:125
✅ short_audio_2: 15.0s → torch:375, onnx:375
✅ medium_audio: 25.0s → torch:625, onnx:625
✅ medium_audio_2: 30.0s → torch:750, onnx:750

speech_tokenizer_v2_25hz Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v3_25hz on Short Audio ===
✅ short_audio_1: 5.0s → torch:125, onnx:125
✅ short_audio_2: 15.0s → torch:375, onnx:375
✅ medium_audio: 25.0s → torch:625, onnx:625
✅ medium_audio_2: 30.0s → torch:750, onnx:750

speech_tokenizer_v3_25hz Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v1 on Long Audio (ONNX Sliding Window) ===
✅ long_audio: 35.0s → torch:1300, onnx:1300
✅ long_audio_2: 56.0s → torch:2600, onnx:2600
✅ very_long_audio: 60.0s → torch:2600, onnx:2600

speech_tokenizer_v1 Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio
.
=== Testing speech_tokenizer_v1_25hz on Long Audio (ONNX Sliding Window) ===
✅ long_audio: 35.0s → torch:650, onnx:650
✅ long_audio_2: 56.0s → torch:1300, onnx:1300
✅ very_long_audio: 60.0s → torch:1300, onnx:1300

speech_tokenizer_v1_25hz Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio
.
=== Testing speech_tokenizer_v2_25hz on Long Audio (ONNX Sliding Window) ===
✅ long_audio: 35.0s → torch:650, onnx:650
❌ long_audio_2: 56.0s → torch:1300, onnx:1300
Miss rate: 0.08%
torch_tokens:
tensor([3361, 6466, 4255, ..., 6442, 6442, 4009])
onnx_tokens:
tensor([3361, 6466, 4255, ..., 6442, 6442, 4009])
❌ very_long_audio: 60.0s → torch:1300, onnx:1300
Miss rate: 0.08%
torch_tokens:
tensor([3361, 4252, 4255, ..., 4036, 4252, 4255])
onnx_tokens:
tensor([3361, 4252, 4255, ..., 4036, 4252, 4255])

speech_tokenizer_v2_25hz Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio
.
=== Testing speech_tokenizer_v3_25hz on Long Audio (ONNX Sliding Window) ===
❌ long_audio: 35.0s → torch:650, onnx:650
Miss rate: 0.15%
torch_tokens:
tensor([ 259, 4276, 4276, 6463, 6463, 4276, 6463, 6463, 6463, 6463, 6382, 6463,
6463, 6463, 6463, 6463, 4276, 4276, 6463, 6463, 4276, 4249, 3520, 4249,
4276, 6463, 6463, 4276, 4249, 4249, 4249, 6463, 6463, 6463, 4276, 4276,
4249, 4276, 3520, 6463, 4249, 6463, 6463, 6463, 3520, 6436, 4276, 6463,
4276, 6436, 6463, 4249, 4249, 4249, 4249, 6463, 4276, 6436, 6436, 4276,
4249, 6463, 6463, 6436, 3547, 5707, 6436, 4276, 4249, 4249, 4249, 4249,
3520, 6463, 6463, 6463, 6463, 4276, 6436, 6436, 6436, 4249, 4276, 4249,
3520, 3520, 4249, 6436, 6463, 4249, 4249, 6463, 3520, 6463, 6463, 6436,
5707, 3520, 5707, 5707, 4249, 5707, 5707, 3547, 3547, 3547, 5734, 6436,
3520, 6463, 6463, 3520, 5707, 3520, 5707, 5707, 3520, 3520, 5734, 3520,
6463, 3520, 3511, 6436, 4249, 3547, 3520, 3520, 5707, 3520, 3520, 3520,
3520, 3520, 3547, 4249, 3520, 3520, 3520, 4249, 3520, 3511, 5707, 3439,
3520, 5707, 5707, 5734, 3520, 3520, 5626, 3520, 6463, 3277, 5707, 3520,
3520, 5707, 5707, 3520, 3520, 3520, 3520, 3520, 3511, 3520, 5707, 3520,
5707, 3439, 3520, 3520, 3520, 5626, 4249, 3439, 5626, 3520, 3520, 5707,
5626, 6355, 3439, 3520, 3520, 3520, 3520, 3520, 6463, 3277, 6463, 3439,
3520, 3430, 3520, 3520, 3520, 3511, 3520, 3520, 3520, 3520, 3520, 3277,
3439, 3196, 3520, 3439, 5707, 5734, 3520, 3520, 5707, 5707, 5707, 3520,
3277, 3520, 5626, 3520, 3520, 3439, 3511, 3520, 3520, 3520, 3520, 3466,
3439, 3520, 3520, 3520, 5626, 3520, 3520, 3439, 3520, 3520, 3520, 3520,
3520, 3520, 3439, 3520, 3277, 3520, 3439, 3520, 3520, 3439, 5707, 3277,
3520, 3277, 3520, 5626, 3520, 3520, 3520, 3268, 3439, 3439, 3520, 3439,
5707, 5707, 3520, 3439, 3520, 3511, 3520, 3511, 3519, 3277, 5707, 3277,
3439, 5626, 3439, 5626, 3277, 3547, 3511, 3520, 3196, 3223, 3439, 3511,
3277, 3511, 3439, 3511, 3511, 3520, 3520, 3439, 3439, 3520, 3520, 3511,
3520, 3277, 3439, 3277, 3196, 3277, 3520, 5707, 3277, 3520, 3439, 3520,
3520, 3277, 3276, 3520, 3520, 3511, 3196, 3439, 3520, 3520, 3439, 3277,
3520, 3439, 3520, 3520, 3439, 3196, 3520, 3439, 3439, 3520, 3268, 3439,
3439, 3439, 3520, 3511, 3439, 3511, 3430, 3439, 3520, 3520, 3520, 3277,
5626, 3268, 3520, 3196, 3520, 3268, 3268, 3277, 3520, 3520, 3511, 3277,
3277, 3277, 3268, 3277, 3520, 3511, 3520, 3187, 3520, 3187, 3277, 3277,
3268, 3268, 3511, 3520, 3520, 3277, 3268, 3268, 3277, 3268, 2539, 3511,
3187, 3268, 3268, 3268, 5698, 3268, 3196, 3511, 3439, 3511, 3187, 3277,
3520, 2539, 3196, 3277, 3438, 3520, 3520, 3196, 3277, 3268, 3439, 3268,
3520, 3277, 3187, 3277, 3268, 3520, 3520, 3511, 3511, 3511, 3267, 3520,
3520, 3511, 3439, 3277, 3511, 3511, 3439, 3439, 3511, 3277, 3520, 3277,
3430, 3268, 3196, 3511, 3511, 3520, 3268, 3277, 3187, 3268, 3439, 3277,
3511, 3196, 3511, 3520, 3268, 3511, 3268, 3268, 3187, 3268, 3520, 3511,
3511, 3277, 3268, 3439, 2539, 3268, 3268, 3430, 3520, 3511, 2539, 3268,
3511, 3268, 3277, 3268, 3196, 2539, 3187, 2539, 3196, 3268, 3187, 3430,
3196, 3196, 2458, 3430, 2512, 2782, 2782, 2539, 3187, 2539, 3268, 3196,
3520, 2782, 2458, 3511, 2782, 3268, 2539, 2701, 3511, 3277, 3277, 3511,
2539, 2539, 3511, 3277, 3268, 3268, 2539, 3187, 3511, 2539, 2539, 2539,
2755, 2782, 3511, 2539, 2458, 3511, 2539, 2539, 2782, 2539, 2539, 2782,
2539, 2539, 2539, 2539, 2539, 2782, 2458, 2539, 2539, 2539, 2539, 3268,
2458, 2539, 2458, 2539, 2539, 2701, 2791, 2458, 2539, 2458, 2539, 2539,
2539, 3511, 2539, 2539, 3268, 2458, 3196, 3430, 3268, 2539, 2782, 2539,
2782, 2782, 3268, 2539, 3520, 2782, 2701, 3511, 2539, 3511, 2701, 2782,
2782, 2539, 2458, 3196, 2539, 3439, 2458, 3268, 2512, 2539, 2755, 2539,
2701, 2539, 2782, 2782, 2539, 2539, 2539, 2548, 2458, 3268, 2539, 2701,
325, 3268, 2782, 2539, 2782, 2539, 2539, 2458, 2458, 2539, 3268, 2539,
2539, 2539, 2539, 2539, 2539, 2539, 2458, 2539, 2458, 2539, 2539, 2539,
2512, 2782, 2539, 2512, 2548, 2782, 2782, 2539, 2458, 2539, 2539, 2539,
2539, 2512, 2458, 2539, 2539, 2539, 2512, 2539, 2512, 2701, 2458, 2539,
2539, 2539])
onnx_tokens:
tensor([ 259, 4276, 4276, 6463, 6463, 4276, 6463, 6463, 6463, 6463, 6382, 6463,
6463, 6463, 6463, 6463, 4276, 4276, 6463, 6463, 4276, 4249, 3520, 4249,
4276, 6463, 6463, 4276, 4249, 4249, 4249, 6463, 6463, 6463, 4276, 4276,
4249, 4276, 3520, 6463, 4249, 6463, 6463, 6463, 3520, 6436, 4276, 6463,
4276, 6436, 6463, 4249, 4249, 4249, 4249, 6463, 4276, 6436, 6436, 4276,
4249, 6463, 6463, 6436, 3547, 5707, 6436, 4276, 4249, 4249, 4249, 4249,
3520, 6463, 6463, 6463, 6463, 4276, 6436, 6436, 6436, 4249, 4276, 4249,
3520, 3520, 4249, 6436, 6463, 4249, 4249, 6463, 3520, 6463, 6463, 6436,
5707, 3520, 5707, 5707, 4249, 5707, 5707, 3547, 3547, 3547, 5734, 6436,
3520, 6463, 6463, 3520, 5707, 3520, 5707, 5707, 3520, 3520, 5734, 3520,
6463, 3520, 3511, 6436, 4249, 3547, 3520, 3520, 5707, 3520, 3520, 3520,
3520, 3520, 3547, 4249, 3520, 3520, 3520, 4249, 3520, 3511, 5707, 3439,
3520, 5707, 5707, 5734, 3520, 3520, 5626, 3520, 6463, 3277, 5707, 3520,
3520, 5707, 5707, 3520, 3520, 3520, 3520, 3520, 3511, 3520, 5707, 3520,
5707, 3439, 3520, 3520, 3520, 5626, 4249, 3439, 5626, 3520, 3520, 5707,
5626, 6355, 3439, 3520, 3520, 3520, 3520, 3520, 6463, 3277, 6463, 3439,
3520, 3430, 3520, 3520, 3520, 3511, 3520, 3520, 3520, 3520, 3520, 3277,
3439, 3196, 3520, 3439, 5707, 5734, 3520, 3520, 5707, 5707, 5707, 3520,
3277, 3520, 5626, 3520, 3520, 3439, 3511, 3520, 3520, 3520, 3520, 3466,
3439, 3520, 3520, 3520, 5626, 3520, 3520, 3439, 3520, 3520, 5707, 3520,
3520, 3520, 3439, 3520, 3277, 3520, 3439, 3520, 3520, 3439, 5707, 3277,
3520, 3277, 3520, 5626, 3520, 3520, 3520, 3268, 3439, 3439, 3520, 3439,
5707, 5707, 3520, 3439, 3520, 3511, 3520, 3511, 3519, 3277, 5707, 3277,
3439, 5626, 3439, 5626, 3277, 3547, 3511, 3520, 3196, 3223, 3439, 3511,
3277, 3511, 3439, 3511, 3511, 3520, 3520, 3439, 3439, 3520, 3520, 3511,
3520, 3277, 3439, 3277, 3196, 3277, 3520, 5707, 3277, 3520, 3439, 3520,
3520, 3277, 3276, 3520, 3520, 3511, 3196, 3439, 3520, 3520, 3439, 3277,
3520, 3439, 3520, 3520, 3439, 3196, 3520, 3439, 3439, 3520, 3268, 3439,
3439, 3439, 3520, 3511, 3439, 3511, 3430, 3439, 3520, 3520, 3520, 3277,
5626, 3268, 3520, 3196, 3520, 3268, 3268, 3277, 3520, 3520, 3511, 3277,
3277, 3277, 3268, 3277, 3520, 3511, 3520, 3187, 3520, 3187, 3277, 3277,
3268, 3268, 3511, 3520, 3520, 3277, 3268, 3268, 3277, 3268, 2539, 3511,
3187, 3268, 3268, 3268, 5698, 3268, 3196, 3511, 3439, 3511, 3187, 3277,
3520, 2539, 3196, 3277, 3438, 3520, 3520, 3196, 3277, 3268, 3439, 3268,
3520, 3277, 3187, 3277, 3268, 3520, 3520, 3511, 3511, 3511, 3267, 3520,
3520, 3511, 3439, 3277, 3511, 3511, 3439, 3439, 3511, 3277, 3520, 3277,
3430, 3268, 3196, 3511, 3511, 3520, 3268, 3277, 3187, 3268, 3439, 3277,
3511, 3196, 3511, 3520, 3268, 3511, 3268, 3268, 3187, 3268, 3520, 3511,
3511, 3277, 3268, 3439, 2539, 3268, 3268, 3430, 3520, 3511, 2539, 3268,
3511, 3268, 3277, 3268, 3196, 2539, 3187, 2539, 3196, 3268, 3187, 3430,
3196, 3196, 2458, 3430, 2512, 2782, 2782, 2539, 3187, 2539, 3268, 3196,
3520, 2782, 2458, 3511, 2782, 3268, 2539, 2701, 3511, 3277, 3277, 3511,
2539, 2539, 3511, 3277, 3268, 3268, 2539, 3187, 3511, 2539, 2539, 2539,
2755, 2782, 3511, 2539, 2458, 3511, 2539, 2539, 2782, 2539, 2539, 2782,
2539, 2539, 2539, 2539, 2539, 2782, 2458, 2539, 2539, 2539, 2539, 3268,
2458, 2539, 2458, 2539, 2539, 2701, 2791, 2458, 2539, 2458, 2539, 2539,
2539, 3511, 2539, 2539, 3268, 2458, 3196, 3430, 3268, 2539, 2782, 2539,
2782, 2782, 3268, 2539, 3520, 2782, 2701, 3511, 2539, 3511, 2701, 2782,
2782, 2539, 2458, 3196, 2539, 3439, 2458, 3268, 2512, 2539, 2755, 2539,
2701, 2539, 2782, 2782, 2539, 2539, 2539, 2548, 2458, 3268, 2539, 2701,
325, 3268, 2782, 2539, 2782, 2539, 2539, 2458, 2458, 2539, 3268, 2539,
2539, 2539, 2539, 2539, 2539, 2539, 2458, 2539, 2458, 2539, 2539, 2539,
2512, 2782, 2539, 2512, 2548, 2782, 2782, 2539, 2458, 2539, 2539, 2539,
2539, 2512, 2458, 2539, 2539, 2539, 2512, 2539, 2512, 2701, 2458, 2539,
2539, 2539])
✅ long_audio_2: 56.0s → torch:1300, onnx:1300
✅ very_long_audio: 60.0s → torch:1300, onnx:1300

speech_tokenizer_v3_25hz Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
s3tokenizer/model_v3.py (1)

320-327: Harden init_from_pt against unsafe torch.load deserialization (duplicate of prior concern).

torch.load(ckpt_path, map_location="cpu", mmap=True) will execute arbitrary code embedded in a crafted checkpoint. Even with weights_only=True, PyTorch ≤ 2.5.1 has a known RCE issue; safest is to both (a) pin torch ≥ 2.6.0 in your dependencies and (b) load weights‑only state dicts.

Within this method, you can at least switch to weights_only=True and make it clear that a state_dict is expected:

     def init_from_pt(self, ckpt_path: str):
-        ckpt = torch.load(ckpt_path, map_location="cpu", mmap=True)
-        self.load_state_dict(ckpt, strict=True)
+        # Expect a state_dict checkpoint and load weights only.
+        ckpt = torch.load(
+            ckpt_path,
+            map_location="cpu",
+            weights_only=True,
+        )
+        self.load_state_dict(ckpt, strict=True)

And separately, update your requirements.txt / pyproject.toml to require torch>=2.6.0 so weights_only actually provides the intended protection.

To double‑check current PyTorch guidance and affected versions, please run a quick web search for the latest torch.load weights_only vulnerability and ensure your pinned torch version is in the patched range.

🧹 Nitpick comments (2)
s3tokenizer/model_v3.py (2)

152-163: Avoid calling ModelConfigV3() in the default argument (Ruff B008).

Using a constructed dataclass instance as a default value means a single shared ModelConfigV3 is reused across all instances, which is brittle if the config is ever mutated. It also triggers Ruff B008.

Consider deferring the construction into the body:

-    def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()):
-        super().__init__()
-        self.name = name
-        self.config = config
+    def __init__(self, name: str, config: Optional[ModelConfigV3] = None):
+        super().__init__()
+        self.name = name
+        self.config = config or ModelConfigV3()

173-314: Mixed‑batch quantization logic is sound; consider normalizing length types for clarity.

The windowing/overlap logic and use of merge_tokenized_segments look correct and consistent with a 30s / 25 Hz setup. To make the control flow more robust and remove subtle int/Tensor mixing, you can convert per‑sample lengths to Python ints once:

-        for batch_idx in range(batch_size):
-            audio_mel = mel[batch_idx]
-            audio_mel_len = mel_len[batch_idx]
-            is_long_audio = long_audio_mask[batch_idx].item()
+        for batch_idx in range(batch_size):
+            audio_mel = mel[batch_idx]
+            audio_mel_len = int(mel_len[batch_idx].item())
+            is_long_audio = bool(long_audio_mask[batch_idx].item())
@@
-            if not is_long_audio:
-                segment = audio_mel[:, :audio_mel_len]
-                seg_len = audio_mel_len.item()
+            if not is_long_audio:
+                segment = audio_mel[:, :audio_mel_len]
+                seg_len = audio_mel_len
@@
-            else:
-                start = 0
-                segment_idx = 0
-                while start < audio_mel_len:
-                    end = min(start + frames_per_window, audio_mel_len)
+            else:
+                start = 0
+                segment_idx = 0
+                while start < audio_mel_len:
+                    end = min(start + frames_per_window, audio_mel_len)

This keeps all indexing and loop conditions in plain Python ints, which is easier to reason about and avoids any future surprises if mel_len’s dtype or device handling changes.

Optionally, you might also derive max_frames in quantize from the same formula used for frames_per_window to make their coupling explicit (30 s @ 100 fps → 3000 frames).

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f36dc21 and 098e1b9.

📒 Files selected for processing (2)
  • s3tokenizer/model_v3.py (1 hunks)
  • test/test_onnx.py (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/test_onnx.py
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py

152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

🔇 Additional comments (1)
s3tokenizer/model_v3.py (1)

40-83: Attention + residual block wiring looks consistent with V2/V3 shape expectations.

Bias patterns on Q/K/V/Out and the LayerNorm+MLP residual structure match the usual transformer block layout and should align well with the V3 weights. No issues spotted here.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
s3tokenizer/model_v3.py (1)

325-327: Prior security concern about torch.load remains unaddressed.

A previous review flagged the security risk of using torch.load without weights_only=True due to CVE-2025-32434 affecting PyTorch ≤2.5.1. This allows arbitrary code execution when loading untrusted checkpoints. The issue has not been resolved.

🧹 Nitpick comments (1)
s3tokenizer/model_v3.py (1)

152-152: Avoid mutable default argument to prevent shared state.

Using ModelConfigV3() as a default argument creates a single shared instance across all calls. If any code mutates the config object, it affects all instances. Use None as the default and instantiate inside the method.

Apply this diff:

-    def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()):
+    def __init__(self, name: str, config: Optional[ModelConfigV3] = None):
         super().__init__()
         self.name = name
-        self.config = config
+        self.config = config if config is not None else ModelConfigV3()

Don't forget to import Optional from typing at the top if not already present.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 098e1b9 and 02e192f.

📒 Files selected for processing (3)
  • s3tokenizer/model_v2.py (3 hunks)
  • s3tokenizer/model_v3.py (1 hunks)
  • test/test_onnx.py (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/test_onnx.py
🧰 Additional context used
🧬 Code graph analysis (2)
s3tokenizer/model_v3.py (3)
s3tokenizer/model.py (13)
  • Conv1d (65-70)
  • LayerNorm (43-52)
  • Linear (55-62)
  • forward (45-52)
  • forward (57-62)
  • forward (96-106)
  • forward (158-165)
  • forward (199-222)
  • forward (329-330)
  • quantize (244-249)
  • quantize (333-360)
  • encode (261-269)
  • encode (295-298)
s3tokenizer/model_v2.py (9)
  • FSMNMultiHeadAttention (152-250)
  • FSQVectorQuantization (120-149)
  • forward (238-250)
  • forward (276-288)
  • forward (321-355)
  • forward (385-387)
  • quantize (390-418)
  • encode (97-112)
  • encode (142-143)
s3tokenizer/utils.py (5)
  • make_non_pad_mask (435-472)
  • mask_to_bias (475-508)
  • merge_tokenized_segments (532-556)
  • onnx2torch_v3 (177-339)
  • padding (511-529)
s3tokenizer/model_v2.py (1)
s3tokenizer/model.py (2)
  • Linear (55-62)
  • LayerNorm (43-52)
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py

152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

🔇 Additional comments (9)
s3tokenizer/model_v2.py (3)

58-59: LGTM! Dtype consistency improved.

The explicit cast to xq.dtype ensures rotary embeddings work correctly with mixed precision (e.g., bfloat16), preventing dtype mismatches during computation.


176-176: Remove redundant self.key definition; parent MultiHeadAttention already defines it with identical settings.

The parent MultiHeadAttention class defines self.key = Linear(n_state, n_state, bias=False) at line 90 of model.py. Since FSMNMultiHeadAttention inherits from it and does not change the bias setting, this override at line 176 is redundant and can be removed.


268-268: Verify LayerNorm epsilon value and its compatibility with existing V2 checkpoints.

The attn_ln LayerNorm uses eps=1e-5 while mlp_ln uses the default. Confirm this difference is intentional and validate compatibility with any existing V2 checkpoint versions, particularly if they were created with different epsilon values.

s3tokenizer/model_v3.py (6)

28-38: LGTM!

The V3 configuration dataclass is well-structured with appropriate defaults for the 12-layer V3 architecture.


55-84: LGTM!

The residual attention block is correctly structured with proper pre-normalization and residual connections.


86-144: LGTM!

The encoder correctly applies the two-stage CNN front-end with proper length tracking and passes the appropriate rotary frequency components to the attention blocks.


146-187: LGTM!

The quantization method correctly handles both short and long audio with an appropriate 30-second threshold (3000 frames).


188-314: LGTM!

The mixed-batch quantization correctly implements sliding-window processing for long audio with appropriate overlap handling and token merging at 25Hz rate.


334-336: LGTM!

The freeze method correctly disables gradient computation for all parameters.

Comment thread s3tokenizer/model_v3.py Resolved
@Mddct

Mddct commented Dec 15, 2025

Copy link
Copy Markdown
Collaborator Author

============================= test session starts ==============================
platform darwin -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/mddct/Downloads/ckpt/cosyvoice3/S3Tokenizer
plugins: anyio-4.11.0, hydra-core-1.3.2
collected 8 items

S3Tokenizer/test/test_onnx.py
=== Testing speech_tokenizer_v1 on Short Audio ===
✅ short_audio_1: 5.0s → torch:250, onnx:250
✅ short_audio_2: 15.0s → torch:750, onnx:750
✅ medium_audio: 25.0s → torch:1250, onnx:1250
✅ medium_audio_2: 30.0s → torch:1500, onnx:1500

speech_tokenizer_v1 Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v1_25hz on Short Audio ===
✅ short_audio_1: 5.0s → torch:125, onnx:125
✅ short_audio_2: 15.0s → torch:375, onnx:375
✅ medium_audio: 25.0s → torch:625, onnx:625
✅ medium_audio_2: 30.0s → torch:750, onnx:750

speech_tokenizer_v1_25hz Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v2_25hz on Short Audio ===
✅ short_audio_1: 5.0s → torch:125, onnx:125
✅ short_audio_2: 15.0s → torch:375, onnx:375
✅ medium_audio: 25.0s → torch:625, onnx:625
✅ medium_audio_2: 30.0s → torch:750, onnx:750

speech_tokenizer_v2_25hz Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v3_25hz on Short Audio ===
✅ short_audio_1: 5.0s → torch:125, onnx:125
✅ short_audio_2: 15.0s → torch:375, onnx:375
✅ medium_audio: 25.0s → torch:625, onnx:625
✅ medium_audio_2: 30.0s → torch:750, onnx:750

speech_tokenizer_v3_25hz Short Audio Summary:
Successful tests: 4/4
.
=== Testing speech_tokenizer_v1 on Long Audio (ONNX Sliding Window) ===
✅ long_audio: 35.0s → torch:1300, onnx:1300
✅ long_audio_2: 56.0s → torch:2600, onnx:2600
✅ very_long_audio: 60.0s → torch:2600, onnx:2600

speech_tokenizer_v1 Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio
.
=== Testing speech_tokenizer_v1_25hz on Long Audio (ONNX Sliding Window) ===
✅ long_audio: 35.0s → torch:650, onnx:650
✅ long_audio_2: 56.0s → torch:1300, onnx:1300
✅ very_long_audio: 60.0s → torch:1300, onnx:1300

speech_tokenizer_v1_25hz Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio
.
=== Testing speech_tokenizer_v2_25hz on Long Audio (ONNX Sliding Window) ===
✅ long_audio: 35.0s → torch:650, onnx:650
✅ long_audio_2: 56.0s → torch:1300, onnx:1300
✅ very_long_audio: 60.0s → torch:1300, onnx:1300

speech_tokenizer_v2_25hz Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio
.
=== Testing speech_tokenizer_v3_25hz on Long Audio (ONNX Sliding Window) ===
✅ long_audio: 35.0s → torch:650, onnx:650
✅ long_audio_2: 56.0s → torch:1300, onnx:1300
✅ very_long_audio: 60.0s → torch:1300, onnx:1300

speech_tokenizer_v3_25hz Long Audio Results:
Total tests: 3
Successful ONNX tests: 3
✅ Torch processing works reliably for all long audio
.

======================== 8 passed in 149.84s (0:02:29) =========================

…d align Eps

Detailed analysis revealed that the Key projection layer in the ONNX model (both V2 and V3) does not have a bias term (unlike Query and Value). This commit sets 'bias=False' for the Key Linear layer in both model_v2.py and model_v3.py, correcting a significant source of numerical divergence.

Additionally, aligned LayerNorm epsilon to 1e-5 for both V2 and V3 models to match ONNX constants.

Findings & Precautions:
1.  **Key Bias**: The ONNX model structure for 'speech_tokenizer_v3' and 'v2' has an asymmetric bias configuration. Query and Value projections have bias terms, but the Key projection does NOT. PyTorch's default 'Linear' includes bias, which caused substantial output mismatch.
2.  **LayerNorm Epsilon**: The V2/V3 ONNX models use 'eps=1e-5', differing from the default '1e-6'. This was corrected.
3.  **BF16 Support**: Fixed 'LayerNorm' and 'RotaryEmbedding' to support BFloat16 inference on CPU without mixed-precision errors.

Final Test Results (PyTorch vs ONNX, 0.5% Threshold):
- speech_tokenizer_v3_25hz:
  - Short Audio (All 4): ✅ 0.00% Miss Rate
  - Long Audio (All 3): ✅ 0.00% Miss Rate
- speech_tokenizer_v2_25hz:
  - Short Audio (All 4): ✅ 0.00% Miss Rate (Improved from 0.27%)
  - Long Audio (All 3): ✅ 0.00% Miss Rate

All tests passed with strict alignment.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
s3tokenizer/model_v3.py (1)

325-327: Security: torch.load requires protection (see existing comment).

This issue was previously flagged with detailed CVE information and mitigation strategies. Please address the security concern by either adding weights_only=True (requires PyTorch ≥2.6.0), switching to state_dict loading, or using safetensors format.

🧹 Nitpick comments (5)
s3tokenizer/model_v3.py (5)

152-152: Consider moving default instantiation inside the function.

While safe here (dataclasses create new instances), calling ModelConfigV3() in the default argument is evaluated at function definition time, which can be surprising and is flagged by linters.

Apply this diff:

-    def __init__(self, name: str, config: ModelConfigV3 = ModelConfigV3()):
+    def __init__(self, name: str, config: Optional[ModelConfigV3] = None):
         super().__init__()
         self.name = name
-        self.config = config
+        self.config = config if config is not None else ModelConfigV3()

178-178: Consider making max_frames threshold configurable.

The 3000-frame threshold for long audio detection is hardcoded. Consider adding it to ModelConfigV3 for easier tuning across different use cases.


195-198: Hardcoded audio parameters should match input assumptions.

The constants sample_rate=16000 and hop_length=160 must match how the mel spectrogram was computed upstream. Consider either:

  1. Adding these to ModelConfigV3 and validating they match the input
  2. Adding documentation specifying the expected mel spectrogram parameters

277-277: Consider optimizing the CPU conversion for performance.

The .cpu().numpy().tolist() conversion moves data off-device and creates Python lists. If merge_tokenized_segments can be refactored to accept tensors directly, this would avoid the device transfer overhead.


292-292: Document the hardcoded token rate.

The token_rate=25 corresponds to the 25Hz output mentioned in the PR title. Consider adding a comment or moving this to a named constant/config for clarity.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 02e192f and eebf924.

📒 Files selected for processing (3)
  • s3tokenizer/model_v2.py (3 hunks)
  • s3tokenizer/model_v3.py (1 hunks)
  • test/test_onnx.py (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/test_onnx.py
  • s3tokenizer/model_v2.py
🧰 Additional context used
🪛 Ruff (0.14.8)
s3tokenizer/model_v3.py

152-152: Do not perform function call ModelConfigV3 in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable

(B008)

🔇 Additional comments (4)
s3tokenizer/model_v3.py (4)

40-53: LGTM: V3 attention correctly configured with bias overrides.

The implementation properly inherits from FSMNMultiHeadAttention and overrides the linear projections with the correct bias settings for V3 (query/value/out with bias, key without).


55-84: LGTM: Residual attention block follows standard transformer architecture.

The implementation correctly applies pre-normalization (LayerNorm before attention/MLP) with residual connections, following modern transformer best practices.


86-144: LGTM: Encoder correctly implements strided convolutions and attention blocks.

The forward pass properly tracks sequence lengths through both convolution layers and applies masking consistently throughout the attention blocks.


316-318: LGTM: Utility methods follow standard patterns.

The device property, load_state_dict wrapper, and freeze method all follow PyTorch conventions correctly.

Also applies to: 329-336

@ylzz1997

Copy link
Copy Markdown

🐮🍺

@Mddct
Mddct merged commit b9b907b into main Dec 18, 2025
2 checks passed
@Mddct
Mddct deleted the dh-cosy3 branch December 18, 2025 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants