docs(tokenless): document compression-rate scenarios and standard test load - #2601
docs(tokenless): document compression-rate scenarios and standard test load#2601Forrest-ly wants to merge 5 commits into
Conversation
![qoderai[bot]](https://avatars.githubusercontent.com/in/1396158?s=60&v=4){kind=small}
There was a problem hiding this comment.
- 英文文档的“Standard test load” 小节仅在括号中写明 “Linux only”，未明确说明该 workspace 不支持 macOS/Windows，易让跨平台用户误以为可以直接运行，建议补充平台约束说明。
- 中文文档对应小节虽提到“仅支持 Linux”，但同样未显式声明在 macOS/Windows 上不支持运行，建议与英文版对齐，追加明确的平台限制句。
🤖 Generated by Qoder • View workflow run
|
|
||
| Real session savings must additionally be multiplied by the share of tool payloads in total session tokens; see [Interpret the saving rate correctly](#interpret-the-saving-rate-correctly) above. | ||
|
|
||
| ## Standard test load |
There was a problem hiding this comment.
[P2] 标准测试负载路径缺少平台约束说明
“Standard test load” 小节仅在括号中标注 “Linux only”，但没有明确说明该 workspace 无法在 macOS/Windows 上运行。对只读文档的用户来说，可能误以为任何平台都可直接进入目录运行 cargo run/./run-benchmarks.sh，在不支持平台上必然失败。建议在括号中或紧随其后增加一句明确平台限制，例如 “Linux only; unsupported on macOS and Windows”。
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
已采纳。已将该小节的平台约束显式写为 "(a standalone Cargo workspace; Linux only — unsupported on macOS and Windows)"，避免读者误以为任意平台都可直接运行 cargo run / ./run-benchmarks.sh。见 commit ffdd307。
|
|
||
| 实际会话收益还要乘以工具 Payload 在会话总 Token 中的占比，见上文[正确解释节省率](#正确解释节省率)。 | ||
|
|
||
| ## 标准测试负载 |
There was a problem hiding this comment.
[P2] 标准测试负载中文描述缺少平台约束说明
“标准测试负载” 小节中仅在括号中写明 “仅支持 Linux”，但未指出该 workspace 在 macOS/Windows 上不支持运行。用户可能误以为任意平台都可进入目录执行 cargo run/./run-benchmarks.sh，在不支持平台上必然失败。建议与英文版保持一致，显式补充“不支持 macOS/Windows”的平台约束语句。
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
已采纳。中文版已同步修改为 "（独立 Cargo workspace；仅支持 Linux，不支持 macOS/Windows）"，与英文版保持一致。见 commit ffdd307。
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d5f09b8e97
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| | Function-calling schema (long descriptions), schema compression alone | ~47% | description truncation; `title`/`examples` and code blocks removed | | ||
| | Mixed workload (response + schema): response compression only | ~62% | responses dominate the savings | | ||
| | Mixed workload: schema + response stacked | ~65% | both payload types reduced | | ||
| | Mixed workload: full stack (schema + response + TOON) | ~63% | little room left for TOON after response compression | |
There was a problem hiding this comment.
This ~63% value comes from the benchmark’s ungated full_stack, which always TOON-encodes both compressed payloads (src/metrics.rs:355-358,387). The deployed compress-toon path instead keeps the original input whenever TOON does not reduce the estimated token count (tokenless-cli/src/main.rs:888-910), and the benchmark itself notes that this fixture therefore remains at the compressed-only rate of about 65%. Presenting the ungated value as the full-stack Tokenless result contradicts the size-guard behavior described below; either report the gated rate or label this as a synthetic ungated measurement.
AGENTS.md reference: AGENTS.md:L337-L343
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Accepted. Verified against the code and the benchmark output: full_stack in src/metrics.rs is ungated (measures 62.9% on the standard fixture), while the deployed compress-toon path applies a size guard (tokenless-cli/src/main.rs) that keeps the original input whenever TOON does not reduce the estimated token count. For the standard fixture TOON inflates after response/schema compression, so the deployed rate equals the compressed-only rate (schema_response, 64.7% ≈ ~65%). The table now reports the gated ~65% as the full-stack result and keeps the ungated ~63% benchmark value as an explicit note for traceability. The Chinese doc was updated the same way. See commit ffdd307.
|
|
||
| ```bash | ||
| cd src/tokenless/benchmark/l1-compressor | ||
| cargo run --release --bin compression_rate # add --json for machine-readable output |
There was a problem hiding this comment.
--json after Cargo's separator
Appending --json as instructed produces error: unexpected argument '--json' found because Cargo consumes options before the binary argument separator. Checked cargo run --help, which explicitly says binary flags must be passed after --; show cargo run --release --bin compression_rate -- --json so the documented machine-readable invocation works.
AGENTS.md reference: AGENTS.md:L337-L343
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Accepted. Reproduced: cargo run --release --bin compression_rate --json fails with error: unexpected argument '--json' found (cargo consumes options before the binary argument separator). The doc now shows cargo run --release --bin compression_rate -- --json, which was verified to build and emit valid JSON. The Chinese doc was updated as well. See commit ffdd307.
|
来自同一任务的另一个并行执行：我提交的重复 PR #2602 已关闭，以本 PR 为准跟进。 一个合并顺序提示供参考：本 PR 中新增的 |
…ore/tokenless-doc-compress-rate-scenarios Bring in the user-manual anchor (#compression-trigger-conditions-and-thresholds) that measuring-savings references, so PR alibaba#2601 passes the website link check before PR alibaba#2600 is merged. Cross-PR dependency fix for CI.
… --json usage - Make the Linux-only constraint of the standard test load explicit (unsupported on macOS/Windows) in EN and ZH docs. - Report the deployed (gated) full-stack rate ~65% instead of the benchmark's ungated full_stack ~63%: the compress-toon size guard keeps the original input when TOON does not reduce estimated tokens, which is what happens for the standard fixture. The ungated benchmark value is kept as a note for traceability. - Show -- --json in the compression_rate invocation; cargo consumes options before the binary argument separator, so the previously documented form errored with 'unexpected argument'.
…ompress-rate-scenarios # Conflicts: # docs/user-guide/en/token-saving/tokenless/framework-integration.md # docs/user-guide/zh/token-saving/tokenless/framework-integration.md
1 participant
改动说明
应客户反馈，在 tokenless 用户文档中补充「压缩率适用场景」说明，并指引使用仓库内置的标准测试负载，使用户能够：1) 了解不同场景下的预期压缩率区间与影响因素；2) 用标准负载复现参考压缩率，自行验证所用版本的效果。
变更内容
docs/user-guide/{zh,en}/token-saving/tokenless/measuring-savings.mdsrc/tokenless/benchmark/l1-compressor/fixtures/下 3 个 canonical fixtures，由python/gen_fixtures.py生成、无随机数、字节级可复现、已提交）；cargo run --release --bin compression_rate（快速报告）与./run-benchmarks.sh --quick（完整质量/对抗测试 + 报告），均已在本地实际执行验证；测试情况
测试范围与实际执行的命令（本次为纯文档变更，共 2 个 Markdown 文件；为保证文档中的数字与操作步骤真实有效，实际运行了标准测试负载全流程）：
cargo build --release --bin compression_rate（l1-compressor 独立 workspace）— 成功cargo run --release --bin compression_rate— 输出与文档参考节省率一致：canonical response 65.8%、canonical schema 47.3%、混合负载 response_only 61.7%、schema_response 64.7%、full_stack 62.9%、toon_only 15.8%./run-benchmarks.sh --quick（完整质量/对抗测试 + 压缩率报告）— 全部通过，退出码 0cargo test --release— 96 passed、0 failed（13 个测试套件，含质量保留、对抗、worst-case、压缩率回归守护）bash scripts/docs-lint.sh（CI 文档门禁：命名规范 + en/zh 目录树镜像）— 通过python3 scripts/docs-link-check.py（相对链接检查）— 通过common/hooks/compress_response_hook.py、compress_toon_hook.py（最小 200 字符）、codex/scripts/compress-response（500/4,000 字符）、tool_categories.json（Layer 2/3 阈值）、benchmark fixtures 与gen_fixtures.py环境概要：Linux x86_64；Rust 1.94.1（cargo 1.94.1）；Python 3.8.17；rtk 0.43.0（报告中 RTK 采样正常输出）。
结果汇总：96 个测试全部通过，0 失败；文档门禁脚本全部通过；文档参考节省率与实测值完全一致。
未运行项及原因：
cargo bench）：未运行 — 本次不涉及性能结论，--quick模式按设计跳过该部分。