docs(tokenless): document compression trigger conditions and thresholds - #2600
docs(tokenless): document compression trigger conditions and thresholds#2600Forrest-ly wants to merge 2 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82232111ee
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| 1. Compression is not switched off. With `compression_enabled=false` or `TOKENLESS_COMPRESSION_ENABLED=0` the run becomes a dry-run: statistics are still calculated, but the original text is returned (see the previous section). | ||
| 2. The tool is not a content-retrieval tool. Read/Glob/Grep/LSP/NotebookRead and their aliases skip response compression so their content stays intact. | ||
| 3. The response reaches the minimum length. The shared response hook, OpenClaw, and Hermes skip responses shorter than 200 characters; Codex skips responses shorter than 500 characters. Length is counted in characters, not bytes. | ||
| 4. The content is valid JSON. Response compression only handles JSON; plain text passes through unchanged. The shared paths additionally skip skill-like text with YAML frontmatter. |
There was a problem hiding this comment.
For Codex responses of at least 500 characters, plain text does not pass through unchanged: adapters/tokenless/codex/scripts/compress-response wraps non-JSON text as {"stdout": raw_text} and runs both response compression and TOON encoding (lines 328–380), potentially appending the result once the original reaches 4,000 characters. Qualify this JSON requirement as applying only to the shared/OpenClaw/Hermes paths, or add the Codex exception, so Codex users are not told that large plain-text shell output bypasses compression.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch — adopted and fixed in dfe1cbe. Verified against the source: src/tokenless/adapters/tokenless/codex/scripts/compress-response computes is_json on the raw text and, when it is not JSON, wraps it as {"stdout": raw_text} before running both compress-response and compress-toon, so Codex plain-text responses that reach the 500-character minimum are compressed too (with the compressed content added to context only for responses of at least 4,000 characters).
Item 4 in both the en and zh user manuals now scopes the plain-text passthrough statement to the shared response hook, OpenClaw, and Hermes paths, calls out the Codex wrapping exception explicitly, and notes that OpenClaw/Hermes frameworks wrap shell output in JSON themselves (e.g. {"stdout": ...}), so that output is still compressed. The YAML-frontmatter sentence now also names the three paths that apply the skip (the Codex hook has no such skip).
![qoderai[bot]](https://avatars.githubusercontent.com/in/1396158?s=60&v=4){kind=small}
There was a problem hiding this comment.
本次仅发现 1 处文档行为描述与 Codex 特殊路径存在轻微偏差，已在英文用户手册对应位置留下建议性注释。未发现其他会影响压缩触发或阈值理解的具体问题。
🤖 Generated by Qoder • View workflow run
| 1. Compression is not switched off. With `compression_enabled=false` or `TOKENLESS_COMPRESSION_ENABLED=0` the run becomes a dry-run: statistics are still calculated, but the original text is returned (see the previous section). | ||
| 2. The tool is not a content-retrieval tool. Read/Glob/Grep/LSP/NotebookRead and their aliases skip response compression so their content stays intact. | ||
| 3. The response reaches the minimum length. The shared response hook, OpenClaw, and Hermes skip responses shorter than 200 characters; Codex skips responses shorter than 500 characters. Length is counted in characters, not bytes. | ||
| 4. The content is valid JSON. Response compression only handles JSON; plain text passes through unchanged. The shared paths additionally skip skill-like text with YAML frontmatter. |
There was a problem hiding this comment.
[P2] JSON 规则需明确 Codex 例外
第 4 条将“响应压缩只处理 JSON；纯文本直接透传”描述为通用行为，但 Codex 路径会先把纯文本封装为 {"stdout": ...} 再参与压缩。建议在此补充 Codex 例外或限定该规则仅适用于共享/OpenClaw/Hermes/独立 CLI 路径，避免 Codex 用户误解大体纯文本输出不会被压缩。
🤖 Generated by Qoder • Fix in Qoder
There was a problem hiding this comment.
Good catch — adopted and fixed in dfe1cbe. Verified against the source: src/tokenless/adapters/tokenless/codex/scripts/compress-response computes is_json on the raw text and, when it is not JSON, wraps it as {"stdout": raw_text} before running both compress-response and compress-toon, so Codex plain-text responses that reach the 500-character minimum are compressed too (with the compressed content added to context only for responses of at least 4,000 characters).
Item 4 in both the en and zh user manuals now scopes the plain-text passthrough statement to the shared response hook, OpenClaw, and Hermes paths, calls out the Codex wrapping exception explicitly, and notes that OpenClaw/Hermes frameworks wrap shell output in JSON themselves (e.g. {"stdout": ...}), so that output is still compressed. The YAML-frontmatter sentence now also names the three paths that apply the skip (the Codex hook has no such skip).
…ore/tokenless-doc-compress-rate-scenarios Bring in the user-manual anchor (#compression-trigger-conditions-and-thresholds) that measuring-savings references, so PR alibaba#2601 passes the website link check before PR alibaba#2600 is merged. Cross-PR dependency fix for CI.
Co-authored-by: multica-agent <github@multica.ai>
…eption
The Codex response hook wraps non-JSON text as {"stdout": ...} before
compression, so plain-text responses that reach the minimum length are
compressed there too. Scope the "plain text passes through unchanged"
statement to the shared response hook, OpenClaw, and Hermes paths, and
note that OpenClaw/Hermes frameworks wrap shell output in JSON
themselves. Also spell out that the YAML-frontmatter skip applies to
the shared response hook, OpenClaw, and Hermes (not Codex).
Co-authored-by: multica-agent <github@multica.ai>
dfe1cbe to
a5a9fa3
Compare
1 participant
改动说明
为 tokenless 用户文档补充「压缩触发阈值」说明（客户反馈：文档缺少对压缩何时触发、阈值多大的解释）。现有文档只在 CLI 参考和框架集成中零散提到阈值数字，没有集中说明压缩的触发条件；本次将其汇总为用户手册中的独立小节，并补充交叉引用。
变更内容
docs/user-guide/{zh,en}/token-saving/tokenless/user-manual.mdtool_categories.json及安全回退；skip_tools/shell_tools覆盖、TOON 独立触发判断、AgentScope 按模式选择阈值。docs/user-guide/{zh,en}/token-saving/tokenless/cli-reference.md：compress-response阈值说明处补充指向新小节的交叉引用。docs/user-guide/{zh,en}/token-saving/tokenless/framework-integration.md：「Adapter 处理规则 / Adapter processing rules」末尾补充指向新小节的交叉引用。所有数字均逐一对照当前代码实现核实：
common/hooks/compress_response_hook.py、common/hooks/compress_toon_hook.py（最小 200 字符、严格变小保护）、codex/scripts/compress-response（500 / 4,000 字符）、common/hooks/tool_categories.json与hook_utils.py（三层阈值）、openclaw/index.ts（200 字符、分类覆盖）、tokenless-schema/src/response_compressor.rs（CLI 默认 4096/32/8）。测试情况
测试范围与实际执行的命令（本次为纯文档变更，共 6 个 Markdown 文件，无代码改动）：
bash scripts/docs-lint.sh（CI Docs 门禁：中文命名规范 + en/zh 目录树镜像）— 通过：✓ Naming convention OK、✓ en/zh tree parity OKpython3 scripts/docs-link-check.py（CI 相对链接检查）— 通过：✓ All relative links resolve#压缩的触发条件与阈值、#compression-trigger-conditions-and-thresholds等，GitHub 锚点规则下成立）环境概要：Linux x86_64；Python 3.8；git/gh CLI。
结果汇总：2 个 CI 门禁脚本全部通过，0 失败；锚点核对通过；数字与源码一致。
未运行项及原因：
cargo check/cargo test：未运行 — 本次无任何 Rust/代码改动，仅文档。