docs(tokenless): document compression-rate scenarios and standard test load

docs/user-guide/en/token-saving/tokenless/cli-reference.md

@@ -125,7 +125,7 @@ Field matching and truncation change the response representation seen by the mod
 
 Stash applies only to truncation of strings, array tails, and deep subtrees. Blacklisted fields, `null`, and empty values are removed without a retrieval marker.
 
-Most adapters override these standalone defaults. Their shared shell profile uses `65536`, `128`, and `8`; the other-structured-tool profile uses `1048576`, `65536`, and `32`. Content-retrieval tools are skipped. See [Framework integration · Adapter processing rules](framework-integration.md#adapter-processing-rules).
+Most adapters override these standalone defaults. Their shared shell profile uses `65536`, `128`, and `8`; the other-structured-tool profile uses `1048576`, `65536`, and `32`. Content-retrieval tools are skipped. See [Framework integration · Adapter processing rules](framework-integration.md#adapter-processing-rules) and [User manual · Compression trigger conditions and thresholds](user-manual.md#compression-trigger-conditions-and-thresholds).
 
 ## `compress-toon` and `decompress-toon`
 

docs/user-guide/en/token-saving/tokenless/framework-integration.md

@@ -43,6 +43,8 @@ The shared response hook, OpenClaw, and Hermes skip inputs shorter than 200 char
 
 Claude Code requires version 2.1.121 or later for `updatedToolOutput`. On older or unknown versions, response compression is disabled to avoid duplicating the original. Structured tool outputs preserve their host schema and do not switch to textual TOON; JSON carried as a string can use TOON when it is smaller.
 
+For the full trigger conditions (compression switch, minimum response length, JSON validation, strictly-smaller guard) and threshold semantics, see [User manual · Compression trigger conditions and thresholds](user-manual.md#compression-trigger-conditions-and-thresholds).
+
 ### DeepSeek Harness native processing
 
 The DSH bundle requires Node.js 22 or later and a compatible DSH profile. Pass

docs/user-guide/en/token-saving/tokenless/measuring-savings.md

@@ -173,6 +173,59 @@ Estimated overall saving rate
 
 For example, a 60% payload compression rate with tool payloads representing 20% of the session gives an estimated overall saving of about 12%. This is still not a provider billing guarantee.
 
+## Where the compression rate applies
+
+The Tokenless compression rate depends on how much removable material a payload contains, and varies widely by scenario. Reference values measured with the standard test load below (measured at commit `2e7d69f1`; re-run locally after upgrading):
+
+| Scenario | Reference savings (estimated tokens) | Notes |
+|----------|--------------------------------------|-------|
+| Structured JSON response (uniform records + redundant fields), response compression alone | ~66% | blacklisted fields and empty values removed; over-long strings/arrays truncated |
+| Function-calling schema (long descriptions), schema compression alone | ~47% | description truncation; `title`/`examples` and code blocks removed |
+| Mixed workload (response + schema): response compression only | ~62% | responses dominate the savings |
+| Mixed workload: schema + response stacked | ~65% | both payload types reduced |
+| Mixed workload: full stack (schema + response + TOON, deployed gating) | ~65% | the deployed `compress-toon` size guard keeps the original input when TOON does not reduce estimated tokens; on this fixture TOON inflates after response/schema compression, so the deployed rate equals the stacked rate above. The benchmark's ungated `full_stack` config measures ~63% but is not what the deployed path emits |
+| TOON encoding only | ~16% | only tabular, regular JSON benefits clearly |
+
+By scenario:
+
+- **High savings**: tools returning many uniform records (lists, tables, search results), payloads carrying redundant fields such as `debug`/`trace`/`logs`, or schemas with verbose descriptions.
+- **Moderate savings**: shell output is truncated only beyond the Layer 2 thresholds (strings 65,536 chars, arrays 128 items, depth 8); output below the thresholds stays mostly unchanged.
+- **Near-zero savings**: responses shorter than the minimum trigger length (200 characters for shared adapters, 500 for Codex); already-compact JSON without redundancy; any input that does not shrink (the size guard keeps the original).
+- **Not compressed at all**: content-retrieval tool output (Read/Glob/Grep and the like), non-JSON text, and skill-like text with a YAML frontmatter header. See [User manual · Compression trigger conditions and thresholds](user-manual.md#compression-trigger-conditions-and-thresholds) for the full trigger rules.
+
+Real session savings must additionally be multiplied by the share of tool payloads in total session tokens; see [Interpret the saving rate correctly](#interpret-the-saving-rate-correctly) above.
+
+## Standard test load
+
+The repository ships a deterministic standard test load under `src/tokenless/benchmark/l1-compressor` (a standalone Cargo workspace; Linux only — unsupported on macOS and Windows). The load is generated by `python/gen_fixtures.py`, uses no randomness, is byte-for-byte reproducible, and is committed to the repository:
+
+| Load file | Content |
+|-----------|---------|
+| `fixtures/records.json` | 1,000 uniform records |
+| `fixtures/tool_response.json` | a realistic tool response (envelope + 60 records + `trace`/`logs` redundant fields) |
+| `fixtures/schema_search.json` | a realistic function-calling schema |
+
+Run the compression-rate report quickly (build required; output covers single-path rates, stacked compression configs, and cost projections):
+
+```bash
+cd src/tokenless/benchmark/l1-compressor
+cargo run --release --bin compression_rate            # human-readable report
+cargo run --release --bin compression_rate -- --json  # machine-readable JSON; binary flags must follow `--`
+```
+
+Run the full quality/adversarial tests plus the rate report (skips criterion performance benches; takes a few minutes):
+
+```bash
+cd src/tokenless/benchmark/l1-compressor
+./run-benchmarks.sh --quick
+```
+
+Notes on using the standard load:
+
+- Token counts use a bytes/4 heuristic estimate; they are meant for relative comparison across versions, and absolute values do not represent billed tokens.
+- Compression rates evolve across versions; cite the corresponding commit or version when quoting numbers.
+- The standard load serves cross-version comparison, not your business data; for real-world savings, measure your own workload with a [dry-run comparison](#run-a-dry-run-comparison).
+
 ## Local AgentSight display
 
 AgentSight's Token savings view can aggregate `~/.tokenless/stats.db` read-only. When both run as the same user and AgentSight can access that database, SLS is not required to display local Tokenless statistics.

docs/user-guide/en/token-saving/tokenless/user-manual.md

@@ -114,6 +114,34 @@ This setting does not disable RTK command rewriting, adapter execution, or retri
 anolisa adapter disable tokenless <framework>
 ```
 
+### Compression trigger conditions and thresholds
+
+Adapters do not compress every tool result. For response compression, compressed content is produced only when all of the following hold:
+
+1. Compression is not switched off. With `compression_enabled=false` or `TOKENLESS_COMPRESSION_ENABLED=0` the run becomes a dry-run: statistics are still calculated, but the original text is returned (see the previous section).
+2. The tool is not a content-retrieval tool. Read/Glob/Grep/LSP/NotebookRead and their aliases skip response compression so their content stays intact.
+3. The response reaches the minimum length. The shared response hook, OpenClaw, and Hermes skip responses shorter than 200 characters; Codex skips responses shorter than 500 characters. Length is counted in characters, not bytes.
+4. The content is valid JSON. Response compression only handles JSON; plain text passes through unchanged. The shared paths additionally skip skill-like text with YAML frontmatter.
+5. The compressed result is strictly smaller. When neither response compression nor TOON encoding makes the content smaller, the original text is kept.
+
+After these checks, truncation strength depends on the tool category. Categories and thresholds are defined in `tool_categories.json` inside the adapter directory (the single source of truth shared by all adapters); built-in safe fallbacks are used when the file is missing or invalid:
+
+| Category | Representative tools | String truncation threshold | Array item cap | Maximum nesting depth |
+|----------|----------------------|------------------------------|----------------|-----------------------|
+| Content retrieval | Read, Glob, Grep, LSP, NotebookRead and aliases | Compression skipped | — | — |
+| Shell/exec | Bash, Shell, exec, terminal, etc. | 65,536 characters | 128 items | 8 |
+| Other structured tools | Any tool not in the two categories above | 1,048,576 characters | 65,536 items | 32 |
+
+Threshold semantics: a string longer than the threshold is cut at the threshold (retrievable through Stash when Stash is enabled); an array over the cap keeps only its leading items while the tail is truncated (also retrievable through Stash when enabled); subtrees nested deeper than the depth cap collapse into a truncation marker.
+
+Per-path differences worth noting:
+
+- Running `tokenless compress-response` standalone uses the CLI's own defaults (4,096 characters / 32 items / depth 8), overridable with `--truncate-strings-at`, `--truncate-arrays-at`, and `--max-depth`; see the [CLI reference](cli-reference.md).
+- Codex adds task-management tools (TodoWrite, Task, TaskStatus) to the shared skip list, and includes compressed content in context only for responses of at least 4,000 characters.
+- The OpenClaw plugin can override the tool categories through `skip_tools` and `shell_tools`; the thresholds themselves still come from `tool_categories.json`. See [Configuration and data privacy](configuration-and-privacy.md) for the options.
+- TOON encoding is a separate trigger decision: it is only adopted when the encoded result is smaller than the current content.
+- The AgentScope framework integration does not use the adapter thresholds above; it selects thresholds by `conservative` / `balanced` / `aggressive` mode. See [Framework integration](framework-integration.md).
+
 ### Reversible compression is conditional
 
 Active response and schema truncation stash the removed payload in `~/.tokenless/stash.db` by default and add a marker such as:
@@ -167,6 +195,7 @@ Command rewriting also changes the shell command submitted by the host. Most ada
 | Build the in-process Python runtime | [This page · Build the Python runtime from source](#build-the-python-runtime-from-source) |
 | Connect an Agent product or integrate AgentScope | [Agent and framework integration](framework-integration.md) |
 | Compress, retrieve, or run MCP manually | [CLI reference](cli-reference.md) |
+| Understand when compression triggers and what the thresholds are | [This page · Compression trigger conditions and thresholds](#compression-trigger-conditions-and-thresholds) |
 | Inspect savings or content changes, or run a dual comparison | [Measuring savings](measuring-savings.md) |
 | Change settings or understand local data | [Configuration and data privacy](configuration-and-privacy.md) |
 | Fix missing statistics, adapter, or Stash issues | [Troubleshooting](troubleshooting.md) |

docs/user-guide/zh/token-saving/tokenless/cli-reference.md

@@ -125,7 +125,7 @@ debug, trace, traces, stack, stacktrace, logs, logging
 
 Stash 只作用于字符串、数组尾部和深层子树截断。黑名单字段、`null` 和空值会直接移除，不会生成取回标记。
 
-大多数 Adapter 会覆盖这些独立 CLI 默认值。共享 Shell 策略使用 `65536`、`128`、`8`；其他结构化工具策略使用 `1048576`、`65536`、`32`。内容读取类工具会被跳过。详见[框架集成 · Adapter 处理规则](framework-integration.md#adapter-处理规则)。
+大多数 Adapter 会覆盖这些独立 CLI 默认值。共享 Shell 策略使用 `65536`、`128`、`8`；其他结构化工具策略使用 `1048576`、`65536`、`32`。内容读取类工具会被跳过。详见[框架集成 · Adapter 处理规则](framework-integration.md#adapter-处理规则)和[用户手册 · 压缩的触发条件与阈值](user-manual.md#压缩的触发条件与阈值)。
 
 ## `compress-toon` 与 `decompress-toon`
 

docs/user-guide/zh/token-saving/tokenless/framework-integration.md

@@ -43,6 +43,8 @@ OpenCode 当前使用下文说明的随附生命周期脚本，本版本尚未
 
 Claude Code 需要 2.1.121 或更高版本才能使用 `updatedToolOutput`。版本更旧或无法确定时，响应压缩会关闭，以免重复注入原文。结构化工具输出会保留宿主 Schema，不会转换成文本 TOON；以字符串承载的 JSON 在 TOON 更小时可以使用 TOON。
 
+完整触发条件（压缩开关、最小响应长度、JSON 校验、严格变小保护）与阈值含义见[用户手册 · 压缩的触发条件与阈值](user-manual.md#压缩的触发条件与阈值)。
+
 ### DeepSeek Harness 原生处理路径
 
 DSH Bundle 要求 Node.js 22 或更高版本，并需要兼容的 DSH profile。应在同一条

docs/user-guide/zh/token-saving/tokenless/measuring-savings.md

@@ -173,6 +173,59 @@ tokenless stats summary \
 
 例如，Payload 压缩率为 60%，但工具 Payload 只占会话总 Token 的 20%，则总体估算收益约为 12%。这个结果仍不是提供商账单保证值。
 
+## 压缩率的适用场景
+
+Tokenless 的压缩率取决于 Payload 中可精简成分的多少，不同场景差异很大。使用下文的标准测试负载测得的参考值（测量于 commit `2e7d69f1`，升级后请在本地重新运行确认）：
+
+| 场景 | 参考节省率（估算 Token） | 说明 |
+|------|--------------------------|------|
+| 结构化 JSON 响应（统一记录 + 冗余字段），单路响应压缩 | 约 66% | 黑名单字段、空值被移除，超长字符串/数组被截断 |
+| 函数调用 Schema（长描述），单路 Schema 压缩 | 约 47% | 描述截断，移除 `title`/`examples` 与代码块 |
+| 混合负载（响应 + Schema）：仅响应压缩 | 约 62% | 响应是主要收益来源 |
+| 混合负载：Schema + 响应压缩叠加 | 约 65% | 两类 Payload 同时精简 |
+| 混合负载：全栈叠加（Schema + 响应 + TOON，部署门控） | 约 65% | 部署路径的 `compress-toon` 尺寸保护会在 TOON 不能减少估算 Token 时保留原输入；本负载下响应/Schema 压缩后再做 TOON 反而膨胀，因此部署后的节省率与上一行叠加结果相同。基准报告中未加门控的 `full_stack` 组合测得约 63%，但部署路径不会输出该结果 |
+| 仅 TOON 编码 | 约 16% | 表格化、规整的 JSON 才有明显收益 |
+
+按场景归纳：
+
+- **收益高**：工具返回大量统一结构的记录（列表、表格、搜索结果），或携带 `debug`/`trace`/`logs` 等冗余字段，或 Schema 描述冗长。
+- **收益中等**：Shell 输出中超过 Layer 2 阈值（字符串 65,536 字符、数组 128 项、深度 8）的部分会被截断；未超阈值的输出基本保持原样。
+- **收益接近零**：短于最小触发长度的响应（共享 Adapter 200 字符、Codex 500 字符）；本身已经紧凑、无冗余的 JSON；任何压缩后不比原文更小的输入（尺寸保护会保留原文）。
+- **不参与压缩**：内容读取类工具（Read/Glob/Grep 等）的输出、非 JSON 文本、带 YAML frontmatter 的 Skill 文本。触发条件详见[用户手册 · 压缩的触发条件与阈值](user-manual.md#压缩的触发条件与阈值)。
+
+实际会话收益还要乘以工具 Payload 在会话总 Token 中的占比，见上文[正确解释节省率](#正确解释节省率)。
+
+## 标准测试负载
+
+仓库内置了确定性的标准测试负载，位于 `src/tokenless/benchmark/l1-compressor`（独立 Cargo workspace；仅支持 Linux，不支持 macOS/Windows）。负载由 `python/gen_fixtures.py` 生成，不含随机数，字节级可复现，并已提交在仓库中：
+
+| 负载文件 | 内容 |
+|----------|------|
+| `fixtures/records.json` | 1,000 条统一结构记录 |
+| `fixtures/tool_response.json` | 典型工具响应（外层信封 + 60 条记录 + `trace`/`logs` 冗余字段） |
+| `fixtures/schema_search.json` | 典型函数调用 Schema |
+
+快速运行压缩率报告（需要先构建，输出包括单路压缩率、各压缩组合的叠加结果与成本估算）：
+
+```bash
+cd src/tokenless/benchmark/l1-compressor
+cargo run --release --bin compression_rate            # 人类可读报告
+cargo run --release --bin compression_rate -- --json  # 机器可读 JSON；二进制参数必须放在 `--` 之后
+```
+
+运行完整质量/对抗测试 + 压缩率报告（跳过 criterion 性能基准，耗时数分钟）：
+
+```bash
+cd src/tokenless/benchmark/l1-compressor
+./run-benchmarks.sh --quick
+```
+
+使用标准负载时请注意：
+
+- Token 数使用字节/4 启发式估算，适合版本间相对比较，绝对值不代表真实计费 Token。
+- 压缩率随版本演进可能变化，引用数字时请注明对应的 commit 或版本号。
+- 标准负载用于横向对比，不代表你的业务数据；评估真实收益仍应使用[双跑对比](#用-dry-run-做双跑对比)在自己的工作负载上测量。
+
 ## AgentSight 本地展示
 
 AgentSight 的 Token savings 页面可以只读聚合 `~/.tokenless/stats.db`。两者由同一用户运行，且 AgentSight 能访问该数据库时，不需要通过 SLS 才能看到本地 Tokenless 统计。