docs(tokenless): add compression-rate scenario guide and standard test payloads

docs/user-guide/en/token-saving/tokenless/QUICKSTART.md

@@ -165,5 +165,6 @@ standalone CLI from source, see
 - [User manual](user-manual.md): behavior boundaries and documentation map
 - [CLI reference](cli-reference.md): all subcommands and options
 - [Measuring savings](measuring-savings.md): statistics, dual runs, and AgentSight/SLS
+- [Compression rates and applicable scenarios](compression-scenarios.md): expected rates per scenario and the standard test payloads
 - [Configuration and data privacy](configuration-and-privacy.md): toggles, storage, and sensitive data
 - [Troubleshooting](troubleshooting.md): common errors, upgrades, and uninstall

docs/user-guide/en/token-saving/tokenless/compression-scenarios.md

@@ -0,0 +1,93 @@
+# Compression Rates and Applicable Scenarios
+
+[中文版](../../../zh/token-saving/tokenless/compression-scenarios.md)
+
+The compression rate Tokenless reports is a per-payload metric. This page explains the expected compression rate and the factors behind it for each strategy in different scenarios, and provides a set of standard test payloads so you can verify compression behavior in your own environment.
+
+## How the compression rate is computed
+
+- Compression rate = (before − after) ÷ before, with sizes measured in UTF-8 bytes.
+- Token counts use the `ceil(bytes ÷ 4)` estimate; no model tokenizer is invoked.
+- Operations with no savings are not recorded: when the estimated token count does not drop, the CLI emits the original text and produces no statistics record.
+- The aggregated rate in `stats summary` and dashboards covers only payloads that passed through Tokenless; it is not the session-wide saving rate. See [Interpret the saving rate correctly](measuring-savings.md#interpret-the-saving-rate-correctly) for the conversion.
+
+## Applicable scenarios and reference rates per strategy
+
+| Strategy | Applicable scenarios | Reference rate | Main factors |
+|----------|----------------------|----------------|--------------|
+| Schema compression | Many Function Calling tool definitions, verbose tool/parameter descriptions, examples present | ~57% | Description length, examples/title presence, parameter count |
+| Response compression | Structured tool/API JSON responses: repetitive record arrays, null/empty values, debug fields, long strings | ~26%–78% | Structural redundancy, array length, string length, truncation thresholds |
+| TOON encoding | Tabular JSON with uniform fields and repetitive records | 15%–40% | Record homogeneity, field count |
+| Command rewriting (RTK) | Noisy build/test/package-manager command output | 60%–90% | Command type, share of noise in the output |
+
+Reference rates are typical values observed on common workloads, not commitments; actual rates are determined by the content itself. Use the standard test payloads below to verify locally.
+
+### Scenarios with high compression rates
+
+- **Repetitive structured responses**: list endpoints, search results, bulk status queries. The longer the array and the more uniform the records, the higher the rate; array tails beyond the truncation threshold go into the Stash and stay retrievable, so compression remains end-to-end lossless.
+- **API responses with redundant fields**: responses carrying `debug`, `trace`, or `logs` fields (default blacklist), `null` values, or empty strings/arrays/objects — that content is removed outright.
+- **Sessions with many tool definitions**: when an agent registers dozens or hundreds of tools, schema compression strips examples, titles, and overly long descriptions from the definitions.
+- **Command-line output**: noisy output from build tools, test frameworks, and package managers is filtered by RTK. RTK is a separate binary and works on command output rather than JSON payloads.
+
+### Scenarios with low or no compression
+
+| Scenario | Why | Expected behavior |
+|----------|-----|-------------------|
+| Short, compact responses | Compression yields no token savings | Original emitted, no statistics recorded (expected) |
+| Natural-language prose (document retrieval, web pages) | Little removable structural redundancy | Low single digits to about ten percent |
+| Source-code-dominated responses | Code itself has low redundancy | Around ten to twenty percent, depending on structure |
+| High-entropy content: base64/binary, compressed or encrypted data, random strings | No redundancy to remove | Almost no savings |
+| Output already trimmed upstream (fields filtered, pages truncated) | Redundancy already removed | Savings depend on the remaining content |
+| Model reasoning output, system prompts, conversation history | Outside what Tokenless touches | Not involved |
+
+Different adapters use different truncation thresholds (shared shell policy `65536`/`128`/`8`, other structured-tool policy `1048576`/`65536`/`32`; see [Adapter processing rules](framework-integration.md#adapter-processing-rules)), so the same content can measure differently through the standalone CLI than inside an agent.
+
+## Verifying with the standard test payloads
+
+The repository ships a set of deterministic standard payloads under [`src/tokenless/benchmark/standard-payload/`](https://github.com/alibaba/anolisa/tree/main/src/tokenless/benchmark/standard-payload), covering the typical scenarios from high to low compression:
+
+| Payload | Scenario | Matching command |
+|---------|----------|------------------|
+| `schema_tools.json` | Function Calling schema array with verbose descriptions | `tokenless compress-schema --batch` |
+| `response_api_records.json` | Structured API response (48 repetitive records, with debug/trace/logs fields) | `tokenless compress-response`, `tokenless compress-toon` |
+| `response_code.json` | Code-search results (content is source code) | `tokenless compress-response` |
+| `response_prose.json` | Document-search results (content is natural-language prose) | `tokenless compress-response` |
+
+All payload content is synthetic and contains no real user data.
+
+### Running
+
+Clone the repository and run the bundled check script (requires an installed tokenless):
+
+```bash
+git clone https://github.com/alibaba/anolisa.git
+cd anolisa/src/tokenless/benchmark/standard-payload
+./run-standard-check.sh
+```
+
+Or download a single payload and run it by hand:
+
+```bash
+curl -fsSL -O https://raw.githubusercontent.com/alibaba/anolisa/main/src/tokenless/benchmark/standard-payload/response_api_records.json
+tokenless compress-response -f response_api_records.json \
+  --session-id stdpay-api
+tokenless stats summary --json
+```
+
+### Reference results
+
+The numbers below were measured with tokenless 0.7.6 and the default truncation thresholds on Linux x86_64. Character and token metrics are content-based and platform-independent, so they should reproduce on other supported platforms.
+
+| Case | Input (bytes) | Output (bytes) | Chars saved | Est. tokens saved |
+|------|---------------|----------------|-------------|-------------------|
+| Schema compression (`schema_tools.json`) | 10,060 | 4,976 | ~50.5% | ~50.7% |
+| Response compression · structured (`response_api_records.json`) | 37,018 | 15,579 | ~57.9% | ~57.9% |
+| Response compression · code (`response_code.json`) | 5,991 | 4,927 | ~17.8% | ~17.8% |
+| Response compression · prose (`response_prose.json`) | 4,697 | 4,410 | ~6.1% | ~6.1% |
+| TOON encoding (`response_api_records.json`) | 37,018 | 29,475 | ~20.4% | ~20.4% |
+
+### How to read the results
+
+- **Standard-payload results differ markedly from the reference table**: first check `tokenless --version`, then confirm the input files match the repository (`gen_standard_payload.py` regenerates them and the output must be byte-identical to the committed files).
+- **Your real workload compresses differently from the reference table**: that is expected — the rate is determined by content redundancy. Use the two scenario tables above to place your workload in the right band.
+- **Estimating session-wide savings**: overall estimated saving ≈ payload compression rate × tool-response share of total session tokens; see [Interpret the saving rate correctly](measuring-savings.md#interpret-the-saving-rate-correctly).

docs/user-guide/en/token-saving/tokenless/measuring-savings.md

@@ -173,6 +173,8 @@ Estimated overall saving rate
 
 For example, a 60% payload compression rate with tool payloads representing 20% of the session gives an estimated overall saving of about 12%. This is still not a provider billing guarantee.
 
+For expected rates per scenario and the standard test payloads used to verify them locally, see [Compression rates and applicable scenarios](compression-scenarios.md).
+
 ## Local AgentSight display
 
 AgentSight's Token savings view can aggregate `~/.tokenless/stats.db` read-only. When both run as the same user and AgentSight can access that database, SLS is not required to display local Tokenless statistics.

docs/user-guide/en/token-saving/tokenless/user-manual.md

@@ -168,6 +168,7 @@ Command rewriting also changes the shell command submitted by the host. Most ada
 | Connect an Agent product or integrate AgentScope | [Agent and framework integration](framework-integration.md) |
 | Compress, retrieve, or run MCP manually | [CLI reference](cli-reference.md) |
 | Inspect savings or content changes, or run a dual comparison | [Measuring savings](measuring-savings.md) |
+| Understand compression-rate scenarios or verify with standard payloads | [Compression rates and applicable scenarios](compression-scenarios.md) |
 | Change settings or understand local data | [Configuration and data privacy](configuration-and-privacy.md) |
 | Fix missing statistics, adapter, or Stash issues | [Troubleshooting](troubleshooting.md) |
 | Upgrade or uninstall | [Troubleshooting · Upgrade and uninstall](troubleshooting.md#upgrade-and-uninstall) |

docs/user-guide/zh/token-saving/tokenless/QUICKSTART.md

@@ -158,5 +158,6 @@ tokenless stats list --limit 1
 - [用户手册](user-manual.md)：能力边界和文档导航
 - [CLI 参考](cli-reference.md)：全部子命令和参数
 - [效果度量](measuring-savings.md)：统计、双跑对比和 AgentSight/SLS
+- [压缩率与适用场景](compression-scenarios.md)：各场景预期压缩率与标准测试负载
 - [配置与数据隐私](configuration-and-privacy.md)：开关、存储和敏感数据
 - [故障排查](troubleshooting.md)：常见错误、升级和卸载

docs/user-guide/zh/token-saving/tokenless/compression-scenarios.md

@@ -0,0 +1,93 @@
+# 压缩率与适用场景
+
+[English](../../../en/token-saving/tokenless/compression-scenarios.md)
+
+Tokenless 报告的压缩率是 Payload 级指标。本页说明各策略在不同场景下的预期压缩率与影响因素，并提供一组标准测试负载，便于你在自己的环境中验证压缩行为。
+
+## 如何计算压缩率
+
+- 压缩率 =（压缩前 − 压缩后）÷ 压缩前，大小以 UTF-8 字节数计算。
+- Token 数使用 `ceil(字节数 ÷ 4)` 近似估算，不调用模型 Tokenizer。
+- 无收益的操作不入库：压缩后估算 Token 数没有下降时，CLI 输出原文，不产生统计记录。
+- `stats summary` 与面板中的聚合压缩率只覆盖经过 Tokenless 的 Payload，不等于会话总体节省率，换算方法见[正确解释节省率](measuring-savings.md#正确解释节省率)。
+
+## 各策略的适用场景与参考压缩率
+
+| 策略 | 适用场景 | 参考压缩率 | 主要影响因素 |
+|------|----------|------------|--------------|
+| Schema 压缩 | Function Calling 工具定义多、工具或参数描述冗长、带示例 | ~57% | 描述长度、examples/title 多少、参数数量 |
+| 响应压缩 | 结构化工具/API JSON 响应：重复记录数组、null/空值、debug 字段、长字符串 | ~26%–78% | 结构冗余度、数组长度、字符串长度、截断阈值 |
+| TOON 编码 | 字段统一、记录重复的表格型 JSON | 15%–40% | 记录同质性、字段数量 |
+| 命令重写（RTK） | 构建、测试、包管理等高噪声命令输出 | 60%–90% | 命令类型、输出中噪声占比 |
+
+参考压缩率是常见负载下的典型值，不是承诺值；实际高低由内容本身决定，可用下文的标准测试负载在本地验证。
+
+### 压缩率较高的典型场景
+
+- **重复结构化响应**：列表接口、搜索结果、批量状态查询。数组越长、记录越同质，压缩率越高；数组超过截断阈值时尾部进入 Stash，可按标记取回，端到端无损。
+- **冗余字段多的 API 响应**：包含 `debug`、`trace`、`logs` 等默认黑名单字段、`null` 值、空字符串/数组/对象的响应，这些内容会被直接移除。
+- **工具定义多的会话**：Agent 注册几十上百个工具时，Schema 压缩移除定义中的示例、标题和超长描述。
+- **命令行输出**：构建工具、测试框架、包管理器的高噪声输出经 RTK 过滤。RTK 是独立二进制，作用于命令输出而非 JSON Payload。
+
+### 压缩率偏低或不适用的场景
+
+| 场景 | 原因 | 预期表现 |
+|------|------|----------|
+| 短响应、结构紧凑 | 压缩无 Token 收益 | 输出原文、不记录统计（预期行为） |
+| 自然语言长文（文档检索、网页正文） | 可移除的结构冗余少 | 低个位数到一成左右 |
+| 源码为主的响应 | 代码自身冗余低 | 一成到两成左右，取决于结构 |
+| 高熵内容：base64/二进制、已压缩或加密数据、随机字符串 | 无冗余可移除 | 几乎无收益 |
+| 已被上游精简的输出（已过滤字段、已分页截断） | 冗余已提前移除 | 收益取决于剩余内容 |
+| 模型推理输出、system prompt、对话历史 | 不在 Tokenless 处理范围 | 不涉及 |
+
+不同 Adapter 使用不同的截断阈值（共享 Shell 策略 `65536`/`128`/`8`，其他结构化工具策略 `1048576`/`65536`/`32`，详见 [Adapter 处理规则](framework-integration.md#adapter-处理规则)），因此同一内容在独立 CLI 与 Agent 内的实测压缩率可能不同。
+
+## 用标准测试负载验证
+
+仓库提供一组确定性标准负载，位于 [`src/tokenless/benchmark/standard-payload/`](https://github.com/alibaba/anolisa/tree/main/src/tokenless/benchmark/standard-payload)，覆盖从高到低的典型场景：
+
+| 负载 | 场景 | 对应命令 |
+|------|------|----------|
+| `schema_tools.json` | 描述冗长的 Function Calling Schema 数组 | `tokenless compress-schema --batch` |
+| `response_api_records.json` | 结构化 API 响应（48 条重复记录，含 debug/trace/logs 字段） | `tokenless compress-response`、`tokenless compress-toon` |
+| `response_code.json` | 代码搜索结果（内容为源码） | `tokenless compress-response` |
+| `response_prose.json` | 文档搜索结果（内容为自然语言长文） | `tokenless compress-response` |
+
+负载内容全部为构造的合成数据，不含真实用户数据。
+
+### 运行
+
+克隆仓库并运行配套检查脚本（需要已安装 tokenless）：
+
+```bash
+git clone https://github.com/alibaba/anolisa.git
+cd anolisa/src/tokenless/benchmark/standard-payload
+./run-standard-check.sh
+```
+
+也可以只下载单个负载手动运行：
+
+```bash
+curl -fsSL -O https://raw.githubusercontent.com/alibaba/anolisa/main/src/tokenless/benchmark/standard-payload/response_api_records.json
+tokenless compress-response -f response_api_records.json \
+  --session-id stdpay-api
+tokenless stats summary --json
+```
+
+### 参考结果
+
+以下数值在 tokenless 0.7.6、默认截断阈值下实测，环境为 Linux x86_64。字符与 Token 指标都是基于内容的度量，与平台无关，在其他受支持平台上应可复现。
+
+| 用例 | 输入（字节） | 输出（字节） | 字符节省 | 估算 Token 节省 |
+|------|--------------|--------------|----------|------------------|
+| Schema 压缩（`schema_tools.json`） | 10,060 | 4,976 | ~50.5% | ~50.7% |
+| 响应压缩 · 结构化（`response_api_records.json`） | 37,018 | 15,579 | ~57.9% | ~57.9% |
+| 响应压缩 · 代码（`response_code.json`） | 5,991 | 4,927 | ~17.8% | ~17.8% |
+| 响应压缩 · 长文（`response_prose.json`） | 4,697 | 4,410 | ~6.1% | ~6.1% |
+| TOON 编码（`response_api_records.json`） | 37,018 | 29,475 | ~20.4% | ~20.4% |
+
+### 如何解读结果
+
+- **标准负载结果与参考表差异明显**：先用 `tokenless --version` 确认版本，再确认输入文件与仓库一致（`gen_standard_payload.py` 可重新生成，输出应与仓库文件逐字节相同）。
+- **真实业务负载的压缩率与参考表不同**：这是正常现象，压缩率由内容冗余度决定。对照上文两个场景表，可以判断自己的负载落在哪一档。
+- **估算会话总体节省**：总体估算节省率 ≈ Payload 压缩率 × 工具响应占会话总 Token 的比例，见[正确解释节省率](measuring-savings.md#正确解释节省率)。

docs/user-guide/zh/token-saving/tokenless/measuring-savings.md

@@ -173,6 +173,8 @@ tokenless stats summary \
 
 例如，Payload 压缩率为 60%，但工具 Payload 只占会话总 Token 的 20%，则总体估算收益约为 12%。这个结果仍不是提供商账单保证值。
 
+不同场景下的预期压缩率，以及用于本地验证的标准测试负载，见[压缩率与适用场景](compression-scenarios.md)。
+
 ## AgentSight 本地展示
 
 AgentSight 的 Token savings 页面可以只读聚合 `~/.tokenless/stats.db`。两者由同一用户运行，且 AgentSight 能访问该数据库时，不需要通过 SLS 才能看到本地 Tokenless 统计。

docs/user-guide/zh/token-saving/tokenless/user-manual.md

@@ -166,6 +166,7 @@ Stash 并不能让所有压缩都可逆。被移除的 `debug`/`trace` 字段、
 | 接入 Agent 产品或集成 AgentScope | [Agent 与框架集成](framework-integration.md) |
 | 手动压缩、取回或运行 MCP | [CLI 参考](cli-reference.md) |
 | 查看节省或内容变化、做双跑对比 | [效果度量](measuring-savings.md) |
+| 了解压缩率适用场景或用标准负载验证 | [压缩率与适用场景](compression-scenarios.md) |
 | 修改配置或了解本地数据 | [配置与数据隐私](configuration-and-privacy.md) |
 | 解决无统计、Adapter 或 Stash 问题 | [故障排查](troubleshooting.md) |
 | 升级或卸载 | [故障排查 · 升级与卸载](troubleshooting.md#升级与卸载) |

src/tokenless/benchmark/README.md

@@ -26,6 +26,11 @@ Each subdirectory is a standalone workspace (its own `Cargo.toml` with an empty
 `[workspace]` table) kept out of the main tokenless workspace on purpose — see
 the per-workspace `README.md` files for build/run instructions and methodology.
 
+In addition to the two benchmark layers, [`standard-payload/`](standard-payload)
+holds the deterministic user-facing payloads behind the user-guide page
+"Compression rates and applicable scenarios" — see its own `README.md` for the
+check script and stability policy.
+
 Each layer also keeps its results in its own `reports/` directory
 (`l1-compressor/reports/`, `l2-module/reports/`) so the two layers' numbers
 never mix. Both directories are gitignored: benchmark reports are