docs(tokenless): add compression-rate scenario guide and standard test payloads

[Forrest-ly]

Summary

Customer feedback asked for (1) documentation explaining when tokenless compression rates apply and what ranges to expect, and (2) a standard test payload that users can run to verify compression behavior in their own environment.

This PR is docs + fixtures only — no Rust code changes.

Changes

New user-guide page: "Compression rates and applicable scenarios"

docs/user-guide/{zh,en}/token-saving/tokenless/compression-scenarios.md

• How the compression rate is computed (UTF-8 byte based, ceil(bytes/4) token estimate, no-savings operations are not recorded).
• Per-strategy applicable scenarios with reference ranges (schema ~57%, response ~26–78%, TOON 15–40%, RTK rewrite 60–90%) and the factors that move the rate.
• Scenarios with low or no compression (short/compact responses, natural-language prose, source-code payloads, high-entropy content, already-trimmed output) and the expected behavior for each.
• Note that adapter truncation thresholds differ from the standalone CLI defaults, so measured rates can differ between the two paths.
• How to run the standard test payloads and how to interpret the results.

Standard test payloads

src/tokenless/benchmark/standard-payload/ — deterministic, synthetic payloads covering the high-to-low compression spectrum, plus tooling:

• schema_tools.json — Function Calling schema array with verbose descriptions (schema compression).
• response_api_records.json — structured API response: envelope plus 48 repetitive records including debug/trace/logs fields (response compression high-savings case; also used for the TOON case; exercises the array-tail stash).
• response_code.json — code-search results whose content is source code (medium/low case).
• response_prose.json — document-search results whose content is natural-language prose (low boundary case).
• gen_standard_payload.py — single source of truth; standard library only, no randomness, byte-identical regeneration.
• run-standard-check.sh — runs each payload through the installed CLI in an isolated TOKENLESS_DATA_DIR (real stats/stash databases untouched) and prints stats summary --json per case.
• README.md — manifest, usage, stability policy.

All payload content is synthetic; no real user data, hosts, or credentials.

Cross-links

• user-manual.md (zh+en): new row in the find-documentation-by-task table.
• measuring-savings.md (zh+en): pointer from the saving-rate interpretation section.
• QUICKSTART.md (zh+en): next-steps entry.
• src/tokenless/benchmark/README.md: pointer to the new directory.

Test report

Environment: Linux x86_64, cargo/rustc 1.94.1, Node.js 22.21.1 (site build), Python 3.8 (generator/check tooling, stdlib only).

Build

• cargo build --release --locked -p tokenless-cli — success, 0 errors.

Documentation validation (all green)

• bash scripts/docs-lint.sh — naming convention OK, en/zh tree parity OK.
• python3 scripts/docs-link-check.py — all relative links resolve.
• npm run validate:locales --prefix website — passed.
• npm run build --prefix website — full Docusaurus build passed for both locales with broken-link checking enabled (onBrokenLinks: throw).
• npm run check:links --prefix website — static link and duplicate-ID validation passed for 164 HTML files (includes fragment checks for the new page's anchors).

Standard payload verification (real CLI runs, tokenless 0.7.6)

• run-standard-check.sh end-to-end: all 5 cases executed and recorded one statistics record each.
• Determinism: regenerated all payloads with gen_standard_payload.py and compared byte-for-byte — identical.
• Reproducibility: re-ran the structured-response case independently — identical metrics (stash keys are fixed-length, so sizes are stable).
• Output validity: compressed schema/response outputs parse as valid JSON; the structured-response output carries a <<tokenless:...>> stash marker for the truncated array tail (retrievable).

Case	Input bytes	Output bytes	Chars saved	Est. tokens saved
Schema compression (schema_tools.json)	10,060	4,976	50.5%	50.7%
Response compression, structured (response_api_records.json)	37,018	15,579	57.9%	57.9%
Response compression, code (response_code.json)	5,991	4,927	17.8%	17.8%
Response compression, prose (response_prose.json)	4,697	4,410	6.1%	6.1%
TOON encoding (response_api_records.json)	37,018	29,475	20.4%	20.4%

These measured values (default truncation thresholds) are what the new doc page publishes as its reference table.

Not run (with reasons)

• cargo test on tokenless crates: this PR changes no Rust code (docs, JSON fixtures, shell and Python tooling only), so no crate test suite is affected.
• RTK command-rewrite measurement: RTK acts on command output rather than the JSON payloads in this suite; the doc page states its reference range without a standard payload for it.

[Forrest-ly]

Closing as duplicate: PR #2601 (from a parallel run of the same task) already covers this issue — compression-rate applicable-scenario documentation plus standard test load guidance. Consolidating there per the no-duplicate-PR rule. The branch on the fork is kept for reference; it contains an alternative user-facing variant (dedicated CLI-runnable payloads + check script) that can serve as optional follow-up material.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 256ea104dc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Make the standard check independent of user configuration

When a user has disabled statistics in ~/.tokenless/config.json or via TOKENLESS_STATS_ENABLED=0, these invocations succeed without recording anything, and the script then misleadingly reports the empty summary as “no estimated token savings.” The check also inherits TOKENLESS_COMPRESSION_ENABLED, TOKENLESS_STATS_DB, and TOKENLESS_STASH_DB; these can change the measured output or redirect writes to the user's real databases despite the isolation claim. Force the required feature flags and unset database overrides for both the compression and summary commands.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Document the CJK-aware token heuristic

For payloads containing CJK or other non-ASCII text, this formula does not match the implementation: tokenless-stats/src/tokenizer.rs::estimate_tokens counts each CJK character as one token and uses ceil(other Unicode characters / 4) for the remainder, rather than dividing UTF-8 bytes by four. The byte formula happens to match these synthetic ASCII fixtures but gives users incorrect expectations for multilingual workloads and can misstate why an operation is or is not recorded. Update both locale pages to describe the character-aware heuristic.

AGENTS.md reference: AGENTS.md:L339-L343

Useful? React with 👍 / 👎.