docs(tokenless): clarify savings-rate field definitions - #2596
docs(tokenless): clarify savings-rate field definitions#2596Forrest-ly wants to merge 2 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 52952401b7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| ## Saving-rate field definitions | ||
|
|
||
| Tokenless always expresses a saving rate as “saved tokens as a share of the original, uncompressed tokens”; only the aggregation scope differs. Every percentage field emitted by `tokenless stats` is defined as follows: |
There was a problem hiding this comment.
The table is not exhaustive: tokenless stats diff ... --json also serializes saved_percent for every chain and stage (src/tokenless/crates/tokenless-stats/src/diff.rs:108-127 and 148-162), using that object's before_tokens and after_tokens. A consumer following this new table could therefore treat saved_percent solely as a comparison-schema field with baseline/tokenless totals, even though diff output uses the same key with a different scope. Add the diff variants and formula, or narrow the claim that this defines every percentage field.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch — adopted. tokenless stats diff --json does serialize saved_percent for every chain and stage, computed from that object's own before_tokens/after_tokens rather than the --compare run totals.
Fixed in 157bb0c: added the diff variants to the field table in both language versions of measuring-savings.md, plus a sentence disambiguating the two saved_percent scopes, so the table stays exhaustive for every percentage field emitted by tokenless stats.
Verified against real CLI output with synthetic records (isolated data dir): stats diff <id> --json and stats diff --session <sid> --json both emit saved_percent per chain and per stage, and every value matches (before_tokens − after_tokens) ÷ before_tokens × 100% recomputed from that same object's own totals exactly.
![qoderai[bot]](https://avatars.githubusercontent.com/in/1396158?s=60&v=4){kind=small}
Customer feedback reported a savings_rate field computed as cached_tokens / total_tokens and attributed it to tokenless-stats. tokenless-stats has never emitted savings_rate or cached_tokens: every percentage it reports is saved amount divided by the original, uncompressed amount. Document the exact formula for each percentage field (chars_saved_percent, tokens_saved_percent, saved_percent), state that the text output percentage uses before_tokens as the denominator, and note explicitly that prompt-cache fields such as savings_rate / cached_tokens / total_cached_tokens are provider-side metrics not produced by tokenless-stats.
84b7838 to
3931626
Compare
tokenless stats diff --json serializes saved_percent for every chain and stage, computed from that object's own before_tokens and after_tokens. Add these variants to the saving-rate field table in both languages and note how their scope differs from the --compare saved_percent, so the table remains exhaustive for every percentage field emitted by tokenless stats.
1 participant
Why
Customer feedback reported a
savings_ratefield computed ascached_tokens / total_tokens, attributed to thetokenless-statsprogram, and asked to either fix the numeric definition of the field or clarify the documentation.Investigation against current
mainand the full history of the tokenless component:tokenless-statshas never emitted asavings_rate,cached_tokens, ortotal_cached_tokensfield — verified by searching the complete history ofsrc/tokenless(git log -S) and inspecting every output path (stats summarytext/JSON,--comparetext/JSON,stats diff, SLS JSONL records).chars_saved_percent= (before_chars − after_chars) ÷ before_chars × 100%tokens_saved_percent= (before_tokens − after_tokens) ÷ before_tokens × 100%saved_percent(compare) = (baseline_tokens − tokenless_tokens) ÷ baseline_tokens × 100%cached_tokens/total_tokensstyle fields are model-provider prompt-cache metrics surfaced by other observability tooling; they are not tokenless compression savings and are not produced by this component.There is therefore no
savings_ratedefinition to fix in code; the right remedy is explicit documentation, which this PR adds.What changed (docs only)
docs/user-guide/{zh,en}/token-saving/tokenless/measuring-savings.mdtokenless stats; clarification that the text-output percentage (Saved: N tokens (X%)) usesbefore_tokens(the original, uncompressed size) as denominator — not the session total and not any provider-side cache metric; and an explicit note thatsavings_rate/cached_tokens/total_cached_tokensare not tokenless-stats output (prompt-cache hit share ≠ tokenless compression savings).docs/user-guide/{zh,en}/token-saving/tokenless/cli-reference.mdstatssection pointing to the field-definition table.Validation (real execution)
Environment: Linux x86_64, Rust cargo/rustc 1.94.1.
cargo build -p tokenless-cli— success (0 errors).cargo test -p tokenless-stats— 143 passed, 0 failed.cargo test -p tokenless-cli— 255 passed, 0 failed, 2 pre-existing#[ignore](env-check tests, unrelated).TOKENLESS_DATA_DIR, synthetic payload):stats summary --json:chars_saved_percent= 12343/25300×100 = 48.7866% ✓;tokens_saved_percent= 3086/6326×100 = 48.7828% ✓; text output printsSaved: 3086 tokens (48.8%)✓stats summary --compare baseline-run active-run --json:saved_percent= (3163−1626)/3163×100 = 48.5931% ✓savings_rate/cached_tokensfields ✓git diff --check— clean.Not run: none skipped — the repository has no dedicated lint gate for these Markdown docs.
Risk and compatibility
Docs-only change; no behavior, schema, or CLI output changes. Revert the commit to undo.
Follow-up: review round 1 (stats diff percentages)
Adopted the review comment:
tokenless stats diff --jsonalso serializessaved_percentfor every chain and stage, using that object's ownbefore_tokens/after_tokens. Added the diff variants to the saving-rate field table (both languages) plus a sentence disambiguating the twosaved_percentscopes (--comparerun totals vs per-chain/stage totals), keeping the "every percentage field" claim exhaustive. Reply posted in the review thread.Validation (real execution, follow-up round)
Environment: Linux x86_64, Rust cargo/rustc 1.94.1.
cargo test -p tokenless-stats— 143 passed, 0 failed.cargo build -p tokenless-cli— success (0 errors).TOKENLESS_DATA_DIR, synthetic payloads):stats diff <record-id> --json: chainsaved_percent= (4380 − 3444) ÷ 4380 × 100% = 21.3698630137% ✓ (matches serialized value exactly); the stage row uses the same formula on the record's own totals ✓stats diff --session <session-id> --json(multi-chain output): every chain and every stage carriessaved_percent; all values match(before_tokens − after_tokens) ÷ before_tokens × 100%recomputed from the same object's own totals ✓git diff --check— clean.Not run: none skipped — the repository has no dedicated lint gate for these Markdown docs (Docs Lint runs in CI).