diff --git a/docs/user-guide/en/token-saving/tokenless/cli-reference.md b/docs/user-guide/en/token-saving/tokenless/cli-reference.md index 6974f7a5cd..67eba71b60 100644 --- a/docs/user-guide/en/token-saving/tokenless/cli-reference.md +++ b/docs/user-guide/en/token-saving/tokenless/cli-reference.md @@ -255,6 +255,7 @@ tokenless env-check --tool Shell --fix ```bash tokenless stats summary tokenless stats summary --json +tokenless stats summary --limit 1000 tokenless stats list --limit 20 tokenless stats show tokenless stats diff @@ -271,6 +272,8 @@ Dual-run comparison: tokenless stats summary --compare ``` +A missing session ID fails with a non-zero exit instead of a 0% comparison, matching `stats diff --session`. `stats summary --limit` must be a positive integer; `--limit 0` is rejected at parse time, matching `stats diff --limit`. + Inspect one record or the verified stages of one tool call: ```bash diff --git a/docs/user-guide/en/token-saving/tokenless/measuring-savings.md b/docs/user-guide/en/token-saving/tokenless/measuring-savings.md index 6732f67a67..2906832b67 100644 --- a/docs/user-guide/en/token-saving/tokenless/measuring-savings.md +++ b/docs/user-guide/en/token-saving/tokenless/measuring-savings.md @@ -69,6 +69,8 @@ The summary reads at most the latest 10,000 records by default. Limit the query tokenless stats summary --limit 1000 ``` +`--limit` must be a positive integer. `--limit 0` is rejected at parse time with a non-zero exit, matching `stats diff --limit`. + ## Inspect individual records List recent records: @@ -194,6 +196,8 @@ tokenless stats summary \ Notes: - `--compare` requires exactly two session IDs in baseline, active order. +- If either session has no records, the command exits with an error instead of reporting 0% savings. +- `--limit` must be a positive integer. `--limit 0` is rejected at parse time instead of looking like a missing session. - The baseline should be a dry-run and the active session should apply compression. The CLI warns on a mode mismatch. - For real agent tasks, keep inputs, tool versions, and the environment as similar as possible. - Dry-run still writes the complete before/after text to the local statistics database. diff --git a/docs/user-guide/zh/token-saving/tokenless/cli-reference.md b/docs/user-guide/zh/token-saving/tokenless/cli-reference.md index 987d855f21..d79a0aa606 100644 --- a/docs/user-guide/zh/token-saving/tokenless/cli-reference.md +++ b/docs/user-guide/zh/token-saving/tokenless/cli-reference.md @@ -249,6 +249,7 @@ tokenless env-check --tool Shell --fix ```bash tokenless stats summary tokenless stats summary --json +tokenless stats summary --limit 1000 tokenless stats list --limit 20 tokenless stats show tokenless stats diff @@ -265,6 +266,8 @@ tokenless stats clear --yes tokenless stats summary --compare ``` +Session ID 不存在时以非零退出码失败，而不是输出 0% 对比，行为与 `stats diff --session` 一致。`stats summary --limit` 必须为正整数；`--limit 0` 会在解析阶段被拒绝，行为与 `stats diff --limit` 一致。 + 查看单条记录，或一次工具调用中可确认衔接的阶段： ```bash diff --git a/docs/user-guide/zh/token-saving/tokenless/measuring-savings.md b/docs/user-guide/zh/token-saving/tokenless/measuring-savings.md index 50b10b45f5..1ff06b0d95 100644 --- a/docs/user-guide/zh/token-saving/tokenless/measuring-savings.md +++ b/docs/user-guide/zh/token-saving/tokenless/measuring-savings.md @@ -69,6 +69,8 @@ tokenless stats summary --json tokenless stats summary --limit 1000 ``` +`--limit` 必须为正整数。`--limit 0` 会在解析阶段以非零退出码被拒绝，行为与 `stats diff --limit` 一致。 + ## 查看单条记录 列出最近记录： @@ -188,6 +190,8 @@ tokenless stats summary \ 注意： - `--compare` 必须提供恰好两个 Session ID，顺序为 baseline、active。 +- 任一 Session 没有记录时，命令以错误退出，而不是报告 0% 节省。 +- `--limit` 必须为正整数。`--limit 0` 会在解析阶段被拒绝，而不会被误报为 Session 缺失。 - baseline 应为 dry-run，active 应为真实压缩；模式不匹配时 CLI 会告警。 - 对真实 Agent 任务做对比时，应尽量使用相同输入、工具版本和环境。 - dry-run 仍会把压缩前后文本写入本地统计数据库。 diff --git a/src/tokenless/README.md b/src/tokenless/README.md index 521072dbdf..9e29e53ddc 100644 --- a/src/tokenless/README.md +++ b/src/tokenless/README.md @@ -323,10 +323,13 @@ echo 'name: Alice\nage: 30' | tokenless decompress-toon ### Inspect token savings -Use `show` to print the complete stored before/after payload, or `diff` to -explain the estimated token saving and highlight only changed lines: +Use `stats summary` for totals, `show` for the stored before/after payload, or +`diff` to explain the estimated token saving and highlight only changed lines: ```bash +tokenless stats summary +tokenless stats summary --limit 1000 +tokenless stats summary --compare tokenless stats show 42 tokenless stats diff 42 tokenless stats diff --session @@ -334,10 +337,13 @@ tokenless stats diff --session --tool-use-id tokenless stats diff 42 --json ``` -Session overviews contain metrics only. Record and tool-use reports include a -unified content diff; consecutive active stages are linked only when their -stored output/input content matches exactly, avoiding duplicate intermediate -token counts. See [Measuring Tokenless Savings](../../docs/user-guide/en/token-saving/tokenless/measuring-savings.md) +`stats summary --limit` must be a positive integer; `--limit 0` is rejected at +parse time. `--compare` fails if either session has no records instead of +reporting 0% savings. Session overviews contain metrics only. Record and +tool-use reports include a unified content diff; consecutive active stages are +linked only when their stored output/input content matches exactly, avoiding +duplicate intermediate token counts. See +[Measuring Tokenless Savings](../../docs/user-guide/en/token-saving/tokenless/measuring-savings.md) for options and measurement limits. ### Database location diff --git a/src/tokenless/README_zh.md b/src/tokenless/README_zh.md index 345dd33792..8803a67599 100644 --- a/src/tokenless/README_zh.md +++ b/src/tokenless/README_zh.md @@ -353,10 +353,13 @@ node npm/scripts/package-npm.js --all ## 查看 Token 节省明细 -`show` 用于原样打印完整的压缩前后内容；`diff` 用于解释估算 Token -节省，并只突出发生变化的行： +`stats summary` 用于查看合计；`show` 用于原样打印完整的压缩前后内容； +`diff` 用于解释估算 Token 节省，并只突出发生变化的行： ```bash +tokenless stats summary +tokenless stats summary --limit 1000 +tokenless stats summary --compare tokenless stats show 42 tokenless stats diff 42 tokenless stats diff --session @@ -364,9 +367,11 @@ tokenless stats diff --session --tool-use-id tokenless stats diff 42 --json ``` -Session 总览只包含指标；单记录和 tool-use 报告包含 unified content -diff。只有相邻 active 阶段的输出与输入内容完全一致时才会串成一条链， -从而避免重复计算中间阶段的 Token。完整选项和度量限制见 +`stats summary --limit` 必须为正整数；`--limit 0` 会在解析阶段被拒绝。 +`--compare` 在任一 Session 没有记录时失败，而不是报告 0% 节省。Session +总览只包含指标；单记录和 tool-use 报告包含 unified content diff。只有相邻 +active 阶段的输出与输入内容完全一致时才会串成一条链，从而避免重复计算中间 +阶段的 Token。完整选项和度量限制见 [Tokenless 效果度量](../../docs/user-guide/zh/token-saving/tokenless/measuring-savings.md)。 ## 数据库位置 diff --git a/src/tokenless/crates/tokenless-cli/src/main.rs b/src/tokenless/crates/tokenless-cli/src/main.rs index e65a5d8956..6b59015257 100644 --- a/src/tokenless/crates/tokenless-cli/src/main.rs +++ b/src/tokenless/crates/tokenless-cli/src/main.rs @@ -181,7 +181,7 @@ impl From for DiffSort { enum StatsCommands { /// Show summary statistics with breakdown by operation Summary { - #[arg(long)] + #[arg(long, value_parser = parse_positive_usize)] limit: Option, /// Output machine-readable JSON #[arg(long)] @@ -699,6 +699,14 @@ fn run_command(command: Commands) -> Result<(), (String, i32)> { let tokenless = recorder .records_by_session(tokenless_sid, limit) .map_err(|e| (format!("Failed to query tokenless: {}", e), 1))?; + if let Some(message) = missing_compare_sessions( + baseline_sid, + tokenless_sid, + &baseline, + &tokenless, + ) { + return Err((message, 1)); + } // Warn if a session's records do not match the expected mode, // i.e. the baseline run was not recorded as dry-run. warn_mode_mismatch("baseline", &baseline, CompressionMode::DryRun); @@ -974,6 +982,27 @@ fn resolve_mode( } } +/// Error text when a `--compare` side has no recorded stats. +/// +/// An empty side used to format as a successful 0% report, which hid typos +/// and made A/B scripts look like "no savings". Fail closed like +/// `stats diff --session` and the Python `TokenlessStats.compare` client. +fn missing_compare_sessions( + baseline_sid: &str, + tokenless_sid: &str, + baseline: &[StatsRecord], + tokenless: &[StatsRecord], +) -> Option { + let mut missing = Vec::new(); + if baseline.is_empty() { + missing.push(format!("baseline session {baseline_sid:?}")); + } + if tokenless.is_empty() { + missing.push(format!("tokenless session {tokenless_sid:?}")); + } + (!missing.is_empty()).then(|| format!("No records found for {}", missing.join(" and "))) +} + /// Warn (to stderr) when a session's records were not recorded in the expected /// mode, e.g. a "baseline" session that was not run with compression disabled. /// A non-blocking sanity hint — comparison still proceeds. diff --git a/src/tokenless/crates/tokenless-cli/src/tests/main_tests.rs b/src/tokenless/crates/tokenless-cli/src/tests/main_tests.rs index 19af3aa407..14b35eeddb 100644 --- a/src/tokenless/crates/tokenless-cli/src/tests/main_tests.rs +++ b/src/tokenless/crates/tokenless-cli/src/tests/main_tests.rs @@ -984,6 +984,38 @@ fn stats_diff_cli_validates_scope_and_limit() { ); } +#[test] +fn stats_summary_cli_rejects_zero_limit() { + let zero = match Cli::try_parse_from(["tokenless", "stats", "summary", "--limit", "0"]) { + Err(error) => error, + Ok(_) => panic!("summary --limit 0 must fail at parse time"), + }; + assert!(zero.to_string().contains("greater than zero")); + + let compare_zero = match Cli::try_parse_from([ + "tokenless", + "stats", + "summary", + "--limit", + "0", + "--compare", + "baseline-run", + "active-run", + ]) { + Err(error) => error, + Ok(_) => panic!("compare --limit 0 must fail at parse time"), + }; + assert!(compare_zero.to_string().contains("greater than zero")); + + let parsed = Cli::try_parse_from(["tokenless", "stats", "summary", "--limit", "1"]).unwrap(); + match parsed.command { + Commands::Stats(StatsCommands::Summary { limit, .. }) => { + assert_eq!(limit, Some(1)); + } + _ => panic!("expected stats summary"), + } +} + #[test] fn run_command_compress_response_large_with_truncation() { let _guard = TempDbGuard::new(); @@ -1197,26 +1229,174 @@ fn run_command_stats_disable_does_not_persist_env_overrides() { assert!(!persisted.compression_enabled); } +#[test] +fn missing_compare_sessions_reports_each_empty_side() { + let record = StatsRecord::new( + OperationType::CompressSchema, + "cli".to_string(), + 100, + 40, + 50, + 20, + ); + assert!( + missing_compare_sessions( + "b", + "t", + std::slice::from_ref(&record), + std::slice::from_ref(&record) + ) + .is_none() + ); + + let both = missing_compare_sessions("b", "t", &[], &[]).unwrap(); + assert!(both.starts_with("No records found for ")); + assert!(both.contains("baseline session \"b\"")); + assert!(both.contains("tokenless session \"t\"")); + assert!(both.contains(" and ")); + + let baseline_only = + missing_compare_sessions("b", "t", &[], std::slice::from_ref(&record)).unwrap(); + assert!(baseline_only.contains("baseline session \"b\"")); + assert!(!baseline_only.contains("tokenless session")); + + let tokenless_only = missing_compare_sessions("b", "t", &[record], &[]).unwrap(); + assert!(tokenless_only.contains("tokenless session \"t\"")); + assert!(!tokenless_only.contains("baseline session")); +} + +#[test] +fn missing_compare_sessions_debug_escapes_control_chars() { + let message = missing_compare_sessions( + "base\u{1b}]0;INJECTED\u{7}", + "tls\u{1b}]52;c;INJECTED\u{7}", + &[], + &[], + ) + .unwrap(); + assert!(!message.contains('\u{1b}')); + assert!(!message.contains('\u{7}')); + assert!(message.contains("INJECTED")); +} + +fn seed_compare_record(session_id: &str, mode: CompressionMode, before: usize, after: usize) { + let recorder = open_recorder().expect("open recorder"); + recorder + .record( + &StatsRecord::new( + OperationType::CompressResponse, + "cli".to_string(), + before * 4, + before, + after * 4, + after, + ) + .with_session_id(session_id) + .with_mode(mode), + ) + .expect("seed compare record"); +} + #[test] fn run_command_stats_compare() { - let _guard = match TempDbGuard::new() { Some(g) => g, None => return }; + let _guard = match TempDbGuard::new() { + Some(g) => g, + None => return, + }; let result = run_command(Commands::Stats(StatsCommands::Summary { limit: None, json: false, - compare: Some(vec!["baseline-sess".to_string(), "tokenless-sess".to_string()]), + compare: Some(vec![ + "baseline-sess".to_string(), + "tokenless-sess".to_string(), + ]), })); - assert!(result.is_ok()); + let err = result.expect_err("empty compare sessions must fail closed"); + assert_eq!(err.1, 1); + assert!(err.0.contains("No records found")); + assert!(err.0.contains("baseline session \"baseline-sess\"")); + assert!(err.0.contains("tokenless session \"tokenless-sess\"")); } #[test] fn run_command_stats_compare_json() { - let _guard = match TempDbGuard::new() { Some(g) => g, None => return }; + let _guard = match TempDbGuard::new() { + Some(g) => g, + None => return, + }; let result = run_command(Commands::Stats(StatsCommands::Summary { limit: None, json: true, - compare: Some(vec!["baseline-sess".to_string(), "tokenless-sess".to_string()]), + compare: Some(vec![ + "baseline-sess".to_string(), + "tokenless-sess".to_string(), + ]), + })); + let err = result.expect_err("empty JSON compare must not emit a 0% report"); + assert_eq!(err.1, 1); + assert!(err.0.contains("No records found")); +} + +#[test] +fn run_command_stats_compare_one_side_missing() { + let _guard = match TempDbGuard::new() { + Some(g) => g, + None => return, + }; + seed_compare_record("tokenless-sess", CompressionMode::Active, 400, 200); + let baseline_missing = run_command(Commands::Stats(StatsCommands::Summary { + limit: None, + json: false, + compare: Some(vec![ + "baseline-sess".to_string(), + "tokenless-sess".to_string(), + ]), + })) + .expect_err("missing baseline must fail"); + assert!(baseline_missing.0.contains("baseline session \"baseline-sess\"")); + assert!(!baseline_missing.0.contains("tokenless session")); + + seed_compare_record("baseline-sess", CompressionMode::DryRun, 400, 200); + let tokenless_missing = run_command(Commands::Stats(StatsCommands::Summary { + limit: None, + json: true, + compare: Some(vec![ + "baseline-sess".to_string(), + "absent-tokenless".to_string(), + ]), + })) + .expect_err("missing tokenless side must fail"); + assert!(tokenless_missing.0.contains("tokenless session \"absent-tokenless\"")); + assert!(!tokenless_missing.0.contains("baseline session")); +} + +#[test] +fn run_command_stats_compare_populated_sessions() { + let _guard = match TempDbGuard::new() { + Some(g) => g, + None => return, + }; + seed_compare_record("baseline-sess", CompressionMode::DryRun, 400, 200); + seed_compare_record("tokenless-sess", CompressionMode::Active, 400, 200); + let result = run_command(Commands::Stats(StatsCommands::Summary { + limit: None, + json: false, + compare: Some(vec![ + "baseline-sess".to_string(), + "tokenless-sess".to_string(), + ]), })); assert!(result.is_ok()); + + let json = run_command(Commands::Stats(StatsCommands::Summary { + limit: None, + json: true, + compare: Some(vec![ + "baseline-sess".to_string(), + "tokenless-sess".to_string(), + ]), + })); + assert!(json.is_ok()); } #[test] diff --git a/src/tokenless/crates/tokenless-cli/tests/cli_integration.rs b/src/tokenless/crates/tokenless-cli/tests/cli_integration.rs index c3a0c11461..c29fad4230 100644 --- a/src/tokenless/crates/tokenless-cli/tests/cli_integration.rs +++ b/src/tokenless/crates/tokenless-cli/tests/cli_integration.rs @@ -3,8 +3,8 @@ use std::process::Command; use tokenless_ccr::StashStore; use tokenless_runtime::{CompressOptions, compress_response_with_store}; use tokenless_stats::{ - OperationType, StatsRecord, StatsRecorder, estimate_tokens, estimate_tokens_from_bytes, - get_home_dir, + CompressionMode, OperationType, StatsRecord, StatsRecorder, estimate_tokens, + estimate_tokens_from_bytes, get_home_dir, }; fn tokenless_bin() -> Command { @@ -1243,3 +1243,189 @@ fn env_check_hard_bypass_json_is_stable_across_processes() { ); } } + +#[test] +fn stats_summary_compare_rejects_missing_sessions() { + let db = match TempStatsDb::new() { + Some(db) => db, + None => return, + }; + let output = db + .command() + .args(["stats", "summary", "--compare", "missing-a", "missing-b"]) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains("No records found")); + assert!(stderr.contains("missing-a")); + assert!(stderr.contains("missing-b")); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!(!stdout.contains("Tokenless Comparison Report")); + assert!(!stdout.contains("saved_percent")); +} + +#[test] +fn stats_summary_compare_json_rejects_one_missing_side() { + let db = match TempStatsDb::new() { + Some(db) => db, + None => return, + }; + let output = db + .command() + .args([ + "stats", + "summary", + "--json", + "--compare", + "missing-baseline", + "integration-session", + ]) + .output() + .unwrap(); + assert_eq!(output.status.code(), Some(1)); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains("baseline session \"missing-baseline\"")); + assert!(!stderr.contains("tokenless session")); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!(!stdout.contains("saved_percent")); +} + +#[test] +fn stats_summary_compare_reports_populated_sessions() { + let db = match TempStatsDb::new() { + Some(db) => db, + None => return, + }; + let recorder = StatsRecorder::new(&db.path).unwrap(); + recorder + .record( + &StatsRecord::new( + OperationType::CompressResponse, + "integration-agent".to_string(), + 1600, + 400, + 800, + 200, + ) + .with_session_id("baseline-run") + .with_mode(CompressionMode::DryRun), + ) + .unwrap(); + recorder + .record( + &StatsRecord::new( + OperationType::CompressResponse, + "integration-agent".to_string(), + 1600, + 400, + 800, + 200, + ) + .with_session_id("active-run") + .with_mode(CompressionMode::Active), + ) + .unwrap(); + + let text = db + .command() + .args([ + "stats", + "summary", + "--compare", + "baseline-run", + "active-run", + ]) + .output() + .unwrap(); + assert!( + text.status.success(), + "compare populated sessions; stderr: {}", + String::from_utf8_lossy(&text.stderr) + ); + let stdout = String::from_utf8_lossy(&text.stdout); + assert!(stdout.contains("Tokenless Comparison Report")); + assert!(stdout.contains("TOTAL")); + + let json = db + .command() + .args([ + "stats", + "summary", + "--json", + "--compare", + "baseline-run", + "active-run", + ]) + .output() + .unwrap(); + assert!(json.status.success()); + let parsed: serde_json::Value = serde_json::from_slice(&json.stdout).unwrap(); + assert_eq!(parsed["baseline_tokens"], 400); + assert_eq!(parsed["tokenless_tokens"], 200); + assert_eq!(parsed["saved_tokens"], 200); +} + +#[test] +fn stats_summary_compare_rejects_zero_limit() { + let db = match TempStatsDb::new() { + Some(db) => db, + None => return, + }; + let recorder = StatsRecorder::new(&db.path).unwrap(); + recorder + .record( + &StatsRecord::new( + OperationType::CompressResponse, + "integration-agent".to_string(), + 1600, + 400, + 800, + 200, + ) + .with_session_id("baseline-run") + .with_mode(CompressionMode::DryRun), + ) + .unwrap(); + recorder + .record( + &StatsRecord::new( + OperationType::CompressResponse, + "integration-agent".to_string(), + 1600, + 400, + 800, + 200, + ) + .with_session_id("active-run") + .with_mode(CompressionMode::Active), + ) + .unwrap(); + + let output = db + .command() + .args([ + "stats", + "summary", + "--limit", + "0", + "--compare", + "baseline-run", + "active-run", + ]) + .output() + .unwrap(); + assert_ne!(output.status.code(), Some(0)); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains("greater than zero"), + "zero limit must fail at parse time; stderr: {stderr}" + ); + assert!( + !stderr.contains("No records found"), + "populated sessions with --limit 0 must not look missing; stderr: {stderr}" + ); + let stdout = String::from_utf8_lossy(&output.stdout); + assert!(!stdout.contains("Tokenless Comparison Report")); + assert!(!stdout.contains("saved_percent")); +}