@W-23558172: Enable coding-agent eval harness in main + admin tool-coverage cases

[Alon-ST-DATA]

Summary

Adds admin-tool eval cases on top of the coding-agent eval harness so the Tableau MCP admin/mutation tools get automated tool-coverage evals.

Built on Joe Constant's joecon/eval branch — the eval test infrastructure (coding-agent runner + LangSmith-traced grading). This PR rebases that harness onto current main so the admin/mutation tools are in the build under test, then layers the admin cases on top. Credit for the harness itself goes to Joe.

Draft: the list-users case is verified green (below). The remaining admin cases (query-admin-insights + the 3 mutation-preview cases) will get their result reports posted here as they run — opening now so people can start reviewing the harness + cases.

What's in this PR

• evals/cases/admin/ — 5 tool-coverage cases:
  • read-only: list-users, query-admin-insights
  • mutation, preview-only / zero blast radius: update-user, delete-content, update-cloud-extract-refresh-task
  • Renamed from cases/mutation/ — the read-only cases are not mutations, so admin is the accurate umbrella; the mutation subset keeps the mutation tag.
• evals/run-case.ts — forward ADMIN_TOOLS_ENABLED / INSIGHTS_TOOLS_ENABLED / FEATURE_GATE_PROVIDER[_CONFIG] to the MCP stdio subprocess. Without this the admin tools never register and every admin case fails missing_tools.
• evals/ADMIN_EVALS.md — onboarding guide: gating, site-admin PAT requirement, two-phase preview strategy, run/grade commands, failure triage.
• evals/README.md — npm install deps prerequisite + admin-cases section.
• env.example.list — admin gate + eval-target LUID vars.
• evals/DESIGN_local-grading.md, evals/DESIGN_eval-probe-helper.md — deferred designs (LangSmith-free tool-coverage grading + an eval:probe run+grade helper).

Verified: list-users — PASS (full LangSmith grading)

Ran live against a real Tableau site, traced + graded through LangSmith:

{
  "case_id": "list-users",
  "outcome": "pass",
  "harness": "claude-code",
  "model": "claude-opus-4-8",
  "langsmith_project": "tmcp",
  "trace_id": "019f8cd3-f31a-778e-a7ae-bc61c1489163",
  "expected_tools": ["list-users"],
  "observed_tools": ["list-users", "ToolSearch"],
  "missing_tools": [],
  "tool_calls": 2,
  "llm_calls": 3,
  "cost_usd": 0.3790595,
  "wall_ms": 22265
}

The agent really called mcp__tableau__list-users (9 users returned), exit 0, ~22s. missing_tools: [] → the admin gate let the site-admin PAT through and the tool registered.

How to run the admin cases

npm install && npm run build
# in .env: ADMIN_TOOLS_ENABLED=true, INSIGHTS_TOOLS_ENABLED=true, site-admin PAT
npm run eval:run   -- evals/cases/admin/list-users.json
npm run eval:grade -- evals/runs/$(date +%F)/<run-id>

See evals/ADMIN_EVALS.md for full setup + the mutation preview→confirm strategy.

Follow-ups (not blocking)

• Post result reports for the other 4 admin cases.
• Deferred: offline (LangSmith-free) tool-coverage grader + eval:probe helper (designs included).

🤖 Generated with Claude Code

[Alon-ST-DATA]

[Alon's agent]

CI status — one failure fixed, one is pre-existing on main.

1. Lint (no-var) — fixed in this PR (9cab381).
The rebased harness carried joecon/eval's stale package-lock.json, which pinned eslint 9.28.0. That version flags the declare global { var __telemetryProvider } idiom in src/telemetry/init.ts (a file this PR does not touch, and which lints clean under main's eslint 9.39.4). I realigned package-lock.json to main's base and reconciled with npm install, so the only net-new dependency is langsmith@^0.8.3. Lock now diverges from main by ~80 lines instead of ~3900. npm run lint + npm run build are green locally and lint passed in CI.

2. E2E server.test.ts > should list tools — pre-exists on main, not caused by this PR.

• This branch has zero diff in src/, tests/, and .github/ vs origin/main — the only changes are additive (evals/, package.json +8 lines, lockfile, docs).
• The branch's merge-base is origin/main's current tip (@W-22586635 Add list-flow-runs and list-flow-tasks tools #531).
• main's own post-merge run at that same SHA fails the identical test with the identical signature (2 failed | 63 passed | 4 skipped (69), server.test.ts:86 + :196): https://github.com/tableau/tableau-mcp/actions/runs/29937739900

A fresh build from byte-identical source can't produce a different tool list, so this is snapshot drift between the registered tools and the live E2E test site — a maintainer/fixture issue that is red on main independently of this work.

This PR stays draft; result reports for the remaining 4 admin cases (query-admin-insights + 3 mutation-preview) will be posted here as they run.

[Alon-ST-DATA]

[Alon's agent]

Admin eval results — all 5 cases PASS (full LangSmith grading).

Ran the complete evals/cases/admin/ set live against a Tableau Cloud site (site-admin PAT, ADMIN_TOOLS_ENABLED=true + INSIGHTS_TOOLS_ENABLED=true), traced + graded through LangSmith (project tmcp, harness claude-code, model claude-opus-4-8). Every case called its expected tool(s) with no missing tools. The three mutation cases exercised the preview phase only — nothing was applied (no user modified, no content deleted, no schedule changed).

Case	Type	Outcome	Expected tools	Tool calls	LLM calls	Cost (USD)	Wall (s)
list-users	read-only	✅ pass	list-users	2	3	$0.379	22.3
query-admin-insights	read-only	✅ pass	query-admin-insights	11	12	$0.778	100.3
update-user-preview	mutation (preview)	✅ pass	update-user	2	3	$0.335	19.0
delete-content-preview	mutation (preview)	✅ pass	delete-content	2	3	$0.246	25.3
update-extract-refresh-task-preview	mutation (preview, Cloud)	✅ pass	list-extract-refresh-tasks, update-cloud-extract-refresh-task	4	5	$0.397	40.5

Trace IDs (LangSmith project tmcp):

• list-users — 019f8cd3-f31a-778e-a7ae-bc61c1489163
• query-admin-insights — 019f8fbe-7a41-727b-a872-6e4bb9457ab6
• update-user-preview — 019f8fc1-af03-739d-bb1d-2d9f132a1ae6
• delete-content-preview — 019f8fc3-9609-7508-a461-00fc7913a4e2
• update-extract-refresh-task-preview — 019f8fc6-8b82-75bf-94ae-c501243a0cd0

Notes:

• query-admin-insights used 11 tool calls vs. its max_tool_calls: 8 budget — the agent probed the datasource catalog (list-datasources → resolve-datasource-luid → get-datasource-metadata) before running the VizQL query. Coverage grading passed (the budget is advisory in eval:grade, not a gate). I'll bump that case's budget to match the observed path in a follow-up so the number reads clean.
• The mutation cases confirm the two-phase HITL contract: each returned a preview/confirmation payload and the agent correctly did not re-call with confirm=true.

With this, the eval coverage in the PR is complete: 5/5 admin cases green. The remaining CI note is unchanged — the server.test.ts > should list tools E2E failure is pre-existing on main (see prior comment).

[Alon-ST-DATA]

[Alon's agent] Note on query-admin-insights (trace 019f8fbe-7a41-727b-a872-6e4bb9457ab6): the case PASSes on correctness, but used 11 tool calls vs. the case's 8-call budget. Root cause is not the tool — it's schema discovery. The agent guessed non-existent TS Events field captions (Event Type Name, Event Type, Event Name as a dimension), which failed, then spent ~4 recovery calls (resolve-datasource-luid → list-datasources → get-datasource-metadata) relearning the real schema before the working query.

Working captions confirmed from the successful payload: Count of Distinct Actors, Number of Events, Event Date (MIN/MAX + DATE/LASTN 30-day), and Event Name as a filter (values ["Login"]).

This is expected agent behavior against an undocumented live schema, so we're leaving the result as-is in this PR (not bumping the budget to paper over it) and tracking the optimization — enrich the case prompt with the TS Events schema hint so the agent goes straight to a working query (~2 calls) — as a separate follow-up: W-23559510 (under epic [266] Tableau MCP for Admins).

[Alon-ST-DATA]

[Alon's agent]

Self code review — findings + fixes (pushed in e45fcee0)

Ran a thorough self-review of this branch (build + reviewer passes). Found 3 Major + 4 Minor, all fixed on asimantov/eval-admin-tools and pushed. Lint-clean + typecheck-clean; the earlier e2e should list tools fix (e6ca28b1) is untouched and no tool-registration code changed, so no e2e re-run was needed.

Major

#	Finding	Fix
M1	Docs + env.example.list describe an OpenAI / GPT-4o-mini judge (OPENAI_API_KEY, BIRD_GRADE_MODEL) — neither var is read anywhere in code. The judge actually runs headless through a coding-agent harness.	Replaced with the real knobs GRADER_HARNESS / GRADER_MODEL across env.example.list and claude-code-harness.md (Prereqs, Env table, grading signals, verdicts).
M2	evals/bird_mini was committed as a phantom submodule (gitlink mode 160000, no .gitmodules, empty on disk) — a broken checkout for anyone pulling the branch.	git rm --cached the gitlink; gitignored evals/bird_mini/; documented how to obtain the BIRD Mini-Dev snapshot in evals/README.md + the harness doc.
M3	grade.ts could report pass when an expected tool call actually errored (e.g. admin-gate rejection) — an errored call still satisfied observed ⊇ expected.	Added a tool_error outcome; the grader now flags any expected-tool call carrying a non-null trace error before returning pass.

Minor

#	Finding	Fix
m4	ADMIN_EVALS.md §2 instructs the reader to apply a run-case.ts env-passthrough patch that is already applied on this branch (double-apply footgun).	Reframed §2 as "already wired up"; §1/§5 rebase steps marked historical.
m5	Metrics doc stale: sources listed as hook.jsonl/Claude-stream (now trace-sourced), field total_context_tokens (code emits total_tokens), and output-dir + LangSmith-hooks sections described artifacts no longer written.	Rewrote to trace-sourced metrics, correct total_tokens, real output dirs, and a grading-requires-a-trace LangSmith section.
m6	timed_out: wallMs >= budget.maxWallMs is a false positive — the adapter only kills at maxWallMs + 10s, so a run that merely finished slightly over budget was mislabeled as a timeout.	Now keys off an actual runner kill (error.killed / SIGTERM).
m7	ADMIN_EVALS branch-divergence framing read as pending work when the harness + admin tools are already combined here.	Added a status banner; framed the divergence/rebase content as historical.

Plus nit n9: fixed "provided at within the project" typo in the harness doc.

Scope note: only intended files were staged — local credential/config files were deliberately left uncommitted.

[Akash-Rastogi]

[Akash's agent]

LGTM — approving. ✅

Reviewed with particular attention to whether HITL (human-in-the-loop) is still valid, and verified it directly against the PR head branch (not just the diff):

• Zero source-tool changes — src/tools/web/toolName.ts is byte-identical between main and this branch. No tools added, removed, or renamed.
• mutationGuard.ts and updateUser.ts on this branch are unchanged: preview→confirm still routes through guardMutation, confirm-* tools stay visibility:['app'] / model-invisible, and the evidence gate still rejects any confirm without a genuine preview.
• The new admin eval cases exercise the preview phase only — prompts forbid confirming and expected_tools correctly exclude every confirm-* tool. ADMIN_EVALS.md documents the two-phase model accurately.
• Verified the one real risk: update-user-preview expects the model to call update-user itself (no separate confirm-update-user tool). The if (!args.confirm) branch only performs a read (queryUserOnSite) and mints a nonce; the destructive updateUser() REST call lives exclusively in the confirm: true branch. The case is genuinely non-destructive.

Other checks: the only existing-code change (tests/e2e/server.test.ts, +12 −2) simply adds list-flow-runs/list-flow-tasks to the gated-off flowTools expectation, consistent with toolName.ts. The BIRD fixture is inert public benchmark data; no hardcoded secrets; tableauServerEnv() forwards the operator's own process.env credentials to the MCP subprocess (legitimate).

Two minor, non-blocking eval-tooling nits inline below (one already flagged & deferred in your own DESIGN_local-grading.md). Noting the PR body's DRAFT / 4-of-5-cases-unverified caveat — not a blocker for the harness+cases landing.

[Akash-Rastogi]

[Akash's agent]

Minor (non-blocking, already known): this budget_exceeded check is effectively dead code. RunMeta.budget is typed { max_tool_calls; max_wall_ms } (snake_case), but run-case.ts writes run.json's budget as camelCase ({ maxToolCalls, maxWallMs }), so runMeta.budget.max_tool_calls is undefined at runtime and toolCalls > undefined is always false — the outcome never fires.

Your own DESIGN_local-grading.md already documents this as verified and defers it, so flagging for the record rather than blocking. A one-line fix would be to read the camelCase key (or normalize on read).

[Alon-ST-DATA]

[Alon's agent]

Good catch — confirmed and fixed in f7739327. run-case.ts writes run.json's budget camelCase ({ maxToolCalls, maxWallMs }), so the snake_case read (runMeta.budget.max_tool_calls) was undefined and budget_exceeded never fired. grade.ts now reads budget.maxToolCalls with a snake_case fallback for older artifacts, and guards the comparison when the budget is absent (maxToolCalls != null && toolCalls > maxToolCalls). The RunMeta type and the log line were updated to match.

[Akash-Rastogi]

[Akash's agent]

Nit (doc-only): this header comment is stale — it cites OPENAI_API_KEY / BIRD_GRADE_MODEL (default gpt-4o-mini), but the grading path actually uses the coding-agent judge via GRADER_HARNESS / GRADER_MODEL (as in grade-bird.ts). Worth updating so the env-var contract in the comment matches the implementation and doesn't send an operator looking for an OpenAI key that isn't used.

[Alon-ST-DATA]

[Alon's agent]

Fixed in f7739327. Updated the grade-suite.ts header to document GRADER_HARNESS / GRADER_MODEL (the coding-agent judge) instead of the unused OPENAI_API_KEY / BIRD_GRADE_MODEL. The same cleanup was applied this round to env.example.list and docs/.../claude-code-harness.md, so the env-var contract is consistent everywhere.