docs: 📝 Add MCP Platform Testing Plan v1 - #1092
Conversation
justincorrigible
left a comment
justincorrigible
left a comment
There was a problem hiding this comment.
some recommended changes that I think will make a difference
|
|
||
| Everything here goes into a **run manifest** written alongside every results file. Two runs are only comparable if their manifests differ in exactly the dimension being studied, so the comparison CLI warns loudly when they differ anywhere else. | ||
|
|
||
| **Test environment and case set.** The fingerprint covers the dataset and the catalogue configuration together, because either one changing invalidates a baseline the same way: per index, the document count, the mapping hash, and a checksum over a fixed aggregation, plus the `/introspection` output hashed. All of it obtained through Arranger, never ES ([§4.1](#41-working-against-the-frozen-dataset)). The case set is hashed separately. |
There was a problem hiding this comment.
A deployment's getServerSideFilter is neither the dataset nor catalogue configuration, so nothing in this section currently pins it.
Two suggested edits:
- add the server-side filter to this section's test environment list
- replicate that new parameter into R6 (line435)
Its detectability splits the four signals cleanly. A filter moves the document count and the aggregation checksum, both obtained through Arranger. It leaves the mapping hash and the introspection hash identical. So the only signals that catch it are the query-derived ones. That is the problem, because R6 pushes the other way. It asks for signals that are not "so sensitive that a harmless restart trips them", and the query-derived signals are exactly the ones that read as over-sensitive.
Drop them and a filter arriving on the testing instance becomes invisible. It then presents as a broad regression across every case at once. That is the failure § 4.1 exists to prevent.
There was a problem hiding this comment.
Another good call, I was unaware of getServerSideFilter and had to do some reading into that. I think it brings up other questions for me regarding how MCP should handle Arranger identity/auth. I'll need to spend some more time thinking about this before I action on this edit suggestion.
Will ping you with my understanding of it and my approach before I make any changes for this one 🙏
|
|
||
| - **Models are open-weight, served from hardware the team controls**, in small, medium, and large tiers. Sampling parameters (temperature, top-p, seed) are configurable, and **the client is ours too, so sampling is fixed at greedy**: temperature 0, one fixed seed, everywhere. Query translation has one right answer, so sampling would add variance with no upside. | ||
| - **Data lives on a separate testing ES instance and Arranger server**, not in the repo. The dataset is real, not synthetic, and it is **frozen**: it is not reindexed or mutated between runs. | ||
| - **The MCP server may be local, or a separately deployed instance.** The suite must support both, selected by configuration. |
There was a problem hiding this comment.
Three suggested edits:
- state somewhere here in §0 that expectations are derived under a single anonymous principal
- pin that in §2, so a later change to it invalidates baselines instead of passing through
- cross-reference the shared access-control conformance corpus, which expresses the same shape: which principal may see which record
There is no principal to define today (sans Usher), and that is the finding rather than an obstacle to it. The MCP server forwards no caller identity, so every session reaches Arranger as the same unauthenticated request (and therefor one same principal), and nothing in the case format at line 273 could set it differently... so this is an unrecorded assumption rather than a missing feature. MCP_API_KEY in § 5.4.1 does not change it either. An API key authenticates the transport. It does not necessarily make a request attributable to a principal.
The risk lands when identity arrives. outcomeMatch becomes principal relative, one intent has several correct totals depending on who asks, and every expectation derived before that point silently answers a different question. A pinned assumption turns that into an invalidated baseline. An unpinned one turns it into a suite that stays green through the change that broke it.
A field in the case format is the eventual shape, once there is something to put in it. It is not worth adding while it has one possible value. Two case sets of the same shape in two repositories will diverge unless they point at each other, which is what the third edit is for.
There was a problem hiding this comment.
Great call, I hadn't considered the future implications of Usher/identity on Arranger when hashing out this spec. Will make the suggested edits 🙏
|
|
||
| ### Phase 5: CI integration | ||
|
|
||
| - **Every PR: L1 only.** Contract tests plus the static-surface token budget per tier. No model server needed; the token budget needs only a tokenizer, which loads without a GPU. |
There was a problem hiding this comment.
Two suggested edits:
- state that the token budget resolves its tokenizer by pinned revision
- state that a tokenizer change is a deliberate re-baseline rather than a build failure
staticSurfaceTokens (line 210) is measured in a tokenizer. §3 already notes that token metrics are tokenizer relative. So the suite's only hard gate depends on an artifact from outside this repository.
Phase 6 names the mechanism already: re-pulling a model tag can change behavior with no change to this repo. Applied to a reported metric, that is a re-baseline. Applied to a gate, it is a build failing on an empty diff. §1.3 argues that exact false failure is what turns a gate into a disabled gate.
§ 2's existing discipline covers the fix: "model repo plus revision, never a friendly name". It needs saying for the CI tokenizer rather than only for the served model.
|
|
||
| This covers the "Did it get the right answer?" and "Did it follow the intended workflow?" groups, which share one piece of instrumentation. The MCP client sees every `tools/call`, so wrapping `client.callTool` to record `{ name, arguments, startedAt, durationMs, isError, resultTokens }` produces the whole trajectory, and every workflow metric is a query over that list. | ||
|
|
||
| - **★ `outcomeMatch`** asserts on the _structured_ result of the final `execute_query`, never the model's prose: exact `total`, exact set of returned primary keys, exact aggregation buckets. The frozen dataset makes these exact and the fingerprint check ([§4.1](#41-working-against-the-frozen-dataset)) licenses the exactness. The strongest single signal in the suite. |
There was a problem hiding this comment.
one more! execute_query accepts aggregationsFilterThemselves as a tool parameter, defaulting to "false". if the model sets the value per call, that will change bucket counts. Two suggested edits:
- record which value of
aggregationsFilterThemselvesthe expectations were derived under - assert that value in the case's
forbiddenchecks (line 277), so a run that used a different one is reported just as a finding rather than as an actual bucket mismatch
why: by default, a facet's own filter clauses are dropped from its own bucket counts, and every other clause still applies. e.g. filter on "sex" and on "disease", and the "sex" buckets are computed with the "sex" clause dropped while the "disease" clause is kept... set the flag "true" and every clause applies to every facet, including its own.
an exact bucket assertion is only defined against a fixed value of that flag. evals:bootstrap-expectations derives buckets by running a SQON directly... at run time the model chooses.
when those differ the buckets differ, and outcomeMatch records it as a comprehension failure. answerDispersion takes this worse. one phrasing leads the model to set the flag and another does not, so dispersion rises with nothing about the question misunderstood.
important: the same semantics are worth stating in §4.1 for whoever reviews the derivation diff. Unfiltered-looking counts beside a filtered query invite a correction that would break the case permanently. i.e "oh this looks broken, let me adjust it manually"
There was a problem hiding this comment.
Great catch! Will update to explicitly state aggregationsFilterThemselves as part of the recorded values for each case, and suggest adding the inverse to a case's forbidden checks 🙂
mistryrn
left a comment
mistryrn
left a comment
There was a problem hiding this comment.
Thanks for the feedback, @justincorrigible ! Initial responses below, before I start working on the changse you suggested 🙏
|
|
||
| - **Models are open-weight, served from hardware the team controls**, in small, medium, and large tiers. Sampling parameters (temperature, top-p, seed) are configurable, and **the client is ours too, so sampling is fixed at greedy**: temperature 0, one fixed seed, everywhere. Query translation has one right answer, so sampling would add variance with no upside. | ||
| - **Data lives on a separate testing ES instance and Arranger server**, not in the repo. The dataset is real, not synthetic, and it is **frozen**: it is not reindexed or mutated between runs. | ||
| - **The MCP server may be local, or a separately deployed instance.** The suite must support both, selected by configuration. |
There was a problem hiding this comment.
Great call, I hadn't considered the future implications of Usher/identity on Arranger when hashing out this spec. Will make the suggested edits 🙏
|
|
||
| Everything here goes into a **run manifest** written alongside every results file. Two runs are only comparable if their manifests differ in exactly the dimension being studied, so the comparison CLI warns loudly when they differ anywhere else. | ||
|
|
||
| **Test environment and case set.** The fingerprint covers the dataset and the catalogue configuration together, because either one changing invalidates a baseline the same way: per index, the document count, the mapping hash, and a checksum over a fixed aggregation, plus the `/introspection` output hashed. All of it obtained through Arranger, never ES ([§4.1](#41-working-against-the-frozen-dataset)). The case set is hashed separately. |
There was a problem hiding this comment.
Another good call, I was unaware of getServerSideFilter and had to do some reading into that. I think it brings up other questions for me regarding how MCP should handle Arranger identity/auth. I'll need to spend some more time thinking about this before I action on this edit suggestion.
Will ping you with my understanding of it and my approach before I make any changes for this one 🙏
|
|
||
| This covers the "Did it get the right answer?" and "Did it follow the intended workflow?" groups, which share one piece of instrumentation. The MCP client sees every `tools/call`, so wrapping `client.callTool` to record `{ name, arguments, startedAt, durationMs, isError, resultTokens }` produces the whole trajectory, and every workflow metric is a query over that list. | ||
|
|
||
| - **★ `outcomeMatch`** asserts on the _structured_ result of the final `execute_query`, never the model's prose: exact `total`, exact set of returned primary keys, exact aggregation buckets. The frozen dataset makes these exact and the fingerprint check ([§4.1](#41-working-against-the-frozen-dataset)) licenses the exactness. The strongest single signal in the suite. |
There was a problem hiding this comment.
Great catch! Will update to explicitly state aggregationsFilterThemselves as part of the recorded values for each case, and suggest adding the inverse to a case's forbidden checks 🙂
35bbb46 to
fe75785
Compare
* Adds v1 of the MCP server platfom testing plan document (`.dev/docs/mcp-platform-testing.md`)
* Added anonymous-principal assumption * Declared access-control posture * Specified that tokenizer is pinned separately * Updated introspection hashing to strip volatile fields * Added `params` field to case sets * Explicitly stated "human as a judge" for evaluating model-judged outputs
fe75785 to
494edef
Compare
2 participants
Summary
Adds v1 of the MCP server platfom testing plan document (
.dev/docs/mcp-platform-testing.md).Issues
Description of Changes
Adds a new dev doc which outlines the v1 plan for harnessed end-to-end testing of the entire Arranger MCP platform, including architecture, research, implementation, risks, and mitigation.
Docs
.dev/docs/mcp-platform-testing.mdSpecial Instructions
N/A
Readiness Checklist
.env.schemafile and documented in the README