diff --git a/.gitignore b/.gitignore index 50fff47..26ac61c 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,6 @@ build mcp.yaml oas/** mise.toml +evals/.venv/ +evals/logs/ +__pycache__/ diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..496a4a2 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,265 @@ +# Skill evals + +See [../EVALS.md](../EVALS.md) for the why and the overall plan. This directory is the +harness ([Inspect AI](https://inspect.aisi.org.uk)). + +## Setup (once) + +```bash +# from the repo root +go build -o .build/chip ./cmd/chip + +cd evals +python3 -m venv .venv +.venv/bin/pip install -r requirements.txt +``` + +Credentials — same resolution as chip itself: `COLLIBRA_MCP_API_URL/USR/PWD` env vars, +falling back to the repo-root `mcp.yaml`. If chip works locally, the scorers work too. + +## Run the happy path + +1. Fill in `fixtures/data_product_create.yaml` (source table + expected outcome). +2. ```bash + export ANTHROPIC_API_KEY= + cd evals && source .venv/bin/activate + inspect eval tasks/data_product_create.py --model anthropic/claude-sonnet-5 + ``` +3. Browse the result: + ```bash + inspect view + ``` +4. Clean up before the next run (duplicate names break reruns): + ```bash + python scripts/cleanup.py # dry run + python scripts/cleanup.py --apply + ``` + +Useful flags: `--epochs 3` (repeat for stability), `--limit 1`, `--log-dir logs/`. + +### Hitting 429 rate limits? + +Every model call carries all 28 chip tool schemas (~20k input tokens), which can +exceed a low-tier key's per-minute input-token budget outright — the run then 429s +forever, no matter how long Inspect backs off. Options, in order of preference: + +1. Raise the key's tier / check workspace limits in the Anthropic console. +2. Trim chip's tool surface to what the skill needs (~halves the request size; + slightly less faithful to real clients, which see every tool): + ```bash + export EVAL_ENABLED_TOOLS="list_collibra_skills,load_collibra_skill,search_asset_keyword,get_asset_details,get_table_semantics,prepare_create_asset,create_asset,edit_asset,list_data_contract,pull_data_contract_manifest,push_data_contract_manifest" + ``` + +## Layout + +```text +tasks/ Inspect task definitions +scorers/ end_state.py + seeded.py (REST assertions), + trajectory.py + skill_choice.py (transcript assertions); + the skill arms use skill_choice.py only +solvers/ seed.py — creates a throwaway star schema per rollout +fixtures/ per-environment values; read ONLY by tasks/data_product_create.py +scripts/ cleanup.py (fixture-based), teardown_seeded.py (seeded arm) +docs/ pipeline.md (start here — the map), then the per-arm writeups: + skill_lookup.md, skill_match.md, skill_adherence.md +``` + +## Test arms + +**New here? Read [docs/pipeline.md](docs/pipeline.md) first** — it draws the eleven +steps between a user's prompt and a created Data Product, brackets the stretch each +arm covers, and links onward to the per-arm docs. + +`data_product_create_happy_path` measures everything at once, so a failure says +*something* is wrong without saying what. `tasks/skill_arms.py` splits it into +arms that each remove one layer of scaffolding — where the score drops tells you +which thing to fix. + +| Arm | Scaffolding | A drop here implicates | Writes? | Rollouts | +| --- | --- | --- | --- | --- | +| `skill_lookup` | chip's **real** `initialize` instructions | `skills.Instructions` in `pkg/skills/register.go` | no | 4 × 5 | +| `skill_lookup_no_instructions` | none — tool schemas only | (the control for the above) | no | 4 × 5 | +| `skill_match` | "discover skills first" | the skill's frontmatter `description` | no | 8 × 5 | +| `skill_adherence` | skill named outright | the `SKILL.md` body | **yes** (seeds + tears down its own) | 1 × 3 | + +The first two are one arm under two instruction conditions. `skill_lookup` is +the production number; the control makes it interpretable, and the *gap* is the +finding: + +| control | production | Means | Do | +| --- | --- | --- | --- | +| low | high | the instructions carry the behaviour | they're load-bearing — don't casually edit | +| high | high | tool descriptions already do the work | the ~2.2k-char text is dead weight on every request | +| low | low | the text is being ignored | rewrite it | + +Pass both task specs to a single `inspect eval` so they share a model, a moment +and an instance — then the gap is attributable to the instructions text alone. +Run on different days and drift between the runs lands in the gap and reads as +signal. + +The two cheap arms run **different datasets**, because they ask different +questions. Both derive from one shared `CASES` pool: `skill_match` runs all of it; +`skill_lookup` runs `LOOKUP_CASES`, the subset +carrying a `request_shape` — one case per *shape* of request (multi-step write, +graph traversal, simple read), which is the only axis that changes whether a model +bothers to look for a skill. Graph traversal carries **two** prompts, because it is +the shape that failed under both instruction conditions and one prompt cannot +separate "the shape defeats the instructions" from "that wording does". Adding routing distractors there would ask the same +question again at extra cost, and adding `no-skill-needed` would be actively +wrong: not looking is the *correct* answer on that prompt, but +`any_skill_discovered` counts "never looked" as a failure, so including it caps a +perfect model at 5/6. Statistical power on this arm comes from epochs, not from +more prompts. + +**Each arm scores only its own question.** `skill_lookup` runs +`skill_discovery_first`; `skill_match` runs `skill_selected`. Reading the two +together separates "never looked" from "looked and chose wrong". `skill_lookup` +deliberately does **not** run `skill_selected`: there it fails both when the model +never looked *and* when it looked and chose wrong, making it roughly +`any_skill_discovered` × `skill_match`'s `skill_selected` — a product of two factors +already measured separately and more cleanly, which cannot be decomposed once +blended. + +`skill_discovery_first` rather than `any_skill_discovered` is the lookup headline +because a skill consulted *after* the first tool call did not guide it — the model +has already committed to an approach. `any_skill_discovered` cannot tell that apart +from healthy behaviour, and as a binary over "ever looked" it has little headroom: +a model that always looks eventually pins it at 1.000 whatever the instructions +say. Both scorers are kept, since discovered-but-late and never-looked need +different fixes. Details in +[docs/skill_lookup.md](docs/skill_lookup.md#why-skill_discovery_first-is-the-headline). + +For the same reason each arm stops as soon as its own question is answered. +`skill_lookup` ends on the first `list_collibra_skills` **or** `load_collibra_skill` +(either answers "did it look?"); routing must see the load itself, or there is no +skill to score. + +One pool rather than a list per arm keeps `any_skill_discovered` comparable across +the two — the arms only line up if their shared prompts are identical — and gives +one source of truth so a reworded prompt can't drift between them unnoticed. + +```bash +# cheap: read-only, parallel-safe, no cleanup needed. +# both instruction conditions in one command, so they share a model and a moment +inspect eval tasks/skill_arms.py@skill_lookup \ + tasks/skill_arms.py@skill_lookup_no_instructions +inspect eval tasks/skill_arms.py@skill_match -T epochs=5 + +# expensive: writes to Collibra, but seeds unique names so it still runs in +# parallel — no --max-samples 1 needed (defaults to 3 epochs) +inspect eval tasks/skill_arms.py@skill_adherence +``` + +Two things make the cheap arms cheap and safe: + +- **Read-only tool surface** — six tools; no `create_asset`, `edit_asset`, or + contract push. They *cannot* mutate Collibra even if the model tries. That also + drops 25 of chip's 31 tool schemas, cutting most of the ~20k-token per-call + overhead behind the 429 problem above. +- **Early stop** — an `on_continue` hook ends the run the moment + `load_collibra_skill` returns, because that is when the routing question is + answered. Without it the model would keep going, work through the skill's + read-only discovery phases, and stop only on reaching a write it cannot perform + — dozens of turns that cost tokens and add no signal. Runs that never load a + skill are *not* cut short (that would manufacture false "never looked" + verdicts); they end via the submit tool or `message_limit`. + +Being write-free is why they can run at high epoch counts, which matters because +editing a skill `description` is the most common change and is fully covered here. + +### The harness's own prompt is trimmed + +Every arm passes `prompt=_sequential_prompt()` to `react()`, which is Inspect's +default assistant prompt minus one sentence: *"Prioritize parallel tool calls: +when operations are independent, run them in one response."* + +That instruction comes from the eval framework, not from chip, and it pushes +against the exact behaviour these arms measure — a skill is a playbook to consult +*before* acting, while batching independent calls means firing +`search_asset_keyword` in the same turn as `list_collibra_skills`, which +`skill_discovery_first` counts as not having waited. It matters for +`skill_adherence` too: parallel writes land in one message, and `write_sequence` +indexes into the call sequence to check ordering. + +It is built by subtraction from upstream's own constants rather than pasted, so a +reworded default still reaches the model, and it raises if the sentence is no +longer where it expects — running with the contamination silently restored is +worse than a loud failure. **Runs from before this change are not comparable with +runs after it** on any ordering metric. + +`skill_lookup` performs one throwaway MCP handshake and uses the +instructions chip actually ships (~2.2k chars) instead of the hand-written +paraphrase in `tasks/data_product_create.py` (~230 chars, more imperative, and +missing the "Exceptions:" paragraph). That is the production-fidelity number. + +### Precision, not just recall + +`CASES` in `tasks/skill_arms.py` includes cases where +`data-product-create` must **not** win — "Create a new Business Term for Churn +Rate" is the sharp one, since it contains *create* and *new*. + +The cases are inline rather than in `fixtures/`, and every one names **invented +assets** ("SALES.ORDERS", "Churn Rate"). Routing is decided from the wording of a +request, not from whether its nouns resolve, so **neither cheap arm reads any +fixture** — both run against any instance with `fixtures/data_product_create.yaml` +left blank. Sourcing a table name from that fixture would pull in its validation of +`expected.data_product_name` and `expected.grouped_tables`, gating the harness's +most portable arms behind its most environment-specific config. + +Without negatives you only measure recall, and a +`description` sharpened until it wins the happy path can start hijacking unrelated +requests. That regression is invisible to a single-skill eval and is a risk on +every skill edit. + +`Case.expect` is a tuple, so a case can accept **more than one** skill. +`ambiguous-governance` ("set an owner for it and make it discoverable to other +teams") legitimately reads as either `asset-edit` or `data-product-create`, and +both pass. What it catches is landing somewhere else entirely — that means the +routing signal is noise rather than a close call. Holding an ambiguous prompt to a +single answer would invent failures. + +## The adherence arm seeds its own data + +`skill_adherence` does not read the fixture. Each rollout creates a throwaway +community → domain → schema → 4 tables → 18 columns in **one** +`POST /import/json-job`, then tears it down with **one** cascading +`DELETE /communities/{id}`. Full details in +[docs/skill_adherence.md](docs/skill_adherence.md). + +```bash +inspect eval tasks/skill_arms.py@skill_adherence +python scripts/teardown_seeded.py # dry run +python scripts/teardown_seeded.py --apply +``` + +Three problems this solves at once, all of which came from the fixture's single +hardcoded table plus a name-based assertion: + +- **A false pass.** Looking a Data Product up *by name* means a leftover product + from an earlier run satisfies the check even when this run wrote nothing (the + agent correctly stops per hard rule 5, and still scored 4/4). `seeded_end_state` + anchors on the seeded table's UUID and walks outward, so another run's product is + simply unreachable. Structural, not procedural — it does not depend on a cleanup + step having run. +- **Forced serialization.** One shared name meant concurrent rollouts raced on + `create_asset`'s duplicate gating. Names are now unique per rollout, so this arm + is safe to run in parallel and with `-T epochs=N`. +- **A fragile assertion.** `expected.data_product_name` required the eval to predict + the model's generative naming. The graph walk needs no name at all. + +Teardown is now **hygiene rather than correctness**: skipping it can only leave +clutter, never cause a false pass. It runs from `logs/seed-manifest.jsonl` rather +than as a task hook, because a hook does not run when a rollout crashes — exactly +when assets get left behind. + +## ⚠️ `data_product_create_happy_path` must run with `--max-samples 1` + +Inspect expands `samples × epochs` into **one concurrently-executed queue** — it +does not run epochs sequentially. The original fixture-based task uses a single +shared asset name, so parallel rollouts race on `create_asset`'s duplicate gating: + +```bash +inspect eval tasks/data_product_create.py --epochs 3 --max-samples 1 +``` + +`skill_adherence` needs no such flag — that is the point of seeding. diff --git a/evals/docs/pipeline.md b/evals/docs/pipeline.md new file mode 100644 index 0000000..c5677f9 --- /dev/null +++ b/evals/docs/pipeline.md @@ -0,0 +1,208 @@ +# The pipeline, and which arm tests which part + +Start here. This is the map; the per-arm docs are the territory. + +Between a user typing a request and a Data Product existing in Collibra there are +twelve steps, and **any one of them can be the thing that broke**. Each arm of the +eval covers a different stretch, so a failing score points at a specific file to +edit rather than at "the skill doesn't work". + +## 1. The pipeline + +No tasks yet — just what actually happens, in order. + +```text + who what + ────────────── ──────────────────────────────────────────────────────── + + ── before the user types anything ── + 0 chip → model MCP handshake: chip returns its `instructions` text + 1 chip → model tools/list: the tool schemas reach the model + + ── the user types ── + 2 user → model "Create a Data Product from the table SALES.ORDERS" + 3 model decides whether this needs a skill at all + 4 model → chip list_collibra_skills → catalog of 7 skills + 5 model reads the descriptions, picks one + 6 model → chip load_collibra_skill(name) → the SKILL.md body + + ── now following the skill's seven phases ── + 7 model → chip Phase 1-3 locate the table, check existing + coverage, suggest related tables [reads] + 8 model Phase 4-5 collect governance metadata, propose + the full set of assets [reads] + 9 user → model Phase 6 single approval gate +10 model → chip Phase 7 create + verify [WRITES] + +11 model → user final answer: "Created , , " +``` + +Three things worth noticing: + +- **Steps 0-1 happen before the prompt.** The instructions text and the tool + schemas are already in the context window when the user types. +- **Step 3 is a decision nothing forces.** No code makes the model look for a + skill; the instructions text is only *persuasion*. +- **Only step 10 writes.** Steps 0-9 are read-only, which is what makes most of + this pipeline cheap and safe to test. + +## 2. `skill_lookup` and `skill_lookup_no_instructions` — steps 0-4 + +Does the model reach for a skill **before doing anything else**? + +```text + 0 chip → model MCP handshake: instructions text ◀── THE VARIABLE + 1 chip → model tools/list: 6 read-only tools + 2 user → model the prompt + 3 model decides whether this needs a skill ◀── MEASURED HERE + 4 model → chip list_collibra_skills ◀── must be the FIRST + action; run stops here +────────────────────────────────────────────────────── never reached ───────── + 5 model picks one + 6 model → chip load_collibra_skill + 7-10 the seven phases +11 model → user final answer +``` + +**"Before anything else" is the whole point.** A model that calls +`search_asset_keyword` first and reaches step 4 afterwards has already taken its +first action unguided — so `skill_discovery_first` is the headline, not +`any_skill_discovered`, which a late look still satisfies. See +[why that is the headline](skill_lookup.md#why-skill_discovery_first-is-the-headline). + +The two tasks are the **same diagram run twice**, differing only at step 0: +`skill_lookup` sends chip's real instructions text, `skill_lookup_no_instructions` +sends nothing. Neither number means much alone — the *gap* between them is what +tells you whether that text is doing any work. + +A drop implicates `skills.Instructions` in +[`pkg/skills/register.go`](../../pkg/skills/register.go). + +→ [**skill_lookup.md**](skill_lookup.md) for the cases, the control's +rationale, and how to read the gap. + +## 3. `skill_match` — steps 5-6 + +Given that it looks, does it pick the **right** skill? + +```text + 0 chip → model MCP handshake + 1 chip → model tools/list: 6 read-only tools + 2 user → model the prompt + 3 model decides whether this needs a skill ◀── HANDED OVER: a system + message orders it to look, + so this is held constant + 4 model → chip list_collibra_skills → 7 descriptions + 5 model reads the descriptions, picks one ◀── MEASURED HERE + 6 model → chip load_collibra_skill(name) ◀── run stops here +────────────────────────────────────────────────────── never reached ───────── + 7-10 the seven phases +11 model → user final answer +``` + +Step 3 is deliberately short-circuited: motivation is pinned at maximum so it +cannot contaminate the routing measurement. Step 6 is where the run stops, because +routing needs to know *which* skill — stopping at step 4 would leave nothing to +score. + +This arm runs **8 prompts, including ones where the skill must lose** — that is how +it measures precision and not just recall. A drop implicates the skill's +`description:` frontmatter. + +→ [**skill_match.md**](skill_match.md) for all eight cases and the +precision/recall trade-off. + +## 4. `skill_adherence` — steps 7-10 + +Given the right skill, is the **procedure** followed? + +```text + 0 chip → model MCP handshake + 1 chip → model tools/list: all 31 tools (writes included) + 2 user → model the prompt — NAMES the skill outright +────────────────────────────────────────────────────── skipped ─────────────── + 3 model decides whether this needs a skill + 4 model → chip list_collibra_skills + 5 model picks one +────────────────────────────────────────────────────────────────────────────── + 6 model → chip load_collibra_skill(named skill) ◀── premise check: must + happen, else this arm + measures nothing + 7 model → chip Phase 1-3 locate, coverage, kin ◀─┐ + 8 model Phase 4-5 metadata, proposal │ MEASURED HERE + 9 user → model Phase 6 approval (pre-granted) │ +10 model → chip Phase 7 create + verify [WRITES] ◀─┘ +11 model → user final answer +``` + +Naming the skill in the prompt deletes steps 3-5 from the run, which is exactly +what makes a failure attributable to the `SKILL.md` **body** and nothing upstream. + +Because step 10 writes, this arm seeds its own throwaway star schema per rollout +and scores the resulting graph by walking outward from the seeded table's UUID — +never by looking a Data Product up by name. + +→ [**skill_adherence.md**](skill_adherence.md) for the seeding, the four end-state +checks, and the teardown. + +## 5. `data_product_create_happy_path` — steps 0-11 + +The original task (PR #108, `tasks/data_product_create.py`). It runs the whole +pipeline and gives you **one** number. + +```text + 0 ─────────────────────────────────────────────────────────────────────── 11 + everything above, as a single pass/fail +``` + +That number answers "does the feature work?" — which is worth knowing. What it +cannot answer is "which part broke?", and that is the entire reason the arms above +exist. + +## Summary + +| Steps | Arm | Headline scorer | A drop implicates | +| --- | --- | --- | --- | +| 0-4 | [`skill_lookup`](skill_lookup.md) | `skill_discovery_first` | the `instructions` text | +| 0-4 | [`skill_lookup_no_instructions`](skill_lookup.md#why-the-control-earns-its-cost) | `skill_discovery_first` | *(nothing — it is the control)* | +| 5-6 | [`skill_match`](skill_match.md) | `skill_selected` | the `description:` frontmatter | +| 7-10 | [`skill_adherence`](skill_adherence.md) | `seeded_end_state` | the `SKILL.md` body | +| 0-11 | `data_product_create_happy_path` | `end_state` | *headline only* — diagnose with the rows above | + +Work **top-down**: the first failing row is the one to fix, because every row below +it starts from a premise the row above just disproved. + +| lookup | match | adherence | Diagnosis | +| --- | --- | --- | --- | +| 30% | 95% | 90% | The skill is fine and never gets invoked. Fix the **instructions text**. | +| 95% | 30% | 90% | It looks, then picks wrong. Fix the **`description`**. | +| 95% | 95% | 40% | Right skill, wrong execution. Fix the **`SKILL.md` body**. | +| 95% | 95% | 95% | Healthy. | + +## Running it + +```bash +# cheap: read-only, 6-tool surface, 1-3 tool calls per rollout +inspect eval tasks/skill_arms.py@skill_lookup \ + tasks/skill_arms.py@skill_lookup_no_instructions +inspect eval tasks/skill_arms.py@skill_match + +# expensive: writes to Collibra, seeds and tears down its own data +inspect eval tasks/skill_arms.py@skill_adherence +python scripts/teardown_seeded.py --apply +``` + +Steps 0-6 are cheap; steps 7-10 are not (~30-50 tool round-trips against the full +tool surface, plus seeding). That asymmetry is why the arms you can afford to run +on every commit are the ones covering the top of the pipeline — which is also where +the most common edit lands, a reworded `description`. + +## See also + +- [`skill_lookup.md`](skill_lookup.md) — steps 0-4: does it reach for a + skill unprompted? +- [`skill_match.md`](skill_match.md) — steps 5-6: having looked, does it choose + correctly, and does it *avoid* choosing when it shouldn't? +- [`skill_adherence.md`](skill_adherence.md) — steps 7-10: given the right skill, is + the procedure followed? +- [`../README.md`](../README.md) — setup, credentials, cost and rate-limit notes. diff --git a/evals/docs/skill_adherence.md b/evals/docs/skill_adherence.md new file mode 100644 index 0000000..e20d020 --- /dev/null +++ b/evals/docs/skill_adherence.md @@ -0,0 +1,349 @@ +# `skill_adherence` — given the right skill, is it followed correctly? + +> Covers **steps 7-10** of the pipeline. See +> [pipeline.md](pipeline.md) for how this arm fits with the others. + +## Why this task exists + +The existing `data_product_create_happy_path` measures everything at once, so a +failure tells you *something* is wrong without telling you *what*. Three +independent things can break: + +1. the model never looked for a skill → the **instructions text** is failing +2. it looked but picked the wrong one → the skill's **`description`** is failing +3. it picked right but executed badly → the **`SKILL.md` body** is failing + +`skill_adherence` isolates **(3)**. The skill is named outright in the prompt, so +routing is removed from the equation and a failure here is the body's fault. + +## What it does + +One sample. Each rollout: + +1. **Seeds** its own throwaway star schema (below). +2. Runs the agent against the full chip tool surface with the skill named. +3. **Scores** the resulting graph by walking outward from the seeded table. +4. Is torn down by `scripts/teardown_seeded.py`. + +```bash +inspect eval tasks/skill_arms.py@skill_adherence +inspect eval tasks/skill_arms.py@skill_adherence -T epochs=3 +python scripts/teardown_seeded.py --apply +``` + +### No system message + +Unlike the two cheap arms, this one ships **no** `system_message`. The cheap arms +use one to tell the model to discover and load the matching skill; here the prompt +already names the exact skill, so that instruction decides nothing. It is also the +*paraphrase* of chip's shipped instructions rather than the real text (see +[skill_lookup.md](skill_lookup.md#two-conditions-two-tasks)), and this arm exists to +isolate the `SKILL.md` **body** — every extra piece of scaffolding is one more +variable that isn't it. + +That makes verifying the load essential rather than optional — see +[`skill_selected` as a precondition](#skill_selected-as-a-precondition). + +## Seeding: one request + +Every rollout creates its own graph, named `TEST__` where `` +comes from the rollout's uuid — so parallel epochs cannot collide: + +```text +TEST__COMMUNITY +├── TEST__DP_CATALOG (Data Product Catalog — empty, the agent writes here) +└── TEST__SALES_DOMAIN (Physical Data Dictionary) + └── TEST__SALES (Schema) + ├── TEST__ORDERS (Table, 6 columns, has a Description) + ├── TEST__CUSTOMER (Table, 4 columns) + ├── TEST__PRODUCT (Table, 4 columns) + └── TEST__STORE (Table, 4 columns) +``` + +`ORDERS` carries `CUSTOMER_ID` / `PRODUCT_ID` / `STORE_ID` deliberately: Phase 3 of +the skill picks related tables by *plain-language reasoning over names*, not by +foreign keys or lineage, so the shared key columns are what give that reasoning +something to work with. + +All of it — community, domain, assets, attributes **and relations** — is one +`POST /rest/2.0/import/json-job`. Verified live: 23 assets, 22 relations, 1 +attribute, 0 errors. + +```text +POST /rest/2.0/import/json-job multipart/form-data + file = @seed.json + fileName = seed.json + continueOnError = false # a half-seeded graph is worse than none + simulation = true # optional dry run; creates nothing +→ 200 {"id": ""} + +GET /rest/2.0/jobs/{jobId} # poll to COMPLETED/ERROR/CANCELED +GET /rest/2.0/import/results/{jobId}/errors # must be {"total": 0} +GET /rest/2.0/assets?domainId= # one call → every seeded UUID +``` + +Relations are declared **on the source asset**, keyed by relation type and +direction. This format is undocumented in the OpenAPI schema — it was confirmed +against a live instance (`RELATION added=22`): + +```json +"relations": { ":TARGET": [ { "name": "...", "domain": {...} } ] } +``` + +### Verified type ids + +All system UUIDs, i.e. out-of-the-box and stable across instances. Probed live +rather than assumed. + +| Thing | Id | +| --- | --- | +| Asset type `Schema` | `00000000-0000-0000-0001-000400000002` | +| Asset type `Table` | `00000000-0000-0000-0000-000000031007` | +| Asset type `Column` | `00000000-0000-0000-0000-000000031008` | +| Domain type `Physical Data Dictionary` | `00000000-0000-0000-0000-000000030011` | +| Domain type `Data Product Catalog` | `00000000-0000-0000-0000-000000050010` | +| `Description` attribute type | `00000000-0000-0000-0000-000000003114` | +| `Schema` **contains** `Table` | `00000000-0000-0000-0000-000000007043` | +| `Column` **is part of** `Table` | `00000000-0000-0000-0000-000000007042` | +| `Data Product Port` **is implemented as** `Table` | `00000000-0000-0000-0000-000000050042` | +| `Data Product` **exposes data as** `Port` | `00000000-0000-0000-0000-000000050040` | +| `Data Contract` **governs functioning of** `Port` | `00000000-0000-0000-0000-000000050044` | + +**Direction is not guessable and matters.** `Table→Column` returns *zero* relation +types — the only one is `Column is part of Table`, with the **column** as source. +Same story for ports: the *Port* is the source of `is implemented as`, and the +*Data Product* is the source of `exposes data as`. Getting any of these backwards +produces a silently empty graph. + +## The agent needs somewhere to write + +`Data Product`, `Data Product Port` and `Data Contract` are only allowed in a +domain of type **Data Product Catalog**. On an instance with none, every +`prepare_create_asset` comes back with + +```text +"domain is required for asset type \"Data Product\". Pick one from domainOptions + and call again. Filtered to Data Product Catalog domains." domainOptions: [] +``` + +and the agent cannot reach Phase 7 at all. So the seed creates an **empty +`TEST__DP_CATALOG` domain** alongside the source domain. Its only job is to +exist. `seed()` verifies it resolves after the import and fails loudly if not — +otherwise a missing domain type would surface as the agent finding nowhere to +write, and the arm would report that as a `SKILL.md` failure. + +### Why the prompt names it + +Phase 4 has the **user** pick the target domain from every domain that accepts a +Data Product, never defaulting silently. Under parallel epochs that list contains +*every other rollout's* seeded catalog, so the agent could write into one of them. +The prompt therefore supplies the domain, exactly as it already supplies the +user's sign-off — `{target_domain}`, substituted by `seed_star_schema` at run +time. + +Two consequences worth knowing: + +- **Everything lands inside the seeded community**, so the teardown cascade + removes it and the sweep stage below becomes a fallback rather than the main + path. +- **The arm no longer tests whether the agent can find a catalog domain unaided.** + That is a deliberate trade: domain choice is Phase 4 *user input*, and what this + arm measures is Phase 7's writes and relations. + +## Scoring: anchored on the seeded table, never on a name + +`seeded_end_state` starts from `metadata["seed"]["fact_table_id"]` and walks +outward. Four equally-weighted checks: + +| Check | How | +| --- | --- | +| `port_exposes_table` | a `Data Product Port` implements the seeded fact table | +| `product_exposes_port` | a `Data Product` exposes that Port | +| `tables_grouped` | the Port also implements **every** seeded dimension table, compared by UUID | +| `contract_governs` | a `Data Contract` governs the Port | + +Why anchored rather than name-based — this is the whole reason the arm was +rewritten. The original looks a Data Product up **by name**, so a leftover product +from an earlier run satisfies `product_exists` even when the current run wrote +nothing at all: the agent correctly stops per hard rule 5 ("existing coverage is a +stop condition"), writes nothing, and still scores 4/4. A false pass in the worst +direction. Anchoring makes that impossible rather than merely unlikely — another +run's product is not reachable from *this* run's table, with no dependence on a +cleanup step having run. + +Both directions are verified: the scorer returns **0.0** on a freshly seeded graph +with no agent run, and **1.0** once a Port/Product/Contract are wired up. +`grouped_fraction` is reported in metadata so a partially-grouped result is visible +even though the check itself is boolean. + +## `skill_selected` as a precondition + +This arm also runs `skill_selected` — **not** as a routing measurement (routing is +handed over by the prompt) but as a check on the arm's own premise. It should read +**~100%**; a dip means the model never loaded the skill it was told to use. + +That matters because without it the failure is invisible. If the model works from +the skill's *name* rather than its contents, the arm silently stops measuring the +`SKILL.md` body and starts measuring what the model can do unaided — and every +other scorer still reports a normal-looking number. Nothing else catches it: +`skill_citation_consistency` only fires when the model *cites* a skill it never +loaded, so a model that simply stays quiet about it passes. + +It is the exact counterpart of `any_skill_discovered` in +[`skill_match`](skill_match.md): a sanity check that should be pinned at the +ceiling, and whose dip invalidates everything measured beside it. + +`skill_selected` checks *which* skill was loaded but not *when*, so this arm also +runs **`skill_discovery_first`**, which requires the load to be the opening move +with nothing beside it. + +Read it as **informational here, not as a premise check.** It is strict about +batching: a model that calls `load_collibra_skill` and `search_asset_keyword` in +the same turn fails it, and that has happened on a rollout which then followed +Phase 7 correctly and scored 4/4 on `seeded_end_state`. So a low number does not +invalidate the body measurement on this arm — it says the model started resolving +the table in parallel with reading the playbook, which is a different (and milder) +concern than never reading it. + +The premise that actually matters here is covered elsewhere: `skill_selected` +confirms the named skill was loaded at all, and `write_sequence` confirms nothing +was written before the procedure ran. + +## `columns_discovered` — a check that exists because the skill has a bug + +[SKILL.md:48](../../pkg/skills/files/collibra/data-product-create/SKILL.md#L48) +tells the model to read the column list "from the **outgoing** `Column` relations". +That is wrong. There is no `Table→Column` relation type in Collibra's model at all; +the only one is `Column is part of Table` with the column as source. So columns +always arrive as **incoming** relations, and `get_asset_details` returns them under +`incomingRelations`. + +A model following Phase 1 literally looks in `outgoingRelations`, finds nothing, +and concludes the table has no columns. That breaks Phase 2's Data Set check and +strips the column context Phase 1 is supposed to feed into generated attributes. + +Crucially, **none of the four end-state checks would notice** — ports, product, +contract and grouped tables do not depend on columns. So the bug degrades quality +invisibly, which is exactly why it needs its own check. `columns_discovered` passes +if the agent names ≥2 of the seeded fact table's columns in its own prose; +deliberately lenient, since the skill only needs "a few" columns for the Data Set +check. + +The skill is **intentionally left unfixed** so the eval demonstrates the bug. + +## `write_sequence` — the skill's write order, not just its end state + +A **transcript** check, so it catches what `seeded_end_state` cannot: a run that +arrives at the right final graph by a route the skill forbids. It asserts the two +orderings Phase 7 states outright: + +| Assertion | Why it matters | +| --- | --- | +| Port→table `is implemented as` links precede `init_data_contract` | Phase 7: *"the tables must be linked to the Port before init so the generated manifest covers them"*. Link afterwards and the manifest silently omits them — invisible in the end state. | +| `init_data_contract` precedes `push_data_contract_manifest` | init produces the base manifest (`0.0.1`) that the push is meant to improve on (`0.0.2`). | + +Plus one assertion that is not about ordering: **a rollout with no `create_asset` +wrote nothing**, which on this arm is a failure rather than a vacuous pass. + +### What it deliberately does not check + +It does **not** require every create to precede every relation edit. Phase 7 +mandates the opposite — create Product and Port, wire their relations, *then* +create the Data Contract, then link it — so that rule fails a correct trajectory. +`scorers/trajectory.py::write_order` (PR #108, untouched and still used by that +task) enforces exactly that non-existent rule, and attributes it to "hard rule 1", +which is about **confirming once**, not about write order. + +`write_order` has a second, independent problem: it matches tool names with a bare +`endswith`, so the read-only `prepare_create_asset` counts as a write. A rollout +that only prepared and created nothing therefore scores CORRECT there — the same +false-pass class `seeded_end_state` was built to eliminate, and worst on the write +arm. `write_sequence` resolves names through `_canonical` (longest match) and reads +`edit_asset`'s structured `operations` payload rather than substring-searching the +argument blob. + +The name differs on purpose: the two are not interchangeable, and a shared name +across arms in a log would imply they were. + +### Known limits + +- **Ordering is by call index**, so a model batching writes into a single message + defeats it. `_sequential_prompt()` removes react's parallel-tool-calls directive + partly for this reason. +- **`create_asset` with `allowDuplicate=false`** is sometimes used purely to + resolve a name to a UUID; it returns `duplicate_found` and writes nothing, yet + still satisfies the no-writes guard. `seeded_end_state` is the robust check for + "nothing was written". +- **Manifest *content* is not checked** — including the version number, so pushing + `0.0.1` instead of the prescribed `0.0.2` passes. + +## Teardown: two stages, and the order matters + +```bash +python scripts/teardown_seeded.py # dry run, lists everything +python scripts/teardown_seeded.py --apply +python scripts/teardown_seeded.py --tag --apply +``` + +1. **Sweep the model-created assets** — Data Contract → Data Product → Port. Now + that the prompt names a seeded target domain these normally land *inside* the + seeded community, so stage 2 already covers them and this stage is a + **fallback**: it catches a run that wrote somewhere else anyway, which is still + possible since nothing forces the agent to obey the named domain. It is only + able to find them by navigating from the seeded fact table, so it must run + **first** — once the community is gone, so is the anchor. +2. **One cascading delete** — `DELETE /rest/2.0/communities/{id}` removes both + domains and every seeded asset. + +Driven by `logs/seed-manifest.jsonl`, written at seed time, rather than a post-run +hook — a hook does not run when a rollout crashes, which is precisely when assets +are left behind. + +Teardown is **hygiene, not correctness**. Because names are unique per rollout, a +skipped teardown can only leave clutter; it can no longer cause a false pass. + +## How it runs + +- **Repetitions:** `epochs=3` by default; `-T epochs=N` to change. Three rather + than the cheap arms' five because a rollout here costs ~100× one of theirs — but + not one, because a single rollout gives an outcome rather than a rate, and both + `seeded_end_state` and `columns_discovered` declare `stderr()`, which is + undefined at n=1. +- **Parallelism:** unlike the fixture-based task this arm needs **no + `--max-samples 1`** — each rollout seeds its own uniquely-named graph, so + concurrent rollouts cannot collide on `create_asset`'s duplicate gating. That is + the point of the seeding design. +- **Cost:** the full flow is ~30–50 tool round-trips against all 31 tool schemas, + plus ~4 seeding requests and a few seconds of job polling — call it $5–15 per + rollout before prompt caching, so ~$15–45 at the default 3 epochs, and 23 seeded + assets per rollout for teardown to sweep. This is the expensive arm; the cheap + ones ([lookup](skill_lookup.md), [match](skill_match.md)) are where a + PR gate belongs. +- `message_limit=120` is a runaway-loop backstop. + +## No fixture + +This arm reads no per-environment values. `fixtures/data_product_create.yaml` +remains only for `tasks/data_product_create.py` (PR #108, untouched). Note that its +committed `source_table` UUID 404s on at least one dev instance, so that task is not +portable between environments — another reason the adherence arm seeds instead. + +## Verification status + +Verified live: one-request import (23 assets / 22 relations / 1 attribute / 0 +errors); relation directions as chip reports them (`incomingRelations` carries the +Schema *and* all columns); prompt substitution of `{source_table}` at setup time; +`seeded_end_state` scoring 0.0 without an agent graph and 1.0 with one; the teardown +sweep finding Contract/Product/Port in dependency order; the cascade leaving 0 +`TEST_*` assets and 0 `TEST_*` communities. + +**Not yet run:** a full agent rollout, which needs `ANTHROPIC_API_KEY` and API +spend. The scorers are verified against real Collibra graphs, not against real model +output. + +## See also + +- [`skill_lookup.md`](skill_lookup.md) — does the model look for a skill at all? +- [`skill_match.md`](skill_match.md) — having looked, does it choose correctly? + +- [`pipeline.md`](pipeline.md) — the map: all eleven steps, and which arm brackets which. diff --git a/evals/docs/skill_lookup.md b/evals/docs/skill_lookup.md new file mode 100644 index 0000000..26cb580 --- /dev/null +++ b/evals/docs/skill_lookup.md @@ -0,0 +1,367 @@ +# `skill_lookup` — does the model reach for a skill on its own? + +> Covers **steps 0-4** of the pipeline. See +> [pipeline.md](pipeline.md) for how this arm fits with the others. + +## Why this task exists + +The existing `data_product_create_happy_path` measures everything at once, so a +failure tells you *something* is wrong without telling you *what*. Three +independent things can break: + +1. the model never looked for a skill → the **instructions text** is failing +2. it looked but picked the wrong one → the skill's **`description`** is failing +3. it picked right but executed badly → the **`SKILL.md` body** is failing + +`skill_lookup` isolates **(1)**. It is the only task that can detect a skill +that is perfectly written and perfectly named but never triggers — which is +arguably the most important failure mode, since a skill that never loads is +indistinguishable from a skill that doesn't exist. + +### Why the original task cannot answer this + +`tasks/data_product_create.py` injects this system message before every run: + +> You are connected to a Collibra MCP server. Before composing any multi-step +> Collibra workflow, discover the relevant skill guide by calling +> `list_collibra_skills` and load the matching one with `load_collibra_skill`, +> then follow it. + +That is a direct order to go and discover a skill. So the original task **cannot** +measure whether the model would have done so unprompted — motivation is handed to +it for free, and its `skill_loaded` scorer can only ever confirm that the model +obeyed an explicit instruction. Every run starts past the point this task is +trying to measure. + +One distinction worth keeping straight, because it decides what each arm is good +for. The message says "load the matching **one**" without naming which, so the +model still has to choose `data-product-create` over `asset-create`, `asset-edit`, +`discovery` and the rest. **Routing is still genuinely tested there.** It is only the *decision to look* that the prompt short-circuits — which is exactly the gap this task +fills, and why `skill_match` reuses that same message rather than removing it. + +The message is also a paraphrase rather than the real thing, which matters +separately — see [Two conditions, two tasks](#two-conditions-two-tasks). + +## What it does + +Four prompts, run against chip with a read-only tool surface, measuring whether +the model reaches for `list_collibra_skills` / `load_collibra_skill` — and whether +it does so **before** taking any other action. Which skill it picks is +[deliberately not scored here](#why-this-arm-does-not-run-skill_selected). + +| Sample id | Request shape | Prompt | Expected skill | +| --- | --- | --- | --- | +| `data-product` | multi-step write | Create a Collibra Data Product from the table SALES.ORDERS | `collibra/data-product-create` | +| `lineage-trace` | graph traversal | Where does the Monthly Recurring Revenue KPI come from? | `collibra/lineage` | +| `lineage-impact` | graph traversal | If we change ORDERS.DISCOUNT_PCT, which reports and dashboards break? | `collibra/lineage` | +| `semantic-discovery` | simple read | What customer data do we have in the catalog? | `collibra/discovery` | + +### Why four, and why these four + +This arm asks *"does the model consult a skill before acting?"* — a question that +varies with the **shape** of the request (how visible the multi-step-ness is), not +with which skill happens to be correct. So it runs one case per shape. Adding more +prompts would ask the same question again at extra cost; statistical power here +comes from **epochs**, not from sample count. + +**Graph traversal is the deliberate exception, with two prompts.** One prompt per +shape cannot separate "this shape defeats the instructions" from "this sentence +does", and traversal is the shape where that ambiguity matters most, so it carries +a second wording. `lineage-impact` differs on every axis available — downstream +rather than upstream, impact-analysis framing, a column rather than a KPI as the +subject — so when the two agree the result is about the shape, not the phrasing. +The other shapes get a second prompt only when their result becomes contested +too. + +`LOOKUP_CASES` is therefore a subset of the shared `CASES` pool — those +carrying a `request_shape`. Two kinds of case are deliberately excluded: + +- **Routing distractors** (`business-term-create`, `attribute-edit`, + `ambiguous-governance`) exist to test *precision*, which this arm doesn't + measure. They belong to `skill_match`. +- **`no-skill-needed`** ("List the asset types available in this instance") would + be **actively wrong** here. chip's own instructions say a single obvious tool + call needs no skill, so *not* looking is the correct behaviour on that prompt — + but `any_skill_discovered` scores "never looked" as INCORRECT. Including it + would cap a perfectly-behaving model at 5/6 and make the arm's ceiling a lie. + +### No environment setup required + +Cases are defined inline in the task file rather than in `fixtures/`, and reference +**invented assets** ("SALES.ORDERS", "Monthly Recurring Revenue"). Nothing here +needs to exist in your Collibra instance: whether the model reaches for a skill is +decided from the *wording* of the request, not from what its nouns resolve to. + +This arm reads no fixture at all. Sourcing a table name from +`fixtures/data_product_create.yaml` would drag in that fixture's validation, which +requires `expected.data_product_name` and `expected.grouped_tables` — write-arm +values irrelevant here — and would gate the cheapest, most portable arm in the +harness behind its most environment-specific config. It runs against any instance +with that fixture left completely blank. + +### Relationship to `skill_match` + +`skill_match` runs all **eight** cases — including the distractors, the ambiguous +case and the negative — because precision and over-triggering are exactly what it +measures. The two arms divide the work cleanly: **this arm scores whether the model +looked; routing scores which skill it chose.** Read together they give a diagnosis +neither gives alone: + +| `skill_discovery_first` here | `skill_selected` in `skill_match` | Diagnosis | +| --- | --- | --- | +| 30% | 95% | The `description` is fine — the model just doesn't look. Fix the **instructions text**. | +| 95% | 30% | It looks and still picks wrong. Fix the **`description` frontmatter**. | +| 95% | 95% | Both healthy. | + +### Why this arm does *not* run `skill_selected` + +It would be a confounded number here. With no system message the model may never +look for a skill, and when it doesn't, `skill_selected` fails too — not because the +`description` is wrong, but because there was nothing to route. So in this arm it +measures P(looks **and** chooses right), which is approximately +`any_skill_discovered` × `skill_match`'s `skill_selected`: a product of two factors that +are each measured separately and more cleanly, blended into one number that can no +longer be decomposed. `skill_match` also runs eight cases against this arm's four, so it +is the better place to measure routing regardless. + +That is also why the run stops at the *listing*: once "which skill" is out of +scope, `list_collibra_skills` on its own answers this arm's question and the +subsequent load is wasted spend. + +One shared `CASES` pool rather than a list per arm keeps `any_skill_discovered` +comparable across the two — the arms only line up if their shared prompts are +identical — and gives one source of truth, so a reworded prompt cannot drift +between them unnoticed. + +See [`skill_match.md`](skill_match.md) for that arm's cases, its per-case +failure-mode table, and how to read the full arm ladder together, and +[`skill_adherence.md`](skill_adherence.md) for the rung below — which seeds its own +Collibra data so it can assert on a graph rather than a name. + +## Two conditions, two tasks + +```bash +# both, in one command — same model, same moment, two clean logs +inspect eval tasks/skill_arms.py@skill_lookup \ + tasks/skill_arms.py@skill_lookup_no_instructions + +inspect eval tasks/skill_arms.py@skill_lookup # production only +``` + +| Task | System message | What the number means | +| --- | --- | --- | +| `skill_lookup` | chip's **real** `initialize` instructions, fetched live over MCP | **Production fidelity.** Reproduces what an actual Claude Code user is in. | +| `skill_lookup_no_instructions` | **nothing** — tool schemas only | **Control.** Do the two skill tools' own descriptions attract a cold model with nothing telling it skills exist? | + +### Why the control earns its cost + +`skill_lookup` alone tells you *whether* it works, not *why*. It's a drug +trial: 90 recoveries out of 100 means nothing until you know how many recover +untreated. The gap is the finding: + +| control | production | Means | Do | +| --- | --- | --- | --- | +| low | high | the instructions carry the behaviour | load-bearing — don't casually edit | +| high | high | the tool descriptions already do the work | the ~2.2k-char text is dead weight on every request | +| low | low | the text is being ignored | rewrite it | + +Passing both task specs to one `inspect eval` keeps that comparison controlled: +same model, same moment, same instance, one variable. Run them on different days +and any drift between them — a model snapshot, a skill edit — lands in the gap and +reads as signal. + +### Why two tasks rather than one + +An earlier version crossed the 3 cases with both conditions into a 6-sample +dataset. That is more precise on paper — one log, guaranteed-identical +conditions — but varying a system message *per sample* is surprisingly expensive +in machinery: `system_message()` is task-level, so it needs a custom solver +reading `metadata`; and Inspect fixes a scorer's metrics at decoration time, so +reporting per-condition numbers needs a second scorer factory wrapping +`grouped(..., all=False)`. Plus the crossing loop and a mode allow-list. + +Two tasks need none of that — plain `system_message()`, plain +`any_skill_discovered()`, one sample per case — and because `inspect eval` accepts +several task specs, they still run together. What you give up is the single +blended log, which you did not want anyway: an average over a control and a +production run describes neither. + +The MCP handshake is `lru_cache`d, so constructing the task fetches the text once +however many times Inspect builds it. + +`skill_lookup` performs one throwaway MCP handshake against `.build/chip` +and reads the `instructions` field off the `initialize` response — the exact +string `pkg/skills/register.go` ships. It raises rather than falling back to a +hardcoded copy: a silent fallback is precisely how prompt drift hides. + +This matters because the harness's existing hand-written `SYSTEM_MESSAGE` is a +**228-character** paraphrase of a **2162-character** shipped text. The paraphrase +is more imperative and drops the `Exceptions:` paragraph — the documented escape +hatch that lets a model skip skill discovery. So it behaves as an *upper bound*, +not a forecast. + +## How to read the results + +The production number predicts real-world behaviour; no user ever connects +without server instructions. The **gap** is the causal contribution of the +instructions text — what writing it bought you. + +- **production ≈ control** → the instructions are on screen and being ignored. + Rewrite them. No other arm can tell you this, because every other arm hands the + model a reason to look for free. +- **production high, control low** → the instructions are load-bearing. Don't + casually edit them, and be aware that MCP clients which truncate or ignore + server instructions give users the floor experience. +- **control already high** → the tool descriptions carry it; instructions are + belt-and-braces. + +Read `skill_discovery_first` as the headline, not `any_skill_discovered` — see +below for why the two differ. + +## Why `skill_discovery_first` is the headline + +Both scorers ask about the same tools; they differ only on **when**. +`any_skill_discovered` passes if the catalog was consulted at any point in the +transcript. `skill_discovery_first` passes only if that was the model's opening +move. + +The stricter one is the arm's question. A skill is a playbook, so one consulted +*after* the first `search_asset_keyword` did not guide that search — the model has +already committed to an approach. `any_skill_discovered` cannot distinguish that +from healthy behaviour, and being a binary over "ever looked" it also has very +little headroom: a model that always looks eventually pins it at 1.000 whatever +the instructions say, leaving no room for a gap to appear in. + +Both are kept, because read together they localise the failure: + +| `discovery_first` | `any_discovered` | Diagnosis | +| --- | --- | --- | +| high | high | healthy — the playbook is consulted before work begins | +| **low** | high | the model looks, but only after acting. The instructions land, but not early enough to steer the first call. | +| low | low | the model never looks at all — the instructions are being ignored outright | + +### Parallel batches count as failures + +Issuing `list_collibra_skills` and `search_asset_keyword` in the **same turn** +scores INCORRECT, whichever order they appear in the batch. The model cannot have +read the playbook it is requesting in that same turn, so it did not wait — and +"did not wait" is the behaviour being measured. + +That makes the metric sensitive to anything nudging the model toward batching, +which is why `_sequential_prompt()` exists (see +[How it runs](#how-it-runs)): Inspect's `react()` ships an assistant prompt +telling the model to *"prioritize parallel tool calls"*, an instruction that comes +from the eval framework rather than from chip and pushes against the very ordering +this arm looks at. Runs made before that prompt was trimmed are not comparable +with runs made after it. + +## How it runs + +- **Repetitions:** `epochs=5` (override with `-T epochs=N`). 4 samples × 5 = + **20 rollouts** *per task*, so 40 when you run the pair. Inspect reduces the 5 + scores per sample with the default **`mean`** reducer, so a sample that routes + correctly 3/5 times scores 0.6. With + 1 epoch every sample is a coin flip and `stderr()` is undefined; 5 gives a rate + you can compare across skill edits. +- **Concurrency:** Inspect uses async concurrency, not threads. `max_samples` + defaults to **adaptive** in inspect-ai 0.3.251 — a dynamic limiter tracking + observed rate limits (`max_connections=10` as reference). Override with + `--max-samples N`. Epochs are **not** sequential: all 20 rollouts go into one + concurrent queue. Safe here because nothing writes. +- **Cost:** ~1–2 tool calls per rollout — the early stop below fires on the + listing, so the load is never paid for — against a 6-tool schema instead of 32. + +## How it terminates, and why it can't create anything + +Two independent mechanisms: + +1. **It has no write tools.** chip is launched with `--enabled-tools` limited to + six read-only tools (`list_collibra_skills`, `load_collibra_skill`, + `search_asset_keyword`, `get_asset_details`, `get_table_semantics`, + `list_asset_types`). `create_asset` isn't blocked — it's *absent from the + menu*, so writing is impossible even if the model tries. Verified: exactly 6 + tools registered, zero write tools. This also drops 25 of chip's 31 tool + schemas, cutting most of the ~20k-token-per-call overhead behind the harness's + documented 429 problem. + +2. **It stops as soon as the question is answered.** An `on_continue` hook ends + the run on the first `list_collibra_skills` **or** `load_collibra_skill` — + either one settles "did it look?", which is all this arm measures. Both count + because a model may skip the listing and load directly. Without the hook, + `react()` runs until the model calls `submit` or hits `message_limit`, so it + would work through the skill's read-only discovery phases and stop only on + reaching a write it can't perform: dozens of turns, zero extra signal. Runs + that never touch either tool are deliberately **not** cut short, since capping + those would manufacture false "never looked" verdicts; they end via `submit` or + the `message_limit=30` backstop. + + `skill_match` uses a stricter gate — it stops only on the load, because it + needs to know *which* skill was chosen. + +## Scorers + +| Scorer | Asserts | Metric | +| --- | --- | --- | +| `skill_discovery_first` | the model's **first** turn was catalog discovery and nothing else — **this arm's headline** | pass/fail | +| `any_skill_discovered` | `list_collibra_skills` or `load_collibra_skill` was called at all, whenever | pass/fail | +| `skill_citation_consistency` | the model never cites a skill in prose that it didn't actually load | pass/fail | + +The first two differ only in *when*, and reading them together localises the +failure: + +| `discovery_first` | `any_discovered` | Diagnosis | +| --- | --- | --- | +| high | high | healthy — the playbook is consulted before work begins | +| **low** | high | the model looks, but only after acting. The instructions are not landing early enough; a skill that arrives after the first tool call did not guide it. | +| low | low | the model never looks at all — the instructions are being ignored outright | + +`skill_selected` is deliberately absent — see +[Why this arm does *not* run `skill_selected`](#why-this-arm-does-not-run-skill_selected). + +`skill_citation_consistency` is **near-vacuous here** and confirmed so live: the +early stop fires before the model writes any prose, so it passed on all 30 +rollouts with `cited: none`. It is kept because it costs nothing and would catch +the model reading skill names out of the `list_collibra_skills` response and then +writing as though it were following one it never loaded — but do not read it as +signal on this arm. + +`skill_citation_consistency` catches the model reconstructing a procedure from a +skill's **name** without reading it — which can produce a plausible transcript for +entirely the wrong reason and is invisible to the end-state and write-order +scorers. It scans assistant messages only: a loaded `SKILL.md` lists siblings in +its `related:` frontmatter and `list_collibra_skills` returns every name, so +scanning tool output would manufacture citations the model never made. + +## Prerequisites + +```bash +go build -o .build/chip ./cmd/chip +cd evals && uv venv --python 3.12 .venv && uv pip install -r requirements.txt +export COLLIBRA_MCP_API_URL=... COLLIBRA_MCP_API_USR=... COLLIBRA_MCP_API_PWD=... +export ANTHROPIC_API_KEY=... +``` + +Python 3.10+ is required (the harness uses `dict | None`). `requirements.txt` now +pins `mcp>=1.23.0,<2`: `mcp` 2.0.0 renamed `McpError` → `MCPError`, which +inspect-ai still imports, so an unpinned install resolves to a combination that +cannot load MCP tools at all. + +## Verification status + +Verified: all tasks discovered by `inspect list tasks`; the live MCP handshake +fetches 2162 chars of real instructions (confirmed ≠ the paraphrase, and confirmed +to contain `Exceptions:`); chip's allow-list restricting to exactly 6 tools with +no write tools; this arm's early-stop predicate returning `False` on either +`list_collibra_skills` or `load_collibra_skill` and `True` on an unrelated read +tool (and `skill_match`'s stopping only on the load); `skill_lookup` building +with a `system_message` solver and `skill_lookup_no_instructions` without one; scorer +behaviour across 26 synthetic-transcript cases. + +**Verified against live rollouts:** both conditions run and score; the injected +system message is chip's real text, `Exceptions:` paragraph included; the early +stop fires on the first catalog call, so no rollout runs past the point being +measured. + +**Reproducibility:** Inspect records the commit in each log's `revision` field and +flags a dirty working tree. Commit before a run whose numbers you intend to cite, +or the log cannot be tied back to a source state. diff --git a/evals/docs/skill_match.md b/evals/docs/skill_match.md new file mode 100644 index 0000000..e303461 --- /dev/null +++ b/evals/docs/skill_match.md @@ -0,0 +1,297 @@ +# `skill_match` — given that it looks, does it pick the right skill? + +> Covers **steps 5-6** of the pipeline. See +> [pipeline.md](pipeline.md) for how this arm fits with the others. + +## Why this task exists + +The existing `data_product_create_happy_path` measures everything at once, so a +failure tells you *something* is wrong without telling you *what*. Three +independent things can break: + +1. the model never looked for a skill → the **instructions text** is failing +2. it looked but picked the wrong one → the skill's **`description`** is failing +3. it picked right but executed badly → the **`SKILL.md` body** is failing + +`skill_match` isolates **(2)**. It is the only task that measures whether a +skill's frontmatter `description` wins the requests it should — *and loses the ones +it shouldn't*. + +### Why `skill_lookup` cannot answer this + +**This is the only arm that runs `skill_selected`**, and that split is deliberate. +In `skill_lookup` the same scorer would be **confounded**: that arm ships no +system message, so the model may never look for a skill at all, and when it doesn't, +`skill_selected` fails too — not because the `description` is wrong, but because +there was nothing to route. It would measure P(looks **and** chooses right), roughly +`any_skill_discovered` × this arm's `skill_selected` — a product of two factors each +measured separately and more cleanly, blended into one number you can no longer +decompose. + +`skill_match` removes that variable by handing motivation over up front, so its +`skill_selected` measures routing and nothing else. `skill_lookup` keeps +`any_skill_discovered`; the two read together give the diagnosis — see +[Reading both arms together](#reading-both-arms-together). + +Second, and the bigger reason: `skill_lookup` runs only 4 unambiguous positive +cases. It deliberately excludes the distractors, and it *cannot* include the +negative case without capping its own ceiling. So three properties are measurable +only here — **precision**, **over-triggering**, and **ambiguity**. + +## What it does + +Eight prompts, run against chip with a read-only tool surface and a system message +that instructs skill discovery. The task measures which skill the model loads +first. + +| Sample id | Prompt | Accepted skill(s) | Tests | +| --- | --- | --- | --- | +| `data-product` | Create a Collibra Data Product from the table SALES.ORDERS | `collibra/data-product-create` | recall | +| `business-term-create` | Create a **new** Business Term for Churn Rate in the Finance domain | `collibra/asset-create` | **precision** | +| `attribute-edit` | Add a definition to the Customer Lifetime Value term and mark me as steward | `collibra/asset-edit` | recall | +| `lineage-trace` | Where does the Monthly Recurring Revenue KPI come from? | `collibra/lineage` | recall | +| `lineage-impact` | If we change ORDERS.DISCOUNT_PCT, which reports and dashboards break? | `collibra/lineage` | recall (2nd traversal wording) | +| `semantic-discovery` | What customer data do we have in the catalog? | `collibra/discovery` | recall | +| `ambiguous-governance` | Set an owner for the CUSTOMER table and make it discoverable to other teams | `asset-edit` **or** `data-product-create` | ambiguity | +| `no-skill-needed` | List the asset types available in this instance | *(none — loading any is a failure)* | **over-triggering** | + +### Why all eight, and why the awkward ones matter + +With only `data-product` you measure **recall** — "is the right skill found when it +applies?" — and never **precision** — "is it found when it doesn't?" Those pull in +opposite directions, and that tension is the whole reason this task has the shape +it does: + +> Broaden a `description` until it reliably wins its own case, and it starts +> winning its neighbours' too. Narrow it to stop poaching, and it stops firing when +> it should. + +You cannot see that trade-off from a single-skill eval. Running both directions in +one task makes it visible in one run, which matters because **editing any +`description` moves both at once**. + +- **`business-term-create` is the sharp one.** It contains *Create* and *new* — the + exact vocabulary that makes `data-product-create` attractive — but the correct + answer is `asset-create`. If `data-product-create` wins here, its description is + too greedy, and that regression is invisible to every other arm. +- **`ambiguous-governance` accepts two answers.** "Set an owner" reads as + `asset-edit`; "make it discoverable to other teams" reads as + `data-product-create`. Both are defensible, so both pass. What it catches is + landing somewhere *else* — `lineage`, `discovery` — which means the routing + signal is noise rather than a close call. Holding an ambiguous prompt to one + "right" answer would invent failures and push you to over-tune a description to + win a question that has no single answer. +- **`no-skill-needed` inverts the assertion.** chip's own instructions say a single + obvious tool call needs no skill, so here loading *anything* is the failure. + `skill_selected` with an empty expectation passes only if nothing substantive was + loaded (`collibra/index` is exempt — it's the documented navigator). + +### Why it keeps the paraphrased system message + +This arm uses the same hand-written `SYSTEM_MESSAGE` as +`tasks/data_product_create.py` — a local copy, deliberately **not** imported: that +file belongs to another author, and their edits should not silently change what +this arm measures. + +> You are connected to a Collibra MCP server. Before composing any multi-step +> Collibra workflow, discover the relevant skill guide by calling +> `list_collibra_skills` and load the matching one with `load_collibra_skill`, then +> follow it. + +That message is a 228-character paraphrase of the 2162-character text chip actually +ships, and it's more imperative than the real thing. For +[`skill_lookup`](skill_lookup.md) that's a defect — it short-circuits the +very thing being measured. **Here it's a feature.** This arm wants motivation held +constant at maximum so it doesn't contaminate the routing measurement, and a blunt +instruction is the most reliable way to do that. + +Note what it does *not* give away: it says "load the matching **one**" without +naming which. The model still has to choose among seven skills whose descriptions +compete. Routing is genuinely tested. + +### No environment setup required + +Cases live inline in the task file, not in `fixtures/`, and reference **invented +assets** ("Churn Rate", "SALES.ORDERS", "the CUSTOMER table"). Nothing here needs +to exist in your Collibra instance, because routing is decided from the *wording* +of a request — the model reads "Create a Collibra Data Product from the table X" +and routes on that sentence shape regardless of what X resolves to. + +This is deliberate. Sourcing the table name from +`fixtures/data_product_create.yaml` would drag in that fixture's validation, which +requires `expected.data_product_name` and `expected.grouped_tables` — write-arm +values with no bearing on routing. That would gate the cheapest, most portable arm +in the harness behind its most environment-specific config. As it stands this arm +runs against any instance with the fixture left completely blank. + +## How to read the results + +**Read per-case, not the aggregate.** A single `skill_selected` average over eight +heterogeneous cases hides everything useful. What you want is the per-sample +breakdown in `inspect view`. + +`skill_selected` records `expected`, `actual` and the full `loaded` list in its +`Score.metadata`, so the per-case results form a **confusion matrix**: for each +failure you can see *which* skill won instead. That's the actionable part — it names +the description that's stealing traffic. + +| Failing case | What it implicates | +| --- | --- | +| `data-product` | `data-product-create`'s description is too weak for its own core use case | +| `business-term-create` → loads `data-product-create` | `data-product-create` is too greedy; a **precision** regression | +| `business-term-create` → loads `asset-edit` | `asset-create` vs `asset-edit` boundary is unclear | +| `attribute-edit` → loads `asset-create` | same boundary, other direction — "add a definition" read as creation | +| `lineage-trace` / `semantic-discovery` | those descriptions lose to louder neighbours | +| `ambiguous-governance` → neither accepted skill | routing is noise, not a close call | +| `no-skill-needed` → anything loaded | some description is too eager; over-triggering | + +Two sanity checks worth glancing at: + +- **`any_skill_discovered` should be ~100% here.** The model was explicitly told to + discover. If it isn't, the model is ignoring a direct instruction and every + `skill_selected` number below it is suspect — investigate that first. +- **`skill_citation_consistency`** failing means the model wrote about a skill it + never loaded, i.e. reconstructed a procedure from the skill's *name*. That makes + a routing "pass" untrustworthy for the wrong reason. + +### Reading both arms together + +Each arm scores one question — `skill_lookup` "did it look?", `skill_match` "which skill?" — +and the pair is the diagnosis: + +| `any_skill_discovered` in `skill_lookup` | `skill_selected` here | Diagnosis | +| --- | --- | --- | +| 30% | 95% | The description is fine — the model just doesn't look. Fix the **instructions text** (`skills.Instructions` in `pkg/skills/register.go`). | +| 95% | 30% | It looks and still picks wrong. Fix the **`description` frontmatter**. | +| 95% | 95% | Both healthy. | + +Neither number distinguishes those rows alone. Note the two arms deliberately do +**not** both run `skill_selected` — in `skill_lookup` it would blend these two factors +into one inseparable figure (see [Why `skill_lookup` cannot answer +this](#why-skill_lookup-cannot-answer-this)). + +Both arms still draw their prompts from one shared `CASES` pool: `any_skill_discovered` +runs in both, so the arms only line up if the shared prompts are identical, and one +source of truth means a reworded prompt cannot drift between them unnoticed. + +Across the full ladder, each arm answers one question and the first failing rung is +the one to fix: + +| Arm | Question | Fix on failure | +| --- | --- | --- | +| `skill_lookup` (vs `skill_lookup_no_instructions`) | Does it look, under production conditions? | the instructions text | +| `skill_match` | Having looked, does it choose correctly — both directions? | the `description` frontmatter | +| `skill_adherence` | Having chosen, does it follow the procedure? | the `SKILL.md` body | +| `data_product_create_happy_path` | Does the whole thing work end to end? | headline number; diagnose via the rungs above | + +Work top-down. A failing `skill_adherence` with a failing `skill_match` above it +usually needs the routing fixed first — the body may be fine and simply never +reached. + +## How it runs + +```bash +inspect eval tasks/skill_arms.py@skill_match +inspect eval tasks/skill_arms.py@skill_match -T epochs=10 +``` + +- **Repetitions:** `epochs=5` (override with `-T epochs=N`). 8 samples × 5 = + **40 rollouts**. Inspect reduces the 5 scores per sample with the default + **`mean`** reducer, so a case that routes correctly 3/5 times scores 0.6. With 1 + epoch every case is a coin flip and `stderr()` is undefined; 5 gives a rate you + can compare across skill edits. `skill_match` is where you most want epochs — a + description change that shifts a case from 100% to 60% is a real regression that + a single run would report as a clean pass. +- **Concurrency:** Inspect uses async concurrency, not threads. `max_samples` + defaults to **adaptive** in inspect-ai 0.3.251 — a dynamic limiter tracking + observed rate limits (`max_connections=10` as reference). Override with + `--max-samples N`. Epochs are **not** sequential: all 40 rollouts go into one + concurrent queue. Safe here because nothing writes. +- **Cost:** ~2–3 tool calls per rollout thanks to the early stop below, against a + 6-tool schema instead of 32. Cheap enough to gate a PR on — which matters, + because editing a skill `description` is the most common change and is fully + covered by this arm. + +## How it terminates, and why it can't create anything + +Two independent mechanisms: + +1. **It has no write tools.** chip is launched with `--enabled-tools` limited to + six read-only tools (`list_collibra_skills`, `load_collibra_skill`, + `search_asset_keyword`, `get_asset_details`, `get_table_semantics`, + `list_asset_types`). `create_asset` isn't blocked — it's *absent from the + menu*, so writing is impossible even if the model tries. This also drops ~26 + tool schemas, cutting most of the ~20k-token-per-call overhead behind the + harness's documented 429 problem. + +2. **It stops as soon as the question is answered.** An `on_continue` hook ends the + run the moment `load_collibra_skill` returns, because that's when routing is + decided. Without it, `react()` runs until the model calls `submit` or hits + `message_limit` — so the model would pick its skill and then keep going, working + through that skill's read-only phases and stopping only on reaching a write it + can't perform. Dozens of turns, zero extra signal. + + This gate is stricter than `skill_lookup`'s, which stops on the *listing* as + well — that arm only needs to know the model looked, whereas this one has to see + which skill it settled on. + + A side effect worth knowing: because the run ends on the *first* load, this arm + structurally sees at most one skill load. "Did it also load other skills?" is + not measurable here — and in the full-flow arms extra loads are usually correct + anyway, since `data-product-create`'s body explicitly directs the model to + `asset-create` and `asset-edit`. + +## Scorers + +| Scorer | Asserts | Metric | +| --- | --- | --- | +| `skill_selected` | the **first** skill loaded is one this case accepts; `collibra/index` is exempt as the documented navigator; an empty expectation passes only if nothing was loaded; a multi-entry expectation passes on any of them | pass/fail | +| `any_skill_discovered` | `list_collibra_skills` or `load_collibra_skill` was called at all — a sanity check here, since the model was told to | pass/fail | +| `skill_citation_consistency` | the model never cites a skill in prose that it didn't actually load | pass/fail | + +`skill_selected` uses *first* load rather than *any* load on purpose: a model that +loads three skills hoping one sticks has not routed correctly, even if the right +one is in the pile. + +`skill_citation_consistency` catches the model reconstructing a procedure from a +skill's **name** without reading it. It scans assistant messages only: a loaded +`SKILL.md` lists siblings in its `related:` frontmatter and `list_collibra_skills` +returns every name, so scanning tool output would manufacture citations the model +never made. + +## Prerequisites + +```bash +go build -o .build/chip ./cmd/chip +cd evals && uv venv --python 3.12 .venv && uv pip install -r requirements.txt +export COLLIBRA_MCP_API_URL=... COLLIBRA_MCP_API_USR=... COLLIBRA_MCP_API_PWD=... +export ANTHROPIC_API_KEY=... +``` + +Python 3.10+ is required (the harness uses `dict | None`). `requirements.txt` pins +`mcp>=1.23.0,<2`: `mcp` 2.0.0 renamed `McpError` → `MCPError`, which inspect-ai +still imports, so an unpinned install resolves to a combination that cannot load +MCP tools at all. + +Credentials are needed even though this arm never writes — chip refuses to start +without an API URL, and the model may legitimately call `search_asset_keyword` or +`get_asset_details` while orienting itself. + +## Verification status + +Verified: the task is discovered by `inspect list tasks`; the dataset builds with +all 7 cases; chip's allow-list restricting to exactly 6 tools with no write tools; +this arm's early-stop predicate returning `True` after a bare `list_collibra_skills` +and `False` only once `load_collibra_skill` fires (and `skill_lookup`'s stopping +on either); `skill_selected` accepting either skill on the ambiguous case and +rejecting a third; the 3 prompts shared with `skill_lookup` being byte-identical +across both arms. + +## See also + +- [`skill_lookup.md`](skill_lookup.md) — the arm above this one on the + ladder: does the model look for a skill at all? +- [`skill_adherence.md`](skill_adherence.md) — the arm below: having chosen, does it + follow the procedure? Seeds its own data and asserts on the resulting graph. + +- [`pipeline.md`](pipeline.md) — the map: all eleven steps, and which arm brackets which. diff --git a/evals/fixtures/data_product_create.yaml b/evals/fixtures/data_product_create.yaml new file mode 100644 index 0000000..c332d40 --- /dev/null +++ b/evals/fixtures/data_product_create.yaml @@ -0,0 +1,23 @@ +# Fixture for the data-product-create happy-path eval. +# +# Fill in `source_table` and `expected` for your test environment before running. +# The eval fails fast with a clear error while `source_table` is empty. + +# The fact table the agent starts from: a name, a DGC UUID, or a Collibra URL. +# e.g. "SALES>ORDERS" or "01924cdc-41c3-772b-94db-9c0fbae84d77" +source_table: "019df8b6-6e10-739d-a129-e38602326f42" + +expected: + # Exact name of the Data Product the skill should create. + # Skill naming rule: human-readable, derived from the table + # (e.g. sales.customer_orders -> "Sales Customer Orders"). + data_product_name: "Sakila Film Catalog" + + # Names of the tables the physical Port must `groups`-link: + # the source table plus every HIGH-confidence dimension in your environment. + grouped_tables: [ "film" , "film_actor", "film_category" ] + # - "ORDERS" + # - "CUSTOMER" + + # 1 normally; 2 when the tables have a semantic layer (physical + UI port). + ports: 1 diff --git a/evals/requirements.txt b/evals/requirements.txt new file mode 100644 index 0000000..1e92842 --- /dev/null +++ b/evals/requirements.txt @@ -0,0 +1,8 @@ +inspect-ai +# mcp 2.0.0 renamed McpError -> MCPError, which inspect-ai still imports, so an +# unpinned install resolves to a combination that cannot load MCP tools at all. +# inspect-ai's own floor is >=1.23.0; 1.23-1.26 satisfy both. +mcp>=1.23.0,<2 +anthropic +httpx +pyyaml diff --git a/evals/scorers/__init__.py b/evals/scorers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/evals/scorers/collibra.py b/evals/scorers/collibra.py new file mode 100644 index 0000000..b63f919 --- /dev/null +++ b/evals/scorers/collibra.py @@ -0,0 +1,103 @@ +"""Thin Collibra DGC REST 2.0 client for eval scorers and cleanup. + +Credentials resolve in chip's own precedence order, so a working local chip +setup needs no extra configuration: + + 1. COLLIBRA_MCP_API_URL / _USR / _PWD environment variables + 2. ./mcp.yaml (repo root) + 3. ~/.config/collibra/mcp.yaml + 4. /etc/collibra/mcp.yaml + +Steps 2-4 mirror the viper search path in cmd/chip/config.go. Earlier +versions stopped after the repo root, which meant a machine configured the +normal way — credentials in ~/.config/collibra — could not run any eval at +all, and the failure looked like missing credentials rather than a client +that never looked for them. + +One deliberate divergence: viper stops at the first config file it finds, +while this fills each field from the first source that supplies it, so a +partial env var set or a partial mcp.yaml is topped up rather than discarded. +That is more forgiving than chip, which means these scorers can authenticate +in a split-config setup where chip itself would not — worth knowing if an eval +passes and the server does not. +""" + +import os +from pathlib import Path + +import httpx +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[2] + +# Same order chip's viper config uses (cmd/chip/config.go). +CONFIG_PATHS = ( + REPO_ROOT / "mcp.yaml", + Path.home() / ".config" / "collibra" / "mcp.yaml", + Path("/etc/collibra/mcp.yaml"), +) + + +def collibra_config() -> tuple[str, str, str]: + url = os.environ.get("COLLIBRA_MCP_API_URL", "") + usr = os.environ.get("COLLIBRA_MCP_API_USR", "") + pwd = os.environ.get("COLLIBRA_MCP_API_PWD", "") + + for config_path in CONFIG_PATHS: + if url and usr and pwd: + break + if not config_path.exists(): + continue + api = (yaml.safe_load(config_path.read_text()) or {}).get("api", {}) or {} + # Per-field fallback, so a partial env var set is topped up rather + # than discarded. + url = url or api.get("url", "") + usr = usr or api.get("username", "") + pwd = pwd or api.get("password", "") + + if not (url and usr and pwd): + searched = "\n ".join(str(p) for p in CONFIG_PATHS) + raise RuntimeError( + "Collibra credentials not found: set COLLIBRA_MCP_API_URL/USR/PWD, " + f"or provide an mcp.yaml at one of:\n {searched}" + ) + return url.rstrip("/"), usr, pwd + + +def client() -> httpx.AsyncClient: + url, usr, pwd = collibra_config() + return httpx.AsyncClient( + base_url=f"{url}/rest/2.0", auth=(usr, pwd), timeout=30.0 + ) + + +def sync_client() -> httpx.Client: + url, usr, pwd = collibra_config() + return httpx.Client(base_url=f"{url}/rest/2.0", auth=(usr, pwd), timeout=30.0) + + +async def find_asset_by_name(c: httpx.AsyncClient, name: str) -> dict | None: + """Exact-name asset lookup; returns the first match or None.""" + r = await c.get("/assets", params={"name": name, "nameMatchMode": "EXACT"}) + r.raise_for_status() + results = r.json().get("results", []) + return results[0] if results else None + + +async def asset_type_name(c: httpx.AsyncClient, asset_id: str) -> str: + r = await c.get(f"/assets/{asset_id}") + r.raise_for_status() + return (r.json().get("type") or {}).get("name", "") + + +async def relations( + c: httpx.AsyncClient, *, source_id: str | None = None, target_id: str | None = None +) -> list[dict]: + params: dict[str, str] = {} + if source_id: + params["sourceId"] = source_id + if target_id: + params["targetId"] = target_id + r = await c.get("/relations", params=params) + r.raise_for_status() + return r.json().get("results", []) diff --git a/evals/scorers/end_state.py b/evals/scorers/end_state.py new file mode 100644 index 0000000..10b87ee --- /dev/null +++ b/evals/scorers/end_state.py @@ -0,0 +1,89 @@ +"""End-state scorer: ignores how the agent got there and asks the test +environment whether the expected asset graph now exists. + +Checks (each contributes equally to the score): + product_exists - a Data Product with the expected name exists + ports_linked - it exposes the expected number of Data Product Ports + tables_grouped - some port groups every expected table + contract_governs - a Data Contract has a relation onto one of the ports +""" + +from inspect_ai.scorer import Score, Target, accuracy, scorer, stderr +from inspect_ai.solver import TaskState + +from . import collibra + + +@scorer(metrics=[accuracy(), stderr()]) +def data_product_end_state(): + async def score(state: TaskState, target: Target) -> Score: + expected = state.metadata["expected"] + checks: dict[str, bool] = {} + notes: list[str] = [] + + async with collibra.client() as c: + product = await collibra.find_asset_by_name( + c, expected["data_product_name"] + ) + checks["product_exists"] = product is not None + if product is None: + notes.append( + f"no asset named {expected['data_product_name']!r} found" + ) + return _result(checks, notes) + + # Ports: relations off the product whose counterpart is a Port asset. + ports: list[dict] = [] + for rel in await collibra.relations(c, source_id=product["id"]): + tgt = rel.get("target") or {} + if tgt and await collibra.asset_type_name(c, tgt["id"]) == ( + "Data Product Port" + ): + ports.append(tgt) + checks["ports_linked"] = len(ports) >= int(expected.get("ports", 1)) + notes.append(f"ports found: {[p.get('name') for p in ports]}") + + # Grouped tables: at least one port must group every expected table. + want = set(expected.get("grouped_tables", [])) + grouped_ok = False + for port in ports: + got = { + (rel.get("target") or {}).get("name", "") + for rel in await collibra.relations(c, source_id=port["id"]) + } + missing = want - got + if not missing: + grouped_ok = True + break + notes.append(f"port {port.get('name')!r} missing tables: {missing}") + checks["tables_grouped"] = bool(want) and grouped_ok + + # Contract: an incoming relation on some port from a Data Contract. + contract_ok = False + for port in ports: + for rel in await collibra.relations(c, target_id=port["id"]): + src = rel.get("source") or {} + if src and await collibra.asset_type_name(c, src["id"]) == ( + "Data Contract" + ): + contract_ok = True + notes.append( + f"contract {src.get('name')!r} governs " + f"port {port.get('name')!r}" + ) + checks["contract_governs"] = contract_ok + + return _result(checks, notes) + + return score + + +def _result(checks: dict[str, bool], notes: list[str]) -> Score: + total = 4 # fixed check count so partial graphs score proportionally + passed = sum(checks.values()) + lines = [f"{'PASS' if ok else 'FAIL'} {name}" for name, ok in checks.items()] + return Score( + value=passed / total, + explanation="\n".join(lines + notes), + metadata={"checks": checks}, + ) diff --git a/evals/scorers/seeded.py b/evals/scorers/seeded.py new file mode 100644 index 0000000..5d5de24 --- /dev/null +++ b/evals/scorers/seeded.py @@ -0,0 +1,215 @@ +"""Scorers for the seeded adherence arm. + +Both key off `state.metadata["seed"]`, which `solvers/seed.py` populates. + +`seeded_end_state` is the graph-anchored replacement for `end_state`: it starts +from the seeded fact table's UUID and walks outward, instead of looking a Data +Product up by name. That difference is the point — a leftover product from +another run is not reachable from *this* run's table, so it cannot be mistaken +for success. Name-based lookup could be satisfied by a previous run's assets even +when the current run wrote nothing at all. + +`columns_discovered` exists because of a bug in the skill. Phase 1 says to read +the column list "from the outgoing `Column` relations", but there is no +Table->Column relation type in Collibra's model at all — the only one is +`Column is part of Table`, with the *column* as source. So columns always arrive +as **incoming** relations, and `get_asset_details` returns them under +`incomingRelations`. A model following Phase 1 literally finds nothing there and +concludes the table has no columns. None of the four end-state checks would +notice: ports, product, contract and grouped tables do not depend on columns. So +the bug degrades quality invisibly, and needs its own check to surface. + +Relation type ids below are system UUIDs, probed live. Traversing by type id +rather than fetching each counterpart's asset type also avoids one GET per +relation. +""" + +from inspect_ai.scorer import CORRECT, INCORRECT, Score, Target, accuracy, scorer, stderr +from inspect_ai.solver import TaskState + +from . import collibra + +REL_PORT_IMPLEMENTED_AS_TABLE = "00000000-0000-0000-0000-000000050042" +REL_PRODUCT_EXPOSES_PORT = "00000000-0000-0000-0000-000000050040" +REL_CONTRACT_GOVERNS_PORT = "00000000-0000-0000-0000-000000050044" + +CHECK_COUNT = 4 # fixed, so a partial graph scores proportionally + + +def _sources(relations: list[dict], type_id: str) -> list[dict]: + return [ + r["source"] + for r in relations + if (r.get("type") or {}).get("id") == type_id and r.get("source") + ] + + +def _targets(relations: list[dict], type_id: str) -> list[dict]: + return [ + r["target"] + for r in relations + if (r.get("type") or {}).get("id") == type_id and r.get("target") + ] + + +@scorer(metrics=[accuracy(), stderr()]) +def seeded_end_state(): + """Did the agent build the expected graph around *our* seeded table? + + product_exposes_port - a Data Product exposes that Port as an output port + port_exposes_table - a Data Product Port implements the seeded fact table + tables_grouped - the Port also implements every seeded dimension table + contract_governs - a Data Contract governs the Port + """ + + async def score(state: TaskState, target: Target) -> Score: + seed = state.metadata.get("seed") + if not seed: + return Score( + value=0.0, + explanation="no seed metadata — did seed_star_schema() run as setup?", + ) + + fact_id = seed["fact_table_id"] + want_tables = set(seed["dimension_table_ids"]) + checks: dict[str, bool] = {} + notes: list[str] = [] + + async with collibra.client() as client: + onto_fact = await collibra.relations(client, target_id=fact_id) + ports = _sources(onto_fact, REL_PORT_IMPLEMENTED_AS_TABLE) + checks["port_exposes_table"] = bool(ports) + if not ports: + notes.append( + f"no Data Product Port implements {seed['fact_table_name']!r} " + f"({len(onto_fact)} relation(s) onto it)" + ) + return _result(checks, notes, {}) + notes.append(f"ports: {[p.get('name') for p in ports]}") + + product_ok = False + contract_ok = False + grouped_best: set[str] = set() + for port in ports: + onto_port = await collibra.relations(client, target_id=port["id"]) + if _sources(onto_port, REL_PRODUCT_EXPOSES_PORT): + product_ok = True + names = [ + p.get("name") + for p in _sources(onto_port, REL_PRODUCT_EXPOSES_PORT) + ] + notes.append(f"product(s) exposing {port.get('name')!r}: {names}") + if _sources(onto_port, REL_CONTRACT_GOVERNS_PORT): + contract_ok = True + names = [ + c.get("name") + for c in _sources(onto_port, REL_CONTRACT_GOVERNS_PORT) + ] + notes.append(f"contract(s) governing {port.get('name')!r}: {names}") + + from_port = await collibra.relations(client, source_id=port["id"]) + implemented = { + t["id"] for t in _targets(from_port, REL_PORT_IMPLEMENTED_AS_TABLE) + } + grouped_best |= implemented & want_tables + + checks["product_exposes_port"] = product_ok + checks["contract_governs"] = contract_ok + checks["tables_grouped"] = bool(want_tables) and grouped_best == want_tables + + missing = want_tables - grouped_best + if missing: + # strict: ids and names come from the same seed manifest and must be + # parallel. If they ever aren't, `by_id[i]` below would raise an + # opaque KeyError instead of naming the real problem. + by_id = dict( + zip( + seed["dimension_table_ids"], + seed["dimension_table_names"], + strict=True, + ) + ) + notes.append(f"dimension tables not grouped: {[by_id[i] for i in missing]}") + extra = { + "grouped_fraction": ( + len(grouped_best) / len(want_tables) if want_tables else 0.0 + ) + } + return _result(checks, notes, extra) + + return score + + +def _result(checks: dict, notes: list[str], extra: dict) -> Score: + passed = sum(checks.values()) + lines = [f"{'PASS' if ok else 'FAIL'} {name}" for name, ok in checks.items()] + return Score( + value=passed / CHECK_COUNT, + explanation="\n".join(lines + notes), + metadata={"checks": checks, **extra}, + ) + + +@scorer(metrics=[accuracy(), stderr()]) +def columns_discovered(): + """Did the agent actually see the seeded table's columns? + + Surfaces the Phase 1 wording bug described in this module's docstring: a + model told to read "outgoing Column relations" finds none, because columns + are incoming. Scored from the agent's own prose — if it discovered the + columns it names at least a couple of them while reasoning about the table. + + Deliberately lenient (>= 2 distinct columns) rather than requiring all of + them: the skill only needs "a few" columns for the Data Set check, so + demanding the full list would fail runs that behaved correctly. + """ + + async def score(state: TaskState, target: Target) -> Score: + seed = state.metadata.get("seed") + if not seed: + return Score(value=INCORRECT, explanation="no seed metadata") + + prose_parts = [] + for message in state.messages: + if getattr(message, "role", None) != "assistant": + continue + content = getattr(message, "content", None) + if isinstance(content, str): + prose_parts.append(content) + elif isinstance(content, list): + for block in content: + text = getattr(block, "text", None) + if isinstance(text, str): + prose_parts.append(text) + prose = "\n".join(prose_parts) + + fact_prefix = seed["fact_table_name"] + "_" + fact_columns = [c for c in seed["column_names"] if c.startswith(fact_prefix)] + # Match the bare column name too: the model usually writes ORDER_TOTAL + # rather than the fully-qualified TEST__ORDERS_ORDER_TOTAL. + found = [ + c + for c in fact_columns + if c in prose or c[len(fact_prefix) :] in prose + ] + + ok = len(found) >= 2 + return Score( + value=CORRECT if ok else INCORRECT, + explanation=( + f"{len(found)}/{len(fact_columns)} seeded fact-table columns named " + f"in the agent's own prose: {[c[len(fact_prefix):] for c in found]}" + + ( + "" + if ok + else " — consistent with Phase 1's 'outgoing Column relations'," + " which finds nothing because columns are incoming" + ) + ), + metadata={ + "found": found, + "expected_any_of": fact_columns, + }, + ) + + return score diff --git a/evals/scorers/skill_choice.py b/evals/scorers/skill_choice.py new file mode 100644 index 0000000..50b0f8b --- /dev/null +++ b/evals/scorers/skill_choice.py @@ -0,0 +1,477 @@ +"""Transcript scorers for the skill arms: did the model reach for a skill, did +it reach for the *right* one, do its prose claims match what it actually called, +and did it write in the order the skill mandates? + +These are cheap and transcript-only — no Collibra access, no writes — so the +arms that use them can run at high epoch counts in parallel. + +Tool names need suffix matching because MCP clients prefix them with the server +name (`collibra_create_asset`), but plain `endswith` is wrong: both +`create_asset` and the read-only `prepare_create_asset` would match a query for +`create_asset` — and so would an underscore-boundary check, since +`collibra_prepare_create_asset` genuinely ends with `_create_asset`. See +_canonical for how that is resolved. Getting this right is what separates +write_sequence below from scorers/trajectory.py::write_order, which it replaces. +""" + +import re + +from inspect_ai.scorer import ( + CORRECT, + INCORRECT, + Score, + Target, + accuracy, + scorer, + stderr, +) +from inspect_ai.solver import TaskState + +LIST_TOOL = "list_collibra_skills" +LOAD_TOOL = "load_collibra_skill" + +# Skill slugs as they appear in the catalog. Matched in prose either fully +# qualified (`collibra/lineage`) or bare (`lineage`); see _cited_skills. +KNOWN_SKILLS = ( + "index", + "discovery", + "lineage", + "asset-create", + "asset-edit", + "data-product-create", + "context", +) + +# chip tool names worth looking for in prose. Longest-first so that +# `prepare_create_asset` is preferred over `create_asset` when both match at +# the same position. +KNOWN_TOOLS = ( + "push_data_contract_manifest", + "pull_data_contract_manifest", + "prepare_create_asset", + "search_asset_keyword", + "get_table_semantics", + "get_asset_details", + "list_collibra_skills", + "load_collibra_skill", + "init_data_contract", + "list_data_contract", + "list_asset_types", + "create_asset", + "edit_asset", +) + + +def _canonical(function: str) -> str | None: + """Resolve an MCP tool call's function name to a known chip tool. + + Suffix matching alone is not enough, and neither is requiring an underscore + boundary: `collibra_prepare_create_asset` ends with `_create_asset`, so both + would report the read-only prepare tool as a write. Resolving to the + *longest* known tool the name ends with disambiguates the pair — and any + future pair like it. + """ + best: str | None = None + for tool in KNOWN_TOOLS: + if function == tool or function.endswith(f"_{tool}"): + if best is None or len(tool) > len(best): + best = tool + return best + + +def _is(call, name: str) -> bool: + """True if `call` is exactly `name`, allowing an MCP server-name prefix.""" + return _canonical(call.function) == name + + +def _tool_calls(state: TaskState) -> list: + return [ + tc + for m in state.messages + if getattr(m, "tool_calls", None) + for tc in m.tool_calls + ] + + +def _has_operation(call, op_type: str) -> bool: + """True if an `edit_asset` call carries an operation of `op_type`. + + Reads the structured `operations: [{"type": ...}]` payload rather than + searching the whole argument blob as text: a `set_attribute` whose *value* + mentions "add_relation" — a Description discussing relations, say — would + otherwise be counted as a relation write. + """ + ops = (call.arguments or {}).get("operations") + if not isinstance(ops, list): + return False + return any(isinstance(op, dict) and op.get("type") == op_type for op in ops) + + +def _loaded_skills(state: TaskState) -> list[str]: + """Skill names actually loaded, in call order. Ground truth — the call + happened and chip returned a body.""" + loaded = [] + for call in _tool_calls(state): + if not _is(call, LOAD_TOOL): + continue + name = (call.arguments or {}).get("skillName", "") + if name: + loaded.append(_normalize(str(name))) + return loaded + + +def _normalize(skill: str) -> str: + """`data-product-create` and `collibra/data-product-create` are the same + skill; compare on the qualified form.""" + skill = skill.strip().strip("`'\"") + return skill if "/" in skill else f"collibra/{skill}" + + +def _assistant_prose(state: TaskState) -> str: + """Concatenated text the model wrote *itself*. + + Deliberately excludes tool result messages: a loaded SKILL.md lists its + siblings in `related:` frontmatter and list_collibra_skills returns every + name, so scanning tool output would manufacture citations the model never + made. + """ + parts: list[str] = [] + for message in state.messages: + if getattr(message, "role", None) != "assistant": + continue + content = getattr(message, "content", None) + if isinstance(content, str): + parts.append(content) + elif isinstance(content, list): + # Content blocks: pull text off whichever ones carry it. + for block in content: + text = getattr(block, "text", None) + if isinstance(text, str): + parts.append(text) + return "\n".join(parts) + + +def _cited_skills(prose: str) -> set[str]: + """Skills the model *claims* to be using. + + Qualified `collibra/` always counts. A bare slug counts only when + it is hyphenated (`data-product-create`, `asset-edit`) — bare `lineage`, + `discovery`, `context`, and `index` are ordinary English words that appear + constantly in this domain and would produce nothing but false positives. + """ + cited = set() + for slug in KNOWN_SKILLS: + if re.search(rf"\bcollibra/{re.escape(slug)}\b", prose): + cited.add(f"collibra/{slug}") + elif "-" in slug and re.search(rf"\b{re.escape(slug)}\b", prose): + cited.add(f"collibra/{slug}") + return cited + + +@scorer(metrics=[accuracy(), stderr()]) +def any_skill_discovered(): + """Did the model reach for the skill catalog at all, unprompted? + + The lookup arms' headline metric: does the model look for a skill, and + does chip's instructions text change whether it does? Each instruction + condition is a separate task (`skill_lookup` / + `skill_lookup_no_instructions`), so this reports one number and the comparison + is between the two runs' numbers. + """ + + async def score(state: TaskState, target: Target) -> Score: + calls = _tool_calls(state) + listed = [i for i, c in enumerate(calls) if _is(c, LIST_TOOL)] + loaded = [i for i, c in enumerate(calls) if _is(c, LOAD_TOOL)] + if not listed and not loaded: + return Score( + value=INCORRECT, + explanation=( + f"neither {LIST_TOOL} nor {LOAD_TOOL} called " + f"in {len(calls)} tool call(s)" + ), + metadata={"tool_calls": [c.function for c in calls]}, + ) + return Score( + value=CORRECT, + explanation=( + f"{LIST_TOOL} at {listed or 'never'}, {LOAD_TOOL} at {loaded or 'never'}" + ), + metadata={"listed_at": listed, "loaded_at": loaded}, + ) + + return score + + +@scorer(metrics=[accuracy(), stderr()]) +def skill_discovery_first(): + """Did the model consult the catalog *before* doing any other Collibra work? + + `any_skill_discovered` asks only whether the model ever looked, and a look + that happens after the first search still passes. That distinction is the + whole measurement: a skill consulted after the first `search_asset_keyword` + did not guide that search, because the model has already committed to an + approach. It also matters for headroom — as a binary over "ever looked", + `any_skill_discovered` pins at 1.000 for any model that looks eventually, + leaving no room for an instructions gap to show up in. + + Scored on the first *message* carrying tool calls, not the first call, so a + parallel batch containing both the skill lookup and a search counts as a + failure. Issuing them together means the model never waited for the playbook, + which is the behaviour this measures however the calls are ordered inside the + batch. + """ + + async def score(state: TaskState, target: Target) -> Score: + batches = [ + [_canonical(c.function) or c.function for c in m.tool_calls] + for m in state.messages + if getattr(m, "tool_calls", None) + ] + if not batches: + return Score( + value=INCORRECT, + explanation="no tool calls at all", + metadata={"first_batch": []}, + ) + + first = batches[0] + discovery = [n for n in first if n in (LIST_TOOL, LOAD_TOOL)] + other = [n for n in first if n not in (LIST_TOOL, LOAD_TOOL)] + meta = {"first_batch": first, "batches": batches} + + if not discovery: + return Score( + value=INCORRECT, + explanation=f"first action was {other}, not skill discovery", + metadata=meta, + ) + if other: + return Score( + value=INCORRECT, + explanation=( + f"skill discovery {discovery} issued in the same turn as " + f"{other} — the model did not wait for the playbook" + ), + metadata=meta, + ) + return Score( + value=CORRECT, + explanation=f"first action was {discovery}", + metadata=meta, + ) + + return score + + +@scorer(metrics=[accuracy(), stderr()]) +def skill_selected(): + """Was the first skill loaded one this sample accepts? + + Reads `expected_skill` from sample metadata: a list of acceptable skills (a + bare string is also accepted). Empty means "no skill applies here" and is + correct only if the model loaded nothing. More than one entry means the + request is genuinely ambiguous and any of them passes — holding an ambiguous + prompt to a single answer invents failures. + + First-loaded rather than any-loaded on purpose: a model that loads three + skills hoping one sticks has not routed correctly, even if the right one + is in the pile. `collibra/index` is skipped — it is the documented + navigator, so loading it on the way to the answer is correct behaviour, + not a wrong choice. + """ + + async def score(state: TaskState, target: Target) -> Score: + raw = state.metadata.get("expected_skill") or [] + if isinstance(raw, str): + raw = [raw] if raw else [] + expected = {_normalize(skill) for skill in raw} + + loaded = _loaded_skills(state) + substantive = [s for s in loaded if s != "collibra/index"] + actual = substantive[0] if substantive else "" + + if not expected: + ok = not substantive + return Score( + value=CORRECT if ok else INCORRECT, + explanation=( + "no skill expected; none loaded" + if ok + else f"no skill expected but loaded {substantive}" + ), + metadata={"expected": [], "actual": actual, "loaded": loaded}, + ) + + want = sorted(expected) + return Score( + value=CORRECT if actual in expected else INCORRECT, + explanation=( + f"expected {'any of ' if len(want) > 1 else ''}{want}, " + f"first substantive load was {actual or 'nothing'} " + f"(all loads: {loaded or 'none'})" + ), + metadata={"expected": want, "actual": actual, "loaded": loaded}, + ) + + return score + + +@scorer(metrics=[accuracy(), stderr()]) +def skill_citation_consistency(): + """Do the model's prose claims match its actual tool calls? + + Fails when the model writes about a skill it never loaded — it is then + reconstructing a plausible-looking procedure from the skill's *name*, + which can produce a passing end state for entirely the wrong reason and is + invisible to both the end-state and write-order scorers. + + The converse (loaded but never named in prose) is reported but not a + failure: silently following a loaded skill is fine. + """ + + async def score(state: TaskState, target: Target) -> Score: + prose = _assistant_prose(state) + cited = _cited_skills(prose) + loaded = set(_loaded_skills(state)) + + fabricated = sorted(cited - loaded) + silent = sorted(loaded - cited) + + notes = [] + if cited: + notes.append(f"cited: {sorted(cited)}") + else: + # Not a pointer to another scorer: which load-based scorer is present + # varies by arm (skill_lookup runs any_skill_discovered, the others + # skill_selected), and naming an absent one sends readers hunting. + notes.append( + "cited: none (vacuous pass — no skill claimed, so nothing to contradict)" + ) + notes.append(f"loaded: {sorted(loaded) or 'none'}") + if silent: + notes.append(f"loaded but never named in prose (allowed): {silent}") + + if fabricated: + return Score( + value=INCORRECT, + explanation="; ".join( + [f"cited without ever loading: {fabricated}"] + notes + ), + metadata={"fabricated": fabricated, "cited": sorted(cited), + "loaded": sorted(loaded)}, + ) + return Score( + value=CORRECT, + explanation="; ".join(notes), + metadata={"fabricated": [], "cited": sorted(cited), + "loaded": sorted(loaded)}, + ) + + return score + + +# Phase 7's relation roles, as the model passes them to edit_asset. +REL_PORT_TO_TABLE = "is implemented as" + +# Init has to see the Port's table links, or the manifest it generates omits them. +INIT_TOOL = "init_data_contract" +PUSH_TOOL = "push_data_contract_manifest" + + +@scorer(metrics=[accuracy(), stderr()]) +def write_sequence(): + """Does Phase 7 happen in the order Phase 7 requires? + + Two orderings, both stated outright in the SKILL.md body: + + * step 2 before step 4 — "the tables must be linked to the Port before init + so the generated manifest covers them". Init reads the Port's + `is implemented as` links to build the base manifest, so linking after + init silently produces a manifest missing those tables. + * step 4 before step 5 — init creates version 0.0.1 and push adds 0.0.2, so + a push with no prior init has no base to improve on. + + Plus a guard that is not about ordering at all: a rollout with **no** + create_asset failed to write, and on this arm that is a failure rather than a + vacuous pass. + + Deliberately does NOT require every create to precede every relation edit. + Phase 7 mandates the opposite: create Product and Port, wire their relations, + *then* create the Data Contract, then link it. scorers/trajectory.py's + write_order enforces that non-existent rule and so fails a correct + trajectory — as does any check written from "hard rule 1", which is about + confirming once, not about write order. + + Two known limits. Ordering is checked by call index, so a model batching + writes into one message defeats it — see _sequential_prompt in + tasks/skill_arms.py. And a `create_asset` sent with `allowDuplicate=false` + purely to resolve a name returns `duplicate_found` without writing, yet still + counts toward the guard; seeded_end_state is the robust check for "nothing was + written". + """ + + async def score(state: TaskState, target: Target) -> Score: + calls = _tool_calls(state) + creates = [i for i, c in enumerate(calls) if _is(c, "create_asset")] + prepares = [i for i, c in enumerate(calls) if _is(c, "prepare_create_asset")] + table_links = [ + i + for i, c in enumerate(calls) + if _is(c, "edit_asset") + and any( + op.get("relationType") == REL_PORT_TO_TABLE + for op in ((c.arguments or {}).get("operations") or []) + if isinstance(op, dict) and op.get("type") == "add_relation" + ) + ] + inits = [i for i, c in enumerate(calls) if _is(c, INIT_TOOL)] + pushes = [i for i, c in enumerate(calls) if _is(c, PUSH_TOOL)] + + if not creates: + return Score( + value=INCORRECT, + explanation=( + f"nothing was created: no create_asset call in {len(calls)} " + f"tool call(s) ({len(prepares)} prepare_create_asset call(s), " + "which write nothing)" + ), + metadata={"creates": [], "prepares": prepares}, + ) + + problems = [] + if inits and table_links and max(table_links) > min(inits): + problems.append( + f"Port linked to a table at #{max(table_links)}, after " + f"init_data_contract at #{min(inits)} — the generated manifest " + "cannot cover that table" + ) + if pushes and not inits: + problems.append( + f"manifest pushed at #{min(pushes)} with no init_data_contract — " + "init produces the base the push is meant to improve" + ) + elif pushes and inits and min(pushes) < min(inits): + problems.append( + f"manifest pushed at #{min(pushes)} before init_data_contract at " + f"#{min(inits)}" + ) + + return Score( + value=INCORRECT if problems else CORRECT, + explanation="; ".join(problems) + or ( + f"{len(creates)} create(s) at {creates}; Port->table links at " + f"{table_links or 'none'}; init at {inits or 'none'}; " + f"push at {pushes or 'none'}" + ), + metadata={ + "creates": creates, + "prepares": prepares, + "table_links": table_links, + "inits": inits, + "pushes": pushes, + }, + ) + + return score diff --git a/evals/scorers/trajectory.py b/evals/scorers/trajectory.py new file mode 100644 index 0000000..5f57e55 --- /dev/null +++ b/evals/scorers/trajectory.py @@ -0,0 +1,102 @@ +"""Trajectory scorers: walk the transcript's tool calls and check the skill's +procedural rules, independent of the final environment state. + +Tool names are matched by suffix because MCP clients may prefix them with the +server name. +""" + +from inspect_ai.scorer import CORRECT, INCORRECT, Score, Target, accuracy, scorer, stderr +from inspect_ai.solver import TaskState + + +def _tool_calls(state: TaskState) -> list: + return [ + tc + for m in state.messages + if getattr(m, "tool_calls", None) + for tc in m.tool_calls + ] + + +def _is(call, name: str) -> bool: + return call.function == name or call.function.endswith(f"_{name}") or ( + call.function.endswith(name) + ) + + +@scorer(metrics=[accuracy(), stderr()]) +def skill_loaded(): + """The agent discovered and loaded the data-product-create skill before + doing the work (cold start — triggering failures show up here, keeping + them distinguishable from execution failures).""" + + async def score(state: TaskState, target: Target) -> Score: + calls = _tool_calls(state) + loaded = [ + i + for i, c in enumerate(calls) + if _is(c, "load_collibra_skill") + and "data-product-create" in str(c.arguments) + ] + if not loaded: + return Score( + value=INCORRECT, + explanation="load_collibra_skill(data-product-create) never called", + ) + first_write = next( + (i for i, c in enumerate(calls) if _is(c, "create_asset")), None + ) + before_writes = first_write is None or loaded[0] < first_write + return Score( + value=CORRECT if before_writes else INCORRECT, + explanation=( + f"skill loaded at call #{loaded[0]}, " + f"first create_asset at #{first_write}" + ), + ) + + return score + + +@scorer(metrics=[accuracy(), stderr()]) +def write_order(): + """Hard rule 1: every create_asset precedes every relation-adding edit, + and the contract push comes after its port exists (last create).""" + + async def score(state: TaskState, target: Target) -> Score: + calls = _tool_calls(state) + creates = [i for i, c in enumerate(calls) if _is(c, "create_asset")] + relation_edits = [ + i + for i, c in enumerate(calls) + if _is(c, "edit_asset") and "add_relation" in str(c.arguments) + ] + contract_pushes = [ + i for i, c in enumerate(calls) if _is(c, "push_data_contract_manifest") + ] + + if not creates: + return Score(value=INCORRECT, explanation="no create_asset calls at all") + + problems = [] + if relation_edits and min(relation_edits) < max(creates): + problems.append( + f"add_relation at #{min(relation_edits)} before " + f"last create_asset at #{max(creates)}" + ) + if contract_pushes and min(contract_pushes) < max(creates): + problems.append( + f"contract pushed at #{min(contract_pushes)} before " + f"last create_asset at #{max(creates)}" + ) + + return Score( + value=INCORRECT if problems else CORRECT, + explanation="; ".join(problems) + or ( + f"{len(creates)} creates, then {len(relation_edits)} relation " + f"edits, {len(contract_pushes)} contract push(es)" + ), + ) + + return score diff --git a/evals/scripts/cleanup.py b/evals/scripts/cleanup.py new file mode 100644 index 0000000..a9882e2 --- /dev/null +++ b/evals/scripts/cleanup.py @@ -0,0 +1,94 @@ +"""Delete the assets a data-product-create eval run created, so back-to-back +runs stay independent (create_asset gates on duplicate names, and the skill +itself stops when it finds an existing product over the same tables). + +Walks outward from the fixture's expected data product name: product -> ports +-> contracts. Tables are never touched. + +Dry-run by default: + + python scripts/cleanup.py # list what would be deleted + python scripts/cleanup.py --apply # actually delete +""" + +import argparse +import os +import sys +from pathlib import Path + +import yaml + +EVALS_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(EVALS_DIR)) + +from scorers.collibra import sync_client + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--apply", action="store_true", help="actually delete") + parser.add_argument( + "--fixture", + default=os.environ.get( + "EVAL_FIXTURE", EVALS_DIR / "fixtures/data_product_create.yaml" + ), + ) + args = parser.parse_args() + + expected = (yaml.safe_load(Path(args.fixture).read_text()) or {}).get( + "expected", {} + ) + product_name = expected.get("data_product_name") + if not product_name: + sys.exit(f"fixture {args.fixture} has no expected.data_product_name") + + with sync_client() as c: + + def type_name(asset_id: str) -> str: + r = c.get(f"/assets/{asset_id}") + r.raise_for_status() + return (r.json().get("type") or {}).get("name", "") + + def related(**params: str) -> list[dict]: + r = c.get("/relations", params=params) + r.raise_for_status() + return r.json().get("results", []) + + r = c.get("/assets", params={"name": product_name, "nameMatchMode": "EXACT"}) + r.raise_for_status() + products = r.json().get("results", []) + if not products: + print(f"nothing to do: no asset named {product_name!r}") + return + + # id -> name, in deletion order: contracts, then ports, then product + to_delete: dict[str, str] = {} + for product in products: + ports = [ + rel["target"] + for rel in related(sourceId=product["id"]) + if rel.get("target") + and type_name(rel["target"]["id"]) == "Data Product Port" + ] + for port in ports: + for rel in related(targetId=port["id"]): + src = rel.get("source") + if src and type_name(src["id"]) == "Data Contract": + to_delete[src["id"]] = src["name"] + for port in ports: + to_delete[port["id"]] = port["name"] + to_delete[product["id"]] = product["name"] + + for asset_id, name in to_delete.items(): + if args.apply: + c.delete(f"/assets/{asset_id}").raise_for_status() + print(f"deleted {name!r} ({asset_id})") + else: + print(f"would delete {name!r} ({asset_id})") + + if not args.apply: + print("\ndry run — rerun with --apply to delete") + + +if __name__ == "__main__": + main() diff --git a/evals/scripts/teardown_seeded.py b/evals/scripts/teardown_seeded.py new file mode 100644 index 0000000..c0d8163 --- /dev/null +++ b/evals/scripts/teardown_seeded.py @@ -0,0 +1,142 @@ +"""Delete everything a seeded execution run created. + +Two stages, and the order matters: + + 1. **Sweep** the assets the *model* created — Data Product, Port(s), Data + Contract. Phase 4 of the skill has the model choose the product's target + domain, so these normally land OUTSIDE the seeded community and the cascade + in stage 2 will not touch them. They are only findable by navigating from + the seeded fact table, so this must run FIRST — once the community is gone, + the anchor is gone with it. + 2. **Cascade** — one `DELETE /communities/{id}` removes the domain and every + seeded asset. Verified: 23/23 assets returned 404 afterwards. + +Teardown is hygiene, not correctness. Because every run's names are unique, a +skipped teardown can no longer cause a false pass — only clutter. That is why +this is a script driven by the seed manifest rather than a post-run hook: a hook +does not run when a rollout crashes, which is exactly when mess is left behind. + + python scripts/teardown_seeded.py # dry run, all runs + python scripts/teardown_seeded.py --apply + python scripts/teardown_seeded.py --tag probe1 --apply +""" + +import argparse +import json +import sys +from pathlib import Path + +EVALS_DIR = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(EVALS_DIR)) + +from scorers.collibra import sync_client +from scorers.seeded import ( + REL_CONTRACT_GOVERNS_PORT, + REL_PORT_IMPLEMENTED_AS_TABLE, + REL_PRODUCT_EXPOSES_PORT, +) +from solvers.seed import MANIFEST + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--apply", action="store_true", help="actually delete") + parser.add_argument("--tag", help="only tear down this seed tag") + parser.add_argument("--manifest", default=str(MANIFEST)) + args = parser.parse_args() + + manifest = Path(args.manifest) + if not manifest.exists(): + print(f"nothing to do: no manifest at {manifest}") + return + + seeds = [json.loads(line) for line in manifest.read_text().splitlines() if line.strip()] + if args.tag: + seeds = [s for s in seeds if s.get("tag") == args.tag] + if not seeds: + print("nothing to do: no matching seed records") + return + + with sync_client() as client: + + def relations(**params) -> list[dict]: + response = client.get("/relations", params={**params, "limit": 500}) + response.raise_for_status() + return response.json().get("results", []) + + def sources(rels, type_id): + return [ + r["source"] + for r in rels + if (r.get("type") or {}).get("id") == type_id and r.get("source") + ] + + for seed in seeds: + tag = seed.get("tag", "?") + print(f"\n=== seed {tag} ({seed.get('community_name')})") + + # Stage 1 — model-created assets, reachable only via the fact table. + swept: dict[str, str] = {} + fact_id = seed.get("fact_table_id") + if fact_id and client.get(f"/assets/{fact_id}").status_code == 200: + ports = sources( + relations(targetId=fact_id), REL_PORT_IMPLEMENTED_AS_TABLE + ) + for port in ports: + onto_port = relations(targetId=port["id"]) + # Contracts first, then the product, then the port itself. + for contract in sources(onto_port, REL_CONTRACT_GOVERNS_PORT): + swept[contract["id"]] = f"Data Contract {contract.get('name')!r}" + for product in sources(onto_port, REL_PRODUCT_EXPOSES_PORT): + swept[product["id"]] = f"Data Product {product.get('name')!r}" + swept[port["id"]] = f"Port {port.get('name')!r}" + elif fact_id: + print(" (seeded fact table already gone — cannot sweep model assets)") + + for asset_id, label in swept.items(): + if args.apply: + response = client.delete(f"/assets/{asset_id}") + if response.status_code not in (200, 204, 404): + response.raise_for_status() + print(f" deleted {label}") + else: + print(f" would delete {label}") + if not swept: + print(" no model-created assets found") + + # Stage 2 — one cascading delete for the seeded subtree. + community_id = seed.get("community_id") + if not community_id: + print(" no community_id recorded; skipping cascade") + continue + if client.get(f"/communities/{community_id}").status_code == 404: + print(f" community {seed.get('community_name')!r} already gone") + continue + if args.apply: + response = client.delete(f"/communities/{community_id}") + if response.status_code not in (200, 204, 404): + response.raise_for_status() + print(f" deleted community {seed.get('community_name')!r} (cascades)") + else: + print( + f" would delete community {seed.get('community_name')!r} " + "(cascades to domain + all seeded assets)" + ) + + if args.apply: + # Only clear records we actually processed, so a --tag run does not drop + # the rest of the manifest. + done = {s.get("tag") for s in seeds} + remaining = [ + line + for line in manifest.read_text().splitlines() + if line.strip() and json.loads(line).get("tag") not in done + ] + manifest.write_text("\n".join(remaining) + ("\n" if remaining else "")) + print(f"\nmanifest: {len(remaining)} record(s) left") + else: + print("\ndry run — rerun with --apply to delete") + + +if __name__ == "__main__": + main() diff --git a/evals/solvers/__init__.py b/evals/solvers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/evals/solvers/seed.py b/evals/solvers/seed.py new file mode 100644 index 0000000..a4912de --- /dev/null +++ b/evals/solvers/seed.py @@ -0,0 +1,346 @@ +"""Seed a throwaway star schema per rollout, in one API request. + +Replaces the fixture's hardcoded source-table UUID. Each rollout gets its own +community → domain → schema → tables → columns, named `TEST__`, so: + + * the Data Product name the skill derives from the table is unique, which + removes `create_asset`'s duplicate gating and lets rollouts run in parallel; + * the scorer can anchor on a known table UUID and walk the graph outward + instead of matching a product by name — so a leftover product from another + run is unreachable and cannot be mistaken for success; + * teardown is one cascading DELETE of the community. + +Everything is created by a single `POST /import/json-job`: the import command +format carries assets, attributes *and* relations inline. Verified live — +1 community, 1 domain, 23 assets, 1 attribute, 22 relations, 0 errors. +""" + +import asyncio +import json +from pathlib import Path + +import httpx +from inspect_ai.solver import Generate, Solver, TaskState, solver + +from scorers import collibra + +EVALS_DIR = Path(__file__).resolve().parents[1] +MANIFEST = EVALS_DIR / "logs" / "seed-manifest.jsonl" + +# System UUIDs — out-of-the-box types, stable across instances. Probed on a live +# instance rather than assumed; see docs/skill_adherence.md. +DOMAIN_TYPE_PHYSICAL = "Physical Data Dictionary" +# Data Product / Data Product Port / Data Contract are only allowed in a domain +# of this type, so without one the skill cannot reach Phase 7 at all: every +# prepare_create_asset returns "domain is required … Filtered to Data Product +# Catalog domains" with an empty option list. Seeding one per rollout is what +# makes this arm independent of how the instance happens to be set up. +DOMAIN_TYPE_DP_CATALOG = "Data Product Catalog" +REL_SCHEMA_CONTAINS_TABLE = "00000000-0000-0000-0000-000000007043" +REL_COLUMN_PART_OF_TABLE = "00000000-0000-0000-0000-000000007042" +ATTR_DESCRIPTION = "00000000-0000-0000-0000-000000003114" + +FACT_TABLE = "ORDERS" +FACT_DESCRIPTION = "Fact table of individual sales order lines." + +# Dimension names and the shared key columns are the whole point: Phase 3 of the +# skill picks related tables by plain-language reasoning over *names*, not by +# foreign keys or lineage. ORDERS carrying CUSTOMER_ID/PRODUCT_ID/STORE_ID is +# what gives that reasoning something to latch onto. +TABLE_COLUMNS = { + FACT_TABLE: [ + "ORDER_ID", + "CUSTOMER_ID", + "PRODUCT_ID", + "STORE_ID", + "ORDER_TOTAL", + "ORDER_DATE", + ], + "CUSTOMER": ["CUSTOMER_ID", "CUSTOMER_NAME", "EMAIL", "CITY"], + "PRODUCT": ["PRODUCT_ID", "PRODUCT_NAME", "CATEGORY", "UNIT_PRICE"], + "STORE": ["STORE_ID", "STORE_NAME", "REGION", "COUNTRY"], +} + + +def names(tag: str) -> dict: + """Every name this seed will create, derived from one dynamic tag.""" + return { + "community": f"TEST_{tag}_COMMUNITY", + "domain": f"TEST_{tag}_SALES_DOMAIN", + # Where the agent writes. Per-rollout and inside the same community, so + # concurrent rollouts cannot write into each other's catalog and the + # community cascade still removes everything in one request. + "dp_domain": f"TEST_{tag}_DP_CATALOG", + "schema": f"TEST_{tag}_SALES", + "fact_table": f"TEST_{tag}_{FACT_TABLE}", + "dimension_tables": [ + f"TEST_{tag}_{t}" for t in TABLE_COLUMNS if t != FACT_TABLE + ], + } + + +def build_commands(tag: str) -> list[dict]: + """The import document: community, domain, and every asset with its relations. + + Relations are declared on the *source* asset via a `:TARGET` + key listing its targets. That format is undocumented in the OpenAPI schema — + it was confirmed against a live instance (the import summary reported + `RELATION added=22`). + """ + n = names(tag) + domain_ref = {"name": n["domain"], "community": {"name": n["community"]}} + + def ident(name: str) -> dict: + return {"name": name, "domain": domain_ref} + + def asset(name: str, type_name: str, relations=None, description=None) -> dict: + cmd = { + "resourceType": "Asset", + "identifier": ident(name), + "name": name, + "type": {"name": type_name}, + } + if relations: + cmd["relations"] = relations + if description: + cmd["attributes"] = {ATTR_DESCRIPTION: [{"value": description}]} + return cmd + + table_names = [f"TEST_{tag}_{t}" for t in TABLE_COLUMNS] + commands = [ + {"resourceType": "Community", "identifier": {"name": n["community"]}}, + { + "resourceType": "Domain", + "identifier": domain_ref, + "type": {"name": DOMAIN_TYPE_PHYSICAL}, + }, + # Empty on purpose: the agent fills it in Phase 7. Its only job is to + # exist, so prepare_create_asset has a legal home to offer for the + # Data Product, its Port and the Data Contract. + { + "resourceType": "Domain", + "identifier": { + "name": n["dp_domain"], + "community": {"name": n["community"]}, + }, + "type": {"name": DOMAIN_TYPE_DP_CATALOG}, + }, + # The schema owns the tables: "Schema contains Table", schema as source. + asset( + n["schema"], + "Schema", + relations={ + f"{REL_SCHEMA_CONTAINS_TABLE}:TARGET": [ident(t) for t in table_names] + }, + ), + ] + + for table, columns in TABLE_COLUMNS.items(): + table_name = f"TEST_{tag}_{table}" + commands.append( + asset( + table_name, + "Table", + description=FACT_DESCRIPTION if table == FACT_TABLE else None, + ) + ) + for column in columns: + # Note the direction: the *column* is the source of + # "Column is part of Table". There is no Table->Column relation type + # at all, so columns necessarily arrive as incoming relations on the + # table. (The skill's Phase 1 says "outgoing Column relations", which + # is why this eval exists — see docs/skill_adherence.md.) + commands.append( + asset( + f"{table_name}_{column}", + "Column", + relations={ + f"{REL_COLUMN_PART_OF_TABLE}:TARGET": [ident(table_name)] + }, + ) + ) + + return commands + + +async def _request(client: httpx.AsyncClient, method: str, url: str, **kwargs): + """One request, retrying transient gateway failures. + + The dev instance returned `503 no healthy upstream` for a few seconds + mid-session. Losing a rollout — and its seeded assets — to a blip that clears + in 5s is not worth it. + """ + last = None + for attempt in range(4): + response = await client.request(method, url, **kwargs) + if response.status_code < 500: + return response + last = response + await asyncio.sleep(2 * (attempt + 1)) + return last + + +async def _await_job(client: httpx.AsyncClient, job_id: str, timeout_s: int = 180): + """Poll an import job to a terminal state.""" + deadline = timeout_s / 1.5 + for _ in range(int(deadline)): + job = (await _request(client, "GET", f"/jobs/{job_id}")).json() + if job.get("state") in ("COMPLETED", "ERROR", "CANCELED"): + return job + await asyncio.sleep(1.5) + raise RuntimeError(f"import job {job_id} did not finish within {timeout_s}s") + + +async def seed(tag: str) -> dict: + """Create the star schema for `tag` and return its identifiers. + + Fails loudly: a partially-seeded graph would be scored as a skill failure, + which is a far more expensive kind of wrong than a crashed setup. + """ + n = names(tag) + commands = build_commands(tag) + expected_assets = sum(1 for c in commands if c["resourceType"] == "Asset") + + async with collibra.client() as client: + response = await _request( + client, + "POST", + "/import/json-job", + files={"file": ("seed.json", json.dumps(commands), "application/json")}, + data={"fileName": "seed.json", "continueOnError": "false"}, + ) + if response.status_code != 200: + raise RuntimeError(f"import job rejected: {response.status_code} {response.text}") + + job_id = response.json()["id"] + job = await _await_job(client, job_id) + errors = ( + await _request(client, "GET", f"/import/results/{job_id}/errors") + ).json() + if job.get("result") != "SUCCESS" or errors.get("total"): + raise RuntimeError( + f"seed import failed (state={job.get('state')} " + f"result={job.get('result')} errors={errors.get('total')}): " + f"{json.dumps(errors.get('results', [])[:3])}" + ) + + # One call resolves every seeded UUID. This is a name lookup, but scoped + # to a domain we just created, so it cannot collide with another run. + domains = ( + await _request(client, "GET", "/domains", params={"name": n["domain"]}) + ).json()["results"] + if not domains: + raise RuntimeError(f"seeded domain {n['domain']!r} not found after import") + domain_id = domains[0]["id"] + + # Checked separately rather than assumed: if the instance's Data Product + # Catalog type is missing or renamed, the import still succeeds and the + # failure would only surface as the agent finding nowhere to write — + # scored as a skill failure, which is the wrong diagnosis. + dp_domains = ( + await _request(client, "GET", "/domains", params={"name": n["dp_domain"]}) + ).json()["results"] + if not dp_domains: + raise RuntimeError( + f"seeded {DOMAIN_TYPE_DP_CATALOG} domain {n['dp_domain']!r} not found " + "after import — the agent would have nowhere to create the Data " + "Product, and the arm would score that as a SKILL.md failure" + ) + dp_domain_id = dp_domains[0]["id"] + + assets = ( + await _request( + client, + "GET", + "/assets", + params={"domainId": domain_id, "limit": 500}, + ) + ).json()["results"] + if len(assets) != expected_assets: + raise RuntimeError( + f"expected {expected_assets} seeded assets, found {len(assets)}" + ) + by_name = {a["name"]: a["id"] for a in assets} + + communities = ( + await _request( + client, "GET", "/communities", params={"name": n["community"]} + ) + ).json()["results"] + + result = { + "tag": tag, + "community_id": communities[0]["id"] if communities else None, + "community_name": n["community"], + "domain_id": domain_id, + "dp_domain_id": dp_domain_id, + "dp_domain_name": n["dp_domain"], + "schema_id": by_name[n["schema"]], + "schema_name": n["schema"], + "fact_table_id": by_name[n["fact_table"]], + "fact_table_name": n["fact_table"], + "dimension_table_ids": [by_name[t] for t in n["dimension_tables"]], + "dimension_table_names": n["dimension_tables"], + "column_names": [ + f"TEST_{tag}_{t}_{c}" for t, cols in TABLE_COLUMNS.items() for c in cols + ], + } + + _record(result) + return result + + +def _record(result: dict) -> None: + """Append to the teardown manifest. + + A file rather than a post-run hook: a hook does not run when a rollout + crashes, which is exactly the case that leaves assets behind. + """ + MANIFEST.parent.mkdir(parents=True, exist_ok=True) + with MANIFEST.open("a") as handle: + handle.write(json.dumps(result) + "\n") + + +def _tag(state: TaskState) -> str: + """A short, filesystem- and Collibra-safe unique tag for this rollout. + + Uses the per-rollout uuid so parallel epochs cannot collide. Not + random/time-based, which would make a resumed run seed a different graph than + the one its cached results describe. + """ + raw = getattr(state, "uuid", None) or f"{state.sample_id}{state.epoch}" + return str(raw).replace("-", "")[:10].lower() + + +@solver +def seed_star_schema( + placeholder: str = "{source_table}", + domain_placeholder: str = "{target_domain}", +) -> Solver: + """Setup solver: seed a fresh star schema, then point the prompt at it. + + The prompt has to be filled in here rather than at dataset construction: + the seeded assets do not exist until this runs, and each epoch of the same + sample needs its own. So the sample's `input` carries both placeholders + verbatim and we substitute the real names once they are known. + + Naming the target domain is what makes concurrent rollouts safe. Phase 4 of + the skill has the *user* pick a domain from every one that accepts a Data + Product, so with several rollouts in flight the agent would be offered every + other rollout's catalog too and could write into one of them. Supplying it is + the same substitution this arm already makes for the user's sign-off, and it + keeps every asset inside the community the cascade deletes. + """ + + async def solve(state: TaskState, generate: Generate) -> TaskState: + result = await seed(_tag(state)) + state.metadata["seed"] = result + + prompt = state.user_prompt + if prompt is not None: + prompt.text = prompt.text.replace( + placeholder, result["fact_table_name"] + ).replace(domain_placeholder, result["dp_domain_name"]) + return state + + return solve diff --git a/evals/tasks/data_product_create.py b/evals/tasks/data_product_create.py new file mode 100644 index 0000000..3c88ac0 --- /dev/null +++ b/evals/tasks/data_product_create.py @@ -0,0 +1,105 @@ +"""Happy-path eval for the collibra/data-product-create skill. + +Fill in fixtures/data_product_create.yaml (or point EVAL_FIXTURE at another +fixture file), then: + + inspect eval tasks/data_product_create.py --model anthropic/claude-sonnet-5 + inspect view +""" + +import os +import sys +from pathlib import Path + +import yaml + +EVALS_DIR = Path(__file__).resolve().parents[1] +REPO_ROOT = EVALS_DIR.parent +sys.path.insert(0, str(EVALS_DIR)) + +from inspect_ai import Task, task +from inspect_ai.agent import react +from inspect_ai.dataset import Sample +from inspect_ai.solver import system_message +from inspect_ai.tool import mcp_server_stdio + +from scorers.end_state import data_product_end_state +from scorers.trajectory import skill_loaded, write_order + +# Real MCP clients receive chip's server-level instructions, which tell the +# model to discover and load skills before multi-step workflows. Inspect does +# not surface those instructions, so we replicate the relevant part here — +# without it we would be measuring Inspect's plumbing, not the skill. +SYSTEM_MESSAGE = """You are connected to a Collibra MCP server. Before composing +any multi-step Collibra workflow, discover the relevant skill guide by calling +list_collibra_skills and load the matching one with load_collibra_skill, then +follow it.""" + +PROMPT_TEMPLATE = """Create a Collibra Data Product from the table {source_table}. + +Include only the dimension tables you identify with high confidence and skip +anything you would normally ask me to confirm. Treat my sign-off on your +proposal as already given — proceed with creation without waiting, and finish +by reporting the UUIDs of everything you created.""" + + +def _load_fixture() -> dict: + path = Path( + os.environ.get("EVAL_FIXTURE", EVALS_DIR / "fixtures/data_product_create.yaml") + ) + fixture = yaml.safe_load(path.read_text()) + if not fixture.get("source_table"): + raise ValueError( + f"fixture {path} has no source_table — fill it in before running " + "(see the comments in the fixture file)" + ) + expected = fixture.get("expected") or {} + for key in ("data_product_name", "grouped_tables"): + if not expected.get(key): + raise ValueError(f"fixture {path} is missing expected.{key}") + return fixture + + +def chip_server(): + args = [ + "--experimental=skills", + # Working-tree skills replace the embedded catalog on name match, + # so SKILL.md edits are picked up with no rebuild. + f"--skills-dir={REPO_ROOT / 'pkg/skills/files'}", + ] + # All 28 tool schemas are ~20odel ck input tokens per mall. Real clients + # see all of them, so unset (the default) is the faithful setup — but on + # low API rate-limit tiers a request that large can 429 forever. Set + # EVAL_ENABLED_TOOLS to trim (see README for the skill's minimal set). + enabled_tools = os.environ.get("EVAL_ENABLED_TOOLS") + if enabled_tools: + args.append(f"--enabled-tools={enabled_tools}") + return mcp_server_stdio( + name="collibra", + command=str(REPO_ROOT / ".build/chip"), + args=args, + cwd=REPO_ROOT, # chip resolves ./mcp.yaml from here; env vars still win + ) + + +@task +def data_product_create_happy_path(): + fixture = _load_fixture() + return Task( + dataset=[ + Sample( + id="happy-path", + input=PROMPT_TEMPLATE.format(source_table=fixture["source_table"]), + target=fixture["expected"]["data_product_name"], + metadata={"expected": fixture["expected"]}, + ) + ], + solver=[ + system_message(SYSTEM_MESSAGE), + react(tools=[chip_server()]), + ], + scorer=[data_product_end_state(), skill_loaded(), write_order()], + # The full flow takes ~30-50 tool round-trips; a runaway loop should + # fail fast instead of burning tokens. + message_limit=120, + ) diff --git a/evals/tasks/skill_arms.py b/evals/tasks/skill_arms.py new file mode 100644 index 0000000..8769436 --- /dev/null +++ b/evals/tasks/skill_arms.py @@ -0,0 +1,601 @@ +"""Ablation arms for skill selection: lookup, match, adherence. + +data_product_create.py measures everything at once, so a failure says +"something is wrong" without saying what. These arms each remove one layer of +scaffolding, so where the score drops tells you which thing to fix: + + skill_lookup chip's real instructions -> the instructions text + skill_lookup_no_instructions no system message -> (the control) + skill_match "discover skills first" -> the `description` + skill_adherence skill named outright -> the SKILL.md body + +The first two are the same arm under two instruction conditions, and the *gap* +between their scores is the finding — see skill_lookup's docstring. Run them +in one command so they share a model and a moment. + +The cheap arms make no writes (they run with a read-only tool surface), need no +cleanup, and cost 1-3 tool calls — so they run at high epoch counts in parallel +and are cheap enough to gate a PR on. Only skill_adherence is a live-write run. + + inspect eval tasks/skill_arms.py@skill_lookup \\ + tasks/skill_arms.py@skill_lookup_no_instructions + inspect eval tasks/skill_arms.py@skill_match -T epochs=5 + inspect eval tasks/skill_arms.py@skill_adherence -T epochs=3 +""" + +import asyncio +import os +import sys +from concurrent.futures import ThreadPoolExecutor +from functools import lru_cache +from pathlib import Path +from typing import NamedTuple + +EVALS_DIR = Path(__file__).resolve().parents[1] +REPO_ROOT = EVALS_DIR.parent +sys.path.insert(0, str(EVALS_DIR)) + +from inspect_ai import Task, task +from inspect_ai.agent import AgentPrompt, react +from inspect_ai.dataset import Sample +from inspect_ai.solver import system_message +from inspect_ai.tool import mcp_server_stdio + +from scorers.seeded import columns_discovered, seeded_end_state +from scorers.skill_choice import ( + LIST_TOOL, + LOAD_TOOL, + _canonical, + any_skill_discovered, + skill_citation_consistency, + skill_discovery_first, + skill_selected, + write_sequence, +) +from solvers.seed import seed_star_schema + +# Real MCP clients receive chip's server-level instructions, which tell the +# model to discover and load skills before multi-step workflows. Inspect does +# not surface those instructions, so we replicate the relevant part here — +# without it we would be measuring Inspect's plumbing, not the skill. +SYSTEM_MESSAGE = """You are connected to a Collibra MCP server. Before composing +any multi-step Collibra workflow, discover the relevant skill guide by calling +list_collibra_skills and load the matching one with load_collibra_skill, then +follow it.""" + +# skill_adherence's user prompt. `{source_table}` and `{target_domain}` are NOT +# formatted here — neither exists until the setup solver seeds it, and every epoch +# needs its own. solvers/seed.py::seed_star_schema substitutes both into the user +# message at run time, once the real names are known. Rename a placeholder here +# and you must rename its counterpart default there too. +# +# Naming the skill up front is what makes this the *adherence* arm: routing is +# handed over, so a failure is the SKILL.md body's fault. The "sign-off already +# given" clause overrides the skill's confirm-once gate, without which an +# unattended run would block forever waiting for approval. +# +# Naming the target domain stands in for the user the same way. Phase 4 has the +# user choose from every domain that accepts a Data Product, which with parallel +# rollouts in flight would include every other rollout's seeded catalog — so the +# agent could write into one of them. Supplying it removes that ambiguity instead +# of hoping the agent resolves it, and keeps every created asset inside the +# community teardown cascades. +ADHERENCE_PROMPT = """Use the collibra/data-product-create skill for this task. + +Create a Collibra Data Product from the table {source_table}. + +Create the Data Product, its Port and its Data Contract in the domain +{target_domain}. + +Include only the dimension tables you identify with high confidence and skip +anything you would normally ask me to confirm. Treat my sign-off on your +proposal as already given — proceed with creation without waiting, and finish +by reporting the UUIDs of everything you created.""" + +# Enough surface to identify tables and read the catalog, with every write tool +# removed. The arms therefore *cannot* mutate Collibra and terminate naturally +# once they run out of read work — no fragile low message_limit needed to force +# an early stop (a truncated run looks identical to a failed one). Also drops 25 +# of chip's 31 tool schemas, cutting the ~20k-token per-call overhead that makes +# low-tier API keys 429 forever. +READ_ONLY_TOOLS = ( + "list_collibra_skills,load_collibra_skill,search_asset_keyword," + "get_asset_details,get_table_semantics,list_asset_types" +) + + +def chip_server(enabled_tools: str | None = None): + """chip as an MCP stdio server, with an optional tool allow-list. + + Mirrors tasks/data_product_create.py::chip_server but takes the tool set as + an argument: the arms need different surfaces within one process, and the + original reads the process-wide EVAL_ENABLED_TOOLS. + """ + args = [ + "--experimental=skills", + # Working-tree skills override the embedded catalog on name match, so + # SKILL.md edits are picked up with no Go rebuild. + f"--skills-dir={REPO_ROOT / 'pkg/skills/files'}", + ] + enabled_tools = enabled_tools or os.environ.get("EVAL_ENABLED_TOOLS") + if enabled_tools: + args.append(f"--enabled-tools={enabled_tools}") + return mcp_server_stdio( + name="collibra", + command=str(REPO_ROOT / ".build/chip"), + args=args, + cwd=REPO_ROOT, # chip resolves ./mcp.yaml from here; env vars still win + ) + + +@lru_cache(maxsize=1) +def _sequential_prompt() -> AgentPrompt: + """react's own prompt, minus its "prioritize parallel tool calls" directive. + + A skill is a playbook to consult *before* acting, so an instruction to batch + independent calls pushes directly against the behaviour these arms measure: + it invites firing search_asset_keyword in the same turn as + list_collibra_skills, which skill_discovery_first scores as not having + waited. The directive comes from Inspect, not from chip, so any number it + moves is an artefact of the harness. + + It also matters for skill_adherence: parallel writes land in one message, and + write_sequence's ordering checks index into the call sequence. + + Built by subtraction from upstream's own constants rather than pasted, so a + reworded default still reaches the model. Raises if the directive is no longer + where it was, because silently running with it back in place would quietly + reintroduce the contamination. + """ + from inspect_ai.agent._types import ( + DEFAULT_ASSISTANT_PROMPT, + PARALLEL_TOOLS_PROMPT, + ) + + if PARALLEL_TOOLS_PROMPT not in DEFAULT_ASSISTANT_PROMPT: + raise RuntimeError( + "inspect_ai's PARALLEL_TOOLS_PROMPT is no longer part of " + "DEFAULT_ASSISTANT_PROMPT — re-check what react() now sends before " + "trusting any ordering metric (skill_discovery_first, write_sequence)." + ) + return AgentPrompt( + assistant_prompt=DEFAULT_ASSISTANT_PROMPT.replace( + f" {PARALLEL_TOOLS_PROMPT}", "" + ) + ) + + +def _run_sync(coro_fn): + """Run an async function from task-construction code. + + Task functions are called synchronously, but Inspect also has an async eval + entry point — and a bare asyncio.run() raises "cannot be called from a + running event loop" if construction ever happens inside one. Falling back to + a private loop on a worker thread makes this correct either way. + """ + try: + asyncio.get_running_loop() + except RuntimeError: + return asyncio.run(coro_fn()) + with ThreadPoolExecutor(max_workers=1) as pool: + return pool.submit(lambda: asyncio.run(coro_fn())).result() + + +@lru_cache(maxsize=1) +def _server_instructions() -> str: + """chip's real `initialize` instructions, fetched over MCP. + + Cached because the text is a property of the binary, not of the run, and + reading it costs a throwaway chip process — worth doing once however many + times a task gets constructed. + + The hand-written SYSTEM_MESSAGE in data_product_create.py is a paraphrase of + pkg/skills/register.go's Instructions: more imperative, missing the + exceptions paragraph, and free to drift out of sync the moment someone edits + the Go string. Fetching the real thing measures the text that actually + ships. + + Raises rather than falling back to the paraphrase — a silent fallback is + exactly how the drift this avoids would hide. + """ + from mcp import ClientSession, StdioServerParameters + from mcp.client.stdio import stdio_client + + async def fetch() -> str: + params = StdioServerParameters( + command=str(REPO_ROOT / ".build/chip"), + args=[ + "--experimental=skills", + f"--skills-dir={REPO_ROOT / 'pkg/skills/files'}", + ], + cwd=str(REPO_ROOT), + ) + async with stdio_client(params) as (read, write): + async with ClientSession(read, write) as session: + result = await session.initialize() + return result.instructions or "" + + instructions = _run_sync(fetch) + if not instructions.strip(): + raise RuntimeError( + "chip returned empty initialize instructions — is --experimental=skills " + "set and .build/chip current? Refusing to fall back to the hand-written " + "paraphrase, which would silently measure the wrong text." + ) + return instructions + + +class Case(NamedTuple): + """One routing question: a prompt, and the skill(s) that should win it.""" + + id: str + prompt: str + # Empty = no skill applies and loading one is over-triggering. More than one + # = the request is genuinely ambiguous and any of them is a defensible read; + # scoring it against a single "right" answer would invent a failure. + expect: tuple[str, ...] + # Set on the cases skill_lookup runs. That arm asks "does the model look + # for a skill at all?", so it needs one prompt per *shape* of request and + # nothing more — see LOOKUP_CASES below for why the rest are excluded. + request_shape: str | None = None + + +# The shared pool both cheap arms draw from — the whole test, readable in one +# place. Deliberately *not* in fixtures/, and deliberately free of any +# environment-specific value: routing is decided from the *wording* of a request, +# not from whether the assets it names exist. Every case names invented assets +# ("Churn Rate", "SALES.ORDERS"), so these arms run against any Collibra instance +# with no setup. +# +# In particular these do NOT read fixtures/data_product_create.yaml. That fixture +# validates expected.data_product_name and expected.grouped_tables, which are +# write-arm concerns — sourcing a table name from it would gate the cheapest, +# most portable arms behind the most environment-specific config in the harness. +# +# One pool rather than one list per arm, because the arms are only comparable if +# their shared prompts are identical. skill_selected is read across both arms to +# separate "the model never looked" from "the model looked and chose wrong"; if a +# prompt drifted between them, each arm would still pass its own checks while that +# inference broke silently. Deriving both views from here makes drift impossible. +# +# The negatives are the point. With only the first case you measure recall ("is +# the right skill found when it applies?") and never precision ("is it found when +# it does not?"). A `description` sharpened until it wins case 1 can start +# hijacking the rest — invisible without them, and a risk on every skill edit. +CASES = [ + Case( + id="data-product", + prompt="Create a Collibra Data Product from the table SALES.ORDERS.", + expect=("collibra/data-product-create",), + request_shape="multi-step write", + ), + # The sharp one: contains "Create" and "new", the exact words that make + # data-product-create attractive, but the target is a plain asset. + Case( + id="business-term-create", + prompt="Create a new Business Term for Churn Rate in the Finance domain.", + expect=("collibra/asset-create",), + ), + Case( + id="attribute-edit", + prompt=( + "Add a definition to the Customer Lifetime Value business term and " + "mark me as its steward." + ), + expect=("collibra/asset-edit",), + ), + Case( + id="lineage-trace", + prompt=( + "Where does the Monthly Recurring Revenue KPI come from? Show me the " + "upstream tables that feed it." + ), + expect=("collibra/lineage",), + request_shape="graph traversal", + ), + # A second graph-traversal prompt, so the shape is not represented by a + # single wording. With one prompt per shape a failure cannot be attributed — + # is it the shape that defeats the instructions, or just that sentence? This + # one differs on every axis available: downstream rather than upstream, + # impact-analysis framing, a column rather than a KPI as the subject. When + # the two agree, the result is about the shape and not the phrasing. + # + # Sharing lineage's expected skill with lineage-trace also gives skill_match a + # second reading on that description's recall, which is a free side benefit + # rather than the reason this exists. + Case( + id="lineage-impact", + prompt=( + "If we change the ORDERS.DISCOUNT_PCT column, which reports and " + "dashboards break?" + ), + expect=("collibra/lineage",), + request_shape="graph traversal", + ), + Case( + id="semantic-discovery", + prompt="What customer data do we have in the catalog?", + expect=("collibra/discovery",), + request_shape="simple read", + ), + # Genuinely ambiguous: "set an owner" is asset-edit, "discoverable to other + # teams" is data-product-create. Both are defensible, so both pass — what + # this case catches is landing somewhere else entirely (lineage, discovery), + # which means the routing signal is noise rather than a close call. + Case( + id="ambiguous-governance", + prompt=( + "I need the CUSTOMER table properly governed — set an owner for it " + "and make it discoverable to other teams." + ), + expect=("collibra/asset-edit", "collibra/data-product-create"), + ), + # chip's own instructions say a single obvious tool call needs no skill, so + # loading one here is over-triggering. + Case( + id="no-skill-needed", + prompt="List the asset types available in this Collibra instance.", + expect=(), + ), +] + +# skill_lookup asks "does the model consult a skill before acting?" — a question +# that varies with the *shape* of the request (is the multi-step-ness visible?), +# not with which skill is correct. So it runs one case per shape and drops: +# +# - the routing distractors (business-term-create, attribute-edit, +# ambiguous-governance) — they exist to test precision, which this arm does +# not measure, so they would only add cost; +# - no-skill-needed — actively harmful here. Not looking is the *correct* +# behaviour on that prompt, but any_skill_discovered scores "never looked" +# as INCORRECT, so including it caps a perfect model at 5/6. +# +# "One case per shape" has one deliberate exception: graph traversal carries two. +# A single prompt per shape cannot separate "this shape defeats the instructions" +# from "this sentence does", so the shape whose result you most need to trust is +# worth a second wording. Give the other shapes a second prompt only when their +# result becomes contested too; until then it buys a duplicate answer, and power +# on this arm comes from epochs. +LOOKUP_CASES = [c for c in CASES if c.request_shape] + + +def _samples(cases: list[Case]) -> list[Sample]: + """Cases as Inspect samples.""" + return [ + Sample( + id=case.id, + input=case.prompt, + target=case.expect[0] if case.expect else "none", + metadata={"expected_skill": list(case.expect)}, + ) + for case in cases + ] + + +def _stop_once(*tool_names: str): + """End the run as soon as one of `tool_names` has been called. + + Each arm stops when *its own* question is answered — nothing more is + measured after that point, and letting the run continue means the model + works through the skill's read-only discovery phases and stops only on + reaching a write it cannot perform: dozens of turns that cost tokens and + tell us nothing. + + Returning False from on_continue breaks react's loop; it is called every + turn. Runs where the tool never fires are not cut short (that would + manufacture false "never looked" verdicts) and end via submit or + message_limit. + """ + + async def on_continue(state) -> bool: + for message in state.messages: + for call in getattr(message, "tool_calls", None) or []: + if _canonical(call.function) in tool_names: + return False + return True + + return on_continue + + +def _stop_once_skill_discovered(): + """skill_lookup's gate: the model reached for the catalog, either way. + + That arm asks only "does it look?", which `list_collibra_skills` answers on + its own — so there is no reason to pay for the subsequent load. Either tool + counts, because a model may skip the listing and load directly, and that + still answers the question. + """ + return _stop_once(LIST_TOOL, LOAD_TOOL) + + +def _stop_once_skill_chosen(): + """skill_match's gate: a skill was actually loaded. + + Routing needs to know *which* skill, so unlike the lookup arm it has to + see the load itself — stopping at the listing would leave nothing to score. + """ + return _stop_once(LOAD_TOOL) + + +def _lookup_task(instructions: str | None, epochs: int) -> Task: + """Arm A's shared body; `instructions` None is the control condition. + + The two conditions are separate tasks rather than one dataset because + varying a system message *per sample* needs a custom solver and a grouped + metric, where two tasks need neither — and Inspect takes several tasks in + one command, so they still run under the same model at the same moment. + """ + return Task( + # One case per request shape, not the full routing matrix — see + # LOOKUP_CASES. Statistical power comes from epochs, not from adding + # prompts that ask the same question again. + dataset=_samples(LOOKUP_CASES), + solver=[ + *([system_message(instructions)] if instructions else []), + react( + tools=[chip_server(READ_ONLY_TOOLS)], + on_continue=_stop_once_skill_discovered(), + prompt=_sequential_prompt(), + ), + ], + # Deliberately NOT skill_selected. In this arm that scorer conflates two + # things — it fails both when the model never looked and when it looked + # and chose wrong — making it approximately + # `any_skill_discovered` x skill_match's `skill_selected`. Both factors + # are measured separately and more cleanly (here and in skill_match + # respectively), so the product adds no information and cannot be + # decomposed. `skill_match` also runs 7 cases against this arm's 3, so it is + # the better place to measure routing regardless. + scorer=[ + # The headline. `any_skill_discovered` is a binary over "ever + # looked", so it pins at 1.000 for any model that looks eventually + # and has no headroom left to show an instructions gap; this + # separates "looked" from "looked *before* acting". Both are kept: + # the pair localises the failure, since discovered-but-not-first is a + # different problem from never-looked. + skill_discovery_first(), + any_skill_discovered(), + # Near-vacuous on this arm — the early stop usually fires before the + # model writes any prose — but it costs nothing and catches the model + # reading skill names out of the list_collibra_skills response and + # then writing as though it were following one it never loaded. + skill_citation_consistency(), + ], + epochs=epochs, + # Read-only, so it ends on its own; this is a runaway-loop backstop. + message_limit=30, + ) + + +@task +def skill_lookup(epochs: int = 5): + """Arm A — does the model reach for a skill at all, under production + conditions? + + The model gets chip's real `initialize` instructions, fetched live from the + binary, which is what an actual MCP client receives. `any_skill_discovered` + is then the production number: how often the shipped setup gets the model to + look for a skill unprompted. + + That number alone does not tell you whether the ~2.2k-character + instructions text earns its place on every request — for that you need + `skill_lookup_no_instructions` and the gap between the two: + + this high, control low the instructions carry the behaviour + both high tool descriptions already do; the text is + dead weight and could be deleted + both low the text is being ignored; rewrite it + + Run both in one command so they share a model, a moment and an instance — + otherwise drift between the runs lands in the gap and reads as signal: + + inspect eval tasks/skill_arms.py@skill_lookup \\ + tasks/skill_arms.py@skill_lookup_no_instructions + """ + return _lookup_task(_server_instructions(), epochs) + + +@task +def skill_lookup_no_instructions(epochs: int = 5): + """Arm A's control — the same thing with no system message at all. + + The model sees only the tool schemas; nothing tells it skills exist. Its + score is the floor `skill_lookup` is measured against, and the absence + of a system message *is* the manipulation, so there is deliberately nothing + here but the react loop. + """ + return _lookup_task(None, epochs) + + +@task +def skill_match(epochs: int = 5): + """Arm B — given that it looks, does it pick the right skill? + + The decision to look is handed to it via SYSTEM_MESSAGE — a local copy of the + text data_product_create.py uses, deliberately not imported: that file belongs + to another author, and their edits should not silently change what this arm + measures. What is left is pure routing against real distractors: asset-create + and asset-edit both compete for "create"/"add" prompts. + + CASES includes negatives, so this measures precision as well as recall — a + `description` sharpened until it wins the happy path can start hijacking + unrelated requests, and that only shows up here. + """ + return Task( + dataset=_samples(CASES), + solver=[ + system_message(SYSTEM_MESSAGE), + react( + tools=[chip_server(READ_ONLY_TOOLS)], + on_continue=_stop_once_skill_chosen(), + prompt=_sequential_prompt(), + ), + ], + scorer=[ + skill_selected(), + any_skill_discovered(), + skill_citation_consistency(), + ], + epochs=epochs, + message_limit=30, + ) + + +@task +def skill_adherence(epochs: int = 3): + """Arm C — given the right skill, is it followed correctly? + + Routing is removed from the equation by naming the skill in the prompt, so + a failure here is the SKILL.md body's fault and nothing else. `skill_selected` + guards that premise: it should read ~100%, and a dip means the model never + loaded the skill it was handed, making the rest of the arm meaningless. + + Each rollout seeds its **own** throwaway star schema (community -> domain -> + schema -> 4 tables -> 18 columns) in a single import request, so: + + * asset names are unique per rollout, which removes create_asset's + duplicate gating — this arm is safe to run in parallel and with epochs, + unlike the fixture-based original; + * seeded_end_state anchors on the seeded table's UUID and walks outward + instead of matching a Data Product by name, so a leftover product from + another run is unreachable and cannot be scored as success; + * teardown is one cascading delete (scripts/teardown_seeded.py), and is + hygiene rather than a correctness requirement. + + `{source_table}` stays unsubstituted here on purpose — seed_star_schema + fills it in once the table exists. + + epochs defaults to 3 rather than the cheap arms' 5: this is the expensive + arm (~30-50 tool round-trips against all 31 tool schemas per rollout, plus + 23 seeded assets to tear down), so 5 would make a bare `inspect eval` a + costly surprise. 3 is the least that still yields a rate rather than a + single outcome — and the scorers declare `stderr()`, which is undefined at + n=1. No `--max-samples 1`: unique per-rollout names are exactly what makes + concurrent epochs safe here. + """ + return Task( + dataset=[ + Sample( + id="execution", + input=ADHERENCE_PROMPT, + target="collibra/data-product-create", + metadata={"expected_skill": "collibra/data-product-create"}, + ) + ], + setup=seed_star_schema(), + solver=[react(tools=[chip_server()], prompt=_sequential_prompt())], + scorer=[ + seeded_end_state(), + columns_discovered(), + write_sequence(), + skill_selected(), + # Guards the *order* skill_selected does not: it checks which skill + # was loaded, not that the load preceded the work. A model that + # starts searching and reads the named playbook afterwards is not + # being guided by the SKILL.md body this arm exists to measure. + skill_discovery_first(), + skill_citation_consistency(), + ], + epochs=epochs, + message_limit=120, + )