Skip to content

docs: describe the current eval API #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TheGreatAxios wants to merge 2 commits into from
Closed

docs: describe the current eval API #30

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ac5f042
docs: describe the current eval API
TheGreatAxios Oct 26, 2026
318fc91
docs: correct how prebuilt rubrics are run
TheGreatAxios Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@

`@corbits/evals` package.

Public call: `runEvaluation({ name, data, task, scorers })`. There is no
`defineEval`, no `evaluate(name, options)`, and no `runEval`. Ground truth
is `expectedOutput`. JSONL keys `expect` and `expected` throw.

The judge model is `judgeModel()` from `@corbits/evals/workflow`: one step,
inference source pinned on the agent. Thresholds are `systemOne`. `jev` is
an alias of `systemOne`, not an endpoint. Do not document `jev({ endpoint })`.

Saved experiments are format 2 (`{ format: 2, name, cases }`). Format 1 is
rejected. `ranSteps([])` fails when `steps` is empty; an observed model-only
run still passes. CLI is `corbits-eval run <file>` and exits 0, 1, or 2.

`includes` and the four prebuilt workflow scorers
(`faithfulness`, `answerRelevancy`, `correctness`, `trajectoryAccuracy`) exist.
Do not document toxicity, bias, or pii.

## Conventions

- Read `README.md` before changing public behavior.
Expand Down
49 changes: 33 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ console.log(report.markdown());

## Checks

`checks` maps a name to a check. Without `checks`, one `matches` check
`scorers` maps a name to a check. Without `scorers`, one `matches` check
compares output with `expectedOutput`. Every check gets the run
`r = { input, output, expectedOutput, steps, ms }`; `output` is whatever the run
returned.
Expand All @@ -47,19 +47,19 @@ returned.
`unsure (p=…)` in between. `{ question, yes, no }` sets its own bars.
- **Helpers**:

| helper | passes when |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `matches(field?)` | output deep-equals expectedOutput, or the same field (key or accessor) of each |
| `includes()` | both strings, and output contains expectedOutput after NFKC, trim, and whitespace collapse; value is 1 or 0 |
| `fitsSchema(schema)` | output satisfies a Standard Schema (arktype, zod, valibot) |
| `mentions(words)` | output contains every word, case-insensitive; value is the fraction found |
| `avoids(words)` | output contains none of the words |
| `ranSteps(names)` | `tool` and `step` names match `names`, order ignored; value is their F1. Agent trajectory comes from `stream()`. A reply-only send without `stream` does not satisfy `ranSteps([])` |
| `stepsAtMost(n)` | the run took at most `n` steps |
| `under({ ms })` | the run finished within `ms` |
| `ask(q, { options, pass })` | Jev picks an option named in `pass` |
| `ask(q, { options, passAt })` | Jev picks an option; its credit (0..1) is the value and passes at `passAt` |
| `ask(q, { levels, pass })` | Jev grades at level `pass` or above (levels lowest first) |
| helper | passes when |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `matches(field?)` | output deep-equals expectedOutput, or the same field (key or accessor) of each |
| `includes()` | both strings, and output contains expectedOutput after NFKC, trim, and whitespace collapse; value is 1 or 0 |
| `fitsSchema(schema)` | output satisfies a Standard Schema (arktype, zod, valibot) |
| `mentions(words)` | output contains every word, case-insensitive; value is the fraction found |
| `avoids(words)` | output contains none of the words |
| `ranSteps(names)` | `tool` and `step` names match `names`, order ignored; value is their F1. Agent trajectory comes from `stream()`. `ranSteps([])` fails when `steps` is empty. An observed model-only run (llm steps, no tool or step names) still passes |
| `stepsAtMost(n)` | the run took at most `n` steps |
| `under({ ms })` | the run finished within `ms` |
| `ask(q, { options, pass })` | Jev picks an option named in `pass` |
| `ask(q, { options, passAt })` | Jev picks an option; its credit (0..1) is the value and passes at `passAt` |
| `ask(q, { levels, pass })` | Jev grades at level `pass` or above (levels lowest first) |

`mentions` and `avoids` also take `(r) => words` for per-case words.
`mentions` and `ranSteps` take `{ atLeast }` to pass on partial credit.
Expand Down Expand Up @@ -104,7 +104,7 @@ space cannot absorb — those fail closed by design.
as a message; output is the reply. When the agent has `stream()`, steps
are recorded from that stream (`inference.done` as `llm`, `tool.start` /
`tool.done` as `tool`). A reply-only send without `stream` leaves `steps`
empty, which does not satisfy `ranSteps([])`.
empty, which fails `ranSteps([])`. An observed model-only run still passes.
- **Workflow**: `workflow(definition, { invokeStep })` from
`@corbits/evals/workflow`. A definition cannot run locally without a
step invoker, and the root never loads `@intx/workflow`, so a bare
Expand All @@ -121,7 +121,8 @@ attempt passes. `concurrency` (default 4) bounds cases in flight;
## Cases

An array of `{ name?, input, expectedOutput?, must?, skip? }`, or the path of a
JSONL file of the same objects. `name` keys the case in reports and
JSONL file of the same objects. The ground-truth key is `expectedOutput`.
`expect` and `expected` throw. `name` keys the case in reports and
compared runs and defaults to the input.

## Passing
Expand Down Expand Up @@ -157,6 +158,8 @@ cases; `.report` holds every result.

An experiment is each case × scorer pass and value — plus a Jev scorer's
probability distribution and confidence — and `about`, no times.
Saved files are format 2: `{ format: 2, name, cases }`. Any other `format`,
including format 1, is rejected.
`compare` only reads; a missing file skips drops. `save` only writes, on
every run. One path for both compares with the last experiment, then replaces
it. `compare` also takes an experiment from an earlier `runEvaluation` of the same
Expand Down Expand Up @@ -193,6 +196,20 @@ Exit codes:

`--format json` (and `--json`) come from incur and print the result as JSON.

## Workflow scorers

`@corbits/evals/workflow` also exports four prebuilt one-step judges. Each
takes a pinned `{ provider, model }` and does not call a model by itself:
`faithfulness`, `answerRelevancy`, `correctness`, and `trajectoryAccuracy`.
Pair them with `judgeModel` when you want those rubrics run. `includes` stays
a code check on the root package.

## Breaking changes

- Saved experiments must be format 2 (`{ format: 2, name, cases }`). Format 1 is rejected.
- JSONL rows use `expectedOutput`. `expect` and `expected` throw.
- `ranSteps([])` fails when `steps` is empty. An observed model-only run still passes.

## Calibrating

Thresholds are per-workload starting points, not constants. `calibrate`
Expand Down
Loading