README.md

PowerContext evaluation

Evaluation suites are grouped by what they measure. Each suite owns its dataset inputs, execution, grading, reports, tests, command entry point, and any console or deployment tools it needs. This directory provides the shared Python environment and build configuration.

Directory	Purpose	Suites
coding/	Coding-task completion and continuation quality	SWE-bench Pro, Work continuity
memory/	Memory retrieval, answer quality, and write-gate evidence	LoCoMo, LoCoMo-Plus, LongMemEval-V2, Memory write gate
performance/	Load, latency, storage growth, and capacity limits	None yet
skills/	Agent instruction routing and authorization regressions	Claude Code skill-up, ZCode native CLI

evaluation/
├── pyproject.toml                 # Shared powercontext-eval environment and build
├── uv.lock
├── coding/
│   ├── swebench_pro/
│   │   ├── src/powercontext_eval_swebench_pro/
│   │   ├── tests/
│   │   ├── web/
│   │   ├── deploy/
│   │   └── docs/
│   └── work_continuity/
│       ├── src/powercontext_eval_work_continuity/
│       ├── tests/
│       ├── locks/
│       └── docs/
├── memory/
│   ├── locomo/
│   ├── locomo_plus/
│   └── longmemeval_v2/
│       ├── src/powercontext_eval_longmemeval_v2/
│       ├── locks/
│       ├── config/
│       ├── docs/
│       ├── scripts/
│       └── tests/
├── performance/
├── zcode_guidance/                 # Native ZCode Skill routing and authorization
└── skills/
    └── skill-up/

Run a suite

Run these commands from the repository root. LoCoMo and LoCoMo-Plus modules use the root project's Python dependencies. SWE-bench Pro, Work continuity, and LongMemEval-V2 share this directory's locked environment and expose independent swebench-pro, work-continuity, and longmemeval-v2 commands. Skill-up has its own validation dependencies and pinned CLI. ZCode guidance uses the root project's dependencies and its pinned native CLI.

# Install the shared environment, then choose a suite command.
uv sync --project evaluation --locked
uv run --project evaluation swebench-pro --help
uv run --project evaluation work-continuity --help
uv run --project evaluation longmemeval-v2 --help

# Memory quality: inspect a bundled, provider-free smoke plan.
uv run python -m evaluation.memory.locomo --help
uv run python -m evaluation.memory.locomo_plus run \
  --dataset-file evaluation/memory/locomo_plus/dataset/locomo_plus_smoke10.json \
  --dry-run

# Skill regression: validate the pinned Skill and controlled fixtures locally.
uv run python evaluation/skills/skill-up/sync_skill.py --check
uv run python evaluation/skills/skill-up/validate_suite.py

# Native ZCode Skill regression: validate the pin and runner without calling a model.
uv run --locked python -m evaluation.zcode_guidance.pin --check
uv run --locked python -m evaluation.zcode_guidance.run --help

The SWE-bench Pro console guide covers its Web and worker deployment. LongMemEval-V2 uses its own suite CLI. Real dataset runs may call model providers, use databases or Docker, create durable namespaces, and incur inference cost. Help, fixture validation, and dry-run plans do not establish real-model or external-backend performance.

Development and results

Run the installed suite checks together:

uv run --directory evaluation pytest -m "not live"
uv run --project evaluation ruff check evaluation/coding evaluation/memory/longmemeval_v2
uv run --project evaluation ruff format --check evaluation/coding evaluation/memory/longmemeval_v2
uv run --directory evaluation ty check
uv build --project evaluation

The powercontext-eval distribution contains three packages: powercontext_eval_swebench_pro, powercontext_eval_work_continuity, and powercontext_eval_longmemeval_v2. Editable installs and wheels expose the independent swebench-pro, work-continuity, and longmemeval-v2 commands and their suite modules. LoCoMo behavior tests remain under tests/, including tests/evaluation/. Skill-up and ZCode guidance keep their own validation tests. The Bub replay harness and product E2E tests cover cross-component acceptance.

Keep fixed datasets, locks, configuration examples, and scoring rules with their owning suite. Store local outputs in a suite's ignored results directory or an explicit directory such as .artifacts/; keep credentials and unredacted provider traces out of version control. Preserve dataset hashes and result schema identifiers so saved evidence remains interpretable.

Add suites under the category describing what they measure. Keep suite-specific execution tools, storage, workers, and Web interfaces inside that suite. Define each suite's command and errors in its own package. Report different datasets, judges, hosts, and protocols as separate experiments.