ENG-3771: add lab project commands

[mrmoxon]

Linear: ENG-3771

Summary

Adds Lab Project support to the Prime CLI and eval client so users can set up a projectized Lab workspace, create/select/clear/inspect/update projects, and attach artifacts to projects from the terminal.

UX

prime lab setup now folds projects into the normal setup flow. When the user is authenticated and no active workspace project exists, setup creates a default project named after the workspace folder and makes it active in .prime/lab/context.json. Users can bind an existing project with prime lab setup --project <project-id-or-slug>, choose the generated project name with prime lab setup --project-name "Alphabet Sort Baselines", or keep setup local-only with prime lab setup --no-project. If the user is not authenticated or the Projects API is unavailable, local setup still succeeds and prints the follow-up prime project create path.

Projects remain compatible with existing Lab flows. Users who do not create or select a project can keep running prime train, prime eval push, and prime eval run as they do today; without an active project, those commands run without project attachment. Once a workspace has an active project, new training runs and evaluations attach to it by default. Users can override that per command with --project <project-id-or-slug> or opt out explicitly with --no-project. Active project context is local to the workspace and ignored when the CLI team or API base URL does not match the current config.

Active project context follows the active CLI account context. A personal CLI context can only set a personal project as active; a team CLI context can only set a project from that same team as active. --team-id is still supported for explicit one-off team operations such as listing, showing, creating with --no-use, assigning, or removing, but it does not switch future prime train / prime eval commands into that team. Users should run prime switch <team-slug-or-id> or set PRIME_TEAM_ID=<team-id> before making a team project active.

Active project lookup is anchored to the nearest Lab workspace marker, so nested paths such as outputs/evals/... still resolve the workspace project, but nested Lab workspaces do not accidentally inherit a parent workspace's active project.

A typical projectized workspace flow looks like:

mkdir alphabet-sort && cd alphabet-sort
prime lab setup
prime project current

prime train rl.toml
prime eval push outputs/evals/gsm8k--gpt-4/abc123
prime eval run gsm8k

For team-owned projects, switch to the team first so setup creates the default project in that team context:

prime switch <team-slug-or-id>
prime lab setup
prime train rl.toml

Users can still switch projects with prime project use <project-id>, stop default attachment with prime project clear, and organize existing artifacts with prime project assign / prime project remove for runs, evals, and adapters.

Details

• Adds prime project commands for creating/listing/showing/using/checking current/clearing/updating/assigning/removing a project.
• Adds default project creation/binding to prime lab setup, with --project, --project-name, and --no-project controls.
• Keeps local Lab setup non-blocking when the user is unauthenticated or project creation fails.
• Adds active project context resolution for training and evaluation flows.
• Guards active project writes so a project can only become the workspace default when its team scope matches the current CLI account/team context.
• Resolves active context from the nearest Lab workspace marker to avoid crossing nested workspace boundaries.
• Keeps project current --output json stable if API detail fetch fails by returning project: null and separate cached context metadata.
• Adds project IDs to eval push/create/update and hosted eval paths.
• Adds project assignment helpers for hosted training runs and adapters.
• Supports additive project assignment for runs/adapters via assign, and clear-or-specific removal via remove.
• Supports eval project assignment/clear. Targeted removal from one named project is intentionally rejected because evaluations have a single project association.
• Adds regression tests for lab setup project creation/binding/skip/fallback, training run project attachment, hosted eval project attachment, eval push forwarding, eval project clearing, and run/adapter assignment operations.
• Documents projectized Lab setup and project CLI usage in the repo READMEs, plus project assignment in the prime-evals README.

Not in this PR

• Project archive/restore is not implemented in the CLI.
• Project environment aggregation, such as prime project environments, is left for a follow-up backend-backed endpoint.

Validation

• uv run pytest packages/prime/tests/test_lab_setup.py -q
• uv run pytest packages/prime/tests/test_lab_setup.py packages/prime/tests/test_projects_cli.py packages/prime/tests/test_eval_push.py packages/prime/tests/test_hosted_eval.py packages/prime/tests/test_rl_api.py -q
• uv run pytest packages/prime/tests packages/prime-evals/tests -q (797 passed, 4 skipped)
• uv run ruff check packages/prime/src/prime_cli/lab_setup.py packages/prime/src/prime_cli/utils/projects.py packages/prime/src/prime_cli/commands/projects.py packages/prime/tests/test_lab_setup.py
• uv run ruff format --check packages/prime/src/prime_cli/lab_setup.py packages/prime/src/prime_cli/utils/projects.py packages/prime/src/prime_cli/commands/projects.py packages/prime/tests/test_lab_setup.py
• uv run ty check packages/prime/src packages/prime-evals/src
• git diff --check
• Local backend smoke: created a team project with --team-id <team-id> --no-use, confirmed no context file was written, confirmed personal context rejects project use <team-project> --team-id <team-id>, then confirmed matching PRIME_TEAM_ID=<team-id> allows project use and writes a team-scoped .prime/lab/context.json.

---

Note

Medium Risk
Touches cross-cutting Lab flows (train, eval, setup) and remote project membership APIs; behavior changes when an active project exists, though --no-project preserves prior opt-out.

Overview
Introduces Lab Projects end-to-end: a new prime project command group, workspace-scoped active project context (.prime/lab/context.json + PRIME_PROJECT_ID), and default project creation/binding during authenticated prime lab setup.

Training & evals now attach to the active project by default. prime train, prime eval push, prime eval run, and post-run hub upload accept --project / --no-project. The prime-evals client gains project_id on create/update (including clear_project via null project_id).

Artifact membership is wired through new API helpers: run and adapter project PATCH with add/remove/clear operations (optional adapter moves on runs), plus eval assign/clear via prime project assign / remove. Team scope is enforced so only projects matching the current CLI account can become active.

Docs and CI are updated (checkout/uv pins on test jobs; README project flows). Large test coverage for setup, CLI, eval push, hosted eval, and RL/adapter payloads.

^{Reviewed by Cursor Bugbot for commit db89b50. Bugbot is set up for automated code reviews on this repo. Configure here.}

[willccbb]

@mrmoxon thoughts on folding this into the setup flow? ie default = single project that shares name with workspace folder, overridable + switchable. a bit worried about abstraction creep, feels like we can have these coupled until users explicitly want to decouple

[mrmoxon]

@mrmoxon thoughts on folding this into the setup flow? ie default = single project that shares name with workspace folder, overridable + switchable. a bit worried about abstraction creep, feels like we can have these coupled until users explicitly want to decouple

Defo agree, am making the lab setup flow more subtle with next commit. Want to minimise user knowledge gradients. Thanks for the suggestion.

[mrmoxon]

Addressed the setup-flow point: prime lab setup now creates an active default project named after the workspace folder when authenticated, with --project to bind an existing project, --project-name to override the generated name, and --no-project for local-only setup. Setup still succeeds without auth/API availability and prints the follow-up project command path.

[mrmoxon]

Rebased onto latest main and addressed the latest Bugbot cleanup by removing the redundant pre-create scope guard. The PR is mergeable again locally/GitHub-side; CI has restarted on head 38348e9. Local validation after the rebase: uv run pytest packages/prime/tests packages/prime-evals/tests -q => 804 passed, 5 skipped; focused project/setup/eval suite, ruff check/format, ty, and git diff --check also passed.

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit db89b50. Configure here.}

[cursor]

Env override skips scope checks

Medium Severity

When PRIME_PROJECT_ID is set, get_active_project_id returns it immediately and never applies the same base_url and team_id checks used for .prime/lab/context.json. Training and eval commands can therefore attach to a project outside the current CLI account/team context.

 

^{Reviewed by Cursor Bugbot for commit db89b50. Configure here.}

[cursor]

Project clear ignores env override

Medium Severity

prime project clear only deletes the workspace context.json file. If PRIME_PROJECT_ID is set in the environment, get_active_project_id still resolves an active project, so runs and evals keep attaching by default after a “clear”.

Additional Locations (1)

• packages/prime/src/prime_cli/utils/projects.py#L124-L127

 

^{Reviewed by Cursor Bugbot for commit db89b50. Configure here.}

[cursor]

Re-setup recreates default project

Medium Severity

After prime project clear, a later prime lab setup with no --project always calls project create with the workspace default name instead of re-binding the existing project. A duplicate-name API error is caught and setup finishes with no active project and no recovery hint.

 

^{Reviewed by Cursor Bugbot for commit db89b50. Configure here.}