Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,9 @@ prime env push my-environment
Prime Lab connects verifiers environments to evaluations, GEPA prompt optimization, and Hosted Training. Start with `prime lab setup` to create a local workspace with starter configs, then use `prime train models` to choose a Hosted Training model with current capacity and pricing.

```bash
# Set up a Lab workspace
# Set up a Lab workspace.
# If authenticated, setup creates an active project named after this folder.
prime lab setup

# Create a Lab project and make it active in this workspace
prime project create "Alphabet Sort Baselines"
prime project current

# List trainable models, capacity, and token pricing
Expand All @@ -165,9 +163,11 @@ prime train models
# Generate a Hosted Training config
prime train init

# Launch the run from the generated config
# Launch the run from the generated config.
# Runs attach to the active project by default.
prime train rl.toml
prime train rl.toml --project <project-id>
prime train rl.toml --no-project

# Inspect and manage Hosted Training runs
prime train list
Expand All @@ -176,10 +176,14 @@ prime train metrics <run-id>
prime train checkpoints <run-id>
```

Lab projects group related training runs, evaluations, and adapters. Use
`prime project use <project-id>` to switch the active workspace project, or
`prime project clear` to clear it. Existing runs and adapters support project
add/remove/clear; evaluations support assign/clear.
Lab projects group related training runs, evaluations, and adapters. By default,
`prime lab setup` creates an active project named after the workspace folder.
Use `prime lab setup --project <project-id>` to bind an existing project,
`prime lab setup --project-name "Alphabet Sort Baselines"` to choose the default
project name, or `prime lab setup --no-project` to keep setup local-only. Later,
use `prime project use <project-id>` to switch the active workspace project, or
`prime project clear` to stop using one by default. Existing runs and adapters
support project add/remove/clear; evaluations support assign/clear.

```bash
# Manage projects
Expand Down Expand Up @@ -237,6 +241,7 @@ prime eval push
# Push specific eval directory (verifiers format)
prime eval push outputs/evals/gsm8k--gpt-4/abc123
prime eval push outputs/evals/gsm8k--gpt-4/abc123 --project <project-id>
prime eval push outputs/evals/gsm8k--gpt-4/abc123 --no-project

# Push a public evaluation (default is private)
prime eval push --public
Expand Down
22 changes: 13 additions & 9 deletions packages/prime/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,11 +114,9 @@ prime sandbox create python:3.11
Prime Lab connects verifiers environments to evaluations, GEPA prompt optimization, and Hosted Training. Start with `prime lab setup` to create a local workspace with starter configs, then use `prime train models` to choose a Hosted Training model with current capacity and pricing.

```bash
# Set up a Lab workspace
# Set up a Lab workspace.
# If authenticated, setup creates an active project named after this folder.
prime lab setup

# Create a Lab project and make it active in this workspace
prime project create "Alphabet Sort Baselines"
prime project current

# List trainable models, capacity, and token pricing
Expand All @@ -127,9 +125,11 @@ prime train models
# Generate a Hosted Training config
prime train init

# Launch the run from the generated config
# Launch the run from the generated config.
# Runs attach to the active project by default.
prime train rl.toml
prime train rl.toml --project <project-id>
prime train rl.toml --no-project

# Inspect and manage Hosted Training runs
prime train list
Expand All @@ -138,10 +138,14 @@ prime train metrics <run-id>
prime train checkpoints <run-id>
```

Lab projects group related training runs, evaluations, and adapters. Use
`prime project use <project-id>` to switch the active workspace project, or
`prime project clear` to clear it. Existing runs and adapters support project
add/remove/clear; evaluations support assign/clear.
Lab projects group related training runs, evaluations, and adapters. By default,
`prime lab setup` creates an active project named after the workspace folder.
Use `prime lab setup --project <project-id>` to bind an existing project,
`prime lab setup --project-name "Alphabet Sort Baselines"` to choose the default
project name, or `prime lab setup --no-project` to keep setup local-only. Later,
use `prime project use <project-id>` to switch the active workspace project, or
`prime project clear` to stop using one by default. Existing runs and adapters
support project add/remove/clear; evaluations support assign/clear.

```bash
# Manage projects
Expand Down
20 changes: 14 additions & 6 deletions packages/prime/src/prime_cli/commands/evals.py
Original file line number Diff line number Diff line change
Expand Up @@ -1223,12 +1223,12 @@ def push_eval(
project: Optional[str] = typer.Option(
None,
"--project",
help="Project ID or slug to attach to this evaluation.",
help="Project ID or slug. Defaults to the active project for this workspace.",
),
no_project: bool = typer.Option(
False,
"--no-project",
help="Do not attach this evaluation to a project.",
help="Do not attach this evaluation to the active project.",
),
) -> None:
"""Push evaluation data to Prime Evals.
Expand Down Expand Up @@ -1260,7 +1260,11 @@ def push_eval(
console.print(" prime eval push outputs/evals/env--model/run-id --eval-id <eval-id>")
raise typer.Exit(1)

project_id = resolve_project_id(project, no_project=no_project)
project_id = resolve_project_id(
project,
no_project=no_project,
use_active_project=eval_id is None,
)
clear_project = bool(eval_id and no_project)

if config_path is None:
Expand Down Expand Up @@ -1510,12 +1514,12 @@ def run_eval_cmd(
project: Optional[str] = typer.Option(
None,
"--project",
help="Project ID or slug to attach to this evaluation.",
help="Project ID or slug. Defaults to the active project for this workspace.",
),
no_project: bool = typer.Option(
False,
"--no-project",
help="Do not attach this evaluation to a project.",
help="Do not attach this evaluation to the active project.",
),
) -> None:
"""Run an evaluation with local-first environment resolution."""
Expand All @@ -1537,7 +1541,11 @@ def run_eval_cmd(

env_dir_path: Optional[str] = None
try:
project_id = resolve_project_id(project, no_project=no_project)
project_id = resolve_project_id(
project,
no_project=no_project,
use_active_project=True,
)
except APIError as exc:
console.print(f"[red]Error:[/red] {exc}")
raise typer.Exit(1) from exc
Expand Down
6 changes: 3 additions & 3 deletions packages/prime/src/prime_cli/commands/projects.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ def _usage_help(*examples: tuple[str, str], json_help_text: Optional[str] = None
),
(
"prime project current",
"Show the active project for this workspace.",
"Show the active project that new Lab runs and evals will use.",
),
(
"prime project use <project-id>",
Expand Down Expand Up @@ -137,7 +137,7 @@ def _usage_help(*examples: tuple[str, str], json_help_text: Optional[str] = None
),
(
"prime project current",
"Confirm which project is active for this workspace.",
"Confirm which project this workspace will use by default.",
),
json_help_text=PROJECT_JSON_HELP,
)
Expand Down Expand Up @@ -173,7 +173,7 @@ def _usage_help(*examples: tuple[str, str], json_help_text: Optional[str] = None
PROJECT_CLEAR_HELP = _usage_help(
(
"prime project clear",
"Clear this workspace's active project.",
"Stop attaching new Lab runs and evals to a workspace project by default.",
),
)

Expand Down
5 changes: 3 additions & 2 deletions packages/prime/src/prime_cli/commands/rl.py
Original file line number Diff line number Diff line change
Expand Up @@ -906,12 +906,12 @@ def create_run(
project: Optional[str] = typer.Option(
None,
"--project",
help="Project ID or slug to attach to this run.",
help="Project ID or slug. Defaults to the active project for this workspace.",
),
no_project: bool = typer.Option(
False,
"--no-project",
help="Do not attach this run to a project.",
help="Do not attach this run to the active project.",
),
yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"),
) -> None:
Expand Down Expand Up @@ -970,6 +970,7 @@ def warn(msg: str) -> None:
project_id = resolve_project_id(
project,
no_project=no_project,
use_active_project=True,
config=app_config,
client=api_client,
)
Expand Down
Loading