Skip to content

feat: integrate main with execution disabled - #179

Draft
evalstate wants to merge 44 commits into
mainfrom
feat/leaderboard-submissions
Draft

evalstate wants to merge 44 commits into
mainfrom
feat/leaderboard-submissions

Conversation

@evalstate

@evalstate evalstate commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Draft visibility PR; execution is intentionally disabled and this is not production-ready.

  • Merge current main (9fb49cb) into the feature branch without rewriting history.
  • Reject Run submissions, direct JobConfig submissions, setup tests, and Run actions before admission, persistence, reservations, or execution credential resolution. Direct service callers also fail closed; runtime uses only read-only Jobs access and never starts the reconciler.
  • Remove control-credential forwarding, the parent worker, private Hub/Harbor Sandbox patches, interrupted-trial deletion, and the worker publication workflow. Retain a CLI-only image scaffold, not an executable runner. HF_TOKEN stays in the control Space; neither persistent secret is forwarded.
  • Workbench configures/saves/loads immutable named native JobConfig fragments. New Run selects reviewed or saved harness configuration, benchmark, model string and cost guidance for local preview only. No preparation Job or task resolution occurs.
  • Preserve installer browser verification and sanitized diagnostics, reconcile native readiness/system responses, and retain first-install build isolation. No old five-profile engine, parallel reconciler, runtime migration, or new credential store.

The old publication-engine-dependent result-submission UI is not carried forward. Its JobConfig-era submission/review flow remains to be reconciled. The Workbench editor now exposes native JSON configuration rather than retaining the old parallel recipe library. Historical Bucket records are not migrated or modified.

Harbor ownership review

Checked Harbor revision dcd0a7ac74b7bd417780d9cb27cd819c7ec82e4e; fresh upstream main had no later commits:

  • src/harbor/cli/agents.py: public harbor agents list --json and harbor agents schema NAME --json own discovery/options.
  • src/harbor/cli/jobs.py and src/harbor/models/job/config.py: native JobConfig, install-only and CLI configuration.
  • src/harbor/job.py: Harbor owns task resolution, locks, retry and resume. Removed the local execution wrapper instead of duplicating these behaviors.
  • src/harbor/environments/hf_sandbox.py: current Sandbox.create integration lacks the needed public ownership-label/namespace boundary. Its authentication requirement does not authorize forwarding the control credential.

A future public Harbor CLI/platform integration and isolated credential boundary are prerequisites to execution. No local Harbor patch, upstream issue, or upstream PR is included. Per-user OAuth scopes, billing attribution and Jobs/inference delegation are deferred research; no forwarding or OAuth implementation is added.

Verification

Passed:

  • npm formatting, lint, strict types, unit tests (398), build, generated contracts, dependency tree and audit.
  • Browser tests: 7 console plus 1 mocked installer test, including saved-config authoring and disabled execution.
  • Root Ruff/format, ty, pytest (46, 87.98% coverage) and pip-audit.
  • Agent-package Ruff/format, ty and pytest (32).
  • Production control Docker build and CLI-only scaffold build, both linux/amd64.
  • Slophammer DRY and structure check without baseline; public privacy and diff checks.

Remaining red checks (not waived):

  • TypeScript coverage: lines 72.83%, statements 70.22%, functions 68.59%, branches 63.14%; each threshold remains 85%.
  • slophammer-py check . --baseline: required baseline file is absent.
  • scripts/check_mutation.py --min-kill-rate 90: script is absent following upstream mutation-tooling retirement; it was not restored.

No HF operational API call, deployment, Job, inference, resource creation, credential transfer, reset, migration, Run action, or result publication was performed. Feature branch only; do not merge this draft into main.

@evalstate evalstate changed the title Feat/leaderboard submissions feat: integrate main with execution disabled Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant