Skip to content

fix(control): check native completion before restarting a parent - #240

Merged
osolmaz merged 1 commit into
mainfrom
fix/guard-finished-parent-restart
Oct 8, 2026
Merged

osolmaz merged 1 commit into
mainfrom
fix/guard-finished-parent-restart

Conversation

@osolmaz

@osolmaz osolmaz commented Sep 26, 2026

Copy link
Copy Markdown
Member

Summary

A parent Job can restart after Harbor has finished if the control service reads an older Bucket listing. This change checks the native result directly just before starting a parent. It avoids an unnecessary restart without changing how Harbor runs or resumes trials.

Related issue: #239.

What Changed

The control service now checks job/result.json with a fresh read after it decides that no parent is live, and before it calls the HF Jobs start API. A native finished_at stops the launch. A missing or unfinished result keeps the existing resume path.

  • Added an optional fresh-read argument to the shared JSON reader, using the ObjectStore's existing cache-bypass option.
  • Added regression tests for an older Bucket listing and an older cached aggregate result.
  • Checked the pinned Harbor source at dcd0a7ac74b7bd417780d9cb27cd819c7ec82e4e: src/harbor/job.py (Job.run()) and src/harbor/models/job/result.py (JobResult.finished_at). Harbor still writes and owns the result and trial resume. No new record, schema, API value, UI value, or trial scheduler was added.

Testing

The stale-listing test failed before the fix and passed after it. The local unit and browser tests passed with en_US.UTF-8 for the existing USD formatting assertions.

  • npm test -- --reporter=dot: 1,719 passed.
  • npm run test:e2e: 77 passed on an isolated local port.
  • npm run typecheck, npm run build, npm run format:check, npm run lint, npm run check:generated, npm audit --audit-level=low, and python3 scripts/check_public_privacy.py .: passed.
  • No paid inference run or hosted deployment was used to test this change.

Risks

Each pending parent launch makes one extra fresh Bucket read. A missing result still allows launch; other read errors stop that reconciliation pass rather than starting a parent on uncertain evidence. This closes the observed stale-listing window, but a result that arrives after the final fresh read can still race with a launch. The hosted control service has not been changed by this PR.

Co-Authored-By: GPT-6 Sol <noreply@pi.dev>
Generated-By: pi 0.87.1 (https://pi.dev)
@osolmaz
osolmaz merged commit 152e4f6 into main Oct 8, 2026
1 check passed
@osolmaz
osolmaz deleted the fix/guard-finished-parent-restart branch October 8, 2026 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant