Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions apps/control-web/e2e/control.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2964,3 +2964,26 @@ test("operator reviews exact infrastructure failures, creates related run and in
page.getByRole("heading", { name: "Original execution", exact: true }),
).toBeVisible();
});

test("parent failure pause explains explicit resume without starting work", async ({
page,
}) => {
const actions: string[] = [];
await mockControl(page, { onAction: (action) => actions.push(action) });
await page.route(`**/api/v1/runs/${runId}`, (route) =>
json(route, {
...run,
status: "paused",
state: {
...state,
desired_state: "paused",
actor: "harbor-hf-parent-failed",
},
}),
);
await page.goto(`/runs/${runId}`);
await expect(
page.getByText("Paused after a parent Job failed.", { exact: false }),
).toBeVisible();
expect(actions).toEqual([]);
});
2 changes: 2 additions & 0 deletions apps/control-web/src/generated/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2770,6 +2770,8 @@ export interface components {
/** @enum {unknown} */
desired_state: "run" | "paused" | "cancelled";
actor: string;
/** @description Owned parent error IDs explicitly acknowledged by operator resume; not Job status or retry state. */
acknowledged_parent_failures?: string[];
parent_jobs: {
id: string;
/** Format: date-time */
Expand Down
7 changes: 7 additions & 0 deletions apps/control-web/src/pages.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -837,6 +837,13 @@ export function RunPage() {
/>
<h2 className="mt-6 font-semibold text-white">Original execution</h2>
<RunSummaryCards run={item} />
{item.state.desired_state === "paused" &&
item.state.actor === "harbor-hf-parent-failed" && (
<p role="status" className="mt-4 text-amber-200">
Paused after a parent Job failed. Review its diagnostics before explicitly
resuming. Resume acknowledges observed parent errors; it does not fix them.
</p>
)}
<RunWaffle run={item} />
<RunPricingCorrections key={`pricing-${item.record.run_id}`} run={item} />
<PricingPanel result={item.result} />
Expand Down
14 changes: 12 additions & 2 deletions docs/2026-09-04-simplification-implementation-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -400,8 +400,9 @@ For each run it applies these rules in order:
4. If Harbor's job result is finished, do not start a parent.
5. If one labeled parent is live, adopt it if needed and wait.
6. Cancel orphaned labeled child Jobs.
7. If capacity is available and the fixed restart delay has passed, start one
parent and append it to `state.json`.
7. If an owned parent has an unacknowledged error, persist desired pause.
Otherwise, if capacity is available and the fixed restart delay has passed,
start one parent and append it to `state.json`.

Stopping parents before children reduces the interval in which a child shutdown
can become an error while its parent still observes it. If Harbor still returns
Expand Down Expand Up @@ -619,3 +620,12 @@ replacement evidence cannot both count. Preserve failed replacement outcomes
and all incurred attempt costs. No replacement launch is implicit in a project
release. Detailed endpoints, validation and limitations are specified in the
[replacement contract](2026-09-11-replacement-backend.md).

## Parent error containment

Explicit resume records observed HF parent error acknowledgments in optional
`state.json.acknowledged_parent_failures`; this is an operator decision, not
mirrored Job status or native retry state. New errors pause again. No schema
version, native field, result writer or projection table is added. See the
[control policy](CONTROL_SERVICE.md#parent-error-containment) and
[ownership and compatibility evidence](replacement-evidence-parent-containment.md).
21 changes: 20 additions & 1 deletion docs/CONTROL_SERVICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ The service reads these Space variables:
| `HARBOR_HF_PARENT_TIMEOUT_SECONDS` | no | `86400` | parent Job timeout |
| `HARBOR_HF_MAX_ACTIVE_JOBS` | no | `16` | live parent Job limit |
| `HARBOR_HF_RECONCILE_INTERVAL_MS` | no | `15000` | reconcile interval |
| `HARBOR_HF_PARENT_RESTART_DELAY_MS` | no | `60000` | failed parent restart delay |
| `HARBOR_HF_PARENT_RESTART_DELAY_MS` | no | `60000` | eligible parent restart delay |
| `HARBOR_HF_PROJECTION_PATH` | no | `/tmp/harbor-hf/control.sqlite` | SQLite projection |
| `HARBOR_HF_AUTH_PATH` | no | `/tmp/harbor-hf/auth.sqlite` | OAuth session store |
| `HARBOR_HF_WEB_ROOT` | no | `./apps/control-web/dist` | built web application |
Expand Down Expand Up @@ -706,3 +706,22 @@ configuration and the exact existing owned scope, then saves explicit approval
through the existing registry API without launching anything. See
[Run-recorded inference access review](run-inference-access-review.md) for the
server-only input, review fencing and policy boundaries.

## Parent error containment

After existing native completion, cost-stop, desired-state, live-parent and orphan
cleanup checks, an owned HF parent `error` pauses execution instead of repeatedly
starting parents. This includes startup failures, but does not claim to classify
errors as deterministic or transient. Logs and native trial exceptions are not
control inputs. Non-error terminal parents retain the existing restart delay.

Explicit operator resume acknowledges only currently observed owned parent errors
in optional `state.json.acknowledged_parent_failures`, a set of Job IDs recording
that operator decision, not mirrored Job status or Harbor retry state. An absent
field means no acknowledgment. New errors require a new review; old acknowledged
errors do not prevent resume. Intent and reconciliation remain under the same run
lock. Failed/unknown recorded-parent inspection prevents acknowledgment or launch.
The parent never writes this field. No error message, retry counter, completion
state, additional resource or projection table is added. Disabled-write runtime
still does not start the reconciler or permit resume. Deployment never resumes a
paused run. See [review evidence](replacement-evidence-parent-containment.md).
15 changes: 13 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,8 @@ For each run it:
finished Harbor job unchanged;
5. adopts an existing live parent;
6. cancels live child Jobs that have no live parent; and
7. starts a new parent after the restart delay when capacity is available.
7. pauses on an unacknowledged HF parent error; otherwise starts a new parent
after the restart delay when capacity is available.

The parent-first stop reduces the child-shutdown race. If Harbor still reports
an in-flight trial as terminal during a controlled stop, the parent preserves
Expand Down Expand Up @@ -311,7 +312,8 @@ per-trial copies; upload failures propagate explicitly. A hard kill can lose
unsaved outputs and cost evidence and cannot guarantee a final copy. Local disk
must fit the working data.

A failed parent can restart after the fixed delay. A cancelled run cannot
An unacknowledged parent error pauses the run for explicit operator review.
A non-error terminal parent can restart after the fixed delay. A cancelled run cannot
resume. A projection rebuild failure, immutable run conflict, unlabeled child,
or Job cancellation failure requires operator review rather than a second
control path.
Expand Down Expand Up @@ -381,3 +383,12 @@ selection excludes constituent subsets and fails closed on incomplete or invalid
assemblies. Native metrics that cannot unambiguously populate its scalar field
remain available in the full combined view rather than being reduced locally.
See [the replacement contract](2026-09-11-replacement-backend.md).

## Parent error containment

Explicit resume records observed HF parent error acknowledgments in optional
`state.json.acknowledged_parent_failures`; this is an operator decision, not
mirrored Job status or native retry state. New errors pause again. No schema
version, native field, result writer or projection table is added. See the
[control policy](CONTROL_SERVICE.md#parent-error-containment) and
[ownership and compatibility evidence](replacement-evidence-parent-containment.md).
10 changes: 10 additions & 0 deletions docs/control-api-v1.openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -1111,6 +1111,16 @@
"minLength": 1,
"maxLength": 320
},
"acknowledged_parent_failures": {
"description": "Owned parent error IDs explicitly acknowledged by operator resume; not Job status or retry state.",
"type": "array",
"uniqueItems": true,
"items": {
"type": "string",
"minLength": 1,
"maxLength": 160
}
},
"parent_jobs": {
"type": "array",
"items": {
Expand Down
151 changes: 151 additions & 0 deletions docs/replacement-evidence-parent-containment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
---
title: Replacement evidence and parent error containment
author: Harbor-HF maintainers
date: 2026-09-11
tags: [harbor, replacements, lifecycle]
---

# Replacement evidence and parent error containment

## Ownership and reviewed source

Harbor revision `dcd0a7ac74b7bd417780d9cb27cd819c7ec82e4e` remains
unchanged. Inspected native `src/harbor/job.py`, `job_plan.py`,
`models/job/config.py`, `models/job/lock.py`, `models/job/result.py`,
`models/trial/result.py`, `models/verifier/result.py`, and
`environments/hf_sandbox.py`, and their pinned history. Relevant history includes
native agent preflight/config validation and the HF Sandbox provider addition
and command-execution correction. The installed native files match the pin.

`Job.create()` performs native preflight and constructs execution;
`JobPlan.from_resolved`, `build_trial_configs`, and `aggregate` own native
planning and results. Native `RetryConfig` controls trial retries, not an external
HF parent Job that fails before `Job.create()`. No native API owns Harbor-HF's
operator-reviewed cross-run fingerprint or external parent restart admission.
`VerifierResult.rewards` is explicitly `float | int`: Pydantic preserves either
input type. Neither native models nor the inspected planning API supply the
transport-equivalence rule required by the already approved replacement bridge.

## Numeric evidence equivalence

JavaScript JSON parsing/stringifying turns integral floats such as `1.0` into
integers. The old review path therefore hashed different native union-field types
than the parent's direct source reader. Both could derive exactly the same child
configuration while the immutable source check rejected it.

Normalize exact integral floats once, at `Evidence.parse`, **before** native model
validation. Harbor then restores its declared types, including explicit float
fields. Keep the existing canonical JSON encoder, hash structure, native field
names and `sha256:` source prefix. The separate public review fingerprint remains
bare hexadecimal and still binds selection and budget.

Do not normalize the final native hash payload: that also changes explicit native
float fields and invalidates existing review hashes. Offline replay against the
saved source confirmed that input normalization matches the existing immutable
review fingerprint and child configuration without rewriting any source bytes,
run record or stored hash. The real parent source reader and preflight succeed.
This is evidence for that reviewed source, not a guarantee for every possible
historical noncanonical number in arbitrary metadata; mismatches still fail
closed. There is no dual hash, fallback reader or evidence migration.

The rule traverses nested JSON maps and arrays. It preserves booleans and
fractional floats, including exponent notation; negative floating zero becomes
integer zero. Arbitrary integers never pass through floating point. Non-finite
values are rejected at the evidence boundary; the existing `allow_nan=False`
encoder remains a second guard. This does not
repair JavaScript precision loss for integers beyond its safe range: changed
integer evidence must still mismatch the parent's exact source, never round into
an accepted identity.

The cross-language regression now reads raw fixture files through the actual TS
`ReplacementEvidence` loader and `NativeLaunch` subprocess transport. The native
launch entry point consumes actual stdin, rather than returning a response
computed before Node serialized the request. Only external admission and task
downloads are stubbed; native validation, fingerprinting, immutable TS storage,
and the independent parent SDK reader/preflight remain real and offline. The
existing native planner/admission tests remain in place.

The bridge's removal condition is unchanged: replace it at the first reviewed
Harbor revision providing equivalent cross-run provenance and coverage APIs. No
future upstream SHA or general native replacement implementation is invented.

## Parent lifecycle policy

HF's typed `ERROR` stage cannot certify whether a failure is deterministic or
transient. The controller therefore pauses on **any unacknowledged owned parent
error**, rather than parsing arbitrary logs or repeatedly spending on startup.
This deliberately replaces automatic retry of failed HF parents; non-error
terminal Jobs retain the fixed restart delay. Native Harbor trial retries are
unchanged. Failed trial evidence in a normally completed parent is not a parent
error and does not trigger this policy.

Existing desired pause/cancel, native completion, cost-stop finalization,
live-parent adoption and orphan cleanup precede the new no-parent launch veto.
Repeated observations persist only one pause. A live parent is not cancelled
because another historical parent errored. Unknown recorded-parent liveness
continues to block launch. Unrelated runs retain their normal safety handling.

Explicit resume acknowledges observed owned parent error IDs while holding the
same run lock used by reconciliation. All recorded parents are inspected during
resume, including when another parent is active. A newly failing parent is never
implicitly acknowledged. Acknowledgments survive projection rebuild; neither
provider timestamps nor SQLite receipt time determine whether an operator has
reviewed an error. Operator review is required even after deployment fixes the
underlying defect, and inference/budget checks still apply on the next start.

## Schema and API comparison

The sole new optional durable value is
`RunState.acknowledged_parent_failures`: unique HF parent Job IDs explicitly
acknowledged by operator resume. It records a Harbor-HF operator decision, not
provider status, a native retry count, or Harbor progress. Existing parent IDs
alone cannot represent that decision; `updated_at` also changes on adoption, and
`actor` must not be overloaded as a hidden restart state machine. The optional
field's absence means no errors were acknowledged, not a legacy reader.

The versioned JSON Schema remains authoritative, with generated TypeScript and
OpenAPI. Existing resume authorization, CSRF/write-mode gates, control intent,
three-table projection and parent ownership are retained. No new API endpoint,
resource, credential, native configuration field, result writer, or scheduler is
added. The parent reads existing desired intent but never writes controller state.
The UI uses the existing paused state and control actor to explain the stop; the
actor is explanatory only, never a launch or resume authorization input.

## Follow-up validation gate closure

The initial 501-test agent run passed tests but measured 76.87% whole-package
coverage; the prescribed Slophammer baseline was also missing. Those historical
failures remain recorded in the project authorization history, not waived.

Offline test-only follow-up adds 97 cases for parent ownership rejection,
immutable receipt replay/conflicts, atomic-write cleanup, nested controlled stops,
DSH session parsing and native trajectory metrics, sandbox upload/install
boundaries, and FX protocol/usage validation. External execution is mocked;
parsers, native models, receipt filesystem operations and validation paths execute
real product code. No production behavior, coverage omission, exclusion, threshold,
provider admission rule or resource changes accompany these tests.

Fresh whole-agent coverage is **87.33%** (598 tests; 2,304 statements, 292 missed).
This is the package's statement-coverage measurement, not a branch-coverage claim.
Root coverage remains **89.10%** with 102 tests. Root and agent Ruff, format and ty
checks pass, as does the root dependency audit. Node formatting, lint, types,
1,575 unit tests, build, generated checks and dependency audit pass; all 76 browser
tests pass. Both Dockerfiles build locally for `linux/amd64` without publication.
Existing native numeric transport and parent/controller regression tests remain in
the passing suites. No live private replay or remote integration was repeated.

The installed Slophammer CLI documents `check . --baseline-write` as its snapshot
command. Its native baseline writer/reader were inspected. An ordinary check
first reported zero findings across all 27 production files; the official writer
then generated `slophammer-baseline.json` with version 1 and an empty findings
array. It contains no paths or suppressed debt. Regeneration is byte-identical.
`uv run slophammer-py check . --baseline` passes with zero baselined and zero new
findings, and `uv run slophammer-py dry .` reports zero candidates. Do not grow the
baseline to hide new findings.

## Release boundary

This is local implementation and offline validation only. Independent review must
precede publication. Matched source/image deployment to existing resources is a
separate release phase. No run launch, resume, inference grant, credential change,
paid canary or resource creation accompanies this change.
6 changes: 6 additions & 0 deletions packages/contracts/schemas/run-state-v1.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@
"updated_at": { "type": "string", "format": "date-time" },
"desired_state": { "enum": ["run", "paused", "cancelled"] },
"actor": { "type": "string", "minLength": 1, "maxLength": 320 },
"acknowledged_parent_failures": {
"description": "Owned parent error IDs explicitly acknowledged by operator resume; not Job status or retry state.",
"type": "array",
"uniqueItems": true,
"items": { "type": "string", "minLength": 1, "maxLength": 160 }
},
"parent_jobs": {
"type": "array",
"items": {
Expand Down
4 changes: 4 additions & 0 deletions packages/contracts/src/generated/run-state-v1.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ revision: number
updated_at: string
desired_state: ("run" | "paused" | "cancelled")
actor: string
/**
* Owned parent error IDs explicitly acknowledged by operator resume; not Job status or retry state.
*/
acknowledged_parent_failures?: string[]
parent_jobs: {
id: string
started_at: string
Expand Down
Loading