Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 47 additions & 9 deletions internal/harness/compose.go
Original file line number Diff line number Diff line change
Expand Up @@ -421,12 +421,54 @@ func matchingAllowedPrefix(rawURL string, allowlist []string) string {
return MatchingAllowedPrefixInList(rawURL, allowlist)
}

// mergeValidationLoop merges two ValidationLoop pointers field by field.
// Child non-zero values win; base fills gaps. Returns the merged result.
// If both are nil, returns nil. If only one is non-nil, returns it as-is.
// If child is non-nil but all fields are zero-valued (e.g., from an explicit
// `validation_loop: {}` in YAML), it is returned as-is so that downstream
// validation (Validate's "script is required" check) can reject it. This
// preserves pre-field-merge behavior where an empty block was a load error
// rather than silent full inheritance.
func mergeValidationLoop(base, child *ValidationLoop) *ValidationLoop {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] — DiffHarness was not updated to match the new field-level merge, breaking the base/child round-trip used by migrate-customizations

ADR-0045's Consequences section states the base: merge rules and their inverse (DiffHarness) must change together ("Changes to merge rules must be reflected in both directions"), but this PR only updates the merge side. DiffHarness (internal/harness/diff.go:179-186) and diffForgeConfig (diff.go:470-473) still treat ValidationLoop as whole-struct: if child.ValidationLoop != nil && !reflect.DeepEqual(child.ValidationLoop, base.ValidationLoop) { result.Child.ValidationLoop = child.ValidationLoop }. When old whole-struct-replace merge was in effect, copying the entire child struct into the diff output was correct. With the new field-level mergeValidationLoop, it's not: if the child's ValidationLoop differs from base in just one field (say Script) while another field (say Schema) is genuinely empty/unset for that child, the whole child struct — including that genuine empty Schema — gets copied into the "minimal" diff. Re-composing that diffed-out child against the same base via mergeBaseIntoChild then backfills Schema from base, silently changing effective validation config.

This is reachable in production: internal/cli/migrate.go:479 calls harness.DiffHarness(&upstreamHarness, &customizedHarness, ...) as the core of the migrate-customizations command (ADR-0064), which converts standalone customized harnesses into base:-composed ones — exactly the diff → later-remerge path described above.

Suggestion: Diff ValidationLoop field-by-field in DiffHarness/diffForgeConfig (mirroring mergeValidationLoop's field set), consistent with how RunnerEnv/Env are already diffed field-by-field elsewhere in the same file. Add a round-trip test where base sets schema/feedback_mode and the differing child intentionally leaves them zero.

(Independently identified by 2 of 3 review agents, both verified by tracing the actual code paths.)

if base == nil && child == nil {
return nil
}
if base == nil {
return child
}
if child == nil {
return base
}

// Non-nil but all-zero child = explicit empty override (e.g., `validation_loop: {}`).
// Return as-is to preserve the pre-field-merge rejection behavior.
if *child == (ValidationLoop{}) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] — New in 36dcd1e1e: this all-zero-child special case (added to fix the validation_loop: {} finding) breaks forge-level inheritance, contradicting validateForge's own comment

This commit's two fixes don't compose. This special case (if *child == (ValidationLoop{}) { return child }) is correct for the base→child composition path it was written for, but mergeValidationLoop is shared by every caller, including mergeForgeConfig (forge.go) and mergeForgeConfigInto (compose.go), where a non-nil-but-all-zero forge-level block should mean "no forge-specific override, inherit everything" — matching how skills/runner_env already treat an empty forge override — not "explicit override, reject."

Concretely, this breaks the exact scenario validateForge's new topHasScript relaxation was written to support: a fully-populated top-level validation_loop plus forge.<platform>.validation_loop: {} (a literal empty mapping, which unmarshals to a non-nil pointer to an all-zero struct, not nil). validateForge lets this load, on the comment's own promise that "the script will be inherited via field-level merge in ResolveForge" — but ResolveForgemergeForgeConfig hits this special case and returns the empty struct as-is, discarding the top-level Script/Schema/MaxIterations/FeedbackMode entirely. Validate() then rejects the load with validation_loop.script is required when validation_loop is set — a confusing failure for a config validateForge itself just approved, on the very promise that just got broken. Confirmed end-to-end via LoadWithOpts; the same defect reproduces through mergeForgeConfigInto's base-chain forge composition. Neither new test (TestLoadWithOpts_ForgePartialValidationLoopInheritsScript, TestLoadWithOpts_ForgeEmptyValidationLoopNoTopLevelScript) covers a fully-empty forge block with a populated top level, so this shipped uncaught.

Suggestion: Move the all-zero-child "reject" special case out of the shared helper and into the mergeBaseIntoChild call site specifically, or add a parameter so mergeForgeConfig/mergeForgeConfigInto can opt out and fall through to normal field-by-field backfill for an all-zero child. Add end-to-end coverage for "top-level populated + forge validation_loop: {}" on both the ResolveForge and LoadWithBase paths.

(Independently confirmed — across the ResolveForge and base-chain forge-composition call sites — by all 4 review agents in this round.)

return child
}

merged := *child // start with child values
Comment thread
waynesun09 marked this conversation as resolved.
if merged.Script == "" {
merged.Script = base.Script
}
if merged.Schema == "" {
merged.Schema = base.Schema
}
if merged.MaxIterations == 0 {
merged.MaxIterations = base.MaxIterations
}
if merged.FeedbackMode == "" {
merged.FeedbackMode = base.FeedbackMode
}
return &merged
}

// mergeBaseIntoChild merges base harness fields into child harness.
// Child values override base values following ADR-0045 merge rules:
// - Scalars: child overrides if non-zero
// - Slices (skills, plugins, providers, api_servers): base + child (concatenated)
// - Maps (runner_env): base merged with child; child keys win
// - Pointer structs (validation_loop, security): child replaces if non-nil
// - Pointer structs (validation_loop): field-level merge; child non-zero fields win

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] premature-decision — ADR-0045 still documents whole-struct replacement for validation_loop; not updated alongside this behavior change

This comment (and its counterpart in forge.go's mergeForgeConfig doc) says the merge follows "ADR-0045 merge rules," but docs/ADRs/0045-forge-portable-harness-schema.md wasn't touched by this PR and still documents the old rule in three places: the forge inheritance table ("validation_loop | Forge value replaces top-level value entirely"), the base-composition merge-rules list ("validation_loop: child replaces base entirely (if non-nil)"), and the Open Questions section (still reasons about validation_loop on the premise "non-nil (including zero struct) = override entirely"). The ADR is the authoritative source these code comments explicitly cite; leaving it stale means the documented architecture no longer matches the code, and the ADR's own nil-vs-empty open question for validation_loop is left looking unresolved when this PR actually changes that behavior (see the other comment on this file re: validation_loop: {}).

Suggestion: Amend ADR-0045's inheritance table, base-merge-rules list, and open-questions note for validation_loop to describe the new field-level merge, in this PR or a fast-follow.

(Flagged independently by 2 of 3 review agents; both quoted the exact stale ADR passages.)

// - Pointer structs (security): child replaces if non-nil
// - host_files: concatenated with last-writer-wins dedup by Dest
// - forge: key-by-key merge; per-platform uses same rules
// - allowed_remote_resources: NOT merged (security; child must declare its own)
Expand Down Expand Up @@ -536,10 +578,8 @@ func mergeBaseIntoChild(base, child *Harness) {
child.Env.mergeEnvFrom(base.Env, false)
}

// Pointer structs: child replaces if non-nil
if child.ValidationLoop == nil {
child.ValidationLoop = base.ValidationLoop
}
// ValidationLoop: field-level merge; child non-zero fields win
child.ValidationLoop = mergeValidationLoop(base.ValidationLoop, child.ValidationLoop)
// Security: child inherits base's config if nil. Note that a base harness
// (even integrity-pinned) could set fail_mode: open. Child authors must
// explicitly set their own security block to prevent inheriting a weaker posture.
Expand Down Expand Up @@ -1272,10 +1312,8 @@ func mergeForgeConfigInto(base, child *ForgeConfig) {
child.Env.mergeEnvFrom(base.Env, false)
}

// ValidationLoop: child replaces if non-nil
if child.ValidationLoop == nil {
child.ValidationLoop = base.ValidationLoop
}
// ValidationLoop: field-level merge; child non-zero fields win
child.ValidationLoop = mergeValidationLoop(base.ValidationLoop, child.ValidationLoop)
}

// FetchAgentHarness fetches a URL-sourced agent harness using the same
Expand Down
202 changes: 200 additions & 2 deletions internal/harness/compose_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ host_files:
assert.Equal(t, "/dest3", h.HostFiles[2].Dest)
}

func TestLoadWithBase_LocalBase_ValidationLoopReplace(t *testing.T) {
func TestLoadWithBase_LocalBase_ValidationLoopAllFieldsOverride(t *testing.T) {
dir := t.TempDir()

writeTestHarness(t, dir, "base.yaml", `
Expand All @@ -319,12 +319,72 @@ validation_loop:
h, _, err := LoadWithBase(context.Background(), path, ComposeOpts{})
require.NoError(t, err)

// ValidationLoop: child replaces entirely
// ValidationLoop: child fields win when all set
require.NotNil(t, h.ValidationLoop)
assert.Equal(t, "child-script.sh", h.ValidationLoop.Script)
assert.Equal(t, 3, h.ValidationLoop.MaxIterations)
}

func TestLoadWithBase_LocalBase_ValidationLoopFieldLevelMerge(t *testing.T) {
dir := t.TempDir()

writeTestHarness(t, dir, "base.yaml", `
agent: agents/test.md
role: test
validation_loop:
script: base-script.sh
max_iterations: 5
feedback_mode: append
schema: base-schema.json
`)

path := writeTestHarness(t, dir, "child.yaml", `
base: base.yaml
validation_loop:
schema: child-schema.json
`)

h, _, err := LoadWithBase(context.Background(), path, ComposeOpts{})
require.NoError(t, err)

// ValidationLoop: child schema wins, base fills gaps
require.NotNil(t, h.ValidationLoop)
assert.Equal(t, "base-script.sh", h.ValidationLoop.Script)
assert.Equal(t, 5, h.ValidationLoop.MaxIterations)
assert.Equal(t, "append", h.ValidationLoop.FeedbackMode)
assert.Equal(t, "child-schema.json", h.ValidationLoop.Schema)
}

func TestLoadWithBase_LocalBase_ValidationLoopPartialScriptOverride(t *testing.T) {
dir := t.TempDir()

writeTestHarness(t, dir, "base.yaml", `
agent: agents/test.md
role: test
validation_loop:
script: base-script.sh
max_iterations: 5
feedback_mode: append
schema: base-schema.json
`)

path := writeTestHarness(t, dir, "child.yaml", `
base: base.yaml
validation_loop:
script: child-script.sh
`)

h, _, err := LoadWithBase(context.Background(), path, ComposeOpts{})
require.NoError(t, err)

// Child sets only script; base schema/iterations/feedback carry through
require.NotNil(t, h.ValidationLoop)
assert.Equal(t, "child-script.sh", h.ValidationLoop.Script)
assert.Equal(t, 5, h.ValidationLoop.MaxIterations)
assert.Equal(t, "append", h.ValidationLoop.FeedbackMode)
assert.Equal(t, "base-schema.json", h.ValidationLoop.Schema)
}

func TestLoadWithBase_LocalBase_ValidationLoopInherit(t *testing.T) {
dir := t.TempDir()

Expand All @@ -350,6 +410,120 @@ model: opus
assert.Equal(t, 5, h.ValidationLoop.MaxIterations)
}

func TestMergeValidationLoop(t *testing.T) {
tests := []struct {
name string
base *ValidationLoop
child *ValidationLoop
expected *ValidationLoop
}{
{
name: "both nil",
base: nil,
child: nil,
expected: nil,
},
{
name: "base nil",
base: nil,
child: &ValidationLoop{
Script: "child.sh",
MaxIterations: 3,
},
expected: &ValidationLoop{
Script: "child.sh",
MaxIterations: 3,
},
},
{
name: "child nil",
base: &ValidationLoop{
Script: "base.sh",
MaxIterations: 5,
},
child: nil,
expected: &ValidationLoop{
Script: "base.sh",
MaxIterations: 5,
},
},
{
name: "child partial override - schema only",
base: &ValidationLoop{
Script: "base.sh",
Schema: "base-schema.json",
MaxIterations: 5,
FeedbackMode: "append",
},
child: &ValidationLoop{
Schema: "child-schema.json",
},
expected: &ValidationLoop{
Script: "base.sh",
Schema: "child-schema.json",
MaxIterations: 5,
FeedbackMode: "append",
},
},
{
name: "child full override",
base: &ValidationLoop{
Script: "base.sh",
Schema: "base-schema.json",
MaxIterations: 5,
FeedbackMode: "append",
},
child: &ValidationLoop{
Script: "child.sh",
Schema: "child-schema.json",
MaxIterations: 3,
FeedbackMode: "replace",
},
expected: &ValidationLoop{
Script: "child.sh",
Schema: "child-schema.json",
MaxIterations: 3,
FeedbackMode: "replace",
},
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := mergeValidationLoop(tt.base, tt.child)
if tt.expected == nil {
assert.Nil(t, result)
} else {
require.NotNil(t, result)
assert.Equal(t, tt.expected.Script, result.Script)
assert.Equal(t, tt.expected.Schema, result.Schema)
assert.Equal(t, tt.expected.MaxIterations, result.MaxIterations)
assert.Equal(t, tt.expected.FeedbackMode, result.FeedbackMode)
}
})
}
}

func TestMergeValidationLoop_AllZeroChild(t *testing.T) {
// An explicit `validation_loop: {}` should NOT inherit from base.
// The all-zero child is returned as-is so Validate() can reject it
// (script is required), matching pre-field-merge behavior.
base := &ValidationLoop{
Script: "base.sh",
Schema: "base-schema.json",
MaxIterations: 5,
FeedbackMode: "append",
}
child := &ValidationLoop{} // all zero — explicit empty override

result := mergeValidationLoop(base, child)
require.NotNil(t, result)
assert.Equal(t, "", result.Script, "all-zero child should not inherit base script")
assert.Equal(t, "", result.Schema, "all-zero child should not inherit base schema")
assert.Equal(t, 0, result.MaxIterations, "all-zero child should not inherit base max_iterations")
assert.Equal(t, "", result.FeedbackMode, "all-zero child should not inherit base feedback_mode")
}

func TestLoadWithBase_ChainedBases(t *testing.T) {
dir := t.TempDir()

Expand Down Expand Up @@ -1038,6 +1212,30 @@ func TestMergeForgeConfigInto_ValidationLoop(t *testing.T) {
assert.Equal(t, 5, child.ValidationLoop.MaxIterations)
}

func TestMergeForgeConfigInto_ValidationLoopFieldLevelMerge(t *testing.T) {
base := &ForgeConfig{
ValidationLoop: &ValidationLoop{
Script: "base-validate.sh",
MaxIterations: 5,
FeedbackMode: "append",
Schema: "base-schema.json",
},
}
child := &ForgeConfig{
ValidationLoop: &ValidationLoop{
Schema: "child-schema.json",
},
}

mergeForgeConfigInto(base, child)

require.NotNil(t, child.ValidationLoop)
assert.Equal(t, "base-validate.sh", child.ValidationLoop.Script)
assert.Equal(t, 5, child.ValidationLoop.MaxIterations)
assert.Equal(t, "append", child.ValidationLoop.FeedbackMode)
assert.Equal(t, "child-schema.json", child.ValidationLoop.Schema)
}

func TestLoadWithBase_InvalidForgeAfterMerge(t *testing.T) {
dir := t.TempDir()

Expand Down
52 changes: 47 additions & 5 deletions internal/harness/diff.go
Original file line number Diff line number Diff line change
Expand Up @@ -175,9 +175,9 @@ func DiffHarness(base, child *Harness, customizedFiles map[string]bool) *DiffRes
hasAny = true
}

// Pointer structs — keep if non-nil and different; abort if child removes.
if child.ValidationLoop != nil && !reflect.DeepEqual(child.ValidationLoop, base.ValidationLoop) {
result.Child.ValidationLoop = child.ValidationLoop
// ValidationLoop — field-level diff to match mergeValidationLoop semantics.
if vlDiff := diffValidationLoop(base.ValidationLoop, child.ValidationLoop); vlDiff != nil {
result.Child.ValidationLoop = vlDiff
hasAny = true
} else if child.ValidationLoop == nil && base.ValidationLoop != nil {
result.Warnings = append(result.Warnings, "validation_loop: child removes block from base; cannot express with base: composition")
Expand Down Expand Up @@ -467,8 +467,8 @@ func diffForgeConfig(base, child *ForgeConfig, platform string) (*ForgeConfig, [
fc.Skills = extras
hasAny = true
}
if child.ValidationLoop != nil && !reflect.DeepEqual(child.ValidationLoop, base.ValidationLoop) {
fc.ValidationLoop = child.ValidationLoop
if vlDiff := diffValidationLoop(base.ValidationLoop, child.ValidationLoop); vlDiff != nil {
fc.ValidationLoop = vlDiff
hasAny = true
}
if d, mapRemoved := diffStringMap(base.RunnerEnv, child.RunnerEnv); len(d) > 0 || mapRemoved {
Expand All @@ -491,3 +491,45 @@ func diffForgeConfig(base, child *ForgeConfig, platform string) (*ForgeConfig, [
}
return fc, warnings
}

// diffValidationLoop returns a ValidationLoop containing only the fields
// where child differs from base. Returns nil when both are nil, both are
// equal, or child is nil (removal is handled by the caller). This mirrors
// mergeValidationLoop's field set so diff→merge round-trips correctly for
// non-zero field values.
func diffValidationLoop(base, child *ValidationLoop) *ValidationLoop {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] — Follow-up on the round-trip finding above (still open): the fix narrows but does not close the gap

36dcd1e1e added diffValidationLoop and switched DiffHarness/diffForgeConfig to field-level diff, matching the merge side's granularity. That closes the specific whole-struct-vs-field-level mismatch, but the underlying round-trip problem is only partially fixed.

diffValidationLoop and mergeValidationLoop both use the Go zero value as the only signal for "not set, inherit from base." They can't distinguish "this field equals base's value, omit it from the diff" from "this field's real value is the zero value, and it differs from a non-zero base value." Concretely: base = {Script:"base.sh", Schema:"base-schema.json", MaxIterations:5, FeedbackMode:"append"}; a customized child overrides Script and intentionally carries no schema: child = {Script:"custom.sh", Schema:"", MaxIterations:5, FeedbackMode:"append"}. diffValidationLoop(base, child) produces {Script:"custom.sh", Schema:"", MaxIterations:0, FeedbackMode:""}. Feeding that back through mergeValidationLoop(base, diff) yields Schema:"base-schema.json" — silently reintroducing schema validation the customized harness never wanted. Still reachable via migrate-customizations (internal/cli/migrate.go's DiffHarness call). The new round-trip tests (TestDiffHarness_ValidationLoopFieldLevelRoundTrip, TestDiffForgeConfig_ValidationLoopFieldLevel) only exercise children whose non-overridden fields already equal base's, so they pass without covering this.

Suggestion: A zero-value-comparison diff/merge can't express "explicitly zero, differs from non-zero base." Track field presence explicitly (pointer fields, or a presence map/bitmask alongside ValidationLoop) instead of inferring it from the value, or explicitly document this as an accepted migrate-customizations limitation and add a test that locks in the (documented) behavior rather than leaving it to silently regress.

(Independently confirmed — with an identical empirical repro — by all 4 review agents in this round.)

if child == nil {
return nil
}
if base == nil {
return child
}
if reflect.DeepEqual(base, child) {
return nil
}

vl := &ValidationLoop{}
hasAny := false

if child.Script != base.Script {
vl.Script = child.Script
hasAny = true
}
if child.Schema != base.Schema {
vl.Schema = child.Schema
hasAny = true
}
if child.MaxIterations != base.MaxIterations {
vl.MaxIterations = child.MaxIterations
hasAny = true
}
if child.FeedbackMode != base.FeedbackMode {
vl.FeedbackMode = child.FeedbackMode
hasAny = true
}

if !hasAny {
return nil
}
return vl
}
Loading
Loading