[webreidi]

Summary

This was pulled out of the C# Refactoring skill as a separate skill to work on an evaluate.

Related issue

Validation

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior.
• I updated CODEOWNERS when adding or moving owned content.
• I updated all marketplace manifests when plugin metadata changed.
• I updated eng/known-domains.txt for any new external domains referenced by skill content.

[github-actions]

This PR has been automatically marked as stale because it has no activity for 30 days. It will be closed if no further activity occurs within another 7 days of this comment. If it is closed, you may reopen it anytime when you're ready again.

[AbhitejJohn]

Current-head migration findings: top-level-config (1).

The repository has moved to Vally 0.18 in #1309. This upgrade includes several breaking evaluation-schema changes.

I checked the existing open PRs against the new schema. This PR adds or changes an eval.yaml file that still uses syntax removed or changed by Vally 0.18. Now that #1309 has merged, the eval will not load or will not preserve its intended behavior until these fields are migrated.

I added inline suggestions for the affected field types. Where a field repeats, apply the same mapping to the remaining occurrences. These are schema-only changes and should preserve the existing prompts, graders, activation contracts, tool restrictions, timeouts, and shard assignments. After applying them, please update this branch from main and rerun the repository eval-quality and Vally corpus checks.

The migration mappings, runtime evidence, and validation results are documented in #1309.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet	setup-local-sdk	4/8	50%

Uncovered: dotnet/setup-local-sdk

• [Pitfall] dotnet app.dll wrong runtime (line 386)
• [CodePattern] [pscustomobject] (line 301)
• [CodePattern] [ordered] (line 301)
• [CodePattern] [guid] (line 104)

[Copilot]

🟡 Changes recommended

Evaluation inputs expose expected answers, and the graders do not reliably enforce compatibility, requested behavior, or routing and no-op preservation.

6 open issues

• Add no-op case to prevent unnecessary rewrites · New
• Exclude eval.yaml from staged agent workspace · New
• Add grader-owned checks for the shipped API contract · New
• Add dormancy test for genuinely excluded fixtures · New
• Add executable coverage for hexadecimal parsing · New
• Test multi-target runtime behavior on both frameworks · New

What changed in this PR

Adds a standalone .NET breaking-changes skill to the dotnet plugin, separating compatibility guidance for independent evaluation.

Changes:

• Adds guidance for public APIs, multi-targeting, generated code, and friend assemblies.
• Adds six evaluation scenarios backed by a Billing library and test fixture.
• Registers the skill in the plugin README and assigns ownership.

File	Description
tests/​dotnet/​dotnet-breaking-changes/​tests/​Billing.Tests/​BillingTests.cs	Adds baseline behavior tests.
tests/​dotnet/​dotnet-breaking-changes/​tests/​Billing.Tests/​Billing.Tests.csproj	Configures the net10.0 test project.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​PublicAPI.Shipped.txt	Records shipped parsing APIs.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​Pricing.cs	Supplies pricing and legacy-wrapper fixtures.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​PlatformInfo.cs	Supplies framework-conditional behavior.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​OrderProcessor.cs	Supplies the local-rename fixture.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​Coupons.g.cs.template	Provides generated partial-type input.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​Coupons.cs	Provides the hand-authored partial type.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​Billing.csproj	Configures multi-targeting and code generation.
tests/​dotnet/​dotnet-breaking-changes/​src/​Billing/​AppSettingsHelper.cs	Supplies duplicated public parsing helpers.
tests/​dotnet/​dotnet-breaking-changes/​Fixture.sln	Connects the library and tests.
tests/​dotnet/​dotnet-breaking-changes/​eval.yaml	Defines compatibility and restraint scenarios.
plugins/​dotnet/​skills/​dotnet-breaking-changes/​SKILL.md	Defines routing and compatibility checks.
plugins/​dotnet/​skills/​dotnet-breaking-changes/​references/​source-generation.md	Explains generated-code editing constraints.
plugins/​dotnet/​skills/​dotnet-breaking-changes/​references/​public-api.md	Explains API validation and compatibility techniques.
plugins/​dotnet/​skills/​dotnet-breaking-changes/​references/​multi-targeting.md	Covers conditional branches and target validation.
plugins/​dotnet/​skills/​dotnet-breaking-changes/​references/​internals-visible-to.md	Covers friend-assembly compatibility.
plugins/​dotnet/​README.md	Lists the new skill.
.github/​CODEOWNERS	Assigns skill and evaluation ownership.

_{Review effort: Balanced}

[webreidi]

/evaluate

[Copilot]

🟡 Changes recommended

Suite-only validation bypasses, incomplete deterministic grading, and an inaccurate golden reference undermine the evaluation’s reliability.

7 open issues

• Suite-only workspaces bypass stimulus patch requirements · New
• Suite-level fixture files bypass patch applicability validation · New
• Signature checks allow incompatible Redeem replacement · New
• Tests do not verify complete unchanged workspace state · New
• Tests do not verify provider-sensitive overload behavior · New
• Tests do not require Describe to provide meaningful labels · New
• Golden answer incorrectly claims Trim changes bool parsing · New

6 resolved since last review

• Add dormancy test for genuinely excluded fixtures
• Add grader-owned checks for the shipped API contract
• Exclude eval.yaml from staged agent workspace
• Add no-op case to prevent unnecessary rewrites
• Test multi-target runtime behavior on both frameworks
• Add executable coverage for hexadecimal parsing

_{Review effort: Balanced}

[github-actions]

👋 @webreidi — this PR has 7 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[Copilot]

🟡 Changes recommended

Required graders reject correct results because of an omitted inherited SDK file and unsupported raw interpolated-string scanning.

3 open issues

• A migrated consumer using $"""{PricingMath.CalculateDiscount(100m, 0.10m)}""" is rejected by this… · New
• The private-app stimulus retains the suite's global.json, but this manifest omits it, so… · New
• The mutation matches every type under src/, so it also rewrites an unrelated helper's instance…

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The preservation grader can accept added compiled source under skipped TestResults directories, weakening the evaluation’s no-edit guarantees.

1 open issue

• Adding src/Billing/TestResults/Extra.cs passes the unchanged-workspace grader because every… · New

3 resolved since last review

• The private-app stimulus retains the suite's global.json, but this manifest omits it, so…
• A migrated consumer using $"""{PricingMath.CalculateDiscount(100m, 0.10m)}""" is rejected by this…
• The mutation matches every type under src/, so it also rewrites an unrelated helper's instance…

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The shared SDK initialization change and custom mutation graders warrant human validation under normal evaluation concurrency.

1 open issue

• This adds a mandatory .NET preflight that can terminate the job before any evaluation starts, but… · New

1 resolved since last review

• Adding src/Billing/TestResults/Extra.cs passes the unchanged-workspace grader because every…

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Custom mutation graders and shared SDK initialization need human confirmation of end-to-end evaluation behavior under normal worker concurrency.

1 open issue

• This adds a mandatory .NET preflight that can terminate the job before any evaluation starts, but…

_{Review effort: Balanced}

[github-actions]

👋 @webreidi — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[Copilot]

🔵 Needs a closer look

Shared evaluation validation, custom mutation grading, and SDK initialization need human confirmation of end-to-end behavior under normal evaluation concurrency.

1 open issue

• This adds a mandatory .NET preflight that can terminate the job before any evaluation starts, but…

_{Review effort: Balanced}

[webreidi]

/evaluate

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 2 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit d349c05463539938df16dcf714f4dc1a10992941; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
dotnet-breaking-changes	claude-sonnet-5	➖ Improvement signal, unproven	n=11; 4W/6T/1L; d=5; p=0.188; net +27.3%; 2 dormancy excluded	🟡 0.25	Activation: isolated 11/11; plugin 9/11	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
dotnet-breaking-changes	gpt-5.6-luna	➖ No preference	n=11; 0W/11T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded	—	Activation: isolated 11/11; plugin 6/11	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Improvement signal, unproven — dotnet-breaking-changes (claude-sonnet-5)

Why: Net win +27.3% (4W/6T/1L over 11 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +21.5% across 13 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 4W/6T/1L; d=5; p=0.188; net +27.3%; 2 dormancy excluded

Warnings: Activation: isolated 11/11; plugin 9/11

Overfit: Moderate (score 0.25)

Repeated-run reliability (not used by the gate): 13 paired runs (5W/7T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a new public method that must be correct on every target framework	Eligible	+0.0%	+0.0%	0/1/0
= Add behavior to a shipped public member without breaking the contract	Eligible	+0.0%	+0.0%	0/1/0
= Fold a shipped duplicate type into its twin without removing it	Eligible	+0.0%	+0.0%	0/1/0
= Keep a shipped parameter name stable when asked to rename it	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-shared parsing alone when asked to consolidate it	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve a shipped nullability contract	Eligible	-100.0%	-100.0%	0/0/1
= Rename a public method in a private single-target app without compatibility ceremony	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a new public method that must be correct on every target framework: The available evidence supports that both agents completed all requested implementation, API-surface, compatibility, and cross-target testing requirements. B's final report gives a little more build-analyzer detail, but that is not a material difference in the delivered result.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ No preference — dotnet-breaking-changes (gpt-5.6-luna)

Why: Net win +0.0% (0W/11T/0L over 11 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 13 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 0W/11T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded

Warnings: Activation: isolated 11/11; plugin 6/11

Repeated-run reliability (not used by the gate): 13 paired runs (0W/13T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a new public method that must be correct on every target framework	Eligible	+0.0%	+0.0%	0/1/0
= Add behavior to a shipped public member without breaking the contract	Eligible	+0.0%	+0.0%	0/1/0
= Extend a shipped method with an extra parameter without a binary break	Eligible	+0.0%	+0.0%	0/1/0
= Fold a shipped duplicate type into its twin without removing it	Eligible	+0.0%	+0.0%	0/1/0
= Keep a local-variable rename proportionate inside a shipped library	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Keep a shipped parameter name stable when asked to rename it	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-shared parsing alone when asked to consolidate it	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped nullability contract	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member of a partial type that a generated part references	Eligible	+0.0%	+0.0%	0/1/0
= Rename a public method in a private single-target app without compatibility ceremony	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Rename a vague public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Review a green pull request for consumer-breaking changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a new public method that must be correct on every target framework: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1191 in dotnet/skills, download eval artifacts with gh run download 40416817331 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/d349c05463539938df16dcf714f4dc1a10992941/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

✅ Evaluation passed for d349c05. cc @webreidi @AbhitejJohn @JanKrivanek — please review.

[Copilot]

🔵 Needs a closer look

Shared runner and grader changes need end-to-end human validation, and a confirmed grader defect remains unresolved.

1 open issue

• This mandatory grader rejects a valid consolidation when both public types delegate to internal… · New

1 resolved since last review

• This adds a mandatory .NET preflight that can terminate the job before any evaluation starts, but…

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The remaining grader defect needs correction, and the custom graders and shared SDK initialization warrant end-to-end validation before approval.

1 open issue

• Consolidate partial type declarations before tracing members · New

1 resolved since last review

• This mandatory grader rejects a valid consolidation when both public types delegate to internal…

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The custom C# mutation graders and shared runner initialization need human confirmation through production evaluation and CI runs.

No open issues 🎉

1 resolved since last review

• Consolidate partial type declarations before tracing members

_{Review effort: Balanced}

[webreidi]

/evaluate

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 1 improved, ➖ 1 result without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 1c3b6bf4fc42fcbd5f0afd0ffa4fb634f7d83146; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
dotnet-breaking-changes	claude-sonnet-5	✅ Improved	n=11; 7W/3T/1L; d=8; p=0.035; net +54.5%; 2 dormancy excluded	✅ 0.17	Activation: isolated 11/11; plugin 7/11	Fix activation gaps.
dotnet-breaking-changes	gpt-5.6-luna	➖ No preference	n=11; 0W/11T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded	—	Activation: isolated 10/11; plugin 6/11	Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ No preference — dotnet-breaking-changes (gpt-5.6-luna)

Why: Net win +0.0% (0W/11T/0L over 11 preference-eligible stimulus vote(s), sign test p=1.000), mean preference +0.0% across 13 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tie rationales and arm outputs; replace inert scenarios rather than adding repeated runs.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 0W/11T/0L; d=0; p=1.000; net +0.0%; 2 dormancy excluded

Warnings: Activation: isolated 10/11; plugin 6/11

Repeated-run reliability (not used by the gate): 13 paired runs (0W/13T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a new public method that must be correct on every target framework	Eligible	+0.0%	+0.0%	0/1/0
= Add behavior to a shipped public member without breaking the contract	Eligible	+0.0%	+0.0%	0/1/0
= Extend a shipped method with an extra parameter without a binary break	Eligible	+0.0%	+0.0%	0/1/0
= Fold a shipped duplicate type into its twin without removing it	Eligible	+0.0%	+0.0%	0/1/0
= Keep a local-variable rename proportionate inside a shipped library	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Keep a shipped parameter name stable when asked to rename it	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-shared parsing alone when asked to consolidate it	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped nullability contract	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member of a partial type that a generated part references	Eligible	+0.0%	+0.0%	0/1/0
= Rename a public method in a private single-target app without compatibility ceremony	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Rename a vague public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Review a green pull request for consumer-breaking changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a new public method that must be correct on every target framework: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — dotnet-breaking-changes (claude-sonnet-5)

Why: Net win +54.5% (7W/3T/1L over 11 preference-eligible stimulus vote(s), sign test p=0.035), mean preference +41.5% across 13 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Fix activation gaps.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=11; 7W/3T/1L; d=8; p=0.035; net +54.5%; 2 dormancy excluded

Warnings: Activation: isolated 11/11; plugin 7/11

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 13 paired runs (7W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a new public method that must be correct on every target framework	Eligible	+0.0%	+0.0%	0/1/0
= Extend a shipped method with an extra parameter without a binary break	Eligible	+0.0%	+0.0%	0/1/0
= Keep a local-variable rename proportionate inside a shipped library	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Leave already-shared parsing alone when asked to consolidate it	Eligible	-100.0%	-100.0%	0/0/1
= Preserve a shipped nullability contract	Eligible	+0.0%	+0.0%	0/1/0
= Rename a public method in a private single-target app without compatibility ceremony	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▲ Rename a vague public method without breaking existing callers	Eligible	+100.0%	+100.0%	1/0/0
▲ Rename an internal member used by a friend assembly	Eligible	+100.0%	+100.0%	1/0/0

Illustrative judge evidence:

• Add a new public method that must be correct on every target framework: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1191 in dotnet/skills, download eval artifacts with gh run download 40536251126 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/1c3b6bf4fc42fcbd5f0afd0ffa4fb634f7d83146/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[Copilot]

🔵 Needs a closer look

Shared evaluation infrastructure changes and custom C# mutation graders warrant maintainer confirmation of end-to-end CI results before merge.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Shared CI initialization and custom mutation graders require human confirmation of end-to-end evaluation behavior.

No open issues 🎉

_{Review effort: Balanced}

[webreidi]

/evaluate

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 2 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit c38effcd11284ba7657427e23029291a646790cb; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
dotnet-breaking-changes	claude-sonnet-5	➖ Improvement signal, unproven	n=11; 6W/4T/1L; d=7; p=0.063; net +45.5%; 2 dormancy excluded	🟡 0.24	Activation: isolated 9/11; plugin 8/11	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
dotnet-breaking-changes	gpt-5.6-luna	➖ Mixed evidence	n=11; 1W/9T/1L; d=2; p=0.750; net +0.0%; 2 dormancy excluded	—	Activation: isolated 9/11; plugin 6/11	Compare winning and losing scenarios to isolate where the target helps versus hurts.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Improvement signal, unproven — dotnet-breaking-changes (claude-sonnet-5)

Why: Net win +45.5% (6W/4T/1L over 11 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +38.5% across 13 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 6W/4T/1L; d=7; p=0.063; net +45.5%; 2 dormancy excluded

Warnings: Activation: isolated 9/11; plugin 8/11

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 13 paired runs (6W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a new public method that must be correct on every target framework	Eligible	+0.0%	+0.0%	0/1/0
= Keep a local-variable rename proportionate inside a shipped library	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Leave already-shared parsing alone when asked to consolidate it	Eligible	-100.0%	-100.0%	0/0/1
= Rename a member of a partial type that a generated part references	Eligible	+0.0%	+0.0%	0/1/0
= Rename a public method in a private single-target app without compatibility ceremony	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Rename an internal member used by a friend assembly	Eligible	+0.0%	+0.0%	0/1/0
= Review a green pull request for consumer-breaking changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Add a new public method that must be correct on every target framework: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Mixed evidence — dotnet-breaking-changes (gpt-5.6-luna)

Why: Net win +0.0% (1W/9T/1L over 11 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 13 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 1W/9T/1L; d=2; p=0.750; net +0.0%; 2 dormancy excluded

Warnings: Activation: isolated 9/11; plugin 6/11

Repeated-run reliability (not used by the gate): 13 paired runs (1W/11T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a new public method that must be correct on every target framework	Eligible	+0.0%	+0.0%	0/1/0
= Add behavior to a shipped public member without breaking the contract	Eligible	+0.0%	+0.0%	0/1/0
= Extend a shipped method with an extra parameter without a binary break	Eligible	+0.0%	+0.0%	0/1/0
= Fold a shipped duplicate type into its twin without removing it	Eligible	+0.0%	+0.0%	0/1/0
= Keep a local-variable rename proportionate inside a shipped library	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Keep a shipped parameter name stable when asked to rename it	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-shared parsing alone when asked to consolidate it	Eligible	+0.0%	+0.0%	0/1/0
= Preserve a shipped nullability contract	Eligible	+0.0%	+0.0%	0/1/0
= Rename a member of a partial type that a generated part references	Eligible	+0.0%	+0.0%	0/1/0
= Rename a public method in a private single-target app without compatibility ceremony	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Rename a vague public method without breaking existing callers	Eligible	+0.0%	+0.0%	0/1/0
▼ Rename an internal member used by a friend assembly	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Add a new public method that must be correct on every target framework: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1191 in dotnet/skills, download eval artifacts with gh run download 40549192176 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/c38effcd11284ba7657427e23029291a646790cb/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[Copilot]

🔵 Needs a closer look

The custom mutation harness and shared evaluation-pipeline changes need maintainer confirmation with production CI evidence.

No open issues 🎉

_{Review effort: Balanced}

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate f74427215e48f95ff309f1986ca3699dd4960cfe to retry this exact commit.

2 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.