[matouskozak]

Keep Test as a historical recheck and warn that its data may overlap with training. During training, report held-out accuracy and log loss plus thresholds, precision, coverage, and accepted counts for 60%, 70%, 80%, 90%, and 95% precision targets.

[Copilot]

🟢 Approval recommended

Static review identified no blocking issues; builds and tests were not executed.

No open issues 🎉

What changed in this PR

Adds saved held-out evaluation to the training and testing tools, enabling offline threshold replay while preserving historical testing.

Changes:

• Saves label-free scores with model and dataset provenance.
• Adds validated offline evaluation and separate score caches.
• Adds regression tests and documents usage and limitations.

File	Description
train/​action.yml	Saves and verifies separate evaluation caches.
test/​action.yml	Adds held-out evaluation mode.
README.md	Documents offline evaluation and limitations.
IssueLabeler/​tests/​Evaluation.Tests/​TrainingEvaluationTests.cs	Tests training integration and offline CLI behavior.
IssueLabeler/​tests/​Evaluation.Tests/​ThresholdEvaluationTests.cs	Tests artifact validation and threshold metrics.
IssueLabeler/​tests/​Evaluation.Tests/​Evaluation.Tests.csproj	Adds the evaluation test project.
IssueLabeler/​src/​Trainer/​Trainer.cs	Saves held-out evidence after training.
IssueLabeler/​src/​Trainer/​HeldOutEvaluation.cs	Collects label-free scores from reloaded models.
IssueLabeler/​src/​Tester/​Tester.cs	Reports offline metrics and distinguishes historical testing.
IssueLabeler/​src/​Tester/​Args.cs	Validates offline evaluation arguments.
IssueLabeler/​src/​Common/​ThresholdEvaluation.cs	Computes threshold and per-label metrics.
IssueLabeler/​src/​Common/​EvaluationArtifact.cs	Stores and validates scores and provenance.
IssueLabeler/​src/​Common/​ArgUtils.cs	Adds reusable argument-parsing overloads.
IssueLabeler/​IssueLabeler.sln	Registers evaluation tests.
IssueLabeler/​Directory.Packages.props	Adjusts test SDK version handling for MSTest SDK.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Concurrent training can publish mismatched model and score caches while reporting success.

1 open issue

• Concurrent cache writes can publish mismatched models and scores · New

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Failed retraining can overwrite an existing model’s held-out dataset, leaving a mismatched pair that invalidates subsequent evaluation.

1 open issue

• Preserve original test data until retraining succeeds · New

1 resolved since last review

• Concurrent cache writes can publish mismatched models and scores

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Failed retraining can leave an existing model paired with a replacement held-out dataset, compromising evaluation correctness.

1 open issue

• Preserve original test data until retraining succeeds

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The workflow’s legacy action pins produce caches incompatible with its new manifest-aware restores.

2 open issues

• Update pinned actions to cache TSV and split manifests · New
• Preserve original test data until retraining succeeds

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The breaking dataset/model compatibility contract requires human validation of coordinated action rollout and dataset/model regeneration.

1 open issue

• Update pinned actions to cache TSV and split manifests

1 resolved since last review

• Preserve original test data until retraining succeeds

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The metrics report incorrectly claims to include all held-out items despite excluding unseen labels.

2 open issues

• Training jobs pin outdated action, ignoring threshold inputs
• Update pinned actions to cache TSV and split manifests

1 previously missed issue

In code that hasn't changed since last review

Clarify metrics exclude holdout-only labels and align report assertions

IssueLabeler/​src/​Trainer/​Trainer.cs:120

A label that occurs only in the holdout is mapped to key 0 because MapValueToKey is fitted on split.TrainSet. ML.NET's evaluator skips these unknown-label rows, so these metrics can report perfect accuracy even when predictions for unseen labels are wrong. Clarify that these metrics exclude labels absent from training, and change the MicroAccuracy description to refer to evaluated items. The precision-target table includes those rows, so distinguish the two populations and update the report-text assertions in TrainerTests.cs.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

Static review identified no blocking issues in evaluation, reporting, or test integration; builds and tests were not run.

2 open issues

• Training jobs pin outdated action, ignoring threshold inputs
• Update pinned actions to cache TSV and split manifests

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The changes match the requested behavior, with relevant tests and no identified blocking issues.

2 open issues

• Training jobs pin outdated action, ignoring threshold inputs
• Update pinned actions to cache TSV and split manifests

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No blocking issues remain from source review; builds and tests were not run.

No open issues 🎉

2 resolved since last review

• Training jobs pin outdated action, ignoring threshold inputs
• Update pinned actions to cache TSV and split manifests

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No blocking issues were identified in evaluation calculations, reporting, warnings, or test integration.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

Static review found no blocking issues in evaluation, reporting, or test integration; tests were not executed.

No open issues 🎉

_{Review effort: Balanced}