Skip to content

fix(egosuite): refuse invalid compare summaries - #272

Merged
kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
Iamnotphage:fix/egosuite-compare-input-errors
Aug 30, 2026
Merged

kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
Iamnotphage:fix/egosuite-compare-input-errors

Conversation

@Iamnotphage

Copy link
Copy Markdown
Contributor

Closes #266.

Summary

  • report missing and malformed comparison summaries as command-line input errors
  • centralize the existing FileNotFoundError, RuntimeError, and ValueError boundary around subcommand dispatch
  • add outcome-focused CLI coverage for both refusal paths and the successful comparison table

Why

compare previously allowed filesystem and JSON decoder failures to escape as Python tracebacks, returning exit code 1 even though no comparison had taken place. Both conditions now exit 2 because the input could not be used, matching the convention documented in docs/FORMAT.md.

The error boundary now wraps the command dispatch itself instead of being copied into a third match arm. This preserves the existing behavior of labels and run while making the same protection the default for future subcommands. Successful comparisons retain their existing computation and output unchanged.

The related #260 was assigned but did not yet have a PR when this change was prepared, so there was no submitted implementation to align with.

Validation

  • uv sync --locked --all-extras — passed
  • targeted regression test before the implementation — 2 failed, 1 passed
  • targeted regression test after the implementation — 3 passed
  • uv run ruff check --fix — passed
  • uv run ruff format — passed
  • uv run ty check — passed
  • uv run pytest -q — 1,195 passed, 6 skipped
  • uv run --locked --project examples/egosuite_evaluation pytest -q examples/egosuite_evaluation/tests — 11 passed
  • git diff --check — passed

Checklist

  • I added outcome-focused tests for the changed behavior.
  • Documentation remains current; this change brings compare into the existing documented exit-code convention.
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran both the root and EgoSuite-specific pytest suites.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • Stored-data compatibility is unchanged; no transform behavior version bump is required.

@kstonekuan kstonekuan left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Hoisting the handler around the match covers the next subcommand too.

Validated on 69de54a: root suite 1,195 passed, 6 skipped; example suite 11 passed; ruff, format, ty clean on both projects. Moved compare back outside the try and confirmed exactly your two refusal tests go red, other nine stay green.

One ask, same as always and not a rule: leave the good first issue pool for newcomers where you can. You've got three merges now, and the starter queue is what keeps new people arriving. Higher-leverage work if you want it: run HFlow against a real corpus, Egocentric-10K or Egocentric-100K on HF, and report what breaks, what's slow, what's awkward. See #287.

@kstonekuan
kstonekuan merged commit 4f05492 into Hebbian-Robotics:main Aug 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EgoSuite evaluation: compare crashes with a traceback on a missing or malformed summary while labels and run refuse cleanly

2 participants