Skip to content

fix: handle invalid compare inputs - #290

Merged
kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
raghulajithn:fix/compare-invalid-input
Aug 30, 2026
Merged

kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
raghulajithn:fix/compare-invalid-input

Conversation

@raghulajithn

Copy link
Copy Markdown
Contributor

Summary

Fixes the compare command so missing or malformed summary files are reported as CLI errors instead of escaping as unhandled tracebacks.

Why

The compare command previously allowed errors from missing or malformed summary files to escape as unhandled tracebacks.

The exception handling is intentionally placed around the match statement rather than inside the compare arm. This keeps the error handling at the CLI boundary and ensures that the same handling applies consistently to all current subcommands and future subcommands without duplicating the error handling logic.

Validation

Ran all of the following validations successfully:

uv sync --locked --all-extras

Resolved 124 packages in 1ms
Uninstalled 61 packages in 142ms

uv run ruff check --fix

All checks passed!

uv run ruff format

195 files left unchanged

uv run ty check

All checks passed!

uv run pytest -q

1244 passed, 6 skipped in 88.32s (0:01:28)

uv run --locked --project examples/build_ai_evaluation pytest -q examples/build_ai_evaluation/tests

23 passed in 4.70s

Closes #260

Checklist

  • I added or updated outcome-focused tests for changed business logic.
  • I updated documentation for changed behavior, flags, formats, or requirements.
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran the relevant pytest suite.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • I preserved stored-data compatibility or documented an explicit version change.

@github-actions

Copy link
Copy Markdown

👋 Hi @raghulajithn — thank you so much for your first contribution to HFlow!

A maintainer will review your pull request as soon as possible. In the meantime:

💡 Tip: one open pull request per contributor at a time. Issues with an assignee are taken; everything else is fair game.

We are excited to have you here and appreciate your help making the project better! 🙌

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Validated on 8840f0d: gate clean, root suite 1,244 passed 6 skipped, example suite 23 passed. Observed:

compare /tmp/nope.json -> exit=2, 2 lines, "No such file or directory: /tmp/nope.json"
compare /tmp/bad.json  -> exit=2, 2 lines, "invalid JSON in /tmp/bad.json: Expecting value..."

Wrapping the JSONDecodeError earns its keep even though it is already a ValueError and would have been caught anyway: the raw one does not name the file, so a multi-file compare would tell you something was malformed without saying which.

Moved compare back outside the shared try and exactly your two refusal tests go red, other 21 stay green.

Merging.

@kstonekuan
kstonekuan merged commit 249eb20 into Hebbian-Robotics:main Aug 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build AI evaluation: compare crashes with a traceback on a missing file while run refuses cleanly

2 participants