Repository navigation
fix(cli): report unknown dataset names without a traceback - #119
Merged
Merged
Conversation
datasets show and datasets infer-policy-expectations now print the registry's message (with its suggestions) to stderr and exit 2 for an unregistered name. Only the lookup is guarded, so a KeyError from a registered dataset's loader still surfaces.
knisar
approved these changes
Oct 23, 2026
knisar
left a comment
Collaborator
There was a problem hiding this comment.
Independently verified 95ab08a against main 20756a2 in a fresh Python 3.13.1 environment.
python -m pytest -q: 722 passed, 16 skipped. The reported environment-related failures do not reproduce here.pip check, Ruff (E9,F63,F7,F82), REUSE and committed-patch whitespace checks pass.- Ten additional checks pass, covering the real CLI entry point for both commands with and without suggestions, valid names, actual registered-loader errors including a misleading error message, and preservation of output files on an unknown name.
- All 18 current-head CI jobs pass.
The lookup guard is narrow, existing successful command behavior is preserved, and unrelated loader errors still propagate. No blocking findings.
This was referenced Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #118
Approach. A small
_unknown_dataset_error(slug)helper incli.pycallsExampleRegistry.getand returns the registry's own message when the lookup raisesKeyError, orNonewhen the name is registered.datasets showanddatasets infer-policy-expectationscheck it first. For an unknown name they printerror: <message>to stderr, the same form the CLI already uses for other input errors, and return 2. Only the lookup is guarded:infer-policy-expectationsthen callsExampleRegistry.loadas before, so aKeyErrorraised by a registered dataset's loader still surfaces as itself. The registry, its suggestions and the successful paths are unchanged.Tests (
tests/test_cli.py, all offline):bbqq: exit 2, empty stdout, the message and thebbqsuggestion on stderr, no traceback;zzzzzzzzzz): exit 2, no "Did you mean";rai datasets show bbq: exit 0, JSON with"slug": "bbq";rai datasets infer-policy-expectations bbq --limit 0: exit 0,[];ExampleRegistry.loadraisesKeyError("question")(monkeypatched): theKeyErrorpropagates and isn't reported as an unknown name.The first three fail on
mainand the rest pass on both.Local checks (Python 3.12, offline in a container):
python -m pytest -q tests/test_cli.py: 12 passed.python -m pytest -q: 704 passed, 16 skipped, 5 failed, 13 errors. The failures and errors are intests/test_reproducible_evaluation.pyand needgit, which my container doesn't have. The list is identical onmain.python -m pip check: no broken requirements.python -m ruff check --select E9,F63,F7,F82 .: all checks passed.reuse --no-multiprocessing lint: compliant.git diff --check origin/main...HEAD: clean.Note: I used Claude Code while working on this.