Skip to content

fix(cli): report unknown dataset names without a traceback - #119

Merged
knisar merged 1 commit into
wandb:mainfrom
bercedev:fix/cli-unknown-dataset-name
Oct 23, 2026
Merged

knisar merged 1 commit into
wandb:mainfrom
bercedev:fix/cli-unknown-dataset-name

Conversation

@bercedev

Copy link
Copy Markdown
Contributor

Closes #118

Approach. A small _unknown_dataset_error(slug) helper in cli.py calls ExampleRegistry.get and returns the registry's own message when the lookup raises KeyError, or None when the name is registered. datasets show and datasets infer-policy-expectations check it first. For an unknown name they print error: <message> to stderr, the same form the CLI already uses for other input errors, and return 2. Only the lookup is guarded: infer-policy-expectations then calls ExampleRegistry.load as before, so a KeyError raised by a registered dataset's loader still surfaces as itself. The registry, its suggestions and the successful paths are unchanged.

$ rai datasets show bbqq
error: Example dataset 'bbqq' not registered. Did you mean: bbq? Available: …
$ echo $?
2

Tests (tests/test_cli.py, all offline):

  • both commands with bbqq: exit 2, empty stdout, the message and the bbq suggestion on stderr, no traceback;
  • an unknown name with no suggestion (zzzzzzzzzz): exit 2, no "Did you mean";
  • rai datasets show bbq: exit 0, JSON with "slug": "bbq";
  • rai datasets infer-policy-expectations bbq --limit 0: exit 0, [];
  • a registered dataset whose ExampleRegistry.load raises KeyError("question") (monkeypatched): the KeyError propagates and isn't reported as an unknown name.

The first three fail on main and the rest pass on both.

Local checks (Python 3.12, offline in a container):

  • python -m pytest -q tests/test_cli.py: 12 passed.
  • python -m pytest -q: 704 passed, 16 skipped, 5 failed, 13 errors. The failures and errors are in tests/test_reproducible_evaluation.py and need git, which my container doesn't have. The list is identical on main.
  • python -m pip check: no broken requirements.
  • python -m ruff check --select E9,F63,F7,F82 .: all checks passed.
  • reuse --no-multiprocessing lint: compliant.
  • git diff --check origin/main...HEAD: clean.

Note: I used Claude Code while working on this.

datasets show and datasets infer-policy-expectations now print the registry's message (with its suggestions) to stderr and exit 2 for an unregistered name. Only the lookup is guarded, so a KeyError from a registered dataset's loader still surfaces.
@knisar knisar added the status: in review Pull request is ready for maintainer review label Oct 23, 2026

@knisar knisar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independently verified 95ab08a against main 20756a2 in a fresh Python 3.13.1 environment.

  • python -m pytest -q: 722 passed, 16 skipped. The reported environment-related failures do not reproduce here.
  • pip check, Ruff (E9,F63,F7,F82), REUSE and committed-patch whitespace checks pass.
  • Ten additional checks pass, covering the real CLI entry point for both commands with and without suggestions, valid names, actual registered-loader errors including a misleading error message, and preservation of output files on an unknown name.
  • All 18 current-head CI jobs pass.

The lookup guard is narrow, existing successful command behavior is preserved, and unrelated loader errors still propagate. No blocking findings.

@knisar
knisar merged commit 7dd0911 into wandb:main Oct 23, 2026
20 checks passed
@knisar knisar removed the status: in review Pull request is ready for maintainer review label Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(cli): report unknown dataset names without a traceback

2 participants