Fast local checks for Carpentries Workbench
lessons. In under a second, wbcheck finds broken front matter, missing
::: blocks, heading problems, broken links and images, and leftover
scaffold text. Run it before you push instead of waiting on the sandpaper CI
build. An optional AI review adds writing and pedagogy findings, each one
quoting the lesson text it's about.
curl -fsSL https://raw.githubusercontent.com/ucla-imls-open-sci/carpentries-workbench-checker/main/install.sh | sh
wbcheck check path/to/lesson # what's wrong
wbcheck fix path/to/lesson # fix it in your editor- Install
- Everyday use
- Checking
- Fixing
- The AI review
- Browsing in the terminal UI
- Filing GitHub issues
- Reports
- Reference: rule codes, ignoring findings, saved results, sandpaper
- Development
One line, macOS or Linux:
curl -fsSL https://raw.githubusercontent.com/ucla-imls-open-sci/carpentries-workbench-checker/main/install.sh | shIt installs pixi if needed, clones the checker into
~/.local/share/wbcheck, builds its environment from the lockfile (nothing
installed system-wide), and puts a wbcheck launcher in ~/.pixi/bin,
which is already on your PATH if you use pixi.
wbcheck doctor # which optional pieces are ready
wbcheck --install-completion # tab completion
wbcheck update # pull the latest and refresh the environmentTo uninstall, delete ~/.local/share/wbcheck and ~/.pixi/bin/wbcheck. To
install a branch, tag, or fork, set WBCHECK_REF (default main),
WBCHECK_REPO, WBCHECK_HOME, or WBCHECK_BIN_DIR before sh, e.g.
... | WBCHECK_REF=my-branch sh.
Optional extras, each needed for one feature only:
| For | Needs |
|---|---|
wbcheck issues, filing from the TUI |
GitHub CLI, logged in (gh auth login) |
report --html / --pdf |
Quarto; PDF also needs LaTeX (quarto install tinytex) |
review --backend claude |
ANTHROPIC_API_KEY |
review --backend ollama |
Ollama running, plus a model (see local models) |
fix, o in the TUI |
$VISUAL or $EDITOR, e.g. export EDITOR=nvim (falls back to vi) |
wbcheck check LESSON # fast checks, saved to LESSON/.wbcheck/
wbcheck fix LESSON # work through them in $EDITOR
wbcheck fix LESSON --apply # let it make the safe mechanical fixes
wbcheck review LESSON --backend claude # optional: add the AI review
wbcheck issues LESSON --preview # optional: see them as GitHub issues
wbcheck explain WB403 # what a rule checks, when to keep the content, its limitsLESSON defaults to the current directory. It can also be a git URL
(wbcheck check https://github.com/librarycarpentry/lc-git.git), which is
cloned to a temporary directory. Every command reads and writes one results
file per lesson, <lesson>/.wbcheck/results.json, which ignores itself, so
the lesson repo needs no changes. wbcheck <command> --help lists every
option.
wbcheck check LESSON
wbcheck check LESSON --episode 03-sharing.md # one episode
wbcheck check LESSON --show-source # show the offending line under each finding (was --source)
wbcheck check LESSON --blame # record who last changed each file with findings
wbcheck check LESSON --fail-on warning # exit 1 on warnings too (also: info, never)
wbcheck check LESSON --fail-on annotations # exit 1 on errors and leftover `fix --annotate` notes
wbcheck check LESSON --quiet # save results without printing
wbcheck check LESSON --changed --since main # only files changed since main or uncommittedFindings are grouped by file, each with its rule code, line,
message, and fix. In terminals with hyperlinks, the code links to the guide
section behind the rule. It exits 1 if anything is at or above --fail-on
(default error), so it works in a pre-commit hook or CI.
Exit codes, the same for every command:
| Code | Meaning |
|---|---|
0 |
Ran, and nothing is at or above --fail-on |
1 |
Ran, and found something at or above --fail-on |
2 |
Could not run: bad option, a path that isn't a lesson, --episode names a missing file, or the mode needs a terminal and there isn't one |
-h works like --help on every command. wbcheck check never writes into a
folder that has neither episodes/ nor config.yaml.
With --changed, only findings in changed files are shown and counted
toward the exit code. That makes this a good pre-push check, holding your
changes to the standard without failing on problems you didn't touch:
wbcheck check . --changed --since main --fail-on warningwbcheck fix re-checks the lesson and opens what it found in your editor.
Narrow it with --code WB401 (repeatable), --file 'episodes/0*',
--min-severity (default warning), --source mechanical|ai, or
--changed / --since REF.
Every finding loads into the quickfix list, and the editor opens on the first one. Don't quit between fixes. Work through the list in place:
- Fix the finding, then
:w. ]q(or:cnext) to jump to the next one.[q/:cprevgoes back.:copenshows the whole list with messages and fixes;enteron a line jumps there.
When you quit, the lesson is re-checked and you get a count of what you fixed. Quickfix entries follow your edits within the session, but they don't know which findings are resolved. To refresh the list mid-session without quitting, re-run the check into it:
:wall | cexpr system('wbcheck fix . --print')--print just writes path:line:col: message lines to stdout, so it also
works for loading findings into an editor that's already open, or
nvim -q <(wbcheck fix . --print). It's worth a mapping:
vim.keymap.set('n', '<leader>wr', function()
vim.cmd("wall | cexpr system('wbcheck fix . --print')")
end, { desc = 'wbcheck: save and refresh quickfix' })With any other editor (or with --step in vim), it goes one finding at a
time. It shows the code, message, quote, and fix, then enter opens the file
at the line, s skips, and q stops. After each edit it re-checks and says
✔ fixed or ✗ still reported.
Findings can show in the editor's own panel without touching any lesson file.
Full setup is in editors/README.md.
- RStudio:
source("editors/wbcheck-markers.R"); wbcheck_markers()loads the saved results into the Markers pane throughrstudioapi::sourceMarkers(). - Positron and VS Code: a task with a problem matcher reads
wbcheck report . --lines -(file:line:col: severity CODE message) into the Problems panel.wbcheck report . --sarif out.sarifwrites SARIF 2.1.0 for the SARIF Viewer extension.
wbcheck fix --annotate writes each finding into the file as a comment above
the flagged block (<!-- wbcheck WB213: ... -->, or a # comment in front
matter). It's opt-in because it edits your files. Running it twice adds
nothing, notes never go inside code fences, and Workbench doesn't render them.
wbcheck fix --clean-annotations removes exactly the notes wbcheck wrote.
While any remain, wbcheck check warns (WB115); --fail-on annotations makes
that a failure for CI. Details and limits: editors/README.md.
Both show each change as a diff before writing, edit only the part the finding is about, and refuse if the line changed since the check ran. Content (prose, placeholders, exercises) is always left to you.
--apply makes safe fixes: unambiguous edits that change no meaning. It
asks y/n/a(ll)/q, and --yes applies them all.
| Code | Fix |
|---|---|
WB103 |
rename a front-matter exercise: typo to exercises: |
WB213 |
set a heading that skips a level to one below the previous heading |
--suggest offers editorial changes that are well defined but yours to
decide. Each asks on its own (default no), and --yes never applies them.
| Code | Suggestion | Why it's your call |
|---|---|---|
WB401 |
replace a vague objective opener with an observable verb (Understand → Explain, Know → Identify, ...), keeping the rest verbatim | it changes what the objective promises, and it still needs an exercise that assesses it |
WB009 |
add an unlisted episode to config.yaml's episodes:, never when the file looks like reference content (WB013, re-checked from the file itself) |
Workbench lets you leave drafts unlisted on purpose; listing one publishes it |
Use both with --apply --suggest, then review with git diff.
wbcheck review LESSON --backend claude # Anthropic API
wbcheck review LESSON --backend ollama # local, free
wbcheck review LESSON --episode 03-sharing.md --backend claude --effort mediumOff unless you run it: it's slow, and the claude backend costs roughly
$0.10 to $0.25 per episode at --effort medium. It adds structured findings
to the saved results (running check first if needed), and re-reviewing an
episode replaces that episode's earlier AI findings.
| Code | Area |
|---|---|
AI201 |
objectives not observable, or not assessed |
AI202 |
exercises without diagnostic power or variety |
AI203 |
difficulty or pacing mismatched to the audience |
AI204 |
too much at once (cognitive load) |
AI205 |
dismissive language, idioms, unexplained jargon |
AI206 |
glossary gap, with a draft definition in the fix |
AI207 |
accessibility (alt text, color-only cues) |
AI208 |
accuracy |
Each finding has a severity, a verbatim quote from the episode, the problem,
a suggested fix, and a scope label that groups related findings into one
pull request's worth of work.
Every finding is checked against the text. A finding whose quote isn't
in the episode is dropped, and the run reports how many were. Matching
tolerates whitespace, smart quotes, markdown emphasis, and ... elisions.
That proves each finding points at real text, not that its judgment is
right: treat AI findings as suggestions to verify.
The rubric is pinned. Every prompt includes
checker/rubric/: the Carpentries Lab reviewer checklist
plus excerpts of CLDT and the Workbench docs. Nothing is fetched at review
time, so the same lesson gets the same rubric every run. The episode and the
lesson's glossary go in the user message, never the system prompt, which
limits (but can't rule out) lesson text steering the model. Terms already in the glossary aren't
flagged as gaps, and the mechanical findings are included so the model
doesn't repeat them.
If any episode fails (backend down, refusal, timeout), the review saves the ones that succeeded and exits 1.
| Backend | Needs | Notes |
|---|---|---|
claude |
ANTHROPIC_API_KEY |
Default model claude-opus-5-5. --effort defaults to high; low/medium are cheaper and faster, xhigh/max more thorough. The rubric and glossary are prompt-cached across episodes; if a safety classifier declines an episode, the API retries it on a fallback model |
ollama |
ollama serve and a pulled model |
Local and free. Output is constrained to the findings schema, with one retry if the model misses it. Calls time out after 15 minutes |
--model overrides the default for either backend.
Install Ollama, run ollama serve, and
ollama pull <model>. wbcheck doctor tells you whether the server is up
and the default model is pulled.
| Model | Download | Use it for |
|---|---|---|
qwen3.5:9b-q4_K_M (default) |
~6.6 GB | General review; best quality for the footprint, with headroom on 16GB |
qwen3.5:4b |
~3.4 GB | Faster checks while drafting; use the 9B for a final pass |
qwen2.5-coder:7b |
~4.7 GB | Lessons heavy on shell, Python, or R code |
gpt-oss:20b |
~14 GB | Best local quality if you can close everything else |
On 16GB, don't run a review alongside another large model or a heavy IDE: swapping slows it down long before memory runs out.
wbcheck tui LESSONA folder → file → rule tree on the left (colored by worst severity), the findings table on the right, and a detail pane with the message, quote, fix, guide links, and surrounding source lines.
| Key | Does |
|---|---|
enter (tree) |
filter to that file or rule code |
space |
select or unselect a finding |
o |
open the file at the finding's line in $VISUAL / $EDITOR; on return it re-checks, and fixed findings drop off |
r |
re-run the mechanical checks, keeping AI findings |
i |
ignore the selection (or current finding) in .wbcheck.toml |
c |
file issues: the selection as one issue, or with nothing selected, the visible warnings and errors grouped as wbcheck issues would. Confirms first and skips anything already filed |
s / a |
cycle minimum severity / source (mechanical, AI) |
/ |
search; esc clears every filter |
q |
quit |
wbcheck issues LESSON # dry run: the issues it would file
wbcheck issues LESSON --preview # ...with each body
wbcheck issues LESSON --create # file them via gh (asks first; -y skips)
wbcheck issues LESSON --group-by file # one issue per file, even for lesson-wide rules
wbcheck issues LESSON --source ai --min-severity info --repo me/my-forkEach issue is sized for one pull request. A rule that shows up in 3 or more
files gets one lesson-wide issue; other mechanical findings get one issue per
file (--group-by file or rule forces one or the other). AI findings are
grouped by file and scope, with an episode's one-offs sharing an
ai-suggested "other suggestions" issue. Each item has a checkbox, a link to
its line at the checked commit, the fix, and a "Why these matter" section
citing the guide. info notes are left out unless you pass
--min-severity info.
Re-running doesn't re-file. Each finding's ID is hidden in the issue
body, and issues skips anything already filed in a wbcheck-labelled
issue, open or closed, so a won't-fix stays closed. Keep the label and the
hidden ID on the issue for this to work. The repo defaults to the lesson's
GitHub origin. If files had uncommitted changes at check time, it warns
first, since their items can't link to GitHub.
wbcheck report LESSON # terminal
wbcheck report LESSON --md report.md --json results.json
wbcheck report LESSON --html report.html --open # needs Quarto
wbcheck report LESSON --pdf report.pdf # needs Quarto + LaTeX
wbcheck report LESSON --sarif out.sarif # SARIF 2.1.0 (`-` for stdout)
wbcheck report LESSON --lines - # file:line:col: severity CODE messageRenders saved results without re-checking. Every format opens with the tool
version and the lesson's metadata from config.yaml and CITATION.cff, so a
report handed to someone else identifies itself. The markdown, HTML, and PDF
reports have three parts: a checklist of files, an action summary with one
row per shared fix (a problem in 16 places is one row), and per-file detail
in editor order. Line references link to GitHub at the checked commit when
possible. PDFs don't render the severity emoji; severity still shows in the
checkbox and bold rule code.
Every check has a stable code, defined once in
checker/rules.py. Codes are never renumbered or reused.
AI201 to AI208 are the AI review's.
wbcheck explain WB403 # or a name: wbcheck explain objectives-not-assessed
wbcheck rules # every rule: code, default severity, topic, title
wbcheck rules --search alt # rules mentioning some text
wbcheck rules --topic accessibilityexplain shows what the check observes, why it matters, what to do, when
keeping the content is reasonable, what the check can't see, whether
wbcheck fix can help, and the exact source sections, with their URLs and
when they were last checked. It works offline, with no lesson needed. The same
explanation appears in the TUI's detail pane, a "Rule explanations" appendix in
markdown/HTML/PDF reports, and filed issues.
Each rule also says what kind of claim it makes: its authority (a Workbench technical requirement, a Carpentries review criterion, a recommendation, or wbcheck's own policy) and its detection (deterministic, a heuristic proxy, or an AI suggestion). Passing wbcheck doesn't mean a lesson is accessible, well taught, or approved by the Carpentries; it's an authoring aid.
| Code | Severity | Checks |
|---|---|---|
WB001 |
error | config.yaml not found |
WB002 |
error | config.yaml is not valid YAML |
WB003 |
error | config.yaml is not a key: value mapping |
WB004 |
error | config.yaml field is empty or still the template value |
WB005 |
warning | created date not set |
WB006 |
info | life_cycle still pre-alpha |
WB007 |
error | episode listed in config.yaml does not exist |
WB008 |
warning | file in episodes/ has no .md/.Rmd extension |
WB009 |
warning | episode file not listed in config.yaml |
WB010 |
info | no local glossary file |
WB011 |
error | no episodes/ directory |
WB012 |
error | --episode named a file that doesn't exist |
WB013 |
warning | episode file looks like reference content |
WB101 |
error | episode has no YAML front matter |
WB102 |
error | front matter is not a key: value mapping |
WB103 |
error | required front-matter field missing |
WB104 |
warning | teaching/exercises is not a number of minutes |
WB105 |
info | episode length outside 20-60 minutes |
WB110 |
error | episode title is still the scaffold default |
WB111 |
warning | episode body still contains scaffold example text |
WB112 |
error | placeholder text in questions/objectives/keypoints |
WB113 |
warning | setup/instructor notes/profiles still the scaffold |
WB114 |
warning | glossary is still the scaffold placeholder |
WB115 |
warning | wbcheck note (from fix --annotate) left in a lesson file |
WB201 |
info | unrecognized div type |
WB202 |
error | closing ::: with no matching open div |
WB203 |
error | div never closed |
WB204 |
error | required questions/objectives/keypoints block missing |
WB205 |
info | more challenges than solutions |
WB210 |
error | episode uses a level-1 heading |
WB211 |
warning | first heading is not level 2 (callout/spoiler titles don't count) |
WB212 |
warning | duplicate heading text |
WB213 |
warning | heading skips a level (h2 → h4), following the rendered outline |
WB301 |
warning | image has no alt text |
WB302 |
error | image file not found |
WB303 |
warning | generic link text ("click here") |
WB304 |
warning | internal link target not found |
WB401 |
warning | objective opens with a hard-to-assess verb (suggests a rewrite) |
WB402 |
info | more than 4 objectives in one episode |
WB403 |
warning | objectives declared but no exercise time |
WB404 |
info | heavy use of contractions |
A few details:
- Div and heading checks skip fenced code blocks, so a lesson that teaches
Markdown or shell
#comments doesn't trip them. Fences follow CommonMark and divs follow Pandoc, including::: {#q .questions}. Reference-style links ([text][ref]) aren't checked yet. - Image paths resolve relative to
episodes/or the lesson root, and.htmllinks resolve to the.mdor.Rmdsource they're rendered from. - Images count as described with caption text, an
{alt='...'}attribute (which may wrap across lines), or the decorative marker{alt=""}. WB301 checks that one is there, not that it describes the figure. - Duplicate headings (WB212) are judged within their hierarchy, as pegboard
does: the same
### Solutionunder two different##sections is fine. - Objectives are the top-level items of the objectives block, with any list
marker (
-,*,+,1.); nested bullets belong to their parent. - Timings must be finite, non-negative numbers of minutes (zero and fractions are fine). WB105 and WB403 are skipped while a timing is invalid.
- WB404 counts contractions in author prose only, skipping code and
blockquotes (
> ...), and its threshold is wbcheck's own. - The objectives, contraction, glossary, and heading-skip checks come from
the Collaborative Lesson Development Training
(CLDT) and the Carpentries Lab reviewer
and editor checklists, not from sandpaper.
wbcheck explaincites the exact section for each.
To ignore one finding where it happens, put a comment above it. The reason is optional but worth writing:
<!-- wbcheck-ignore WB404: quoted learner data, not prose -->
This line is the one WB404 would report.
<!-- wbcheck-ignore-file WB213, WB401: transcript of a recorded session -->wbcheck-ignore CODE covers that code on the next non-blank line.
wbcheck-ignore-file CODE[, CODE] covers the whole file. In front matter use
a # comment. Comments inside code blocks are ignored, and wbcheck's own
checks never read these comment lines. Ignored findings are counted in the
saved results, as with .wbcheck.toml.
Put a .wbcheck.toml at the lesson root and commit it so collaborators
share it:
[ignore]
codes = ["WB404"] # a rule, everywhere
paths = ["episodes/all_exercises.md"] # every finding in these files (globs ok)
ids = ["8289307d05d3"] # single findings, by IDMatching findings are dropped before saving, and reports say how many. The
TUI's i key adds IDs here, rewriting the file in this form, so comments in
it aren't kept.
- Mechanical checks re-run on every
check,review,fix, and TUI re-check, so they always match the files as they are. - AI findings are kept between runs, but each file is fingerprinted. If a
file changes after its review, its AI findings are marked stale: shown
with a note and never filed as issues until you re-run
review. - Results record the lesson and git commit they belong to. Results for a
different lesson are replaced, never merged.
check --episodekeeps the rest of the lesson's results only if those files haven't changed; otherwise they're marked partial until the next full check. - Each finding has a 12-character ID, a hash of its code, file, and message (not the line number), so it survives unrelated edits. AI finding IDs come from the rule, file, and quoted text, so a reworded re-review keeps the same ID. IDs are never reassigned, which is what ignoring and issue de-duplication rely on.
- When a check's wording changes, its findings keep their old IDs: the
finding carries a frozen
identity_anchorbuilt from the old wording, so.wbcheck.tomlignores and filed issues still match. - The results file is format version 3 (it adds
identity_anchor). wbcheck reads versions 1-3. An older wbcheck refuses a version-3 file with "not supported... re-runwbcheck check"; doing that rebuilds it in the old format. - For a git URL, results are saved under
./.wbcheck/<host_owner_repo>/.
The Carpentries' CI (sandpaper::validate_lesson() and pegboard's
validate_divs(), validate_headings(), and validate_links()) is
authoritative but takes several minutes, and only runs after you push.
wbcheck mirrors those rules locally in under a second and adds the CLDT and
Carpentries Lab checks. It's an approximation: sandpaper is still the final
word.
git clone https://github.com/ucla-imls-open-sci/carpentries-workbench-checker.git
cd carpentries-workbench-checker
pixi install
pixi run wbcheck check path/to/lesson # or `pixi shell`, then plain `wbcheck`The package is installed in editable mode, so code changes take effect immediately.
| Task | Does |
|---|---|
pixi run test |
the test suite; no network, models, gh, or Quarto needed (all faked) |
pixi run lint / lint-fix |
ruff |
pixi run refresh-rubric |
re-fetch the AI review's guidance into checker/rubric/; read the diff before committing. Fails loudly if an upstream section moved |
pixi run ollama-serve / pull-models |
run Ollama and pull the local models from the pixi environment |
CI runs the tests and lint, then installs with install.sh on Linux and
macOS and runs wbcheck from outside the repo, so a file missing from the
package fails the build. pixi run format exists but isn't applied
wholesale, since it would reflow intentionally formatted prose strings.
Design notes are in design/, starting with
modernization-assessment-2026-09-29.md.
Changes are in CHANGELOG.md. HTML and PDF reports use a
bundled Quarto extension,
checker/quarto/_extensions/checker-report/,
which owns the look while report.py owns the content.
The original flag-based CLI (pixi run check LESSON ...) still works from a
development checkout but is superseded by wbcheck.
legacy/proposal_analysis.ipynb is unrelated to lesson checking and kept for
reference.
See CONTRIBUTING.md. Everyone participating is expected to follow the Code of Conduct.
wbcheck began in 2023 as a project of UCLA DataSquad students Lawrence Lee and Eric Huang, and it evolved over time as the UCLA-IMLS Lessons for Librarians in Open Science program learned what collaborative lesson development involves. In 2026 Tim Dennis filled in the implementation with AI assistance. See CITATION.cff for how to cite it.
Zhiyuan Yao coordinated the program. Loretta Hu, Hyerin Lee, Emily Gong, and Madeline Kim contributed to its website, documentation, and lessons. All are authors in CITATION.cff. Roles follow the CRediT taxonomy in CREDITS.md.