Correct the compendium format docs, test the links, and make CI use the locked ruff

[gaurav]

Documentation fixes found while working out a clean division of labour between Babel's docs and
NodeNorm's, plus a test that keeps the cross-repo links from rotting again. Companion to
NCATSTranslator/NodeNormalization#403.

Corrections

preferred_name is used by NodeNorm. DataFormats.md said it was "not currently used by
NodeNorm, but will be in the future". NodeNorm has consumed it since its #299 and returns it as the
label of the normalized clique. The entry now says so, and keeps the more useful warning that the
preferred name is not necessarily the clique leader's label.

Synonym files and GeneProtein conflation. The note that synonym files are built with GeneProtein
conflation turned off is still true of the per-type files, but v1.13 added a separate
synonyms/GeneProteinConflated.txt.gz so NameRes can be loaded with that conflation on. Both are
now described.

Architecture.md listed a clique-level descriptions field. write_compendium() does not emit
one — descriptions are per-identifier, in the d key, sorted shortest first. Replaced with a
description of what the identifiers list actually carries, and clarified taxa as the union of
the per-identifier t values.

Removed a pointer to docs/debugging/Conflation.md, which does not exist.

Cross-references

Conflation.md explains what NodeNorm does with the conflation files but never says how a caller
turns a conflation on, so it now points at NodeNorm's API documentation for the flags.

Deployment.md step 3.1 says to "split the Babel outputs into smaller files to improve load times"
without saying what the sizes are or why it matters. That is documented in detail on the NodeNorm
side, so this links there rather than growing a second copy that can drift.

Link hygiene, and a test that enforces it

Several cross-repo links pointed at master, which none of these repositories uses any more:
README.md, docs/Babel.ipynb, docs/Deployment.md, and a provenance URL reported to users from
src/snakefiles/diseasephenotype.snakefile. They resolve today only through GitHub's branch-rename
redirect. CITATION.cff still named TranslatorSRI/Babel as repository-code, which is the URL
Zenodo reads. Two release notes linked [May 2024](TranslatorMay2024) with no .md, so they never
resolved at all.

tests/test_docs_links.py now enforces four rules across every tracked Markdown file, and — for
the URL rules — every tracked file that can carry a GitHub link (snakefiles, CITATION.cff,
notebooks, YAML, Python):

• relative links resolve on disk
• heading anchors match a real heading
• no link is pinned to the master branch of an NCATSTranslator repository
• no link names the pre-rename TranslatorSRI org for Babel, NameRes or NodeNorm

Both bans are deliberately scoped by name: biolink-model really does default to master, and
TranslatorSRI/babel-validation really is still under that org, so a blanket ban would push someone
to "fix" correct links into 404s.

Everything is offline — nothing fetches a URL, because a test that fails when GitHub is slow is a
test people learn to ignore. It is marked unit, so per-PR CI runs it, and it discovers files
through git ls-files rather than walking the repo, which keeps it out of babel_downloads/ and
babel_outputs/. Ported from NameResolution's test of the same name (NameResolution#262,
NodeNormalization#403); the three copies should be kept in sync.

CI runs the ruff in uv.lock now

The formatting job installed whatever ruff was newest rather than the project's, so it was checking
the repo with a version nobody has locally — 0.16.0 in CI against 0.15.21 in the lockfile. That is
how this PR, which touches no Python at all, came to fail on three Markdown files: ruff 0.16 began
formatting the Python inside fenced blocks. version-file: "uv.lock" makes the job run the locked
version, so a formatter upgrade arrives as a lockfile change that can be reformatted for in the
same PR instead of as a red X on an unrelated one. #981 has since landed the other half of this —
extend-exclude = ["*.md"], so ruff leaves the fragments in our docs alone — and CONTRIBUTING.md
now says where a bump's reformatting is due.

While in the file: astral-sh/ruff-action v3 → v4.1.0, which runs on node24 and so drops the
"Node.js 20 is deprecated" warning. It is pinned exactly because releases from v4 on are immutable
— there is no moving v4 tag, and @v4 does not resolve.

The actions/setup-python step is gone: ruff is a standalone binary and never used the Python it
installed. That was the repository's only use of the action, so this supersedes #966 (Dependabot
bumping it 6 → 7), which can be closed.

Also removed

Three files committed by accident at some point, belonging to none of the above: .DS_Store (now
gitignored), input_data/manual_concords/drugchemical.numbers (an Apple Numbers working copy of
drugchemical.tsv that nothing references), and
docs/sources/EMAPA/impact-report/modified-cliques.csv, which #781 had already decided not to
commit.

Deliberately not changed

DataFormats.md says there are "only two conflation files". I initially had this down as wrong
because DrugChemicalSmaller.txt.gz appears in two release notes — but it is a SapBERT
training-data export (src/exporters/sapbert.py), not a conflation file. The existing text is
correct.

Conflation.md documents NodeNorm's query-time behaviour, which looks at first like it belongs in
the other repo. Leaving it: it is where readers actually ask the question, and NodeNorm now
restates the contract from its own side.

Verification

• uv run pytest -m unit — 406 passed (402 before; the four new link tests are the difference)
• ruff check, ruff format --check, snakefmt --check, rumdl check . — all clean, re-run
  against ruff 0.16.0 after merging main
• uvx ruff@0.16.0 format --check . and the locked 0.15.21 agreed at 191 files before Upgrade all dependencies, and document how to do it #981 raised
  the lock to 0.16.0

Follow-ups filed

#971 (three documented paths that do not exist: docs/pipelines/, docs/sources/EMAPA/*.md, and
the docs/debugging/Conflation.md walkthrough removed here — the walkthrough still seems worth
writing), #972 (the "General releases" section of releases/README.md is an empty TODO, which
matters because that index is effectively the provenance table behind NodeNorm's /status
babel_version).

🤖 Generated with Claude Code

[gaurav]

Two link commits here.

Cross-repo links now point at main (fe6fee45). Babel, NodeNormalization and NameResolution all default to main; a /blob/master/ URL resolves only through GitHub's post-rename redirect, which is the kind of thing that disappears quietly. Seven of them: two notebook links in README.md, one in docs/Babel.ipynb, one in docs/Conflation.md, two in docs/Deployment.md, and the disease-concord URL that diseasephenotype.snakefile reports to users. Also CITATION.cff, which still named TranslatorSRI/Babel as repository-code — the field Zenodo and GitHub's citation widget read.

Deliberately not touched: every TranslatorSRI/babel-validation reference. That repository really does still live under TranslatorSRI (NCATSTranslator/babel-validation is a 404), so those are correct rather than stale. The TranslatorSRI strings in input_data/manual_concords/drugchemical.tsv are data.

A test to stop it drifting back (5b2e9aaa), ported from NameRes so all three repos enforce the same rules: relative links resolve, heading anchors exist, nothing is pinned to master. Offline, so a slow GitHub cannot make it flaky. Adapted rather than copied — Babel has no API server, so the endpoint-anchor check has no equivalent, and instead the master ban covers the snakefiles, src/ and CITATION.cff. The pattern is scoped to NCATSTranslator so it does not flag biolink-model or babel-validation.

It found two dead links immediately: releases/TranslatorFuguJuly2024.md and releases/TranslatorMay2024.md both link to the previous release without the .md extension, so both render as dead links on GitHub. Fixed.

Verified: all eight rewritten URLs return 200, the notebook is still valid JSON, CITATION.cff still parses, rumdl is clean, and the suite passes (405 passed, 100 skipped).

[Copilot]

Pull request overview

This PR updates Babel’s documentation to accurately reflect the current Babel→NodeNormalization contract (especially around compendium fields, synonym/conflation variants, and descriptions), and replaces drift-prone restatements with links to canonical NodeNorm docs. It also adds an offline test to keep documentation/source links consistent across Babel/NodeNorm/NameRes.

Changes:

• Correct compendium/schema documentation details (e.g., preferred_name usage, description/taxa semantics) and remove a dangling reference to a non-existent doc.
• Replace stale master/old-org GitHub links with main and NCATSTranslator, and add cross-repo doc links to NodeNorm where appropriate.
• Add an offline documentation link checker test intended to enforce shared link rules across repos.

Reviewed changes

Copilot reviewed 12 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_docs_links.py	Adds offline tests to validate relative links, heading anchors, and ban stale GitHub URL forms.
src/snakefiles/diseasephenotype.snakefile	Updates a GitHub URL from blob/master to blob/main.
releases/TranslatorMay2024.md	Fixes an internal release-note link to include the .md extension.
releases/TranslatorFuguJuly2024.md	Fixes an internal release-note link to include the .md extension.
README.md	Updates NodeNorm/NameRes notebook links from master to main.
docs/Understanding.md	Clarifies where descriptions exist (compendia-only) and how downstream frontends can expose them.
docs/Deployment.md	Links to NodeNorm loader documentation instead of duplicating split-size rationale; updates NameRes deployment link to main.
docs/DataFormats.md	Corrects/clarifies preferred_name usage, synonym file variants re: GeneProtein conflation, and removes a pointer to a non-existent debugging doc.
docs/Conflation.md	Links to NodeNorm API docs for how to enable conflation flags at query time.
docs/Babel.ipynb	Updates a GitHub link from blob/master to blob/main.
docs/Architecture.md	Corrects compendium field descriptions (identifier-level descriptions/taxa; removes nonexistent clique-level descriptions).
CITATION.cff	Updates repository URL to https://github.com/NCATSTranslator/Babel.

Comments suppressed due to low confidence (1)

tests/test_docs_links.py:43

• SOURCE_WITH_LINKS = REPO_ROOT.rglob("*") still traverses excluded directories like .git, babel_outputs/, and babel_downloads/ (the filter happens after traversal). On developer machines and even in CI, that can make this unit test unexpectedly slow and/or hit the global timeout.

Consider walking with os.walk() and pruning EXCLUDED dirs, so excluded trees are never descended into.

SOURCE_WITH_LINKS = sorted(
    p for p in REPO_ROOT.rglob("*")
    if p.is_file() and _ours(p)
    and (p.suffix in LINK_BEARING_SUFFIXES or p.name.startswith("Dockerfile"))
)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.