Correct the compendium format docs, test the links, and make CI use the locked ruff

.github/workflows/check-formatting.yml

@@ -9,12 +9,20 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v7
-      - uses: actions/setup-python@v6
-      - uses: astral-sh/ruff-action@v3
+      # version-file so CI runs the ruff in uv.lock. Without it the action installs the latest
+      # release, so a ruff that nobody has locally can fail this job on a PR that did not touch
+      # any Python -- 0.16 did exactly that by adding Markdown formatting. ruff is a standalone
+      # binary, so no setup-python step is needed.
+      #
+      # Pinned to an exact version because ruff-action publishes immutable releases from v4 on:
+      # there is no moving `v4` tag to track, and `@v4` fails to resolve.
+      - uses: astral-sh/ruff-action@v4.1.0
         with:
+          version-file: "uv.lock"
           args: "check --output-format github"
-      - uses: astral-sh/ruff-action@v3
+      - uses: astral-sh/ruff-action@v4.1.0
         with:
+          version-file: "uv.lock"
           args: "format --check"
 
   check-formatting-with-snakefmt:

.gitignore

@@ -32,6 +32,9 @@ docs/sources/*/impact-report/modified-cliques.json
 # Build metadata regenerated by `uv sync` / setuptools; never committed.
 *.egg-info/
 
+# macOS Finder metadata, which turns up in any directory browsed in Finder.
+.DS_Store
+
 # Coverage data written by `pytest --cov`.
 .coverage
 .coverage.*

CITATION.cff

@@ -32,7 +32,7 @@ authors:
     family-names: Yi
     affiliation: Renaissance Computing Institute, University of North Carolina
     orcid: https://orcid.org/0000-0002-5699-1259
-repository-code: 'https://github.com/TranslatorSRI/Babel'
+repository-code: 'https://github.com/NCATSTranslator/Babel'
 abstract: >-
   Babel creates cliques of equivalent identifiers across
   many biomedical vocabularies.

CONTRIBUTING.md

@@ -55,8 +55,10 @@ PYTHONPATH=. uv run pytest     # then the four linter commands above
 
 No `npm-check-updates`-style step is needed: every dependency in `pyproject.toml` is an unbounded
 `>=`, so nothing caps a package below its latest release and `uv lock --upgrade` already goes
-there. Run the formatters as well as the tests — a `ruff` or `rumdl` bump can change their output
-and fail the formatting workflow on an otherwise untouched file.
+there. Run the formatters as well as the tests — a `ruff` or `rumdl` bump can change their output,
+and reformatting for it is part of the upgrade. The ruff job reads its version from `uv.lock`, so
+that reformatting is due in the upgrade PR itself rather than landing on whichever unrelated PR
+opens next.
 
 When a package refuses to move, `uv lock --upgrade-package 'name==version' --dry-run` prints the
 conflict. `bmt` is held at 1.4.6 this way: 1.4.8 pulls in `biolink-model`, which requires

README.md

@@ -113,7 +113,7 @@ other APIs intended primarily for Translator users.
 
 You can find out more about NodeNorm at its
 [Swagger interface](https://nodenormalization-sri.renci.org/docs) or
-[in this Jupyter Notebook](https://github.com/NCATSTranslator/NodeNormalization/blob/master/documentation/NodeNormalization.ipynb).
+[in this Jupyter Notebook](https://github.com/NCATSTranslator/NodeNormalization/blob/main/documentation/NodeNormalization.ipynb).
 
 ### What is the Name Resolver (NameRes)?
 
@@ -125,7 +125,7 @@ autocomplete text fields, it has also been used for named-entity linkage.
 
 You can find out more about NameRes at its
 [Swagger interface](https://name-resolution-sri.renci.org/docs) or
-[in this Jupyter Notebook](https://github.com/NCATSTranslator/NameResolution/blob/master/documentation/NameResolution.ipynb).
+[in this Jupyter Notebook](https://github.com/NCATSTranslator/NameResolution/blob/main/documentation/NameResolution.ipynb).
 
 ## Understanding Babel outputs
 

docs/Architecture.md

@@ -79,11 +79,12 @@ equivalence cliques.
 
 Each line of a compendium file is a JSON object representing one clique. A clique includes:
 
-- `identifiers` — list of all equivalent CURIEs, in preferred-prefix order
+- `identifiers` — list of all equivalent CURIEs, in preferred-prefix order. Each entry carries its
+  own label (`l`), descriptions (`d`, collected from UberGraph and sorted shortest first) and taxa
+  (`t`)
 - `ic` — information content score (from UberGraph)
-- `taxa` — associated taxa (for genes, proteins, etc.)
+- `taxa` — associated taxa (for genes, proteins, etc.); the union of the per-identifier `t` values
 - `preferred_name` — the preferred human-readable label for the clique
-- `descriptions` — descriptions collected from UberGraph
 - `type` — Biolink semantic type
 
 The first identifier in `identifiers` is the preferred identifier for the clique. See

docs/Babel.ipynb

@@ -13,7 +13,7 @@
    "id": "13384de2-1c99-4aa6-8792-de8ba393589e",
    "metadata": {},
    "source": [
-    "This Jupyter Notebook shows you what it looks like when you [run Babel](https://github.com/NCATSTranslator/Babel/blob/master/docs/Downloads.md)."
+    "This Jupyter Notebook shows you what it looks like when you [run Babel](https://github.com/NCATSTranslator/Babel/blob/main/docs/Downloads.md)."
    ]
   },
   {

docs/Conflation.md

@@ -26,6 +26,9 @@ on, NodeNorm will:
    identifier is the first identifier in the clique, and which consists of all the identifiers from
    all the cliques included in that conflation.
 
+For how to turn each conflation on when querying, see the
+[NodeNorm API documentation](https://github.com/NCATSTranslator/NodeNormalization/blob/main/documentation/API.md).
+
 ## How are types handled for conflated cliques?
 
 Babel does not assign a type to any conflations. When NodeNorm is called with a particular

docs/DataFormats.md

@@ -63,7 +63,7 @@ This entry consists of the following fields:
 | identifiers[0].l | G6PC1                                                       | A label for this identifier. This will almost always be from the source of the CURIE (in this case, the label is from the NCBI Gene database).                                                                                                                        |
 | identifiers[0].d | (blank in this example, but usually 1-3 sentences)          | A description of this identifier or concept from this source.                                                                                                                                                                                                         |
 | identifiers[0].t | ["NCBITaxon:9606"]                                          | A list of taxa that this concept is found in as NCBITaxon CURIEs. NCBITaxon:9606 refers to the species _Homo sapiens_.                                                                                                                                                |
-| preferred_name   | G6PC1                                                       | The preferred name for this clique. This is not currently used by NodeNorm, but will be in the future.                                                                                                                                                                |
+| preferred_name   | G6PC1                                                       | The preferred name for this clique. NodeNorm returns this as the `label` of the normalized clique. Note that it is not necessarily the label of the clique leader — see below.                                                                                        |
 | taxa             | ["NCBITaxon:9606"]                                          | A list of taxa that this concept is found in as NCBITaxon CURIEs. This is combined from all the individual taxa from each identifier.                                                                                                                                 |
 | type             | biolink:Gene                                                | The Biolink type of this concept. Must be a class from the [Biolink model](https://biolink.github.io/biolink-model/) with a `biolink:` prefix.                                                                                                                        |
 
@@ -140,8 +140,10 @@ This entry consists of the following fields:
 | taxon_specific          | true or false                      | True if this concept is associated with one or more specific taxa; false if it is not taxon-specific.                                                                                                                                       |
 | types                   | ["Gene", "GeneOrGeneProduct", ...] | A list of Biolink types (without the `biolink:` prefix) for this concept. This is arranged in the same order provided by the Biolink Model Toolkit, starting with the narrowest concept, expanding to the broadest, followed by mixins.     |
 
-Note that the synonym files are generated with DrugChemical conflation turned on, but GeneProtein
-conflation turned off.
+Note that the per-type synonym files are generated with DrugChemical conflation turned on, but
+GeneProtein conflation turned off. A separate `synonyms/GeneProteinConflated.txt.gz` is also
+produced, containing the same concepts with GeneProtein conflation applied, so that NameRes can be
+loaded with that conflation turned on.
 
 ## Conflation files
 
@@ -350,7 +352,6 @@ If the edge you expected is absent, the link was never generated upstream (a com
 RxCUI-typing problem); if it is present but the CURIEs still aren't conflated, the pair was dropped
 by a conflation filter — look up the reason in the paired per-run exclusion report
 `babel_outputs/reports/drugchemical/excluded_pairs.tsv.gz` (its `reason` column names the filter).
-`docs/debugging/Conflation.md` walks through this two-report diagnosis flow end to end.
 
 `Identifier.parquet` — one row per identifier extracted into an `ids/` file:
 

docs/Deployment.md

@@ -16,13 +16,14 @@ to users who aren't system administrators for these tools:
 2. Store the Babel outputs alongside other Babel releases on Hatteras.
 3. Deploy a new NodeNorm instance
    1. Split the Babel outputs into smaller files to improve load times and put them on a public web
-      server.
+      server. The split sizes and the reason for splitting are documented in
+      [NodeNorm's loader documentation](https://github.com/NCATSTranslator/NodeNormalization/blob/main/documentation/Loader.md).
    2. Update the Translator-devops repo with the URL to these Babel output files.
    3. Create a [Redis R3 External] instance to store identifiers.
    4. Run the [NodeNorm loader] to load the Babel outputs into a Redis instance.
    5. Create a [NodeNorm web server] to share the data in a Redis instance.
 4. Deploy a new NameRes instance (either
-   [locally](https://github.com/NCATSTranslator/NameResolution/blob/master/documentation/Deployment.md)
+   [locally](https://github.com/NCATSTranslator/NameResolution/blob/main/documentation/Deployment.md)
    or
    [on Kubernetes](https://github.com/helxplatform/translator-devops/tree/ed25b5f5bfe2383ade8457da97341c90500f5291/helm/name-lookup))
    1. Create an empty Apache Solr instance.

docs/Understanding.md

@@ -69,13 +69,17 @@ conflated.
 
 ## Where do the clique descriptions come from?
 
-Currently, all descriptions for NodeNorm concepts come from
+Currently, all descriptions come from
 [UberGraph](https://github.com/INCATools/ubergraph/). You will note that
 descriptions are collected for every identifier within a clique, and then the
 description associated with the most preferred identifier is provided for the
-preferred identifier. Descriptions are not included in NameRes, but the
-`description` flag can be used to include any descriptions when returning
-cliques from NodeNorm.
+preferred identifier.
+
+Descriptions are written to the [compendium files](./DataFormats.md#compendia-files)
+only: the [synonym files](./DataFormats.md#synonym-files) have no description field.
+Whether a frontend exposes them therefore depends on which files it is built from —
+NodeNorm reads the compendia and can return them via its `description` flag, while
+anything built from the synonym files has nothing to return.
 
 ## What are "information content" values?
 

releases/TranslatorFuguJuly2024.md

@@ -29,6 +29,6 @@ Next release: [TranslatorGuppyAugust2024](./TranslatorGuppyAugust2024.md)
 * Updated PANTHER pathways from SequenceAssociationPathway3.6.7.txt to
   SequenceAssociationPathway3.6.8.txt.
 
-## Releases since [May 2024](TranslatorMay2024)
+## Releases since [May 2024](TranslatorMay2024.md)
 
 * No official releases

releases/TranslatorMay2024.md

@@ -18,7 +18,7 @@ Next release: [Translator "Fugu" July 2024](TranslatorFuguJuly2024.md)
   in synonyms file.
 * Minor fixes.
 
-## Releases since [December 2023](TranslatorDecember2023)
+## Releases since [December 2023](TranslatorDecember2023.md)
 
 * [Babel v1.5.0](https://github.com/NCATSTranslator/Babel/releases/tag/v1.5.0):
   * Normalize DrugChemical conflation IDs by @gaurav in #250

src/snakefiles/diseasephenotype.snakefile

@@ -250,7 +250,7 @@ rule disease_manual_concord:
                     "url": "https://github.com/NCATSTranslator/Babel",
                 }
             ],
-            url="https://github.com/NCATSTranslator/Babel/blob/master/input_data/manual_concords/disease.txt",
+            url="https://github.com/NCATSTranslator/Babel/blob/main/input_data/manual_concords/disease.txt",
             concord_filename=output.outfile,
         )
 

tests/README.md

@@ -249,6 +249,13 @@ and how to add new checks or vocabularies.
   NCBI FTP server. Covers pulling plain text and gzipped files to memory or disk
   with optional decompression. Requires `--network` to run.
 
+- **`test_docs_links.py`** (`unit`) — Checks the links in every tracked Markdown file and in the
+  source that reports URLs to users (snakefiles, `CITATION.cff`, notebooks): relative links and
+  heading anchors must resolve on disk, and no link may point at a `master` branch or at the
+  pre-rename `TranslatorSRI` org. Entirely offline — nothing fetches a URL, because a test that
+  fails when GitHub is slow is a test people learn to ignore. Ported from NameResolution's test of
+  the same name; keep the three repos' copies in sync.
+
 - **`test_uber.py`** (`network`) — Tests the `UberGraph` class for querying ontology
   subclasses and cross-references via SPARQL. Covers direct and indirect subclass retrieval,
   filtering by cross-reference presence, and exact-match label queries. Tests may xfail at