-
Notifications
You must be signed in to change notification settings - Fork 4
Improve NameRes documentation: Translator Guide, Babel data guide, and doc drift fixes #262
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
gaurav
wants to merge
50
commits into
main
Choose a base branch
from
improve-documentation
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 2 commits
Commits
Show all changes
50 commits
Select commit
Hold shift + click to select a range
5785147
Add Translator Guide and improve README for issue #133
gaurav 17fb4a1
Document conflations in OpenAPI spec and /status endpoint
gaurav 27c74ad
Fix descriptions of the `limit` GET/POST parameter. Closes #269.
gaurav bd26427
Fix descriptions of the `limit` GET/POST parameter (#271)
gaurav 5c7a1a4
Add checked-in Solr configset and parallel data loader with doc-count…
gaurav bb59c41
Switch Solr to standalone mode and self-contained core backups
gaurav 625f8cc
Document the new data-loading process
gaurav c7f6221
Apply suggestion from @gaurav
gaurav 5ba1178
Fix /status when Solr runs in standalone mode
gaurav 6eb52a2
Harden the load guard: count deltas, bound the wait, survive odd file…
gaurav 6c7ffc0
Make the backup build restart-safe and stop gzipping on one core
gaurav cb32758
Keep the schema's single source of truth out of configoverlay.json
gaurav fb83dbb
Check the load guard and the backup roundtrip in CI
gaurav 4eec268
Track the Solr 9.10 line when serving, and refresh the restore docs
gaurav 5e86aac
Activated publishing NameRes Loading on PR for testing.
gaurav 12f49c0
Size the load to the CPUs we actually have, and stop over-sizing the …
gaurav 95b79ba
Resize the data-loading pod and document how to tune it
gaurav 1c82b3d
Keep each stamp file on the volume whose state it describes
gaurav 541a913
Put the loading index on a node-local NVMe ephemeral volume
gaurav c43ca90
Give the loading pod 32 CPUs and 128Gi
gaurav c4cad93
Raise ramBufferSizeMB to 2G for the parallel load
gaurav 4434391
Cache the data-loading image build
gaurav ad0aa98
Point the loading pod at the PR image until this is released
gaurav b21ad58
Document that the pipeline ships inside the image
gaurav 69095b5
Updated Babel version to 2026jul22.
gaurav 2ad61ca
Cut the configset down to what NameRes actually chose
gaurav a9387ac
Take Solr from the official image instead of downloading it
gaurav 0ead597
Fix two bugs that would have failed the load at step 2 and step 3
gaurav 56d361f
Go back to a persistent Solr volume until the namespace can do NVMe
gaurav 92a3a28
Address review: backup ownership, additive-load guard, and doc drift
gaurav c563ded
Removed on:pull_request trigger.
gaurav b17e04b
Correct index-size rationale for 600Gi Solr PVC
gaurav b6458cc
Updated NameRes Data Loading tag to latest.
gaurav 210ff54
Pin queryResultCache to CaffeineCache
gaurav 60c1847
Tolerate whitespace when extracting Solr's numFound
gaurav 6ddb07d
Trim image-fingerprint checks from the k8s loading README
gaurav 6fc21a5
Speed up and simplify Solr data loading; fix /status for standalone (…
gaurav a48765b
Incremented version to v1.7.0.
gaurav 4072a54
Incremented version to v1.7.0 (#282)
gaurav 733ff78
Fix four broken relative links in documentation/
gaurav 9704650
Add documentation/Babel.md and route cross-repo links through it
gaurav 6774fc4
Correct documentation that had drifted from the code
gaurav b8ce87a
Sharpen two field descriptions against a live index
gaurav 2ff973c
Link the /lookup vs /synonyms divergences to issue #291
gaurav 84dcf3b
Merge the Translator Guide and /status conflation work
gaurav 223af96
Point every cross-repo link at main, not master
gaurav 814649f
Cover /status's conflation and environment reporting
gaurav 914da54
Fix the follow-ups this branch turned up
gaurav b8aff05
Refresh the documented /status example and keep it from drifting again
gaurav 6831d7d
Use the current org in release-note links, and ban the old one
gaurav File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,185 @@ | ||
| # NameRes Translator Guide | ||
|
|
||
| This guide is aimed at Translator developers and users who are integrating NameRes into their workflows. | ||
| It covers what to do when results are unexpected, how `/synonyms` (reverse-lookup) relates to NodeNorm, | ||
| and tips for improving performance. | ||
|
|
||
| ## What to do when a name lookup returns unexpected results | ||
|
|
||
| NameRes ranks results by a [Solr TF*IDF score](./Scoring.md) — the top result is the best *textual* match, | ||
| not necessarily the biologically intended concept. If the results don't look right, try these steps. | ||
|
|
||
| ### 1. Use `highlighting` to understand what matched | ||
|
|
||
| Set `highlighting=true` on a `/lookup` call to see which label or synonym drove the match: | ||
|
|
||
| ``` | ||
| GET /lookup?string=cold&highlighting=true&limit=5 | ||
| ``` | ||
|
|
||
| This tells you which synonym triggered the match, which helps diagnose why an unexpected concept ranked high. | ||
|
|
||
| ### 2. Filter by Biolink type | ||
|
|
||
| Use `biolink_type` to restrict results to the category you expect. Multiple types are combined with OR logic: | ||
|
|
||
| ``` | ||
| GET /lookup?string=cold&biolink_type=Disease&biolink_type=PhenotypicFeature | ||
| ``` | ||
|
|
||
| Common types: `Disease`, `Gene`, `ChemicalEntity`, `PhenotypicFeature`, `BiologicalProcess`, `AnatomicalEntity`. | ||
| Types can be specified with or without the `biolink:` prefix. | ||
|
|
||
| ### 3. Restrict to trusted prefixes | ||
|
|
||
| Use `only_prefixes` to limit results to a specific ontology, or `exclude_prefixes` to drop a noisy one. | ||
| Prefixes are pipe-separated and case-sensitive: | ||
|
|
||
| ``` | ||
| # Only MONDO disease identifiers | ||
| GET /lookup?string=diabetes&biolink_type=Disease&only_prefixes=MONDO | ||
|
|
||
| # Exclude UMLS (often produces many ambiguous matches) | ||
| GET /lookup?string=NIH&exclude_prefixes=UMLS | ||
| ``` | ||
|
|
||
| Common trusted prefixes by category: | ||
|
|
||
| | Category | Recommended prefixes | | ||
| |---|---| | ||
| | Disease | `MONDO`, `OMIM`, `ORPHANET` | | ||
| | Gene | `NCBIGene`, `HGNC` | | ||
| | Chemical/Drug | `CHEBI`, `DRUGBANK` | | ||
| | Phenotype | `HP`, `MP` | | ||
| | Anatomy | `UBERON`, `CL` | | ||
|
|
||
| ### 4. Filter by taxon for gene/protein queries | ||
|
|
||
| When searching for a gene or protein, results may include entries from multiple species. Use `only_taxa` | ||
| to restrict to a specific organism. The value is a pipe-separated list of NCBI Taxon CURIEs: | ||
|
|
||
| ``` | ||
| # Human genes only | ||
| GET /lookup?string=APOE&biolink_type=Gene&only_taxa=NCBITaxon:9606 | ||
|
|
||
| # Human and mouse | ||
| GET /lookup?string=APOE&only_taxa=NCBITaxon:9606|NCBITaxon:10090 | ||
| ``` | ||
|
|
||
| Common taxa: human `NCBITaxon:9606`, mouse `NCBITaxon:10090`, rat `NCBITaxon:10116`, zebrafish `NCBITaxon:7955`. | ||
|
|
||
| ### 5. Try autocomplete mode for partial strings | ||
|
|
||
| If your search string is a fragment of a name (e.g., typed by a user mid-word), set `autocomplete=true`. | ||
| This expands the final word with a wildcard so that `"diab"` matches `"diabetes"`, `"diabetic"`, etc.: | ||
|
|
||
| ``` | ||
| GET /lookup?string=diab&autocomplete=true&limit=5 | ||
| ``` | ||
|
|
||
| Without `autocomplete`, `"diab"` will only match documents that literally contain the token `"diab"`. | ||
|
|
||
| ### 6. If the correct concept is consistently missing | ||
|
|
||
| If your filtering is correct but the expected result never appears, the concept may be missing from the | ||
| Babel data that NameRes is built from. Consider filing an issue on: | ||
| - [NameRes GitHub](https://github.com/NCATSTranslator/NameResolution/issues) — for search/ranking problems | ||
| - [Babel GitHub](https://github.com/NCATSTranslator/Babel/issues) — for missing synonyms or identifiers | ||
|
|
||
| --- | ||
|
|
||
| ## Using `/synonyms` (reverse-lookup) vs. NodeNorm | ||
|
|
||
| These two services answer different questions. | ||
|
|
||
| ### Use `/synonyms` when you want to inspect synonyms for a known CURIE | ||
|
|
||
| The `/synonyms` endpoint returns all names and synonyms that NameRes knows for a given concept, along with | ||
| its Biolink types, taxa, and clique identifier count. This is useful for verifying synonym coverage or | ||
| debugging why a particular name did or did not match. | ||
|
|
||
| ``` | ||
| GET /synonyms?preferred_curies=NCBIGene:1756 | ||
| ``` | ||
|
|
||
| **Important:** `/synonyms` requires the *preferred* (normalized) CURIE. If you pass a non-preferred | ||
| identifier (e.g. a UniProtKB accession for a gene), you will get an empty result. Before calling | ||
| `/synonyms`, normalize your CURIE with NodeNorm (see below). | ||
|
|
||
| You can look up multiple CURIEs in one request: | ||
|
|
||
| ``` | ||
| GET /synonyms?preferred_curies=MONDO:0005148&preferred_curies=NCBIGene:1756 | ||
| ``` | ||
|
|
||
| ### Use NodeNorm when you need identifier normalization or equivalent identifiers | ||
|
|
||
| The [Node Normalization service](https://nodenormalization-sri.renci.org/) is the right tool when you need to: | ||
|
|
||
| - Convert a non-preferred identifier to its preferred CURIE | ||
| - Find all equivalent identifiers for a concept across ontologies | ||
| - Check which Biolink types a CURIE maps to | ||
| - Determine whether two CURIEs refer to the same concept | ||
|
|
||
| To normalize a CURIE before passing it to `/synonyms`, call NodeNorm with GeneProtein and DrugChemical | ||
| conflation enabled (to match the conflation used by NameRes): | ||
|
|
||
| ``` | ||
| GET https://nodenormalization-sri.renci.org/get_normalized_nodes?curie=UniProtKB:A0A0S2Z3B5&conflate=true&drug_chemical=true | ||
|
|
||
| ``` | ||
|
|
||
| The `id.identifier` field in the response is the preferred CURIE you can then pass to `/synonyms`. | ||
|
|
||
| ### Quick decision guide | ||
|
|
||
| | Question | Tool | | ||
| |---|---| | ||
| | What synonyms does NameRes know for this CURIE? | `/synonyms` | | ||
| | What is the preferred identifier for this concept? | NodeNorm | | ||
| | Are these two CURIEs equivalent? | NodeNorm | | ||
| | What Biolink types does this CURIE have? | NodeNorm | | ||
| | Why didn't a particular name match in `/lookup`? | `/synonyms` + `highlighting` | | ||
| | Which conflations are active in this NameRes deployment? | `/status` (`conflations` field) | | ||
|
|
||
| --- | ||
|
|
||
| ## Performance tips | ||
|
|
||
| ### Batch multiple queries with `/bulk-lookup` | ||
|
|
||
| Instead of making N separate `/lookup` calls, send them all in one POST request to `/bulk-lookup`. | ||
| It returns a dictionary keyed by input string: | ||
|
|
||
| ```json | ||
| POST /bulk-lookup | ||
| { | ||
| "strings": ["diabetes", "hypertension", "asthma"], | ||
| "limit": 5, | ||
| "biolink_types": ["Disease"] | ||
| } | ||
| ``` | ||
|
|
||
| This is significantly more efficient than sequential individual requests. | ||
|
|
||
| ### Add filters before processing results | ||
|
|
||
| Apply `biolink_type`, `only_prefixes`, and `only_taxa` at query time rather than filtering the response | ||
| yourself. Server-side filtering reduces the result set before it is serialized and transmitted. | ||
|
|
||
| ### Set `limit` to what you actually need | ||
|
|
||
| The default `limit` is 10 and the maximum is 1000. If you only need the top result, set `limit=1`. | ||
| If you need to page through a large result set, use `offset` for server-side pagination rather than | ||
| requesting a large `limit` and slicing client-side. | ||
|
|
||
| ### Cache results between Babel data releases | ||
|
|
||
| NameRes results are stable between Babel data releases (which happen a few times per year). If your | ||
| application calls NameRes repeatedly for the same input strings, cache the results locally. Check the | ||
| `/status` endpoint to detect when the Babel version changes and invalidate your cache accordingly: | ||
|
|
||
| ``` | ||
| GET /status | ||
| ``` | ||
|
|
||
| The `babel_version` field in the response changes with each data release. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.