VarFish DB Downloader

The purpose of this repository is to collect various data form public sources that is eventually used in VarFish for annotation and display to the user. This repository contains a Snakemake workflow with supporting code for downloading the data and preparing it for being used with VarFish.

Quick Facts

• License: MIT
• Programming Language: Python / Snakemake

Running

Use the utility rule help to get a list of all available rules:

# snakemake --cores=1 help

Run them all with all:

# snakemake --cores=1 all

Note that this will take a long time, use a lot of disk space, and download a lot of data.

To run on a Slurm cluster, you can use the Snakemake --slurm option. See run-slurm.sh for an example.

Development Setup

Prerequisites: Install mamba for Conda Package Management

Install conda, ideally via miniforge. A quickstart:

# wget -O /tmp/Mambaforge-Linux-x86_64.sh \
    https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh
# bash /tmp/Mambaforge-Linux-x86_64.sh -b -p ~/mambaforge3 -s
# source ~/mambaforge3/bin/activate

Clone Project

# git clone git@github.com:bihealth/varfish-db-downloader.git
# cd varfish-db-downloader

Setup Environment and Install Tools

This will setup the conda environment:

# mamba env create --file environment.yml
# conda activate varfish-db-downloader

This will install the varfish-db-downloader tools:

# pip install -e .

Developer Rules

Download Commands

We use wget and aria2c only and not curl. The rationale is that for the test mode, we are overriding the two executables with helper commands.

Development Subsets

Besides the full output, we also build a subset of the data suitable for development. At the moment of writing, the subset is to the BRCA1 gene only. The rationale is that this gene and its variants are heavily annotated as breast cancer predisposition screening is a common task and users/data is plenty.

annonars from its Docker image

Outside of CI test mode, the workflow runs annonars from the Docker image ghcr.io/varfish-org/annonars:0.45.2 via Apptainer instead of the bioconda package (annonars-image-bin/annonars, put first on PATH by the Snakefile). The image links RocksDB 8.9, which writes table format_version 5. The bioconda build bundles RocksDB 11.8.1, which writes format_version 7, and readers with RocksDB < 10.4 (e.g. varfish-server-worker 0.17) cannot open that.

• Apptainer must be available on the nodes that run the jobs.
• Fill the Apptainer cache once before starting the workflow, so that the jobs do not convert the image in parallel:

  # apptainer exec docker://ghcr.io/varfish-org/annonars:0.45.2 annonars --version
  

• Paths outside the working directory and $HOME must be bound, e.g. with export APPTAINER_BIND=/data02.
• Check the format of built databases; databases above format_version 6 have to be rebuilt:

  # varfish-db-downloader rocksdb check-format output/full/annonars/*/rocksdb output/full/mehari/freqs-*/rocksdb
  

Running in Test Mode

TODO: This part is currently disabled in CI (Run-Test-Mode in .github/workflows/main.yml). TODO: pre-mehari workflow files are missing.

The download of files can be disabled to enable a test mode. Instead, the files in excerpt-data are used when CI=true is set in the environment.

This is done by overriding the download executables wget and aria2 in the Snakemake file when CI=true has been set. This again is done by overriding the PATH environment variable.

The files can be updated by calling

# varfish-db-downloader wget urls-download

The known URLs are managed in download_urls.yml.

Manually maintained excerpts

Some files have to be put manually in the excerpt-data folder. Some need to be reduced in size.

• dbNSFP(a): https://usf.box.com/shared/static/2hzcx5s6p1xui7oen16xqzndfrkt8l9l
• dbNSFP(c): https://usf.box.com/shared/static/03xsrpna0nzgrytfo2pzk326t8jad4oc
• dbscSNV: https://usf.box.com/shared/static/ffwlywsat3q5ijypvunno3rg6steqfs8
• https://storage.googleapis.com/public-download-files/hgnc/archive/archive/quarterly/json/hgnc_complete_set_2026-10-06.json
• https://storage.googleapis.com/adult-gtex/bulk-gex/v8/rna-seq/GTEx_Analysis_2017-06-05_v8_RNASeQCv1.1.9_gene_tpm.gct.gz
• https://github.com/varfish-org/clinvar-data-jsonl/releases/download/clinvar-weekly-20260104/clinvar-data-extract-vars-20260104+0.18.5.tar.gz
  • Extract the tar.gz, take the first 1000 lines of the containing files and zip them again.
• https://ftp.ncbi.nlm.nih.gov/asn1-converters/by_program/gene2xml/linux64.gene2xml.gz
• https://ftp.ncbi.nih.gov/gene/DATA/ASN_BINARY/Mammalia/Homo_sapiens.ags.gz
• https://github.com/SACGF/cdot/releases/download/data_v0.2.36/cdot-0.2.36.Homo_sapiens_GRCh37_RefSeq_105.20220307.gff.json.gz
• https://github.com/SACGF/cdot/releases/download/data_v0.2.36/cdot-0.2.36.Homo_sapiens_GRCh38_RefSeq_RS_2025_08.gff.json.gz

Manually downloaded data

Some source files can't be downloaded by the pipeline and have to be downloaded manually and committed to the data folder.

• MITOMAP: https://mitomap.org/cgi-bin/polymorphisms.cgi?format=vcf
  • The URL is behind a Cloudflare browser check, so wget gets HTTP 403. Download the file in a browser.
  • Save it as data/mitomap/<fileDate>/polymorphisms.vcf, where <fileDate> is the ##fileDate from the VCF header.
  • Set mitomap in varfish_db_downloader/versions.py to the same <fileDate>.

Managing GitHub Project with Terraform

# export GITHUB_OWNER=bihealth
# export GITHUB_TOKEN=ghp_<thetoken>

# cd utils/terraform
# terraform init
# terraform import github_repository.varfish-db-downloader varfish-db-downloader

# terraform validate
# terraform fmt
# terraform plan
# terraform apply

Uploading Data to S3

For example, as follows

# s5cmd --dry-run --profile ext-varfish-public \
    --endpoint-url https://ceph-s3-ext.cubi.bihealth.org \
    sync \
    'output/full/mehari/genes-txs-grch3*' \
    s3://varfish-public/public/

Semantic Commits

Generally, follow Semantic Commits v1, also see examples.

Here is a list of the commit message prefixes that we use:

prefix	description
feat	Features
fix	Bug Fixes
perf	Performance Improvements
deps	Dependencies
revert	Reverts
docs	Documentation
style	Styles
chore	Miscellaneous Chores
refactor	Code Refactoring
test	Tests
build	Build System
ci	Continuous Integration