Add RDAS_RDOMICS project

RDAS_RDOMICS/.gitignore

@@ -0,0 +1,28 @@
+# Python
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.mypy_cache/
+.venv/
+venv/
+.ipynb_checkpoints/
+
+# Environment and local overrides
+.env
+scripts/config/paths.local.yaml
+
+# Logs
+logs/
+*.log
+scripts/sample_characteristics_harmonization/logs/
+
+# macOS
+.DS_Store
+
+# Generated pipeline outputs
+scripts/data/geo_matrix_files/
+scripts/data/geo_final_tables/
+scripts/data/geo_final_tables_testing/
+scripts/data/node_csv_files/
+scripts/data/node_json_files/
+scripts/sample_characteristics_harmonization/sample_characteristics_harmonization_output/sample_import_resume.json

RDAS_RDOMICS/README.md

@@ -0,0 +1,60 @@
+# RDAS_RDOMICS
+
+RDAS_RDOMICS is a rare-disease omics data extraction and graph-construction pipeline centered on GEO studies. The project takes disease-driven queries, gathers matching GEO accessions, extracts experiment and sample metadata, normalizes selected fields, and reshapes the results into graph-ready node and relationship tables that can be imported into Neo4j or Memgraph.
+
+The codebase supports two connected workflows. The first is the main RD-OMICS build pipeline, which moves from GEO search through metadata extraction, graph table generation, graph import, and experiment-property normalization. The second is a downstream sample-characteristics harmonization workflow that cleans noisy sample labels, applies rule-based and LLM-assisted grouping, and writes harmonized sample properties back onto graph `Sample` nodes.
+
+At a high level, the repository does the following:
+
+- searches GEO with rare-disease terms and records matched GSE studies
+- extracts study, platform, project, publication, and sample metadata from GEO/NCBI pages
+- converts extracted tables into graph node files and relationship mapping files
+- imports graph-ready exports into Neo4j or Memgraph
+- normalizes experiment assay fields for cleaner graph properties
+- harmonizes sample-characteristics labels into curated categories and subcategories
+
+## Repository Layout
+
+- `scripts/`: core pipeline scripts plus sample-characteristics harmonization utilities
+- `scripts/config/paths.yaml`: tracked default config with repo-relative paths
+- `scripts/data/`: small source inputs and templates that the pipeline reads; large generated graph outputs are intentionally not versioned here
+- `paper_materials/`: figures and supplementary files for the paper
+- `SCHEMA.md`: graph labels, identifiers, and relationship types
+
+## Setup
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+Optional LLM/HPC workflow:
+
+```bash
+pip install -r requirements-llm.txt
+```
+
+The LLM harmonization scripts under `scripts/sample_characteristics_harmonization/` assume a GPU-capable environment and local access to a vLLM-compatible model.
+
+## Configuration
+
+The default config lives at `scripts/config/paths.yaml` and now uses repo-relative paths. You can point the pipeline at a different config file with `--config` or by setting `RDAS_RDOMICS_CONFIG`.
+
+For the optional LLM steps, set `settings.llm_model_path` in the config or export `RDAS_LLM_MODEL_PATH`.
+For NCBI Entrez calls, set `ENTREZ_EMAIL` in your environment instead of editing source files.
+
+Example:
+
+```bash
+python -m scripts.main --step3-download-gse-number --step4-extract-to-table
+python -m scripts.main --step5-generate-node-mappings --step6-import-to-neo4j
+python -m scripts.main --step7-experiment-normalization
+python -m scripts.main --step7-1-import-normalized-experiment-properties
+```
+
+## Pipeline Order
+
+1. Core extraction and graph-building pipeline: `step1` through `step7.1` in `scripts/`
+2. Sample-characteristics harmonization: the scripts under `scripts/sample_characteristics_harmonization/`
+

RDAS_RDOMICS/SCHEMA.md

@@ -0,0 +1,34 @@
+# RD-OMICS Graph Schema
+
+This repository builds a graph-oriented export with six node types and six relationship types. The definitions below are inferred from `scripts/step6_import_to_neo4j.py`.
+
+## Node Labels
+
+| Node file | Graph label | Primary identifier |
+| --- | --- | --- |
+| `publication_node.csv` | `Publication` | `Pubmed_id` |
+| `project_node.csv` | `Project` | `Project_id` |
+| `experiment_node.csv` | `Experiment` | `Experiment_id` |
+| `sample_node.csv` | `Sample` | `Sample_id` |
+| `platform_node.csv` | `Platform` | `Platform_id` |
+| `condition_node.csv` | `Condition` | `GardId` |
+
+## Relationship Types
+
+The importer derives relationship type names from mapping filenames by uppercasing the basename without `_mapping`.
+
+| Mapping file | Start node | End node | Relationship type |
+| --- | --- | --- | --- |
+| `sample_platform_mapping.csv` | `Sample` | `Platform` | `SAMPLE_PLATFORM` |
+| `condition_project_mapping.csv` | `Condition` | `Project` | `CONDITION_PROJECT` |
+| `publication_project_mapping.csv` | `Publication` | `Project` | `PUBLICATION_PROJECT` |
+| `project_experiment_mapping.csv` | `Project` | `Experiment` | `PROJECT_EXPERIMENT` |
+| `experiment_platform_mapping.csv` | `Experiment` | `Platform` | `EXPERIMENT_PLATFORM` |
+| `sample_experiment_mapping.csv` | `Sample` | `Experiment` | `SAMPLE_EXPERIMENT` |
+
+## Pipeline Notes
+
+- `scripts/step5_generate_node_mappings.py` creates the node CSVs and relationship mapping CSVs.
+- `scripts/step6_import_to_neo4j.py` converts those CSVs to JSON and imports them into Memgraph/Neo4j.
+- `scripts/step7_experiment_normalization.py` adds normalized assay fields to `Experiment`.
+- `scripts/sample_characteristics_harmonization/7_import_to_sample_node.py` adds harmonized sample property fields to `Sample`.

RDAS_RDOMICS/paper_materials/supplementary_files/Supp.file_4_Category_reference_information.csv

@@ -0,0 +1,11 @@
+standarized_term,synonym,Example_entrys,Definition
+External_sample_id,"patient_id, patient no, patient id, patient', Patient, patients, patient id, patientid, patient_id, patient id no","50,371,018",internal IDs for the patients involved in clinical trials. This id should be included as external sample id only.
+Biospecimen Organism,"species, cell organism, host organism, organism_status, source_organism, microorganism_source_of_infection, microorganism_blood","Homo sapiens, Mus musculus, Danio rerio",Any living biological system (from bacteria to plants to humans)
+Biospecimen Type,"sample type, Specimen type, type of sample, specimen type, physical sample type, specimen_type, Sample Type, Type of Biospecimen, Specimen Type, Biospecimen Type, physical_sample_type, SPEC, tissue type, cell lline, tissue location, cell type, cell name, organismpart, tissue anatomic site, Tissue type, name of tissue, tissue origin, tumor tissue source, tissue/cell info, tissue/cell type, cell line source tissue, cell line/tissue source, tissue location, brain tissue, tissue zone","blood, cell, tissue, feces, Cerebrospinal fluid, stem cell, iPSC, NK cell, liver tissue, brain tissue, red blood cell","The type of a material sample taken from a biological entity for testing, diagnostic, propagation, treatment or research purposes. This includes particular types of cellular molecules, cells, tissues, organs, body fluids, embryos, and body excretory substances. This category also include sub-type of biospecimen, such as cells, tissues, fluids"
+Biospecimen Age,"age, sample collection time, sample age, donor age, cell culture time, age in year, age(year), age of onset, age_weeks, age patient, subject age, age in yrs, age at draw, age at diagnosis","30 years old,24 hours",the age at the time of the collection of biospecimen. Years / months for human donor and hours / days for cell culture 
+Biospecimen Sex,"sex, gender, sugject gender, gender source, patient gender, source patient gender, donor gender, cell line source gender, sex/age, donor sex, animal sex, detal sex, biological sex, SEX, infantSex, sext of nodor mice","male, female",the sex at birth of the donor if the biospecimen is collected from a human donors or cell lines are derived from human tissues
+Biospecimen Race,"race, race/ethnicity, donor race, Ethnicity, self reported Ethnicity, donor ethnicity, race_ethinicity","Caucasian, Asian American, Hispanic, Latin-Americans",the race of the donor if the biospecimen is collected from human donors or cell lines are derived from human tissues. the Ethnicity of the donor if the biospecimen is collected from human donors or cell lines are derived from human tissues
+Biospecimen Disease Condition,"condition, disease status, diagnosis, cohort, stage, subject status, patient status, developmental stage, primary disease, disease classification, diesase group, stage of disease, disease subtpe, lung diseaes, disease condition, disease phase, disease subset, patient condition, sample condition, clinical condition, participant condition","disease (such as cancer), control, healthy, healthy contol, pseudo surgery, simulattion",the disease condition of the donor or other samples
+Treatment,"treatment, intervention, treatment condition, culture condition, growth condition, labeling condition, treatment type, treatment/group, genotype/treatment, culture/treatment, no treatment, treatment, active ingredient, drug treatment, antibody treatment, chemical treatment, treatment agent, treatment/agent, compound treatment, treatment term name, polarization treatment, growth medium/treatment, surgical treatment category, treatment art, treatment condition/type","treatment, placebo, oberservation, drug name, placebo, saline","the treatment group of specimen donor or cell line or animial models, and the name of the treatment"
+Treatment Dosage Regimen,"dosage, treatment description, treatment protocol, treatment dose, treatment_dose, treatment tming, time of drug treatment, days under treatment, duration of treatment, days under treatment, infectious_dose, dose group, vaccine doses, drug dose, lung radiation dose,  treatment dose, drug concentration, drug administration, agent concentration","treatment duration, treatment time, concentration","A Dosage Regimen is the schedule of doses of a medicine outlining the instructions for its safe and effective administration. It includes dose size frequency of administration time interval between administrations duration of treatment administration instructions route of administration dosage form and dosage modifications."
+Other,,,Other labels that do not belong to any other categories. 

RDAS_RDOMICS/requirements-llm.txt

@@ -0,0 +1,2 @@
+-r requirements.txt
+vllm

RDAS_RDOMICS/requirements.txt

@@ -0,0 +1,10 @@
+beautifulsoup4
+biopython
+neo4j
+nltk
+numpy
+openpyxl
+pandas
+PyYAML
+requests
+tqdm

RDAS_RDOMICS/scripts/__init__.py

@@ -0,0 +1 @@
+"""RDAS_RDOMICS pipeline scripts."""

RDAS_RDOMICS/scripts/check_unique_comb/check_amount_of_unique_normalized_combo.py

@@ -0,0 +1,68 @@
+import pandas as pd
+import numpy as np
+from pathlib import Path
+import sys
+
+CURRENT_DIR = Path(__file__).resolve().parent
+SCRIPTS_DIR = CURRENT_DIR.parent
+if str(SCRIPTS_DIR) not in sys.path:
+    sys.path.insert(0, str(SCRIPTS_DIR))
+
+from utils import load_paths
+
+paths = load_paths()
+df = pd.read_csv(paths['experiment_normalization_rules'])
+
+print("=== Dataset Overview ===")
+print(f"Total rows in dataset: {len(df)}")
+print(f"Dataset shape: {df.shape}")
+print()
+
+print("=== Column Information ===")
+print("Columns in the dataset:")
+for col in df.columns:
+    print(f"  - {col}")
+print()
+
+print("=== Unique Values Analysis ===")
+print(f"Unique values in 'Omics_type_norm': {df['Omics_type_norm'].nunique()}")
+print(f"Unique values in 'Sequencing_library_norm': {df['Sequencing_library_norm'].nunique()}")
+print()
+
+print("=== Unique Combinations ===")
+# Get unique combinations of the two columns
+unique_combinations = df[['Omics_type_norm', 'Sequencing_library_norm']].drop_duplicates()
+print(f"Total unique combinations of 'Omics_type_norm' and 'Sequencing_library_norm': {len(unique_combinations)}")
+print()
+
+print("=== All Unique Combinations ===")
+print("Omics_type_norm | Sequencing_library_norm")
+print("-" * 50)
+for idx, row in unique_combinations.iterrows():
+    omics_type = row['Omics_type_norm']
+    seq_lib = row['Sequencing_library_norm']
+    # Handle NaN values
+    if pd.isna(seq_lib):
+        seq_lib = "NaN/Empty"
+    print(f"{omics_type} | {seq_lib}")
+
+print()
+print("=== Frequency of Each Combination ===")
+combination_counts = df.groupby(['Omics_type_norm', 'Sequencing_library_norm']).size().reset_index(name='count')
+combination_counts = combination_counts.sort_values('count', ascending=False)
+print("Omics_type_norm | Sequencing_library_norm | Count")
+print("-" * 60)
+for idx, row in combination_counts.iterrows():
+    omics_type = row['Omics_type_norm']
+    seq_lib = row['Sequencing_library_norm']
+    count = row['count']
+    # Handle NaN values
+    if pd.isna(seq_lib):
+        seq_lib = "NaN/Empty"
+    print(f"{omics_type} | {seq_lib} | {count}")
+
+print()
+print("=== Summary Statistics ===")
+print(f"Most common combination appears {combination_counts['count'].max()} times")
+print(f"Least common combination appears {combination_counts['count'].min()} times")
+print(f"Average frequency per combination: {combination_counts['count'].mean():.2f}")

RDAS_RDOMICS/scripts/check_unique_comb/check_unique_omics_type_comb.py

@@ -0,0 +1,25 @@
+import pandas as pd
+from pathlib import Path
+import sys
+
+CURRENT_DIR = Path(__file__).resolve().parent
+SCRIPTS_DIR = CURRENT_DIR.parent
+if str(SCRIPTS_DIR) not in sys.path:
+    sys.path.insert(0, str(SCRIPTS_DIR))
+
+from utils import load_paths
+
+paths = load_paths()
+df = pd.read_csv(f"{paths['node_csv_files']}/experiment_node.csv")
+
+# Select the relevant columns
+df_subset = df[['Omics_type', 'Sequencing_type', 'Sequencing_library']]
+
+# Get unique combinations, including rows with NaN in either column
+unique_combinations = df_subset.drop_duplicates()
+
+# Sort (optional)
+unique_combinations = unique_combinations.sort_values(by=['Omics_type', 'Sequencing_type', 'Sequencing_library'])
+
+# Save to CSV
+unique_combinations.to_csv("omics_sequencing_combinations2.csv", index=False)

RDAS_RDOMICS/scripts/check_unique_comb/check_unique_sample_characteristics.py

@@ -0,0 +1,34 @@
+import pandas as pd
+from collections import Counter
+from pathlib import Path
+import sys
+
+CURRENT_DIR = Path(__file__).resolve().parent
+SCRIPTS_DIR = CURRENT_DIR.parent
+if str(SCRIPTS_DIR) not in sys.path:
+    sys.path.insert(0, str(SCRIPTS_DIR))
+
+from utils import load_paths
+
+paths = load_paths()
+df = pd.read_csv(f"{paths['node_csv_files']}/sample_node.csv")
+
+# Initialize a counter for keys
+key_counter = Counter()
+
+# Iterate through the 'Sample_characteristics' column
+for entry in df['Sample_characteristics'].dropna():
+    for item in entry.split(';'):
+        item = item.strip()
+        if ':' in item:
+            key = item.split(':', 1)[0].strip()
+            key_counter[key] += 1
+
+# Convert to DataFrame
+key_df = pd.DataFrame(key_counter.items(), columns=['name', 'count'])
+
+# Sort by count descending (optional)
+key_df = key_df.sort_values(by='count', ascending=False)
+
+# Save to CSV
+key_df.to_csv("sample_characteristics_key_count.csv", index=False)