WHO Tuberculosis PT Data Import

statvar_imports/tuberculosis_preventive_treatment/README.md

@@ -0,0 +1,71 @@
+<<<<<<< HEAD
+# WHO Tuberculosis: Percentage of household contacts (or all close contacts) who were started on TB preventive treatment out of those eligible
+=======
+# WHO Tuberculosis: Percentage of household contacts (or all close contacts) who were started on TB preventive treatment out of those eligible
+>>>>>>> 52fd568515958ab33ac62e11b3b07a03ae2933d3
+
+## Overview
+This dataset provides the percentage of household contacts (or close contacts) of people diagnosed with a new episode of bacteriologically confirmed pulmonary TB disease who started TB preventive treatment, out of those eligible.
+
+## Data Source
+
+**Source URL:**
+https://data.who.int/indicators/i/45274BD/F5556F8
+
+The data comes from the official WHO reporting database and includes comprehensive, country-level health metrics detailing annual Tuberculosis notifications and case classifications.
+
+## How To Download Input Data
+To download the data, you'll need to run the provided download script `tb_data_download_who.py`. This script automatically queries the WHO API for the indicator, merges it with the WHO geographical master list to append standard `iso3` country codes, and saves the cleaned `Tuberculosis_preventive_treatment_input.csv` file inside an "input_files" folder.
+
+type of place: Country.
+
+statvars: Health / Tuberculosis.
+
+years: 2010 to 2022
+
+place_resolution: manually.
+
+release_frequency: P1Y
+
+## Processing Instructions
+To process the WHO Tuberculosis data and generate statistical variables, use the following commands from your root `data` directory:
+
+**Download input file**
+```bash	
+python3 statvar_imports/tuberculosis_preventive_treatment/tb_data_download_who.py
+```
+
+**For Test Data Run**
+```bash
+	python3 tools/statvar_importer/stat_var_processor.py \
+<<<<<<< HEAD
+  --input_data="statvar_imports/tuberculosis_preventive_treatment/testdata/Tuberculosis_preventive_treatment.csv" \
+  --pv_map="statvar_imports/tuberculosis_preventive_treatment/tuberculosis_PreventiveTreatment_pvmap.csv" \
+  --output_path="statvar_imports/tuberculosis_preventive_treatment/output_files/tuberculosis_PreventiveTreatment" \
+  --config_file="statvar_imports/tuberculosis_preventive_treatment/tuberculosis_PreventiveTreatment_metadata.csv" \
+  --existing_statvar_mcf="gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+```
+=======
+  --input_data="statvar_imports/tuberculosis_preventive_treatment/testdata/Tuberculosis_preventive_treatment_input.csv" \
+  --pv_map="statvar_imports/tuberculosis_preventive_treatment/tuberculosis_PreventiveTreatment_pv_mapping.csv" \
+  --output_path="statvar_imports/tuberculosis_preventive_treatment/output_files/tuberculosis_PreventiveTreatment" \
+  --config_file="statvar_imports/tuberculosis_preventive_treatment/tuberculosis_PreventiveTreatment_metadata.csv" \
+  --existing_statvar_mcf="gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+>>>>>>> 52fd568515958ab33ac62e11b3b07a03ae2933d3
+
+**For Main data run**
+```bash
+	python3 tools/statvar_importer/stat_var_processor.py \
+<<<<<<< HEAD
+  --input_data="statvar_imports/tuberculosis_preventive_treatment/input_files/Tuberculosis_preventive_treatment.csv" \
+  --pv_map="statvar_imports/tuberculosis_preventive_treatment/tuberculosis_PreventiveTreatment_pvmap.csv" \
+=======
+  --input_data="statvar_imports/tuberculosis_preventive_treatment/source_files/Tuberculosis_preventive_treatment.csv" \
+  --pv_map="statvar_imports/tuberculosis_preventive_treatment/tuberculosis_PreventiveTreatment_pv_mapping.csv" \
+>>>>>>> 52fd568515958ab33ac62e11b3b07a03ae2933d3
+  --output_path="statvar_imports/tuberculosis_preventive_treatment/output_files/tuberculosis_PreventiveTreatment" \
+  --config_file="statvar_imports/tuberculosis_preventive_treatment/tuberculosis_PreventiveTreatment_metadata.csv" \
+  --existing_statvar_mcf="gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+```
+
+#### Refresh type: Fully Autorefresh

statvar_imports/tuberculosis_preventive_treatment/manifest.json

@@ -0,0 +1,30 @@
+{
+    "import_specifications": [
+        {
+            "import_name": "WHO_TuberculosisPreventiveTreatment",
+            "curator_emails": [
+                "support@datacommons.org"
+            ],
+            "provenance_url": "https://data.who.int/indicators/i/45274BD/F5556F8",
+            "provenance_description": "Tuberculosis: Percentage of household contacts (or all close contacts) who were started on TB preventive treatment out of those eligible",
+            "scripts": [
+                "tb_data_download_who.py",
+<<<<<<< HEAD
+                "../../../tools/statvar_importer/stat_var_processor.py --input_data=input_files/Tuberculosis_preventive_treatment.csv --pv_map=tuberculosis_PreventiveTreatment_pvmap.csv --config_file=tuberculosis_PreventiveTreatment_metadata.csv --output_path=output/tuberculosis_preventive_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+=======
+                "../../../tools/statvar_importer/stat_var_processor.py --input_data=input_files/Tuberculosis_preventive_treatment.csv --pv_map=tuberculosis_PreventiveTreatment_pv_mapping.csv --config_file=tuberculosis_PreventiveTreatment_metadata.csv --output_path=output/tuberculosis_preventive_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+>>>>>>> 52fd568515958ab33ac62e11b3b07a03ae2933d3
+            ],
+            "import_inputs": [
+                {
+                    "template_mcf": "output/tuberculosis_preventive_output.tmcf",
+                    "cleaned_csv": "output/tuberculosis_preventive_output.csv"
+                }
+            ],
+            "source_files": [
+                "input_files/Tuberculosis_preventive_treatment.csv"
+            ],
+            "cron_schedule": "0 10 10,21 * *"
+        }
+    ]
+}

statvar_imports/tuberculosis_preventive_treatment/tb_data_download_who.py

@@ -0,0 +1,82 @@
+import os
+import requests
+import io
+import pandas as pd
+import logging
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+
+def download_tb_percentage_data():
+    # 1. Get the Clean Data from the API using the new Indicator ID
+    api_url = "https://xmart-api-public.who.int/DATA_/RELAY_TB_DATA"
+    params = {
+        "$filter": "IND_ID eq '45274BDF5556F8'",
+        "$select": "IND_ID,INDICATOR_NAME,YEAR,COUNTRY,DISAGGR_1,VALUE",
+        "$format": "csv"
+    }
+
+    logging.info("1. Fetching clean percentage data from WHO API...")
+<<<<<<< HEAD
+    try:
+        api_response = requests.get(api_url, params=params)
+        api_response.raise_for_status()
+    except requests.exceptions.RequestException as e:
+        logging.error(f"Failed to fetch API data: {e}")
+=======
+    api_response = requests.get(api_url, params=params)
+
+    if api_response.status_code != 200:
+        logging.error(f"Failed to fetch API data. HTTP {api_response.status_code}")
+>>>>>>> 52fd568515958ab33ac62e11b3b07a03ae2933d3
+        return
+
+    # Load the clean API data into a pandas table
+    api_df = pd.read_csv(io.StringIO(api_response.text))
+
+    # 2. Get ONLY the iso3 code from the master database
+    logging.info("2. Fetching country iso3 codes from WHO master database...")
+    master_url = "https://extranet.who.int/tme/generateCSV.asp?ds=notifications"
+<<<<<<< HEAD
+    try:
+        master_response = requests.get(master_url)
+        master_response.raise_for_status()
+    except requests.exceptions.RequestException as e:
+        logging.error(f"Failed to fetch master data: {e}")
+=======
+    master_response = requests.get(master_url)
+    if master_response.status_code != 200:
+        logging.error(f"Failed to fetch master data. HTTP {master_response.status_code}")
+>>>>>>> 52fd568515958ab33ac62e11b3b07a03ae2933d3
+        return
+
+    # We only pull the 'country' (for matching) and 'iso3' columns
+    geo_columns = ['country', 'iso3']
+    master_df = pd.read_csv(io.StringIO(master_response.text), usecols=geo_columns).drop_duplicates()
+
+    # 3. Merge the two datasets together based on the country name
+    logging.info("3. Merging data and formatting...")
+    # The API uses uppercase 'COUNTRY', the master uses lowercase 'country'
+    merged_df = pd.merge(api_df, master_df, left_on='COUNTRY', right_on='country', how='left')
+
+    # Drop the duplicate lowercase 'country' column used for joining
+    merged_df = merged_df.drop(columns=['country'])
+
+    # Reorder columns so the iso3 code sits right next to the Country name
+    final_columns = [
+        'IND_ID', 'INDICATOR_NAME', 'YEAR', 'COUNTRY', 'iso3','DISAGGR_1', 'VALUE'
+    ]
+    merged_df = merged_df[final_columns]
+
+    # 4. Save to CSV in a new folder
+    output_dir = "statvar_imports/tuberculosis_preventive_treatment/input_files"
+    filename = os.path.join(output_dir, "Tuberculosis_preventive_treatment.csv")
+
+    os.makedirs(output_dir, exist_ok=True)
+
+    # Save without the pandas index column
+    merged_df.to_csv(filename, index=False)
+    logging.info(f"Success! Data saved locally as '{filename}'")
+
+if __name__ == "__main__":
+    download_tb_percentage_data()