Eurostat Data Import - Death by week and sex & Divorce indicators

statvar_imports/eurostat/deaths_by_week_and_sex/README.md

@@ -0,0 +1,51 @@
+# Eurostat Deaths By Week And Sex Import
+
+## Overview
+This dataset contains weekly death indicators at national and regional levels, sourced from Eurostat. The data tracks the total number of deaths broken down by sex and week across various European countries and regions.
+
+**type of place:** Country, NUTS Regions (Level 0-3)
+**years:** Historical data to present (2000-present, recorded weekly e.g., 2000-W01) 
+**place_resolution:** Resolved to DCIDs (e.g., dcid:country/FRA, dcid:nuts/AT113)
+
+## Data Source
+**Source URL:**
+https://ec.europa.eu/eurostat/databrowser/view/demo_r_mwk_ts/default/table?lang=en
+
+**Provenance Description:**
+The data is provided by Eurostat, the statistical office of the European Union. It belongs to the "Demography, population stock and balance" database under the "Deaths by week - special data collection" statistical theme, specifically the "Deaths by week and sex" (DEMO_R_MWK_TS) dataset.
+
+## Refresh Type
+Automatic Refresh
+
+The refresh is automated using the provided `run.sh` script, which handles both data download and processing.
+
+## How To Run Import
+To execute the complete import process (download and processing), run:
+```bash
+./run.sh
+
+
+### Script Details:
+- **Download**: Uses `curl` to fetch the latest SDMX-CSV data from Eurostat's dissemination API.
+- **Processing**: Uses `stat_var_processor.py` to map raw data to Data Commons StatVarObservations using the PV map and metadata configuration.
+
+## Key Files
+- `run.sh`: Main execution script for download and processing.
+- `deaths_by_week_and_sex_pvmap.csv`: Property-Value mapping for StatVar definitions and dimensions.
+- `deaths_by_week_and_sex_metadata.csv`: Configuration parameters for the processor.
+- `places_resolved.csv`: Mapping of place codes to Data Commons DCIDs.
+- `deaths_by_week_and_sex_output.csv`: Processed statistical observations.
+- `deaths_by_week_and_sex_output.tmcf`: Template MCF mapping the CSV columns to Data Commons schema.
+
+## Validation
+To validate the generated data, use the Data Commons import tool (lint mode):
+```bash
+java -jar datacommons-import-tool.jar lint deaths_by_week_and_sex_output.csv deaths_by_week_and_sex_output.tmcf
+```
+The resulting reports (`report.json`, `summary_report.html`) in `dc_generated/` provide detailed insights into data quality and validation status.
+
+## Testing
+Testing is performed using the `test_data` directory:
+- Raw Input: `test_data/deaths_by_week_and_sex_data_raw.csv`
+- Expected Output: `test_data/deaths_by_week_and_sex_output.csv`
+- Expected TMCF: `test_data/deaths_by_week_and_sex_output.tmcf`

statvar_imports/eurostat/deaths_by_week_and_sex/deaths_by_week_and_sex_metadata.csv

@@ -0,0 +1,2 @@
+parameter,value
+header_rows,1

statvar_imports/eurostat/deaths_by_week_and_sex/deaths_by_week_and_sex_pvmap.csv

@@ -0,0 +1,35 @@
+key,property1,value1,property2,value2,property3,value3,property4,value4
+# Dataflow mapping
+DATAFLOW,measuredProperty,dcs:count,populationType,dcs:Person,mortalityStatus,dcs:Deceased,statType,dcs:measuredValue
+
+# --- AUTOMATED CLEANUP: Ignored Columns ---
+LAST UPDATE,#ignore,"",,,,,,
+CONF_STATUS,#ignore,"",,,,,,
+
+# --- AUTOMATED CLEANUP: Global Geo Logic & Ignores ---
+geo:DE_TOT,#ignore,"",,,,,,
+geo:EA19,#ignore,"",,,,,,
+geo:EA20,#ignore,"",,,,,,
+geo:EA21,#ignore,"",,,,,,
+geo:EEA30_2007,#ignore,"",,,,,,
+geo:EEA31,#ignore,"",,,,,,
+geo:EFTA,#ignore,"",,,,,,
+geo:EU27_2007,#ignore,"",,,,,,
+geo:EU27_2020,#ignore,"",,,,,,
+geo:EU28,#ignore,"",,,,,,
+# Frequency dimension
+freq:W,observationPeriod,P1W,,,,,,
+# Sex dimension
+sex:T,gender,"",,,,,,
+sex:M,gender,dcs:Male,,,,,,
+sex:F,gender,dcs:Female,,,,,,
+# Unit dimension
+unit:NR,unit,dcs:Number,,,,,,
+# Geography dimension
+geo,observationAbout,{Data},,,,,,
+# Time Period - Using #Regex to match every cell value
+TIME_PERIOD,#Regex,"(?P<Year>[0-9]{4})-W(?P<Week>[0-9]{2})",observationDate,{Year},measurementMethod,dcs:EurostatWeekly_Week{Week}
+# Observation Value
+OBS_VALUE,value,{Number},,,,,,
+# Observation Flag (Status)
+OBS_FLAG:p,observationStatus,dcs:Provisional,,,,,,

statvar_imports/eurostat/deaths_by_week_and_sex/manifest.json

@@ -0,0 +1,26 @@
+{
+  "import_specifications": [
+    {
+      "import_name": "EuroStat_Deaths_by_week_and_sex",
+      "curator_emails": [
+        "support@datacommons.org"
+      ],
+      "provenance_url": "https://ec.europa.eu/eurostat/databrowser/view/demo_r_mwk_ts/default/table?lang=en",
+      "provenance_description": "This data is a voluntary, granular collection of weekly deaths by sex, age, and region across European countries from 2000 onward, established by Eurostat to monitor mortality trends during and after the COVID-19 pandemic.",
+      "scripts": [
+        "run.sh"
+      ],
+      "import_inputs": [
+        {
+          "template_mcf": "deaths_by_week_and_sex_output.tmcf",
+          "cleaned_csv": "deaths_by_week_and_sex_output.csv"
+        }
+      ],
+      "source_files": [
+        "input_files/*.csv"
+      ],
+      "cron_schedule": "5 1 1,15 * *",
+      "resource_limits": {"cpu": 4, "memory": 8, "disk":100}
+        }
+   ]
+}

statvar_imports/eurostat/deaths_by_week_and_sex/places_resolved.csv

@@ -0,0 +1,51 @@
+"place_name","dcid","placeId","wikidataId","name","alternateName","typeOf","containedInPlace"
+"AD","country/AND","","","","","",""
+"AL","country/ALB","","","","","",""
+"AM","country/ARM","","","","","",""
+"AT","country/AUT","","","","","",""
+"AZ","country/AZE","","","","","",""
+"BA","country/BIH","","","","","",""
+"BE","country/BEL","","","","","",""
+"BG","country/BGR","","","","","",""
+"BY","country/BLR","","","","","",""
+"CH","country/CHE","","","","","",""
+"CY","country/CYP","","","","","",""
+"CZ","country/CZE","","","","","",""
+"DE","country/DEU","","","","","",""
+"DK","country/DNK","","","","","",""
+"EE","country/EST","","","","","",""
+"EL","country/GRC","","","","","",""
+"ES","country/ESP","","","","","",""
+"FI","country/FIN","","","","","",""
+"FR","country/FRA","","","","","",""
+"FX","country/FXX","","","","","",""
+"GE","country/GEO","","","","","",""
+"HR","country/HRV","","","","","",""
+"HU","country/HUN","","","","","",""
+"IE","country/IRL","","","","","",""
+"IS","country/ISL","","","","","",""
+"IT","country/ITA","","","","","",""
+"LI","country/LIE","","","","","",""
+"LT","country/LTU","","","","","",""
+"LU","country/LUX","","","","","",""
+"LV","country/LVA","","","","","",""
+"MC","country/MCO","","","","","",""
+"MD","country/MDA","","","","","",""
+"ME","country/MNE","","","","","",""
+"MK","country/MKD","","","","","",""
+"MT","country/MLT","","","","","",""
+"NL","country/NLD","","","","","",""
+"NO","country/NOR","","","","","",""
+"PL","country/POL","","","","","",""
+"PT","country/PRT","","","","","",""
+"RO","country/ROU","","","","","",""
+"RS","country/SRB","","","","","",""
+"RU","country/RUS","","","","","",""
+"SE","country/SWE","","","","","",""
+"SI","country/SVN","","","","","",""
+"SK","country/SVK","","","","","",""
+"SM","country/SMR","","","","","",""
+"TR","country/TUR","","","","","",""
+"UA","country/UKR","","","","","",""
+"UK","country/GBR","","","","","",""
+"XK","country/XKX","","","","","",""

statvar_imports/eurostat/deaths_by_week_and_sex/run.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Step 1: Data Download
+mkdir -p input_files
+curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_R_MWK_TS/?format=SDMX-CSV&compressed=false" -o sdmx/Deaths_by_week_and_sex/deaths_by_week_and_sex_data_raw.csv
+
+# Step 2: Data Processing
+python3 ../../../tools/statvar_importer/stat_var_processor.py \
+  "--input_data=./input_files/*.csv" \
+  "--pv_map=./deaths_by_week_and_sex_pvmap.csv" \
+  "--config_file=./deaths_by_week_and_sex_metadata.csv" \
+  "--generate_statvar_name=True" \
+  "--skip_constant_csv_columns=False" \
+  "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \
+  "--output_path=./deaths_by_week_and_sex_output" \
+  "--places_resolved_csv=./places_resolved.csv" \
+  "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" \
+