Skip to content

feat(nowcast): detail IO loader router for Step 8 (Phase A) - #806

Open
bl-young wants to merge 2 commits into
nowcastfrom
by_nowcast_pipeline
Open

feat(nowcast): detail IO loader router for Step 8 (Phase A)#806
bl-young wants to merge 2 commits into
nowcastfrom
by_nowcast_pipeline

Conversation

@bl-young

@bl-young bl-young commented Sep 1, 2026

Copy link
Copy Markdown
Member

cc:

Related: #586

What changed? Why?

Step 8 does not add a new Cornerstone conversion module. Existing derive_cornerstone_* / cornerstone_disagg_pipeline already apply industry_corresp() @ V @ commodity_corresp().T. This PR adds the loader router and config so those paths can read either published BEA 2017 Detail tables or nowcast MUT artifacts (when wired).

Config (USAConfig)

  • usa_detail_io_source: 'bea_published' (default) | 'nowcast'
  • nowcast_mut_vintage: artifact build label; required when nowcast
  • usa_base_io_data_year: extended to 2012-2024; validated per source (bea_published -> 2012/2017 only; nowcast -> 2017-2024 and must equal model_base_year)
  • apply_io_year_adjustments forbidden when nowcast
  • usa_detail_original_year: returns usa_base_io_data_year on the nowcast path, else 2017

New modules

Rewired consumers (default bea_published -> same load_2017_* inputs as before)

  • derived_cornerstone.py
  • cornerstone_disagg_pipeline.py
  • cornerstone_bea_intermediates.py
  • derive_PRO_to_PUR_ratio.py

Scaling

  • derive_cornerstone_Aq_scaled returns derive_cornerstone_Aq() unchanged when usa_detail_io_source == 'nowcast' (detail IO is already at the IO calendar year; no summary-ratio scaling or PI inflation). Documented alongside the existing scale_a_matrix_with_useeio_method shortcut.

No production config enables nowcast yet. Default configs are bit-identical to pre-change behavior.

Follow-ups (Phase B+)

Merge note: #744 (jv_electricity_reaggregation) touches the same Cornerstone files for a different feature -- rebase that branch onto this one and re-apply load_detail_* imports at conflict sites.

Testing

uv run pytest bedrock/extract/iot/__tests__/test_detail_io.py -v

Router regression: under default bea_published, each load_detail_* matches the corresponding load_2017_*.

uv run black --check bedrock/extract/iot/detail_io.py bedrock/extract/iot/nowcast_mut_storage.py bedrock/extract/iot/__tests__/test_detail_io.py bedrock/utils/config/usa_config.py bedrock/transform/eeio/derived_cornerstone.py bedrock/transform/eeio/cornerstone_disagg_pipeline.py bedrock/transform/eeio/cornerstone_bea_intermediates.py bedrock/transform/iot/derive_PRO_to_PUR_ratio.py
uv run ruff check bedrock/extract/iot/detail_io.py bedrock/extract/iot/nowcast_mut_storage.py bedrock/extract/iot/__tests__/test_detail_io.py bedrock/utils/config/usa_config.py bedrock/transform/eeio/derived_cornerstone.py bedrock/transform/eeio/cornerstone_disagg_pipeline.py bedrock/transform/eeio/cornerstone_bea_intermediates.py bedrock/transform/iot/derive_PRO_to_PUR_ratio.py
uv run mypy bedrock/extract/iot/detail_io.py bedrock/extract/iot/nowcast_mut_storage.py bedrock/extract/iot/__tests__/test_detail_io.py bedrock/utils/config/usa_config.py bedrock/transform/eeio/derived_cornerstone.py bedrock/transform/eeio/cornerstone_disagg_pipeline.py bedrock/transform/eeio/cornerstone_bea_intermediates.py bedrock/transform/iot/derive_PRO_to_PUR_ratio.py

Introduce usa_detail_io_source and load_detail_* routing so Cornerstone
derivation can switch between published BEA 2017 tables and nowcast MUT
artifacts. Default bea_published preserves existing behavior; nowcast
loaders raise NotImplementedError until GCS storage lands (#592).

Extend usa_base_io_data_year validators, add nowcast_mut_vintage, and
bypass year-scaling in derive_cornerstone_Aq_scaled when source is
nowcast. Rewire derived_cornerstone, cornerstone_disagg_pipeline,
cornerstone_bea_intermediates, and derive_PRO_to_PUR_ratio.
Comment thread bedrock/utils/config/usa_config.py Outdated

@WesIngwersen WesIngwersen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great one small suggested change to clarify it's DETAIL years in that param - needs to be propagated (i just noted it in one place)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants