feat(nowcast): detail IO loader router for Step 8 (Phase A) - #806
Open
bl-young wants to merge 2 commits into
Open
feat(nowcast): detail IO loader router for Step 8 (Phase A)#806bl-young wants to merge 2 commits into
bl-young wants to merge 2 commits into
Conversation
Introduce usa_detail_io_source and load_detail_* routing so Cornerstone derivation can switch between published BEA 2017 tables and nowcast MUT artifacts. Default bea_published preserves existing behavior; nowcast loaders raise NotImplementedError until GCS storage lands (#592). Extend usa_base_io_data_year validators, add nowcast_mut_vintage, and bypass year-scaling in derive_cornerstone_Aq_scaled when source is nowcast. Rewire derived_cornerstone, cornerstone_disagg_pipeline, cornerstone_bea_intermediates, and derive_PRO_to_PUR_ratio.
WesIngwersen
reviewed
Sep 1, 2026
WesIngwersen
requested changes
Sep 1, 2026
WesIngwersen
left a comment
Member
There was a problem hiding this comment.
This looks great one small suggested change to clarify it's DETAIL years in that param - needs to be propagated (i just noted it in one place)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc:
Related: #586
What changed? Why?
Step 8 does not add a new Cornerstone conversion module. Existing
derive_cornerstone_*/cornerstone_disagg_pipelinealready applyindustry_corresp() @ V @ commodity_corresp().T. This PR adds the loader router and config so those paths can read either published BEA 2017 Detail tables or nowcast MUT artifacts (when wired).Config (
USAConfig)usa_detail_io_source:'bea_published'(default) |'nowcast'nowcast_mut_vintage: artifact build label; required whennowcastusa_base_io_data_year: extended to 2012-2024; validated per source (bea_published-> 2012/2017 only;nowcast-> 2017-2024 and must equalmodel_base_year)apply_io_year_adjustmentsforbidden whennowcastusa_detail_original_year: returnsusa_base_io_data_yearon the nowcast path, else2017New modules
bedrock/extract/iot/detail_io.py--load_detail_*routers (V, Utot, Uimp, margins, Ytot, value_added)bedrock/extract/iot/nowcast_mut_storage.py--resolve_nowcast_mut_uriandload_nowcast_detail_*raiseNotImplementedErroruntil GCS layout is finalized (Step 9: store the nowcast Make, Use, Import and Margins tables on GCS #592)Rewired consumers (default
bea_published-> sameload_2017_*inputs as before)derived_cornerstone.pycornerstone_disagg_pipeline.pycornerstone_bea_intermediates.pyderive_PRO_to_PUR_ratio.pyScaling
derive_cornerstone_Aq_scaledreturnsderive_cornerstone_Aq()unchanged whenusa_detail_io_source == 'nowcast'(detail IO is already at the IO calendar year; no summary-ratio scaling or PI inflation). Documented alongside the existingscale_a_matrix_with_useeio_methodshortcut.No production config enables
nowcastyet. Default configs are bit-identical to pre-change behavior.Follow-ups (Phase B+)
load_nowcast_detail_*tobuild_nowcast_mut_{before,after}_redef(depends on Step 6/7 landing -- Implement 2017 after-redefinitions MUTs #775, feat(iot): before-redef Use loaders and the MUT replay scorer #799 stack)bea_publishedvsnowcastCornerstone diffMerge note: #744 (
jv_electricity_reaggregation) touches the same Cornerstone files for a different feature -- rebase that branch onto this one and re-applyload_detail_*imports at conflict sites.Testing
Router regression: under default
bea_published, eachload_detail_*matches the correspondingload_2017_*.