Implement reaggregation

[jvendries]

cc:
Closes: cornerstone-data/methods#92

What changed? Why?

Adds a new implement_electricity_reaggregation flag and supporting pipeline that collapses the post-3-way-split G/T/D sectors (221110, 221121, 221122) back into a single monetary 221100 sector, producing a published 405-sector model. This sits after the existing reallocation + 3-way disaggregation steps but does not use mixed physical units, making it mutually exclusive with implement_electricity_mixed_units.

The reaggregation step:

• Sums child commodity/industry entries into 221100 across A, B, V, U, VA, x, q, and final demand (Y) matrices using q-weighted or block-sum collapse helpers in cornerstone_disagg_pipeline.py
• Exposes new cached derived functions (derive_cornerstone_Aq_reaggregated, derive_cornerstone_B_reaggregated, derive_cornerstone_V_reaggregated, derive_cornerstone_U_set_reaggregated, derive_cornerstone_VA_reaggregated, derive_cornerstone_x_reaggregated, derive_disagg_Ytot_reaggregated, derive_cornerstone_y_nab_reaggregated) in derived_cornerstone.py
• Routes get_V, get_U, get_Udom, get_x, get_A, get_Adom, get_B, get_D, get_q, get_Phi, and get_Rho in model_objects.py through the reaggregated path when the flag is active
• Sets phi[221100] = 1.0 and restores rho[221100] from the 221110 price index panel under reaggregation
• Adds two new YAML configs: 2025_usa_cornerstone_v0_3_electricity_reaggregation (production) and test_usa_config_waste_disagg_electricity_reaggregation (test)
• Snapshot generation and derive_E_usa now use the published 405-column path under reaggregation; load_E_from_flowsa remains the internal 407 path so per-child E/x is available before the q-weighted B collapse
• derive_phi_cornerstone_usa_panel_published is introduced to apply electricity producer-price overrides (G/T/D → 1.0 under disaggregation; 221100 → 1.0 under reaggregation) and reindex to 405 before writing snapshots or populating get_Phi

The diagnostics deck renames the current implementation to mecs_mixed_units throughout (pairs, tests, README, CLI help text) and adds a new reaggregated_vs_production pair. This fifth pair uses custom table_steps and hist_steps ending at reaggregation instead of mixed_units, and replaces slide 1's class-MWh table with a 221100 q/x (USD) table via a new qx_grids function. Histogram rendering now reads pair.hist_steps instead of a module-level constant, allowing per-pair step sequences. The EIA G/T/D class-MWh table in tables.yaml is expanded from a combined Com+Ind+Trans+Exports row to individual rows matching CLASS_ORDER, and EIA_CLASS_GROUPS is updated accordingly.

A new ceda_electricity diagnostics subpackage is added with two PPTX decks: Deck A compares the electricity reaggregation run against the CEDA v8.1 baseline (N/D histograms, D/A-effect scatter, focus-sector table for key countries), and Deck B shows a cumulative g1→g5e ABSR + electricity ladder with per-step N/D histogram panels, BLy net waterfall charts, and a USA 221100 callout comparing g5, g5+electricity, and electricity-only results.

Testing

• New test_electricity_reaggregation.py covers: block-collapse identity, no-op when flag is off, error when children are missing, U reconstruction from collapsed A/q, q-weighted B collapse, published axis shape (221100 present, G/T/D absent), integration identities (q sum, U = A·q, B·q invariant, L·y = q), Phi/Rho values on the product YAML, and a check that reaggregated D/N differ from non-disagg production
• New test_derive_E_usa_reagg.py covers column collapse and passthrough behavior of derive_E_usa under the reaggregation flag
• test_usa_config.py extended with parsing, prerequisite-flag, and mutual-exclusion tests for implement_electricity_reaggregation
• test_phi_helpers.py extended with tests for derive_phi_cornerstone_usa_panel_published under both reaggregation-on and disaggregation-on paths
• Existing deck tests updated to use renamed mecs_mixed_units keys; new deck tests cover reaggregated_vs_production pair schema, shared ladder steps, qx_grids output, load_impl_bundle step filtering, derive_step routing, EF grid headers, and full PPTX write with slide-title assertion
• A guard asserts no legacy current* keys remain in IMPLEMENTATIONS or PAIRS

[jvendries]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Implement reaggregation #744 👈 (View in Graphite)
• Implement industry mecs allocation for electricity disaggregation #736
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[bl-young]

Reaggregation appears to be publish-layer only, not used in the snapshot

The *_reaggregated getters are wired through model_objects.py (and diagnostics/deck), but generate_snapshots.py still writes derive_Aq_usa() / derive_B_usa_non_finetuned() - i.e. the 407-sector G/T/D-disaggregated matrices, with no collapse to 221100.

For a config with reaggregation, that means:

• Published Excel/SEF: 405-sector model
• Snapshot parquets + diagnostics "old" baseline: 407-sector model

I don't believe that to be the intended outcome?

[jvendries]

Reaggregation appears to be publish-layer only, not used in the snapshot

The *_reaggregated getters are wired through model_objects.py (and diagnostics/deck), but generate_snapshots.py still writes derive_Aq_usa() / derive_B_usa_non_finetuned() - i.e. the 407-sector G/T/D-disaggregated matrices, with no collapse to 221100.

For a config with reaggregation, that means:

• Published Excel/SEF: 405-sector model
• Snapshot parquets + diagnostics "old" baseline: 407-sector model

I don't believe that to be the intended outcome?

Yes, that was a gap. The USA snapshot derives (derive_Aq_usa, derive_B_usa_non_finetuned, y_nab, Y/trade) now return the published 405 collapse when the reaggregation flag is on, so generate_snapshots matches Excel/SEF. The 407 G/T/D internals are unchanged. E and Phi snapshots collapse the same way. A new snapshot still has to be generated for a reaggregation config before diagnostics “old” is 405.

[bl-young]

I think the PR description might be a bit out of date.

I was a bit surprised to see the manipulations to E and phi in generate_snapshots.py. My intuition is that these objects should be handled as they are derived and not just when generating the snapshot. So I have two follow up suggestions:

1) E

Today derive_E_usa() is just return load_E_from_flowsa() — they do the same thing. On a reaggregation config both return 407 G/T/D columns. Snapshot generation works around that with a one-off collapse in generate_snapshots.py; anything else that calls derive_E_usa() (diagnostics, decompose_d_E_x, etc.) still sees 407.

The suggestion is to split those two functions so they have different jobs:

Function	Role
load_E_from_flowsa()	Load and map E from the GHG FBS — 407 when G/T/D disagg is on. Internal input to B construction.
derive_E_usa()	Published E for snapshots, diagnostics, comparisons — 405 when reaggregation is on (column-sum collapse G/T/D → 221100, reindex to CORNERSTONE_INDUSTRIES).

Concrete changes:

1. Add reaggregation collapse to derive_E_usa() (same rule as the snapshot block today).
2. Point derive_cornerstone_B_via_vnorm() at load_E_from_flowsa() instead of derive_E_usa(), so B is still built from 407 E before q-weight collapse.
3. Remove the E special-case from generate_snapshots.py; snapshots call derive_E_usa() like A/B/y.

Why the split: B is computed at 407 as (E/x) @ Vnorm, then collapsed to 405 with q-weights from post-reanchor scaled_q. That q-weight step needs per-child G/T/D E/x at 407. Collapsing E first and recomputing B would be a different published B (sum of E / sum of x ≠ q-weighted average of per-child B columns). Right now the alias between derive_E_usa and load_E_from_flowsa forces every caller to share one sector count; splitting them matches what you already did for B (derive_B_usa_non_finetuned() → published 405; derive_cornerstone_B_non_finetuned() → internal 407).

2) Phi

Runtime EF adjustment is already correct — phi_for_sectors() handles disagg (G/T/D → 1.0) and reaggregation (221100 → 1.0).

Publish and snapshots still duplicate logic: get_Phi() partially overrides 221100, and generate_snapshots.py drops G/T/D rows and sets 221100 = 1.0 again.

Suggested change: one published helper in derive_PRO_to_PUR_ratio.py (e.g. derive_phi_cornerstone_usa_panel_published()) that builds the margin panel, reindexes to the published commodity list, and applies the electricity overrides. get_Phi() and snapshot generation both call it.

No internal/public split needed for Phi — margins are 405-native; reaggregation is just 221100 = 1.0, not a matrix collapse.

[jvendries]

I think the PR description might be a bit out of date.

I was a bit surprised to see the manipulations to E and phi in generate_snapshots.py. My intuition is that these objects should be handled as they are derived and not just when generating the snapshot. So I have two follow up suggestions:

1) E ...

2) Phi ...

Implemented the fixes suggested for E and Phi handling, and updated the PR description.

[jvendries]

Merge activity

• Sep 1, 3:52 PM UTC: A user started a stack merge that includes this pull request via Graphite.