Implement industry mecs allocation for electricity disaggregation

[jvendries]

cc: cornerstone-data/methods#90
Closes:

What changed? Why?

Introduces MECS Table 7.7 purchased-kWh shares as the default weighting method for Industrial manufacturing purchasers in the EIA-anchored G/T/D electricity allocation, replacing the previous dollar-bill weighting inside manufacturing. The allocation now splits the Industrial class into two pools: manufacturing sectors (weighted by Table 7.7 kWh) and residual Industrial sectors (weighted by dollar bills). A dollars fallback is preserved for diagnostics and tests that must not load Table 7.7. EIA MECS Energy extraction is extended to parse Table 7.7 (million kWh) for both 2018 and 2022 survey years, with correct unit assignment distinct from Table 7.2 (money), and the 2022 MECS year is added to the raw data extraction script. The table was uploaded to GCS as excel and parquet FBA files.

The original freeze parquets under historical/original_elec_disagg_implementation/output/ are removed from git. In their place, a lightweight published pack is introduced under historical/original_vs_eia_anchored_deck/, containing a tables.yaml extract of class MWh and per-step D/N EF values and four committed histogram PNGs (original vs v0.2 footing and EIA G/T/D pre-MECS vs v0.3.1 footing) sourced from the published PPTX comparison deck.

A five-slide PPTX comparison deck (diagnostics/deck/) is added to visualize differences across four implementation pairs: current (post-MECS) vs. EIA G/T/D pre-MECS, current vs. original electricity disaggregation, EIA G/T/D pre-MECS vs. original, and current vs. Cornerstone v0.3 production (non-disagg). Each deck includes class MWh tables, per-step D/N EF tables with same/N/A cell labeling, and two-row histogram panels using frozen archived PNGs where available. The current_vs_production pair uses a shared production baseline rather than each implementation's own footing.

The purchaser_tables module is updated to rename d0_* terminology to class_mwh_targets/class_target_mwh throughout, add a manufacturing_mecs_vs_dollar_frame for dual-run MECS vs. dollar comparison, and expose p_share_from_allocation. The vs_original_elec_disagg comparison module is removed as the freeze it depended on is no longer tracked. Cache reset logic is extended to clear the new mecs_purchased_kwh LRU cache alongside existing electricity caches. Existing tests that exercise the allocation path are wrapped with a dollar_industrial_weights context manager so they do not trigger Table 7.7 loading.

Testing

• New unit tests in test_electricity_gtd_mecs.py cover MECS year mapping, NAICS overlay deduplication, 3-digit suppressed-value imputation, two-pool Industrial MWh identity, dollar-path isolation, zero-bill and zero-residual edge cases, cache mutation safety, and cache reset behavior.
• New unit tests in test_purchaser_tables.py cover p_share_from_allocation, manufacturing_mecs_vs_dollar_frame flags, and the renamed class_mwh_targets_frame.
• New unit tests in test_deck.py cover data helpers, histogram frame construction, frozen panel stacking, table grid generation, and end-to-end PPTX assembly (five slides, at least two media files) for both current_vs_eia_gtd and current_vs_production pairs.
• An integration test (pytest.mark.eeio_integration) validates post-overlay NAICS presence in live Table 7.7 data for 2018 and 2022.

[jvendries]

• Implement reaggregation #744
• Implement industry mecs allocation for electricity disaggregation #736 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[WesIngwersen]

1. Justify why you didn't produce an FBS (although you do build from an FBA)
2. The allocation of MECS across NAICS needs to be compared to the GHG MECS allocation of industrial combustion -- not by flow because its not allocating electricity; but by how it treats NAICS/BEA sectors with the allocation because we need consistency across the board in how we interpret/attribute the MECS data since its translating flows from the same industry pool to BEA industries.

[jvendries]

1. Justify why you didn't produce an FBS (although you do build from an FBA)

No MECS Energy FBS exists in bedrock. Tables 2.2/3.2 stay FBA and enter GHG (and CAP/HAP) FBS build as attribution sources. Table 7.7 is also left as FBA: it only supplies manufacturing purchased-kWh weights inside the electricity (Generation/Transmission/Distribution) purchaser allocation. We do not build an FBS because that path would run estimate_suppressed_mecs_energy and a generic NAICS→BEA crosswalk, both wrong for table 7.7 for the following reasons:

A) estimate_suppressed_mecs_energy drops every row marked D or Q (i.e., treats them as “no more information than an industry that is not in the table”) and replaces * with 0.25, because in those tables * means “less than 0.5 Trillion Btu”. Table 7.7 is million kWh, not TBtu. Putting 0.25 on * would be a TBtu-scale guess in the wrong unit.

Dropping Q/D is worse: EIA still publishes the manufacturing total 31-33, so a withheld 3-digit industry is recoverable. If we drop it, those kWh disappear from manufacturing shares and that NAICS gets a zero electricity weight even though the table identity tells us the amount.

What this path does instead:

• * → 0 (too small to publish, not 0.25 TBtu)
• Q/D at 3-digit NAICS → residual fill (below)
• other Q/D → 0

B) 3-digit Q/D residual fill: Table 7.7 is hierarchical: 31-33 is all manufacturing purchased electricity; under that sit 3-digit industries (311 food, 331 primary metals, 337 furniture, …), then 6-digit rows. When one 3-digit cell is Q or D, EIA still prints 31-33, which means we can calculate the leftover:

leftover = 31-33 − (sum of 3-digit industries that are published)

If exactly one 3-digit industry is Q/D, that leftover is that industry’s kWh. If leftover is not ~0 and there is not exactly one Q/D 3-digit, it hard-errors instead of guessing. This is indeed the case for 2022 MECS, which is the survey this implementation uses for model years after 2017:

Live US Table 7.7 Electricity total:

Survey	3-digit Q/D	31-33 − published 3-digit
2018	none	0 million kWh
2022	exactly one: 337 (furniture), marked Q	9,224 million kWh

We need that leftover because manufacturing G/T/D MWh shares are normalized 7.7 kWh weights. Losing a whole 3-digit industry would reassign its electricity to every other manufacturer. Recovering it from 31-33 keeps the withheld industry in the manufacturing pool with the quantity EIA implied.

C) Mapping: Using a default FBS map (BEA_2017_Detail) would not apply the 3.1 subtraction (e.g. 331313 = 3313 − 331314/331315/331318) and could attach parent rows like 31-33 unless they are explicitly excluded. GHG combustion already uses the Cornerstone 3.1 hand map, not that generic crosswalk, which means that table 7.7 must use that same hand map, which the usual FBS path would not provide.

[jvendries]

2. The allocation of MECS across NAICS needs to be compared to the GHG MECS allocation of industrial combustion -- not by flow because its not allocating electricity; but by how it treats NAICS/BEA sectors with the allocation because we need consistency across the board in how we interpret/attribute the MECS data since its translating flows from the same industry pool to BEA industries.

Summary of comparison

The treatment of table 7.7 here and MECS for GHG interpret the same MECS industry pool onto BEA the same way. On that, the methods align: same 3.1 hand map, same manufacturing vs NON_MECS split. We are not inventing a second NAICS story for electricity.

What is different is GHG’s table-specific mechanics (static 2018 survey year use, Q/D→0, split by fuel Use). Those would be the wrong analogue for purchased kWh.

The electricity method is: map with the Cornerstone 3.1 dictionaries (same); split multi-IO rows by Use of that row’s commodity (same concept, different commodity); use the MECS survey that better matches the electricity year (different); read 7.7’s 31-33 identity instead of dropping a recoverable 3-digit cell (different).

Comparison of MECS across NAICS when using table 7.7 for electricity vs. GHG MECS allocation of industrial combustion

Same

What	Method
Hand map	Which NAICS belong to which BEA, including 3.1 subtraction
Industry pools	Manufacturing vs NON_MECS membership

The hand map of MECS across NAICS is consistent between the GHG MECS allocation and the table 7.7 allocation. Both paths use CORNERSTONE_INDUSTRY_TO_MECS_3_1_NAICS_* (plus NON_MECS_INDUSTRIES for residual industrial). The use of the 2022 survey requires the MECS_7_7_NAICS_OVERLAY, which relabels four 2018 NAICS codes so 2022 Table 7.7 can be read.

Different

What	Method (7.7 vs GHG MECS)
Survey Used	7.7 follows EIA/eGRID year (2018 or 2022) vs 3.1 pinned 2018
Suppression treatment	3-digit Q/D leftover vs Q/D → 0
Multi-IO (and residual) split weights	electricity bills vs BEA fuel Use

Does it make sense to align 7.7 for electricity with GHG MECS?

Survey year: No, not by pinning 7.7 to 2018. G/T/D is tied to the EIA/eGRID year (2017 → MECS 2018; model year → MECS 2022). Using 2018 shares on a 2024 electricity allocation would freeze manufacturing kWh in the previous survey while class MWh and eGRID move, which would mix two survey years in one allocation when a more recent alternative is available.

Suppression treatment: Align only if you value cross-method identity over this table. load_mecs_3_1 zeroes Q/D. On 7.7 that would set 2022 337 to 0 and dump 9,224 million kWh onto every other manufacturer, even though 31-33 still implies furniture’s total. The leftover fill makes sense for this table.

Split weights: No. GHG does not have a generic “use BEA fuel Use” rule. It splits a MECS coal (or gas) total by BEA Use of that fuel. The parallel for 7.7 is BEA Use of electricity (purchaser bills on 221100), which is what this path already does. Splitting purchased kWh by coal/gas Use would assign furniture vs chemicals electricity with the wrong commodity. Residual NON_MECS is the same idea: leftover industrial electricity should follow electricity bills, not fuel Use.

[bl-young]

for future reference we generally don't add unit tests for extraction of specific FBAs

[jvendries]

Removed bedrock/extract/eia/__tests__/test_eia_mecs_energy_units.py, the unit tests for extraction of specific FBAs.

[WesIngwersen]

The other source that you should look at for the split beyond MECS is the source the nowcasting is now using for annual electricity which is the Cost of Electricity in the annual surveys. We are looking for physical units here, so on the surface that is less valuable, but the manufacturing and service sector annual surveys get cost of electricity at NAICS-6. The FBAs for these data are in the nowcast branch -- here are yamls for these FBAs

bedrock.extract.census.Census_AIES.yaml. See EXPS_ELEC_VAL (2023)
bedrock.extract.census.Census_AIES_Services.yaml See EXPS_ELEC_VAL (2023)
bedrock.extract.census.Census_EC.yaml See CSTELEC (2017, 2022)
bedrock.extract.census.Census_ASM.yaml See CSTELEC (2018-2021)
bedrock.extract.census.Census_SAS_Expenses.yaml, See 'EXPS_ELEC_VAL` (2018-2022)

[bl-young]

it would be good to confirm that CORNERSTONE_INDUSTRY_TO_MECS_3_1_NAICS_ + NON_MECS_INDUSTRIES is equivalent to the list of industries you intend to target with the "Industrial" portion of the EIA data to make sure nothing is getting dropped

[jvendries]

The EIA Industrial mapping to Cornerstone IO sectors includes every commodity in CORNERSTONE_INDUSTRY_TO_MECS_3_1_NAICS_* plus NON_MECS_INDUSTRIES, as well as:

• NAICS 22 utilities: electricity G/T/D (221110, 221121, 221122), gas distribution (221200), and water (221300)
• F02E00, nonresidential equipment investment

Nothing in that Industrial class is dropped. The one 3.1 mapping that never appears as a purchaser is 331314, an industry-only code with no matching commodity column in A.

[bl-young]

I spent a lot of time on this one as I want to think through how we are approaching this given both past use of MECS in old GHG FBS, and the current choice to build from the CEDA approach in the current GHG model, which is currently being updated in #688. There are definitely some decisions we need to resolve across the board regarding the best way to implement MECS wherever its used, and we need to do so consistently.

[bl-young]

I'm also concerned about the handling of these data as it relates to the 2022 NAICS codes used in the newer version. I'm not sure if that's coming up with @catherinebirney's work or not in #688

[bl-young]

In response to why not an FBS above (here) (I wish we could thread comments)

We do not build an FBS because that path would run estimate_suppressed_mecs_energy and a generic NAICS→BEA crosswalk, both wrong for table 7.7 for the following reasons:

To be clear I don't think the decision to not use an FBS is based on this above. The FBS would be NAICS based and wouldn't have to use the estimate suppressed function unless we tell it to. In both other places we are using MECS, the old attribution source, and the newer faux-FBA developed based off the old CEDA approach for the current GHG, they remain FBAs.

[bl-young]

In response to

The allocation of MECS across NAICS needs to be compared to the GHG MECS allocation of industrial combustion ... in how it treats NAICS/BEA sectors with the allocation because we need consistency across the board

I agree 100%. I think to simplify a bit on @jvendries answer, yes the use of the same mappings generally confers the same approach. Though there are differences in how the data suppression is handled it appears and also different from the old flowsa way (which I think may ultimately be the superior approach). Recall that we were worried about deviation from the original CEDA EFs for these sectors, and so for lack of a clear winner just defaulted to the CEDA MECS approach at the time (with the expectation that we would be adjusting the stationary combustion plan down the line anyway.

The other two points, I think are non -issues. Survey years is just a relic - we want to update stationary combustion to 2022 as well, and Split weights I think is just semantics - yes we use electricity use as the basis for electricity and use of other fuels as the basis for those other fuels.

All of which is to say, if we want the closest direct alignment with the approach as currently implemented (pending #688), I think we should aim to tweak this PR to align more closely with say industrial_natural_gas.py and eia_mecs_energy_ceda_parse. Alternatively we can step back and re-evaluate the use of MECS across the board and land-on which implementation is our preferred one.

[jvendries]

I'm also concerned about the handling of these data as it relates to the 2022 NAICS codes used in the newer version. I'm not sure if that's coming up with @catherinebirney's work or not in #688

To be clear I don't think the decision to not use an FBS is based on this above...

I agree 100%. I think to simplify a bit on @jvendries, yes the use of the same mappings generally confers the same approach...

I will wait to modify the implementation here until we have settled on a broader MECS approach.

[jvendries]

The other source that you should look at for the split beyond MECS is the source the nowcasting is now using for annual electricity which is the Cost of Electricity in the annual surveys. We are looking for physical units here, so on the surface that is less valuable, but the manufacturing and service sector annual surveys get cost of electricity at NAICS-6. The FBAs for these data are in the nowcast branch -- here are yamls for these FBAs

bedrock.extract.census.Census_AIES.yaml. See EXPS_ELEC_VAL (2023) bedrock.extract.census.Census_AIES_Services.yaml See EXPS_ELEC_VAL (2023) bedrock.extract.census.Census_EC.yaml See CSTELEC (2017, 2022) bedrock.extract.census.Census_ASM.yaml See CSTELEC (2018-2021) bedrock.extract.census.Census_SAS_Expenses.yaml, See 'EXPS_ELEC_VAL` (2018-2022)

I think we could probably use the Census data to allocate within some of the EIA Commercial end-use mapped IO sectors, similar to how we use MECS table 7.7 to allocate within some of the EIA Industrial end-use mapped IO sectors. It would also be useful for Transportation. We would keep MECS table 7.7 for industrial mapped allocation. I will take a look and see if I can implement this quickly on a follow up PR; if not, then this could potentially be something we implement at a later date.