feat(sam): reproducible ingestion from the SAM public monthly extract - #146
Open
ekim1394 wants to merge 1 commit into
Open
feat(sam): reproducible ingestion from the SAM public monthly extract#146ekim1394 wants to merge 1 commit into
ekim1394 wants to merge 1 commit into
Conversation
The SAM Entity API is daily-quota-gated (~1K req/day) and pagination-capped at ~5K, so the live reader can only sample the ~885K entities. This adds spicy_regs/ingest_sam_extract.py to parse the public monthly extract flat file (142 pipe-delimited fields) into the full sam_entities table — one file = full coverage. Confirmed field map + code->desc mappings documented; the 3 fields the extract doesn't expose (entity_type_desc, profit_structure_desc, exclusion_status_flag) are published NULL. Hermetic test with a 142-field fixture. (The 2026-07 extract, 885,266 entities, is already published to R2 via this parser.)
ekim1394
force-pushed
the
feat/sam-extract-ingest
branch
from
July 18, 2026 16:25
87a3cdd to
36b8b68
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full
sam_entitiescoverage from SAM's public monthly entity extract, sidestepping the API's daily quota (~1K req/day) and ~5K pagination cap that capped the live reader at a small sample.Already applied: the 2026-07 extract is parsed and published to R2 —
sam_entitieswent 4,685 → 885,266 entities, and the UEI join tousaspending_recipientsjumped 5 → 60,558 overlapping UEIs (the org-resolution join is now real).This PR makes it reproducible:
scripts/ingest_sam_extract.pyparses the 142-field pipe-delimited flat file → the 18-colsam_entitiesschema, with the confirmed field-position map and code→description mappings documented in the module. Run monthly:Get the file from SAM.gov → Data Services → Entity Registration → Public.
Known extract limits (documented):
entity_type_desc,profit_structure_desc,exclusion_status_flagaren't cleanly in the extract → published NULL;entity_structure_desccarries the raw structure code. Hermetic test covers the parse/mapping. ruff +ty+ test green.