Foundation: fetch-if-not-cached data-sourcing helper

[WestonVoglesonger]

A reusable fetch-if-not-cached data-sourcing helper, the mechanism that turns hardcoded paths into data-product handles.

Scope: generalize the one live network fetcher (COVID19_USA:model_input/preprocess/preprocess_nhsn_age5band.py, CDC Socrata paginated fetch) and the skip_if_exists cache into a resolver: given a data-product handle (URL or source plus parser), fetch if not cached locally, transform, and expose a local path. Replaces git-committed artifacts and the manual cache.

Namespace: process (or a shared utility).

DECISION for @pearsonca: cache/artifact-store contract (cache dir, content-hash keys, re-fetch policy).

Depends on: nothing. Blocks: the parameter, calibration, and reporting-regime modules.

Acceptance: the resolver fetches and caches a real dataset (mockable in tests); the cache contract is documented; unit tests with a stubbed source.

Part of #14.

[pearsonca]

I think this wants to be two things roughly:

• some sort of process module defined by fetching a target file
• that offers some standard 1-liner-few-args options (curl, wget, scp, ...)
• but also offers a spot for full custom script, with a required API that will write to the target
• in terms of caching behavior, should look for the file being absent (run) vs present (do not run, notify about next option) vs present-but-given-force-arg (run, warning about overwrite)

Second thing:

• a "transform" process, which indicates a script, source files, optionally arguments in the config and then a resulting processed file
• basically calls script with the inputs in a structured way, yields the target file
• same caching management as above

Nice to have: eventually want to have this working akin to dependency / build systems, ala make, just, etc - open to config syntax conventions that will smooth the way for that (e.g. maybe enabling the process target names to be files)

[WestonVoglesonger]

Core dependency ordering and cache no-op semantics are now tracked in ACCIDDA/flepimop2#323. Work that can proceed here without pre-empting that contract: specify the fetch and transform provider interfaces, their explicit inputs/outputs, and tests for the absent / present / forced behavior at the provider boundary. DAG traversal and cross-step cache ownership remain in core.

[WestonVoglesonger]

Initial provider contract, separated from the core DAG in flepimop2#323:

flepimop2.process.fetch

• source: source identifier understood by the selected fetch strategy
• target: Path: the single file the process promises to materialize
• force: bool = FALSE
• one simple built-in HTTP strategy using Python's standard library, plus an explicit command/script escape hatch; curl, wget, and scp should not silently become hard runtime dependencies
• success requires that target exists after execution; partial downloads should use a temporary sibling and atomic rename

flepimop2.process.transform

• sources: list[Path]: explicit input files
• script: Path and structured args
• target: Path: the processed file the script must materialize
• force: bool = FALSE
• fail before invoking the script when an input is absent; success requires target

Shared provider-level behavior

• target absent: execute
• target present, force = FALSE: no-op and report how to force
• target present, force = TRUE: warn, execute into a temporary sibling, then replace
• dry-run validates source/input availability and destination writability without writes
• tests use a local HTTP server or stub fetcher and a tiny transform script; no live external dataset in the unit suite

This keeps provider I/O concrete now while leaving cross-step dependency ordering, target identity, and provenance/freshness policy to flepimop2#323. If this contract looks right, the next implementation step is to create separate flepimop2-fetch and flepimop2-transform workspace packages so either can be installed independently.

[WestonVoglesonger]

Handoff: implementation is now in draft PR #25. I’m leaving ACCIDDA next week, so @pearsonca and @jc-macdonald, please review the provider contracts and carry the PR through revisions/merge. The PR follows the comment-level fetch/transform split, structured script interface, target no-op/force behavior, validation, and atomic replacement.