Prominent warning about data revisions for generating and scoring retrospective forecasts

[micokoch]

Add prominent warnings about data revisions when using hub data for retrospective forecast generation and evaluation

Note: Changes and updates have been made based on feedback. Please scroll to the bottom to find the latest proposed language.

Background

This comes out of a suggestion from the Delphi group to have hubs include a warning prominently that care must be taken to actually use historical revision behavior for retrospective forecast evaluation.

I'm starting this as a discussion (rather than an issue) because it spans several repos and the immediate goal is to gather feedback on the wording, placement, and links before we commit to implementing anything. Once we've converged here, we can spin off concrete issues in the individual repos and link back to this thread.

The problem

Real-time forecasting hubs store the forecasts that were generated with the data available at the time each forecast was made. Surveillance/target data, however, is frequently revised after its initial release — reporting delays, backfill, and retroactive corrections mean that the observed value for a given target date can change (sometimes substantially) in later data versions.

If an analyst runs a retrospective evaluation using the latest / finalized version of the target data, they are no longer reproducing the conditions the forecasters faced. Models can look artificially better or worse than they were in real time, and comparisons between models can be distorted. Faithful retrospective evaluation requires using the version of the target data that was available as of each forecast's reference date.

The hubverse target-data format already supports this: time-series target data may include an optional as_of column, and target-data.json has a versioned flag. But:

• not every hub populates versioned data, and
• even where versioning exists, an analyst has to knowingly filter to the correct as_of version rather than pulling the current snapshot.

So this is primarily a documentation / prominent-warning effort, not a format change: make the risk visible everywhere hub data is offered for download or evaluation, tell analysts what to do, and point them to versioned-data archives (e.g. Delphi's) where a hub doesn't retain its own revision history.

Proposed solution

Add a short, consistent warning — with a link to guidance and to versioned-data archives — in the places where someone is most likely to grab hub data and evaluate it. Use one canonical long-form version in the docs and shorter pointers elsewhere that link back to it.

Where to put it

Repo	Location	Which variant	Notes
hubDocs (docs.hubverse.io)	user-guide/target-data.md, as an admonition near the as_of / versioned discussion	Full (A)	Canonical home for the warning; everything else links here.
hubDocs	Evaluation-related content (e.g. the evaluations/hubEvals docs, dashboard-predevals page)	Short (B), linking to A	Catches people who arrive via the evaluation path.
hubTemplate	README.md — add a standard "Using this data for retrospective evaluation" section to the template boilerplate	README (C)	New hubs inherit the warning automatically.
hub-dashboard-template	An evaluation/about page in pages/, and/or near predevals-config.yml docs	Short (B)	Dashboards are a common entry point for casual users.
Existing real-time hub repos (and archives, e.g. covid19-forecast-hub-archive)	README.md	README (C)	The direct ask — warn future analysts pulling from live/archived hubs.
hubverse-site (hubverse.io)	A line + link on the data/tools page	Link to A	Lightweight; mostly a signpost.

Once wording and placement are agreed here, the plan is to open a concrete issue per repo (starting with hubDocs, which is the canonical home for the warning) and link each back to this discussion. The rollout list below is the starting point for those issues.

---

Draft warning text

Variant A — Full (docs admonition)

⚠️ Data revisions and retrospective evaluation

Forecasts in a real-time hub were made using the data available at the time. Surveillance data is often revised after its initial release (reporting delays, backfill, and corrections), so the observed value for a past date can change in later data versions.

If you evaluate these forecasts retrospectively using the latest version of the target data, you are not reproducing the information forecasters actually had, and your results may be misleading. For a faithful evaluation, use the version of the target data that was available as of each forecast's reference date.

• If this hub publishes versioned target data (an as_of column / versioned: true in target-data.json), filter to the version available at forecast time rather than using the current snapshot.
• If versioned data is not available here, use an external archive that preserves data revisions — for example, Delphi's epidata archives (exact link to be confirmed with Delphi).

Variant B — Short (evaluation pages / dashboards)

⚠️ Evaluating retrospectively? Surveillance data gets revised after it's first reported. Using the latest data to score forecasts that were made in real time can produce misleading results — use the data version available as of each forecast's reference date. See Data revisions and retrospective evaluation for details and links to versioned-data archives.

Variant C — README boilerplate (template & real-time hubs)

⚠️ A note on retrospective evaluation

The forecasts in this hub were produced with the data available at the time they were made. Surveillance data is frequently revised afterward, so scoring these forecasts against the current target data does not reflect what forecasters actually knew and can give misleading results.

If you use this data for retrospective evaluation, use the version of the target data that was available as of each forecast's reference date. See the hubverse guidance on data revisions and retrospective evaluation and, where this hub does not retain versioned data, versioned-data archives such as Delphi's (link to be confirmed).

---

Open questions / for discussion

1. Exact Delphi link(s). Which resource does the Delphi team want us to point at — the Epidata API as_of/issues docs, the epiprocess epi_archive tooling, epidatr/epidatpy, or a landing page? (Placeholders above.)
2. Tone/length of the docs version. Is Variant A the right level of detail, or do we want a fuller worked example (e.g. the 420→446 revision example already in target-data.md)?
3. Scope of "existing real-time hub repos." Which repos are in scope? Confirm the archive repos (e.g. covid19-forecast-hub-archive) should carry the README note too.
4. hubverse-site — worth a line there, or keep it to docs + repos to avoid dilution?
5. Should we also surface this at data-access time (e.g. a note in hubData / Python hub-data docs where connect_target_*() functions are described), since that's where analysts programmatically pull target data?

Proposed plan (for discussion)

Rough rollout, to become per-repo issues once we've agreed on the text:

• Finalize wording with Delphi feedback (Variants A/B/C + Delphi links)
• hubDocs: add Variant A to user-guide/target-data.md
• hubDocs: add Variant B pointer to evaluation/dashboard-predevals docs
• hubTemplate: add Variant C section to README boilerplate
• hub-dashboard-template: add Variant B to an eval/about page
• Add Variant C to existing real-time hub + archive READMEs
• hubverse-site: add signpost line + link
• (Optional) note in hubData / hub-data target-data access docs

[jbracher]

I've been pointed to this issue as I've recently had similar discussions with various people. One thought: the above reads like we need to use target data as of the time of forecasting for scoring ("If you evaluate these forecasts retrospectively using the latest version of the target data", "so scoring these forecasts against the current target data does not reflect what forecasters actually knew"). I would argue we need to use the data as it was available for forecast generation, which is something different. For scoring it is fine to use the data as available at the end of the season - or whatever was defined as the target in advance.

One alternative to data as of end of season as the target is to give each data point a fixed amount of time to stabilize. So for instance "predict the incidence value after four weeks of potential retrospective revisions". That makes sense when revisions primarily stem from reporting delays, as the models used to correct for this usually need some pre-defined maximum reporting delay to take into account. In our German Hubs this is actually our preferred solution, see our discussion in Wolffram et al.

I feel like currently the targets aren't always well-defined as it is often not stated explicitly which data version is the actual target. But maybe this is defined somewhere for the US Hubs, I don't know the documentation well enough.

I wonder if we also need well-defined exclusion criteria for state-weeks with too strong revisions to be meaningfully scored. IMO the revisions to look at to this end are the ones in the latest data points forecasters had available at a given point in time, not the ones in the data points they had to predict. I think the latter is what we did in Sherratt et al, but I'm not so convinced anymore. Again it depends on whether the issue is delayed reporting and thus predictable revisions, or occasional major and unpredictable revisions / backfill. For predictable revisions we'd probably say they are just part of the game.

[nickreich]

Thanks for starting this conversation, @micokoch ! I agree with @jbracher's points. So a specific suggestion for edits to Variant A could be

If you evaluate these forecasts retrospectively using the latest version of the target data, you are not reproducing the information forecasters actually had, and your results may be misleading. For a faithful evaluation, use the version of the target data that was available as of each forecast's reference date.

becomes

If you attempt to generate new forecasts retrospectively using the latest version of the target data, you are not reproducing the information forecasters actually had, and your results may be misleading. For a faithful replication, use the version of the target data that was available as of each forecast's reference date to generate new forecasts and score them using an appropriate "finalized" version of the data. This "final" version may vary depending on the hub setup.

With similar adjustments to B and C.

[micokoch]

Thank you @jbracher and @nickreich for the comments and suggestions. Here are the three reworked variants building on Nick's suggested language for Variant A (the short variants B and C are meant to link back to the canonical explanation in Variant A once it lives in the hubverse docs).

---

Variant A — Full (docs admonition)

⚠️ Data revisions, retrospective forecasting, and evaluation

Forecasts in a real-time hub were made using the data available at the time. Surveillance data is often revised after its initial release (reporting delays, backfill, and corrections), so the observed value for a given target date can change in later data versions.

Because of this, the data version used to generate forecasts and the data version used to score them are two different things:

• Generating forecasts retrospectively. If you attempt to generate new forecasts retrospectively using the latest version of the target data, you are not reproducing the information forecasters actually had, and your results may be misleading. For a faithful replication, use the version of the target data that was available as of each forecast's reference date to generate the new forecasts.
• Scoring. Score forecasts against an appropriate "finalized" version of the data — the version that was defined in advance as the prediction target. This "final" version may vary depending on the hub setup: for example, the data as of the end of the season, or each data point given a fixed amount of time to stabilize (e.g. the value after a set number of weeks of revisions). Check the hub's documentation for how its target is defined.

Where a hub publishes versioned target data (an as_of column / versioned: true in target-data.json), you can reconstruct the version available at any past date. Where versioned data is not retained here, use an external archive that preserves data revisions — for example, Delphi's epidata archives (exact link to be confirmed with Delphi).

Variant B — Short (evaluation pages / dashboards)

⚠️ Forecasting or evaluating retrospectively? Surveillance data gets revised after it's first reported, so the data version you use matters. To re-generate forecasts, use the data version available as of each forecast's reference date — using the latest data lets models "see" information the original forecasters didn't. To score forecasts, use the finalized target defined by the hub (which may differ from both the as-of-forecast-time data and the current snapshot). See the hubverse guidance on data revisions and retrospective evaluation for details and links to versioned-data archives.

Variant C — README boilerplate (template & real-time hubs)

⚠️ A note on retrospective forecasting and evaluation

The forecasts in this hub were produced with the data available at the time they were made. Surveillance data is frequently revised afterward, so the data version you use matters:

• To re-generate forecasts retrospectively, use the version of the target data available as of each forecast's reference date. Using the current data lets models "see" information the original forecasters did not, which can give misleading results.
• To score forecasts, use the finalized target version defined for this hub (e.g. end-of-season data, or data given a fixed time to stabilize) — [hub maintainers: state or link your hub's target definition here].

See the hubverse guidance on data revisions and retrospective evaluation, and — where this hub does not retain versioned data — versioned-data archives such as Delphi's (link to be confirmed).

---

[dsweber2]

I thought I'd commented a couple of weeks ago, but apparently not! sorry about that. I wrote similar PR's for flusight and covidhub.

Anyway I think it may be best to point at Delphi's epidata API. Otherwise the wording looks good to me and glad to have this in there!

[nickreich]

Thanks @micokoch ! I think your suggested language looks good. And thanks for the additional comments @dsweber2 !

[micokoch]

Thanks @dsweber2 and @nickreich! I've updated the Delphi reference to point at Delphi's Epidata API as suggested by David, and aligned the wording with the FluSight (#3688) and covidhub (#1645) PRs (adding the epidatr/epidatpy clients, the git-reconstruction option, and a note on how large the effect can be). Here is the final version of the three variants, ready for implementation. I will proceed with implementation as well.

---

Variant A — Full (docs admonition)

⚠️ Data revisions, retrospective forecasting, and evaluation

Forecasts in a real-time hub were made using the data available at the time. Surveillance data is often revised after its initial release (reporting delays, backfill, and corrections), so the observed value for a given target date can change in later data versions. In practice, generating forecasts from finalized data and then evaluating them against that same data can make retrospective performance look substantially better than it really was in real time.

Because of this, the data version used to generate forecasts and the data version used to score them are two different things:

• Generating forecasts retrospectively. If you attempt to generate new forecasts retrospectively using the latest version of the target data, you are not reproducing the information forecasters actually had, and your results may be misleading. For a faithful replication, restrict each forecast's inputs to the data as it was available as of that forecast's reference date.
• Scoring. Score forecasts against an appropriate "finalized" version of the data — the version that was defined in advance as the prediction target. This "final" version may vary depending on the hub setup: for example, the data as of the end of the season, or each data point given a fixed amount of time to stabilize (e.g. the value after a set number of weeks of revisions). Check the hub's documentation for how its target is defined.

To obtain the data as it was reported on a past date:

• If this hub publishes versioned target data (an as_of column / versioned: true in target-data.json) or versions its target data in the repository's git history, reconstruct the version available at the relevant date from the hub itself.
• Otherwise, use an external archive that preserves data revisions — for example, Delphi's Epidata API, accessible via the epidatr (R) and epidatpy (Python) clients.

Variant B — Short (evaluation pages / dashboards)

⚠️ Forecasting or evaluating retrospectively? Surveillance data gets revised after it's first reported, so the data version you use matters — generating forecasts from finalized data and then evaluating against it can make performance look substantially better than it really was. To re-generate forecasts, restrict each forecast's inputs to the data available as of its reference date. To score forecasts, use the finalized target defined by the hub (which may differ from both the as-of-forecast-time data and the current snapshot). See the hubverse guidance on data revisions and retrospective evaluation for details and links to versioned-data archives.

Variant C — README boilerplate (template & real-time hubs)

⚠️ A note on retrospective forecasting and evaluation

The forecasts in this hub were produced with the data available at the time they were made. Surveillance data is frequently revised afterward, so the data version you use matters — generating forecasts from the latest data and then evaluating against it can make retrospective performance look substantially better than it really was:

• To re-generate forecasts retrospectively, restrict each forecast's inputs to the version of the target data available as of its reference date. Using the current data lets models "see" information the original forecasters did not.
• To score forecasts, use the finalized target version defined for this hub (e.g. end-of-season data, or data given a fixed time to stabilize) — [hub maintainers: state or link your hub's target definition here].

To get the data as it was on a past date, reconstruct it from this hub's versioned target data / git history where available, or use an external archive such as Delphi's Epidata API via the epidatr (R) and epidatpy (Python) clients. See the hubverse guidance on data revisions and retrospective evaluation for more.