This page is intended to provide teams with all the information they need to submit forecasts. All forecasts should be submitted directly to the data-processed/ folder. Data in this directory should be added to the repository through a pull request so that automatic data validation checks are run.
These instructions provide detail about the data format as well as validation that you can do prior to this pull request. In addition, we describe meta-data that each model should provide.
Table of Contents
- What is a forecast
- ground truth data
- data formatting
- forecast file format
- forecast data validation
- retractions
- weekly ensemble and visualization deployment
- policy on late submissions
Models at the COVID-19 Forecast Hub are asked to make specific quantitative forecasts about data that will be observed in the future. These forecasts are interpreted as "unconditional" predictions about the future. That is, they are not predictions only for a limited set of possible future scenarios in which a certain set of conditions (e.g. vaccination uptake is strong, or new social-distancing mandates are put in place) hold about the future -- rather, they should characterize uncertainty across all reasonable future scenarios. In practice, all forecasting models make some assumptions about how current trends in data may change and impact the forecasted outcome; some teams select a "most likely" scenario or combine predictions across multiple scenarios that may occur. Forecasts submitted to the COVID-19 Forecast Hub will be evaluated against observed data.
We note that other efforts, such as the COVID-19 Scenario Modeling Hub, have been launched to collect and aggregate model outputs from "scenario projection" models. These models create longer-term projections under a specific set of assumptions about how the main drivers of the pandemic (such as non-pharmaceutical intervention compliance, or vaccination uptake) may change over time.
The COVID-19 Forecast Hub treats case and death data on COVID-19 from JHU CSSE as "ground truth" data. Slightly different versions of these data are also available from USA FACTS and the NY Times. Hospitalization ground truth data are from HealthData.gov. We create processed versions of these data that are stored in this repository.
Details on how ground truth data are defined can be found in the data-truth folder README file.
Technical details about how and when the truth data are updated and checked for validity can be found on the Hub Wiki page about truth data.
The automatic check validates both the filename and file contents to ensure the file can be used in the visualization and ensemble forecasting.
Each subdirectory within the data-processed/ directory has the format
team-model
where
teamis the teamname andmodelis the name of your model.
Both team and model should be less than 15 characters and not include hyphens.
Within each subdirectory, there should be a metadata file, a license file (optional), and a set of forecasts.
The metadata file should have the following format
metadata-team-model.txt
and here is the structure of the metadata file.
If you would like to include a license file, please use the following format
LICENSE.txt
If you are not using one of the standard licenses, then you must include a license file.
Each forecast file within the subdirectory should have the following format
YYYY-MM-DD-team-model.csv
where
YYYYis the 4 digit year,MMis the 2 digit month,DDis the 2 digit day,teamis the teamname, andmodelis the name of your model.
The date YYYY-MM-DD is the forecast_date.
The team and model in this file must match the team and model in
the directory this file is in. Both team and model should be less
than 15 characters, alpha-numeric and underscores only, with no spaces
or hyphens.
The file must be a comma-separated value (csv) file with the following columns (in any order):
forecast_datetargettarget_end_datelocationtypequantilevalue
No additional columns are allowed.
Each row in the file is either a point or quantile forecast for a location on a particular date for a particular target.
Values in the forecast_date column must be a date in the format
YYYY-MM-DD
This is the date on which the submitted forecast were available. This
will typically be the date on which the computation finishes running and
produces the standard formatted file. forecast_date should correspond
and be redundant with the date in the filename, but is included here by
request from some analysts. We will enforce that the forecast_date for
a file must be either the date on which the file was submitted to the
repository or the previous day. Exceptions will be made for legitimate
extenuating circumstances.
Values in the target column must be a character (string) and be one of
the following specific targets:
- “N wk ahead cum death” where N is a number between 1 and 20
- “N wk ahead inc death” where N is a number between 1 and 20
- “N wk ahead inc case” where N is a number between 1 and 8
- “N day ahead inc hosp” where N is a number between 0 and 130
For county locations, the only target should be “N wk ahead inc case”.
For week-ahead forecasts, we will use the specification of epidemiological weeks (EWs) defined by the US CDC which run Sunday through Saturday. There are standard software packages to convert from dates to epidemic weeks and vice versa. E.g. MMWRweek for R and pymmwr and epiweeks for python.
We have created a csv file describing forecast collection dates and dates for which forecasts refer to can be found.
For week-ahead forecasts with forecast_date of Sunday or Monday of
EW12, a 1 week ahead forecast corresponds to EW12 and should have
target_end_date of the Saturday of EW12. For week-ahead forecasts with
forecast_date of Tuesday through Saturday of EW12, a 1 week ahead
forecast corresponds to EW13 and should have target_end_date of the
Saturday of EW13.
This target is the cumulative number of deaths predicted by the model up
to and including N weeks after forecast_date.
A week-ahead forecast should represent the cumulative number of deaths reported on the Saturday of a given epiweek.
Predictions for this target will be evaluated compared to the cumulative of the number of new reported cases, as recorded by JHU CSSE.
This target is the incident (weekly) number of deaths predicted by the
model during the week that is N weeks after forecast_date.
A week-ahead forecast should represent the total number of new deaths reported during a given epiweek (from Sunday through Saturday, inclusive).
Predictions for this target will be evaluated compared to the number of new reported cases, as recorded by JHU CSSE.
This target is the incident (weekly) number of cases predicted by the
model during the week that is N weeks after forecast_date.
A week-ahead forecast should represent the total number of new cases reported during a given epiweek (from Sunday through Saturday, inclusive).
Predictions for this target will be evaluated compared to the number of new reported cases, as recorded by JHU CSSE.
This target is the number of new daily hospitalizations predicted by the
model on day N after forecast_date.
As an example, for day-ahead forecasts with a forecast_date of a
Monday, a 1 day ahead inc hosp forecast corresponds to the number of
incident hospitalizations on Tuesday, 2 day ahead to Wednesday, etc….
Currently there is no “gold standard” for hospitalization data.
On 2020-06-06, these targets were removed:
- N day ahead inc death
- N day ahead cum death
Values in the target_end_date column must be a date in the format
YYYY-MM-DD
This is the date for the forecast target. For “# day” targets,
target_end_date will be # days after forecast_date. For “# wk”
targets, target_end_date will be the Saturday at the end of the week
time period.
Values in the location column must be one of the “locations” in this
FIPS numeric code file which includes
numeric FIPS codes for U.S. states, counties, territories, and districts
as well as “US” for national forecasts.
Please note that when writing FIPS codes, they should be written in as a character string to preserve any leading zeroes.
Values in the type column are either
- “point” or
- “quantile”.
This value indicates whether that row corresponds to a point forecast or a quantile forecast. Point forecasts are used in visualization while quantile forecasts are used in visualization and in ensemble construction.
Forecasts must include exactly 1 “point” forecast for every location-target pair.
Values in the quantile column are either “NA” (if type is “point”)
or a quantile in the format
0.###
For quantile forecasts, this value indicates the quantile for the
value in this row.
Teams should provide the following 23 quantiles:
c(0.01, 0.025, seq(0.05, 0.95, by = 0.05), 0.975, 0.99)
## [1] 0.010 0.025 0.050 0.100 0.150 0.200 0.250 0.300 0.350 0.400 0.450 0.500
## [13] 0.550 0.600 0.650 0.700 0.750 0.800 0.850 0.900 0.950 0.975 0.990
for all targets except “N wk ahead inc case” target. For the “N wk
ahead inc case” target, teams should provide the following 7 quantiles:
c(0.025, 0.100, 0.250, 0.500, 0.750, 0.900, 0.975)
## [1] 0.025 0.100 0.250 0.500 0.750 0.900 0.975
Values in the value column are non-negative numbers indicating the
“point” or “quantile” prediction for this row. For a “point” prediction,
value is simply the value of that point prediction for the target
and location associated with that row. For a “quantile” prediction,
value is the inverse of the cumulative distribution function (CDF) for
the target, location, and quantile associated with that row.
An example inverse CDF is below.
From 4/22/2021 we will be accepting NULL values in this column. The purpose
of NULL is to indicate the retraction of previously forecasted values. More
details can be found in the retractions section.
To ensure proper data formatting, pull requests for new data in data-processed/ will be automatically run.
When a pull request is submitted, the data are validated through Github Actions which runs the tests present in the validations repository. The intent for these tests are to validate the requirements above and specifically enumerated on the wiki. Please let us know if the wiki is inaccurate or if you're facing issues while running the tests.
If the pull request fails, please follow these instructions for details on how to troubleshoot.
To run these checks locally rather than waiting for the results from a pull request, follow these instructions.
If you cannot get the python checks to run, you can use these instructions to run some checks in R. These checks are no longer maintained, but may still be of use to teams working with R.
Conforming to new rules being enforced starting 4/22/2021 with the introduction
of retractions, newer/updated forecast files that have the same forecast date in
the file name must now include all previously forecasted points; i.e., the updated
forecast file cannot contain fewer rows than the previous one, and must include
all (forecast_date, target, target_end_date, location, type, quantiles)
combinations that were present in the previous forecast file. In case one of these rows are to be retracted, follow the instructions under in the next section.
From 4/22/2021 we will be formally introducing the idea of a retracted forecast.
A retracted forecast point is a updated forecast point with a NULL value (previously non-NULL)
but the same forecast_date, target, target_end_date, location, type,
and/or quantiles (if applicable); A retracted forecast is a new forecast file with the same
forecast date in the file name that contains such retracted points. Forecast teams can both retract
and update forecasts in one forecast file.
The purpose of this new idea is to enable the original data to have an explicit track-record of forecasts that were made in earlier versions and then subsequently were removed for any reason. This way, forecasting teams will have a way to retract previously made forecasts but evaluators will not lose the ability to retrieve previously retracted forecasts. A detailed description and discussion of this idea can be found on this page of the Zoltar documentation website, starting from 4/22/2021.
All forecasts containing NULL values will be subjected to review for the foreseeable future, as we understand
this is a big change and the correct semantics of NULL values may not be immediately clear.
Every Monday at 3pm ET, we will update our COVID Forecast Hub ensemble forecast using the most recent valid forecast from each team. Additional details on model eligibility are available on the page describing the ensemble. Details on which models were included each week in the ensemble are available in the ensemble metadata folder.
In order to ensure that forecasting is done in real-time, all forecasts should be submitted to the forecast hub within 1 day of the forecast date. We do not accepting late forecasts due to technical issues, missed deadlines, or updated modeling methods. We will accept updated forecasts if there was a bug in the original file. If you need to submit an updated forecast for this reason, please include a comment in your pull request confirming that there was a bug and that the forecast was fit only to data available at the time. We also accept late forecasts from new teams if they can provide publicly available information showing that the forecasts were made in real-time (e.g. GitHub commit history).