Refreshing the ensemble at the beginning of a season

This page documents the steps for refreshing the ensemble at the beginning of a new season.

1. Update the participation guidelines, as appropriate.
2. By reaching out to teams, establish a working sense of what models will be submitted.
3. Before making any changes to the repository, create a tagged version and/or branch marking files that represent the end of the last season. This makes it easy to switch back to files as they were at a certain point in time.
4. For models that were used last season and will be used again use ./scripts/move-last-years-files.R to migrate files from the ./model-forecasts/real-time-component-models to the ./model-forecasts/component-models directory.
5. Update the "ground truth" data in the ./scores/target-multivals.csv file by running ./scripts/calculate-targets.R.
6. Recalculate all scores for component models in the master branch using ./scripts/generate-scores.js. See Code Documentation for more details.
7. Update model-forecasts/component-models/complete-modelids.csv and model-forecasts/component-models/model-id-map.csv. Currently this is done using scripts/generate-id-mappings.js (As of Oct 2019, we are not sure if any piece of visualization infrastructure relies on the definition of a model being "complete" as used by these scripts. We don't think so.)
8. Establish a list of candidate ensemble models to test for the given season. Ideally, write this down on GitHub somewhere, so we have a time-stamp of the "data analysis plan", including the method we will use to choose an ensemble.
9. Generate weights for all candidate ensemble models. For the 2019/2020 season, this requires using a combination of the scripts/calculate-weights.R and the makefile in scripts/static_adaptive_ensemble/. (According to Tom, using make rebuild staticAdaptive should re-estimate the weight file.)
10. Confirm that ensemble weights for all ensemble specifications are saved and stored in GitHub.
11. Generate leave-one-season-out cross-validation forecast files for all ensemble models using this the set of ensemble models defined above. This will require minor updates to and running of the scripts/make-cv-ensemble-forecast-files.R file.
12. Re-generate scores (see scoring instructions above) so that the updated ensemble forecasts are now scored as well.
13. Summarize scores and pick one ensemble to submit.
14. Re-organize the real-time-ensemble-models folder to make sure that there is a folder for each model. Also, put last year's files into the real-time-ensemble-models\archive-20xx-20xx folder.
15. Copy the files from last year's submissions folder into the submissions-archive folder.
16. Wait for the season to start and once it does, use the Creating ensemble submission guidelines to create weekly submissions.