Removal of all unused legacy _qc variables

[kerfoot]

The original DAC NetCDF specification provided a set of reserved variables that could be used by the data providers to store the results of an aggregate qc algorithm for a geophysical variables. These reserved names are:

1. density_qc
2. depth_qc
3. conductivity_qc
4. lat_qc
5. lon_qc
6. latitude_qc
7. longitude_qc
8. precise_lat_qc
9. precise_lon_qc
10. precise_time_qc
11. pressure_qc
12. salinity_qc
13. temperature_qc
14. time_qc
15. time_uv_qc

These variables appear to be hardcoded into the script that builds the ERDDAP catalog:

glider-dac/scripts/build_erddap_catalog.py

Line 516 in 22d6322

# variables which need to have the variable {var_name}_qc present in the

which resulted in all data sets containing these empty variables, regardless of whether they are variables that are included in the data provider files. Upon inspection of over 1600 data sets, it has been determined that these variables are not included in the vast majority (>99%) of user submitted data sets.

The following steps are suggested:

1. The hardcoded variables listed above should be removed from the XML generation code to prevent them from being included in data sets that do not include the variables in the submitted files.
2. All elements defining these variables in data sets for which they were not included should be removed from existing data sets. This will likely require a full rewrite of the datasets.xml catalog.

These variables will be replaced by a set of reserved variable names that will contain the results of QARTOD tests. A comprehensive discussion of this can be found in #268. The resulting aggregate flags provided by the QARTOD test suite will be used by NDBC to determine suitability for release to GTS as described in #277.

[leilabbb]

See 911b4a1

The following were addressed:

• [ √] 1. The hardcoded variables listed above should be removed from the XML generation code to prevent them from being included in data sets that do not include the variables in the submitted files.
  --- unused _qc variables were removed from the code

• [ √] These variables will be replaced by a set of reserved variable names that will contain the results of QARTOD tests. A comprehensive discussion of this can be found in Inconsistent Application of QARTOD QC to data sets #268.
  --- list of legacy qartod variables is added to the code

[leilabbb]

The following is being investigated:

• [ √] 2- All elements defining these variables in data sets for which they were not included should be removed from existing data sets. This will likely require a full rewrite of the datasets.xml catalog.
• --- how to separate the providers' supplied qc-variables from the system added qc-variables: is it safe to assume that non-empty qc-variables were the providers work to store results?

[leilabbb]

In progress,

Testing a code to remove all empty _qc variables from the DAC files.

[kbailey-noaa]

This sounded like it was finished recently, and that NDBC was able to test and update their code accordingly. Can this be closed?

[leilabbb]

The legacy _qc variables still present in the system datasets are presumed to be from the providers' originally submitted files and will continue to exist in the system. Issue #372 has been opened to verify this.