Authors: Hana Hourston (@hhourston) , Di Wan (@onedwd), Lu Guan (@guanlu129), Maxim Krassovski
Note: L2 processing routine is currently out-of-date
For performing "level 0" (L0), "level 1" (L1), and "level 2" (L2) processing on raw
moored ADCP data in Python using the UHDAS pycurrents
package and plotting the output netCDF data.
Teledyne RDI Workhorse, Sentinel V, and Broadband instruments are supported.
L0 processing does not include any processing. Raw ADCP data is combined with metadata from a csv file and exported in netCDF format.
L1 processing contains minimal processing. Raw ADCP data is also combined with metadata from a csv file and exported in netCDF format. The difference from L0 is that L1 processing comprises:
- Corrections for magnetic declination
- Calculation of sea surface height from pressure values and latitude
- Rotation into enu coordinates if this is not already the coordinate system of the dataset
- Removing leading and trailing ensembles from before and after deployment
- Flagging negative pressure values
- Optional: Splitting the dataset into segments if there are pressure changes due to mooring strikes. The user must provide the date-times or ensembles at which to split the dataset
L2 processing contains:
- Flagging data in bins where calculated pressure is negative
- Flagging data by backscatter increases in upward-facing ADCPs
- Flagging data below the depth of the sea floor in downward-facing ADCPs
- Optional: Calculation of pressure data from CTD pressure data from the same deployment (if the ADCP was missing a pressure sensor)
If the time data in the raw file are garbled, then use generate_time_range.py to create a csv file containing regularly-spaced time data.
ADCP_IOS_header_file.py produces an IOS Shell header file for each netCDF file that makes the netCDF file searchable on the IOS Water Properties website.
plot_westcoast_nc_LX.py contains code for the following types of plots:
- pressure data time series (check potential ADCP depth change)
- North/East (LCNSAP01/LCEWAP01) and along-/cross-shore velocity profile time series
- Low-pass filtered versions of the above plots (Godin or X-hour rolling mean, e.g., 30 hour)
- Eastward (LCEWAP01) bin 1 time series (check bad data close to ADCP)
- Diagnostic plots for mean backscatter, mean velocity, and principal axis (orientation)
- Feather plots for select bins: current vectors plotted over time
- Rotary spectra for selected bins
- Depth profile of rotary spectrum
- Depth profile of tidal ellipses
- Single bin North/East velocity plots
Reference (now partially outdated): New tools for ADCP processing. Canadian Technical Report of Hydrography and Ocean Sciences 336.
- Before creating a virtual environment for the package, create a folder for the virtual environment and enter the folder in terminal, e.g. "adcp"
- Create a virtual environment called "adcp37" with Python version 3.7:
conda create -n adcp37 python=3.7
- Activate the virtual environment:
conda activate adcp37
- Add the conda-forge to your channel:
conda config --add channels conda-forge
conda config --set channel_priority strict
- Install required packages:
conda install numpy scipy pip pandas netCDF4 xarray gsw matplotlib=3.5 shapely
pip install datetime ruamel.yaml
- Install ttide_py from github
git clone https://github.com/moflaher/ttide_py
Navigate to the newly created ttide_py folder, then
python setup.py install
- Clone pycurrents with Mercurial:
hg clone --verbose http://currents.soest.hawaii.edu/hg/pycurrents
- Install pycurrents:
pip install -e ./pycurrents
- Clone this
pycurrents_ADCP_processing
repository with git:
git clone https://github.com/IOS-OSD-DPG/pycurrents_ADCP_processing.git
- cd to the
pycurrents_ADCP_processing
directory and run "setup.py":
python setup.py install
- Linux (or Unix-like) environment
- Python 3.7
Sample usage of ADCP_processing_L1.py, ADCP_processing_L0.py, add_var2nc.py and ADCP_IOS_Header_file.py is laid out in create_nc.py. An example of how to create uniform time data (for replacing invalid time data in a raw ADCP file) can be found in generate_time_range.py. Sample usage of the plotting functions in plot_westcoast_nc_LX.py is given in the file example_plot_westcoast.py.
ADCP_pycurrents_L1.py is based off of the gist, RdiToNetcdf.ipynb, by Jody Klymak (https://github.com/jklymak), and includes contributions from Di Wan (https://github.com/onedwd) and Eric Firing (https://github.com/efiring). add_var2nc.py was written by Di Wan. ADCP_IOS_Header_file.py was written by Lu Guan (https://github.com/guanlu129).
A web app based off this package can be found on the IOS Data Management Apps (DM Apps) page. Credits: Tom Roe.
Documentation:
pycurrents
package and log: https://currents.soest.hawaii.edu/hgstage/pycurrents/file/tip- netCDF documentation: https://www.unidata.ucar.edu/software/netcdf/docs/index.html
- RDI Broadband primer: https://www.comm-tec.com/Docs/Manuali/RDI/BBPRIME.pdf
- RDI Ocean Surveyor technical manual: http://www.teledynemarine.com/Documents/Brand%20Support/RD%20INSTRUMENTS/Technical%20Resources/Manuals%20and%20Guides/Ocean%20Surveyor_Observer/Ocean%20Surveyor%20Technical%20Manual_Apr18.pdf
- Raw data variable terminology starts on page 148;
pycurrents
outputs variables with these names
- Raw data variable terminology starts on page 148;
Conventions:
- BODC SeaDataNet quality flags: https://www.bodc.ac.uk/data/documents/series/37006/#QCflags
- BODC SeaDataNet P01 vocabulary search: http://seadatanet.maris2.nl/v_bodc_vocab_v2/search.asp?lib=p01&screen=0
- BODC SeaDataNet P06 data storage unit search: http://seadatanet.maris2.nl/v_bodc_vocab_v2/search.asp?lib=p06&screen=0
- CF Conventions standard name table: https://cfconventions.org/Data/cf-standard-names/current/build/cf-standard-name-table.html
- GF3 codes (no longer maintained): https://www.nodc.noaa.gov/woce/woce_v3/wocedata_1/sss/documents/liste_param.htm
Tidal analysis:
Pawlowicz, R., B. Beardsley, and S. Lentz, "Classical Tidal Harmonic Analysis Including Error Estimates in MATLAB
using T_TIDE", Computers and Geosciences, 28, 929-937 (2002).