METRIC Processing Status

[NadyaAlexander]

Hi @qjhart and @jamorande,

I am expecting to work until August 13th, but there is a possibility that I could go in for a C-section any day between now and then.

In order to facilitate our workflow with the current METRIC project, I am going to create a status diary here that I will update whenever new work is done. The purpose of the diary will be to give an update of where I am with all image processing, where the inputs and results are, what data is final, and what data is incomplete. These information might not be obvious looking through the folders in Kaweah, so I want to create a record here.

If I do have to take leave unexpectedly before we finish processing the water year for the SSJ-BD project Andres you will be responsible for running METRIC. I will write you a procedure today and tomorrow and it will be very straightforward - between Quinn's work with Model 1 and my modifications to the spreadsheet, the directions will be very easy to follow. Quinn will be able to assist you with the hot and cold pixel selector and other pieces of code that he has built.

[NadyaAlexander]

2016.07.21 Current Processing Status [DRAFT]

(1) Information on Model 1 Outputs
These are already in place for the 2014-2015 WY run. You would only need to modify this if we re-ran Model 1 and Quinn handed over new results.

The METRIC model is run up through the end of Model 1 in Earth Engine. Quinn downloads those outputs to Kaweah in images/FromEarthEngine as zip files. The zip files need to be moved into their respective date folders inside the "products" folder. Quinn has named them all appropriately so that they are automatically read by Model 2 in Erdas.

The only exception to this is the landcover raster. The landcover raster needs to be renamed and moved to the "images" folder. The renaming adds an underscore between the mmdd and yyyy and deletes the L8 name. You can see the example of the correct naming currently in the images folders.

(2) Running METRIC
In order to run METRIC, you only need to populate Model 2 and Model 3 using the spreadsheet and press "Run" in Erdas. What is missing from the spreadsheet will change from day to day, I cover that in (3) below. This step will discuss how to do the running of METRIC.

Besides the inputs covered in (1) above, METRIC needs the spreadsheet, a DEM, and an AOI. The DEM and AOI have already been created and are in the images folder, where they are supposed to be. I have redesigned the spreadsheet so that it is very easy to add new data to it. All the data that needs to be added should be added to the first two purple sheets. They are automatically linked to the rest of the spreadsheet. Once those parts of the spreadsheet are filled in, you will go to the "VB Populate" tab, check "cf02" and "cf03", and press the populate button.

Once you have done that, you will open Erdas Model Maker, as you have been doing to look at model equations, and first open and run cf02 and then cf03. The spreadsheet writes the new models to their respective image dates in the "model" folder, so that is the version you need to open.

There is one quirk in Erdas. You shouldn't have to do this, but you do. Before you run the model, go to Model -> Set Window. Change Set Window to "Specify". Check the box next to "Use file for map extent". Under File Name choose the NDVI raster in the "products" folder and click Okay. The model is supposed to only run over the AOI, but for some reason sometimes Erdas runs over a much larger area than the AOI and it messes everything up. So we have to use this backup method as a secondary constraint on the processing area.

That's it for running METRIC. The lightening bolt is the run button. The only other things that need to be done are in the spreadsheet in (3) below.

[NadyaAlexander]

2016.07.21 Spreadsheet Status [DRAFT]

(3) Finalizing the Spreadsheet

All the changes in the spreadsheet are made to the first two tabs, the "Climate" and "Data Entry" tabs. You only paste data or make changes into the orange cells. When you paste data, please paste as "values only" or "keep original/destination formatting" to preserve the formatting of the spreadsheet.

The current status of the spreadsheet is as follows:

(a) The spreadsheet is not complete. Therefore, the most up-to-date version is only in the P44_R33_34_2014_base folder. After we get most of the data in, you will need to copy the spreadsheet to the individual image date folders into the correct subfolder.

(b) The data on the "Climate" tab has not been updated, what is shown there is for a previous project. The data on this tab is a combination of data from CIMIS, data Quinn downloads from Spatial CIMIS, and data that I calculate in R. The details on the Climate tab are covered in more detail in (4) below.

(c) On the "Data Entry" tab, there are certain cells you will change for each image date to update the correct date and the hot and cold pixel information. The details on the hot and cold pixels are covered in detail in (5) below. The cells that you will change are E1, E4, and B10. You will also fill in B28:H29 with the hot and cold pixel information. All the other information, except for the section titled "Soil Water Balance Info" is up to date and correct and you do not need to make changes to it for different dates.

(d) "Soil Water Balance Info" is covered in (4) below.

(4) Climate Tab (and Soil Water Balance Info)

Although we are running METRIC for the water year, which is October - September, the METRIC spreadsheet is designed for a calendar year. It is not easy to break links inside the spreadsheet because of all their links and macros. Therefore, I have decided that it is easier to run the calendar year of 2014 and the calendar year of 2015 separately for weather data and ET. You will paste the data for the correct year into the correct spreadsheet.

(a) Climate Tab - Daily Data
For columns L,M,N,O, and P, you are pasting data that you download directly from CIMIS. That data comes in a "daily.csv" file. It is in the Evapotranspiration folder I shared with you, inside the Twitchell folder. There is one for 2014 and one for 2015. The data has not been added to the spreadsheet yet because it needs to be QAQC'd. The way we are doing that is that if the data doesn't look good (missing or a zero where there shouldn't be a zero), we are going to put spatial CIMIS data in there. Column Q already has the correct values in it, from my spreadsheet called Rs_QAQC_ID140_2014 in the folder I shared with you. You will need to change column Q when you create the 2015 spreadsheet, using the data on the "DataForR" tab in the Rs_QAQC_ID140_2015 spreadsheet.

(b) Climate Tab - Hourly Data
For columns A-E and G-I you are pasting data that you download directly from CIMIS. That data comes in a "hourly.csv" file. It is in the Evapotranspiration folder I shared with you, inside the Twitchell folder.

(c) Data Entry Tab - Precipitation (column K)
This comes from CIMIS and needs to be pasted from the "daily.csv" file. No QAQC.

(d) Hourly and Daily ETr.
For Hourly and Daily ETr, you will need to run my R program. It is in the Evapotranspiration folder I shared with you. The instructions in the R code should be self-explanatory. You only need modify two parts of the code - "CIMIS definition" and "Write Results". We want to run alfalfa reference. All you need to do to run this code is make a folder with the R Markdown script, the daily and hourly .csv files (QAQC'd to fix problems with missing data), and a file called "SolRad.csv". SolRad.csv was made from the "DataForR" tab in the Rs_QAQC_ID140_2014" spreadsheet. It exists for both years in the Evapotranspiration folder. It is up to Quinn if he wants to do additional QAQC on Rs, don't worry about that part if I'm not working on it. The Rs in there right now is from spatial CIMIS.

You will run the code and it will give you an output file for hourly and daily ETr. That is the data you will paste into Climate Tab column F (hourly) and Data Entry Tab column M (daily). The code actually outputs a file with all the things you need for METRIC, so you can paste the things for (a) - (c) above from here too. It's the same numbers.

The reason why we paste into column M and not column L is because if there is an ETr = 0 it messes up the whole METRIC spreadsheet. So I modified the data to make ETr = 0 into ETr = 0.01.

(5) Hot and Cold pixel selection.
The hot and cold pixel selection is a piece of Earth Engine code Quinn wrote. You will have to coordinate with him on how to run it. When you run it, you will get a map that shows good hot and cold candidates. You will try to choose one that is in a relatively homogenous field close to the weather station (look at the weather station on the CIMIS website station map).

Earth Engine will give you all the values you need to enter into the B28:H29 cells. Ignore the greyed out hot/cold pixel info below that, that's just more rows if you want to record multiple candidates before you decide. You will need to convert the lat/long from EE into X and Y using this website:
http://www.latlong.net/lat-long-utm.html

Once you have filled in everything above, you are ready to populate the model as discussed in (2) above.

I will keep adding new comments as these steps get filled in over the next week, to let you know what is ready and what still needs to be done.

[jamorande]

Thanks. Let's try to discuss this further tomorrow.

A

---

From: NadyaAlexander [email protected]
Sent: Thursday, July 21, 2016 1:32:12 PM
To: ssj-delta-cu/ssj-ucd-metric
Cc: Jorge Andres Morande; Assign
Subject: Re: [ssj-delta-cu/ssj-ucd-metric] METRIC Processing Status (#5)

2016.07.21 Spreadsheet Status [DRAFT]

(3) Finalizing the Spreadsheet

The current status of the spreadsheet is as follows:

(a) The spreadsheet is not complete. Therefore, the most up-to-date version is only in the P44_R33_34_2014_base folder. After we get most of the data in, you will need to copy the spreadsheet to the individual image date folders into the correct subfolder.

(b) The data on the climate tab has not been updated, what is shown there is for a previous project. The data on this tab is a combination of data from CIMIS, data Quinn downloads from Spatial CIMIS, and data that I calculate in R. The details on the climate tab is covered in more detail in (4) below.

(c)

You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/5#issuecomment-234374905, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ATZ6Un4ucnjONYpc6JOep8_fNuh77COyks5qX9dMgaJpZM4JRJoa.

[NadyaAlexander]

@qjhart @jamorande
Expecting to have the baby in the next couple days. Here is the current status of the processing:

I have run most of the 2015 WY. I haven't looked at all the rasters yet. We need to pull out some statistics for crops like alfalfa to see if the numbers seem reasonable.

I'm not sure about open water and riparian. We can work more to refine this in phase II. There seems to be some big changes in ET around these areas, Quinn will have to decide how we should include it.

In terms of the instructions above:
A. All the spreadsheets have been updated with the 2014 and 2015 weather data and ET calculations. The base folder has been removed. This means that (3) and (4) above have been finished, and (5) is done for all dates except what I mention below.

The dates that are not finished are:

1. We talked about running a date in September 2014 but I don't have Earth Engine outputs for that.
2. October 12, 2014 has crazy high ET during the overpass time. It looks correct - I checked it against Rick Allen's code. The windspeed is like insane, if I remember correctly. So we need to look into that and either get a different ETr number for that timeframe or maybe skip that day, or at least look into the spreadsheet and how that is affecting the hot and cold pixel calibration.
3. I can't get the hot and cold pixel selector code to run for 12/31/14, so that date isn't done yet. The only thing missing from the spreadsheet is the hot and cold pixel information.
4. There was a glitch in 2015 where the lookup function stopped working for the hourly climate data. Those spreadsheets have the cells highlighted in bright yellow on the first purple tab, the Data Tab, to show where they are. I haven't solved it yet, so I just copied and pasted the values in there manually from the second purple tab, the Climate Tab.
5. The last date we have for the WY (I think 9/29) is too cloudy.

Everything else has been run. If Andrés needs to do some tweaking, maybe put in new ET numbers or something, make sure to only tweak those first two purple sheets in the spreadsheet. All the other sheets are linked together and super sensitive.

If you want to run the R code again, re-download it, I made a couple small changes.

Quinn - I didn't finish pulling apart the PM ET equation you sent me. It looks right, but it's doing more things in one step and I haven't followed all the pieces all the way through yet to make sure they're exactly the same. My R code follows the 2005 ASCE paper exactly, some of the equations are noted with equation numbers but I didn't put them all in there yet. So maybe Andrés or someone else can look at that.