Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setting output directory to tmp_dir #655

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

setting output directory to tmp_dir #655

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
56bed4b
- updating file download location
ddurden Apr 25, 2023
69c9146
- fixing formatting issues and removing warning messages
ddurden Apr 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 18 additions & 14 deletions tutorials/R/eddy-covariance/diel-cycle-flux/flux-diel-cycle.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
## ----install-load, results='hide'-------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----setup, include=FALSE-----------------------------------------------------------------------------------------------------------------------------------------------------
knitr::opts_chunk$set(warning = FALSE, message = FALSE)


## ----install-load, results='hide'---------------------------------------------------------------------------------------------------------------------------------------------

#Required R packages
packReq <- c("BiocManager", "rhdf5", 'neonUtilities', 'ggplot2','tidyverse', "lubridate")
Expand All @@ -14,7 +18,7 @@ options(stringsAsFactors=F)



## ----download, results="hide", message=FALSE--------------------------------------------------------------------------------------------------------------------------------------------------
## ----download, results="hide", message=FALSE----------------------------------------------------------------------------------------------------------------------------------

#Target dates
startDate <- "2022-04"
Expand All @@ -24,7 +28,7 @@ endDate <- "2022-09"
site <- c("STEI", "TREE")

#File directory
dirFile <- c(tempdir(),"/home/ddurden/eddy/tmp/tutorial")[2]
dirFile <- tempdir()

zipsByProduct(dpID="DP4.00200.001", package="basic",
site=site,
Expand All @@ -34,20 +38,20 @@ zipsByProduct(dpID="DP4.00200.001", package="basic",



## ----stack-dp04, results="hide"---------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----stack-dp04, results="hide"-----------------------------------------------------------------------------------------------------------------------------------------------

flux <- neonUtilities::stackEddy(filepath=paste0(dirFile,"/filesToStack00200"),
level="dp04")



## ----see-variables----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----see-variables------------------------------------------------------------------------------------------------------------------------------------------------------------

flux$variables



## ----plot-fluxes------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----plot-fluxes--------------------------------------------------------------------------------------------------------------------------------------------------------------

flux$STEI$Site <- "STEI"
flux$TREE$Site <- "TREE"
Expand All @@ -64,13 +68,13 @@ dfFlux %>%



## ----mean-qfqm--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----mean-qfqm----------------------------------------------------------------------------------------------------------------------------------------------------------------

dfFlux %>% group_by(Site) %>% summarise(mean(qfqm.fluxCo2.turb.qfFinl))



## ----plot-qfqm--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----plot-qfqm----------------------------------------------------------------------------------------------------------------------------------------------------------------
dfFlux %>%
select(contains("qfqm")) %>%
pivot_longer(cols = everything(), names_to = "var") %>%
Expand All @@ -85,7 +89,7 @@ dfFlux %>%



## ----qfqm-remove------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----qfqm-remove--------------------------------------------------------------------------------------------------------------------------------------------------------------

dfFlux %>% select(contains("qfqm") & contains("fluxCO2")) %>% summarise_each(sum)
dfFlux %>% select(contains("data") & contains("fluxCO2")) %>% summarise_each(funs(sum(is.na(.))))
Expand All @@ -100,7 +104,7 @@ dfFlux %>% select(contains("data") & contains("fluxCO2")) %>% summarise_each(fun



## ----plot-diel-cycle--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----plot-diel-cycle----------------------------------------------------------------------------------------------------------------------------------------------------------

dfFlux$hour <- factor(lubridate::hour(dfFlux$timeBgn))

Expand All @@ -110,7 +114,7 @@ ggplot(dfFlux, aes(x = hour, y = data.fluxCo2.turb.flux, fill = Site)) +



## ----plot-diel-lst----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----plot-diel-lst------------------------------------------------------------------------------------------------------------------------------------------------------------

fileMeta <- list.files(dirFile, pattern = paste0(".*",site[1],".*.h5"), recursive = TRUE, full.names = TRUE)[1]

Expand All @@ -128,7 +132,7 @@ ggplot(dfFlux, aes(x = hourLst, y = data.fluxCo2.turb.flux, fill = Site)) +



## ----plot-diel-stor---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----plot-diel-stor-----------------------------------------------------------------------------------------------------------------------------------------------------------

ggplot(dfFlux, aes(x = hourLst, y = data.fluxCo2.stor.flux, fill = Site)) +
geom_boxplot() +
Expand All @@ -137,7 +141,7 @@ ggplot(dfFlux, aes(x = hourLst, y = data.fluxCo2.stor.flux, fill = Site)) +



## ----plot-diel-nsae---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----plot-diel-nsae-----------------------------------------------------------------------------------------------------------------------------------------------------------

ggplot(dfFlux, aes(x = hourLst, y = data.fluxCo2.nsae.flux, fill = Site)) +
geom_boxplot() +
Expand All @@ -146,7 +150,7 @@ ggplot(dfFlux, aes(x = hourLst, y = data.fluxCo2.nsae.flux, fill = Site)) +



## ----mean-nsae--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
## ----mean-nsae----------------------------------------------------------------------------------------------------------------------------------------------------------------


dfFlux %>%
Expand Down
19 changes: 8 additions & 11 deletions tutorials/R/eddy-covariance/diel-cycle-flux/flux-diel-cycle.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ tutorialSeries:
urlTitle: eddy-diel-cycle
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```

This data tutorial provides an overview of exploring NEON carbon
flux data, using the `neonUtilities` R package. If you are just starting to work with NEON data it is recommended that you also checkout the more general tutorials, such as the
<a href="https://www.neonscience.org/neonDataStackR" target="_blank">neonUtilities tutorial</a>
Expand Down Expand Up @@ -69,7 +73,7 @@ endDate <- "2022-09"
site <- c("STEI", "TREE")

#File directory
dirFile <- c(tempdir(),"/home/ddurden/eddy/tmp/tutorial")[2]
dirFile <- tempdir()

zipsByProduct(dpID="DP4.00200.001", package="basic",
site=site,
Expand All @@ -84,7 +88,7 @@ zipsByProduct(dpID="DP4.00200.001", package="basic",
There are five levels of data contained in the eddy flux bundle. For full
details, refer to the <a href="https://data.neonscience.org/documents/10179/2403599/NEON.DOC.004571vC/4c72353a-35fb-1136-ef9f-cbdc514711ad" target="_blank">NEON algorithm document</a>.

In this tutorial we will only be focusing on Level 4 (`dp04`) flux data products; however, additional data products from Level 0' (`dp0p`) calibrated raw data to Level 3 (`dp03`) spatially interpolated vertical profiles used to derive the storage flux are available in the EC bundled HDF5 files. Information can be found on the <a href="<a href="https://data.neonscience.org/data-products/DP4.00200.001" target="_blank">NEON algorithm document</a>" target="_blank">NEON Bundled data product - eddy covariance</a> webpage. The <a href="https://www.neonscience.org/resources/learning-hub/tutorials/eddy-data-intro" target="_blank">Introduction to working with NEON eddy flux data</a> tutorial dives into additional detail regarding the other data product levels, but this tutoiral will focus exclusively on flux data.
In this tutorial we will only be focusing on Level 4 (`dp04`) flux data products; however, additional data products from Level 0' (`dp0p`) calibrated raw data to Level 3 (`dp03`) spatially interpolated vertical profiles used to derive the storage flux are available in the EC bundled HDF5 files. Information can be found on the <a href="<a href="https://data.neonscience.org/data-products/DP4.00200.001" target="_blank">NEON algorithm document</a> webpage. The <a href="https://www.neonscience.org/resources/learning-hub/tutorials/eddy-data-intro" target="_blank">Introduction to working with NEON eddy flux data</a> tutorial dives into additional detail regarding the other data product levels, but this tutoiral will focus exclusively on flux data.

To extract the `dp04` data from the HDF5 files and merge them into a
single table, we'll use the `stackEddy()` function. We provide the function to input arguments, `filepath` and `level`. The `filepath` will be the file directory (`dirFile`) used for the data download via `zipsByProduct()` combined with the `filestoStack00200` folder created by the function. To grab just the flux data products `level = dp04`:
Expand Down Expand Up @@ -272,25 +276,18 @@ ggplot(dfFlux, aes(x = hourLst, y = data.fluxCo2.turb.flux, fill = Site)) +
```

That looks better, now we can clearly see the diel cycle of the turbulent
COsub>2</sub> flux at our STEI and TREE sites with peak carbon uptake around noon.
CO<sub>2</sub> flux at our STEI and TREE sites with peak carbon uptake around noon.
The carbon uptake should align with peak solar angle and incoming
photosynthetically active radiation (PAR). The boxplot also provides us information
about the variability of the fluxes throughout the day across the growing season
with interquartile range (IQR) represented by the box and the whiskers indicating
the threshold for outliers (black dots) as $ 1.5 x IQR$ subtracted from 1st quantile (`Q1`)
the threshold for outliers (black dots) as 1.5 * IQR subtracted from 1st quantile (`Q1`)
and added to 3rd quantile (`Q3`):

$$

Q1 - 1.5 * IQR

$$
$$

Q3 + 1.5 * IQR

$$

In the boxplots above the outlier points were greatly reduced after the qfFinal
data removal; however, some outliers remained. To focus our attention on the
bulk of the data, we scaled the y-axis using `scale_y_continuous()` in conjunction
Expand Down
40 changes: 15 additions & 25 deletions tutorials/R/eddy-covariance/diel-cycle-flux/flux-diel-cycle.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,6 @@



<!-- MathJax scripts -->
<script type="text/javascript" src="https://cdn.bootcss.com/mathjax/2.7.0/MathJax.js?config=TeX-MML-AM_CHTML">
</script>


<style type="text/css">
Expand Down Expand Up @@ -215,7 +212,7 @@ <h2>1. Setup</h2>
site &lt;- c(&quot;STEI&quot;, &quot;TREE&quot;)

#File directory
dirFile &lt;- c(tempdir(),&quot;/home/ddurden/eddy/tmp/tutorial&quot;)[2]
dirFile &lt;- tempdir()

zipsByProduct(dpID=&quot;DP4.00200.001&quot;, package=&quot;basic&quot;,
site=site,
Expand All @@ -229,7 +226,7 @@ <h2>2. Stacking Level 4 Flux Data</h2>
<p>There are five levels of data contained in the eddy flux bundle. For full
details, refer to the <a href="https://data.neonscience.org/documents/10179/2403599/NEON.DOC.004571vC/4c72353a-35fb-1136-ef9f-cbdc514711ad" target="_blank">NEON algorithm document</a>.</p>

<p>In this tutorial we will only be focusing on Level 4 (<code>dp04</code>) flux data products; however, additional data products from Level 0&#39; (<code>dp0p</code>) calibrated raw data to Level 3 (<code>dp03</code>) spatially interpolated vertical profiles used to derive the storage flux are available in the EC bundled HDF5 files. Information can be found on the <a href="<a href="https://data.neonscience.org/data-products/DP4.00200.001" target="_blank">NEON algorithm document</a>&ldquo; target=&rdquo;_blank&quot;&gt;NEON Bundled data product - eddy covariance</a> webpage. The <a href="https://www.neonscience.org/resources/learning-hub/tutorials/eddy-data-intro" target="_blank">Introduction to working with NEON eddy flux data</a> tutorial dives into additional detail regarding the other data product levels, but this tutoiral will focus exclusively on flux data.</p>
<p>In this tutorial we will only be focusing on Level 4 (<code>dp04</code>) flux data products; however, additional data products from Level 0&#39; (<code>dp0p</code>) calibrated raw data to Level 3 (<code>dp03</code>) spatially interpolated vertical profiles used to derive the storage flux are available in the EC bundled HDF5 files. Information can be found on the <a href="<a href="https://data.neonscience.org/data-products/DP4.00200.001" target="_blank">NEON algorithm document</a> webpage. The <a href="https://www.neonscience.org/resources/learning-hub/tutorials/eddy-data-intro" target="_blank">Introduction to working with NEON eddy flux data</a> tutorial dives into additional detail regarding the other data product levels, but this tutoiral will focus exclusively on flux data.</p>

<p>To extract the <code>dp04</code> data from the HDF5 files and merge them into a
single table, we&#39;ll use the <code>stackEddy()</code> function. We provide the function to input arguments, <code>filepath</code> and <code>level</code>. The <code>filepath</code> will be the file directory (<code>dirFile</code>) used for the data download via <code>zipsByProduct()</code> combined with the <code>filestoStack00200</code> folder created by the function. To grab just the flux data products <code>level = dp04</code>:</p>
Expand Down Expand Up @@ -383,8 +380,8 @@ <h3>Quality flags</h3>
## # A tibble: 2 x 2
## Site `mean(qfqm.fluxCo2.turb.qfFinl)`
## &lt;fct&gt; &lt;dbl&gt;
## 1 STEI 0.219
## 2 TREE 0.261
## 1 STEI 0.222
## 2 TREE 0.277
</code></pre>

<p>Now, lets plot the qfFinal failed percentage for all our flux data products:</p>
Expand Down Expand Up @@ -415,12 +412,12 @@ <h3>Removing data with failed qfFinal</h3>
<pre><code>dfFlux %&gt;% select(contains(&quot;qfqm&quot;) &amp; contains(&quot;fluxCO2&quot;)) %&gt;% summarise_each(sum)

## qfqm.fluxCo2.nsae.qfFinl qfqm.fluxCo2.stor.qfFinl qfqm.fluxCo2.turb.qfFinl
## 1 9322 7886 3526
## 1 10901 8903 4382

dfFlux %&gt;% select(contains(&quot;data&quot;) &amp; contains(&quot;fluxCO2&quot;)) %&gt;% summarise_each(funs(sum(is.na(.))))

## data.fluxCo2.nsae.flux data.fluxCo2.stor.flux data.fluxCo2.turb.flux
## 1 3174 2334 1366
## 1 3541 2508 1619

dfFlux$data.fluxCo2.turb.flux[(which(dfFlux$qfqm.fluxCo2.turb.qfFinl== 1))] &lt;- NaN
dfFlux$data.fluxCo2.stor.flux[(which(dfFlux$qfqm.fluxCo2.stor.qfFinl== 1))] &lt;- NaN
Expand All @@ -429,12 +426,12 @@ <h3>Removing data with failed qfFinal</h3>
dfFlux %&gt;% select(contains(&quot;qfqm&quot;) &amp; contains(&quot;fluxCO2&quot;)) %&gt;% summarise_each(sum)

## qfqm.fluxCo2.nsae.qfFinl qfqm.fluxCo2.stor.qfFinl qfqm.fluxCo2.turb.qfFinl
## 1 9322 7886 3526
## 1 10901 8903 4382

dfFlux %&gt;% select(contains(&quot;data&quot;) &amp; contains(&quot;fluxCO2&quot;)) %&gt;% summarise_each(funs(sum(is.na(.))))

## data.fluxCo2.nsae.flux data.fluxCo2.stor.flux data.fluxCo2.turb.flux
## 1 9713 7913 4343
## 1 11386 8948 5349
</code></pre>

<p>We see from the summary of the <code>fluxCo2</code> qfFinal and data NA&#39;s, that we have
Expand Down Expand Up @@ -507,13 +504,13 @@ <h2>3. Plotting the diel cycle of CO<sub>2</sub> fluxes</h2>
## [1] 6
##
## $`Pf$AngEnuXaxs`
## [1] 0.01544
## [1] 0.020097
##
## $`Pf$AngEnuYaxs`
## [1] -0.019366
## [1] -0.024191
##
## $`Pf$Ofst`
## [1] 0.05764
## [1] 0.016017
##
## $TimeDiffUtcLt
## [1] -6
Expand Down Expand Up @@ -542,25 +539,18 @@ <h2>3. Plotting the diel cycle of CO<sub>2</sub> fluxes</h2>
<p><img src="https://raw.githubusercontent.com/NEONScience/NEON-Data-Skills/main/tutorials/R/eddy-covariance/diel-cycle-flux/rfigs/plot-diel-lst-1.png" alt=" "/></p>

<p>That looks better, now we can clearly see the diel cycle of the turbulent
COsub&gt;2</sub> flux at our STEI and TREE sites with peak carbon uptake around noon.
CO<sub>2</sub> flux at our STEI and TREE sites with peak carbon uptake around noon.
The carbon uptake should align with peak solar angle and incoming
photosynthetically active radiation (PAR). The boxplot also provides us information
about the variability of the fluxes throughout the day across the growing season
with interquartile range (IQR) represented by the box and the whiskers indicating
the threshold for outliers (black dots) as $ 1.5 x IQR$ subtracted from 1st quantile (<code>Q1</code>)
the threshold for outliers (black dots) as 1.5 * IQR subtracted from 1st quantile (<code>Q1</code>)
and added to 3rd quantile (<code>Q3</code>):</p>

<p>$$</p>

<p>Q1 - 1.5 * IQR </p>

<p>\[
\]</p>

<p>Q3 + 1.5 * IQR</p>

<p>$$</p>

<p>In the boxplots above the outlier points were greatly reduced after the qfFinal
data removal; however, some outliers remained. To focus our attention on the
bulk of the data, we scaled the y-axis using <code>scale_y_continuous()</code> in conjunction
Expand Down Expand Up @@ -607,8 +597,8 @@ <h2>3. Plotting the diel cycle of CO<sub>2</sub> fluxes</h2>
## # A tibble: 2 x 4
## Site data.fluxCo2.nsae.flux data.fluxCo2.stor.flux data.fluxCo2.turb.flux
## &lt;fct&gt; &lt;dbl&gt; &lt;dbl&gt; &lt;dbl&gt;
## 1 STEI -2.72 -0.0259 -3.83
## 2 TREE -2.54 -0.0911 -3.14
## 1 STEI -2.18 -0.0216 -3.19
## 2 TREE -2.04 -0.0716 -2.68
</code></pre>

<p>As expected from the plots, we see that on average the STEI site takes up more
Expand Down
Loading