ropensci/tidyhydatPublic

NotificationsYou must be signed in to change notification settings
Fork22
Star71

An R package to import Water Survey of Canada hydrometric data and make it tidy

License

Apache-2.0 license

71 stars 22 forks Branches Tags Activity

Star

Notifications

You must be signed in to change notification settings

Branches Tags

Folders and files

Name		Name	Last commit message	Last commit date
Latest commit History 746 Commits
.github		.github
R		R
data-raw		data-raw
data		data
inst		inst
man		man
presentations		presentations
revdep		revdep
tests		tests
vignettes		vignettes
.Rbuildignore		.Rbuildignore
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
CRAN-SUBMISSION		CRAN-SUBMISSION
DESCRIPTION		DESCRIPTION
LICENSE		LICENSE
NAMESPACE		NAMESPACE
NEWS.md		NEWS.md
README.Rmd		README.Rmd
README.md		README.md
cran-comments.md		cran-comments.md
release_process.md		release_process.md
tidyhydat.Rproj		tidyhydat.Rproj
use_cases.md		use_cases.md

Repository files navigation

tidyhydat

What does`tidyhydat` do?

Provides functions (hy_*) that access hydrometric data from theHYDAT database, a national archive of Canadian hydrometric data andreturn tidy data.
Provides functions (realtime_*) that access Environment and ClimateChange Canada’s real-time hydrometric data source.
Provides functions (ws_*) that access historical daily data via webservice without requiring HYDAT download.
Provides functions (search_*) that can search through theapproximately 7000 stations in the database and aid in generatingstation vectors
Keep functions as simple as possible. For example, for daily flows,thehy_daily_flows() function queries the database,tidies thedata and returns atibble of dailyflows.

Installation

You can installtidyhydat from CRAN:

install.packages("tidyhydat")

To install the development version of thetidyhydat package, you caninstall directly from the rOpenSci development server:

install.packages("tidyhydat", repos = "https://dev.ropensci.org")

Usage

More documentation ontidyhydat can found at the rOpenSci doc page:https://docs.ropensci.org/tidyhydat/

When you installtidyhydat, several other packages will be installedas well. One of those packages,dplyr, is useful for datamanipulations and is used regularly here. To use actually usedplyr ina session you must explicitly load it. A helpfuldplyr tutorial can befoundhere.

library(tidyhydat)library(dplyr)

HYDAT download

To use many of the functions in thetidyhydat package you will need todownload a version of the HYDAT database, Environment and Climate ChangeCanada’s database of historical hydrometric data then tell R where tofind the database. Convenientlytidyhydat does all this for you via:

download_hydat()

This downloads (with your permission) the most recent version of HYDATand then saves it in a location on your computer wheretidyhydat’sfunction will look for it. Do be patient though as this can take a longtime! To see where HYDAT was saved you can runhy_default_db(). Nowthat you have HYDAT downloaded and ready to go, you are all set to beginlooking at Canadian hydrometric data.

Real-time

To download real-time data using the datamart we can use approximatelythe same conventions discussed above. Usingrealtime_dd() we caneasily select specific stations by supplying a station of interest:

realtime_dd(station_number = "08MF005")#>   Queried on: 2025-10-29 17:49:41.08772 (UTC)#>   Date range: 2025-09-29 to 2025-10-29 #> # A tibble: 17,502 × 8#>    STATION_NUMBER PROV_TERR_STATE_LOC Date                Parameter Value Grade#>    <chr>          <chr>               <dttm>              <chr>     <dbl> <chr>#>  1 08MF005        BC                  2025-09-29 08:00:00 Flow       1390 <NA> #>  2 08MF005        BC                  2025-09-29 08:05:00 Flow       1390 <NA> #>  3 08MF005        BC                  2025-09-29 08:10:00 Flow       1390 <NA> #>  4 08MF005        BC                  2025-09-29 08:15:00 Flow       1390 <NA> #>  5 08MF005        BC                  2025-09-29 08:20:00 Flow       1390 <NA> #>  6 08MF005        BC                  2025-09-29 08:25:00 Flow       1390 <NA> #>  7 08MF005        BC                  2025-09-29 08:30:00 Flow       1390 <NA> #>  8 08MF005        BC                  2025-09-29 08:35:00 Flow       1400 <NA> #>  9 08MF005        BC                  2025-09-29 08:40:00 Flow       1390 <NA> #> 10 08MF005        BC                  2025-09-29 08:45:00 Flow       1390 <NA> #> # ℹ 17,492 more rows#> # ℹ 2 more variables: Symbol <chr>, Code <chr>

Or we can userealtime_ws:

realtime_ws(  station_number = "08MF005",  parameters = c(46, 5), ## see param_id for a list of codes  start_date = Sys.Date() - 14,  end_date = Sys.Date())#>   Queried on: 2025-10-29 17:49:42.453161 (UTC)#>   Date range: 2025-10-15 to 2025-10-29 #>   Station(s) returned: 1#>   All stations successfully retrieved.#>   All parameters successfully retrieved.#> # A tibble: 4,593 × 12#>    STATION_NUMBER Date                Name_En  Value Unit  Grade Symbol Approval#>    <chr>          <dttm>              <chr>    <dbl> <chr> <lgl> <chr>  <chr>   #>  1 08MF005        2025-10-15 00:00:00 Water t…  12.1 °C    NA    <NA>   Provisi…#>  2 08MF005        2025-10-15 01:00:00 Water t…  12.1 °C    NA    <NA>   Provisi…#>  3 08MF005        2025-10-15 02:00:00 Water t…  12.1 °C    NA    <NA>   Provisi…#>  4 08MF005        2025-10-15 03:00:00 Water t…  12.1 °C    NA    <NA>   Provisi…#>  5 08MF005        2025-10-15 04:00:00 Water t…  12.0 °C    NA    <NA>   Provisi…#>  6 08MF005        2025-10-15 05:00:00 Water t…  12.0 °C    NA    <NA>   Provisi…#>  7 08MF005        2025-10-15 06:00:00 Water t…  12.0 °C    NA    <NA>   Provisi…#>  8 08MF005        2025-10-15 07:00:00 Water t…  12.0 °C    NA    <NA>   Provisi…#>  9 08MF005        2025-10-15 08:00:00 Water t…  12.0 °C    NA    <NA>   Provisi…#> 10 08MF005        2025-10-15 09:00:00 Water t…  12.0 °C    NA    <NA>   Provisi…#> # ℹ 4,583 more rows#> # ℹ 4 more variables: Parameter <dbl>, Code <chr>, Qualifier <chr>,#> #   Qualifiers <lgl>

Historical web service

For smaller queries where downloading the entire HYDAT database isunnecessary,tidyhydat provides web service functions that accesshistorical daily data directly:

ws_daily_flows(  station_number = "08MF005",  start_date = "2020-01-01",  end_date = "2020-12-31")#>   Queried on: 2025-10-29 17:49:43.567461 (UTC)#>   Date range: 2020-01-01 to 2020-12-31 #>   Station(s) returned: 1#>   All stations successfully retrieved.#> # A tibble: 366 × 5#>    STATION_NUMBER Date       Parameter       Value Symbol#>    <chr>          <date>     <chr>           <dbl> <chr> #>  1 08MF005        2020-01-01 discharge/débit  1340 <NA>  #>  2 08MF005        2020-01-02 discharge/débit  1330 <NA>  #>  3 08MF005        2020-01-03 discharge/débit  1310 <NA>  #>  4 08MF005        2020-01-04 discharge/débit  1420 <NA>  #>  5 08MF005        2020-01-05 discharge/débit  1350 <NA>  #>  6 08MF005        2020-01-06 discharge/débit  1310 <NA>  #>  7 08MF005        2020-01-07 discharge/débit  1280 <NA>  #>  8 08MF005        2020-01-08 discharge/débit  1320 <NA>  #>  9 08MF005        2020-01-09 discharge/débit  1230 <NA>  #> 10 08MF005        2020-01-10 discharge/débit  1210 <NA>  #> # ℹ 356 more rows

Similarly for water levels:

ws_daily_levels(  station_number = "08MF005",  start_date = "2020-01-01",  end_date = "2020-12-31")#>   Queried on: 2025-10-29 17:49:44.687726 (UTC)#>   Date range: 2020-01-01 to 2020-12-31 #>   Station(s) returned: 1#>   All stations successfully retrieved.#> # A tibble: 366 × 5#>    STATION_NUMBER Date       Parameter          Value Symbol#>    <chr>          <date>     <chr>              <dbl> <chr> #>  1 08MF005        2020-01-01 water level/niveau  3.72 <NA>  #>  2 08MF005        2020-01-02 water level/niveau  3.72 <NA>  #>  3 08MF005        2020-01-03 water level/niveau  3.69 <NA>  #>  4 08MF005        2020-01-04 water level/niveau  3.81 <NA>  #>  5 08MF005        2020-01-05 water level/niveau  3.74 <NA>  #>  6 08MF005        2020-01-06 water level/niveau  3.69 <NA>  #>  7 08MF005        2020-01-07 water level/niveau  3.66 <NA>  #>  8 08MF005        2020-01-08 water level/niveau  3.70 <NA>  #>  9 08MF005        2020-01-09 water level/niveau  3.60 <NA>  #> 10 08MF005        2020-01-10 water level/niveau  3.58 <NA>  #> # ℹ 356 more rows

Compare realtime_ws and realtime_dd

tidyhydat provides two methods to download realtime data.realtime_dd() provides a function to import .csv files fromhere.realtime_ws() isan client for a web service hosted by ECCC.realtime_ws() has severaldifference torealtime_dd(). These include:

Speed: Therealtime_ws() is much faster for larger queries(i.e. many stations). For single station queries torealtime_dd() ismore appropriate.
Length of record:realtime_ws() records goes back further in time.
Type of parameters:realtime_dd() are restricted to river flow(either flow and level) data. In contrastrealtime_ws() can downloadseveral different parameters depending on what is available for thatstation. Seedata("param_id") for a list and explanation of theparameters.
Date/Time filtering:realtime_ws() provides argument to select adate range. Selecting a data range withrealtime_dd() is notpossible until after all files have been downloaded.

Plotting

Plot methods are also provided to quickly visualize realtime data:

realtime_ex <- realtime_dd(station_number = "08MF005")plot(realtime_ex)

and also historical data:

hy_ex <- hy_daily_flows(station_number = "08MF005", start_date = "2013-01-01")plot(hy_ex)

Getting Help or Reporting an Issue

To report bugs/issues/feature requests, please file anissue.

These are very welcome!

How to Contribute

If you would like to contribute to the package, please see ourCONTRIBUTINGguidelines.

Please note that this project is released with aContributor Code ofConduct.By participating in this project you agree to abide by its terms.

Citation

Get citation information fortidyhydat in R by running:

To cite package 'tidyhydat' in publications use:  Albers S (2017). "tidyhydat: Extract and Tidy Canadian Hydrometric  Data." _The Journal of Open Source Software_, *2*(20).  doi:10.21105/joss.00511 <https://doi.org/10.21105/joss.00511>,  <http://dx.doi.org/10.21105/joss.00511>.A BibTeX entry for LaTeX users is  @Article{,    title = {tidyhydat: Extract and Tidy Canadian Hydrometric Data},    author = {Sam Albers},    doi = {10.21105/joss.00511},    url = {http://dx.doi.org/10.21105/joss.00511},    year = {2017},    publisher = {The Open Journal},    volume = {2},    number = {20},    journal = {The Journal of Open Source Software},  }

License

Licensed under the Apache License, Version 2.0 (the “License”); you maynot use this file except in compliance with the License. You may obtaina copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an “AS IS” BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.