
Core Functions in tidyquant
MattDancho
2025-11-01
Source:vignettes/TQ01-core-functions-in-tidyquant.RmdTQ01-core-functions-in-tidyquant.RmdA few core functions with a lot of power
Overview
Thetidyquant package has afew core functionswith a lot of power. Few functions means less of a learningcurve for the user, which is why there are only a handful of functionsthe user needs to learn to perform the vast majority of financialanalysis tasks. The main functions are:
Get a Stock Index,
tq_index(), or a StockExchange,tq_exchange(): Returns the stock symbolsand various attributes for every stock in an index or exchange. Eighteenindexes and three exchanges are available.Get Quantitative Data,
tq_get(): Aone-stop shop to get data from various web-sources.Transmute,
tq_transmute(), and Mutate,tq_mutate(), Quantitative Data: Perform and scalefinancial calculations completely within thetidyverse.These workhorse functions integrate thexts,zoo,quantmod, andTTRpackages.Performance analysis,
tq_performance(), andportfolio aggregation,tq_portfolio(): ThePerformanceAnalyticsintegration enables analyzingperformance of assets and portfolios. Because of the breadth of thistopic, refer toPerformanceAnalysis with tidyquant for a tutorial on these functions.
1.0 Retrieve Consolidated Symbol Data
1.1 Stock Indexes
A wide range of stock index / exchange lists can be retrieved usingtq_index(). To get a full list of the options, usetq_index_options().
## [1] "DOW" "DOWGLOBAL" "SP400" "SP500" "SP600"Setx as one of the options in the list of options aboveto get the desired stock index / exchange.
tq_index("SP500")The data source is State Street Global Advisors.
1.2 Stock Exchanges
Stock lists for three stock exchanges are available: NASDAQ, NYSE,and AMEX. If you forget, just usetq_exchange_options(). Wecan easily get the full list of stocks on the NASDAQ exchange.
tq_exchange("NASDAQ")1.0 Get Quantitative Data
Thetq_get() function is used to collect data bychanging theget argument. The data sources:
- Yahoo Finance - Daily stock data
- FRED - Economic data
- Quandl - Economic, Energy, & Financial DataAPI
- Tiingo - Financial API with sub-daily stock dataand crypto-currency
- Alpha Vantage - Financial API with sub-daily,ForEx, and crypto-currency data
- Bloomberg - Financial API. Paid account isrequired.
Usetq_get_options() to see the full list.
## [1] "stock.prices" "stock.prices.japan" "dividends"## [4] "splits" "economic.data" "quandl"## [7] "quandl.datatable" "tiingo" "tiingo.iex"## [10] "tiingo.crypto" "alphavantager" "alphavantage"## [13] "rblpapi"2.1 Yahoo! Finance
The stock prices can be retrieved succinctly usingget = "stock.prices". This returns stock price data fromYahoo Finance.
aapl_prices<-tq_get("AAPL", get="stock.prices", from=" 1990-01-01")aapl_prices### A tibble: 9,026 × 8## symbol date open high low close volume adjusted##<chr><date><dbl><dbl><dbl><dbl><dbl><dbl>## 1 AAPL 1990-01-02 0.315 0.335 0.312 0.333 183198400 0.261## 2 AAPL 1990-01-03 0.339 0.339 0.335 0.335 207995200 0.263## 3 AAPL 1990-01-04 0.342 0.346 0.333 0.336 221513600 0.264## 4 AAPL 1990-01-05 0.337 0.342 0.330 0.337 123312000 0.265## 5 AAPL 1990-01-08 0.335 0.339 0.330 0.339 101572800 0.266## 6 AAPL 1990-01-09 0.339 0.339 0.330 0.336 86139200 0.264## 7 AAPL 1990-01-10 0.336 0.336 0.319 0.321 199718400 0.252## 8 AAPL 1990-01-11 0.324 0.324 0.308 0.308 211052800 0.242## 9 AAPL 1990-01-12 0.306 0.310 0.301 0.308 171897600 0.242##10 AAPL 1990-01-15 0.308 0.319 0.306 0.306 161739200 0.240### ℹ 9,016 more rowsWe can get multiple stocks:
### A tibble: 3,024 × 8## symbol date open high low close volume adjusted##<chr><date><dbl><dbl><dbl><dbl><dbl><dbl>## 1 AAPL 2013-01-02 19.8 19.8 19.3 19.6 560518000 16.6## 2 AAPL 2013-01-03 19.6 19.6 19.3 19.4 352965200 16.4## 3 AAPL 2013-01-04 19.2 19.2 18.8 18.8 594333600 16.0## 4 AAPL 2013-01-07 18.6 18.9 18.4 18.7 484156400 15.9## 5 AAPL 2013-01-08 18.9 19.0 18.6 18.8 458707200 15.9## 6 AAPL 2013-01-09 18.7 18.8 18.4 18.5 407604400 15.7## 7 AAPL 2013-01-10 18.9 18.9 18.4 18.7 601146000 15.9## 8 AAPL 2013-01-11 18.6 18.8 18.5 18.6 350506800 15.8## 9 AAPL 2013-01-14 18.0 18.1 17.8 17.9 734207600 15.2##10 AAPL 2013-01-15 17.8 17.8 17.3 17.4 876772400 14.7### ℹ 3,014 more rowsYahoo Japan stock prices can be retrieved using a similar call,get = "stock.prices.japan".
x8411T<-tq_get("8411.T", get="stock.prices.japan", from="2016-01-01", to="2016-12-31")The data source is Yahoo Finance (https://finance.yahoo.com/) and Yahoo Finance Japan.
2.2 FRED Economic Data
A wealth of economic data can be extracted from the Federal ReserveEconomic Data (FRED) database. The FRED contains over 10K data sets thatare free to use. See theFRED categories tonarrow down the data base and to get data codes. TheWTI Crude OilPrices are shown below.
wti_price_usd<-tq_get("DCOILWTICO", get="economic.data")wti_price_usd### A tibble: 2,823 × 3## symbol date price##<chr><date><dbl>## 1 DCOILWTICO 2015-01-01NA## 2 DCOILWTICO 2015-01-02 52.7## 3 DCOILWTICO 2015-01-05 50.0## 4 DCOILWTICO 2015-01-06 48.0## 5 DCOILWTICO 2015-01-07 48.7## 6 DCOILWTICO 2015-01-08 48.8## 7 DCOILWTICO 2015-01-09 48.4## 8 DCOILWTICO 2015-01-12 46.1## 9 DCOILWTICO 2015-01-13 45.9##10 DCOILWTICO 2015-01-14 48.5### ℹ 2,813 more rows2.3 Nasdaq Data Link (Quandl) API
Quandl provides access to avast number of financial and economic databases. The Quandl packagesmust be installed separately.
install.packages("Quandl")Authentication
To make full use of the integration we recommend you set your apikey. To do this create or sign into your Quandl account and go to youraccount api key page.
quandl_api_key("<your-api-key>")Search
Searching Quandl from within the R console is possible withquandl_search(), a wrapper forQuandl::Quandl.search(). An example search is shown below.The only required argument isquery. You can also visit theQuandl Search webpage tosearch for available database codes.
quandl_search(query="Oil", database_code="NSE", per_page=3)Getting Quandl Data
Getting data is integrated intotq_get(). Two getoptions exist to retrieve Quandl data:
get = "quandl": Get’s Quandl time series data. Awrapper forQuandl().get = "quandl.datatable": Gets Quandl datatables(larger data sets that may not be time series). A wrapper forQuandl.datatable().
Getting data from Quandl can be achieved in much the same way as theother “get” options. Just pass the “codes” for the data along withdesired arguments for the underlying function.
The following usesget = "quandl" and the “WIKI”database to download daily stock prices for AAPL in 2016. The output isa tidy data frame.
The following time series options are available to be passed to theunderlyingQuandl() function:
start_date(from) = “yyyy-mm-dd” |end_date(to) = “yyyy-mm-dd”column_index= numeric column number (e.g. 1)rows= numeric row number indicating first n rows(e.g. 100)collapse= “none”, “daily”, “weekly”, “monthly”,“quarterly”, “annual”transform= “none”, “diff”, “rdiff”, “cumul”,“normalize”
Here’s an example to get period returns of the adj.close (columnindex 11) using thecolumn_index,collapse andtransform arguments.
"WIKI/AAPL"%>%tq_get(get="quandl", from="2007-01-01", to="2016-12-31", column_index=11, collapse="annual", transform="rdiff")Datatables are larger data sets. These can be downloaded usingget = "quandl.datatable". Note that the time seriesarguments do not work with data tables.
Here’s several examples ofZacksFundamentals Collection B
# Zacks Fundamentals Collection B (DOW 30 Available to non subscribers)tq_get("ZACKS/FC", get="quandl.datatable")# Zacks Fundamentals Condensedtq_get("ZACKS/FR", get="quandl.datatable")# Zacks Fundamental Ratiostq_get("ZACKS/MT", get="quandl.datatable")# Zacks Master Tabletq_get("ZACKS/MKTV", get="quandl.datatable")# Zacks Market Value Supplementtq_get("ZACKS/SHRS", get="quandl.datatable")# Zacks Shares Out Supplement2.4 Tiingo API
The Tiingo API is a free source for stock prices, cryptocurrencies,and intraday feeds from the IEX (Investors Exchange). This can serve asan alternate source of data to Yahoo! Finance.
Authentication
To make full use of the integration you need to get an API key andthen set your api key. If you don’t have one already, go to Tiingoaccount and get your FREE API key. You can then set it as follows:
tiingo_api_key('<your-api-key>')Getting Tiingo Data
Thetidyquant package provides convenient wrappers totheriingo package (R interface to Tiingo). Here’s howtq_get() maps toriingo:
- Tiingo Prices:
tq_get(get = "tiingo") = riingo::riingo_prices() - Tiingo IEX Data:
tq_get(get = "tiingo.iex") = riingo::riingo_iex_prices() - Tiingo Crypto Data:
tq_get(get = "tiingo.crypto") = riingo::riingo_crypto_prices()
# Tiingo Prices (Free alternative to Yahoo Finance!)tq_get(c("AAPL","GOOG"), get="tiingo", from="2010-01-01")# Sub-daily prices from IEX ----tq_get(c("AAPL","GOOG"), get="tiingo.iex", from="2020-01-01", to="2020-01-15", resample_frequency="5min")# Tiingo Bitcoin in USD ----tq_get(c("btcusd"), get="tiingo.crypto", from="2020-01-01", to="2020-01-15", resample_frequency="5min")2.5 Alpha Vantage API
Alpha Vantage providesaccess to a real-time and historical financial data. Thealphavantager package, a lightweight R interface, has beenintegrated intotidyquant as follows. The benefit of theintegration is thescalability since we can now get multiplesymbols returned in a tidy format. You will need to install itfirst.
install.packages("alphavantager")Authentication
To make full use of the integration you need to get an API key andthen set your api key. If you don’t have one already, go toAlpha Vantage account and getyour FREE API key. You can then set it as follows:
# install.packages("alphavantager")av_api_key("<your-api-key>")Getting Alpha Vantage Data
Getting data is simple as the structure follows theAlpha Vantage APIdocumentation. For example, if you wish to retrieve intraday data at5 minute intervals for META and MSFT, you can build the parametersx = c("META", "MSFT"), get = "alphavantager", av_fun = "TIME_SERIES_INTRADAY", interval = "5min".The familiarx andget are the same as youalways use. Theav_fun argument comes fromalphavantager::av_get() and the Alpha Vantagedocumentation. Theinterval argument comes from the docs aswell.
2.6 Bloomberg
Bloombergprovides access to arguably the most comprehensive financial data and isactively used by most major financial institutions that work withfinancial data. TheRblpapi package, an R interface toBloomberg, has been integrated intotidyquant as follows.The benefit of the integration is thescalability since we cannow get multiple symbols returned in a tidy format.
Authentication
To make full use of the integration you need to have a BloombergTerminal account (Note this is not a free service). If you haveBloomberg Terminal running on your machine, you can connect asfollows:
# install.packages("Rblpapi")Rblpapi::blpConnect()Getting Bloomberg Data
Getting data is simple as the structure follows theRblpapi APIdocumentation. For example, if you wish to retrieve monthly data forSPX Index and AGTHX Equity, you can build thetq_getparameters as follows:
x = c('SPX Index','ODMAX Equity')get = "rblpapi"rblpapi_fun = "bdh"Note that “bdh” is the default, andoptions include “bdh” (Bloomberg Data History), “bds” (Bloomberg DataSet), and “bdp” (Bloomberg Data Point)from / toThese get passed tostart.dateandend.dateand can be provided in “YYYY-MM-DD” characterformat. Note thatstart.dateandend.datefromRblpapican be used but must be converted to date ordatetime.- Other arguments: These are options that depend on the
rblpapi_fun. SeeRblpapidocumentation.
3.0 Mutate Quantitative Data
Mutating functions enable thexts/zoo,quantmod andTTR functions to shine. We’lltouch on the mutation functions briefly using theFANG dataset, which consists of daily prices for META, AMZN, GOOG, and NFLX fromthe beginning of 2013 to the end of 2016. We’ll apply the functions togrouped data sets to get a feel for how each works
FANG### A tibble: 4,032 × 8## symbol date open high low close volume adjusted##<chr><date><dbl><dbl><dbl><dbl><dbl><dbl>## 1 META 2013-01-02 27.4 28.2 27.4 28 69846400 28## 2 META 2013-01-03 27.9 28.5 27.6 27.8 63140600 27.8## 3 META 2013-01-04 28.0 28.9 27.8 28.8 72715400 28.8## 4 META 2013-01-07 28.7 29.8 28.6 29.4 83781800 29.4## 5 META 2013-01-08 29.5 29.6 28.9 29.1 45871300 29.1## 6 META 2013-01-09 29.7 30.6 29.5 30.6 104787700 30.6## 7 META 2013-01-10 30.6 31.5 30.3 31.3 95316400 31.3## 8 META 2013-01-11 31.3 32.0 31.1 31.7 89598000 31.7## 9 META 2013-01-14 32.1 32.2 30.6 31.0 98892800 31.0##10 META 2013-01-15 30.6 31.7 29.9 30.1 173242600 30.1### ℹ 4,022 more rowsFor a detailed walkthrough of the compatible functions, see the nextvignette in the series,R Quantitative AnalysisPackage Integrations in tidyquant.
3.1 Transmute Quantitative Data, tq_transmute
Transmute the results oftq_get(). Transmute here holdsalmost the same meaning as indplyr, only the newly createdcolumns will be returned, but withtq_transmute(), thenumber of rows returned can be different than the original data frame.This is important for changing periodicity. An example is periodicityaggregation from daily to monthly.
FANG%>%group_by(symbol)%>%tq_transmute(select=adjusted, mutate_fun=to.monthly, indexAt="lastof")### A tibble: 192 × 3### Groups: symbol [4]## symbol date adjusted##<chr><date><dbl>## 1 META 2013-01-31 31.0## 2 META 2013-02-28 27.2## 3 META 2013-03-31 25.6## 4 META 2013-04-30 27.8## 5 META 2013-05-31 24.4## 6 META 2013-06-30 24.9## 7 META 2013-07-31 36.8## 8 META 2013-08-31 41.3## 9 META 2013-09-30 50.2##10 META 2013-10-31 50.2### ℹ 182 more rowsLet’s go through what happened.select allows you toeasily choose what columns get passed tomutate_fun. Inexample above,adjusted selects the “adjusted” column fromdata, and sends it to the mutate function,to.monthly, which mutates the periodicity from daily tomonthly. Additional arguments can be passed to themutate_fun by way of.... We are passing theindexAt argument to return a date that matches the firstdate in the period.
Working with non-OHLC data
Returns from FRED, Oanda, and other sources do not have open, high,low, close (OHLC) format. However, this is not a problem withselect. The following example shows how to transmute WTICrude daily prices to monthly prices. Since we only have a single columnto pass, we can leave theselect argument asNULL which selects all columns by default. This sends theprice column to theto.period mutate function.
wti_prices<-tq_get("DCOILWTICO", get="economic.data")wti_prices%>%tq_transmute(mutate_fun=to.period, period="months", col_rename="WTI Price")### A tibble: 130 × 2## date `WTI Price`##<date><dbl>## 1 2015-01-30 47.8## 2 2015-02-27 49.8## 3 2015-03-31 47.7## 4 2015-04-30 59.6## 5 2015-05-29 60.2## 6 2015-06-30 59.5## 7 2015-07-31 47.1## 8 2015-08-31 49.2## 9 2015-09-30 45.1##10 2015-10-30 46.6### ℹ 120 more rows3.2 Mutate Quantitative Data, tq_mutate
Adds a column or set of columns to the tibble with the calculatedattributes (hence the original tibble is returned, mutated with theadditional columns). An example is getting theMACD fromclose, which mutates the original input by adding MACD andSignal columns. Note that we can quickly rename the columns using thecol_rename argument.
### A tibble: 4,032 × 10### Groups: symbol [4]## symbol date open high low close volume adjusted MACD Signal##<chr><date><dbl><dbl><dbl><dbl><dbl><dbl><dbl><dbl>## 1 META 2013-01-02 27.4 28.2 27.4 28 69846400 28NANA## 2 META 2013-01-03 27.9 28.5 27.6 27.8 63140600 27.8NANA## 3 META 2013-01-04 28.0 28.9 27.8 28.8 72715400 28.8NANA## 4 META 2013-01-07 28.7 29.8 28.6 29.4 83781800 29.4NANA## 5 META 2013-01-08 29.5 29.6 28.9 29.1 45871300 29.1NANA## 6 META 2013-01-09 29.7 30.6 29.5 30.6 104787700 30.6NANA## 7 META 2013-01-10 30.6 31.5 30.3 31.3 95316400 31.3NANA## 8 META 2013-01-11 31.3 32.0 31.1 31.7 89598000 31.7NANA## 9 META 2013-01-14 32.1 32.2 30.6 31.0 98892800 31.0NANA##10 META 2013-01-15 30.6 31.7 29.9 30.1 173242600 30.1NANA### ℹ 4,022 more rowsNote that a mutation can occur if, and only if, the mutation has thesame structure of the original tibble. In other words, the calculationmust have the same number of rows and row.names (or date fields),otherwise the mutation cannot be performed.
Mutate rolling regressions with rollapply
A very powerful example is applyingcustom functionsacross a rolling window usingrollapply. A specific exampleis using therollapply function to compute a rollingregression. This example is slightly more complicated so it will bebroken down into three steps:
- Get returns
- Create a custom function
- Apply the custom function across a rolling window using
tq_mutate(mutate_fun = rollapply)
Step 1: Get Returns
First, get combined returns. The asset and baseline returns should bein wide format, which is needed for thelm function in thenext step.
fb_returns<-tq_get("META", get="stock.prices", from="2016-01-01", to="2016-12-31")%>%tq_transmute(adjusted,periodReturn, period="weekly", col_rename="fb.returns")xlk_returns<-tq_get("XLK", from="2016-01-01", to="2016-12-31")%>%tq_transmute(adjusted,periodReturn, period="weekly", col_rename="xlk.returns")returns_combined<-left_join(fb_returns,xlk_returns, by="date")returns_combined### A tibble: 52 × 3## date fb.returns xlk.returns##<date><dbl><dbl>## 1 2016-01-08 -0.0478 -0.0516## 2 2016-01-15 -0.0242 -0.0187## 3 2016-01-22 0.0313 0.0264## 4 2016-01-29 0.146 0.0213## 5 2016-02-05 -0.0725 -0.0422## 6 2016-02-12 -0.0198 -0.00582## 7 2016-02-19 0.0251 0.0354## 8 2016-02-26 0.0320 0.0148## 9 2016-03-04 0.00436 0.0281##10 2016-03-11 0.00941 0.0106### ℹ 42 more rowsStep 2: Create a custom function
Next, create a custom regression function, which will be used toapply over the rolling window in Step 3. An important point is that the“data” will be passed to the regression function as anxtsobject. Thetimetk::tk_tbl function takes care ofconverting to a data frame for thelm function to workproperly with the columns “fb.returns” and “xlk.returns”.
Step 3: Apply the custom function
Now we can usetq_mutate() to apply the customregression function over a rolling window usingrollapplyfrom thezoo package. Internally, since we leftselect = NULL, thereturns_combined data frameis being passed automatically to thedata argument of therollapply function. All you need to specify is themutate_fun = rollapply and any additional argumentsnecessary to apply therollapply function. We’ll specify a12 week window viawidth = 12. TheFUNargument is our custom regression function,regr_fun. It’sextremely important to specifyby.column = FALSE, whichtellsrollapply to perform the computation using the dataas a whole rather than apply the function to each column independently.Thecol_rename argument is used to rename the addedcolumns.
returns_combined%>%tq_mutate(mutate_fun=rollapply, width=12, FUN=regr_fun, by.column=FALSE, col_rename=c("coef.0","coef.1"))### A tibble: 52 × 5## date fb.returns xlk.returns coef.0 coef.1##<date><dbl><dbl><dbl><dbl>## 1 2016-01-08 -0.0478 -0.0516NANA## 2 2016-01-15 -0.0242 -0.0187NANA## 3 2016-01-22 0.0313 0.0264NANA## 4 2016-01-29 0.146 0.0213NANA## 5 2016-02-05 -0.0725 -0.0422NANA## 6 2016-02-12 -0.0198 -0.00582NANA## 7 2016-02-19 0.0251 0.0354NANA## 8 2016-02-26 0.0320 0.0148NANA## 9 2016-03-04 0.00436 0.0281NANA##10 2016-03-11 0.00941 0.0106NANA### ℹ 42 more rowsreturns_combined### A tibble: 52 × 3## date fb.returns xlk.returns##<date><dbl><dbl>## 1 2016-01-08 -0.0478 -0.0516## 2 2016-01-15 -0.0242 -0.0187## 3 2016-01-22 0.0313 0.0264## 4 2016-01-29 0.146 0.0213## 5 2016-02-05 -0.0725 -0.0422## 6 2016-02-12 -0.0198 -0.00582## 7 2016-02-19 0.0251 0.0354## 8 2016-02-26 0.0320 0.0148## 9 2016-03-04 0.00436 0.0281##10 2016-03-11 0.00941 0.0106### ℹ 42 more rowsAs shown above, the rolling regression coefficients were added to thedata frame.
3.3 _xy Variants, tq_mutate_xy and tq_transmute_xy
Enables working with mutation functions that require two primaryinputs (e.g. EVWMA, VWAP, etc).
Mutate with two primary inputs
EVWMA (exponential volume-weighted moving average) requires twoinputs, price and volume. To work with these columns, we can switch tothe xy variants,tq_transmute_xy() andtq_mutate_xy(). The only difference is instead of theselect argument, you usex andyarguments to pass the columns needed based on themutate_fun documentation.
FANG%>%group_by(symbol)%>%tq_mutate_xy(x=close, y=volume, mutate_fun=EVWMA, col_rename="EVWMA")### A tibble: 4,032 × 9### Groups: symbol [4]## symbol date open high low close volume adjusted EVWMA##<chr><date><dbl><dbl><dbl><dbl><dbl><dbl><dbl>## 1 META 2013-01-02 27.4 28.2 27.4 28 69846400 28NA## 2 META 2013-01-03 27.9 28.5 27.6 27.8 63140600 27.8NA## 3 META 2013-01-04 28.0 28.9 27.8 28.8 72715400 28.8NA## 4 META 2013-01-07 28.7 29.8 28.6 29.4 83781800 29.4NA## 5 META 2013-01-08 29.5 29.6 28.9 29.1 45871300 29.1NA## 6 META 2013-01-09 29.7 30.6 29.5 30.6 104787700 30.6NA## 7 META 2013-01-10 30.6 31.5 30.3 31.3 95316400 31.3NA## 8 META 2013-01-11 31.3 32.0 31.1 31.7 89598000 31.7NA## 9 META 2013-01-14 32.1 32.2 30.6 31.0 98892800 31.0NA##10 META 2013-01-15 30.6 31.7 29.9 30.1 173242600 30.1 30.1### ℹ 4,022 more rows