CSV Data Feed Development

backtrader already offers a Generic CSV Data feed and some specific CSV DataFeeds. Summarizing:

• GenericCSVData

• VisualChartCSVData

• YahooFinanceData (for online downloads)

• YahooFinanceCSVData (for already downloaded data)

• BacktraderCSVData (in-house … for testing purposed, but can be used)

But even with that, the end user may wish to develop support for a specific CSVData Feed.

The usual motto would be: “It’s easier said than done”. Actually the structureis meant to make it easy.

Steps:

• Inherit frombacktrader.CSVDataBase

• Define anyparams if needed

• Do any initialization in thestart method

• Do any clean-up in thestop method

• Define a_loadline method where the actual work happens

  This method receives a single argument: linetokens.

  As the name suggests this contains the tokens after the current line hasbeen splitten according to theseparator parameter (inherited from thebase class)

  If after doing its work there is new data … fill up the correspondinglines and returnTrue

  If nothing is available and therefore the parsing has come to an end: returnFalse

  ReturningFalse may not even be needed if the behind the scenes codewhich is reading the file lines finds out there are no more lines to parse.

Things which are already taken into account:

• Opening the file (or receiving a file-like object)

• Skipping the headers row if indicated as present

• Reading the lines

• Tokenizing the lines

• Preloading support (to load the entire data feed at once in memory)

Usually an example is worth a thousand requirement descriptions. Let’s use asimplified version of the in-house defined CSV parsing code fromBacktraderCSVData. This one needs no initialization or clean-up (this couldbe opening a socket and closing it later, for example).

If your strategy/algorithm or simple data perusal only needs, for example theclosing prices you can leave the others untouched (each iteration fills themautomatically with a float(‘NaN’) value before the end user code has a chanceto do anything.

In this example only a daily format is supported:

importitertools...importbacktraderasbtclassMyCSVData(bt.CSVDataBase):defstart(self):# Nothing to do for this data feed typepassdefstop(self):# Nothing to do for this data feed typepassdef_loadline(self,linetokens):i=itertools.count(0)dttxt=linetokens[next(i)]# Format is YYYY-MM-DDy=int(dttxt[0:4])m=int(dttxt[5:7])d=int(dttxt[8:10])dt=datetime.datetime(y,m,d)dtnum=date2num(dt)self.lines.datetime[0]=dtnumself.lines.open[0]=float(linetokens[next(i)])self.lines.high[0]=float(linetokens[next(i)])self.lines.low[0]=float(linetokens[next(i)])self.lines.close[0]=float(linetokens[next(i)])self.lines.volume[0]=float(linetokens[next(i)])self.lines.openinterest[0]=float(linetokens[next(i)])returnTrue

The code expects all fields to be in place and be convertible to floats, exceptfor the datetime which has a fixed YYYY-MM-DD format and can be parsed withoutusingdatetime.datetime.strptime.

More complex needs can be covered by adding just a few lines of code to accountfor null values, date format parsing. TheGenericCSVData does that.

Caveat Emptor

Using theGenericCSVData existing feed and inheritance a lot can beacomplished in order to support formats.