Repository files navigation

A collection of postprocessinghelper functions and examples forFlat Data.

These examples and functions are written inDeno, a new language created by the same founders of Node.js and meant to improve on many aspects of Node.

Note: If you're noticing your scripts failing recently, try updating to0.0.15.More info here

When writing aFlat Data Action, you can specify a path to a postprocessing Deno script that can manipulate the data downloaded by Flat even further.

-name:Fetch datauses:githubocto/flat@v2with:http_url:http://api.coindesk.com/v2/bpi/currentprice.json# The endpoint to fetchdownloaded_filename:btc-price.json# The http_url gets saved and renamed in our repository as btc-price.jsonpostprocess:postprocess.ts# A postprocessing javascript or typescript file written in Deno

This is an example of a postprocessing script. Notice the use ofDeno.args[0] to pass in the path of thedownloaded_filename.

// The Flat Data postprocessing libraries can be found at https://deno.land/x/flat/mod.ts// Replace 'x' with latest library versionimport{readJSON,writeJSON}from'https://deno.land/x/flat@0.0.x/mod.ts'constfilename=Deno.args[0]// equivalent to writing `const filename = 'btc-price.json'`constdata=awaitreadJSON(filename)// pluck a specific key off and write it out to a new fileconstnewfile=`postprocessed_${filename}`awaitwriteJSON(newfile,data.path.to.something)

Can be found in the examples folder. Once youinstall Deno you can run these examples with:

• deno run -A examples/csv/csv-example.ts
• deno run -A examples/csv/arquero-example.ts
• deno run -A --unstable examples/image/image-example.ts
• deno run -A examples/json/json-example.ts
• deno run -A examples/sheets/sheets-example.ts
• deno run -A examples/xlsx/xlsx-example.ts
• deno run -A --unstable examples/zip/zip-example.ts

Deno can run javascript or typescript files, so you can easily convert any of these examples to javascript and run them in the same way:

deno run -A examples/csv/csv-example.js

While our examples use a Deno file to run postprocessing tasks, you can also use Python as specified in this example:https://github.com/pierrotsmnrd/flat_data_py_example. Thank you@pierrotsmnrd!

You can also use bash as specified in this example:https://github.com/aborruso/flat_data_bash_example. By@aborruso

The Flat Data postprocessing library can be found at:https://deno.land/x/flat/mod.ts

You can import and use these helper functions directly, or treat them as a starting point for writing your own postprocessing scripts.

readCSV(path:string,options?:ParseOptions):Promise<Record<string,unknown>[]>

Args:

• path: path to a local CSV file
• options:options for parsing the CSV file

Usage:

const csv = await readCSV('./path/to/file.csv')

writeCSV(path:string,data:Record<string,unknown>[]|string,options?:Deno.WriteFileOptions)

Args:

• path: path to a local CSV file
• data: string or object array to store
• options:options for writing the CSV file

Usage:

constdata=[{age:70,name:'Rick'},{age:14,name:'Smith'}]awaitwriteCSV('./path/to/file.csv',data)

readTXT(path:string):string

Args:

• path: path to a local TXT file

Usage:

consttext=awaitreadTXT('./path/to/file.txt')

writeTXT(path:string,text:string,options?:Deno.WriteFileOptions):void

Args:

• path: path to a local TXT file
• text: text to write to file
• options:options for writing the TXT file

Usage:

awaitwriteTXT('./path/to/file.txt','Content for the file')

readJSON(path:string):JSON

Args:

• path: path to a local JSON file

Usage:

constjson=awaitreadJSON('./path/to/file.json')

readJSONFromURL(url:string):JSON

Args:

• url: URL to a json file

Usage:

constjson=awaitreadJSON('www.url.com/file.json')

writeJSON(path:string,data: any,replacer?: any,space?:string|number):void

Args:

• path: path to a local JSON file
• data: data to store as JSON
• replacer:replacer function that transforms the results or an array of strings and numbers that acts as an approved list for selecting the object properties that will be stringified
• space: adds indentation, white space, and line break characters to the to the JSON text to make it easier to read.

Usage:

constdata={age:40}awaitwriteJSON('./path/to/file.json',data)awaitwriteJSON('./path/to/file-with-indentation',data,null,2)

Our library relies onSheetJS, a library for parsing various spreadsheet formats. In addition to a simplereadXLSX function you can access the corexlsx module by importing it directly.

import{xlsx,readXLSX}from'https://deno.land/x/flat/mod.ts'

xlsx provides many moreutility functions.

readXLSX(path:string):XLSX.WorkBook

Args:

• path: path to a local XLSX file

Usage:

constworkbook=awaitreadXLSX('./path/to/file.xlsx')constsheetData=workbook.Sheets[workbook.SheetNames[0]]constcsvString=awaitxlsx.utils.sheet_to_csv(sheetData)

We recommend using a library likeimagescript for more advanced image manipulation. See an examplehere.

readImageFromFile(path:string):Promise<Uint8Array>

Args:

• path: path to a local image file

Usage:

constbytes=awaitreadImageFromFile('./path/to/image.jpeg')

readImageFromURL(url:string):Promise<{bytes:Uint8Array;name:string;}>

Args:

• url: url string to an image

Usage:

constimage=awaitreadImageFromURL('www.url.com/image.jpg')constbytes=image.bytesconstname=image.name

writeImage(imageBytes:Uint8Array,path:string):void

Args:

• imageBytes: a byte array
• path: path and name to write the image file

Usage:

awaitwriteImage(bytes,'./path/to/image.jpeg')

unZipFromFile(filePath: string,destinationPath: string|null="./",options:any={},):Promise<string|false>

Args:

• filePath: a path to a local zip file
• destinationPath: a folder path to unzip the files
• options: option.includeFileName can be true or false

Usage:

constresult=awaitunZipFromFile('./path/to/folder.zip','./unzip/path')constoutput=result ?'File unzipped successfully' :'Error unzipping'

unZipFromURL(fileURL: string,destinationPath: string|null="./",options:any={},):Promise<string|false>

Args:

• filePath: a path to a local zip file
• destinationPath: a folder path to unzip the files
• options: option.includeFileName can be true or false

Usage:

constresult=awaitunZipFromURL('www.url.com/file.zip','./unzip/path')constoutput=result ?'File unzipped successfully' :'Error unzipping'

removeFile(path:string):void

Args:

• path: path to a local file to delete

Usage:

awaitremoveFile('/path/to/file.x')

Run all the tests:

deno test -A --unstable tests/*

Run separate tests

deno test -A --unstable tests/csv-test.ts

MIT