Apache Arrow - v20.0.0

Apache Arrow in JS

Arrow is a set of technologies that enable big data systems to process and transfer data quickly.

Install`apache-arrow` from NPM

npm install apache-arrow oryarn add apache-arrow

(read about how wepackage apache-arrow below)

Powering Columnar In-Memory Analytics

Apache Arrow is a columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. The Arrow spec aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors.

Apache Arrow is the emerging standard for large in-memory columnar data (Spark,Pandas,Drill,Graphistry, ...). By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication.

Get Started

Check out ourAPI documentation to learn more about how to use Apache Arrow's JS implementation. You can also learn by example by checking out some of the following resources:

/js/test/unit - Unit tests for Table and Vector

Cookbook

Get a table from an Arrow file on disk (in IPC format)

import {readFileSync }from'fs';
import {tableFromIPC }from'apache-arrow';

constarrow =readFileSync('simple.arrow');
consttable =tableFromIPC(arrow);

console.table(table.toArray());

/*
 foo,  bar,  baz
   1,    1,   aa
null, null, null
   3, null, null
   4,    4,  bbb
   5,    5, cccc
*/

Create a Table when the Arrow file is split across buffers

import {readFileSync }from'fs';
import {tableFromIPC }from'apache-arrow';

consttable =tableFromIPC([
'latlong/schema.arrow',
'latlong/records.arrow'
].map((file)=>readFileSync(file)));

console.table([...table]);

/*
        origin_lat,         origin_lon
35.393089294433594,  -97.6007308959961
35.393089294433594,  -97.6007308959961
35.393089294433594,  -97.6007308959961
29.533695220947266, -98.46977996826172
29.533695220947266, -98.46977996826172
*/

Create a Table from JavaScript arrays

import {tableFromArrays }from'apache-arrow';

constLENGTH =2000;

constrainAmounts =Float32Array.from(
    {length:LENGTH },
    ()=>Number((Math.random() *20).toFixed(1)));

constrainDates =Array.from(
    {length:LENGTH },
    (_,i)=>newDate(Date.now() -1000 *60 *60 *24 *i));

constrainfall =tableFromArrays({
precipitation:rainAmounts,
date:rainDates
});

console.table([...rainfall]);

Load data with`fetch`

import {tableFromIPC }from"apache-arrow";

consttable =awaittableFromIPC(fetch("/simple.arrow"));

console.table([...table]);

Vectors look like JS Arrays

You can create vector from JavaScript typed arrays withmakeVector and from JavaScript arrays withvectorFromArray.makeVector is a lot faster and does not require a copy.

import {makeVector }from"apache-arrow";

constLENGTH =2000;

constrainAmounts =Float32Array.from(
    {length:LENGTH },
    ()=>Number((Math.random() *20).toFixed(1)));

constvector =makeVector(rainAmounts);

consttyped =vector.toArray()

assert(typedinstanceofFloat32Array);

for (leti = -1,n =vector.length; ++i <n;) {
assert(vector.get(i) ===typed[i]);
}

String vectors

Strings can be encoded as UTF-8 or dictionary encoded UTF-8. Dictionary encoding encodes repeated values more efficiently. You can create a dictionary encoded string conveniently withvectorFromArray or efficiently withmakeVector.

import {makeVector,vectorFromArray,Dictionary,Uint8,Utf8 }from"apache-arrow";

constutf8Vector =vectorFromArray(['foo','bar','baz'],newUtf8);

constdictionaryVector1 =vectorFromArray(
    ['foo','bar','baz','foo','bar']
);

constdictionaryVector2 =makeVector({
data: [0,1,2,0,1],// indexes into the dictionary
dictionary:utf8Vector,
type:newDictionary(newUtf8,newUint8)
});

Getting involved

SeeDEVELOP.md

Even if you do not plan to contribute to Apache Arrow itself or Arrowintegrations in other projects, we'd be happy to have you involved:

Join the mailing list: send an email todev-subscribe@arrow.apache.org. Share your ideas and use cases for theproject
Follow our activity onGitHub issues
Learn the format
Contribute code to one of the reference implementations

We prefer to receive contributions in the form of GitHub pull requests.Please send pull requests against thegithub.com/apache/arrow repository.

If you are looking for some ideas on what to contribute, check out theGitHubissues for the Apache Arrow project. Comment on the issue and/or contactdev@arrow.apache.orgwith your questions and ideas.

If you’d like to report a bug but don’t have time to fix it, you can still postit on GitHub issues, or email the mailing listdev@arrow.apache.org

Packaging

apache-arrow is written in TypeScript, but the project is compiled to multiple JS versions and common module formats.

The baseapache-arrow package includes all the compilation targets for convenience, but if you're conscientious about yournode_modules footprint, we got you.

The targets are also published under the@apache-arrow namespace:

npminstallapache-arrow# <-- combined es2015/CommonJS/ESModules/UMD + esnext/UMD
npminstall@apache-arrow/ts# standalone TypeScript package
npminstall@apache-arrow/es5-cjs# standalone es5/CommonJS package
npminstall@apache-arrow/es5-esm# standalone es5/ESModules package
npminstall@apache-arrow/es5-umd# standalone es5/UMD package
npminstall@apache-arrow/es2015-cjs# standalone es2015/CommonJS package
npminstall@apache-arrow/es2015-esm# standalone es2015/ESModules package
npminstall@apache-arrow/es2015-umd# standalone es2015/UMD package
npminstall@apache-arrow/esnext-cjs# standalone esNext/CommonJS package
npminstall@apache-arrow/esnext-esm# standalone esNext/ESModules package
npminstall@apache-arrow/esnext-umd# standalone esNext/UMD package

Why we package like this

The JS community is a diverse group with a varied list of target environments and tool chains. Publishing multiple packages accommodates projects of all stripes.

If you think we missed a compilation target and it's a blocker for adoption, please open an issue.

Supported Browsers and Platforms

The bundles we compile support moderns browser released in the last 5 years. This includes supported versions ofFirefox, Chrome, Edge, and Safari. We do not actively support Internet Explorer.Apache Arrow also works onmaintained versions of Node.

People

Full list of broader Apache Arrowcommitters.

Brian Hulette,committer
Paul Taylor,committer
Dominik Moritz,committer

Powered By Apache Arrow in JS

Full list of broader Apache Arrowprojects & organizations.

Open Source Projects

Apache Arrow -- Parent project for Powering Columnar In-Memory Analytics, including affiliated open source projects
Perspective -- Perspective is an interactive analytics and data visualization component well-suited for large and/or streaming datasets. Perspective leverages Arrow C++ compiled to WebAssembly.
Falcon is a visualization tool for linked interactions across multiple aggregate visualizations of millions or billions of records.
Vega is an ecosystem of tools for interactive visualizations on the web. The Vega team implemented anArrow loader.
Arquero is a library for query processing and transformation of array-backed data tables.
OmniSci is a GPU database. Its JavaScript connector returns Arrow dataframes.

License

Apache 2.0

Settings

Member Visibility

Inherited

Theme

Movatterモバイル変換

Apache Arrow - v20.0.0

Apache Arrow in JS

Install`apache-arrow` from NPM

Powering Columnar In-Memory Analytics

Get Started

Cookbook

Get a table from an Arrow file on disk (in IPC format)

Create a Table when the Arrow file is split across buffers

Create a Table from JavaScript arrays

Load data with`fetch`

Vectors look like JS Arrays

String vectors

Getting involved

Packaging

Why we package like this

Supported Browsers and Platforms

People

Powered By Apache Arrow in JS

Open Source Projects

License

Settings

On This Page

Movatterモバイル変換

Apache Arrow - v20.0.0

Apache Arrow in JS

Installapache-arrow from NPM

Powering Columnar In-Memory Analytics

Get Started

Cookbook

Get a table from an Arrow file on disk (in IPC format)

Create a Table when the Arrow file is split across buffers

Create a Table from JavaScript arrays

Load data withfetch

Vectors look like JS Arrays

String vectors

Getting involved

Packaging

Why we package like this

Supported Browsers and Platforms

People

Powered By Apache Arrow in JS

Open Source Projects

License

Settings

On This Page

Install`apache-arrow` from NPM

Load data with`fetch`