Repository files navigation

The easiest, most efficient way to render large geospatial datasets indeck.gl, viaGeoArrow.

This is just aglue library to deck.gl. It generates the same layer objects as upstream deck.gl does, but uses alow-level binary interface for best performance. Using the binary interface directly is really easy to mess up. Instead, the layer classes exposed by@geoarrow/deck.gl-layers focus on making the process easy to use and validating user input, and under the hood pass buffers to deck.gl's binary interface.

3.2 million points rendered with aGeoArrowScatterplotLayer.

• Fast: copies binary buffers directly from anArrow JSTable object to the GPU usingdeck.gl's binary data interface.
• Memory-efficient: no intermediate data representation and no garbage-collector overhead.
• Full layer customization: Use the same layer properties as in the upstream deck.gl layer documentation. Anyaccessor (layer property prefixed withget*) can be passed an ArrowVector.
• Input validation. Validation can be turned off via the_validate property on most layer types.
• Multi-threaded polygon triangulation. When rendering polygon layers, a process calledpolygon triangulation must happen on the CPU before data can be copied to the GPU. Ordinarily, this can block the main thread for several seconds, but theGeoArrowSolidPolygonLayer will perform this process off the main thread, on up to 8 web workers.
• Progressive rendering support. For streaming-capable data formats like Arrow IPC and Parquet, you can render a GeoArrow layer per chunk as the data loads.

Standalone examples exist in theexamples/ directory. Create an issue if you have trouble running them.

More hosted examples on Observable are planned.

All deck.gl layers have two types of properties:"Render Options" — constant properties across a layer — and "Data Accessors" — properties that can vary across rows. An accessor is any property prefixed withget, likeGeoArrowScatterplotLayer'sgetFillColor.

With@geoarrow/deck.gl-layers specifically, there are two ways to pass these data accessors, either as pre-computed columns or with function callbacks on Arrow data.

If you have an Arrow column (Vector in Arrow JS terminology), you can pass that directly into a layer:

import{Table}from"apache-arrow";import{GeoArrowScatterplotLayer}from"@geoarrow/deck.gl-layers";consttable=newTable(...);constdeckLayer=newGeoArrowScatterplotLayer({id:"scatterplot",data:table,/// Geometry columngetPosition:table.getChild("geometry")!,/// Column of type FixedSizeList[3] or FixedSizeList[4], with child type Uint8getFillColor:table.getChild("colors")!,});

For example,lonboard computes Arrow columns on the Python side for all attributes so that end users have available the full capabilities Python. Then those columns are serialized to Python and the resultingarrow.Vector is passed into the relevant layer.

GeoArrow layers accept a callback that takes an object withindex anddata.data is anarrow.RecordBatch object (a vertical section of the inputTable), andindex is the positional index of the current row of that batch. In TypeScript, you should see accurate type checking.

constdeckLayer=newGeoArrowPathLayer({id:"geoarrow-path",data:table,getColor:({ index, data, target})=>{constrecordBatch=data.data;constrow=recordBatch.get(index)!;returnCOLORS_LOOKUP[row["scalerank"]];},}),

The full example is inexamples/multilinestring/app.tsx.

You can also use assign to thetarget prop to reduce garbage collector overhead, as described in thedeck.gl performance guide.

To create deck.gl layers using this library, you need to first get GeoArrow-formatted data into the browser, discussed below.

OGR/GDAL is useful for converting among data formats on the backend, and it includes bothGeoArrow andGeoParquet drivers. Pass-lco GEOMETRY_ENCODING=GEOARROW when converting to Arrow or Parquet files in order to store geometries in a GeoArrow-native geometry column.

If you already have Arrow IPC files (also called Feather files) with a GeoArrow geometry column, you can useapache-arrow to load those files.

import{tableFromIPC}from"apache-arrow";import{GeoArrowScatterplotLayer}from"@geoarrow/deck.gl-layers";constresp=awaitfetch("url/to/file.arrow");constjsTable=awaittableFromIPC(resp);constdeckLayer=newGeoArrowScatterplotLayer({id:"scatterplot",data:jsTable,/// Replace with the correct geometry column namegetPosition:jsTable.getChild("geometry")!,});

Note those IPC files must be saveduncompressed (at least not internally compressed). As of v14, Arrow JS does not currently support loading IPC files with internal compression.

If you have a Parquet file where the geometry column is stored asGeoArrow encoding (i.e. not as a binary column with WKB-encoded geometries), you can use the stableparquet-wasm library to load those files.

import{readParquet}from"parquet-wasm"import{tableFromIPC}from"apache-arrow";import{GeoArrowScatterplotLayer}from"@geoarrow/deck.gl-layers";constresp=awaitfetch("url/to/file.parquet");constarrayBuffer=awaitresp.arrayBuffer();constwasmTable=readParquet(newUint8Array(arrayBuffer));constjsTable=tableFromIPC(wasmTable.intoIPCStream());constdeckLayer=newGeoArrowScatterplotLayer({id:"scatterplot",data:jsTable,/// Replace with the correct geometry column namegetPosition:jsTable.getChild("geometry")!,});

See below for instructions to load GeoParquet 1.0 files, which have WKB-encoded geometries that need to be decoded before they can be used with@geoarrow/deck.gl-layers.

An initial version of the@geoarrow/geoparquet-wasm library is published, which reads a GeoParquet file to GeoArrow memory.

import{readGeoParquet}from"@geoarrow/geoparquet-wasm";import{tableFromIPC}from"apache-arrow";import{GeoArrowScatterplotLayer}from"@geoarrow/deck.gl-layers";constresp=awaitfetch("url/to/file.parquet");constarrayBuffer=awaitresp.arrayBuffer();constwasmTable=readGeoParquet(newUint8Array(arrayBuffer));constjsTable=tableFromIPC(wasmTable.intoTable().intoIPCStream());constdeckLayer=newGeoArrowScatterplotLayer({id:"scatterplot",data:jsTable,/// Replace with the correct geometry column namegetPosition:jsTable.getChild("geometry")!,});

If you hit a bug with@geoarrow/geoparquet-wasm, please create a reproducible bug reporthere.

An initial version of the@geoarrow/flatgeobuf-wasm library is published, which reads a FlatGeobuf file to GeoArrow memory. As of version 0.2.0-beta.1, this library does not yet support remote files, and expects the full FlatGeobuf file to exist in memory.

import{readFlatGeobuf}from"@geoarrow/flatgeobuf-wasm";import{tableFromIPC}from"apache-arrow";import{GeoArrowScatterplotLayer}from"@geoarrow/deck.gl-layers";constresp=awaitfetch("url/to/file.fgb");constarrayBuffer=awaitresp.arrayBuffer();constwasmTable=readFlatGeobuf(newUint8Array(arrayBuffer));constjsTable=tableFromIPC(wasmTable.intoTable().intoIPCStream());constdeckLayer=newGeoArrowScatterplotLayer({id:"scatterplot",data:jsTable,/// Replace with the correct geometry column namegetPosition:jsTable.getChild("geometry")!,});

If you hit a bug with@geoarrow/flatgeobuf-wasm, please create a reproducible bug reporthere.