Acero - Streaming Execution Engine#

Warning

Acero is experimental and a stable API is not yet guaranteed.

Acero is a streaming query engine, which allows the computation to be expressedas an “execution plan” (constructed using theDeclaration interface).This enables to create a computation composed ofpyarrow.compute functionsand to execute this efficiently in a batched manner.

Declaration(factory_name, ...[, inputs])

Helper class for declaring the nodes of an ExecPlan.

ExecNodeOptions

Base class for the node options.

TableSourceNodeOptions(table)

A Source node which accepts a table.

ScanNodeOptions(dataset, **kwargs)

A Source node which yields batches from a Dataset scan.

FilterNodeOptions(filter_expression)

Make a node which excludes some rows from batches passed through it.

ProjectNodeOptions(expressions[, names])

Make a node which executes expressions on input batches, producing batches of the same length with new columns.

AggregateNodeOptions(aggregates[, keys])

Make a node which aggregates input batches, optionally grouped by keys.

OrderByNodeOptions([sort_keys, null_placement])

Make a node which applies a new ordering to the data.

HashJoinNodeOptions(join_type, left_keys, ...)

Make a node which implements join operation using hash join strategy.

See also

Acero C++ user guide

Substrait

Alternative way to run Acero from a standardized Substrait plan.