pyarrow.RecordBatch#

classpyarrow.RecordBatch#

Bases:_Tabular

Batch of rows of columns of equal length

Warning

Do not call this class’s constructor directly, use one of theRecordBatch.from_* functions instead.

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>names=["n_legs","animals"]

Constructing a RecordBatch from arrays:

>>>pa.RecordBatch.from_arrays([n_legs,animals],names=names)pyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,2,4,4,5,100]animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]>>>pa.RecordBatch.from_arrays([n_legs,animals],names=names).to_pandas()   n_legs        animals0       2       Flamingo1       2         Parrot2       4            Dog3       4          Horse4       5  Brittle stars5     100      Centipede

Constructing a RecordBatch from pandas DataFrame:

>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2021,2022],...'month':[3,5,7,9],...'day':[1,5,9,13],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>pa.RecordBatch.from_pandas(df)pyarrow.RecordBatchyear: int64month: int64day: int64n_legs: int64animals: string----year: [2020,2022,2021,2022]month: [3,5,7,9]day: [1,5,9,13]n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]>>>pa.RecordBatch.from_pandas(df).to_pandas()   year  month  day  n_legs        animals0  2020      3    1       2       Flamingo1  2022      5    5       4          Horse2  2021      7    9       5  Brittle stars3  2022      9   13     100      Centipede

Constructing a RecordBatch from pylist:

>>>pylist=[{'n_legs':2,'animals':'Flamingo'},...{'n_legs':4,'animals':'Dog'}]>>>pa.RecordBatch.from_pylist(pylist).to_pandas()   n_legs   animals0       2  Flamingo1       4       Dog

You can also construct a RecordBatch usingpyarrow.record_batch():

>>>pa.record_batch([n_legs,animals],names=names).to_pandas()   n_legs        animals0       2       Flamingo1       2         Parrot2       4            Dog3       4          Horse4       5  Brittle stars5     100      Centipede

>>>pa.record_batch(df)pyarrow.RecordBatchyear: int64month: int64day: int64n_legs: int64animals: string----year: [2020,2022,2021,2022]month: [3,5,7,9]day: [1,5,9,13]n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]

__init__(*args,**kwargs)#

Methods

__init__(*args, **kwargs)
add_column(self, int i, field_, column)	Add column to RecordBatch at position i.
append_column(self, field_, column)	Append column at end of columns.
cast(self, Schema target_schema[, safe, options])	Cast record batch values to another schema.
column(self, i)	Select single column from Table or RecordBatch.
copy_to(self, destination)	Copy the entire RecordBatch to destination device.
drop_columns(self, columns)	Drop one or more columns and return a new Table or RecordBatch.
drop_null(self)	Remove rows that contain missing values from a Table or RecordBatch.
equals(self, other, bool check_metadata=False)	Check if contents of two record batches are equal.
field(self, i)	Select a schema field by its column name or numeric index.
filter(self, mask[, null_selection_behavior])	Select rows from the table or record batch based on a boolean mask.
from_arrays(list arrays[, names, schema, ...])	Construct a RecordBatch from multiple pyarrow.Arrays
from_pandas(cls, df, Schema schema=None[, ...])	Convert pandas.DataFrame to an Arrow RecordBatch
from_pydict(cls, mapping[, schema, metadata])	Construct a Table or RecordBatch from Arrow arrays or columns.
from_pylist(cls, mapping[, schema, metadata])	Construct a Table or RecordBatch from list of rows / dictionaries.
from_struct_array(StructArray struct_array)	Construct a RecordBatch from a StructArray.
get_total_buffer_size(self)	The sum of bytes in each buffer referenced by the record batch
itercolumns(self)	Iterator over all columns in their numerical order.
remove_column(self, int i)	Create new RecordBatch with the indicated column removed.
rename_columns(self, names)	Create new record batch with columns renamed to provided names.
replace_schema_metadata(self[, metadata])	Create shallow copy of record batch by replacing schema key-value metadata with the indicated new metadata (which may be None, which deletes any existing metadata
select(self, columns)	Select columns of the RecordBatch.
serialize(self[, memory_pool])	Write RecordBatch to Buffer as encapsulated IPC message, which does not include a Schema.
set_column(self, int i, field_, column)	Replace column in RecordBatch at position.
slice(self[, offset, length])	Compute zero-copy slice of this RecordBatch
sort_by(self, sorting, **kwargs)	Sort the Table or RecordBatch by one or multiple columns.
take(self, indices)	Select rows from a Table or RecordBatch.
to_pandas(self[, memory_pool, categories, ...])	Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
to_pydict(self, *[, maps_as_pydicts])	Convert the Table or RecordBatch to a dict or OrderedDict.
to_pylist(self, *[, maps_as_pydicts])	Convert the Table or RecordBatch to a list of rows / dictionaries.
to_string(self, *[, show_metadata, preview_cols])	Return human-readable string representation of Table or RecordBatch.
to_struct_array(self)	Convert to a struct array.
to_tensor(self, bool null_to_nan=False, ...)	Convert to aTensor.
validate(self, *[, full])	Perform validation checks.

Attributes

column_names	Names of the Table or RecordBatch columns.
columns	List of all columns in numerical order.
device_type	The device type where the arrays in the RecordBatch reside.
is_cpu	Whether the RecordBatch's arrays are CPU-accessible.
nbytes	Total number of bytes consumed by the elements of the record batch.
num_columns	Number of columns
num_rows	Number of rows
schema	Schema of the RecordBatch and its columns
shape	Dimensions of the table or record batch: (#rows, #columns).

__dataframe__(self,nan_as_null:bool=False,allow_copy:bool=True)#

Return the dataframe interchange object implementing the interchange protocol.

Parameters:

nan_as_nullbool, defaultFalse

Whether to tell the DataFrame to overwrite null values in the datawithNaN (orNaT).

allow_copybool, defaultTrue

Whether to allow memory copying when exporting. If set to Falseit would cause non-zero-copy exports to fail.

Returns:

DataFrameinterchange object

The object which consuming library can use to ingress the dataframe.

Notes

Details on the interchange protocol:https://data-apis.org/dataframe-protocol/latest/index.htmlnan_as_null currently has no effect; once support for nullable extensiondtypes is added, this value should be propagated to columns.

add_column(self,inti,field_,column)#

Add column to RecordBatch at position i.

A new record batch is returned with the column added, the original record batchobject is left unchanged.

Parameters:

iint

Index to place the column at.

field_str orField

If a string is passed then the type is deduced from the columndata.

columnArray orvalue coercible toarray

Column data.

Returns:

RecordBatch

New record batch with the passed column added.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>batch=pa.RecordBatch.from_pandas(df)

Add column:

>>>year=[2021,2022,2019,2021]>>>batch.add_column(0,"year",year)pyarrow.RecordBatchyear: int64n_legs: int64animals: string----year: [2021,2022,2019,2021]n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]

Original record batch is left unchanged:

>>>batchpyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]

append_column(self,field_,column)#

Append column at end of columns.

Parameters:

field_str orField

If a string is passed then the type is deduced from the columndata.

columnArray orvalue coercible toarray

Column data.

Returns:

Table orRecordBatch

New table or record batch with the passed column added.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Append column at the end:

>>>year=[2021,2022,2019,2021]>>>table.append_column('year',[year])pyarrow.Tablen_legs: int64animals: stringyear: int64----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]year: [[2021,2022,2019,2021]]

cast(self,Schematarget_schema,safe=None,options=None)#

Cast record batch values to another schema.

Parameters:

target_schemaSchema

Schema to cast to, the names and order of fields must match.

safebool, defaultTrue

Check for overflows or other unsafe conversions.

optionsCastOptions, defaultNone

Additional checks pass by CastOptions

Returns:

RecordBatch

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>batch=pa.RecordBatch.from_pandas(df)>>>batch.scheman_legs: int64animals: string-- schema metadata --pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ...

Define new schema and cast batch values:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.duration('s')),...pa.field('animals',pa.string())]...)>>>batch.cast(target_schema=my_schema)pyarrow.RecordBatchn_legs: duration[s]animals: string----n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]

column(self,i)#

Select single column from Table or RecordBatch.

Parameters:

iint orstr

The index or name of the column to retrieve.

Returns:

columnArray (forRecordBatch) orChunkedArray (forTable)

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Select a column by numeric index:

>>>table.column(0)<pyarrow.lib.ChunkedArray object at ...>[  [    2,    4,    5,    100  ]]

Select a column by its name:

>>>table.column("animals")<pyarrow.lib.ChunkedArray object at ...>[  [    "Flamingo",    "Horse",    "Brittle stars",    "Centipede"  ]]

column_names#

Names of the Table or RecordBatch columns.

Returns:

list ofstr

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>table=pa.Table.from_arrays([[2,4,5,100],...["Flamingo","Horse","Brittle stars","Centipede"]],...names=['n_legs','animals'])>>>table.column_names['n_legs', 'animals']

columns#

List of all columns in numerical order.

Returns:

columnslist ofArray (forRecordBatch) orlist ofChunkedArray (forTable)

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.columns[<pyarrow.lib.ChunkedArray object at ...>[  [    null,    4,    5,    null  ]], <pyarrow.lib.ChunkedArray object at ...>[  [    "Flamingo",    "Horse",    null,    "Centipede"  ]]]

copy_to(self,destination)#

Copy the entire RecordBatch to destination device.

This copies each column of the record batch to createa new record batch where all underlying buffers for the columns havebeen copied to the destination MemoryManager.

Parameters:

destinationpyarrow.MemoryManager orpyarrow.Device

The destination device to copy the array to.

Returns:

RecordBatch

device_type#

The device type where the arrays in the RecordBatch reside.

Returns:

DeviceAllocationType

drop_columns(self,columns)#

Drop one or more columns and return a new Table or RecordBatch.

Parameters:

columnsstr orlist[str]

Field name(s) referencing existing column(s).

Returns:

Table orRecordBatch

A tabular object without the column(s).

Raises:

KeyError

If any of the passed column names do not exist.

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Drop one column:

>>>table.drop_columns("animals")pyarrow.Tablen_legs: int64----n_legs: [[2,4,5,100]]

Drop one or more columns:

>>>table.drop_columns(["n_legs","animals"])pyarrow.Table...----

drop_null(self)#

Remove rows that contain missing values from a Table or RecordBatch.

Seepyarrow.compute.drop_null() for full usage.

Returns:

Table orRecordBatch

A tabular object with the same schema, with rows containingno missing values.

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'year':[None,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.drop_null()pyarrow.Tableyear: doublen_legs: int64animals: string----year: [[2022,2021]]n_legs: [[4,100]]animals: [["Horse","Centipede"]]

equals(self,other,boolcheck_metadata=False)#

Check if contents of two record batches are equal.

Parameters:

otherpyarrow.RecordBatch

RecordBatch to compare against.

check_metadatabool, defaultFalse

Whether schema metadata equality should be checked as well.

Returns:

are_equalbool

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>batch_0=pa.record_batch([])>>>batch_1=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"],...metadata={"n_legs":"Number of legs per animal"})>>>batch.equals(batch)True>>>batch.equals(batch_0)False>>>batch.equals(batch_1)True>>>batch.equals(batch_1,check_metadata=True)False

field(self,i)#

Select a schema field by its column name or numeric index.

Parameters:

iint orstr

The index or name of the field to retrieve.

Returns:

Field

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.field(0)pyarrow.Field<n_legs: int64>>>>table.field(1)pyarrow.Field<animals: string>

filter(self,mask,null_selection_behavior='drop')#

Select rows from the table or record batch based on a boolean mask.

The Table can be filtered based on a mask, which will be passed topyarrow.compute.filter() to perform the filtering, or it canbe filtered through a booleanExpression

Parameters:

maskArray orarray-like orExpression

The boolean mask or theExpression to filter the table with.

null_selection_behaviorstr, default “drop”

How nulls in the mask should be handled, does nothing ifanExpression is used.

Returns:

filteredTable orRecordBatch

A tabular object of the same schema, with only the rows selectedby applied filtering

Examples

Using a Table (works similarly for RecordBatch):

>>>importpyarrowaspa>>>table=pa.table({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})

Define an expression and select rows:

>>>importpyarrow.computeaspc>>>expr=pc.field("year")<=2020>>>table.filter(expr)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2019]]n_legs: [[2,5]]animals: [["Flamingo","Brittle stars"]]

Define a mask and select rows:

>>>mask=[True,True,False,None]>>>table.filter(mask)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2022]]n_legs: [[2,4]]animals: [["Flamingo","Horse"]]>>>table.filter(mask,null_selection_behavior='emit_null')pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2022,null]]n_legs: [[2,4,null]]animals: [["Flamingo","Horse",null]]

staticfrom_arrays(listarrays,names=None,schema=None,metadata=None)#

Construct a RecordBatch from multiple pyarrow.Arrays

Parameters:

arrayslist ofpyarrow.Array

One for each field in RecordBatch

nameslist ofstr, optional

Names for the batch fields. If not passed, schema must be passed

schemaSchema, defaultNone

Schema for the created batch. If not passed, names must be passed

metadatadict or Mapping, defaultNone

Optional metadata for the schema (if inferred).

Returns:

pyarrow.RecordBatch

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>names=["n_legs","animals"]

Construct a RecordBatch from pyarrow Arrays using names:

>>>pa.RecordBatch.from_arrays([n_legs,animals],names=names)pyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,2,4,4,5,100]animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]>>>pa.RecordBatch.from_arrays([n_legs,animals],names=names).to_pandas()   n_legs        animals0       2       Flamingo1       2         Parrot2       4            Dog3       4          Horse4       5  Brittle stars5     100      Centipede

Construct a RecordBatch from pyarrow Arrays using schema:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"n_legs":"Number of legs per animal"})>>>pa.RecordBatch.from_arrays([n_legs,animals],schema=my_schema).to_pandas()   n_legs        animals0       2       Flamingo1       2         Parrot2       4            Dog3       4          Horse4       5  Brittle stars5     100      Centipede>>>pa.RecordBatch.from_arrays([n_legs,animals],schema=my_schema).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

classmethodfrom_pandas(cls,df,Schemaschema=None,preserve_index=None,nthreads=None,columns=None)#

Convert pandas.DataFrame to an Arrow RecordBatch

Parameters:

dfpandas.DataFrame

schemapyarrow.Schema, optional

The expected schema of the RecordBatch. This can be used toindicate the type of columns if we cannot infer it automatically.If passed, the output will have exactly this schema. Columnsspecified in the schema that are not found in the DataFrame columnsor its index will raise an error. Additional columns or indexlevels in the DataFrame which are not specified in the schema willbe ignored.

preserve_indexbool, optional

Whether to store the index as an additional column in the resultingRecordBatch. The default of None will store the index as acolumn, except for RangeIndex which is stored as metadata only. Usepreserve_index=True to force it to be stored as a column.

nthreadsint, defaultNone

If greater than 1, convert columns to Arrow in parallel usingindicated number of threads. By default, this followspyarrow.cpu_count() (may use up to system CPU count threads).

columnslist, optional

List of column to be converted. If None, use all columns.

Returns:

pyarrow.RecordBatch

Examples

>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2021,2022],...'month':[3,5,7,9],...'day':[1,5,9,13],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})

Convert pandas DataFrame to RecordBatch:

>>>importpyarrowaspa>>>pa.RecordBatch.from_pandas(df)pyarrow.RecordBatchyear: int64month: int64day: int64n_legs: int64animals: string----year: [2020,2022,2021,2022]month: [3,5,7,9]day: [1,5,9,13]n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]

Convert pandas DataFrame to RecordBatch using schema:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"n_legs":"Number of legs per animal"})>>>pa.RecordBatch.from_pandas(df,schema=my_schema)pyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]

Convert pandas DataFrame to RecordBatch specifying columns:

>>>pa.RecordBatch.from_pandas(df,columns=["n_legs"])pyarrow.RecordBatchn_legs: int64----n_legs: [2,4,5,100]

classmethodfrom_pydict(cls,mapping,schema=None,metadata=None)#

Construct a Table or RecordBatch from Arrow arrays or columns.

Parameters:

mappingdict or Mapping

A mapping of strings to Arrays or Python lists.

schemaSchema, defaultNone

If not passed, will be inferred from the Mapping values.

metadatadict or Mapping, defaultNone

Optional metadata for the schema (if inferred).

Returns:

Table orRecordBatch

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>pydict={'n_legs':n_legs,'animals':animals}

Construct a Table from a dictionary of arrays:

>>>pa.Table.from_pydict(pydict)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]>>>pa.Table.from_pydict(pydict).scheman_legs: int64animals: string

Construct a Table from a dictionary of arrays with metadata:

>>>my_metadata={"n_legs":"Number of legs per animal"}>>>pa.Table.from_pydict(pydict,metadata=my_metadata).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

Construct a Table from a dictionary of arrays with pyarrow schema:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"n_legs":"Number of legs per animal"})>>>pa.Table.from_pydict(pydict,schema=my_schema).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

classmethodfrom_pylist(cls,mapping,schema=None,metadata=None)#

Construct a Table or RecordBatch from list of rows / dictionaries.

Parameters:

mappinglist of dicts of rows

A mapping of strings to row values.

schemaSchema, defaultNone

If not passed, will be inferred from the first row of themapping values.

metadatadict or Mapping, defaultNone

Optional metadata for the schema (if inferred).

Returns:

Table orRecordBatch

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>pylist=[{'n_legs':2,'animals':'Flamingo'},...{'n_legs':4,'animals':'Dog'}]

Construct a Table from a list of rows:

>>>pa.Table.from_pylist(pylist)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4]]animals: [["Flamingo","Dog"]]

Construct a Table from a list of rows with metadata:

>>>my_metadata={"n_legs":"Number of legs per animal"}>>>pa.Table.from_pylist(pylist,metadata=my_metadata).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

Construct a Table from a list of rows with pyarrow schema:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"n_legs":"Number of legs per animal"})>>>pa.Table.from_pylist(pylist,schema=my_schema).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

staticfrom_struct_array(StructArraystruct_array)#

Construct a RecordBatch from a StructArray.

Each field in the StructArray will become a column in the resultingRecordBatch.

Parameters:

struct_arrayStructArray

Array to construct the record batch from.

Returns:

pyarrow.RecordBatch

Examples

>>>importpyarrowaspa>>>struct=pa.array([{'n_legs':2,'animals':'Parrot'},...{'year':2022,'n_legs':4}])>>>pa.RecordBatch.from_struct_array(struct).to_pandas()  animals  n_legs    year0  Parrot       2     NaN1    None       4  2022.0

get_total_buffer_size(self)#

The sum of bytes in each buffer referenced by the record batch

An array may only reference a portion of a buffer.This method will overestimate in this case and return thebyte size of the entire buffer.

If a buffer is referenced multiple times then it willonly be counted once.

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>batch.get_total_buffer_size()120

is_cpu#

Whether the RecordBatch’s arrays are CPU-accessible.

itercolumns(self)#

Iterator over all columns in their numerical order.

Yields:

Array (forRecordBatch) orChunkedArray (forTable)

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>foriintable.itercolumns():...print(i.null_count)...21

nbytes#

Total number of bytes consumed by the elements of the record batch.

In other words, the sum of bytes from all buffer ranges referenced.

Unlikeget_total_buffer_size this method will account for arrayoffsets.

If buffers are shared between arrays then the sharedportion will only be counted multiple times.

The dictionary of dictionary arrays will always be counted in theirentirety even if the array only references a portion of the dictionary.

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>batch.nbytes116

num_columns#

Number of columns

Returns:

int

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>batch.num_columns2

num_rows#

Number of rows

Due to the definition of a RecordBatch, all columns have the samenumber of rows.

Returns:

int

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>batch.num_rows6

remove_column(self,inti)#

Create new RecordBatch with the indicated column removed.

Parameters:

iint

Index of column to remove.

Returns:

Table

New record batch without the column.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>batch=pa.RecordBatch.from_pandas(df)>>>batch.remove_column(1)pyarrow.RecordBatchn_legs: int64----n_legs: [2,4,5,100]

rename_columns(self,names)#

Create new record batch with columns renamed to provided names.

Parameters:

nameslist[str] ordict[str,str]

List of new column names or mapping of old column names to new column names.

If a mapping of old to new column names is passed, then all columns which arefound to match a provided old column name will be renamed to the new column name.If any column names are not found in the mapping, a KeyError will be raised.

Returns:

RecordBatch

Raises:

KeyError

If any of the column names passed in the names mapping do not exist.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>batch=pa.RecordBatch.from_pandas(df)>>>new_names=["n","name"]>>>batch.rename_columns(new_names)pyarrow.RecordBatchn: int64name: string----n: [2,4,5,100]name: ["Flamingo","Horse","Brittle stars","Centipede"]>>>new_names={"n_legs":"n","animals":"name"}>>>batch.rename_columns(new_names)pyarrow.RecordBatchn: int64name: string----n: [2,4,5,100]name: ["Flamingo","Horse","Brittle stars","Centipede"]

replace_schema_metadata(self,metadata=None)#

Create shallow copy of record batch by replacing schemakey-value metadata with the indicated new metadata (which may be None,which deletes any existing metadata

Parameters:

metadatadict, defaultNone

Returns:

shallow_copyRecordBatch

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])

Constructing a RecordBatch with schema and metadata:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64())],...metadata={"n_legs":"Number of legs per animal"})>>>batch=pa.RecordBatch.from_arrays([n_legs],schema=my_schema)>>>batch.scheman_legs: int64-- schema metadata --n_legs: 'Number of legs per animal'

Shallow copy of a RecordBatch with deleted schema metadata:

>>>batch.replace_schema_metadata().scheman_legs: int64

schema#

Schema of the RecordBatch and its columns

Returns:

pyarrow.Schema

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>batch.scheman_legs: int64animals: string

select(self,columns)#

Select columns of the RecordBatch.

Returns a new RecordBatch with the specified columns, and metadatapreserved.

Parameters:

columnslist-like

The column names or integer indices to select.

Returns:

RecordBatch

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.record_batch([n_legs,animals],...names=["n_legs","animals"])

Select columns my indices:

>>>batch.select([1])pyarrow.RecordBatchanimals: string----animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]

Select columns by names:

>>>batch.select(["n_legs"])pyarrow.RecordBatchn_legs: int64----n_legs: [2,2,4,4,5,100]

serialize(self,memory_pool=None)#

Write RecordBatch to Buffer as encapsulated IPC message, which does notinclude a Schema.

To reconstruct a RecordBatch from the encapsulated IPC message Bufferreturned by this function, a Schema must be passed separately. SeeExamples.

Parameters:

memory_poolMemoryPool, defaultNone

Uses default memory pool if not specified

Returns:

serializedBuffer

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>buf=batch.serialize()>>>buf<pyarrow.Buffer address=0x... size=... is_cpu=True is_mutable=True>

Reconstruct RecordBatch from IPC message Buffer and original Schema

>>>pa.ipc.read_record_batch(buf,batch.schema)pyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,2,4,4,5,100]animals: ["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]

set_column(self,inti,field_,column)#

Replace column in RecordBatch at position.

Parameters:

iint

Index to place the column at.

field_str orField

If a string is passed then the type is deduced from the columndata.

columnArray orvalue coercible toarray

Column data.

Returns:

RecordBatch

New record batch with the passed column set.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>batch=pa.RecordBatch.from_pandas(df)

Replace a column:

>>>year=[2021,2022,2019,2021]>>>batch.set_column(1,'year',year)pyarrow.RecordBatchn_legs: int64year: int64----n_legs: [2,4,5,100]year: [2021,2022,2019,2021]

shape#

Dimensions of the table or record batch: (#rows, #columns).

Returns:

(int,int)

Number of rows and number of columns.

Examples

>>>importpyarrowaspa>>>table=pa.table({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table.shape(4, 2)

slice(self,offset=0,length=None)#

Compute zero-copy slice of this RecordBatch

Parameters:

offsetint, default 0

Offset from start of record batch to slice

lengthint, defaultNone

Length of slice (default is until end of batch starting fromoffset)

Returns:

slicedRecordBatch

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>batch=pa.RecordBatch.from_arrays([n_legs,animals],...names=["n_legs","animals"])>>>batch.to_pandas()   n_legs        animals0       2       Flamingo1       2         Parrot2       4            Dog3       4          Horse4       5  Brittle stars5     100      Centipede>>>batch.slice(offset=3).to_pandas()   n_legs        animals0       4          Horse1       5  Brittle stars2     100      Centipede>>>batch.slice(length=2).to_pandas()   n_legs   animals0       2  Flamingo1       2    Parrot>>>batch.slice(offset=3,length=1).to_pandas()   n_legs animals0       4   Horse

sort_by(self,sorting,**kwargs)#

Sort the Table or RecordBatch by one or multiple columns.

Parameters:

sortingstr orlist[tuple(name,order)]

Name of the column to use to sort (ascending), ora list of multiple sorting conditions whereeach entry is a tuple with column nameand sorting order (“ascending” or “descending”)

**kwargsdict, optional

Additional sorting options.As allowed bySortOptions

Returns:

Table orRecordBatch

A new tabular object sorted according to the sort keys.

Examples

Table (works similarly for RecordBatch)

>>>importpandasaspd>>>importpyarrowaspa>>>df=pd.DataFrame({'year':[2020,2022,2021,2022,2019,2021],...'n_legs':[2,2,4,4,5,100],...'animal':["Flamingo","Parrot","Dog","Horse",..."Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.sort_by('animal')pyarrow.Tableyear: int64n_legs: int64animal: string----year: [[2019,2021,2021,2020,2022,2022]]n_legs: [[5,100,4,2,4,2]]animal: [["Brittle stars","Centipede","Dog","Flamingo","Horse","Parrot"]]

take(self,indices)#

Select rows from a Table or RecordBatch.

Seepyarrow.compute.take() for full usage.

Parameters:

indicesArray orarray-like

The indices in the tabular object whose rows will be returned.

Returns:

Table orRecordBatch

A tabular object with the same schema, containing the taken rows.

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.take([1,3])pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2022,2021]]n_legs: [[4,100]]animals: [["Horse","Centipede"]]

to_pandas(self,memory_pool=None,categories=None,boolstrings_to_categorical=False,boolzero_copy_only=False,boolinteger_object_nulls=False,booldate_as_object=True,booltimestamp_as_object=False,booluse_threads=True,booldeduplicate_objects=True,boolignore_metadata=False,boolsafe=True,boolsplit_blocks=False,boolself_destruct=False,strmaps_as_pydicts=None,types_mapper=None,boolcoerce_temporal_nanoseconds=False)#

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters:

memory_poolMemoryPool, defaultNone

Arrow MemoryPool to use for allocations. Uses the default memorypool if not passed.

categorieslist, defaultempty

List of fields that should be returned as pandas.Categorical. Onlyapplies to table-like data structures.

strings_to_categoricalbool, defaultFalse

Encode string (UTF8) and binary types to pandas.Categorical.

zero_copy_onlybool, defaultFalse

Raise an ArrowException if this function call would require copyingthe underlying data.

integer_object_nullsbool, defaultFalse

Cast integers with nulls to objects

date_as_objectbool, defaultTrue

Cast dates to objects. If False, convert to datetime64 dtype withthe equivalent time unit (if supported). Note: in pandas version< 2.0, only datetime64[ns] conversion is supported.

timestamp_as_objectbool, defaultFalse

Cast non-nanosecond timestamps (np.datetime64) to objects. This isuseful in pandas version 1.x if you have timestamps that don’t fitin the normal date range of nanosecond timestamps (1678 CE-2262 CE).Non-nanosecond timestamps are supported in pandas version 2.0.If False, all timestamps are converted to datetime64 dtype.

use_threadsbool, defaultTrue

Whether to parallelize the conversion using multiple threads.

deduplicate_objectsbool, defaultTrue

Do not create multiple copies Python objects when created, to saveon memory use. Conversion will be slower.

ignore_metadatabool, defaultFalse

If True, do not use the ‘pandas’ metadata to reconstruct theDataFrame index, if present

safebool, defaultTrue

For certain data types, a cast is needed in order to store thedata in a pandas DataFrame or Series (e.g. timestamps are alwaysstored as nanoseconds in pandas). This option controls whether itis a safe cast or not.

split_blocksbool, defaultFalse

If True, generate one internal “block” for each column whencreating a pandas.DataFrame from a RecordBatch or Table. While thiscan temporarily reduce memory note that various pandas operationscan trigger “consolidation” which may balloon memory use.

self_destructbool, defaultFalse

EXPERIMENTAL: If True, attempt to deallocate the originating Arrowmemory while converting the Arrow object to pandas. If you use theobject after calling to_pandas with this option it will crash yourprogram.

Note that you may not see always memory usage improvements. Forexample, if multiple columns share an underlying allocation,memory can’t be freed until all columns are converted.

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.This can change the ordering of (key, value) pairs, and willdeduplicate multiple keys, resulting in a possible loss of data.

If ‘lossy’, this key deduplication results in a warning printedwhen detected. If ‘strict’, this instead results in an exceptionbeing raised when detected.

types_mapperfunction, defaultNone

A function mapping a pyarrow DataType to a pandas ExtensionDtype.This can be used to override the default pandas type for conversionof built-in pyarrow types or in absence of pandas_metadata in theTable schema. The function receives a pyarrow DataType and isexpected to return a pandas ExtensionDtype orNone if thedefault conversion should be used for that type. If you havea dictionary mapping, you can passdict.get as function.

coerce_temporal_nanosecondsbool, defaultFalse

Only applicable to pandas version >= 2.0.A legacy option to coerce date32, date64, duration, and timestamptime units to nanoseconds when converting to pandas. This is thedefault behavior in pandas version 1.x. Set this option to True ifyou’d like to use this coercion when using pandas version >= 2.0for backwards compatibility (not recommended otherwise).

Returns:

pandas.Series orpandas.DataFrame depending ontype of object

Examples

>>>importpyarrowaspa>>>importpandasaspd

Convert a Table to pandas DataFrame:

>>>table=pa.table([...pa.array([2,4,5,100]),...pa.array(["Flamingo","Horse","Brittle stars","Centipede"])...],names=['n_legs','animals'])>>>table.to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede>>>isinstance(table.to_pandas(),pd.DataFrame)True

Convert a RecordBatch to pandas DataFrame:

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>batch=pa.record_batch([n_legs,animals],...names=["n_legs","animals"])>>>batchpyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]>>>batch.to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede>>>isinstance(batch.to_pandas(),pd.DataFrame)True

Convert a Chunked Array to pandas Series:

>>>importpyarrowaspa>>>n_legs=pa.chunked_array([[2,2,4],[4,5,100]])>>>n_legs.to_pandas()0      21      22      43      44      55    100dtype: int64>>>isinstance(n_legs.to_pandas(),pd.Series)True

to_pydict(self,*,maps_as_pydicts=None)#

Convert the Table or RecordBatch to a dict or OrderedDict.

Parameters:

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.

If ‘lossy’, whenever duplicate keys are detected, a warning will be printed.The last seen value of a duplicate key will be in the Python dictionary.If ‘strict’, this instead results in an exception being raised when detected.

Returns:

dict

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>table=pa.Table.from_arrays([n_legs,animals],names=["n_legs","animals"])>>>table.to_pydict(){'n_legs': [2, 2, 4, 4, 5, 100], 'animals': ['Flamingo', 'Parrot', ..., 'Centipede']}

to_pylist(self,*,maps_as_pydicts=None)#

Convert the Table or RecordBatch to a list of rows / dictionaries.

Parameters:

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.

If ‘lossy’, whenever duplicate keys are detected, a warning will be printed.The last seen value of a duplicate key will be in the Python dictionary.If ‘strict’, this instead results in an exception being raised when detected.

Returns:

list

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>data=[[2,4,5,100],...["Flamingo","Horse","Brittle stars","Centipede"]]>>>table=pa.table(data,names=["n_legs","animals"])>>>table.to_pylist()[{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 4, 'animals': 'Horse'}, ...

to_string(self,*,show_metadata=False,preview_cols=0)#

Return human-readable string representation of Table or RecordBatch.

Parameters:

show_metadatabool, defaultFalse

Display Field-level and Schema-level KeyValueMetadata.

preview_colsint, default 0

Display values of the columns for the first N columns.

Returns:

str

to_struct_array(self)#

Convert to a struct array.

to_tensor(self,boolnull_to_nan=False,boolrow_major=True,MemoryPoolmemory_pool=None)#

Convert to aTensor.

RecordBatches that can be converted have fields of type signed or unsignedinteger or float, including all bit-widths.

null_to_nan isFalse by default and this method will raise an error in caseany nulls are present. RecordBatches with nulls can be converted withnull_to_nanset toTrue. In this case null values are converted toNaN and integer typearrays are promoted to the appropriate float type.

Parameters:

null_to_nanbool, defaultFalse

Whether to write null values in the result asNaN.

row_majorbool, defaultTrue

Whether resulting Tensor is row-major or column-major

memory_poolMemoryPool, defaultNone

For memory allocations, if required, otherwise use default pool

Examples

>>>importpyarrowaspa>>>batch=pa.record_batch(...[...pa.array([1,2,3,4,None],type=pa.int32()),...pa.array([10,20,30,40,None],type=pa.float32()),...],names=["a","b"]...)

>>>batchpyarrow.RecordBatcha: int32b: float----a: [1,2,3,4,null]b: [10,20,30,40,null]

Convert a RecordBatch to row-major Tensor with null valueswritten as``NaN``s

>>>batch.to_tensor(null_to_nan=True)<pyarrow.Tensor>type: doubleshape: (5, 2)strides: (16, 8)>>>batch.to_tensor(null_to_nan=True).to_numpy()array([[ 1., 10.],       [ 2., 20.],       [ 3., 30.],       [ 4., 40.],       [nan, nan]])

Convert a RecordBatch to column-major Tensor

>>>batch.to_tensor(null_to_nan=True,row_major=False)<pyarrow.Tensor>type: doubleshape: (5, 2)strides: (8, 40)>>>batch.to_tensor(null_to_nan=True,row_major=False).to_numpy()array([[ 1., 10.],       [ 2., 20.],       [ 3., 30.],       [ 4., 40.],       [nan, nan]])

validate(self,*,full=False)#

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Passfull=Truefor thorough validation checks (potentially O(n)).

Parameters:

fullbool, defaultFalse

If True, run expensive checks, otherwise cheap checks only.

Raises:

ArrowInvalid