pyarrow.Table#

classpyarrow.Table#

Bases:_Tabular

A collection of top-level named, equal length Arrow arrays.

Warning

Do not call this class’s constructor directly, use one of thefrom_*methods instead.

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>names=["n_legs","animals"]

Construct a Table from arrays:

>>>pa.Table.from_arrays([n_legs,animals],names=names)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from a RecordBatch:

>>>batch=pa.record_batch([n_legs,animals],names=names)>>>pa.Table.from_batches([batch])pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from pandas DataFrame:

>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>pa.Table.from_pandas(df)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2022,2019,2021]]n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from a dictionary of arrays:

>>>pydict={'n_legs':n_legs,'animals':animals}>>>pa.Table.from_pydict(pydict)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]>>>pa.Table.from_pydict(pydict).scheman_legs: int64animals: string

Construct a Table from a dictionary of arrays with metadata:

>>>my_metadata={"n_legs":"Number of legs per animal"}>>>pa.Table.from_pydict(pydict,metadata=my_metadata).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

Construct a Table from a list of rows:

>>>pylist=[{'n_legs':2,'animals':'Flamingo'},{'year':2021,'animals':'Centipede'}]>>>pa.Table.from_pylist(pylist)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,null]]animals: [["Flamingo","Centipede"]]

Construct a Table from a list of rows with pyarrow schema:

>>>my_schema=pa.schema([...pa.field('year',pa.int64()),...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"year":"Year of entry"})>>>pa.Table.from_pylist(pylist,schema=my_schema).schemayear: int64n_legs: int64animals: string-- schema metadata --year: 'Year of entry'

Construct a Table withpyarrow.table():

>>>pa.table([n_legs,animals],names=names)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

__init__(*args,**kwargs)#

Methods

__init__(*args, **kwargs)
add_column(self, int i, field_, column)	Add column to Table at position.
append_column(self, field_, column)	Append column at end of columns.
cast(self, Schema target_schema[, safe, options])	Cast table values to another schema.
column(self, i)	Select single column from Table or RecordBatch.
combine_chunks(self, MemoryPool memory_pool=None)	Make a new table by combining the chunks this table has.
drop(self, columns)	Drop one or more columns and return a new table.
drop_columns(self, columns)	Drop one or more columns and return a new Table or RecordBatch.
drop_null(self)	Remove rows that contain missing values from a Table or RecordBatch.
equals(self, Table other, ...)	Check if contents of two tables are equal.
field(self, i)	Select a schema field by its column name or numeric index.
filter(self, mask[, null_selection_behavior])	Select rows from the table or record batch based on a boolean mask.
flatten(self, MemoryPool memory_pool=None)	Flatten this Table.
from_arrays(arrays[, names, schema, metadata])	Construct a Table from Arrow arrays.
from_batches(batches, Schema schema=None)	Construct a Table from a sequence or iterator of Arrow RecordBatches.
from_pandas(cls, df, Schema schema=None[, ...])	Convert pandas.DataFrame to an Arrow Table.
from_pydict(cls, mapping[, schema, metadata])	Construct a Table or RecordBatch from Arrow arrays or columns.
from_pylist(cls, mapping[, schema, metadata])	Construct a Table or RecordBatch from list of rows / dictionaries.
from_struct_array(struct_array)	Construct a Table from a StructArray.
get_total_buffer_size(self)	The sum of bytes in each buffer referenced by the table.
group_by(self, keys[, use_threads])	Declare a grouping over the columns of the table.
itercolumns(self)	Iterator over all columns in their numerical order.
join(self, right_table, keys[, right_keys, ...])	Perform a join between this table and another one.
join_asof(self, right_table, on, by, tolerance)	Perform an asof join between this table and another one.
remove_column(self, int i)	Create new Table with the indicated column removed.
rename_columns(self, names)	Create new table with columns renamed to provided names.
replace_schema_metadata(self[, metadata])	Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.
select(self, columns)	Select columns of the Table.
set_column(self, int i, field_, column)	Replace column in Table at position.
slice(self[, offset, length])	Compute zero-copy slice of this Table.
sort_by(self, sorting, **kwargs)	Sort the Table or RecordBatch by one or multiple columns.
take(self, indices)	Select rows from a Table or RecordBatch.
to_batches(self[, max_chunksize])	Convert Table to a list of RecordBatch objects.
to_pandas(self[, memory_pool, categories, ...])	Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
to_pydict(self, *[, maps_as_pydicts])	Convert the Table or RecordBatch to a dict or OrderedDict.
to_pylist(self, *[, maps_as_pydicts])	Convert the Table or RecordBatch to a list of rows / dictionaries.
to_reader(self[, max_chunksize])	Convert the Table to a RecordBatchReader.
to_string(self, *[, show_metadata, preview_cols])	Return human-readable string representation of Table or RecordBatch.
to_struct_array(self[, max_chunksize])	Convert to a chunked array of struct type.
unify_dictionaries(self, ...)	Unify dictionaries across all chunks.
validate(self, *[, full])	Perform validation checks.

Attributes

column_names	Names of the Table or RecordBatch columns.
columns	List of all columns in numerical order.
is_cpu	Whether all ChunkedArrays are CPU-accessible.
nbytes	Total number of bytes consumed by the elements of the table.
num_columns	Number of columns in this table.
num_rows	Number of rows in this table.
schema	Schema of the table and its columns.
shape	Dimensions of the table or record batch: (#rows, #columns).

__dataframe__(self,nan_as_null:bool=False,allow_copy:bool=True)#

Return the dataframe interchange object implementing the interchange protocol.

Parameters:

nan_as_nullbool, defaultFalse

Whether to tell the DataFrame to overwrite null values in the datawithNaN (orNaT).

allow_copybool, defaultTrue

Whether to allow memory copying when exporting. If set to Falseit would cause non-zero-copy exports to fail.

Returns:

DataFrameinterchange object

The object which consuming library can use to ingress the dataframe.

Notes

Details on the interchange protocol:https://data-apis.org/dataframe-protocol/latest/index.htmlnan_as_null currently has no effect; once support for nullable extensiondtypes is added, this value should be propagated to columns.

add_column(self,inti,field_,column)#

Add column to Table at position.

A new table is returned with the column added, the original tableobject is left unchanged.

Parameters:

iint

Index to place the column at.

field_str orField

If a string is passed then the type is deduced from the columndata.

columnArray,list ofArray, or values coercible to arrays

Column data.

Returns:

Table

New table with the passed column added.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Add column:

>>>year=[2021,2022,2019,2021]>>>table.add_column(0,"year",[year])pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2021,2022,2019,2021]]n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Original table is left unchanged:

>>>tablepyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

append_column(self,field_,column)#

Append column at end of columns.

Parameters:

field_str orField

If a string is passed then the type is deduced from the columndata.

columnArray orvalue coercible toarray

Column data.

Returns:

Table orRecordBatch

New table or record batch with the passed column added.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Append column at the end:

>>>year=[2021,2022,2019,2021]>>>table.append_column('year',[year])pyarrow.Tablen_legs: int64animals: stringyear: int64----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]year: [[2021,2022,2019,2021]]

cast(self,Schematarget_schema,safe=None,options=None)#

Cast table values to another schema.

Parameters:

target_schemaSchema

Schema to cast to, the names and order of fields must match.

safebool, defaultTrue

Check for overflows or other unsafe conversions.

optionsCastOptions, defaultNone

Additional checks pass by CastOptions

Returns:

Table

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.scheman_legs: int64animals: string-- schema metadata --pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ...

Define new schema and cast table values:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.duration('s')),...pa.field('animals',pa.string())]...)>>>table.cast(target_schema=my_schema)pyarrow.Tablen_legs: duration[s]animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

column(self,i)#

Select single column from Table or RecordBatch.

Parameters:

iint orstr

The index or name of the column to retrieve.

Returns:

columnArray (forRecordBatch) orChunkedArray (forTable)

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Select a column by numeric index:

>>>table.column(0)<pyarrow.lib.ChunkedArray object at ...>[  [    2,    4,    5,    100  ]]

Select a column by its name:

>>>table.column("animals")<pyarrow.lib.ChunkedArray object at ...>[  [    "Flamingo",    "Horse",    "Brittle stars",    "Centipede"  ]]

column_names#

Names of the Table or RecordBatch columns.

Returns:

list ofstr

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>table=pa.Table.from_arrays([[2,4,5,100],...["Flamingo","Horse","Brittle stars","Centipede"]],...names=['n_legs','animals'])>>>table.column_names['n_legs', 'animals']

columns#

List of all columns in numerical order.

Returns:

columnslist ofArray (forRecordBatch) orlist ofChunkedArray (forTable)

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.columns[<pyarrow.lib.ChunkedArray object at ...>[  [    null,    4,    5,    null  ]], <pyarrow.lib.ChunkedArray object at ...>[  [    "Flamingo",    "Horse",    null,    "Centipede"  ]]]

combine_chunks(self,MemoryPoolmemory_pool=None)#

Make a new table by combining the chunks this table has.

All the underlying chunks in the ChunkedArray of each column areconcatenated into zero or one chunk.

Parameters:

memory_poolMemoryPool, defaultNone

For memory allocations, if required, otherwise use default pool.

Returns:

Table

Examples

>>>importpyarrowaspa>>>n_legs=pa.chunked_array([[2,2,4],[4,5,100]])>>>animals=pa.chunked_array([["Flamingo","Parrot","Dog"],["Horse","Brittle stars","Centipede"]])>>>names=["n_legs","animals"]>>>table=pa.table([n_legs,animals],names=names)>>>tablepyarrow.Tablen_legs: int64animals: string----n_legs: [[2,2,4],[4,5,100]]animals: [["Flamingo","Parrot","Dog"],["Horse","Brittle stars","Centipede"]]>>>table.combine_chunks()pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,2,4,4,5,100]]animals: [["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]]

drop(self,columns)#

Drop one or more columns and return a new table.

Alias of Table.drop_columns, but kept for backwards compatibility.

Parameters:

columnsstr orlist[str]

Field name(s) referencing existing column(s).

Returns:

Table

New table without the column(s).

drop_columns(self,columns)#

Drop one or more columns and return a new Table or RecordBatch.

Parameters:

columnsstr orlist[str]

Field name(s) referencing existing column(s).

Returns:

Table orRecordBatch

A tabular object without the column(s).

Raises:

KeyError

If any of the passed column names do not exist.

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Drop one column:

>>>table.drop_columns("animals")pyarrow.Tablen_legs: int64----n_legs: [[2,4,5,100]]

Drop one or more columns:

>>>table.drop_columns(["n_legs","animals"])pyarrow.Table...----

drop_null(self)#

Remove rows that contain missing values from a Table or RecordBatch.

Seepyarrow.compute.drop_null() for full usage.

Returns:

Table orRecordBatch

A tabular object with the same schema, with rows containingno missing values.

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'year':[None,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.drop_null()pyarrow.Tableyear: doublen_legs: int64animals: string----year: [[2022,2021]]n_legs: [[4,100]]animals: [["Horse","Centipede"]]

equals(self,Tableother,boolcheck_metadata=False)#

Check if contents of two tables are equal.

Parameters:

otherpyarrow.Table

Table to compare against.

check_metadatabool, defaultFalse

Whether schema metadata equality should be checked as well.

Returns:

bool

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>names=["n_legs","animals"]>>>table=pa.Table.from_arrays([n_legs,animals],names=names)>>>table_0=pa.Table.from_arrays([])>>>table_1=pa.Table.from_arrays([n_legs,animals],...names=names,...metadata={"n_legs":"Number of legs per animal"})>>>table.equals(table)True>>>table.equals(table_0)False>>>table.equals(table_1)True>>>table.equals(table_1,check_metadata=True)False

field(self,i)#

Select a schema field by its column name or numeric index.

Parameters:

iint orstr

The index or name of the field to retrieve.

Returns:

Field

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.field(0)pyarrow.Field<n_legs: int64>>>>table.field(1)pyarrow.Field<animals: string>

filter(self,mask,null_selection_behavior='drop')#

Select rows from the table or record batch based on a boolean mask.

The Table can be filtered based on a mask, which will be passed topyarrow.compute.filter() to perform the filtering, or it canbe filtered through a booleanExpression

Parameters:

maskArray orarray-like orExpression

The boolean mask or theExpression to filter the table with.

null_selection_behaviorstr, default “drop”

How nulls in the mask should be handled, does nothing ifanExpression is used.

Returns:

filteredTable orRecordBatch

A tabular object of the same schema, with only the rows selectedby applied filtering

Examples

Using a Table (works similarly for RecordBatch):

>>>importpyarrowaspa>>>table=pa.table({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})

Define an expression and select rows:

>>>importpyarrow.computeaspc>>>expr=pc.field("year")<=2020>>>table.filter(expr)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2019]]n_legs: [[2,5]]animals: [["Flamingo","Brittle stars"]]

Define a mask and select rows:

>>>mask=[True,True,False,None]>>>table.filter(mask)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2022]]n_legs: [[2,4]]animals: [["Flamingo","Horse"]]>>>table.filter(mask,null_selection_behavior='emit_null')pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2022,null]]n_legs: [[2,4,null]]animals: [["Flamingo","Horse",null]]

flatten(self,MemoryPoolmemory_pool=None)#

Flatten this Table.

Each column with a struct type is flattenedinto one column per struct field. Other columns are left unchanged.

Parameters:

memory_poolMemoryPool, defaultNone

For memory allocations, if required, otherwise use default pool

Returns:

Table

Examples

>>>importpyarrowaspa>>>struct=pa.array([{'n_legs':2,'animals':'Parrot'},...{'year':2022,'n_legs':4}])>>>month=pa.array([4,6])>>>table=pa.Table.from_arrays([struct,month],...names=["a","month"])>>>tablepyarrow.Tablea: struct<animals: string, n_legs: int64, year: int64>  child 0, animals: string  child 1, n_legs: int64  child 2, year: int64month: int64----a: [  -- is_valid: all not null  -- child 0 type: string["Parrot",null]  -- child 1 type: int64[2,4]  -- child 2 type: int64[null,2022]]month: [[4,6]]

Flatten the columns with struct field:

>>>table.flatten()pyarrow.Tablea.animals: stringa.n_legs: int64a.year: int64month: int64----a.animals: [["Parrot",null]]a.n_legs: [[2,4]]a.year: [[null,2022]]month: [[4,6]]

staticfrom_arrays(arrays,names=None,schema=None,metadata=None)#

Construct a Table from Arrow arrays.

Parameters:

arrayslist ofpyarrow.Array orpyarrow.ChunkedArray

Equal-length arrays that should form the table.

nameslist ofstr, optional

Names for the table columns. If not passed, schema must be passed.

schemaSchema, defaultNone

Schema for the created table. If not passed, names must be passed.

metadatadict or Mapping, defaultNone

Optional metadata for the schema (if inferred).

Returns:

Table

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>names=["n_legs","animals"]

Construct a Table from arrays:

>>>pa.Table.from_arrays([n_legs,animals],names=names)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from arrays with metadata:

>>>my_metadata={"n_legs":"Number of legs per animal"}>>>pa.Table.from_arrays([n_legs,animals],...names=names,...metadata=my_metadata)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]>>>pa.Table.from_arrays([n_legs,animals],...names=names,...metadata=my_metadata).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

Construct a Table from arrays with pyarrow schema:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"animals":"Name of the animal species"})>>>pa.Table.from_arrays([n_legs,animals],...schema=my_schema)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]>>>pa.Table.from_arrays([n_legs,animals],...schema=my_schema).scheman_legs: int64animals: string-- schema metadata --animals: 'Name of the animal species'

staticfrom_batches(batches,Schemaschema=None)#

Construct a Table from a sequence or iterator of Arrow RecordBatches.

Parameters:

batchessequence or iterator ofRecordBatch

Sequence of RecordBatch to be converted, all schemas must be equal.

schemaSchema, defaultNone

If not passed, will be inferred from the first RecordBatch.

Returns:

Table

Examples

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>names=["n_legs","animals"]>>>batch=pa.record_batch([n_legs,animals],names=names)>>>batch.to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede

Construct a Table from a RecordBatch:

>>>pa.Table.from_batches([batch])pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

Construct a Table from a sequence of RecordBatches:

>>>pa.Table.from_batches([batch,batch])pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100],[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"],["Flamingo","Horse","Brittle stars","Centipede"]]

classmethodfrom_pandas(cls,df,Schemaschema=None,preserve_index=None,nthreads=None,columns=None,boolsafe=True)#

Convert pandas.DataFrame to an Arrow Table.

The column types in the resulting Arrow Table are inferred from thedtypes of the pandas.Series in the DataFrame. In the case of non-objectSeries, the NumPy dtype is translated to its Arrow equivalent. In thecase ofobject, we need to guess the datatype by looking at thePython objects in this Series.

Be aware that Series of theobject dtype don’t carry enoughinformation to always lead to a meaningful Arrow type. In the case thatwe cannot infer a type, e.g. because the DataFrame is of length 0 orthe Series only contains None/nan objects, the type is set tonull. This behavior can be avoided by constructing an explicit schemaand passing it to this function.

Parameters:

dfpandas.DataFrame

schemapyarrow.Schema, optional

The expected schema of the Arrow Table. This can be used toindicate the type of columns if we cannot infer it automatically.If passed, the output will have exactly this schema. Columnsspecified in the schema that are not found in the DataFrame columnsor its index will raise an error. Additional columns or indexlevels in the DataFrame which are not specified in the schema willbe ignored.

preserve_indexbool, optional

Whether to store the index as an additional column in the resultingTable. The default of None will store the index as a column,except for RangeIndex which is stored as metadata only. Usepreserve_index=True to force it to be stored as a column.

nthreadsint, defaultNone

If greater than 1, convert columns to Arrow in parallel usingindicated number of threads. By default, this followspyarrow.cpu_count() (may use up to system CPU count threads).

columnslist, optional

List of column to be converted. If None, use all columns.

safebool, defaultTrue

Check for overflows or other unsafe conversions.

Returns:

Table

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>pa.Table.from_pandas(df)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

classmethodfrom_pydict(cls,mapping,schema=None,metadata=None)#

Construct a Table or RecordBatch from Arrow arrays or columns.

Parameters:

mappingdict or Mapping

A mapping of strings to Arrays or Python lists.

schemaSchema, defaultNone

If not passed, will be inferred from the Mapping values.

metadatadict or Mapping, defaultNone

Optional metadata for the schema (if inferred).

Returns:

Table orRecordBatch

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>pydict={'n_legs':n_legs,'animals':animals}

Construct a Table from a dictionary of arrays:

>>>pa.Table.from_pydict(pydict)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]>>>pa.Table.from_pydict(pydict).scheman_legs: int64animals: string

Construct a Table from a dictionary of arrays with metadata:

>>>my_metadata={"n_legs":"Number of legs per animal"}>>>pa.Table.from_pydict(pydict,metadata=my_metadata).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

Construct a Table from a dictionary of arrays with pyarrow schema:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"n_legs":"Number of legs per animal"})>>>pa.Table.from_pydict(pydict,schema=my_schema).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

classmethodfrom_pylist(cls,mapping,schema=None,metadata=None)#

Construct a Table or RecordBatch from list of rows / dictionaries.

Parameters:

mappinglist of dicts of rows

A mapping of strings to row values.

schemaSchema, defaultNone

If not passed, will be inferred from the first row of themapping values.

metadatadict or Mapping, defaultNone

Optional metadata for the schema (if inferred).

Returns:

Table orRecordBatch

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>pylist=[{'n_legs':2,'animals':'Flamingo'},...{'n_legs':4,'animals':'Dog'}]

Construct a Table from a list of rows:

>>>pa.Table.from_pylist(pylist)pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4]]animals: [["Flamingo","Dog"]]

Construct a Table from a list of rows with metadata:

>>>my_metadata={"n_legs":"Number of legs per animal"}>>>pa.Table.from_pylist(pylist,metadata=my_metadata).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

Construct a Table from a list of rows with pyarrow schema:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"n_legs":"Number of legs per animal"})>>>pa.Table.from_pylist(pylist,schema=my_schema).scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'

staticfrom_struct_array(struct_array)#

Construct a Table from a StructArray.

Each field in the StructArray will become a column in the resultingTable.

Parameters:

struct_arrayStructArray orChunkedArray

Array to construct the table from.

Returns:

pyarrow.Table

Examples

>>>importpyarrowaspa>>>struct=pa.array([{'n_legs':2,'animals':'Parrot'},...{'year':2022,'n_legs':4}])>>>pa.Table.from_struct_array(struct).to_pandas()  animals  n_legs    year0  Parrot       2     NaN1    None       4  2022.0

get_total_buffer_size(self)#

The sum of bytes in each buffer referenced by the table.

An array may only reference a portion of a buffer.This method will overestimate in this case and return thebyte size of the entire buffer.

If a buffer is referenced multiple times then it willonly be counted once.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.get_total_buffer_size()76

group_by(self,keys,use_threads=True)#

Declare a grouping over the columns of the table.

Resulting grouping can then be used to perform aggregationswith a subsequentaggregate() method.

Parameters:

keysstr orlist[str]

Name of the columns that should be used as the grouping key.

use_threadsbool, defaultTrue

Whether to use multithreading or not. When set to True (thedefault), no stable ordering of the output is guaranteed.

Returns:

TableGroupBy

See also

TableGroupBy.aggregate

Examples

>>>importpandasaspd>>>importpyarrowaspa>>>df=pd.DataFrame({'year':[2020,2022,2021,2022,2019,2021],...'n_legs':[2,2,4,4,5,100],...'animal':["Flamingo","Parrot","Dog","Horse",..."Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.group_by('year').aggregate([('n_legs','sum')])pyarrow.Tableyear: int64n_legs_sum: int64----year: [[2020,2022,2021,2019]]n_legs_sum: [[2,6,104,5]]

is_cpu#

Whether all ChunkedArrays are CPU-accessible.

itercolumns(self)#

Iterator over all columns in their numerical order.

Yields:

Array (forRecordBatch) orChunkedArray (forTable)

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>foriintable.itercolumns():...print(i.null_count)...21

join(self,right_table,keys,right_keys=None,join_type='leftouter',left_suffix=None,right_suffix=None,coalesce_keys=True,use_threads=True)#

Perform a join between this table and another one.

Result of the join will be a new Table, where furtheroperations can be applied.

Parameters:

right_tableTable

The table to join to the current one, acting as the right tablein the join operation.

keysstr orlist[str]

The columns from current table that should be used as keysof the join operation left side.

right_keysstr orlist[str], defaultNone

The columns from the right_table that should be used as keyson the join operation right side.WhenNone use the same key names as the left table.

join_typestr, default “left outer”

The kind of join that should be performed, one of(“left semi”, “right semi”, “left anti”, “right anti”,“inner”, “left outer”, “right outer”, “full outer”)

left_suffixstr, defaultNone

Which suffix to add to left column names. This prevents confusionwhen the columns in left and right tables have colliding names.

right_suffixstr, defaultNone

Which suffix to add to the right column names. This prevents confusionwhen the columns in left and right tables have colliding names.

coalesce_keysbool, defaultTrue

If the duplicated keys should be omitted from one of the sidesin the join result.

use_threadsbool, defaultTrue

Whether to use multithreading or not.

Returns:

Table

Examples

>>>importpandasaspd>>>importpyarrowaspa>>>df1=pd.DataFrame({'id':[1,2,3],...'year':[2020,2022,2019]})>>>df2=pd.DataFrame({'id':[3,4],...'n_legs':[5,100],...'animal':["Brittle stars","Centipede"]})>>>t1=pa.Table.from_pandas(df1)>>>t2=pa.Table.from_pandas(df2)

Left outer join:

>>>t1.join(t2,'id').combine_chunks().sort_by('year')pyarrow.Tableid: int64year: int64n_legs: int64animal: string----id: [[3,1,2]]year: [[2019,2020,2022]]n_legs: [[5,null,null]]animal: [["Brittle stars",null,null]]

Full outer join:

>>>t1.join(t2,'id',join_type="full outer").combine_chunks().sort_by('year')pyarrow.Tableid: int64year: int64n_legs: int64animal: string----id: [[3,1,2,4]]year: [[2019,2020,2022,null]]n_legs: [[5,null,null,100]]animal: [["Brittle stars",null,null,"Centipede"]]

Right outer join:

>>>t1.join(t2,'id',join_type="right outer").combine_chunks().sort_by('year')pyarrow.Tableyear: int64id: int64n_legs: int64animal: string----year: [[2019,null]]id: [[3,4]]n_legs: [[5,100]]animal: [["Brittle stars","Centipede"]]

Right anti join

>>>t1.join(t2,'id',join_type="right anti")pyarrow.Tableid: int64n_legs: int64animal: string----id: [[4]]n_legs: [[100]]animal: [["Centipede"]]

join_asof(self,right_table,on,by,tolerance,right_on=None,right_by=None)#

Perform an asof join between this table and another one.

This is similar to a left-join except that we match on nearest key ratherthan equal keys. Both tables must be sorted by the key. This type of joinis most useful for time series data that are not perfectly aligned.

Optionally match on equivalent keys with “by” before searching with “on”.

Result of the join will be a new Table, where furtheroperations can be applied.

Parameters:

right_tableTable

The table to join to the current one, acting as the right tablein the join operation.

onstr

The column from current table that should be used as the “on” keyof the join operation left side.

An inexact match is used on the “on” key, i.e. a row is considered amatch if and only if left_on - tolerance <= right_on <= left_on.

The input dataset must be sorted by the “on” key. Must be a singlefield of a common type.

Currently, the “on” key must be an integer, date, or timestamp type.

bystr orlist[str]

The columns from current table that should be used as the keysof the join operation left side. The join operation is then doneonly for the matches in these columns.

toleranceint

The tolerance for inexact “on” key matching. A right row is considereda match with the left rowright.on-left.on<=tolerance. Thetolerance may be:

• negative, in which case a past-as-of-join occurs;

• or positive, in which case a future-as-of-join occurs;

• or zero, in which case an exact-as-of-join occurs.

The tolerance is interpreted in the same units as the “on” key.

right_onstr orlist[str], defaultNone

The columns from the right_table that should be used as the on keyon the join operation right side.WhenNone use the same key name as the left table.

right_bystr orlist[str], defaultNone

The columns from the right_table that should be used as keyson the join operation right side.WhenNone use the same key names as the left table.

Returns:

Table

nbytes#

Total number of bytes consumed by the elements of the table.

In other words, the sum of bytes from all buffer ranges referenced.

Unlikeget_total_buffer_size this method will account for arrayoffsets.

If buffers are shared between arrays then the sharedportion will only be counted multiple times.

The dictionary of dictionary arrays will always be counted in theirentirety even if the array only references a portion of the dictionary.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.nbytes72

num_columns#

Number of columns in this table.

Returns:

int

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.num_columns2

num_rows#

Number of rows in this table.

Due to the definition of a table, all columns have the same number ofrows.

Returns:

int

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.num_rows4

remove_column(self,inti)#

Create new Table with the indicated column removed.

Parameters:

iint

Index of column to remove.

Returns:

Table

New table without the column.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.remove_column(1)pyarrow.Tablen_legs: int64----n_legs: [[2,4,5,100]]

rename_columns(self,names)#

Create new table with columns renamed to provided names.

Parameters:

nameslist[str] ordict[str,str]

List of new column names or mapping of old column names to new column names.

If a mapping of old to new column names is passed, then all columns which arefound to match a provided old column name will be renamed to the new column name.If any column names are not found in the mapping, a KeyError will be raised.

Returns:

Table

Raises:

KeyError

If any of the column names passed in the names mapping do not exist.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>new_names=["n","name"]>>>table.rename_columns(new_names)pyarrow.Tablen: int64name: string----n: [[2,4,5,100]]name: [["Flamingo","Horse","Brittle stars","Centipede"]]>>>new_names={"n_legs":"n","animals":"name"}>>>table.rename_columns(new_names)pyarrow.Tablen: int64name: string----n: [[2,4,5,100]]name: [["Flamingo","Horse","Brittle stars","Centipede"]]

replace_schema_metadata(self,metadata=None)#

Create shallow copy of table by replacing schemakey-value metadata with the indicated new metadata (which may be None),which deletes any existing metadata.

Parameters:

metadatadict, defaultNone

Returns:

Table

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Constructing a Table with pyarrow schema and metadata:

>>>my_schema=pa.schema([...pa.field('n_legs',pa.int64()),...pa.field('animals',pa.string())],...metadata={"n_legs":"Number of legs per animal"})>>>table=pa.table(df,my_schema)>>>table.scheman_legs: int64animals: string-- schema metadata --n_legs: 'Number of legs per animal'pandas: ...

Create a shallow copy of a Table with deleted schema metadata:

>>>table.replace_schema_metadata().scheman_legs: int64animals: string

Create a shallow copy of a Table with new schema metadata:

>>>metadata={"animals":"Which animal"}>>>table.replace_schema_metadata(metadata=metadata).scheman_legs: int64animals: string-- schema metadata --animals: 'Which animal'

schema#

Schema of the table and its columns.

Returns:

Schema

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.scheman_legs: int64animals: string-- schema metadata --pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' ...

select(self,columns)#

Select columns of the Table.

Returns a new Table with the specified columns, and metadatapreserved.

Parameters:

columnslist-like

The column names or integer indices to select.

Returns:

Table

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.select([0,1])pyarrow.Tableyear: int64n_legs: int64----year: [[2020,2022,2019,2021]]n_legs: [[2,4,5,100]]>>>table.select(["year"])pyarrow.Tableyear: int64----year: [[2020,2022,2019,2021]]

set_column(self,inti,field_,column)#

Replace column in Table at position.

Parameters:

iint

Index to place the column at.

field_str orField

If a string is passed then the type is deduced from the columndata.

columnArray,list ofArray, or values coercible to arrays

Column data.

Returns:

Table

New table with the passed column set.

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Replace a column:

>>>year=[2021,2022,2019,2021]>>>table.set_column(1,'year',[year])pyarrow.Tablen_legs: int64year: int64----n_legs: [[2,4,5,100]]year: [[2021,2022,2019,2021]]

shape#

Dimensions of the table or record batch: (#rows, #columns).

Returns:

(int,int)

Number of rows and number of columns.

Examples

>>>importpyarrowaspa>>>table=pa.table({'n_legs':[None,4,5,None],...'animals':["Flamingo","Horse",None,"Centipede"]})>>>table.shape(4, 2)

slice(self,offset=0,length=None)#

Compute zero-copy slice of this Table.

Parameters:

offsetint, default 0

Offset from start of table to slice.

lengthint, defaultNone

Length of slice (default is until end of table starting fromoffset).

Returns:

Table

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.slice(length=3)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2020,2022,2019]]n_legs: [[2,4,5]]animals: [["Flamingo","Horse","Brittle stars"]]>>>table.slice(offset=2)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2019,2021]]n_legs: [[5,100]]animals: [["Brittle stars","Centipede"]]>>>table.slice(offset=2,length=1)pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2019]]n_legs: [[5]]animals: [["Brittle stars"]]

sort_by(self,sorting,**kwargs)#

Sort the Table or RecordBatch by one or multiple columns.

Parameters:

sortingstr orlist[tuple(name,order)]

Name of the column to use to sort (ascending), ora list of multiple sorting conditions whereeach entry is a tuple with column nameand sorting order (“ascending” or “descending”)

**kwargsdict, optional

Additional sorting options.As allowed bySortOptions

Returns:

Table orRecordBatch

A new tabular object sorted according to the sort keys.

Examples

Table (works similarly for RecordBatch)

>>>importpandasaspd>>>importpyarrowaspa>>>df=pd.DataFrame({'year':[2020,2022,2021,2022,2019,2021],...'n_legs':[2,2,4,4,5,100],...'animal':["Flamingo","Parrot","Dog","Horse",..."Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.sort_by('animal')pyarrow.Tableyear: int64n_legs: int64animal: string----year: [[2019,2021,2021,2020,2022,2022]]n_legs: [[5,100,4,2,4,2]]animal: [["Brittle stars","Centipede","Dog","Flamingo","Horse","Parrot"]]

take(self,indices)#

Select rows from a Table or RecordBatch.

Seepyarrow.compute.take() for full usage.

Parameters:

indicesArray orarray-like

The indices in the tabular object whose rows will be returned.

Returns:

Table orRecordBatch

A tabular object with the same schema, containing the taken rows.

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'year':[2020,2022,2019,2021],...'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)>>>table.take([1,3])pyarrow.Tableyear: int64n_legs: int64animals: string----year: [[2022,2021]]n_legs: [[4,100]]animals: [["Horse","Centipede"]]

to_batches(self,max_chunksize=None)#

Convert Table to a list of RecordBatch objects.

Note that this method is zero-copy, it merely exposes the same dataunder a different API.

Parameters:

max_chunksizeint, defaultNone

Maximum number of rows for each RecordBatch chunk. Individual chunksmay be smaller depending on the chunk layout of individual columns.

Returns:

list[RecordBatch]

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Convert a Table to a RecordBatch:

>>>table.to_batches()[0].to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede

Convert a Table to a list of RecordBatches:

>>>table.to_batches(max_chunksize=2)[0].to_pandas()   n_legs   animals0       2  Flamingo1       4     Horse>>>table.to_batches(max_chunksize=2)[1].to_pandas()   n_legs        animals0       5  Brittle stars1     100      Centipede

to_pandas(self,memory_pool=None,categories=None,boolstrings_to_categorical=False,boolzero_copy_only=False,boolinteger_object_nulls=False,booldate_as_object=True,booltimestamp_as_object=False,booluse_threads=True,booldeduplicate_objects=True,boolignore_metadata=False,boolsafe=True,boolsplit_blocks=False,boolself_destruct=False,unicodemaps_as_pydicts=None,types_mapper=None,boolcoerce_temporal_nanoseconds=False)#

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters:

memory_poolMemoryPool, defaultNone

Arrow MemoryPool to use for allocations. Uses the default memorypool if not passed.

categorieslist, defaultempty

List of fields that should be returned as pandas.Categorical. Onlyapplies to table-like data structures.

strings_to_categoricalbool, defaultFalse

Encode string (UTF8) and binary types to pandas.Categorical.

zero_copy_onlybool, defaultFalse

Raise an ArrowException if this function call would require copyingthe underlying data.

integer_object_nullsbool, defaultFalse

Cast integers with nulls to objects

date_as_objectbool, defaultTrue

Cast dates to objects. If False, convert to datetime64 dtype withthe equivalent time unit (if supported). Note: in pandas version< 2.0, only datetime64[ns] conversion is supported.

timestamp_as_objectbool, defaultFalse

Cast non-nanosecond timestamps (np.datetime64) to objects. This isuseful in pandas version 1.x if you have timestamps that don’t fitin the normal date range of nanosecond timestamps (1678 CE-2262 CE).Non-nanosecond timestamps are supported in pandas version 2.0.If False, all timestamps are converted to datetime64 dtype.

use_threadsbool, defaultTrue

Whether to parallelize the conversion using multiple threads.

deduplicate_objectsbool, defaultTrue

Do not create multiple copies Python objects when created, to saveon memory use. Conversion will be slower.

ignore_metadatabool, defaultFalse

If True, do not use the ‘pandas’ metadata to reconstruct theDataFrame index, if present

safebool, defaultTrue

For certain data types, a cast is needed in order to store thedata in a pandas DataFrame or Series (e.g. timestamps are alwaysstored as nanoseconds in pandas). This option controls whether itis a safe cast or not.

split_blocksbool, defaultFalse

If True, generate one internal “block” for each column whencreating a pandas.DataFrame from a RecordBatch or Table. While thiscan temporarily reduce memory note that various pandas operationscan trigger “consolidation” which may balloon memory use.

self_destructbool, defaultFalse

EXPERIMENTAL: If True, attempt to deallocate the originating Arrowmemory while converting the Arrow object to pandas. If you use theobject after calling to_pandas with this option it will crash yourprogram.

Note that you may not see always memory usage improvements. Forexample, if multiple columns share an underlying allocation,memory can’t be freed until all columns are converted.

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.This can change the ordering of (key, value) pairs, and willdeduplicate multiple keys, resulting in a possible loss of data.

If ‘lossy’, this key deduplication results in a warning printedwhen detected. If ‘strict’, this instead results in an exceptionbeing raised when detected.

types_mapperfunction, defaultNone

A function mapping a pyarrow DataType to a pandas ExtensionDtype.This can be used to override the default pandas type for conversionof built-in pyarrow types or in absence of pandas_metadata in theTable schema. The function receives a pyarrow DataType and isexpected to return a pandas ExtensionDtype orNone if thedefault conversion should be used for that type. If you havea dictionary mapping, you can passdict.get as function.

coerce_temporal_nanosecondsbool, defaultFalse

Only applicable to pandas version >= 2.0.A legacy option to coerce date32, date64, duration, and timestamptime units to nanoseconds when converting to pandas. This is thedefault behavior in pandas version 1.x. Set this option to True ifyou’d like to use this coercion when using pandas version >= 2.0for backwards compatibility (not recommended otherwise).

Returns:

pandas.Series orpandas.DataFrame depending ontype of object

Examples

>>>importpyarrowaspa>>>importpandasaspd

Convert a Table to pandas DataFrame:

>>>table=pa.table([...pa.array([2,4,5,100]),...pa.array(["Flamingo","Horse","Brittle stars","Centipede"])...],names=['n_legs','animals'])>>>table.to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede>>>isinstance(table.to_pandas(),pd.DataFrame)True

Convert a RecordBatch to pandas DataFrame:

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>batch=pa.record_batch([n_legs,animals],...names=["n_legs","animals"])>>>batchpyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]>>>batch.to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede>>>isinstance(batch.to_pandas(),pd.DataFrame)True

Convert a Chunked Array to pandas Series:

>>>importpyarrowaspa>>>n_legs=pa.chunked_array([[2,2,4],[4,5,100]])>>>n_legs.to_pandas()0      21      22      43      44      55    100dtype: int64>>>isinstance(n_legs.to_pandas(),pd.Series)True

to_pydict(self,*,maps_as_pydicts=None)#

Convert the Table or RecordBatch to a dict or OrderedDict.

Parameters:

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.

If ‘lossy’, whenever duplicate keys are detected, a warning will be printed.The last seen value of a duplicate key will be in the Python dictionary.If ‘strict’, this instead results in an exception being raised when detected.

Returns:

dict

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>n_legs=pa.array([2,2,4,4,5,100])>>>animals=pa.array(["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"])>>>table=pa.Table.from_arrays([n_legs,animals],names=["n_legs","animals"])>>>table.to_pydict(){'n_legs': [2, 2, 4, 4, 5, 100], 'animals': ['Flamingo', 'Parrot', ..., 'Centipede']}

to_pylist(self,*,maps_as_pydicts=None)#

Convert the Table or RecordBatch to a list of rows / dictionaries.

Parameters:

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.

If ‘lossy’, whenever duplicate keys are detected, a warning will be printed.The last seen value of a duplicate key will be in the Python dictionary.If ‘strict’, this instead results in an exception being raised when detected.

Returns:

list

Examples

Table (works similarly for RecordBatch)

>>>importpyarrowaspa>>>data=[[2,4,5,100],...["Flamingo","Horse","Brittle stars","Centipede"]]>>>table=pa.table(data,names=["n_legs","animals"])>>>table.to_pylist()[{'n_legs': 2, 'animals': 'Flamingo'}, {'n_legs': 4, 'animals': 'Horse'}, ...

to_reader(self,max_chunksize=None)#

Convert the Table to a RecordBatchReader.

Note that this method is zero-copy, it merely exposes the same dataunder a different API.

Parameters:

max_chunksizeint, defaultNone

Maximum number of rows for each RecordBatch chunk. Individual chunksmay be smaller depending on the chunk layout of individual columns.

Returns:

RecordBatchReader

Examples

>>>importpyarrowaspa>>>importpandasaspd>>>df=pd.DataFrame({'n_legs':[2,4,5,100],...'animals':["Flamingo","Horse","Brittle stars","Centipede"]})>>>table=pa.Table.from_pandas(df)

Convert a Table to a RecordBatchReader:

>>>table.to_reader()<pyarrow.lib.RecordBatchReader object at ...>

>>>reader=table.to_reader()>>>reader.scheman_legs: int64animals: string-- schema metadata --pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, ...>>>reader.read_all()pyarrow.Tablen_legs: int64animals: string----n_legs: [[2,4,5,100]]animals: [["Flamingo","Horse","Brittle stars","Centipede"]]

to_string(self,*,show_metadata=False,preview_cols=0)#

Return human-readable string representation of Table or RecordBatch.

Parameters:

show_metadatabool, defaultFalse

Display Field-level and Schema-level KeyValueMetadata.

preview_colsint, default 0

Display values of the columns for the first N columns.

Returns:

str

to_struct_array(self,max_chunksize=None)#

Convert to a chunked array of struct type.

Parameters:

max_chunksizeint, defaultNone

Maximum number of rows for ChunkedArray chunks. Individual chunksmay be smaller depending on the chunk layout of individual columns.

Returns:

ChunkedArray

unify_dictionaries(self,MemoryPoolmemory_pool=None)#

Unify dictionaries across all chunks.

This method returns an equivalent table, but where all chunks ofeach column share the same dictionary values. Dictionary indicesare transposed accordingly.

Columns without dictionaries are returned unchanged.

Parameters:

memory_poolMemoryPool, defaultNone

For memory allocations, if required, otherwise use default pool

Returns:

Table

Examples

>>>importpyarrowaspa>>>arr_1=pa.array(["Flamingo","Parrot","Dog"]).dictionary_encode()>>>arr_2=pa.array(["Horse","Brittle stars","Centipede"]).dictionary_encode()>>>c_arr=pa.chunked_array([arr_1,arr_2])>>>table=pa.table([c_arr],names=["animals"])>>>tablepyarrow.Tableanimals: dictionary<values=string, indices=int32, ordered=0>----animals: [  -- dictionary:["Flamingo","Parrot","Dog"]  -- indices:[0,1,2],  -- dictionary:["Horse","Brittle stars","Centipede"]  -- indices:[0,1,2]]

Unify dictionaries across both chunks:

>>>table.unify_dictionaries()pyarrow.Tableanimals: dictionary<values=string, indices=int32, ordered=0>----animals: [  -- dictionary:["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]  -- indices:[0,1,2],  -- dictionary:["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]  -- indices:[3,4,5]]

validate(self,*,full=False)#

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Passfull=Truefor thorough validation checks (potentially O(n)).

Parameters:

fullbool, defaultFalse

If True, run expensive checks, otherwise cheap checks only.

Raises:

ArrowInvalid