pyarrow.StructArray#

classpyarrow.StructArray#

Bases:Array

Concrete class for Arrow arrays of a struct data type.

__init__(*args,**kwargs)#

Methods

__init__(*args, **kwargs)
buffers(self)	Return a list of Buffer objects pointing to this array's physical storage.
cast(self[, target_type, safe, options, ...])	Cast array values to another data type
copy_to(self, destination)	Construct a copy of the array with all buffers on destination device.
dictionary_encode(self[, null_encoding])	Compute dictionary-encoded representation of array.
diff(self, Array other)	Compare contents of this array against another one.
drop_null(self)	Remove missing values from an array.
equals(self, Array other)	Parameters:
field(self, index)	Retrieves the child array belonging to field.
fill_null(self, fill_value)	Seepyarrow.compute.fill_null() for usage.
filter(self, mask, *[, null_selection_behavior])	Select values from an array.
flatten(self, MemoryPool memory_pool=None)	Return one individual array for each field in the struct.
format(self, **kwargs)	DEPRECATED, use pyarrow.Array.to_string
from_arrays(arrays[, names, fields, mask, ...])	Construct StructArray from collection of arrays representing each field in the struct.
from_buffers(DataType type, length, buffers)	Construct an Array from a sequence of buffers.
from_pandas(obj[, mask, type])	Convert pandas.Series to an Arrow Array.
get_total_buffer_size(self)	The sum of bytes in each buffer referenced by the array.
index(self, value[, start, end, memory_pool])	Find the first index of a value.
is_nan(self)	Return BooleanArray indicating the NaN values.
is_null(self, *[, nan_is_null])	Return BooleanArray indicating the null values.
is_valid(self)	Return BooleanArray indicating the non-null values.
slice(self[, offset, length])	Compute zero-copy slice of this array.
sort(self[, order, by])	Sort the StructArray
sum(self, **kwargs)	Sum the values in a numerical array.
take(self, indices)	Select values from an array.
to_numpy(self[, zero_copy_only, writable])	Return a NumPy view or copy of this array.
to_pandas(self[, memory_pool, categories, ...])	Convert to a pandas-compatible NumPy array or DataFrame, as appropriate
to_pylist(self, *[, maps_as_pydicts])	Convert to a list of native Python objects.
to_string(self, *, int indent=2, ...)	Render a "pretty-printed" string representation of the Array.
tolist(self)	Alias of to_pylist for compatibility with NumPy.
unique(self)	Compute distinct elements in array.
validate(self, *[, full])	Perform validation checks.
value_counts(self)	Compute counts of unique elements in array.
view(self, target_type)	Return zero-copy "view" of array as another data type.

Attributes

device_type	The device type where the array resides.
is_cpu	Whether the array is CPU-accessible.
nbytes	Total number of bytes consumed by the elements of the array.
null_count
offset	A relative position into another array's data.
statistics	Statistics of the array.
type

buffers(self)#

Return a list of Buffer objects pointing to this array’s physicalstorage.

To correctly interpret these buffers, you need to also apply the offsetmultiplied with the size of the stored data type.

cast(self,target_type=None,safe=None,options=None,memory_pool=None)#

Cast array values to another data type

Seepyarrow.compute.cast() for usage.

Parameters:

target_typeDataType, defaultNone

Type to cast array to.

safebool, defaultTrue

Whether to check for conversion errors such as overflow.

optionsCastOptions, defaultNone

Additional checks pass by CastOptions

memory_poolMemoryPool, optional

memory pool to use for allocations during function execution.

Returns:

castArray

copy_to(self,destination)#

Construct a copy of the array with all buffers on destinationdevice.

This method recursively copies the array’s buffers and those of itschildren onto the destination MemoryManager device and returns thenew Array.

Parameters:

destinationpyarrow.MemoryManager orpyarrow.Device

The destination device to copy the array to.

Returns:

Array

device_type#

The device type where the array resides.

Returns:

DeviceAllocationType

dictionary_encode(self,null_encoding='mask')#

Compute dictionary-encoded representation of array.

Seepyarrow.compute.dictionary_encode() for full usage.

Parameters:

null_encodingstr, default “mask”

How to handle null entries.

Returns:

encodedDictionaryArray

A dictionary-encoded version of this array.

diff(self,Arrayother)#

Compare contents of this array against another one.

Return a string containing the result of diffing this array(on the left side) against the other array (on the right side).

Parameters:

otherArray

The other array to compare this array with.

Returns:

diffstr

A human-readable printout of the differences.

Examples

>>>importpyarrowaspa>>>left=pa.array(["one","two","three"])>>>right=pa.array(["two",None,"two-and-a-half","three"])>>>print(left.diff(right))

@@ -0, +0 @@-“one”@@ -2, +1 @@+null+”two-and-a-half”

drop_null(self)#

Remove missing values from an array.

equals(self,Arrayother)#

Parameters:

otherpyarrow.Array

Returns:

bool

field(self,index)#

Retrieves the child array belonging to field.

Parameters:

indexUnion[int,str]

Index / position or name of the field.

Returns:

resultArray

fill_null(self,fill_value)#

Seepyarrow.compute.fill_null() for usage.

Parameters:

fill_valueany

The replacement value for null entries.

Returns:

resultArray

A new array with nulls replaced by the given value.

filter(self,mask,*,null_selection_behavior='drop')#

Select values from an array.

Seepyarrow.compute.filter() for full usage.

Parameters:

maskArray orarray-like

The boolean mask to filter the array with.

null_selection_behaviorstr, default “drop”

How nulls in the mask should be handled.

Returns:

filteredArray

An array of the same type, with only the elements selected bythe boolean mask.

flatten(self,MemoryPoolmemory_pool=None)#

Return one individual array for each field in the struct.

Parameters:

memory_poolMemoryPool, defaultNone

For memory allocations, if required, otherwise use default pool.

Returns:

resultList[Array]

format(self,**kwargs)#

DEPRECATED, use pyarrow.Array.to_string

Parameters:

**kwargsdict

Returns:

str

staticfrom_arrays(arrays,names=None,fields=None,mask=None,memory_pool=None,type=None)#

Construct StructArray from collection of arrays representingeach field in the struct.

Either field names, field instances or a struct type must be passed.

Parameters:

arrayssequence ofArray

namesList[str] (optional)

Field names for each struct child.

fieldsList[Field] (optional)

Field instances for each struct child.

maskpyarrow.Array[bool] (optional)

Indicate which values are null (True) or not null (False).

memory_poolMemoryPool (optional)

For memory allocations, if required, otherwise uses default pool.

typepyarrow.StructType (optional)

Struct type for name and type of each child.

Returns:

resultStructArray

staticfrom_buffers(DataTypetype,length,buffers,null_count=-1,offset=0,children=None)#

Construct an Array from a sequence of buffers.

The concrete type returned depends on the datatype.

Parameters:

typeDataType

The value type of the array.

lengthint

The number of values in the array.

buffersList[Buffer]

The buffers backing this array.

null_countint, default -1

The number of null entries in the array. Negative value means thatthe null count is not known.

offsetint, default 0

The array’s logical offset (in values, not in bytes) from thestart of each buffer.

childrenList[Array], defaultNone

Nested type children with length matching type.num_fields.

Returns:

arrayArray

staticfrom_pandas(obj,mask=None,type=None,boolsafe=True,MemoryPoolmemory_pool=None)#

Convert pandas.Series to an Arrow Array.

This method uses Pandas semantics about what values indicatenulls. See pyarrow.array for more general conversion from arrays orsequences to Arrow arrays.

Parameters:

objndarray,pandas.Series,array-like

maskarray (bool), optional

Indicate which values are null (True) or not null (False).

typepyarrow.DataType

Explicit type to attempt to coerce to, otherwise will be inferredfrom the data.

safebool, defaultTrue

Check for overflows or other unsafe conversions.

memory_poolpyarrow.MemoryPool, optional

If not passed, will allocate memory from the currently-set defaultmemory pool.

Returns:

arraypyarrow.Array orpyarrow.ChunkedArray

ChunkedArray is returned if object data overflows binary buffer.

Notes

Localized timestamps will currently be returned as UTC (pandas’s nativerepresentation). Timezone-naive data will be implicitly interpreted asUTC.

get_total_buffer_size(self)#

The sum of bytes in each buffer referenced by the array.

An array may only reference a portion of a buffer.This method will overestimate in this case and return thebyte size of the entire buffer.

If a buffer is referenced multiple times then it willonly be counted once.

index(self,value,start=None,end=None,*,memory_pool=None)#

Find the first index of a value.

Seepyarrow.compute.index() for full usage.

Parameters:

valueScalar or object

The value to look for in the array.

startint, optional

The start index where to look forvalue.

endint, optional

The end index where to look forvalue.

memory_poolMemoryPool, optional

A memory pool for potential memory allocations.

Returns:

indexInt64Scalar

The index of the value in the array (-1 if not found).

is_cpu#

Whether the array is CPU-accessible.

is_nan(self)#

Return BooleanArray indicating the NaN values.

Returns:

arrayboolArray

is_null(self,*,nan_is_null=False)#

Return BooleanArray indicating the null values.

Parameters:

nan_is_nullbool (optional, defaultFalse)

Whether floating-point NaN values should also be considered null.

Returns:

arrayboolArray

is_valid(self)#

Return BooleanArray indicating the non-null values.

nbytes#

Total number of bytes consumed by the elements of the array.

In other words, the sum of bytes from all bufferranges referenced.

Unlikeget_total_buffer_size this method will account for arrayoffsets.

If buffers are shared between arrays then the sharedportion will be counted multiple times.

The dictionary of dictionary arrays will always be counted in theirentirety even if the array only references a portion of the dictionary.

null_count#

offset#

A relative position into another array’s data.

The purpose is to enable zero-copy slicing. This value defaults to zerobut must be applied on all operations with the physical storagebuffers.

slice(self,offset=0,length=None)#

Compute zero-copy slice of this array.

Parameters:

offsetint, default 0

Offset from start of array to slice.

lengthint, defaultNone

Length of slice (default is until end of Array starting fromoffset).

Returns:

slicedArray

An array with the same datatype, containing the sliced values.

sort(self,order='ascending',by=None,**kwargs)#

Sort the StructArray

Parameters:

orderstr, default “ascending”

Which order to sort values in.Accepted values are “ascending”, “descending”.

bystr orNone, defaultNone

If to sort the array by one of its fieldsor by the whole array.

**kwargsdict, optional

Additional sorting options.As allowed bySortOptions

Returns:

resultStructArray

statistics#

Statistics of the array.

sum(self,**kwargs)#

Sum the values in a numerical array.

Seepyarrow.compute.sum() for full usage.

Parameters:

**kwargsdict, optional

Options to pass topyarrow.compute.sum().

Returns:

sumScalar

A scalar containing the sum value.

take(self,indices)#

Select values from an array.

Seepyarrow.compute.take() for full usage.

Parameters:

indicesArray orarray-like

The indices in the array whose values will be returned.

Returns:

takenArray

An array with the same datatype, containing the taken values.

to_numpy(self,zero_copy_only=True,writable=False)#

Return a NumPy view or copy of this array.

By default, tries to return a view of this array. This is onlysupported for primitive arrays with the same memory layout as NumPy(i.e. integers, floating point, ..) and without any nulls.

For the extension arrays, this method simply delegates to theunderlying storage array.

Parameters:

zero_copy_onlybool, defaultTrue

If True, an exception will be raised if the conversion to a numpyarray would require copying the underlying data (e.g. in presenceof nulls, or for non-primitive types).

writablebool, defaultFalse

For numpy arrays created with zero copy (view on the Arrow data),the resulting array is not writable (Arrow data is immutable).By setting this to True, a copy of the array is made to ensureit is writable.

Returns:

arraynumpy.ndarray

to_pandas(self,memory_pool=None,categories=None,boolstrings_to_categorical=False,boolzero_copy_only=False,boolinteger_object_nulls=False,booldate_as_object=True,booltimestamp_as_object=False,booluse_threads=True,booldeduplicate_objects=True,boolignore_metadata=False,boolsafe=True,boolsplit_blocks=False,boolself_destruct=False,unicodemaps_as_pydicts=None,types_mapper=None,boolcoerce_temporal_nanoseconds=False)#

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters:

memory_poolMemoryPool, defaultNone

Arrow MemoryPool to use for allocations. Uses the default memorypool if not passed.

categorieslist, defaultempty

List of fields that should be returned as pandas.Categorical. Onlyapplies to table-like data structures.

strings_to_categoricalbool, defaultFalse

Encode string (UTF8) and binary types to pandas.Categorical.

zero_copy_onlybool, defaultFalse

Raise an ArrowException if this function call would require copyingthe underlying data.

integer_object_nullsbool, defaultFalse

Cast integers with nulls to objects

date_as_objectbool, defaultTrue

Cast dates to objects. If False, convert to datetime64 dtype withthe equivalent time unit (if supported). Note: in pandas version< 2.0, only datetime64[ns] conversion is supported.

timestamp_as_objectbool, defaultFalse

Cast non-nanosecond timestamps (np.datetime64) to objects. This isuseful in pandas version 1.x if you have timestamps that don’t fitin the normal date range of nanosecond timestamps (1678 CE-2262 CE).Non-nanosecond timestamps are supported in pandas version 2.0.If False, all timestamps are converted to datetime64 dtype.

use_threadsbool, defaultTrue

Whether to parallelize the conversion using multiple threads.

deduplicate_objectsbool, defaultTrue

Do not create multiple copies Python objects when created, to saveon memory use. Conversion will be slower.

ignore_metadatabool, defaultFalse

If True, do not use the ‘pandas’ metadata to reconstruct theDataFrame index, if present

safebool, defaultTrue

For certain data types, a cast is needed in order to store thedata in a pandas DataFrame or Series (e.g. timestamps are alwaysstored as nanoseconds in pandas). This option controls whether itis a safe cast or not.

split_blocksbool, defaultFalse

If True, generate one internal “block” for each column whencreating a pandas.DataFrame from a RecordBatch or Table. While thiscan temporarily reduce memory note that various pandas operationscan trigger “consolidation” which may balloon memory use.

self_destructbool, defaultFalse

EXPERIMENTAL: If True, attempt to deallocate the originating Arrowmemory while converting the Arrow object to pandas. If you use theobject after calling to_pandas with this option it will crash yourprogram.

Note that you may not see always memory usage improvements. Forexample, if multiple columns share an underlying allocation,memory can’t be freed until all columns are converted.

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.This can change the ordering of (key, value) pairs, and willdeduplicate multiple keys, resulting in a possible loss of data.

If ‘lossy’, this key deduplication results in a warning printedwhen detected. If ‘strict’, this instead results in an exceptionbeing raised when detected.

types_mapperfunction, defaultNone

A function mapping a pyarrow DataType to a pandas ExtensionDtype.This can be used to override the default pandas type for conversionof built-in pyarrow types or in absence of pandas_metadata in theTable schema. The function receives a pyarrow DataType and isexpected to return a pandas ExtensionDtype orNone if thedefault conversion should be used for that type. If you havea dictionary mapping, you can passdict.get as function.

coerce_temporal_nanosecondsbool, defaultFalse

Only applicable to pandas version >= 2.0.A legacy option to coerce date32, date64, duration, and timestamptime units to nanoseconds when converting to pandas. This is thedefault behavior in pandas version 1.x. Set this option to True ifyou’d like to use this coercion when using pandas version >= 2.0for backwards compatibility (not recommended otherwise).

Returns:

pandas.Series orpandas.DataFrame depending ontype of object

Examples

>>>importpyarrowaspa>>>importpandasaspd

Convert a Table to pandas DataFrame:

>>>table=pa.table([...pa.array([2,4,5,100]),...pa.array(["Flamingo","Horse","Brittle stars","Centipede"])...],names=['n_legs','animals'])>>>table.to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede>>>isinstance(table.to_pandas(),pd.DataFrame)True

Convert a RecordBatch to pandas DataFrame:

>>>importpyarrowaspa>>>n_legs=pa.array([2,4,5,100])>>>animals=pa.array(["Flamingo","Horse","Brittle stars","Centipede"])>>>batch=pa.record_batch([n_legs,animals],...names=["n_legs","animals"])>>>batchpyarrow.RecordBatchn_legs: int64animals: string----n_legs: [2,4,5,100]animals: ["Flamingo","Horse","Brittle stars","Centipede"]>>>batch.to_pandas()   n_legs        animals0       2       Flamingo1       4          Horse2       5  Brittle stars3     100      Centipede>>>isinstance(batch.to_pandas(),pd.DataFrame)True

Convert a Chunked Array to pandas Series:

>>>importpyarrowaspa>>>n_legs=pa.chunked_array([[2,2,4],[4,5,100]])>>>n_legs.to_pandas()0      21      22      43      44      55    100dtype: int64>>>isinstance(n_legs.to_pandas(),pd.Series)True

to_pylist(self,*,maps_as_pydicts=None)#

Convert to a list of native Python objects.

Parameters:

maps_as_pydictsstr, optional, defaultNone

Valid values areNone, ‘lossy’, or ‘strict’.The default behavior (None), is to convert Arrow Map arrays toPython association lists (list-of-tuples) in the same order as theArrow Map, as in [(key1, value1), (key2, value2), …].

If ‘lossy’ or ‘strict’, convert Arrow Map arrays to native Python dicts.

If ‘lossy’, whenever duplicate keys are detected, a warning will be printed.The last seen value of a duplicate key will be in the Python dictionary.If ‘strict’, this instead results in an exception being raised when detected.

Returns:

lstlist

to_string(self,*,intindent=2,inttop_level_indent=0,intwindow=10,intcontainer_window=2,boolskip_new_lines=False)#

Render a “pretty-printed” string representation of the Array.

Note: for data on a non-CPU device, the full array is copied to CPUmemory.

Parameters:

indentint, default 2

How much to indent the internal items in the string tothe right, by default2.

top_level_indentint, default 0

How much to indent right the entire content of the array,by default0.

windowint

How many primitive items to preview at the begin and endof the array when the array is bigger than the window.The other items will be ellipsed.

container_windowint

How many container items (such as a list in a list array)to preview at the begin and end of the array when the arrayis bigger than the window.

skip_new_linesbool

If the array should be rendered as a single line of textor if each element should be on its own line.

tolist(self)#

Alias of to_pylist for compatibility with NumPy.

type#

unique(self)#

Compute distinct elements in array.

Returns:

uniqueArray

An array of the same data type, with deduplicated elements.

validate(self,*,full=False)#

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Passfull=Truefor thorough validation checks (potentially O(n)).

Parameters:

fullbool, defaultFalse

If True, run expensive checks, otherwise cheap checks only.

Raises:

ArrowInvalid

value_counts(self)#

Compute counts of unique elements in array.

Returns:

StructArray

An array of <input type “Values”, int64 “Counts”> structs

view(self,target_type)#

Return zero-copy “view” of array as another data type.

The data types must have compatible columnar buffer layouts

Parameters:

target_typeDataType

Type to construct view as.

Returns:

viewArray