ExtensionDtype#

Apandas.api.extensions.ExtensionDtype is similar to anumpy.dtype object. It describes thedata type. Implementers are responsible for a few unique items like the name.

One particularly important item is thetype property. This should be theclass that is the scalar type for your data. For example, if you were writing anextension array for IP Address data, this might beipaddress.IPv4Address.

See theextension dtype source for interface definition.

pandas.api.extensions.ExtensionDtype can be registered to pandas to allow creation via a string dtype name.This allows one to instantiateSeries and.astype() with a registered string name, forexample'category' is a registered string accessor for theCategoricalDtype.

See theextension dtype dtypes for more on how to register dtypes.

ExtensionArray#

This class provides all the array-like functionality. ExtensionArrays arelimited to 1 dimension. An ExtensionArray is linked to an ExtensionDtype via thedtype attribute.

pandas makes no restrictions on how an extension array is created via its__new__ or__init__, and puts no restrictions on how you store yourdata. We do require that your array be convertible to a NumPy array, even ifthis is relatively expensive (as it is forCategorical).

They may be backed by none, one, or many NumPy arrays. For example,pandas.Categorical is an extension array backed by two arrays,one for codes and one for categories. An array of IPv6 addresses maybe backed by a NumPy structured array with two fields, one for thelower 64 bits and one for the upper 64 bits. Or they may be backedby some other storage type, like Python lists.

See theextension array source for the interface definition. The docstringsand comments contain guidance for properly implementing the interface.

ExtensionArray operator support#

By default, there are no operators defined for the classExtensionArray.There are two approaches for providing operator support for your ExtensionArray:

1. Define each of the operators on yourExtensionArray subclass.

2. Use an operator implementation from pandas that depends on operators that are already definedon the underlying elements (scalars) of the ExtensionArray.

For the first approach, you define selected operators, e.g.,__add__,__le__, etc. thatyou want yourExtensionArray subclass to support.

The second approach assumes that the underlying elements (i.e., scalar type) of theExtensionArrayhave the individual operators already defined. In other words, if yourExtensionArraynamedMyExtensionArray is implemented so that each element is an instanceof the classMyExtensionElement, then if the operators are definedforMyExtensionElement, the second approach will automaticallydefine the operators forMyExtensionArray.

A mixin class,ExtensionScalarOpsMixin supports this secondapproach. If developing anExtensionArray subclass, for exampleMyExtensionArray,can simply includeExtensionScalarOpsMixin as a parent class ofMyExtensionArray,and then call the methods_add_arithmetic_ops() and/or_add_comparison_ops() to hook the operators intoyourMyExtensionArray class, as follows:

frompandas.api.extensionsimportExtensionArray,ExtensionScalarOpsMixinclassMyExtensionArray(ExtensionArray,ExtensionScalarOpsMixin):passMyExtensionArray._add_arithmetic_ops()MyExtensionArray._add_comparison_ops()

For arithmetic operations, this implementation will try to reconstruct a newExtensionArray with the result of the element-wise operation. Whetheror not that succeeds depends on whether the operation returns a resultthat’s valid for theExtensionArray. If anExtensionArray cannotbe reconstructed, an ndarray containing the scalars returned instead.

For ease of implementation and consistency with operations between pandasand NumPy ndarrays, we recommendnot handling Series and Indexes in your binary ops.Instead, you should detect these cases and returnNotImplemented.When pandas encounters an operation likeop(Series,ExtensionArray), pandaswill

1. unbox the array from theSeries (Series.array)

2. callresult=op(values,ExtensionArray)

3. re-box the result in aSeries

NumPy universal functions#

Series implements__array_ufunc__. As part of the implementation,pandas unboxes theExtensionArray from theSeries, applies the ufunc,and re-boxes it if necessary.

If applicable, we highly recommend that you implement__array_ufunc__ in yourextension array to avoid coercion to an ndarray. Seethe NumPy documentationfor an example.

As part of your implementation, we require that you defer to pandas when a pandascontainer (Series,DataFrame,Index) is detected ininputs.If any of those is present, you should returnNotImplemented. pandas will take care ofunboxing the array from the container and re-calling the ufunc with the unwrapped input.

Testing extension arrays#

We provide a test suite for ensuring that your extension arrays satisfy the expectedbehavior. To use the test suite, you must provide several pytest fixtures and inheritfrom the base test class. The required fixtures are found inpandas-dev/pandas.

To use a test, subclass it:

frompandas.tests.extensionimportbaseclassTestConstructors(base.BaseConstructorsTests):pass

Seepandas-dev/pandasfor a list of all the tests available.

Compatibility with Apache Arrow#

AnExtensionArray can support conversion to / frompyarrow arrays(and thus support for example serialization to the Parquet file format)by implementing two methods:ExtensionArray.__arrow_array__ andExtensionDtype.__from_arrow__.

TheExtensionArray.__arrow_array__ ensures thatpyarrow knowns howto convert the specific extension array into apyarrow.Array (also whenincluded as a column in a pandas DataFrame):

classMyExtensionArray(ExtensionArray):...def__arrow_array__(self,type=None):# convert the underlying array values to a pyarrow Arrayimportpyarrowreturnpyarrow.array(...,type=type)

TheExtensionDtype.__from_arrow__ method then controls the conversionback from pyarrow to a pandas ExtensionArray. This method receives a pyarrowArray orChunkedArray as only argument and is expected to return theappropriate pandasExtensionArray for this dtype and the passed values:

class ExtensionDtype:    ...    def __from_arrow__(self, array: pyarrow.Array/ChunkedArray) -> ExtensionArray:        ...

See more in theArrow documentation.

Those methods have been implemented for the nullable integer and string extensiondtypes included in pandas, and ensure roundtrip to pyarrow and the Parquet file format.

Override constructor properties#

Each data structure has severalconstructor properties for returning a newdata structure as the result of an operation. By overriding these properties,you can retain subclasses throughpandas data manipulations.

There are 3 possible constructor properties to be defined on a subclass:

• DataFrame/Series._constructor: Used when a manipulation result has the same dimension as the original.

• DataFrame._constructor_sliced: Used when aDataFrame (sub-)class manipulation result should be aSeries (sub-)class.

• Series._constructor_expanddim: Used when aSeries (sub-)class manipulation result should be aDataFrame (sub-)class, e.g.Series.to_frame().

Below example shows how to defineSubclassedSeries andSubclassedDataFrame overriding constructor properties.

classSubclassedSeries(pd.Series):@propertydef_constructor(self):returnSubclassedSeries@propertydef_constructor_expanddim(self):returnSubclassedDataFrameclassSubclassedDataFrame(pd.DataFrame):@propertydef_constructor(self):returnSubclassedDataFrame@propertydef_constructor_sliced(self):returnSubclassedSeries

>>>s=SubclassedSeries([1,2,3])>>>type(s)<class '__main__.SubclassedSeries'>>>>to_framed=s.to_frame()>>>type(to_framed)<class '__main__.SubclassedDataFrame'>>>>df=SubclassedDataFrame({"A":[1,2,3],"B":[4,5,6],"C":[7,8,9]})>>>df   A  B  C0  1  4  71  2  5  82  3  6  9>>>type(df)<class '__main__.SubclassedDataFrame'>>>>sliced1=df[["A","B"]]>>>sliced1   A  B0  1  41  2  52  3  6>>>type(sliced1)<class '__main__.SubclassedDataFrame'>>>>sliced2=df["A"]>>>sliced20    11    22    3Name: A, dtype: int64>>>type(sliced2)<class '__main__.SubclassedSeries'>

Define original properties#

To let original data structures have additional properties, you should letpandas know what properties are added.pandas maps unknown properties to data names overriding__getattribute__. Defining original properties can be done in one of 2 ways:

1. Define_internal_names and_internal_names_set for temporary properties which WILL NOT be passed to manipulation results.

2. Define_metadata for normal properties which will be passed to manipulation results.

Below is an example to define two original properties, “internal_cache” as a temporary property and “added_property” as a normal property

classSubclassedDataFrame2(pd.DataFrame):# temporary properties_internal_names=pd.DataFrame._internal_names+["internal_cache"]_internal_names_set=set(_internal_names)# normal properties_metadata=["added_property"]@propertydef_constructor(self):returnSubclassedDataFrame2

>>>df=SubclassedDataFrame2({"A":[1,2,3],"B":[4,5,6],"C":[7,8,9]})>>>df   A  B  C0  1  4  71  2  5  82  3  6  9>>>df.internal_cache="cached">>>df.added_property="property">>>df.internal_cachecached>>>df.added_propertyproperty# properties defined in _internal_names is reset after manipulation>>>df[["A","B"]].internal_cacheAttributeError: 'SubclassedDataFrame2' object has no attribute 'internal_cache'# properties defined in _metadata are retained>>>df[["A","B"]].added_propertyproperty