Goals#

• Expose an ABI-stable interface.

• Make it easy for third-party projects to implement support for (including partialsupport where sufficient), with little initial investment.

• Allow zero-copy sharing of Arrow data between independent runtimesand components running in the same process.

• Match the Arrow array concepts closely to avoid the development ofyet another marshalling layer.

• Avoid the need for one-to-one adaptation layers such as the limitedJPype-based bridge between Java and Python.

• Enable integration without an explicit dependency (either at compile-timeor runtime) on the Arrow software project.

Ideally, the Arrow C data interface can become a low-levellingua francafor sharing columnar data at runtime and establish Arrow as the universalbuilding block in the columnar processing ecosystem.

Non-goals#

• Expose a C API mimicking operations available in higher-level runtimes(such as C++, Java…).

• Data sharing between distinct processes or storage persistence.

Comparison with the Arrow IPC format#

Pros of the C data interface vs. the IPC format:

• No dependency on Flatbuffers.

• No buffer reassembly (data is already exposed in logical Arrow format).

• Zero-copy by design.

• Easy to reimplement from scratch.

• Minimal C definition that can be easily copied into other codebases.

• Resource lifetime management through a custom release callback.

Pros of the IPC format vs. the data interface:

• Works across processes and machines.

• Allows data storage and persistence.

• Being a streamable format, the IPC format has room for composing more features(such as integrity checks, compression…).

• Does not require explicit C data access.

Examples#

• A dictionary-encodeddecimal128(precision=12,scale=5) arraywithint16 indices has format strings, and its dependent dictionaryarray has format stringd:12,5.

• Alist<uint64> array has format string+l, and its single childhas format stringL.

• Alarge_list_view<uint64> array has format string+vL, and its singlechild has format stringL.

• Astruct<ints:int32,floats:float32> has format string+s; its twochildren have namesints andfloats, and format stringsi andf respectively.

• Amap<string,float64> array has format string+m; its single childhas nameentries and format string+s; its two grandchildren have nameskey andvalue, and format stringsu andg respectively.

• Asparse_union<ints:int32,floats:float32> with type ids4,5has format string+us:4,5; its two children have namesints andfloats, and format stringsi andf respectively.

• Arun_end_encoded<int32,float32> has format string+r; its twochildren have namesrun_ends andvalues, and format stringsi andf respectively.

The ArrowSchema structure#

TheArrowSchema structure describes the type and metadata of an exportedarray or record batch. It has the following fields:

constchar*ArrowSchema.format#

Mandatory. A null-terminated, UTF8-encoded string describingthe data type. If the data type is nested, child types are notencoded here but in theArrowSchema.children structures.

Consumers MAY decide not to support all data types, but theyshould document this limitation.

constchar*ArrowSchema.name#

Optional. A null-terminated, UTF8-encoded string of the fieldor array name. This is mainly used to reconstruct child fieldsof nested types.

Producers MAY decide not to provide this information, and consumersMAY decide to ignore it. If omitted, MAY be NULL or an empty string.

constchar*ArrowSchema.metadata#

Optional. A binary string describing the type’s metadata.If the data type is nested, child types are not encoded here butin theArrowSchema.children structures.

This string is not null-terminated but follows a specific format:

int32:numberofkey/valuepairs(notedNbelow)int32:bytelengthofkey0key0(notnull-terminated)int32:bytelengthofvalue0value0(notnull-terminated)...int32:bytelengthofkeyN-1keyN-1(notnull-terminated)int32:bytelengthofvalueN-1valueN-1(notnull-terminated)

Integers are stored in native endianness. For example, the metadata[('key1','value1')] is encoded on a little-endian machine as:

\x01\x00\x00\x00\x04\x00\x00\x00key1\x06\x00\x00\x00value1

On a big-endian machine, the same example would be encoded as:

\x00\x00\x00\x01\x00\x00\x00\x04key1\x00\x00\x00\x06value1

If omitted, this field MUST be NULL (not an empty string).

Consumers MAY choose to ignore this information.

int64_tArrowSchema.flags#

Optional. A bitfield of flags enriching the type description.Its value is computed by OR’ing together the flag values.The following flags are available:

• ARROW_FLAG_NULLABLE: whether this field is semantically nullable(regardless of whether it actually has null values).

• ARROW_FLAG_DICTIONARY_ORDERED: for dictionary-encoded types,whether the ordering of dictionary indices is semantically meaningful.

• ARROW_FLAG_MAP_KEYS_SORTED: for map types, whether the keys withineach map value are sorted.

If omitted, MUST be 0.

Consumers MAY choose to ignore some or all of the flags. Even then,they SHOULD keep this value around so as to propagate its informationto their own consumers.

int64_tArrowSchema.n_children#

Mandatory. The number of children this type has.

ArrowSchema**ArrowSchema.children#

Optional. A C array of pointers to each child type of this type.There must beArrowSchema.n_children pointers.

MAY be NULL only ifArrowSchema.n_children is 0.

ArrowSchema*ArrowSchema.dictionary#

Optional. A pointer to the type of dictionary values.

MUST be present if the ArrowSchema represents a dictionary-encoded type.MUST be NULL otherwise.

void(*ArrowSchema.release)(structArrowSchema*)#

Mandatory. A pointer to a producer-provided release callback.

See below for memory management and release callback semantics.

void*ArrowSchema.private_data#

Optional. An opaque pointer to producer-provided private data.

Consumers MUST not process this member. Lifetime of this memberis handled by the producer, and especially by the release callback.

The ArrowArray structure#

TheArrowArray describes the data of an exported array or record batch.For theArrowArray structure to be interpreted type, the array typeor record batch schema must already be known. This is either done byconvention – for example a producer API that always produces the same datatype – or by passing aArrowSchema on the side.

It has the following fields:

int64_tArrowArray.length#

Mandatory. The logical length of the array (i.e. its number of items).

int64_tArrowArray.null_count#

Mandatory. The number of null items in the array. MAY be -1 if notyet computed.

int64_tArrowArray.offset#

Mandatory. The logical offset inside the array (i.e. the number of itemsfrom the physical start of the buffers). MUST be 0 or positive.

Producers MAY specify that they will only produce 0-offset arrays toease implementation of consumer code.Consumers MAY decide not to support non-0-offset arrays, but theyshould document this limitation.

int64_tArrowArray.n_buffers#

Mandatory. The number of physical buffers backing this array. Thenumber of buffers is a function of the data type, as described in theColumnar format specification, except for thethe binary or utf-8 view type, which has one additional buffer comparedto the Columnar format specification (seeBinary view arrays).

Buffers of children arrays are not included.

constvoid**ArrowArray.buffers#

Mandatory. A C array of pointers to the start of each physical bufferbacking this array. Eachvoid* pointer is the physical start ofa contiguous buffer. There must beArrowArray.n_buffers pointers.

The producer MUST ensure that each contiguous buffer is large enough torepresentlength + offset values encoded according to theColumnar format specification.

It is recommended, but not required, that the memory addresses of thebuffers be aligned at least according to the type of primitive data thatthey contain. Consumers MAY decide not to support unaligned memory.

The buffer pointers MAY be null only in two situations:

1. for the null bitmap buffer, ifArrowArray.null_count is 0;

2. for any buffer, if the size in bytes of the corresponding buffer would be 0.

Buffers of children arrays are not included.

int64_tArrowArray.n_children#

Mandatory. The number of children this array has. The number of childrenis a function of the data type, as described in theColumnar format specification.

ArrowArray**ArrowArray.children#

Optional. A C array of pointers to each child array of this array.There must beArrowArray.n_children pointers.

MAY be NULL only ifArrowArray.n_children is 0.

ArrowArray*ArrowArray.dictionary#

Optional. A pointer to the underlying array of dictionary values.

MUST be present if the ArrowArray represents a dictionary-encoded array.MUST be NULL otherwise.

void(*ArrowArray.release)(structArrowArray*)#

Mandatory. A pointer to a producer-provided release callback.

See below for memory management and release callback semantics.

void*ArrowArray.private_data#

Optional. An opaque pointer to producer-provided private data.

Consumers MUST not process this member. Lifetime of this memberis handled by the producer, and especially by the release callback.

Dictionary-encoded arrays#

For dictionary-encoded arrays, theArrowSchema.format stringencodes theindex type. The dictionaryvalue type can be readfrom theArrowSchema.dictionary structure.

The same holds forArrowArray structure: while the parentstructure points to the index data, theArrowArray.dictionarypoints to the dictionary values array.

Extension arrays#

For extension arrays, theArrowSchema.format string encodes thestorage type. Information about the extension type is encoded in theArrowSchema.metadata string, similarly to theIPC format. Specifically, themetadata keyARROW:extension:name encodes the extension type name,and the metadata keyARROW:extension:metadata encodes theimplementation-specific serialization of the extension type (forparameterized extension types).

TheArrowArray structure exported from an extension array simply pointsto the storage data of the extension array.

Binary view arrays#

For binary or utf-8 view arrays, an extra buffer is appended which storesthe lengths of each variadic data buffer asint64_t. This buffer isnecessary since these buffer lengths are not trivially extractable fromother data in an array of binary or utf-8 view type.

Memory management#

TheArrowSchema andArrowArray structures follow the same conventionsfor memory management. The term“base structure” below refers to theArrowSchema orArrowArray that is passed between producer and consumer– not any child structure thereof.

staticvoidReleaseExportedArray(structArrowArray*array){// This should not be called on already released arrayassert(array->release!=NULL);// Release childrenfor(int64_ti=0;i<array->n_children;++i){structArrowArray*child=array->children[i];if(child->release!=NULL){child->release(child);assert(child->release==NULL);}}// Release dictionarystructArrowArray*dict=array->dictionary;if(dict!=NULL&&dict->release!=NULL){dict->release(dict);assert(dict->release==NULL);}// TODO here: release and/or deallocate all data directly owned by// the ArrowArray struct, such as the private_data.// Mark array releasedarray->release=NULL;}

Record batches#

A record batch can be trivially considered as an equivalent struct array. Inthis case the metadata of the top-levelArrowSchema can be used for theschema-level metadata of the record batch.

Mutability#

Both the producer and the consumer SHOULD consider the exported data(that is, the data reachable through thebuffers member ofArrowArray)to be immutable, as either party could otherwise see inconsistent data whilethe other is mutating it.

Exporting a simpleint32 array#

Export a non-nullableint32 type with empty metadata. In this case,allArrowSchema members point to statically-allocated data, so therelease callback is trivial.

staticvoidrelease_int32_type(structArrowSchema*schema){// Mark releasedschema->release=NULL;}voidexport_int32_type(structArrowSchema*schema){*schema=(structArrowSchema){// Type description.format="i",.name="",.metadata=NULL,.flags=0,.n_children=0,.children=NULL,.dictionary=NULL,// Bookkeeping.release=&release_int32_type};}

Export a C-malloc()ed array of the same type as a Arrow array, transferringownership to the consumer through the release callback:

staticvoidrelease_int32_array(structArrowArray*array){assert(array->n_buffers==2);// Free the buffers and the buffers arrayfree((void*)array->buffers[1]);free(array->buffers);// Mark releasedarray->release=NULL;}voidexport_int32_array(constint32_t*data,int64_tnitems,structArrowArray*array){// Initialize primitive fields*array=(structArrowArray){// Data description.length=nitems,.offset=0,.null_count=0,.n_buffers=2,.n_children=0,.children=NULL,.dictionary=NULL,// Bookkeeping.release=&release_int32_array};// Allocate list of buffersarray->buffers=(constvoid**)malloc(sizeof(void*)*array->n_buffers);assert(array->buffers!=NULL);array->buffers[0]=NULL;// no nulls, null bitmap can be omittedarray->buffers[1]=data;}

Exporting astruct<float32,utf8> array#

Export the array type as aArrowSchema with C-malloc()ed children:

staticvoidrelease_malloced_type(structArrowSchema*schema){inti;for(i=0;i<schema->n_children;++i){structArrowSchema*child=schema->children[i];if(child->release!=NULL){child->release(child);}free(child);}free(schema->children);// Mark releasedschema->release=NULL;}voidexport_float32_utf8_type(structArrowSchema*schema){structArrowSchema*child;//// Initialize parent type//*schema=(structArrowSchema){// Type description.format="+s",.name="",.metadata=NULL,.flags=0,.n_children=2,.dictionary=NULL,// Bookkeeping.release=&release_malloced_type};// Allocate list of children typesschema->children=malloc(sizeof(structArrowSchema*)*schema->n_children);//// Initialize child type #0//child=schema->children[0]=malloc(sizeof(structArrowSchema));*child=(structArrowSchema){// Type description.format="f",.name="floats",.metadata=NULL,.flags=ARROW_FLAG_NULLABLE,.n_children=0,.dictionary=NULL,.children=NULL,// Bookkeeping.release=&release_malloced_type};//// Initialize child type #1//child=schema->children[1]=malloc(sizeof(structArrowSchema));*child=(structArrowSchema){// Type description.format="u",.name="strings",.metadata=NULL,.flags=ARROW_FLAG_NULLABLE,.n_children=0,.dictionary=NULL,.children=NULL,// Bookkeeping.release=&release_malloced_type};}

Export C-malloc()ed arrays in Arrow-compatible layout as an Arrow struct array,transferring ownership to the consumer:

staticvoidrelease_malloced_array(structArrowArray*array){inti;// Free childrenfor(i=0;i<array->n_children;++i){structArrowArray*child=array->children[i];if(child->release!=NULL){child->release(child);}free(child);}free(array->children);// Free buffersfor(i=0;i<array->n_buffers;++i){free((void*)array->buffers[i]);}free(array->buffers);// Mark releasedarray->release=NULL;}voidexport_float32_utf8_array(int64_tnitems,constuint8_t*float32_nulls,constfloat*float32_data,constuint8_t*utf8_nulls,constint32_t*utf8_offsets,constuint8_t*utf8_data,structArrowArray*array){structArrowArray*child;//// Initialize parent array//*array=(structArrowArray){// Data description.length=nitems,.offset=0,.null_count=0,.n_buffers=1,.n_children=2,.dictionary=NULL,// Bookkeeping.release=&release_malloced_array};// Allocate list of parent buffersarray->buffers=malloc(sizeof(void*)*array->n_buffers);array->buffers[0]=NULL;// no nulls, null bitmap can be omitted// Allocate list of children arraysarray->children=malloc(sizeof(structArrowArray*)*array->n_children);//// Initialize child array #0//child=array->children[0]=malloc(sizeof(structArrowArray));*child=(structArrowArray){// Data description.length=nitems,.offset=0,.null_count=-1,.n_buffers=2,.n_children=0,.dictionary=NULL,.children=NULL,// Bookkeeping.release=&release_malloced_array};child->buffers=malloc(sizeof(void*)*child->n_buffers);child->buffers[0]=float32_nulls;child->buffers[1]=float32_data;//// Initialize child array #1//child=array->children[1]=malloc(sizeof(structArrowArray));*child=(structArrowArray){// Data description.length=nitems,.offset=0,.null_count=-1,.n_buffers=3,.n_children=0,.dictionary=NULL,.children=NULL,// Bookkeeping.release=&release_malloced_array};child->buffers=malloc(sizeof(void*)*child->n_buffers);child->buffers[0]=utf8_nulls;child->buffers[1]=utf8_offsets;child->buffers[2]=utf8_data;}