Other Data Structures#
OurFlatbuffers protocol definition files have metadata for some other datastructures defined to allow other kinds of applications to take advantage ofcommon interprocess communication machinery. These data structures are notconsidered to be part of the columnar format.
An Arrow columnar implementation is not required to implement thesetypes.
Tensor (Multi-dimensional Array)#
TheTensor message types provides a way to write amultidimensional array of fixed-size values (such as a NumPy ndarray).
When writing a standalone encapsulated tensor message, we use theencapsulated IPC format defined in theColumnar Specification, but additionally align the starting offset of thetensor body to be a multiple of 64 bytes:
<metadataprefixandmetadata><PADDING><tensorbody>
Sparse Tensor#
SparseTensor represents a multidimensional array whose elementsare generally almost all zeros.
When writing a standalone encapsulated sparse tensor message, we usethe encapsulated IPC format defined in theColumnar Specification, but additionally align the starting offsets of thesparse index and the sparse tensor body (if writing to a shared memoryregion) to be multiples of 64 bytes:
<metadataprefixandmetadata><PADDING><sparseindex><PADDING><sparsetensorbody>
The contents of the sparse tensor index depends on what kind of sparseformat is used.