Reading and writing the Arrow IPC format#
Arrow C++ provides readers and writers for the Arrow IPC format which wraplower level input/output, handled through theIO interfaces.For reading, there is also an event-driven API that enables feedingarbitrary data into the IPC decoding layer asynchronously.
Reading IPC streams and files#
Synchronous reading#
For most cases, it is most convenient to use theRecordBatchStreamReaderorRecordBatchFileReader class, depending on which variant of the IPCformat you want to read. The former requires aInputStreamsource, while the latter requires aRandomAccessFile.
Reading Arrow IPC data is inherently zero-copy if the source allows it.For example, aBufferReader orMemoryMappedFilecan typically be zero-copy. Exceptions are when the data must be transformedon the fly, e.g. when buffer compression has been enabled on the IPC streamor file.
Event-driven reading#
When it is necessary to process the IPC format without blocking (for exampleto integrate Arrow with an event loop), or if data is coming from an unusualsource, use the event-drivenStreamDecoder. You will need to definea subclass ofListener and implement the virtual methods for thedesired events (for example, implementListener::OnRecordBatchDecoded()to be notified of each incomingRecordBatch).
Writing IPC streams and files#
Use one of the factory functions,MakeStreamWriter() orMakeFileWriter(), to obtain aRecordBatchWriter instance forthe given IPC format variant.
Configuring#
Various aspects of reading and writing the IPC format can be configuredusing theIpcReadOptions andIpcWriteOptions classes,respectively.