Memory Management#
See also
Buffers#
To avoid passing around raw data pointers with varying and non-obviouslifetime rules, Arrow provides a generic abstraction calledarrow::Buffer.A Buffer encapsulates a pointer and data size, and generally also ties itslifetime to that of an underlying provider (in other words, a Buffer shouldalways point to valid memory till its destruction). Buffers are untyped:they simply denote a physical memory area regardless of its intended meaningor interpretation.
Buffers may be allocated by Arrow itself , or by third-party routines.For example, it is possible to pass the data of a Python bytestring as a Arrowbuffer, keeping the Python object alive as necessary.
In addition, buffers come in various flavours: mutable or not, resizable ornot. Generally, you will hold a mutable buffer when building up a pieceof data, then it will be frozen as an immutable container such as anarray.
Note
Some buffers may point to non-CPU memory, such as GPU-backed memoryprovided by a CUDA context. If you’re writing a GPU-aware application,you will need to be careful not to interpret a GPU memory pointer asa CPU-reachable pointer, or vice-versa.
Accessing Buffer Memory#
Buffers provide fast access to the underlying memory using thesize() anddata() accessors(ormutable_data() for writable access to a mutablebuffer).
Slicing#
It is possible to make zero-copy slices of buffers, to obtain a bufferreferring to some contiguous subset of the underlying data. This is doneby calling thearrow::SliceBuffer() andarrow::SliceMutableBuffer()functions.
Allocating a Buffer#
You can allocate a buffer yourself by calling one of thearrow::AllocateBuffer() orarrow::AllocateResizableBuffer()overloads:
arrow::Result<std::unique_ptr<Buffer>>maybe_buffer=arrow::AllocateBuffer(4096);if(!maybe_buffer.ok()){// ... handle allocation error}std::shared_ptr<arrow::Buffer>buffer=*std::move(maybe_buffer);uint8_t*buffer_data=buffer->mutable_data();memcpy(buffer_data,"hello world",11);
Allocating a buffer this way ensures it is 64-bytes aligned and paddedas recommended by theArrow memory specification.
Building a Buffer#
You can also allocateand build a Buffer incrementally, using thearrow::BufferBuilder API:
BufferBuilderbuilder;builder.Resize(11);// reserve enough space for 11 bytesbuilder.Append("hello ",6);builder.Append("world",5);automaybe_buffer=builder.Finish();if(!maybe_buffer.ok()){// ... handle buffer allocation error}std::shared_ptr<arrow::Buffer>buffer=*maybe_buffer;
If a Buffer is meant to contain values of a given fixed-width type (forexample the 32-bit offsets of a List array), it can be more convenient touse the templatearrow::TypedBufferBuilder API:
TypedBufferBuilder<int32_t>builder;builder.Reserve(2);// reserve enough space for two int32_t valuesbuilder.Append(0x12345678);builder.Append(-0x765643210);automaybe_buffer=builder.Finish();if(!maybe_buffer.ok()){// ... handle buffer allocation error}std::shared_ptr<arrow::Buffer>buffer=*maybe_buffer;
Memory Pools#
When allocating a Buffer using the Arrow C++ API, the buffer’s underlyingmemory is allocated by aarrow::MemoryPool instance. Usually thiswill be the process-widedefault memory pool, but many Arrow APIs allowyou to pass another MemoryPool instance for their internal allocations.
Memory pools are used for large long-lived data such as array buffers.Other data, such as small C++ objects and temporary workspaces, usuallygoes through the regular C++ allocators.
Default Memory Pool#
The default memory pool depends on how Arrow C++ was compiled:
Overriding the Default Memory Pool#
One can override the above selection algorithm by setting theARROW_DEFAULT_MEMORY_POOL environment variable.
STL Integration#
If you wish to use a Arrow memory pool to allocate the data of STL containers,you can do so using thearrow::stl::allocator wrapper.
Conversely, you can also use a STL allocator to allocate Arrow memory,using thearrow::stl::STLMemoryPool class. However, this may be lessperformant, as STL allocators don’t provide a resizing operation.
Devices#
Many Arrow applications only access host (CPU) memory. However, in some casesit is desirable to handle on-device memory (such as on-board memory on a GPU)as well as host memory.
Arrow represents the CPU and other devices using thearrow::Device abstraction. The associated classarrow::MemoryManagerspecifies how to allocate on a given device. Each device has a default memory manager, butadditional instances may be constructed (for example, wrapping a customarrow::MemoryPool the CPU).arrow::MemoryManager instances which specify how to allocatememory on a given device (for example, using a particulararrow::MemoryPool on the CPU).
Device-Agnostic Programming#
If you receive a Buffer from third-party code, you can query whether it isCPU-readable by calling itsis_cpu() method.
You can also view the Buffer on a given device, in a generic way, by callingarrow::Buffer::View() orarrow::Buffer::ViewOrCopy(). This willbe a no-operation if the source and destination devices are identical.Otherwise, a device-dependent mechanism will attempt to construct a memoryaddress for the destination device that gives access to the buffer contents.Actual device-to-device transfer may happen lazily, when reading the buffercontents.
Similarly, if you want to do I/O on a buffer without assuming a CPU-readablebuffer, you can callarrow::Buffer::GetReader() andarrow::Buffer::GetWriter().
For example, to get an on-CPU view or copy of an arbitrary buffer, you cansimply do:
std::shared_ptr<arrow::Buffer>arbitrary_buffer=...;std::shared_ptr<arrow::Buffer>cpu_buffer=arrow::Buffer::ViewOrCopy(arbitrary_buffer,arrow::default_cpu_memory_manager());
Memory Profiling#
On Linux, detailed profiles of memory allocations can be generated usingperfrecord, without any need to modify the binaries. These profiles canshow the traceback in addition to allocation size. This does require debugsymbols, from either a debug build or a release with debug symbols build.
Note
If you are profiling Arrow’s tests on another platform, you can run thefollowing Docker container using Archery to access a Linux environment:
archerydockerrunubuntu-cppbash# Inside the Docker container.../arrow/ci/scripts/cpp_build.sh/arrow/buildcdbuild/cpp/debug./arrow-array-test# Run a testapt-getupdateapt-getinstall-ylinux-tools-genericaliasperf=/usr/lib/linux-tools/<version-path>/perf
To track allocations, create probe points on each of the allocator methods used.Collecting$params allows us to record the size of the allocationsrequested, while collecting$retval allows us to record the address ofrecorded allocations, so we can correlate them with the call to free/de-allocate.
perfprobe-xlibarrow.soje_arrow_mallocx'$params'perfprobe-xlibarrow.soje_arrow_mallocx%return'$retval'perfprobe-xlibarrow.soje_arrow_rallocx'$params'perfprobe-xlibarrow.soje_arrow_rallocx%return'$retval'perfprobe-xlibarrow.soje_arrow_dallocx'$params'PROBE_ARGS="-e probe_libarrow:je_arrow_mallocx \ -e probe_libarrow:je_arrow_mallocx__return \ -e probe_libarrow:je_arrow_rallocx \ -e probe_libarrow:je_arrow_rallocx__return \ -e probe_libarrow:je_arrow_dallocx"
perfprobe-xlibarrow.somi_malloc_aligned'$params'perfprobe-xlibarrow.somi_malloc_aligned%return'$retval'perfprobe-xlibarrow.somi_realloc_aligned'$params'perfprobe-xlibarrow.somi_realloc_aligned%return'$retval'perfprobe-xlibarrow.somi_free'$params'PROBE_ARGS="-e probe_libarrow:mi_malloc_aligned \ -e probe_libarrow:mi_malloc_aligned__return \ -e probe_libarrow:mi_realloc_aligned \ -e probe_libarrow:mi_realloc_aligned__return \ -e probe_libarrow:mi_free"
Once probes have been set, you can record calls with associated tracebacks usingperfrecord. In this example, we are running the StructArray unit tests inArrow:
perfrecord-g--call-graphdwarf\$PROBE_ARGS\./arrow-array-test--gtest_filter=StructArray*
If you want to profile a running process, you can runperfrecord-p<PID>and it will record until you interrupt with CTRL+C. Alternatively, you can doperfrecord-P<PID>sleep10 to record for 10 seconds.
The resulting data can be processed with standard tools to work with perf orperfscript can be used to pipe a text format of the data to custom scripts.The following script parsesperfscript output and prints the output innew lines delimited JSON for easier processing.
importsysimportreimportjson# Example non-traceback line# arrow-array-tes 14344 [003] 7501.073802: probe_libarrow:je_arrow_mallocx: (7fbcd20bb640) size=0x80 flags=6current={}current_traceback=''defnew_row():globalcurrent_tracebackcurrent['traceback']=current_tracebackprint(json.dumps(current))current_traceback=''forlineinsys.stdin:ifline=='\n':continueelifline[0]=='\t':# traceback linecurrent_traceback+=line.strip("\t")else:line=line.rstrip('\n')ifnotlen(current)==0:new_row()parts=re.sub(' +',' ',line).split(' ')parts.reverse()parts.pop()# fileparts.pop()# "14344"parts.pop()# "[003]"current['time']=float(parts.pop().rstrip(":"))current['event']=parts.pop().rstrip(":")parts.pop()# (7fbcd20bddf0)ifparts[-1]=="<-":parts.pop()parts.pop()params={}forpairinparts:key,value=pair.split("=")params[key]=valuecurrent['params']=params
Here’s an example invocation of that script, with a preview of output data:
$perfscript|python3/arrow/process_perf_events.py>processed_events.jsonl$headprocessed_events.jsonl|cut-c-120{"time": 14814.954378, "event": "probe_libarrow:je_arrow_mallocx", "params": {"flags": "6", "size": "0x80"}, "traceback"{"time": 14814.95443, "event": "probe_libarrow:je_arrow_mallocx__return", "params": {"arg1": "0x7f4a97e09000"}, "traceba{"time": 14814.95448, "event": "probe_libarrow:je_arrow_mallocx", "params": {"flags": "6", "size": "0x40"}, "traceback":{"time": 14814.954486, "event": "probe_libarrow:je_arrow_mallocx__return", "params": {"arg1": "0x7f4a97e0a000"}, "traceb{"time": 14814.954502, "event": "probe_libarrow:je_arrow_rallocx", "params": {"flags": "6", "size": "0x40", "ptr": "0x7f{"time": 14814.954507, "event": "probe_libarrow:je_arrow_rallocx__return", "params": {"arg1": "0x7f4a97e0a040"}, "traceb{"time": 14814.954796, "event": "probe_libarrow:je_arrow_mallocx", "params": {"flags": "6", "size": "0x40"}, "traceback"{"time": 14814.954805, "event": "probe_libarrow:je_arrow_mallocx__return", "params": {"arg1": "0x7f4a97e0a080"}, "traceb{"time": 14814.954817, "event": "probe_libarrow:je_arrow_mallocx", "params": {"flags": "6", "size": "0x40"}, "traceback"{"time": 14814.95482, "event": "probe_libarrow:je_arrow_mallocx__return", "params": {"arg1": "0x7f4a97e0a0c0"}, "traceba
From there one can answer a number of questions. For example, the followingscript will find which allocations were never freed, and print the associatedtracebacks along with the count of dangling allocations:
'''Find tracebacks of allocations with no corresponding free'''importsysimportjsonfromcollectionsimportdefaultdictallocated=dict()forlineinsys.stdin:line=line.rstrip('\n')data=json.loads(line)ifdata['event']=="probe_libarrow:je_arrow_mallocx__return":address=data['params']['arg1']allocated[address]=data['traceback']elifdata['event']=="probe_libarrow:je_arrow_rallocx":address=data['params']['ptr']delallocated[address]elifdata['event']=="probe_libarrow:je_arrow_rallocx__return":address=data['params']['arg1']allocated[address]=data['traceback']elifdata['event']=="probe_libarrow:je_arrow_dallocx":address=data['params']['ptr']ifaddressinallocated:delallocated[address]elifdata['event']=="probe_libarrow:mi_malloc_aligned__return":address=data['params']['arg1']allocated[address]=data['traceback']elifdata['event']=="probe_libarrow:mi_realloc_aligned":address=data['params']['p']delallocated[address]elifdata['event']=="probe_libarrow:mi_realloc_aligned__return":address=data['params']['arg1']allocated[address]=data['traceback']elifdata['event']=="probe_libarrow:mi_free":address=data['params']['p']ifaddressinallocated:delallocated[address]traceback_counts=defaultdict(int)fortracebackinallocated.values():traceback_counts[traceback]+=1fortraceback,countinsorted(traceback_counts.items(),key=lambdax:-x[1]):print("Num of dangling allocations:",count)print(traceback)
The script can be invoked like so:
$catprocessed_events.jsonl|python3/arrow/count_tracebacks.pyNum of dangling allocations: 1 7fc945e5cfd2 arrow::(anonymous namespace)::JemallocAllocator::ReallocateAligned+0x13b (/build/cpp/debug/libarrow.so.700.0.0) 7fc945e5fe4f arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::JemallocAllocator>::Reallocate+0x93 (/build/cpp/debug/libarrow.so.700.0.0) 7fc945e618f7 arrow::PoolBuffer::Resize+0xed (/build/cpp/debug/libarrow.so.700.0.0) 55a38b163859 arrow::BufferBuilder::Resize+0x12d (/build/cpp/debug/arrow-array-test) 55a38b163bbe arrow::BufferBuilder::Finish+0x48 (/build/cpp/debug/arrow-array-test) 55a38b163e3a arrow::BufferBuilder::Finish+0x50 (/build/cpp/debug/arrow-array-test) 55a38b163f90 arrow::BufferBuilder::FinishWithLength+0x4e (/build/cpp/debug/arrow-array-test) 55a38b2c8fa7 arrow::TypedBufferBuilder<int, void>::FinishWithLength+0x4f (/build/cpp/debug/arrow-array-test) 55a38b2bcce7 arrow::NumericBuilder<arrow::Int32Type>::FinishInternal+0x107 (/build/cpp/debug/arrow-array-test) 7fc945c065ae arrow::ArrayBuilder::Finish+0x5a (/build/cpp/debug/libarrow.so.700.0.0) 7fc94736ed41 arrow::ipc::internal::json::(anonymous namespace)::Converter::Finish+0x123 (/build/cpp/debug/libarrow.so.700.0.0) 7fc94737426e arrow::ipc::internal::json::ArrayFromJSON+0x299 (/build/cpp/debug/libarrow.so.700.0.0) 7fc948e98858 arrow::ArrayFromJSON+0x64 (/build/cpp/debug/libarrow_testing.so.700.0.0) 55a38b6773f3 arrow::StructArray_FlattenOfSlice_Test::TestBody+0x79 (/build/cpp/debug/arrow-array-test) 7fc944689633 testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>+0x68 (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc94468132a testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>+0x5d (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc9446555eb testing::Test::Run+0xf1 (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc94465602d testing::TestInfo::Run+0x13f (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc944656947 testing::TestSuite::Run+0x14b (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc9446663f5 testing::internal::UnitTestImpl::RunAllTests+0x433 (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc94468ab61 testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>+0x68 (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc944682568 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>+0x5d (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc944664b0c testing::UnitTest::Run+0xcc (/build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0) 7fc9446d0299 RUN_ALL_TESTS+0x14 (/build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0) 7fc9446d021b main+0x42 (/build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0) 7fc9441e70b2 __libc_start_main+0xf2 (/usr/lib/x86_64-linux-gnu/libc-2.31.so) 55a38b10a50d _start+0x2d (/build/cpp/debug/arrow-array-test)