Getting Started#

Arrow’s memory management is built around the needs of the columnar format and using off-heap memory.Arrow Java has its own independent implementation. It does not wrap the C++ implementation, although the framework is flexible enoughto be used with memory allocated in C++ that is used by Java code.

Arrow provides multiple modules: the core interfaces, and implementations of the interfaces.Users need the core interfaces, and exactly one of the implementations.

• memory-core: Provides the interfaces used by the Arrow libraries and applications.

• memory-netty: An implementation of the memory interfaces based on theNetty library.

• memory-unsafe: An implementation of the memory interfaces based on thesun.misc.Unsafe library.

ArrowBuf#

ArrowBuf represents a single, contiguous region ofdirect memory. It consists of an address and a length,and provides low-level interfaces for working with the contents, similar to ByteBuffer.

Unlike (Direct)ByteBuffer, it has reference counting built in, as discussed later.

BufferAllocator#

TheBufferAllocator is primarily an arena or nursery used for accounting of buffers (ArrowBuf instances).As the name suggests, it can allocate new buffers associated with itself, but it can alsohandle the accounting for buffers allocated elsewhere. For example, it handles the Java-side accounting formemory allocated in C++ and shared with Java using the C-Data Interface. In the code below it performs an allocation:

importorg.apache.arrow.memory.ArrowBuf;importorg.apache.arrow.memory.BufferAllocator;importorg.apache.arrow.memory.RootAllocator;try(BufferAllocatorbufferAllocator=newRootAllocator(8*1024)){ArrowBufarrowBuf=bufferAllocator.buffer(4*1024);System.out.println(arrowBuf);arrowBuf.close();}

ArrowBuf[2],address:140363641651200,length:4096

The concrete implementation of the BufferAllocator interface isRootAllocator. Applications should generally createone RootAllocator at the start of the program, and use it through the BufferAllocator interface. Allocators implementAutoCloseable and must be closed after the application is done with them; this will check that all outstanding memoryhas been freed (see the next section).

Arrow provides a tree-based model for memory allocation. The RootAllocator is created first, then more allocatorsare created as children of an existing allocator vianewChildAllocator. When creating a RootAllocator or a childallocator, a memory limit is provided, and when allocating memory, the limit is checked. Furthermore, when allocatingmemory from a child allocator, those allocations are also reflected in all parent allocators. Hence, the RootAllocatoreffectively sets the program-wide memory limit, and serves as the master bookkeeper for all memory allocations.

Child allocators are not strictly required, but can help better organize code. For instance, a lower memory limit canbe set for a particular section of code. The child allocator can be closed when that section completes,at which point it checks that that section didn’t leak any memory.Child allocators can also be named, which makes it easier to tell where an ArrowBuf came from during debugging.

Reference counting#

Because direct memory is expensive to allocate and deallocate, allocators may share direct buffers. To manage shared buffersdeterministically, we use manual reference counting instead of the garbage collector.This simply means that each buffer has a counter keeping track of the number of references tothe buffer, and the user is responsible for properly incrementing/decrementing the counter as the buffer is used.

In Arrow, each ArrowBuf has an associatedReferenceManager that tracks the reference count. You can retrieveit with ArrowBuf.getReferenceManager(). The reference count is updated usingReferenceManager.release to decrement the count,andReferenceManager.retain to increment it.

Of course, this is tedious and error-prone, so instead of directly working with buffers, we typically usehigher-level APIs like ValueVector. Such classes generally implement Closeable/AutoCloseable and will automaticallydecrement the reference count when closed.

Allocators implement AutoCloseable as well. In this case, closing the allocator will check that all buffersobtained from the allocator are closed. If not,close() method will raise an exception; this helps trackmemory leaks from unclosed buffers.

Reference counting needs to be handled carefully. To ensure that anindependent section of code has fully cleaned up all allocated buffers, use a new child allocator.

Development Guidelines#

Applications should generally:

• Use the BufferAllocator interface in APIs instead of RootAllocator.

• Create one RootAllocator at the start of the program and explicitly pass it when needed.

• close() allocators after use (whether they are child allocators or the RootAllocator), either manually or preferably via a try-with-resources statement.

Debugging Memory Leaks/Allocation#

InDEBUG mode, the allocator and supporting classes will record additionaldebug tracking information to better track down memory leaks and issues. Toenable DEBUG mode pass the following system property to the VM when starting-Darrow.memory.debug.allocator=true.

When DEBUG is enabled, a log will be kept of allocations. Configure SLF4J to see these logs (e.g. via Logback/Apache Log4j).Consider the following example to see how it helps us with the tracking of allocators:

importorg.apache.arrow.memory.ArrowBuf;importorg.apache.arrow.memory.BufferAllocator;importorg.apache.arrow.memory.RootAllocator;try(BufferAllocatorbufferAllocator=newRootAllocator(8*1024)){ArrowBufarrowBuf=bufferAllocator.buffer(4*1024);System.out.println(arrowBuf);}

Without the debug mode enabled, when we close the allocator, we get this:

11:56:48.944[main]INFOo.apache.arrow.memory.BaseAllocator-Debugmodedisabled.ArrowBuf[2],address:140508391276544,length:409616:28:08.847[main]ERRORo.apache.arrow.memory.BaseAllocator-Memorywasleakedbyquery.Memoryleaked:(4096)Allocator(ROOT)0/4096/4096/8192(res/actual/peak/limit)

Enabling the debug mode, we get more details:

11:56:48.944[main]INFOo.apache.arrow.memory.BaseAllocator-Debugmodeenabled.ArrowBuf[2],address:140437894463488,length:4096Exceptioninthread"main"java.lang.IllegalStateException:Allocator[ROOT]closedwithoutstandingbuffersallocated(1).Allocator(ROOT)0/4096/4096/8192(res/actual/peak/limit)childallocators:0ledgers:1ledger[1]allocator:ROOT),isOwning:,size:,references:1,life:261438177096661..0,allocatorManager:[,life:]holds1buffers.ArrowBuf[2],address:140437894463488,length:4096reservations:0

Additionally, in debug mode,ArrowBuf.print() can be used to obtain a debug string.This will include information about allocation operations on the buffer with stack traces, such as when/where the buffer was allocated.

importorg.apache.arrow.memory.ArrowBuf;importorg.apache.arrow.memory.BufferAllocator;importorg.apache.arrow.memory.RootAllocator;try(finalBufferAllocatorallocator=newRootAllocator()){try(finalArrowBufbuf=allocator.buffer(1024)){finalStringBuildersb=newStringBuilder();buf.print(sb,/*indent*/0);System.out.println(sb.toString());}}

ArrowBuf[2], address:140433199984656, length:1024 event log for: ArrowBuf[2]   675959093395667 create()      at org.apache.arrow.memory.util.HistoricalLog$Event.<init>(HistoricalLog.java:175)      at org.apache.arrow.memory.util.HistoricalLog.recordEvent(HistoricalLog.java:83)      at org.apache.arrow.memory.ArrowBuf.<init>(ArrowBuf.java:96)      at org.apache.arrow.memory.BufferLedger.newArrowBuf(BufferLedger.java:271)      at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:300)      at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:276)      at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)      at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:240)      at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)      at REPL.$JShell$14.do_it$($JShell$14.java:10)      at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)      at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)      at java.lang.reflect.Method.invoke(Method.java:566)      at jdk.jshell.execution.DirectExecutionControl.invoke(DirectExecutionControl.java:209)      at jdk.jshell.execution.RemoteExecutionControl.invoke(RemoteExecutionControl.java:116)      at jdk.jshell.execution.DirectExecutionControl.invoke(DirectExecutionControl.java:119)      at jdk.jshell.execution.ExecutionControlForwarder.processCommand(ExecutionControlForwarder.java:144)      at jdk.jshell.execution.ExecutionControlForwarder.commandLoop(ExecutionControlForwarder.java:262)      at jdk.jshell.execution.Util.forwardExecutionControl(Util.java:76)      at jdk.jshell.execution.Util.forwardExecutionControlAndIO(Util.java:137)      at jdk.jshell.execution.RemoteExecutionControl.main(RemoteExecutionControl.java:70)

The BufferAllocator also provides aBufferAllocator.toVerboseString() which can be used inDEBUG mode to get extensive stacktrace information and events associated with various Allocator behaviors.

Finally, enabling theTRACE logging level will automatically provide this stack trace when the allocator is closed:

// Assumes use of Logback; adjust for Log4j, etc. as appropriateimportch.qos.logback.classic.Level;importch.qos.logback.classic.Logger;importorg.apache.arrow.memory.ArrowBuf;importorg.apache.arrow.memory.BufferAllocator;importorg.apache.arrow.memory.RootAllocator;importorg.slf4j.LoggerFactory;// Set log level to TRACE to get tracebacks((Logger)LoggerFactory.getLogger("org.apache.arrow")).setLevel(Level.TRACE);try(finalBufferAllocatorallocator=newRootAllocator()){// Leak bufferallocator.buffer(1024);}

|  Exception java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding buffers allocated (1).Allocator(ROOT) 0/1024/1024/9223372036854775807 (res/actual/peak/limit)  child allocators: 0  ledgers: 1    ledger[1] allocator: ROOT), isOwning: , size: , references: 1, life: 712040870231544..0, allocatorManager: [, life: ] holds 1 buffers.        ArrowBuf[2], address:139926571810832, length:1024     event log for: ArrowBuf[2]       712040888650134 create()              at org.apache.arrow.memory.util.StackTrace.<init>(StackTrace.java:34)              at org.apache.arrow.memory.util.HistoricalLog$Event.<init>(HistoricalLog.java:175)              at org.apache.arrow.memory.util.HistoricalLog.recordEvent(HistoricalLog.java:83)              at org.apache.arrow.memory.ArrowBuf.<init>(ArrowBuf.java:96)              at org.apache.arrow.memory.BufferLedger.newArrowBuf(BufferLedger.java:271)              at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:300)              at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:276)              at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)              at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:240)              at org.apache.arrow.memory.RootAllocator.buffer(RootAllocator.java:29)              at REPL.$JShell$18.do_it$($JShell$18.java:13)              at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)              at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)              at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)              at java.lang.reflect.Method.invoke(Method.java:566)              at jdk.jshell.execution.DirectExecutionControl.invoke(DirectExecutionControl.java:209)              at jdk.jshell.execution.RemoteExecutionControl.invoke(RemoteExecutionControl.java:116)              at jdk.jshell.execution.DirectExecutionControl.invoke(DirectExecutionControl.java:119)              at jdk.jshell.execution.ExecutionControlForwarder.processCommand(ExecutionControlForwarder.java:144)              at jdk.jshell.execution.ExecutionControlForwarder.commandLoop(ExecutionControlForwarder.java:262)              at jdk.jshell.execution.Util.forwardExecutionControl(Util.java:76)              at jdk.jshell.execution.Util.forwardExecutionControlAndIO(Util.java:137)  reservations: 0|        at BaseAllocator.close (BaseAllocator.java:405)|        at RootAllocator.close (RootAllocator.java:29)|        at (#8:1)

Sometimes, explicitly passing allocators around is difficult. For example, itcan be hard to pass around extra state, like an allocator, through layers ofexisting application or framework code. A global or singleton allocator instancecan be useful here, though it should not be your first choice.

How this works:

1. Set up a global allocator in a singleton class.

2. Provide methods to create child allocators from the global allocator.

3. Give child allocators proper names to make it easier to figure out whereallocations occurred in case of errors.

4. Ensure that resources are properly closed.

5. Check that the global allocator is empty at some suitable point, such asright before program shutdown.

6. If it is not empty, review the above allocation bugs.

//1privatestaticfinalBufferAllocatorallocator=newRootAllocator();privatestaticfinalAtomicIntegerchildNumber=newAtomicInteger(0);...//2publicstaticBufferAllocatorgetChildAllocator(){returnallocator.newChildAllocator(nextChildName(),0,Long.MAX_VALUE);}...//3privatestaticStringnextChildName(){return"Allocator-Child-"+childNumber.incrementAndGet();}...//4: Business codetry(BufferAllocatorallocator=GlobalAllocator.getChildAllocator()){...}...//5publicstaticvoidcheckGlobalCleanUpResources(){...if(!allocator.getChildAllocators().isEmpty()){thrownewIllegalStateException(...);}elseif(allocator.getAllocatedMemory()!=0){thrownewIllegalStateException(...);}}

Design Principles#

Arrow’s memory model is based on the following basic concepts:

• Memory can be allocated up to some limit. That limit could be a reallimit (OS/JVM) or a locally imposed limit.

• Allocation operates in two phases: accounting then actual allocation.Allocation could fail at either point.

• Allocation failure should be recoverable. In all cases, the Allocatorinfrastructure should expose memory allocation failures (OS orinternal limit-based) asOutOfMemoryExceptions.

• Any allocator can reserve memory when created. This memory shall beheld such that this allocator will always be able to allocate thatamount of memory.

• A particular application component should work to use a localallocator to understand local memory usage and better debug memoryleaks.

• The same physical memory can be shared by multiple allocators and theallocator must provide an accounting paradigm for this purpose.

Reserving Memory#

Arrow provides two different ways to reserve memory:

• BufferAllocator accounting reservations: When a new allocator (otherthan theRootAllocator) is initialized, it can set aside memorythat it will keep locally for its lifetime. This is memory that willnever be released back to its parent allocator until the allocator isclosed.

• AllocationReservation via BufferAllocator.newReservation():Allows a short-term preallocation strategy so that a particularsubsystem can ensure future memory is available to support aparticular request.

Reference Counting Details#

Typically, the ReferenceManager implementation used is an instance ofBufferLedger.A BufferLedger is a ReferenceManager that also maintains the relationship between anAllocationManager,aBufferAllocator and one or more individualArrowBufs

All ArrowBufs (direct or sliced) related to a single BufferLedger/BufferAllocator combinationshare the same reference count and either all will be valid or all will be invalid.For simplicity of accounting, we treat that memory as being used by oneof the BufferAllocators associated with the memory. When that allocatorreleases its claim on that memory, the memory ownership is then moved toanother BufferLedger belonging to the same AllocationManager.

Allocation Details#

There are several Allocator types in Arrow Java:

• BufferAllocator - The public interface application users should be leveraging

• BaseAllocator - The base implementation of memory allocation, contains the meat of the Arrow allocator implementation

• RootAllocator - The root allocator. Typically only one created for a JVM. It serves as the parent/ancestor for child allocators

• ChildAllocator - A child allocator that derives from the root allocator

Many BufferAllocators can reference the same piece of physical memory at the sametime. It is the AllocationManager’s responsibility to ensure that in this situation,all memory is accurately accounted for from the Root’s perspectiveand also to ensure that the memory is correctly released once allBufferAllocators have stopped using that memory.

For simplicity of accounting, we treat that memory as being used by oneof the BufferAllocators associated with the memory. When that allocatorreleases its claim on that memory, the memory ownership is then moved toanother BufferLedger belonging to the same AllocationManager. Note thatbecause a ArrowBuf.release() is what actually causes memory ownershiptransfer to occur, we always proceed with ownership transfer (even ifthat violates an allocator limit). It is the responsibility of theapplication owning a particular allocator to frequently confirm whetherthe allocator is over its memory limit (BufferAllocator.isOverLimit())and if so, attempt to aggressively release memory to ameliorate thesituation.

Object Hierarchy#

There are two main ways that someone can look at the object hierarchyfor Arrow’s memory management scheme. The first is a memory basedperspective as below:

+ AllocationManager||-- UnsignedDirectLittleEndian (One per AllocationManager)||-+ BufferLedger 1 ==> Allocator A (owning)| ` - ArrowBuf 1|-+ BufferLedger 2 ==> Allocator B (non-owning)| ` - ArrowBuf 2|-+ BufferLedger 3 ==> Allocator C (non-owning)  | - ArrowBuf 3  | - ArrowBuf 4  ` - ArrowBuf 5

+ RootAllocator|-+ ChildAllocator 1| | - ChildAllocator 1.1| ` ...||-+ ChildAllocator 2|-+ ChildAllocator 3| || |-+ BufferLedger 1 ==> AllocationManager 1 (owning) ==> UDLE| | `- ArrowBuf 1| `-+ BufferLedger 2 ==> AllocationManager 2 (non-owning)==> UDLE|   `- ArrowBuf 2||-+ BufferLedger 3 ==> AllocationManager 1 (non-owning)==> UDLE| ` - ArrowBuf 3|-+ BufferLedger 4 ==> AllocationManager 2 (owning) ==> UDLE  | - ArrowBuf 4  | - ArrowBuf 5  ` - ArrowBuf 6