What is NUMA?¶

This question can be answered from a couple of perspectives: thehardware view and the Linux software view.

From the hardware perspective, a NUMA system is a computer platform thatcomprises multiple components or assemblies each of which may contain 0or more CPUs, local memory, and/or IO buses. For brevity and todisambiguate the hardware view of these physical components/assembliesfrom the software abstraction thereof, we’ll call the components/assemblies‘cells’ in this document.

Each of the ‘cells’ may be viewed as an SMP [symmetric multi-processor] subsetof the system--although some components necessary for a stand-alone SMP systemmay not be populated on any given cell. The cells of the NUMA system areconnected together with some sort of system interconnect--e.g., a crossbar orpoint-to-point link are common types of NUMA system interconnects. Both ofthese types of interconnects can be aggregated to create NUMA platforms withcells at multiple distances from other cells.

For Linux, the NUMA platforms of interest are primarily what is known as CacheCoherent NUMA or ccNUMA systems. With ccNUMA systems, all memory is visibleto and accessible from any CPU attached to any cell and cache coherencyis handled in hardware by the processor caches and/or the system interconnect.

Memory access time and effective memory bandwidth varies depending on how faraway the cell containing the CPU or IO bus making the memory access is from thecell containing the target memory. For example, access to memory by CPUsattached to the same cell will experience faster access times and higherbandwidths than accesses to memory on other, remote cells. NUMA platformscan have cells at multiple remote distances from any given cell.

Platform vendors don’t build NUMA systems just to make software developers’lives interesting. Rather, this architecture is a means to provide scalablememory bandwidth. However, to achieve scalable memory bandwidth, system andapplication software must arrange for a large majority of the memory references[cache misses] to be to “local” memory--memory on the same cell, if any--orto the closest cell with memory.

This leads to the Linux software view of a NUMA system:

Linux divides the system’s hardware resources into multiple softwareabstractions called “nodes”. Linux maps the nodes onto the physical cellsof the hardware platform, abstracting away some of the details for somearchitectures. As with physical cells, software nodes may contain 0 or moreCPUs, memory and/or IO buses. And, again, memory accesses to memory on“closer” nodes--nodes that map to closer cells--will generally experiencefaster access times and higher effective bandwidth than accesses to moreremote cells.

For some architectures, such as x86, Linux will “hide” any node representing aphysical cell that has no memory attached, and reassign any CPUs attached tothat cell to a node representing a cell that does have memory. Thus, onthese architectures, one cannot assume that all CPUs that Linux associates witha given node will see the same local memory access times and bandwidth.

In addition, for some architectures, again x86 is an example, Linux supportsthe emulation of additional nodes. For NUMA emulation, linux will carve upthe existing nodes--or the system memory for non-NUMA platforms--into multiplenodes. Each emulated node will manage a fraction of the underlying cells’physical memory. NUMA emulation is useful for testing NUMA kernel andapplication features on non-NUMA platforms, and as a sort of memory resourcemanagement mechanism when used together with cpusets.[seeCPUSETS]

For each node with memory, Linux constructs an independent memory managementsubsystem, complete with its own free page lists, in-use page lists, usagestatistics and locks to mediate access. In addition, Linux constructs foreach memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE],an ordered “zonelist”. A zonelist specifies the zones/nodes to visit when aselected zone/node cannot satisfy the allocation request. This situation,when a zone has no available memory to satisfy a request, is called“overflow” or “fallback”.

Because some nodes contain multiple zones containing different types ofmemory, Linux must decide whether to order the zonelists such that allocationsfall back to the same zone type on a different node, or to a different zonetype on the same node. This is an important consideration because some zones,such as DMA or DMA32, represent relatively scarce resources. Linux choosesa default Node ordered zonelist. This means it tries to fallback to other zonesfrom the same node before using remote nodes which are ordered by NUMA distance.

By default, Linux will attempt to satisfy memory allocation requests from thenode to which the CPU that executes the request is assigned. Specifically,Linux will attempt to allocate from the first node in the appropriate zonelistfor the node where the request originates. This is called “local allocation.”If the “local” node cannot satisfy the request, the kernel will examine othernodes’ zones in the selected zonelist looking for the first zone in the listthat can satisfy the request.

Local allocation will tend to keep subsequent access to the allocated memory“local” to the underlying physical resources and off the system interconnect--as long as the task on whose behalf the kernel allocated some memory does notlater migrate away from that memory. The Linux scheduler is aware of theNUMA topology of the platform--embodied in the “scheduling domains” datastructures [seeScheduler Domains]--and the schedulerattempts to minimize task migration to distant scheduling domains. However,the scheduler does not take a task’s NUMA footprint into account directly.Thus, under sufficient imbalance, tasks can migrate between nodes, remotefrom their initial node and kernel data structures.

System administrators and application designers can restrict a task’s migrationto improve NUMA locality using various CPU affinity command line interfaces,such as taskset(1) and numactl(1), and program interfaces such assched_setaffinity(2). Further, one can modify the kernel’s default localallocation behavior using Linux NUMA memory policy. [seeNUMA Memory Policy].

System administrators can restrict the CPUs and nodes’ memories that a non-privileged user can specify in the scheduling or NUMA commands and functionsusing control groups and CPUsets. [seeCPUSETS]

On architectures that do not hide memoryless nodes, Linux will include onlyzones [nodes] with memory in the zonelists. This means that for a memorylessnode the “local memory node”--the node of the first zone in CPU’s node’szonelist--will not be the node itself. Rather, it will be the node that thekernel selected as the nearest node with memory when it built the zonelists.So, default, local allocations will succeed with the kernel supplying theclosest available memory. This is a consequence of the same mechanism thatallows such allocations to fallback to other nearby nodes when a node thatdoes contain memory overflows.

Some kernel allocations do not want or cannot tolerate this allocation fallbackbehavior. Rather they want to be sure they get memory from the specified nodeor get notified that the node has no free memory. This is usually the case whena subsystem allocates per CPU memory resources, for example.

A typical model for making such an allocation is to obtain the node id of thenode to which the “current CPU” is attached using one of the kernel’snuma_node_id() orCPU_to_node() functions and then request memory from onlythe node id returned. When such an allocation fails, the requesting subsystemmay revert to its own fallback path. The slab kernel memory allocator is anexample of this. Or, the subsystem may choose to disable or not to enableitself on allocation failure. The kernel profiling subsystem is an example ofthis.

If the architecture supports--does not hide--memoryless nodes, then CPUsattached to memoryless nodes would always incur the fallback path overheador some subsystems would fail to initialize if they attempted to allocatedmemory exclusively from a node without memory. To support sucharchitectures transparently, kernel subsystems can use thenuma_mem_id()orcpu_to_mem() function to locate the “local memory node” for the calling orspecified CPU. Again, this is the same node from which default, local pageallocations will be attempted.