Incomputer architecture, thememory hierarchy separatescomputer storage into a hierarchy based onresponse time. Since response time,complexity, andcapacity are related, the levels may also be distinguished by theirperformance and controlling technologies.^[1] Memory hierarchy affects performance in computer architectural design, algorithm predictions, and lower levelprogramming constructs involvinglocality of reference.

Designing for high performance requires considering the restrictions of the memory hierarchy, i.e. the size and capabilities of each component. Each of the various components can be viewed as part of a hierarchy of memories(m₁,m₂, ...,m_n) in which each memberm_i is typically smaller and faster than the next highest memberm_i+1 of the hierarchy. To limit waiting by higher levels, a lower level will respond by filling a buffer and then signaling for activating the transfer.

There are four major storage levels.^[1]

• Internal – processor registers andcache.
• Main – the systemRAM and controller cards.
• On-line mass storage – secondary storage.
• Off-line bulk storage – tertiary and off-line storage.

This is a general memory hierarchy structuring. Many other structures are useful. For example, a paging algorithm may be considered as a level forvirtual memory when designing acomputer architecture, and one can include a level ofnearline storage between online and offline storage.

• Adding complexity slows thememory hierarchy.^[2]
• CMOx memory technology stretches the flash space in the memory hierarchy^[3]
• One of the main ways to increase system performance is minimising how far down the memory hierarchy one has to go to manipulate data.^[4]
• Latency and bandwidth are two metrics associated with caches. Neither of them is uniform, but is specific to a particular component of the memory hierarchy.^[5]
• Predicting where in the memory hierarchy the data resides is difficult.^[5]
• The location in the memory hierarchy dictates the time required for the prefetch to occur.^[5]

The number of levels in the memory hierarchy and the performance at each level has increased over time. The type of memory or storage components also change historically.^[6]

Some CPUs include additional levels of cache between L3 and memory. For example, theHaswell microarchitecture includes an L4 cache of 128 MB on mobile units.^[13][14]

The lower levels of the hierarchy – from mass storage downwards – are also known astiered storage. The formal distinction between online, nearline, and offline storage is:^[15]

• Online storage is immediately available for I/O.
• Nearline storage is not immediately available, but can be made online quickly without human intervention.
• Offline storage is not immediately available, and requires some human intervention to bring online.

For example, always-on spinning disks are online, while spinning disks that spin down, such as massive arrays of idle disk (MAID), are nearline. Removable media such as tape cartridges that can be automatically loaded, as in atape library, are nearline, while cartridges that must be manually loaded are offline.

Most modernCPUs are so fast that, for most program workloads, thebottleneck is thelocality of reference of memory accesses and the efficiency of thecaching and memory transfer between different levels of the hierarchy^{[citation needed]}. As a result, the CPU spends much of its time idling, waiting for memory I/O to complete. This is sometimes called thespace cost, as a larger memory object is more likely to overflow a small and fast level and require use of a larger, slower level. The resulting load on memory use is known aspressure (respectivelyregister pressure,cache pressure, and (main)memory pressure). Terms for data being missing from a higher level and needing to be fetched from a lower level are, respectively:register spilling (due toregister pressure: register to cache),cache miss (cache to main memory), and (hard)page fault (real main memory tovirtual memory, i.e. mass storage, commonly referred to asdisk regardless of the actual mass storage technology used).

Modernprogramming languages mainly assume two levels of memory, main (working) memory and mass storage. The exception is the relatively low-levelassembly language and in theinline assemblers of higher-level languages such asC. Taking optimal advantage of the memory hierarchy requires the cooperation of programmers, hardware, and compilers (as well as underlying support from the operating system):

• Programmers are responsible for moving data between disk and memory through file I/O.
• Hardware is responsible for moving data between memory and caches.
• Optimizing compilers are responsible for generating code that, when executed, will cause the hardware to use caches and registers efficiently.

Many programmers assume one level of memory. This works fine until the application hits a performance wall. At that point, the programmer needs to change the code's memory access patterns to that it works well with cache resources. A classic illustration of the effect of locality and caching is in the form of changing the order of iterating a three-dimensional array.Computer Systems: A Programmer's Perspective is a classic textbook that deals with this aspect of systems programming.^[16]

• Cache hierarchy
• Use of spatial and temporal locality: hierarchical memory
• Buffer vs. cache
• Cache hierarchy in a modern processor
• Memory wall
• Computer memory
• Hierarchical storage management
• Cloud storage
• Memory access pattern
• Communication-avoiding algorithm

• Computer architecture
• Computer data storage
• Hierarchy

• CS1 errors: missing periodical
• Articles with short description
• Short description is different from Wikidata
• All articles with unsourced statements
• Articles with unsourced statements from September 2009