FAQ¶

• Where’s the value?

When a workload starts swapping, performance falls through the floor.Frontswap significantly increases performance in many such workloads byproviding a clean, dynamic interface to read and write swap pages to“transcendent memory” that is otherwise not directly addressable to the kernel.This interface is ideal when data is transformed to a different formand size (such as with compression) or secretly moved (as might beuseful for write-balancing for some RAM-like devices). Swap pages (andevicted page-cache pages) are a great use for this kind of slower-than-RAM-but-much-faster-than-disk “pseudo-RAM device” and the frontswap (andcleancache) interface to transcendent memory provides a nice way to readand write – and indirectly “name” – the pages.

Frontswap – and cleancache – with a fairly small impact on the kernel,provides a huge amount of flexibility for more dynamic, flexible RAMutilization in various system configurations:

In the single kernel case, aka “zcache”, pages are compressed andstored in local memory, thus increasing the total anonymous pagesthat can be safely kept in RAM. Zcache essentially trades off CPUcycles used in compression/decompression for better memory utilization.Benchmarks have shown little or no impact when memory pressure islow while providing a significant performance improvement (25%+)on some workloads under high memory pressure.

“RAMster” builds on zcache by adding “peer-to-peer” transcendent memorysupport for clustered systems. Frontswap pages are locally compressedas in zcache, but then “remotified” to another system’s RAM. Thisallows RAM to be dynamically load-balanced back-and-forth as needed,i.e. when system A is overcommitted, it can swap to system B, andvice versa. RAMster can also be configured as a memory server somany servers in a cluster can swap, dynamically as needed, to a singleserver configured with a large amount of RAM… without pre-configuringhow much of the RAM is available for each of the clients!

In the virtual case, the whole point of virtualization is to statisticallymultiplex physical resources across the varying demands of multiplevirtual machines. This is really hard to do with RAM and efforts to doit well with no kernel changes have essentially failed (except in somewell-publicized special-case workloads).Specifically, the Xen Transcendent Memory backend allows otherwise“fallow” hypervisor-owned RAM to not only be “time-shared” between multiplevirtual machines, but the pages can be compressed and deduplicated tooptimize RAM utilization. And when guest OS’s are induced to surrenderunderutilized RAM (e.g. with “selfballooning”), sudden unexpectedmemory pressure may result in swapping; frontswap allows those pagesto be swapped to and from hypervisor RAM (if overall host system memoryconditions allow), thus mitigating the potentially awful performance impactof unplanned swapping.

A KVM implementation is underway and has been RFC’ed to lkml. And,using frontswap, investigation is also underway on the use of NVM asa memory extension technology.

• Sure there may be performance advantages in some situations, butwhat’s the space/time overhead of frontswap?

If CONFIG_FRONTSWAP is disabled, every frontswap hook compiles intonothingness and the only overhead is a few extra bytes per swapon’edswap device. If CONFIG_FRONTSWAP is enabled but no frontswap “backend”registers, there is one extra global variable compared to zero forevery swap page read or written. If CONFIG_FRONTSWAP is enabledAND a frontswap backend registers AND the backend fails every “store”request (i.e. provides no memory despite claiming it might),CPU overhead is still negligible – and since every frontswap failprecedes a swap page write-to-disk, the system is highly likelyto be I/O bound and using a small fraction of a percent of a CPUwill be irrelevant anyway.

As for space, if CONFIG_FRONTSWAP is enabled AND a frontswap backendregisters, one bit is allocated for every swap page for every swapdevice that is swapon’d. This is added to the EIGHT bits (whichwas sixteen until about 2.6.34) that the kernel already allocatesfor every swap page for every swap device that is swapon’d. (HughDickins has observed that frontswap could probably steal one ofthe existing eight bits, but let’s worry about that minor optimizationlater.) For very large swap disks (which are rare) on a standard4K pagesize, this is 1MB per 32GB swap.

When swap pages are stored in transcendent memory instead of writtenout to disk, there is a side effect that this may create more memorypressure that can potentially outweigh the other advantages. Abackend, such as zcache, must implement policies to carefully (butdynamically) manage memory limits to ensure this doesn’t happen.

• OK, how about a quick overview of what this frontswap patch doesin terms that a kernel hacker can grok?

Let’s assume that a frontswap “backend” has registered duringkernel initialization; this registration indicates that thisfrontswap backend has access to some “memory” that is not directlyaccessible by the kernel. Exactly how much memory it provides isentirely dynamic and random.

Whenever a swap-device is swapon’d frontswap_init() is called,passing the swap device number (aka “type”) as a parameter.This notifies frontswap to expect attempts to “store” swap pagesassociated with that number.

Whenever the swap subsystem is readying a page to write to a swapdevice (c.f swap_writepage()), frontswap_store is called. Frontswapconsults with the frontswap backend and if the backend says it does NOThave room, frontswap_store returns -1 and the kernel swaps the pageto the swap device as normal. Note that the response from the frontswapbackend is unpredictable to the kernel; it may choose to never accept apage, it could accept every ninth page, or it might accept everypage. But if the backend does accept a page, the data from the pagehas already been copied and associated with the type and offset,and the backend guarantees the persistence of the data. In this case,frontswap sets a bit in the “frontswap_map” for the swap devicecorresponding to the page offset on the swap device to which it wouldotherwise have written the data.

When the swap subsystem needs to swap-in a page (swap_readpage()),it first calls frontswap_load() which checks the frontswap_map tosee if the page was earlier accepted by the frontswap backend. Ifit was, the page of data is filled from the frontswap backend andthe swap-in is complete. If not, the normal swap-in code isexecuted to obtain the page of data from the real swap device.

So every time the frontswap backend accepts a page, a swap device readand (potentially) a swap device write are replaced by a “frontswap backendstore” and (possibly) a “frontswap backend loads”, which are presumably muchfaster.

• Can’t frontswap be configured as a “special” swap device that isjust higher priority than any real swap device (e.g. like zswap,or maybe swap-over-nbd/NFS)?

No. First, the existing swap subsystem doesn’t allow for any kind ofswap hierarchy. Perhaps it could be rewritten to accommodate a hierarchy,but this would require fairly drastic changes. Even if it wererewritten, the existing swap subsystem uses the block I/O layer whichassumes a swap device is fixed size and any page in it is linearlyaddressable. Frontswap barely touches the existing swap subsystem,and works around the constraints of the block I/O subsystem to providea great deal of flexibility and dynamicity.

For example, the acceptance of any swap page by the frontswap backend isentirely unpredictable. This is critical to the definition of frontswapbackends because it grants completely dynamic discretion to thebackend. In zcache, one cannot know a priori how compressible a page is.“Poorly” compressible pages can be rejected, and “poorly” can itself bedefined dynamically depending on current memory constraints.

Further, frontswap is entirely synchronous whereas a real swapdevice is, by definition, asynchronous and uses block I/O. Theblock I/O layer is not only unnecessary, but may perform “optimizations”that are inappropriate for a RAM-oriented device including delayingthe write of some pages for a significant amount of time. Synchrony isrequired to ensure the dynamicity of the backend and to avoid thorny raceconditions that would unnecessarily and greatly complicate frontswapand/or the block I/O subsystem. That said, only the initial “store”and “load” operations need be synchronous. A separate asynchronous threadis free to manipulate the pages stored by frontswap. For example,the “remotification” thread in RAMster uses standard asynchronouskernel sockets to move compressed frontswap pages to a remote machine.Similarly, a KVM guest-side implementation could do in-guest compressionand use “batched” hypercalls.

In a virtualized environment, the dynamicity allows the hypervisor(or host OS) to do “intelligent overcommit”. For example, it canchoose to accept pages only until host-swapping might be imminent,then force guests to do their own swapping.

There is a downside to the transcendent memory specifications forfrontswap: Since any “store” might fail, there must always be a realslot on a real swap device to swap the page. Thus frontswap must beimplemented as a “shadow” to every swapon’d device with the potentialcapability of holding every page that the swap device might have heldand the possibility that it might hold no pages at all. This meansthat frontswap cannot contain more pages than the total of swapon’dswap devices. For example, if NO swap device is configured on someinstallation, frontswap is useless. Swapless portable devicescan still use frontswap but a backend for such devices must configuresome kind of “ghost” swap device and ensure that it is never used.

• Why this weird definition about “duplicate stores”? If a pagehas been previously successfully stored, can’t it always besuccessfully overwritten?

Nearly always it can, but no, sometimes it cannot. Consider an examplewhere data is compressed and the original 4K page has been compressedto 1K. Now an attempt is made to overwrite the page with data thatis non-compressible and so would take the entire 4K. But the backendhas no more space. In this case, the store must be rejected. Wheneverfrontswap rejects a store that would overwrite, it also must invalidatethe old data and ensure that it is no longer accessible. Since theswap subsystem then writes the new data to the read swap device,this is the correct course of action to ensure coherency.

• What is frontswap_shrink for?

When the (non-frontswap) swap subsystem swaps out a page to a realswap device, that page is only taking up low-value pre-allocated diskspace. But if frontswap has placed a page in transcendent memory, thatpage may be taking up valuable real estate. The frontswap_shrinkroutine allows code outside of the swap subsystem to force pages outof the memory managed by frontswap and back into kernel-addressable memory.For example, in RAMster, a “suction driver” thread will attemptto “repatriate” pages sent to a remote machine back to the local machine;this is driven using the frontswap_shrink mechanism when memory pressuresubsides.

• Why does the frontswap patch create the new include file swapfile.h?

The frontswap code depends on some swap-subsystem-internal datastructures that have, over the years, moved back and forth betweenstatic and global. This seemed a reasonable compromise: Definethem as global but declare them in a new include file that isn’tincluded by the large number of source files that include swap.h.

Dan Magenheimer, last updated April 9, 2012