Inline Encryption

Background

Inline encryption hardware sits logically between memory and disk, and canen/decrypt data as it goes in/out of the disk. For each I/O request, softwarecan control exactly how the inline encryption hardware will en/decrypt the datain terms of key, algorithm, data unit size (the granularity of en/decryption),and data unit number (a value that determines the initialization vector(s)).

Some inline encryption hardware accepts all encryption parameters including rawkeys directly in low-level I/O requests. However, most inline encryptionhardware instead has a fixed number of “keyslots” and requires that the key,algorithm, and data unit size first be programmed into a keyslot. Eachlow-level I/O request then just contains a keyslot index and data unit number.

Note that inline encryption hardware is very different from traditional cryptoaccelerators, which are supported through the kernel crypto API. Traditionalcrypto accelerators operate on memory regions, whereas inline encryptionhardware operates on I/O requests. Thus, inline encryption hardware needs to bemanaged by the block layer, not the kernel crypto API.

Inline encryption hardware is also very different from “self-encrypting drives”,such as those based on the TCG Opal or ATA Security standards. Self-encryptingdrives don’t provide fine-grained control of encryption and provide no way toverify the correctness of the resulting ciphertext. Inline encryption hardwareprovides fine-grained control of encryption, including the choice of key andinitialization vector for each sector, and can be tested for correctness.

Objective

We want to support inline encryption in the kernel. To make testing easier, wealso want support for falling back to the kernel crypto API when actual inlineencryption hardware is absent. We also want inline encryption to work withlayered devices like device-mapper and loopback (i.e. we want to be able to usethe inline encryption hardware of the underlying devices if present, or elsefall back to crypto API en/decryption).

Constraints and notes

  • We need a way for upper layers (e.g. filesystems) to specify an encryptioncontext to use for en/decrypting a bio, and device drivers (e.g. UFSHCD) needto be able to use that encryption context when they process the request.Encryption contexts also introduce constraints on bio merging; the block layerneeds to be aware of these constraints.

  • Different inline encryption hardware has different supported algorithms,supported data unit sizes, maximum data unit numbers, etc. We call theseproperties the “crypto capabilities”. We need a way for device drivers toadvertise crypto capabilities to upper layers in a generic way.

  • Inline encryption hardware usually (but not always) requires that keys beprogrammed into keyslots before being used. Since programming keyslots may beslow and there may not be very many keyslots, we shouldn’t just program thekey for every I/O request, but rather keep track of which keys are in thekeyslots and reuse an already-programmed keyslot when possible.

  • Upper layers typically define a specific end-of-life for crypto keys, e.g.when an encrypted directory is locked or when a crypto mapping is torn down.At these times, keys are wiped from memory. We must provide a way for upperlayers to also evict keys from any keyslots they are present in.

  • When possible, device-mapper devices must be able to pass through the inlineencryption support of their underlying devices. However, it doesn’t makesense for device-mapper devices to have keyslots themselves.

Basic design

We introducestructblk_crypto_key to represent an inline encryption key andhow it will be used. This includes the type of the key (raw orhardware-wrapped); the actual bytes of the key; the size of the key; thealgorithm and data unit size the key will be used with; and the number of bytesneeded to represent the maximum data unit number the key will be used with.

We introducestructbio_crypt_ctx to represent an encryption context. Itcontains a data unit number and a pointer to a blk_crypto_key. We add pointersto a bio_crypt_ctx tostructbio andstructrequest; this allows usersof the block layer (e.g. filesystems) to provide an encryption context whencreating a bio and have it be passed down the stack for processing by the blocklayer and device drivers. Note that the encryption context doesn’t explicitlysay whether to encrypt or decrypt, as that is implicit from the direction of thebio; WRITE means encrypt, and READ means decrypt.

We also introducestructblk_crypto_profile to contain all generic inlineencryption-related state for a particular inline encryption device. Theblk_crypto_profile serves as the way that drivers for inline encryption hardwareadvertise their crypto capabilities and provide certain functions (e.g.,functions to program and evict keys) to upper layers. Each device driver thatwants to support inline encryption will construct a blk_crypto_profile, thenassociate it with the disk’s request_queue.

The blk_crypto_profile also manages the hardware’s keyslots, when applicable.This happens in the block layer, so that users of the block layer can justspecify encryption contexts and don’t need to know about keyslots at all, nor dodevice drivers need to care about most details of keyslot management.

Specifically, for each keyslot, the block layer (via the blk_crypto_profile)keeps track of which blk_crypto_key that keyslot contains (if any), and how manyin-flight I/O requests are using it. When the block layer creates astructrequest for a bio that has an encryption context, it grabs a keyslotthat already contains the key if possible. Otherwise it waits for an idlekeyslot (a keyslot that isn’t in-use by any I/O), then programs the key into theleast-recently-used idle keyslot using the function the device driver provided.In both cases, the resulting keyslot is stored in thecrypt_keyslot field ofthe request, where it is then accessible to device drivers and is released afterthe request completes.

structrequest also contains a pointer to the original bio_crypt_ctx.Requests can be built from multiple bios, and the block layer must take theencryption context into account when trying to merge bios and requests. For twobios/requests to be merged, they must have compatible encryption contexts: bothunencrypted, or both encrypted with the same key and contiguous data unitnumbers. Only the encryption context for the first bio in a request isretained, since the remaining bios have been verified to be merge-compatiblewith the first bio.

To make it possible for inline encryption to work with request_queue basedlayered devices, when a request is cloned, its encryption context is cloned aswell. When the cloned request is submitted, it is then processed as usual; thisincludes getting a keyslot from the clone’s target device if needed.

blk-crypto-fallback

It is desirable for the inline encryption support of upper layers (e.g.filesystems) to be testable without real inline encryption hardware, andlikewise for the block layer’s keyslot management logic. It is also desirableto allow upper layers to just always use inline encryption rather than have toimplement encryption in multiple ways.

Therefore, we also introduceblk-crypto-fallback, which is an implementationof inline encryption using the kernel crypto API. blk-crypto-fallback is builtinto the block layer, so it works on any block device without any special setup.Essentially, when a bio with an encryption context is submitted to ablock_device that doesn’t support that encryption context, the block layer willhandle en/decryption of the bio using blk-crypto-fallback.

For encryption, the data cannot be encrypted in-place, as callers usually relyon it being unmodified. Instead, blk-crypto-fallback allocates bounce pages,fills a new bio with those bounce pages, encrypts the data into those bouncepages, and submits that “bounce” bio. When the bounce bio completes,blk-crypto-fallback completes the original bio. If the original bio is toolarge, multiple bounce bios may be required; see the code for details.

For decryption, blk-crypto-fallback “wraps” the bio’s completion callback(bi_complete) and private data (bi_private) with its own, unsets thebio’s encryption context, then submits the bio. If the read completessuccessfully, blk-crypto-fallback restores the bio’s original completioncallback and private data, then decrypts the bio’s data in-place using thekernel crypto API. Decryption happens from a workqueue, as it may sleep.Afterwards, blk-crypto-fallback completes the bio.

In both cases, the bios that blk-crypto-fallback submits no longer have anencryption context. Therefore, lower layers only see standard unencrypted I/O.

blk-crypto-fallback also defines its own blk_crypto_profile and has its own“keyslots”; its keyslots containstructcrypto_skcipher objects. The reasonfor this is twofold. First, it allows the keyslot management logic to be testedwithout actual inline encryption hardware. Second, similar to actual inlineencryption hardware, the crypto API doesn’t accept keys directly in requests butrather requires that keys be set ahead of time, and setting keys can beexpensive; moreover, allocating a crypto_skcipher can’t happen on the I/O pathat all due to the locks it takes. Therefore, the concept of keyslots stillmakes sense for blk-crypto-fallback.

Note that regardless of whether real inline encryption hardware orblk-crypto-fallback is used, the ciphertext written to disk (and hence theon-disk format of data) will be the same (assuming that both the inlineencryption hardware’s implementation and the kernel crypto API’s implementationof the algorithm being used adhere to spec and function correctly).

blk-crypto-fallback is optional and is controlled by theCONFIG_BLK_INLINE_ENCRYPTION_FALLBACK kernel configuration option.

API presented to users of the block layer

blk_crypto_config_supported() allows users to check ahead of time whetherinline encryption with particular crypto settings will work on a particularblock_device -- either via hardware or via blk-crypto-fallback. This functiontakes in astructblk_crypto_config which is like blk_crypto_key, but omitsthe actual bytes of the key and instead just contains the algorithm, data unitsize, etc. This function can be useful if blk-crypto-fallback is disabled.

blk_crypto_init_key() allows users to initialize a blk_crypto_key.

Users must callblk_crypto_start_using_key() before actually starting to usea blk_crypto_key on a block_device (even ifblk_crypto_config_supported()was called earlier). This is needed to initialize blk-crypto-fallback if itwill be needed. This must not be called from the data path, as this may have toallocate resources, which may deadlock in that case.

Next, to attach an encryption context to a bio, users should callbio_crypt_set_ctx(). This function allocates a bio_crypt_ctx and attachesit to a bio, given the blk_crypto_key and the data unit number that will be usedfor en/decryption. Users don’t need to worry about freeing the bio_crypt_ctxlater, as that happens automatically when the bio is freed or reset.

Finally, when done using inline encryption with a blk_crypto_key on ablock_device, users must callblk_crypto_evict_key(). This ensures thatthe key is evicted from all keyslots it may be programmed into and unlinked fromany kernel data structures it may be linked into.

In summary, for users of the block layer, the lifecycle of a blk_crypto_key isas follows:

  1. blk_crypto_config_supported() (optional)

  2. blk_crypto_init_key()

  3. blk_crypto_start_using_key()

  4. bio_crypt_set_ctx() (potentially many times)

  5. blk_crypto_evict_key() (after all I/O has completed)

  6. Zeroize the blk_crypto_key (this has no dedicated function)

If a blk_crypto_key is being used on multiple block_devices, thenblk_crypto_config_supported() (if used),blk_crypto_start_using_key(),andblk_crypto_evict_key() must be called on each block_device.

API presented to device drivers

A device driver that wants to support inline encryption must set up ablk_crypto_profile in the request_queue of its device. To do this, it firstmust callblk_crypto_profile_init() (or its resource-managed variantdevm_blk_crypto_profile_init()), providing the number of keyslots.

Next, it must advertise its crypto capabilities by setting fields in theblk_crypto_profile, e.g.modes_supported andmax_dun_bytes_supported.

It then must set function pointers in thell_ops field of theblk_crypto_profile to tell upper layers how to control the inline encryptionhardware, e.g. how to program and evict keyslots. Most drivers will need toimplementkeyslot_program andkeyslot_evict. For details, see thecomments forstructblk_crypto_ll_ops.

Once the driver registers a blk_crypto_profile with a request_queue, I/Orequests the driver receives via that queue may have an encryption context. Allencryption contexts will be compatible with the crypto capabilities declared inthe blk_crypto_profile, so drivers don’t need to worry about handlingunsupported requests. Also, if a nonzero number of keyslots was declared in theblk_crypto_profile, then all I/O requests that have an encryption context willalso have a keyslot which was already programmed with the appropriate key.

If the driver implements runtime suspend and its blk_crypto_ll_ops don’t workwhile the device is runtime-suspended, then the driver must also set thedevfield of the blk_crypto_profile to point to thestructdevice that will beresumed before any of the low-level operations are called.

If there are situations where the inline encryption hardware loses the contentsof its keyslots, e.g. device resets, the driver must handle reprogramming thekeyslots. To do this, the driver may callblk_crypto_reprogram_all_keys().

Finally, if the driver usedblk_crypto_profile_init() instead ofdevm_blk_crypto_profile_init(), then it is responsible for callingblk_crypto_profile_destroy() when the crypto profile is no longer needed.

Layered Devices

Request queue based layered devices like dm-rq that wish to support inlineencryption need to create their own blk_crypto_profile for their request_queue,and expose whatever functionality they choose. When a layered device wants topass a clone of that request to another request_queue, blk-crypto willinitialize and prepare the clone as necessary.

Interaction between inline encryption and blk integrity

At the time of this patch, there is no real hardware that supports both thesefeatures. However, these features do interact with each other, and it’s notcompletely trivial to make them both work together properly. In particular,when a WRITE bio wants to use inline encryption on a device that supports bothfeatures, the bio will have an encryption context specified, after whichits integrity information is calculated (using the plaintext data, sincethe encryption will happen while data is being written), and the data andintegrity info is sent to the device. Obviously, the integrity info must beverified before the data is encrypted. After the data is encrypted, the devicemust not store the integrity info that it received with the plaintext datasince that might reveal information about the plaintext data. As such, it mustre-generate the integrity info from the ciphertext data and store that on diskinstead. Another issue with storing the integrity info of the plaintext data isthat it changes the on disk format depending on whether hardware inlineencryption support is present or the kernel crypto API fallback is used (sinceif the fallback is used, the device will receive the integrity info of theciphertext, not that of the plaintext).

Because there isn’t any real hardware yet, it seems prudent to assume thathardware implementations might not implement both features together correctly,and disallow the combination for now. Whenever a device supports integrity, thekernel will pretend that the device does not support hardware inline encryption(by setting the blk_crypto_profile in the request_queue of the device to NULL).When the crypto API fallback is enabled, this means that all bios with andencryption context will use the fallback, and IO will complete as usual. Whenthe fallback is disabled, a bio with an encryption context will be failed.

Hardware-wrapped keys

Motivation and threat model

Linux storage encryption (dm-crypt, fscrypt, eCryptfs, etc.) traditionallyrelies on the raw encryption key(s) being present in kernel memory so that theencryption can be performed. This traditionally isn’t seen as a problem becausethe key(s) won’t be present during an offline attack, which is the main type ofattack that storage encryption is intended to protect from.

However, there is an increasing desire to also protect users’ data from othertypes of attacks (to the extent possible), including:

  • Cold boot attacks, where an attacker with physical access to a system suddenlypowers it off, then immediately dumps the system memory to extract recentlyin-use encryption keys, then uses these keys to decrypt user data on-disk.

  • Online attacks where the attacker is able to read kernel memory without fullycompromising the system, followed by an offline attack where any extractedkeys can be used to decrypt user data on-disk. An example of such an onlineattack would be if the attacker is able to run some code on the system thatexploits a Meltdown-like vulnerability but is unable to escalate privileges.

  • Online attacks where the attacker fully compromises the system, but their dataexfiltration is significantly time-limited and/or bandwidth-limited, so inorder to completely exfiltrate the data they need to extract the encryptionkeys to use in a later offline attack.

Hardware-wrapped keys are a feature of inline encryption hardware that isdesigned to protect users’ data from the above attacks (to the extent possible),without introducing limitations such as a maximum number of keys.

Note that it is impossible tofully protect users’ data from these attacks.Even in the attacks where the attacker “just” gets read access to kernel memory,they can still extract any user data that is present in memory, includingplaintext pagecache pages of encrypted files. The focus here is just onprotecting the encryption keys, as those instantly give access toall userdata in any following offline attack, rather than just some of it (where whichdata is included in that “some” might not be controlled by the attacker).

Solution overview

Inline encryption hardware typically has “keyslots” into which software canprogram keys for the hardware to use; the contents of keyslots typically can’tbe read back by software. As such, the above security goals could be achievedif the kernel simply erased its copy of the key(s) after programming them intokeyslot(s) and thereafter only referred to them via keyslot number.

However, that naive approach runs into a couple problems:

  • It limits the number of unlocked keys to the number of keyslots, whichtypically is a small number. In cases where there is only one encryption keysystem-wide (e.g., a full-disk encryption key), that can be tolerable.However, in general there can be many logged-in users with many differentkeys, and/or many running applications with application-specific encryptedstorage areas. This is especially true if file-based encryption (e.g.fscrypt) is being used.

  • Inline crypto engines typically lose the contents of their keyslots if thestorage controller (usually UFS or eMMC) is reset. Resetting the storagecontroller is a standard error recovery procedure that is executed if certaintypes of storage errors occur, and such errors can occur at any time.Therefore, when inline crypto is being used, the operating system must alwaysbe ready to reprogram the keyslots without user intervention.

Thus, it is important for the kernel to still have a way to “remind” thehardware about a key, without actually having the raw key itself.

Somewhat less importantly, it is also desirable that the raw keys are nevervisible to software at all, even while being initially unlocked. This wouldensure that a read-only compromise of system memory will never allow a key to beextracted to be used off-system, even if it occurs when a key is being unlocked.

To solve all these problems, some vendors of inline encryption hardware havemade their hardware supporthardware-wrapped keys. Hardware-wrapped keysare encrypted keys that can only be unwrapped (decrypted) and used by hardware-- either by the inline encryption hardware itself, or by a dedicated hardwareblock that can directly provision keys to the inline encryption hardware.

(We refer to them as “hardware-wrapped keys” rather than simply “wrapped keys”to add some clarity in cases where there could be other types of wrapped keys,such as in file-based encryption. Key wrapping is a commonly used technique.)

The key which wraps (encrypts) hardware-wrapped keys is a hardware-internal keythat is never exposed to software; it is either a persistent key (a “long-termwrapping key”) or a per-boot key (an “ephemeral wrapping key”). The long-termwrapped form of the key is what is initially unlocked, but it is erased frommemory as soon as it is converted into an ephemerally-wrapped key. In-usehardware-wrapped keys are always ephemerally-wrapped, not long-term wrapped.

As inline encryption hardware can only be used to encrypt/decrypt data on-disk,the hardware also includes a level of indirection; it doesn’t use the unwrappedkey directly for inline encryption, but rather derives both an inline encryptionkey and a “software secret” from it. Software can use the “software secret” fortasks that can’t use the inline encryption hardware, such as filenamesencryption. The software secret is not protected from memory compromise.

Key hierarchy

Here is the key hierarchy for a hardware-wrapped key:

               Hardware-wrapped key                        |                        |                  <Hardware KDF>                        |          -----------------------------          |                           |Inline encryption key           Software secret

The components are:

  • Hardware-wrapped key: a key for the hardware’s KDF (Key DerivationFunction), in ephemerally-wrapped form. The key wrapping algorithm is ahardware implementation detail that doesn’t impact kernel operation, but astrong authenticated encryption algorithm such as AES-256-GCM is recommended.

  • Hardware KDF: a KDF (Key Derivation Function) which the hardware uses toderive subkeys after unwrapping the wrapped key. The hardware’s choice of KDFdoesn’t impact kernel operation, but it does need to be known for testingpurposes, and it’s also assumed to have at least a 256-bit security strength.All known hardware uses the SP800-108 KDF in Counter Mode with AES-256-CMAC,with a particular choice of labels and contexts; new hardware should use thisalready-vetted KDF.

  • Inline encryption key: a derived key which the hardware directly provisionsto a keyslot of the inline encryption hardware, without exposing it tosoftware. In all known hardware, this will always be an AES-256-XTS key.However, in principle other encryption algorithms could be supported too.Hardware must derive distinct subkeys for each supported encryption algorithm.

  • Software secret: a derived key which the hardware returns to software sothat software can use it for cryptographic tasks that can’t use inlineencryption. This value is cryptographically isolated from the inlineencryption key, i.e. knowing one doesn’t reveal the other. (The KDF ensuresthis.) Currently, the software secret is always 32 bytes and thus is suitablefor cryptographic applications that require up to a 256-bit security strength.Some use cases (e.g. full-disk encryption) won’t require the software secret.

Example: in the case of fscrypt, the fscrypt master key (the key that protects aparticular set of encrypted directories) is made hardware-wrapped. The inlineencryption key is used as the file contents encryption key, while the softwaresecret (rather than the master key directly) is used to key fscrypt’s KDF(HKDF-SHA512) to derive other subkeys such as filenames encryption keys.

Note that currently this design assumes a single inline encryption key perhardware-wrapped key, without any further key derivation. Thus, in the case offscrypt, currently hardware-wrapped keys are only compatible with the “inlineencryption optimized” settings, which use one file contents encryption key perencryption policy rather than one per file. This design could be extended tomake the hardware derive per-file keys using per-file nonces passed down thestorage stack, and in fact some hardware already supports this; future work isplanned to remove this limitation by adding the corresponding kernel support.

Kernel support

The inline encryption support of the kernel’s block layer (“blk-crypto”) hasbeen extended to support hardware-wrapped keys as an alternative to raw keys,when hardware support is available. This works in the following way:

  • Akey_types_supported field is added to the crypto capabilities instructblk_crypto_profile. This allows device drivers to declare thatthey support raw keys, hardware-wrapped keys, or both.

  • structblk_crypto_key can now contain a hardware-wrapped key as analternative to a raw key; akey_type field is added tostructblk_crypto_config to distinguish between the different key types.This allows users of blk-crypto to en/decrypt data using a hardware-wrappedkey in a way very similar to using a raw key.

  • A new methodblk_crypto_ll_ops::derive_sw_secret is added. Device driversthat support hardware-wrapped keys must implement this method. Users ofblk-crypto can callblk_crypto_derive_sw_secret() to access this method.

  • The programming and eviction of hardware-wrapped keys happens viablk_crypto_ll_ops::keyslot_program andblk_crypto_ll_ops::keyslot_evict, just like it does for raw keys. If adriver supports hardware-wrapped keys, then it must handle hardware-wrappedkeys being passed to these methods.

blk-crypto-fallback doesn’t support hardware-wrapped keys. Therefore,hardware-wrapped keys can only be used with actual inline encryption hardware.

All the above deals with hardware-wrapped keys in ephemerally-wrapped form only.To get such keys in the first place, new block device ioctls have been added toprovide a generic interface to creating and preparing such keys:

  • BLKCRYPTOIMPORTKEY converts a raw key to long-term wrapped form. It takesin a pointer to astructblk_crypto_import_key_arg. The caller must setraw_key_ptr andraw_key_size to the pointer and size (in bytes) of theraw key to import. On success,BLKCRYPTOIMPORTKEY returns 0 and writesthe resulting long-term wrapped key blob to the buffer pointed to bylt_key_ptr, which is of maximum sizelt_key_size. It also updateslt_key_size to be the actual size of the key. On failure, it returns -1and sets errno. An errno ofEOPNOTSUPP indicates that the block devicedoes not support hardware-wrapped keys. An errno ofEOVERFLOW indicatesthat the output buffer did not have enough space for the key blob.

  • BLKCRYPTOGENERATEKEY is likeBLKCRYPTOIMPORTKEY, but it has thehardware generate the key instead of importing one. It takes in a pointer toastructblk_crypto_generate_key_arg.

  • BLKCRYPTOPREPAREKEY converts a key from long-term wrapped form toephemerally-wrapped form. It takes in a pointer to astructblk_crypto_prepare_key_arg. The caller must setlt_key_ptr andlt_key_size to the pointer and size (in bytes) of the long-term wrappedkey blob to convert. On success,BLKCRYPTOPREPAREKEY returns 0 and writesthe resulting ephemerally-wrapped key blob to the buffer pointed to byeph_key_ptr, which is of maximum sizeeph_key_size. It also updateseph_key_size to be the actual size of the key. On failure, it returns -1and sets errno. Errno values ofEOPNOTSUPP andEOVERFLOW mean thesame as they do forBLKCRYPTOIMPORTKEY. An errno ofEBADMSG indicatesthat the long-term wrapped key is invalid.

Userspace needs to use eitherBLKCRYPTOIMPORTKEY orBLKCRYPTOGENERATEKEYonce to create a key, and thenBLKCRYPTOPREPAREKEY each time the key isunlocked and added to the kernel. Note that these ioctls have no relevance forraw keys; they are only for hardware-wrapped keys.

Testability

Both the hardware KDF and the inline encryption itself are well-definedalgorithms that don’t depend on any secrets other than the unwrapped key.Therefore, if the unwrapped key is known to software, these algorithms can bereproduced in software in order to verify the ciphertext that is written to diskby the inline encryption hardware.

However, the unwrapped key will only be known to software for testing if the“import” functionality is used. Proper testing is not possible in the“generate” case where the hardware generates the key itself. The correctoperation of the “generate” mode thus relies on the security and correctness ofthe hardware RNG and its use to generate the key, as well as the testing of the“import” mode as that should cover all parts other than the key generation.

For an example of a test that verifies the ciphertext written to disk in the“import” mode, see the fscrypt hardware-wrapped key tests in xfstests, orAndroid’s vts_kernel_encryption_test.