Trusting the developers, not infrastructure¶

Ever since the 2011 compromise of core kernel.org systems, the mainoperating principle of the Kernel Archives project has been to assumethat any part of the infrastructure can be compromised at any time. Forthis reason, the administrators have taken deliberate steps to emphasizethat trust must always be placed with developers and never with the codehosting infrastructure, regardless of how good the security practicesfor the latter may be.

The above guiding principle is the reason why this guide is needed. Wewant to make sure that by placing trust into developers we do not merelyshift the blame for potential future security incidents to someone else.The goal is to provide a set of guidelines developers can use to createa secure working environment and safeguard the PGP keys used toestablish the integrity of the Linux kernel itself.

Use GnuPG 2.4 or later¶

Your distro should already have GnuPG installed by default, you justneed to verify that you are using a reasonably recent version of it.To check, run:

$ gpg --version | head -n1

If you have version 2.4 or above, then you are good to go. If you havean earlier version, then you are using a release of GnuPG that is nolonger maintained and some commands from this guide may not work.

# set to 30 minutes for regular ttl, and 2 hours for max ttldefault-cache-ttl 1800max-cache-ttl 7200

Understanding PGP Subkeys¶

A PGP key rarely consists of a single keypair -- usually it is acollection of independent subkeys that can be used for differentpurposes based on their capabilities, assigned at their creation time.PGP defines four capabilities that a key can have:

• [S] keys can be used for signing

• [E] keys can be used for encryption

• [A] keys can be used for authentication

• [C] keys can be used for certifying other keys

The key with the[C] capability is often called the “master” key,but this terminology is misleading because it implies that the Certifykey can be used in place of any of other subkey on the same chain (likea physical “master key” can be used to open locks made for other keys).Since this is not the case, this guide will refer to it as “the Certifykey” to avoid any ambiguity.

It is critical to fully understand the following:

1. All subkeys are fully independent from each other. If you lose aprivate subkey, it cannot be restored or recreated from any otherprivate key on your chain.

2. With the exception of the Certify key, there can be multiple subkeyswith identical capabilities (e.g. you can have 2 valid encryptionsubkeys, 3 valid signing subkeys, but only one valid certificationsubkey). All subkeys are fully independent -- a message encrypted toone[E] subkey cannot be decrypted with any other[E] subkeyyou may also have.

3. A single subkey may have multiple capabilities (e.g. your[C] keycan also be your[S] key).

The key carrying the[C] (certify) capability is the only key thatcan be used to indicate relationship with other keys. Only the[C]key can be used to:

• add or revoke other keys (subkeys) with S/E/A capabilities

• add, change or revoke identities (uids) associated with the key

• add or change the expiration date on itself or any subkey

• sign other people’s keys for web of trust purposes

By default, GnuPG creates the following when generating new keys:

• One subkey carrying both Certify and Sign capabilities ([SC])

• A separate subkey with the Encryption capability ([E])

If you used the default parameters when generating your key, then thatis what you will have. You can verify by runninggpg--list-secret-keys,for example:

sec   ed25519 2022-12-20 [SC] [expires: 2024-12-19]      000000000000000000000000AAAABBBBCCCCDDDDuid           [ultimate] Alice Dev <adev@kernel.org>ssb   cv25519 2022-12-20 [E] [expires: 2024-12-19]

The long line under thesec entry is your key fingerprint --whenever you see[fpr] in the examples below, that 40-characterstring is what it refers to.

Ensure your passphrase is strong¶

GnuPG uses passphrases to encrypt your private keys before storing them ondisk. This way, even if your.gnupg directory is leaked or stolen inits entirety, the attackers cannot use your private keys without firstobtaining the passphrase to decrypt them.

It is absolutely essential that your private keys are protected by astrong passphrase. To set it or change it, use:

$ gpg --change-passphrase [fpr]

Create a separate Signing subkey¶

Our goal is to protect your Certify key by moving it to offline media,so if you only have a combined[SC] key, then you should create aseparate signing subkey:

$ gpg --quick-addkey [fpr] ed25519 sign

Back up your Certify key for disaster recovery¶

The more signatures you have on your PGP key from other developers, themore reasons you have to create a backup version that lives on somethingother than digital media, for disaster recovery reasons.

A good way to create a printable hardcopy of your private key is byusing thepaperkey software written for this very purpose. Seemanpaperkey for more details on the output format and its benefits overother solutions. Paperkey should already be packaged for mostdistributions.

Run the following command to create a hardcopy backup of your privatekey:

$ gpg --export-secret-key [fpr] | paperkey -o /tmp/key-backup.txt

Print out that file, then take a pen and write your passphrase on themargin of the paper.This is strongly recommended because the keyprintout is still encrypted with that passphrase, and if you ever changeit you will not remember what it used to be when you had created thebackup --guaranteed.

Put the resulting printout and the hand-written passphrase into an envelopeand store in a secure and well-protected place, preferably away from yourhome, such as your bank vault.

Back up your whole GnuPG directory¶

It is important to have a readily available backup of your PGP keysshould you need to recover them. This is different from thedisaster-level preparedness we did withpaperkey. You will also relyon these external copies whenever you need to use your Certify key --such as when making changes to your own key or signing other people’skeys after conferences and summits.

Start by getting an external media card (preferably two!) that you willuse for backup purposes. You will need to create an encrypted partitionon this device using LUKS -- refer to your distro’s documentation on howto accomplish this.

For the encryption passphrase, you can use the same one as on yourPGP key.

Once the encryption process is over, re-insert your device and make sureit gets properly mounted. Copy your entire.gnupg directory over tothe encrypted storage:

$ cp -a ~/.gnupg /media/disk/foo/gnupg-backup

You should now test to make sure everything still works:

$ gpg --homedir=/media/disk/foo/gnupg-backup --list-key [fpr]

If you don’t get any errors, then you should be good to go. Unmount thedevice, distinctly label it so you don’t overwrite it by accident, andput in a safe place -- but not too far away, because you’ll need to useit every now and again for things like editing identities, adding orrevoking subkeys, or signing other people’s keys.

Remove the Certify key from your homedir¶

The files in our home directory are not as well protected as we like tothink. They can be leaked or stolen via many different means:

• by accident when making quick homedir copies to set up a new workstation

• by systems administrator negligence or malice

• via poorly secured backups

• via malware in desktop apps (browsers, pdf viewers, etc)

• via coercion when crossing international borders

Protecting your key with a good passphrase greatly helps reduce the riskof any of the above, but passphrases can be discovered via keyloggers,shoulder-surfing, or any number of other means. For this reason, therecommended setup is to remove your Certify key from your home directoryand store it on offline storage.

First, identify the “keygrip” of your Certify key:

$ gpg --with-keygrip --list-key [fpr]

The output will be something like this:

pub   ed25519 2022-12-20 [SC] [expires: 2022-12-19]      000000000000000000000000AAAABBBBCCCCDDDD      Keygrip = 1111000000000000000000000000000000000000uid           [ultimate] Alice Dev <adev@kernel.org>sub   cv25519 2022-12-20 [E] [expires: 2022-12-19]      Keygrip = 2222000000000000000000000000000000000000sub   ed25519 2022-12-20 [S]      Keygrip = 3333000000000000000000000000000000000000

Find the keygrip entry that is beneath thepub line (right under theCertify key fingerprint). This will correspond directly to a file in your~/.gnupg directory:

$ cd ~/.gnupg/private-keys-v1.d$ ls1111000000000000000000000000000000000000.key2222000000000000000000000000000000000000.key3333000000000000000000000000000000000000.key

It is sufficient to remove the .key file that corresponds to the Certifykey keygrip:

$ cd ~/.gnupg/private-keys-v1.d$ rm 1111000000000000000000000000000000000000.key

Now, if you issue the--list-secret-keys command, it will show thatthe Certify key is missing (the# indicates it is not available):

$ gpg --list-secret-keyssec#  ed25519 2022-12-20 [SC] [expires: 2024-12-19]      000000000000000000000000AAAABBBBCCCCDDDDuid           [ultimate] Alice Dev <adev@kernel.org>ssb   cv25519 2022-12-20 [E] [expires: 2024-12-19]ssb   ed25519 2022-12-20 [S]

You should also remove anysecring.gpg files in the~/.gnupgdirectory, which may be left over from previous versions of GnuPG.

The benefits of smartcards¶

A smartcard contains a cryptographic chip that is capable of storingprivate keys and performing crypto operations directly on the carditself. Because the key contents never leave the smartcard, theoperating system of the computer into which you plug in the hardwaredevice is not able to retrieve the private keys themselves. This is verydifferent from the encrypted media storage device we used earlier forbackup purposes -- while that device is plugged in and mounted, theoperating system is able to access the private key contents.

Using external encrypted media is not a substitute to having asmartcard-capable device.

Available smartcard devices¶

Unless all your laptops and workstations have smartcard readers, theeasiest is to get a specialized USB device that implements smartcardfunctionality. There are several options available:

• Nitrokey Start: Open hardware and Free Software, based on FSIJapan’sGnuk. One of the cheapest options, but offers fewestsecurity features (such as resistance to tampering or someside-channel attacks).

• Nitrokey 3: Similar to the Nitrokey Start, but moretamper-resistant and offers more security features and USBform-factors. Supports ECC cryptography (ED25519 and NISTP).

• Yubikey 5: proprietary hardware and software, but cheaper thanNitrokey with a similar set of features. Supports ECC cryptography(ED25519 and NISTP).

Your choice will depend on cost, shipping availability in yourgeographical region, and open/proprietary hardware considerations.

Configure your smartcard device¶

Your smartcard device should Just Work (TM) the moment you plug it intoany modern Linux workstation. You can verify it by running:

$ gpg --card-status

If you see full smartcard details, then you are good to go.Unfortunately, troubleshooting all possible reasons why things may notbe working for you is way beyond the scope of this guide. If you arehaving trouble getting the card to work with GnuPG, please seek help viausual support channels.

To configure your smartcard, you will need to use the GnuPG menu system, asthere are no convenient command-line switches:

$ gpg --card-edit[...omitted...]gpg/card> adminAdmin commands are allowedgpg/card> passwd

You should set the user PIN (1), Admin PIN (3), and the Reset Code (4).Please make sure to record and store these in a safe place -- especiallythe Admin PIN and the Reset Code (which allows you to completely wipethe smartcard). You so rarely need to use the Admin PIN, that you willinevitably forget what it is if you do not record it.

Getting back to the main card menu, you can also set other values (suchas name, gender, login data, etc), but it’s not necessary and willadditionally leak information about your smartcard should you lose it.

Move the subkeys to your smartcard¶

Exit the card menu (using “q”) and save all changes. Next, let’s moveyour subkeys onto the smartcard. You will need both your PGP keypassphrase and the admin PIN of the card for most operations:

$ gpg --edit-key [fpr]Secret subkeys are available.pub  ed25519/AAAABBBBCCCCDDDD     created: 2022-12-20  expires: 2024-12-19  usage: SC     trust: ultimate      validity: ultimatessb  cv25519/1111222233334444     created: 2022-12-20  expires: never       usage: Essb  ed25519/5555666677778888     created: 2017-12-07  expires: never       usage: S[ultimate] (1). Alice Dev <adev@kernel.org>gpg>

Using--edit-key puts us into the menu mode again, and you willnotice that the key listing is a little different. From here on, allcommands are done from inside this menu mode, as indicated bygpg>.

First, let’s select the key we’ll be putting onto the card -- you dothis by typingkey1 (it’s the first one in the listing, the[E]subkey):

gpg> key 1

In the output, you should now seessb* on the[E] key. The*indicates which key is currently “selected.” It works as atoggle,meaning that if you typekey1 again, the* will disappear andthe key will not be selected any more.

Now, let’s move that key onto the smartcard:

gpg> keytocardPlease select where to store the key:   (2) Encryption keyYour selection? 2

Since it’s our[E] key, it makes sense to put it into the Encryptionslot. When you submit your selection, you will be prompted first foryour PGP key passphrase, and then for the admin PIN. If the commandreturns without an error, your key has been moved.

Important: Now typekey1 again to unselect the first key, andkey2 to select the[S] key:

gpg> key 1gpg> key 2gpg> keytocardPlease select where to store the key:   (1) Signature key   (3) Authentication keyYour selection? 1

You can use the[S] key both for Signature and Authentication, butwe want to make sure it’s in the Signature slot, so choose (1). Onceagain, if your command returns without an error, then the operation wassuccessful:

gpg> qSave changes? (y/N) y

Saving the changes will delete the keys you moved to the card from yourhome directory (but it’s okay, because we have them in our backupsshould we need to do this again for a replacement smartcard).

$ gpg --list-secret-keyssec#  ed25519 2022-12-20 [SC] [expires: 2024-12-19]      000000000000000000000000AAAABBBBCCCCDDDDuid           [ultimate] Alice Dev <adev@kernel.org>ssb>  cv25519 2022-12-20 [E] [expires: 2024-12-19]ssb>  ed25519 2022-12-20 [S]

$ cd ~/.gnupg/private-keys-v1.d$ strings *.key | grep 'private-key'

$ echo "Hello world" | gpg --clearsign > /tmp/test.asc$ gpg --verify /tmp/test.asc

Other common GnuPG operations¶

Here is a quick reference for some common operations you’ll need to dowith your PGP key.

$ export GNUPGHOME=/media/disk/foo/gnupg-backup$ gpg --list-secret-keys

$ gpg --quick-set-expire [fpr] 1y

$ gpg --quick-set-expire [fpr] 2038-07-01

$ gpg --send-key [fpr]

$ gpg --export | gpg --homedir ~/.gnupg --import$ unset GNUPGHOME

Configure git to use your PGP key¶

If you only have one secret key in your keyring, then you don’t reallyneed to do anything extra, as it becomes your default key. However, ifyou happen to have multiple secret keys, you can tell git which keyshould be used ([fpr] is the fingerprint of your key):

$ git config --global user.signingKey [fpr]

How to work with signed tags¶

To create a signed tag, pass the-s switch to the tag command:

$ git tag -s [tagname]

Our recommendation is to always sign git tags, as this allows otherdevelopers to ensure that the git repository they are pulling from hasnot been maliciously altered.

$ git verify-tag [tagname]

$ git pull [url] tags/sometag

Merge tag 'sometag' of [url][Tag message]# gpg: Signature made [...]# gpg: Good signature from [...]

$ git config --global tag.forceSignAnnotated true

How to work with signed commits¶

It is also possible to create signed commits, but they have limitedusefulness in Linux kernel development. The kernel contribution workflowrelies on sending in patches, and converting commits to patches does notpreserve git commit signatures. Furthermore, when rebasing your ownrepository on a newer upstream, PGP commit signatures will end updiscarded. For this reason, most kernel developers don’t bother signingtheir commits and will ignore signed commits in any externalrepositories that they rely upon in their work.

That said, if you have your working git tree publicly available at somegit hosting service (kernel.org, infradead.org, ozlabs.org, or others),then the recommendation is that you sign all your git commits even ifupstream developers do not directly benefit from this practice.

We recommend this for the following reasons:

1. Should there ever be a need to perform code forensics or track codeprovenance, even externally maintained trees carrying PGP commitsignatures will be valuable for such purposes.

2. If you ever need to re-clone your local repository (for example,after reinstalling your system), this lets you verify the repositoryintegrity before resuming your work.

3. If someone needs to cherry-pick your commits, this allows them toquickly verify their integrity before applying them.

$ git commit -S

git config --global commit.gpgSign true

How to work with signed patches¶

It is possible to use your PGP key to sign patches sent to kerneldeveloper mailing lists. Since existing email signature mechanisms(PGP-Mime or PGP-inline) tend to cause problems with regular codereview tasks, you should use the tool kernel.org created for thispurpose that puts cryptographic attestation signatures into messageheaders (a-la DKIM):

• Patatt Patch Attestation

patatt install-hook

$ b4 am 20220720205013.890942-1-broonie@kernel.org[...]Checking attestation on all messages, may take a moment...---  ✓ [PATCH v1 1/3] kselftest/arm64: Correct buffer allocation for SVE Z registers  ✓ [PATCH v1 2/3] arm64/sve: Document our actual ABI for clearing registers on syscall  ✓ [PATCH v1 3/3] kselftest/arm64: Enforce actual ABI for SVE syscalls  ---  ✓ Signed: openpgp/broonie@kernel.org  ✓ Signed: DKIM/kernel.org

Configure auto-key-retrieval using WKD and DANE¶

If you are not already someone with an extensive collection of otherdevelopers’ public keys, then you can jumpstart your keyring by relyingon key auto-discovery and auto-retrieval. GnuPG can piggyback on otherdelegated trust technologies, namely DNSSEC and TLS, to get you going ifthe prospect of starting your own Web of Trust from scratch is toodaunting.

Add the following to your~/.gnupg/gpg.conf:

auto-key-locate wkd,dane,localauto-key-retrieve

DNS-Based Authentication of Named Entities (“DANE”) is a method forpublishing public keys in DNS and securing them using DNSSEC signedzones. Web Key Directory (“WKD”) is the alternative method that useshttps lookups for the same purpose. When using either DANE or WKD forlooking up public keys, GnuPG will validate DNSSEC or TLS certificates,respectively, before adding auto-retrieved public keys to your localkeyring.

Kernel.org publishes the WKD for all developers who have kernel.orgaccounts. Once you have the above changes in yourgpg.conf, you canauto-retrieve the keys for Linus Torvalds and Greg Kroah-Hartman (if youdon’t already have them):

$ gpg --locate-keys torvalds@kernel.org gregkh@kernel.org

If you have a kernel.org account, then you shouldadd the kernel.orgUID to your key to make WKD more useful to other kernel developers.

Web of Trust (WOT) vs. Trust on First Use (TOFU)¶

PGP incorporates a trust delegation mechanism known as the “Web ofTrust.” At its core, this is an attempt to replace the need forcentralized Certification Authorities of the HTTPS/TLS world. Instead ofvarious software makers dictating who should be your trusted certifyingentity, PGP leaves this responsibility to each user.

Unfortunately, very few people understand how the Web of Trust works.While it is still an important part of the OpenPGP specification,recent versions of GnuPG (2.2 and above) have implemented an alternativemechanism called “Trust on First Use” (TOFU). You can think of TOFU as“the SSH-like approach to trust.” With SSH, the first time you connectto a remote system, its key fingerprint is recorded and remembered. Ifthe key changes in the future, the SSH client will alert you and refuseto connect, forcing you to make a decision on whether you choose totrust the changed key or not. Similarly, the first time you importsomeone’s PGP key, it is assumed to be valid. If at any point in thefuture GnuPG comes across another key with the same identity, both thepreviously imported key and the new key will be marked for verificationand you will need to manually figure out which one to keep.

We recommend that you use the combined TOFU+PGP trust model (which isthe new default in GnuPG v2). To set it, add (or modify) thetrust-model setting in~/.gnupg/gpg.conf:

trust-model tofu+pgp

Using the kernel.org web of trust repository¶

Kernel.org maintains a git repository with developers’ public keys as areplacement for replicating keyserver networks that have gone mostlydark in the past few years. The full documentation for how to set upthat repository as your source of public keys can be found here:

• Kernel developer PGP Keyring

If you are a kernel developer, please consider submitting your key forinclusion into that keyring.