Repository files navigation

Problem: "I can manage all my K8s config in git, except Secrets."

Solution: Encrypt your Secret into a SealedSecret, whichis safeto store - even inside a public repository. The SealedSecret can bedecrypted only by the controller running in the target cluster andnobody else (not even the original author) is able to obtain theoriginal Secret from the SealedSecret.

• Overview

  • SealedSecrets as templates for secrets
  • Public key / Certificate
  • Scopes

• Installation

  • Controller
    • Kustomize
    • Helm Chart
  • Kubeseal
    • Homebrew
    • MacPorts
    • Linux
    • Installation from source

• Upgrade

• Usage

  • Managing existing secrets
  • Patching existing secrets
  • Update existing secrets
  • Raw mode (experimental)
  • Validate a Sealed Secret

• Secret Rotation

  • Sealing key renewal
  • User secret rotation
  • Early key renewal
  • Common misconceptions about key renewal
  • Manual key management (advanced)
  • Re-encryption (advanced)

• Details (advanced)

  • Crypto

• Developing

• FAQ

  • Will you still be able to decrypt if you no longer have access to your cluster?
  • How can I do a backup of my SealedSecrets?
  • Can I decrypt my secrets offline with a backup key?
  • What flags are available for kubeseal?
  • How do I update parts of JSON/YAML/TOML/.. file encrypted with sealed secrets?
  • Can I bring my own (pre-generated) certificates?
  • How to use kubeseal if the controller is not running within thekube-system namespace?
  • How to verify the images?
  • How to use one controller for a subset of namespaces
  • Can I configure the controller unseal retries

• Community

  • Related projects

Sealed Secrets is composed of two parts:

• A cluster-side controller / operator
• A client-side utility:kubeseal

Thekubeseal utility uses asymmetric crypto to encrypt secrets that only the controller can decrypt.

These encrypted secrets are encoded in aSealedSecret resource, which you can see as a recipe for creatinga secret. Here is how it looks:

apiVersion:bitnami.com/v1alpha1kind:SealedSecretmetadata:name:mysecretnamespace:mynamespacespec:encryptedData:foo:AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

Once unsealed this will produce a secret equivalent to this:

apiVersion:v1kind:Secretmetadata:name:mysecretnamespace:mynamespacedata:foo:YmFy# <- base64 encoded "bar"

This normalkubernetes secret will appear in the clusterafter a few seconds you can use it as you would use any secret that you would have created directly (e.g. reference it from aPod).

Jump to theInstallation section to get up and running.

TheUsage section explores in more detail how you craftSealedSecret resources.

The previous example only focused on the encrypted secret items themselves, but the relationship between aSealedSecret custom resource and theSecret it unseals into is similar in many ways (but not in all of them) to the familiarDeployment vsPod.

In particular, the annotations and labels of aSealedSecret resource are not the same as the annotations of theSecret that gets generated out of it.

To capture this distinction, theSealedSecret object has atemplate section which encodes all the fields you want the controller to put in the unsealedSecret.

TheSprig function library is available (except forenv,expandenv andgetHostByName) in addition to the default Go Text Template functions.

Themetadata block is copied as is (theownerReference field will be updatedunless disabled).

Other secret fields are handled individually. Thetype andimmutable fields are copied, and thedata field can be used totemplate complex values on theSecret. All other fields are currently ignored.

apiVersion:bitnami.com/v1alpha1kind:SealedSecretmetadata:name:mysecretnamespace:mynamespaceannotations:"kubectl.kubernetes.io/last-applied-configuration":....spec:encryptedData:.dockerconfigjson:AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....template:type:kubernetes.io/dockerconfigjsonimmutable:true# this is an example of labels and annotations that will be added to the output secretmetadata:labels:"jenkins.io/credentials-type":usernamePasswordannotations:"jenkins.io/credentials-description":credentials from Kubernetes

The controller would unseal that into something like:

apiVersion:v1kind:Secretmetadata:name:mysecretnamespace:mynamespacelabels:"jenkins.io/credentials-type":usernamePasswordannotations:"jenkins.io/credentials-description":credentials from KubernetesownerReferences:  -apiVersion:bitnami.com/v1alpha1controller:truekind:SealedSecretname:mysecretuid:5caff6a0-c9ac-11e9-881e-42010aac003etype:kubernetes.io/dockerconfigjsonimmutable:truedata:.dockerconfigjson:ewogICJjcmVk...

As you can see, the generatedSecret resource is a "dependent object" of theSealedSecret and as suchit will be updated and deleted whenever theSealedSecret object gets updated or deleted.

The key certificate (public key portion) is used for sealing secrets,and needs to be available whereverkubeseal is going to beused. The certificate is not secret information, although you need toensure you are using the correct one.

kubeseal will fetch the certificate from the controller at runtime(requires secure access to the Kubernetes API server), which isconvenient for interactive use, but it's known to be brittle when usershave clusters with special configurations such asprivate GKE clusters that havefirewalls between control plane and nodes.

An alternative workflowis to store the certificate somewhere (e.g. local disk) withkubeseal --fetch-cert >mycert.pem,and use it offline withkubeseal --cert mycert.pem.The certificate is also printed to the controller log on startup.

Since v0.9.x certificates get automatically renewed every 30 days. It's good practice that you and your teamupdate your offline certificate periodically. To help you with that, since v0.9.2kubeseal accepts URLs too. You can set up your internal automation to publish certificates somewhere you trust.

kubeseal --cert https://your.intranet.company.com/sealed-secrets/your-cluster.cert

It also recognizes theSEALED_SECRETS_CERT env var. (pro-tip: see alsodirenv).

NOTE: we are working on providing key management mechanisms that offload the encryption to HSM based modules or managed cloud crypto solutions such as KMS.

SealedSecrets are from the POV of an end user a "write only" device.

The idea is that the SealedSecret can be decrypted only by the controller running in the target cluster andnobody else (not even the original author) is able to obtain the original Secret from the SealedSecret.

The user may or may not have direct access to the target cluster.More specifically, the user might or might not have access to the Secret unsealed by the controller.

There are many ways to configure RBAC on k8s, but it's quite common to forbid low-privilege usersfrom reading Secrets. It's also common to give users one or more namespaces where they have higher privileges,which would allow them to create and read secrets (and/or create deployments that can reference those secrets).

EncryptedSealedSecret resources are designed to be safe to be looked at without gaining any knowledge about the secrets it conceals. This implies that we cannot allow users to read a SealedSecret meant for a namespace they wouldn't have access toand just push a copy of it in a namespace where they can read secrets from.

Sealed-secrets thus behavesas if each namespace had its own independent encryption key and thus once youseal a secret for a namespace, it cannot be moved in another namespace and decrypted there.

We don't technically use an independent private key for each namespace, but instead weinclude the namespace nameduring the encryption process, effectively achieving the same result.

Furthermore, namespaces are not the only level at which RBAC configurations can decide who can see which secret. In fact, it's possible that users can access a secret calledfoo in a given namespace but not any other secret in the same namespace. We cannot thus by default let users freely renameSealedSecret resources otherwise a malicious user would be able to decrypt any SealedSecret for that namespace by just renaming it to overwrite the one secret user does have access to. We use the same mechanism used to include the namespace in the encryption key to also include the secret name.

That said, there are many scenarios where you might not care about this level of protection. For example, the only people who have access to your clusters are either admins or they cannot read anySecret resource at all. You might have a use case for moving a sealed secret to other namespaces (e.g. you might not know the namespace name upfront), or you might not know the name of the secret (e.g. it could contain a unique suffix based on the hash of the contents etc).

These are the possible scopes:

• strict (default): the secret must be sealed with exactly the samename andnamespace. These attributes becomepart of the encrypted data and thus changing name and/or namespace would lead to "decryption error".
• namespace-wide: you can freelyrename the sealed secret within a given namespace.
• cluster-wide: the secret can be unsealed inany namespace and can be givenany name.

In contrast to the restrictions ofname andnamespace, secretitems (i.e. JSON object keys likespec.encryptedData.my-key) can be renamed at will without losing the ability to decrypt the sealed secret.

The scope is selected with the--scope flag:

kubeseal --scope cluster-wide<secret.yaml>sealed-secret.json

It's also possible to request a scope via annotations in the input secret you pass tokubeseal:

• sealedsecrets.bitnami.com/namespace-wide: "true" -> fornamespace-wide
• sealedsecrets.bitnami.com/cluster-wide: "true" -> forcluster-wide

The lack of any of such annotations meansstrict mode. If both are set,cluster-wide takes precedence.

NOTE: Next release will consolidate this into a singlesealedsecrets.bitnami.com/scope annotation.

Seehttps://github.com/bitnami-labs/sealed-secrets/releases for the latest release and detailed installation instructions.

Cloud platform specific notes and instructions:

• GKE

Once you deploy the manifest it will create theSealedSecret resourceand install the controller intokube-system namespace, create a serviceaccount and necessary RBAC roles.

After a few moments, the controller will start, generate a key pair,and be ready for operation. If it does not, check the controller logs.

The official controller manifest installation mechanism is just a YAML file.

In some cases you might need to apply your own customizations, like set a custom namespace or set some env variables.

kubectl has native support for that, seekustomize.

The Sealed Secrets helm chart is now officially supported and hosted in this GitHub repo.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets

NOTE: The versioning scheme of the helm chart differs from the versioning scheme of the sealed secrets project itself.

Originally the helm chart was maintained by the community and the first version adopted a major version of 1 while thesealed secrets project itself is still at major 0.This is ok because the version of the helm chart itself is not meant to be necessarily the version of the app itself.However this is confusing, so our current versioning rule is:

1. TheSealedSecret controller version scheme: 0.X.Y
2. The helm chart version scheme: 1.X.Y-rZ

There can be thus multiple revisions of the helm chart, with fixes that apply only to the helm chart withoutaffecting the static YAML manifests or the controller image itself.

NOTE: The helm chart readme still contains a deprecation notice, but it no longer reflects reality and will be removed upon the next release.

NOTE: The helm chart by default installs the controller with the namesealed-secrets, while thekubeseal command line interface (CLI) tries to access the controller with the namesealed-secrets-controller. You can explicitly pass--controller-name to the CLI:

kubeseal --controller-name sealed-secrets<args>

Alternatively, you can setfullnameOverride when installing the chart to override the name. Note also thatkubeseal assumes that the controller is installed within thekube-system namespace by default. So if you want to use thekubeseal CLI without having to pass the expected controller name and namespace you should install the Helm Chart like this:

helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets

In some companies you might be given access only to a single namespace, not a full cluster.

One of the most restrictive environments you can encounter is:

• Anamespace was allocated to you with someservice account.
• You do not have access to the rest of the cluster, not even cluster CRDs.
• You may not even be able to create further service accounts or roles in your namespace.
• You are required to include resource limits in all your deployments.

Even with these restrictions you can still install the sealed secrets Helm Chart, there is only one pre-requisite:

• The cluster must already have the sealed secrets CRDs installed.

Once your admins installed the CRDs, if they were not there already, you can install the chart by preparing a YAML config file such as this:

serviceAccount:  create:false  name: {allocated-service-account}rbac:  create:false  clusterRole:falseresources:  limits:    cpu: 150m    memory: 256Mi

Note that:

• No service accounts are created, instead the one allocated to you will be used.
  • {allocated-service-account} is the name of theservice account you were allocated on the cluster.
• No RBAC roles are created neither in the namespace nor the cluster.
• Resource limits must be specified.
  • The limits are samples that should work, but you might want to review them in your particular setup.

Once that file is ready, if you named itconfig.yaml you now can install the sealed secrets Helm Chart like this:

helm install sealed-secrets -n {allocated-namespace} sealed-secrets/sealed-secrets --skip-crds -f config.yaml

Where{allocated-namespace} is the name of thenamespace you were allocated in the cluster.

Thekubeseal client is also available onhomebrew:

brew install kubeseal

Thekubeseal client is also available onMacPorts:

port install kubeseal

Thekubeseal client is also available onNixpkgs: (DISCLAIMER: Not maintained by bitnami-labs)

nix-env -iA nixpkgs.kubeseal

Thekubeseal client can be installed on Linux, using the below commands:

KUBESEAL_VERSION=''# Set this to, for example, KUBESEAL_VERSION='0.23.0'curl -OL"https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION:?}/kubeseal-${KUBESEAL_VERSION:?}-linux-amd64.tar.gz"tar -xvzf kubeseal-${KUBESEAL_VERSION:?}-linux-amd64.tar.gz kubesealsudo install -m 755 kubeseal /usr/local/bin/kubeseal

If you havecurl andjq installed on your machine, you can get the version dynamically this way. This can be useful for environments used in automation and such.

# Fetch the latest sealed-secrets version using GitHub APIKUBESEAL_VERSION=$(curl -s https://api.github.com/repos/bitnami-labs/sealed-secrets/tags | jq -r '.[0].name' | cut -c 2-)# Check if the version was fetched successfullyif [ -z "$KUBESEAL_VERSION" ]; then    echo "Failed to fetch the latest KUBESEAL_VERSION"    exit 1ficurl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz kubesealsudo install -m 755 kubeseal /usr/local/bin/kubeseal

whereKUBESEAL_VERSION is theversion tag of the kubeseal release you want to use. For example:v0.18.0.

If you just want the latest client tool, it can be installed into$GOPATH/bin with:

go install github.com/bitnami-labs/sealed-secrets/cmd/kubeseal@main

You can specify a release tag or a commit SHA instead ofmain.

Thego install command will place thekubeseal binary at$GOPATH/bin:

$(go env GOPATH)/bin/kubeseal

Don't forget to check therelease notes for guidance aboutpossible breaking changes when you upgrade the client tooland/or the controller.

Currently, only the latest version of Sealed Secrets is supported for production environments.

The Sealed Secrets controller ensures compatibility with different versions of Kubernetes by relying on a stable Kubernetes API. Typically, Kubernetes versions above 1.16 are considered compatible. However, we officially support thecurrently recommended Kubernetes versions. Additionally, versions above 1.24 undergo thorough verification through our CI process with every release.

# Create a json/yaml-encoded Secret somehow:# (note use of `--dry-run` - this is just a local file!)echo -n bar| kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json>mysecret.json# This is the important bit:kubeseal -f mysecret.json -w mysealedsecret.json# At this point mysealedsecret.json is safe to upload to Github,# post on Twitter, etc.# Eventually:kubectl create -f mysealedsecret.json# Profit!kubectl get secret mysecret

Note theSealedSecret andSecret must havethe same namespace andname. This is a feature to prevent other users on the same clusterfrom re-using your sealed secrets. See theScopes section for more info.

kubeseal reads the namespace from the input secret, accepts an explicit--namespace argument, and usesthekubectl default namespace (in that order). Any labels,annotations, etc on the originalSecret are preserved, but notautomatically reflected in theSealedSecret.

By design, this schemedoes not authenticate the user. In otherwords,anyone can create aSealedSecret containing anySecretthey like (provided the namespace/name matches). It is up to yourexisting config management workflow, cluster RBAC rules, etc to ensurethat only the intendedSealedSecret is uploaded to the cluster. Theonly change from existing Kubernetes is that thecontents of theSecret are now hidden while outside the cluster.

If you want the Sealed Secrets controller to manage an existingSecret, you can annotate yourSecret with thesealedsecrets.bitnami.com/managed: "true" annotation. The existingSecret will be overwritten when unsealing aSealedSecret with the same name and namespace, and theSealedSecret will take ownership of theSecret (so that when theSealedSecret is deleted theSecret will also be deleted).

New in v0.23.0

There are some use cases in which you don't want to replace the wholeSecret but just add or modify some keys from the existingSecret. For this, you can annotate yourSecret withsealedsecrets.bitnami.com/patch: "true". Using this annotation will make sure that secret keys, labels and annotations in theSecret that are not present in theSealedSecret won't be deleted, and those present in theSealedSecret will be added to theSecret (secret keys, labels and annotations that exist both in theSecret and theSealedSecret will be modified by theSealedSecret).

This annotation does not make theSealedSecret take ownership of theSecret. You can add both thepatch andmanaged annotations to obtain the patching behavior while also taking ownership of theSecret.

If you wantSealedSecret and theSecret to be independent, which mean when you delete theSealedSecret theSecret won't disappear with it, then you have to annotate that Secret with the annotationsealedsecrets.bitnami.com/skip-set-owner-references: "true" ahead of applying the Usage steps. You still may also addsealedsecrets.bitnami.com/managed: "true" to yourSecret so that your secret will be updated whenSealedSecret is updated.

If you want to add or update existing sealed secrets without having the cleartext for the other items,you can just copy&paste the new encrypted data items and merge it into an existing sealed secret.

You must take care of sealing the updated items with a compatible name and namespace (see note about scopes above).

You can use the--merge-into command to update an existing sealed secrets if you don't want to copy&paste:

echo -n bar| kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o json \| kubeseal> mysealedsecret.jsonecho -n baz| kubectl create secret generic mysecret --dry-run=client --from-file=bar=/dev/stdin -o json \| kubeseal --merge-into mysealedsecret.json

Creating temporary Secret with thekubectl command, only to throw it away once piped tokubeseal canbe a quite unfriendly user experience. We're working on an overhaul of the CLI experience. In the meantime,we offer an alternative mode where kubeseal only cares about encrypting a value to stdout, and it's your responsibility to put it inside aSealedSecret resource (not unlike any of the other k8s resources).

It can also be useful as a building block for editor/IDE integrations.

The downside is that you have to be careful to be consistent with the sealing scope, the namespace and the name.

SeeScopes

strict scope (default):

$echo -n foo| kubeseal --raw --namespace bar --name mysecretAgBChHUWLMx...

namespace-wide scope:

$echo -n foo| kubeseal --raw --namespace bar --scope namespace-wideAgAbbFNkM54...

Include thesealedsecrets.bitnami.com/namespace-wide annotation in theSealedSecret

metadata:annotations:sealedsecrets.bitnami.com/namespace-wide:"true"

cluster-wide scope:

$echo -n foo| kubeseal --raw --scope cluster-wideAgAjLKpIYV+...

Include thesealedsecrets.bitnami.com/cluster-wide annotation in theSealedSecret

metadata:annotations:sealedsecrets.bitnami.com/cluster-wide:"true"

If you want to validate an existing sealed secret,kubeseal has the flag--validate to help you.

Giving a file namedsealed-secrets.yaml containing the following sealed secret:

apiVersion:bitnami.com/v1alpha1kind:SealedSecretmetadata:name:mysecretnamespace:mynamespacespec:encryptedData:foo:AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

You can validate if the sealed secret was properly created or not:

$cat sealed-secrets.yaml| kubeseal --validate

In case of an invalid sealed secret,kubeseal will show:

$cat sealed-secrets.yaml| kubeseal --validateerror: unable to decrypt sealed secret

You should always rotate your secrets. But since your secrets are encrypted with another secret,you need to understand how these two layers relate to take the right decisions.

TL;DR:

If asealing private key is compromised, you need to follow the instructions below in "Early key renewal"section before rotating any of your actual secret values.

SealedSecret key renewal and re-encryption features arenot a substitute for periodical rotation of your actual secret values.

Sealing keys are automatically renewed every 30 days. Which means a new sealing key is created and appended to the set of active sealing keys the controller can use to unsealSealedSecret resources.

The most recently created sealing key is the one used to seal new secrets when you usekubeseal and it's the one whose certificate is downloaded when you usekubeseal --fetch-cert.

The renewal time of 30 days is a reasonable default, but it can be tweaked as neededwith the--key-renew-period=<value> flag for the command in the pod template of theSealedSecret controller. Thevalue field can be given as golangduration flag (eg:720h30m). Assuming that you've installed Sealed Secrets into thekube-system namespace, use the following command to edit the Deployment controller, and add the--key-renew-period parameter. Once you close your text editor, and the Deployment controller has been modified, a new Pod will be automatically created to replace the old Pod.

kubectl edit deployment/sealed-secrets-controller --namespace=kube-system

A value of0 will deactivate automatic key renewal. Of course, you may have a valid use case for deactivating automatic sealing key renewal but experience has shown that new users often tend to jump to conclusions that they want control over key renewal, before fully understanding how sealed secrets work. Read more about this in thecommon misconceptions section below.

Unfortunately, you cannot use e.g. "d" as a unit for days because that's not supported by the Go stdlib. Instead of hitting your face with a palm, take this as an opportunity to meditate on thefalsehoods programmers believe about time.

A common misunderstanding is that key renewal is often thought of as a form of key rotation, where the old key is not only obsolete but actually bad and that you thus want to get rid of it.It doesn't help that this feature has been historically called "key rotation", which can add to the confusion.

Sealed secrets are not automatically rotated and old keys are not deletedwhen new keys are generated. OldSealedSecret resources can be still decrypted (that's because old sealing keys are not deleted).

When the controller starts, it will initialize the key registry. The most recent key is used to seal secrets. By default, this certificate is chosen based on the NotBefore attribute of the certificate. If you want to change the priority order of the keys in the registry, you can use the--key-order-priority flag.

The--key-order-priority flag accepts the following values:

• CertNotBefore: (default) The key registry will be ordered based on the NotBefore attribute of the key certificate.
• SecretCreationTimestamp: The key registry will be ordered based on the creation timestamp of the secret.

This flag influences the public key used to encrypt secrets and the certificate retrieved bykubeseal --fetch-cert.

Thesealing key renewal and SealedSecret rotation arenot a substitute for rotating your actual secrets.

A core value proposition of this tool is:

Encrypt your Secret into a SealedSecret, whichis safe to store - even inside a public repository.

If you store anything in a version control storage, and in a public one in particular, you must assumeyou cannot ever delete that information.

If a sealing key somehow leaks out of the cluster you must consider all yourSealedSecret resourcesencrypted with that key as compromised. No amount of sealing key rotation in the cluster or even re-encryption of existing SealedSecrets files can change that.

The best practice is to periodically rotate all your actual secrets (e.g. change the password)and craft newSealedSecret resources with those new secrets.

But if theSealedSecret controller was not renewing thesealing key that rotation would be moot,since the attacker could just decrypt the new secrets as well. Thus, you need to do both: periodically renew the sealing key and rotate your actual secrets!

If you know or suspect asealing key has been compromised you should renew the key ASAP before youstart sealing your new rotated secrets, otherwise you'll be giving attackers access to your new secrets as well.

A key can be generated early by passing the current timestamp to the controller into a flag called--key-cutoff-time or an env var calledSEALED_SECRETS_KEY_CUTOFF_TIME. The expected format is RFC1123, you can generate it with thedate -R unix command.

Sealed secrets sealing keys are not access control keys (e.g. a password). They are more like the GPG key you might use to read encrypted mail sent to you. Let's continue with the email analogy for a bit:

Imagine you have reasons to believe your private GPG key might have been compromised. You'd have more to lose than to gain if the first thing you do is just delete your private key. All the previous emails sent with that key are no longer accessible to you (unless you have a decrypted copy of those emails), nor are new emails sent by your friends whom you have not yet managed to tell to use the new key.

Sure, the content of those encrypted emails is not secure, as an attacker might now be able to decrypt them, but what's done is done. Your sudden loss of the ability to read those emails surely doesn't undo the damage. If anything, it's worse because you no longer know for sure what secret the attacker got to know. What you really want to do is to make sure that your friend stops using your old key and that from now on all further communication is encrypted with a new key pair (i.e. your friend must know about that new key).

The same logic applies to SealedSecrets. The ultimate goal is to secure your actual "user" secrets. The "sealing" secrets are just a mechanism, an "envelope". If a secret is leaked there is no going back, what's done is done.

You first need to ensure that new secrets don't get encrypted with that old compromised key (in the email analogy above that's: create a new key pair and give all your friends your new public key).

The second logical step is to neutralize the damage, which depends on the nature of the secret. A simple example is a database password: if you accidentally leak your database password, the thing you're supposed to do is simply to change your database password (on the database; and revoke the old one!)and update theSealedSecret resource with the new password (i.e. runningkubeseal again).

Both steps are described in the previous sections, albeit in a less verbose way. There is no shame in reading them again, now that you have a more in-depth grasp of the underlying rationale.

TheSealedSecret controller and the associated workflow are designed to keep old sealing keys around and periodically add new ones. You should not delete old keys unless you know what you're doing.

That said, if you want you can manually manage (create, move, delete)sealing keys. They are just normal k8s secrets living in the same namespace where theSealedSecret controller lives (usuallykube-system, but it's configurable).

There are advanced use cases that you can address by creative management of the sealing keys.For example, you can share the same sealing key among a few clusters so that you can apply exactly the same sealed secret in multiple clusters.Since sealing keys are just normal k8s secrets you can even use sealed secrets themselves and use a GitOps workflow to manage your sealing keys (useful when you want to share the same key among different clusters)!

Labeling asealing key secret with anything other thanactive effectively deletesthe key from theSealedSecret controller, but it is still available in k8s formanual encryption/decryption if need be.

NOTESealedSecret controller currently does not automatically pick up manually created, deleted or relabeled sealing keys. An admin must restart the controller before the effect will apply.

Before you can get rid of some old sealing keys you need to re-encrypt your SealedSecrets with the latest private key.

kubeseal --re-encrypt<my_sealed_secret.json>tmp.json \&& mv tmp.json my_sealed_secret.json

The invocation above will produce a new sealed secret file freshly encrypted withthe latest key, without making the secrets leave the cluster to the client. You can then save that filein your version control system (kubeseal --re-encrypt doesn't update the in-cluster object).

Currently, old keys are not garbage collected automatically.

It's a good idea to periodically re-encrypt your SealedSecrets. But as mentioned above, don't lull yourself in a false sense of security: you must assume the old version of theSealedSecret resource (the one encrypted with a key you think of as dead) is still potentially around and accessible to attackers. I.e. re-encryption is not a substitute for periodically rotating your actual secrets.

This controller adds a newSealedSecret custom resource. Theinteresting part of aSealedSecret is a base64-encodedasymmetrically encryptedSecret.

The controller maintains a set of private/public key pairs as kubernetessecrets. Keys are labeled withsealedsecrets.bitnami.com/sealed-secrets-keyand identified in the label as eitheractive orcompromised. On startup,The sealed secrets controller will...

1. Search for these keys and add them to its local store if they arelabeled as active.
2. Create a new key
3. Start the key rotation cycle

More details about crypto can be foundhere.

Developing guidelines can be foundin the Developer Guide.

Yes, you can! Drop as many secrets as you like in one file. Make sure to separate them via--- for YAML and as extra, single objects in JSON.

No, the private keys are only stored in the Secret managed by the controller (unless you have some other backup of your k8s objects). There are no backdoors - without that private key used to encrypt a given SealedSecrets, you can't decrypt it. If you can't get to the Secrets with the encryption keys, and you also can't get to the decrypted versions of your Secrets live in the cluster, then you will need to regenerate new passwords for everything, seal them again with a new sealing key, etc.

If you do want to make a backup of the encryption private keys, it's easy to do from an account with suitable access:

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml>main.keyecho"---">> main.keykubectl get secret -n kube-system sealed-secrets-key -o yaml>>main.key

NOTE: You need the second statement only if you ever installed sealed-secrets older than version 0.9.x on your cluster.

NOTE: This file will contain the controller's public + private keys and should be kept omg-safe!

NOTE: After sealing key renewal you should recreate your backup. Otherwise, your backup won't be able to decrypt new sealed secrets.

To restore from a backup after some disaster, just put that secrets back before starting the controller - or if the controller was already started, replace the newly-created secrets and restart the controller:

• For Helm deployment:

  kubectl apply -f main.keykubectl delete pod -n kube-system -l app.kubernetes.io/name=sealed-secrets

• For deployment viacontroller.yaml manifest

  kubectl apply -f main.keykubectl delete pod -n kube-system -l name=sealed-secrets-controller

While treating sealed-secrets as long term storage system for secrets is not the recommended use case, some peopledo have a legitimate requirement for being able to recover secrets when the k8s cluster is down and restoring a backup into a newSealedSecret controller deployment is not practical.

If you have backed up one or more of your private keys (see previous question), you can use thekubeseal --recovery-unseal --recovery-private-key file1.key,file2.key,... command to decrypt a sealed secrets file.

You can check the flags available usingkubeseal --help.

A kubernetesSecret resource contains multiple items, basically a flat map of key/value pairs.SealedSecrets operate at that level, and does not care what you put in the values. In other wordsit cannot make sense of any structured configuration file you might have put in a secret and thuscannot help you update individual fields in it.

Since this is a common problem, especially when dealing with legacy applications, we do offer anexample of a possible workaround.

Yes, you can provide the controller with your own certificates, and it will consume them.Please checkhere for a workaround.

If you installed the controller in a different namespace than the defaultkube-system, you need to provide this namespaceto thekubeseal commandline tool. There are two options:

1. You can specify the namespace via the command line option--controller-namespace <namespace>:

kubeseal --controller-namespace sealed-secrets<mysecret.json>mysealedsecret.json

2. Via the environment variableSEALED_SECRETS_CONTROLLER_NAMESPACE:

export SEALED_SECRETS_CONTROLLER_NAMESPACE=sealed-secretskubeseal<mysecret.json>mysealedsecret.json

Our images are being signed usingcosign. The signatures have been saved in ourGitHub Container Registry.

Images up to and including v0.20.2 were signed using Cosign v1. Newer images are signed with Cosign v2.

It is pretty simple to verify the images:

# export the COSIGN_VARIABLE setting up the GitHub container registry signs pathexport COSIGN_REPOSITORY=ghcr.io/bitnami-labs/sealed-secrets-controller/signs# verify the image uploaded in GHCRcosign verify --key .github/workflows/cosign.pub ghcr.io/bitnami-labs/sealed-secrets-controller:latest# verify the image uploaded in Dockerhubcosign verify --key .github/workflows/cosign.pub docker.io/bitnami/sealed-secrets-controller:latest

If you want to use one controller for more than one namespace, but not all namespaces, you can provide additional namespaces using the command line flag--additional-namespaces=<namespace1>,<namespace2>,<...>. Make sure you provide appropriate roles and rolebindings in the target namespaces, so the controller can manage the secrets in there.

The answer is yes, you can configure the number of retries in your controller using the flag--max-unseal-retries. This flag allows you to configure the number of maximum retries to unseal your Sealed Secrets.

• #sealed-secrets on Kubernetes Slack

Clickhere to sign up to the Kubernetes Slack org.

• kubeseal-convert:https://github.com/EladLeev/kubeseal-convert
• Visual Studio Code extension:https://marketplace.visualstudio.com/items?itemName=codecontemplator.kubeseal
• WebSeal: generates secrets in the browser:https://socialgouv.github.io/webseal
• HybridEncrypt TypeScript implementation:https://github.com/SocialGouv/aes-gcm-rsa-oaep
• [DEPRACATED] Sealed Secrets Operator:https://github.com/disposab1e/sealed-secrets-operator-helm