Kustomization

The GitOps Toolkit Custom Resource Definitions documentation.

TheKustomization API defines a pipeline for fetching, decrypting, building,validating and applying Kustomize overlays or plain Kubernetes manifests.TheKustomization Custom Resource Definition is thecounterpart of Kustomize’skustomization.yaml config file.

Example

The following is an example of a Flux Kustomization that reconciles theKubernetes manifests stored in a Git repository.

apiVersion:source.toolkit.fluxcd.io/v1kind:GitRepositorymetadata:name:podinfonamespace:defaultspec:interval:5murl:https://github.com/stefanprodan/podinforef:branch:master---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:podinfonamespace:defaultspec:interval:10mtargetNamespace:defaultsourceRef:kind:GitRepositoryname:podinfopath:"./kustomize"prune:truetimeout:1m

In the above example:

A Flux GitRepository namedpodinfo is created that clones themasterbranch and makes the repository content available as an Artifact inside the cluster.
A Flux Kustomization namedpodinfo is created that watches theGitRepository for Artifact changes.
The Kustomization builds the YAML manifests located at the specified.spec.path,sets the namespace of all objects to the.spec.targetNamespace,validates the objects against the Kubernetes API and finally applies them onthe cluster.
As specified by.spec.interval, every ten minutes, the Kustomization runs aserver-side apply dry-run to detect and correct drift inside the cluster.
When the Git revision changes, the manifests are reconciled automatically. Ifpreviously applied objects are missing from the current revision, theseobjects are deleted from the cluster when.spec.prune is enabled.

You can run this example by saving the manifest intopodinfo.yaml.

Apply the resource on the cluster:
```
kubectl apply -f podinfo.yaml
```

Runkubectl get gitrepositories to see the source status:

NAME      URL                                       READY   STATUSpodinfo   https://github.com/stefanprodan/podinfo   True    stored artifact for revision 'master@sha1:450796ddb2ab6724ee1cc32a4be56da032d1cca0'

Runkubectl get kustomizations to see the reconciliation status:

NAME      READY   STATUSpodinfo   True    Applied revision: master@sha1:450796ddb2ab6724ee1cc32a4be56da032d1cca0

Runkubectl describe kustomization podinfo to see the reconciliation statusconditions and events:

... Status:   Conditions:     Last Transition Time:  2023-03-07T11:14:41Z     Message:               Applied revision: master@sha1:450796ddb2ab6724ee1cc32a4be56da032d1cca0     Reason:                ReconciliationSucceeded     Status:                True     Type:                  Ready Events:   Type    Reason       Age    From                  Message   ----    ------       ----   ----                  -------   Normal  Progressing  1m48s  kustomize-controller  Service/default/podinfo created                                                     Deployment/default/podinfo created                                                     HorizontalPodAutoscaler/default/podinfo created   Normal  ReconciliationSucceeded  1m48s  kustomize-controller  Reconciliation finished in 176.163666ms, next run in 10m0s

Writing a Kustomization spec

As with all other Kubernetes config, a Kustomization needsapiVersion,kind, andmetadata fields. The name of a Kustomization object must be avalidDNS subdomain name.

A Kustomization also needs a.spec section.

Source reference

.spec.sourceRef is used to refer to the Source object which has the requiredArtifact containing the YAML manifests. It has two required fields:

kind: The Kind of the referred Source object. Supported Source types:
name: The Name of the referred Source object.

Cross-namespace references

By default, the Source object is assumed to be in the same namespace as theKustomization. To refer to a Source object in a different namespace, specifythe namespace using.spec.sourceRef.namespace.

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:webappnamespace:appsspec:interval:5mpath:"./deploy"sourceRef:kind:GitRepositoryname:webappnamespace:shared

On multi-tenant clusters, platform admins can disable cross-namespace referencesby starting kustomize-controller with the--no-cross-namespace-refs=true flag.

Prune

.spec.prune is a required boolean field to enable/disable garbage collectionfor a Kustomization.

Garbage collection means that the Kubernetes objects that were previouslyapplied on the cluster but are missing from the current source revision, areremoved from the cluster automatically. Garbage collection is also performedwhen a Kustomization object is deleted, triggering a removal of all Kubernetesobjects previously applied on the cluster. The removal of the Kubernetesobjects is done in the background, i.e. it doesn’t block the reconciliation ofthe Kustomization.

To enable garbage collection for a Kustomization, set this field totrue.

You can disable pruning for certain resources by either labelling orannotating them with:

kustomize.toolkit.fluxcd.io/prune:disabled

For details on how the controller tracks Kubernetes objects and determines whatto garbage collect, see.status.inventory.

Deletion policy

.spec.deletionPolicy is an optional field that allows control overgarbage collection when a Kustomization object is deleted. The default behavioris to mirror the configuration of.spec.prune.

Valid values:

MirrorPrune (default) - The managed resources will be deleted ifprune istrue and orphaned iffalse.
Delete - Ensure the managed resources are deleted before the Kustomizationis deleted.
WaitForTermination - Ensure the managed resources are deleted and wait fortermination before the Kustomization is deleted.
Orphan - Leave the managed resources when the Kustomization is deleted.

TheWaitForTermination deletion policy blocks and waits for the managedresources to be removed from etcd by the Kubernetes garbage collector.The wait time is determined by the.spec.timeout field. If a timeout occurs,the controller will stop waiting for the deletion of the resources,log an error and will allow the Kustomization to be deleted.

For special cases when the managed resources are removed by other means (e.g.the deletion of the namespace specified with.spec.targetNamespace), you can set the deletion policytoOrphan:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:appnamespace:defaultspec:# ...omitted for brevitytargetNamespace:app-namespaceprune:truedeletionPolicy:Orphan

Interval

.spec.interval is a required field that specifies the interval at which theKustomization is reconciled, i.e. the controller fetches the source with theKubernetes manifests, builds the Kustomization and applies it on the cluster,correcting any existing drift in the process. The minimum value should be 60seconds.

After successfully reconciling the object, the controller requeues it forinspection after the specified interval. The value must be in aGo recognized duration string format,e.g.10m0s to reconcile the object every 10 minutes.

If the.metadata.generation of a resource changes (due to e.g. a change tothe spec) or the Source revision changes (which generates a Kubernetes event),this is handled instantly outside the interval window.

Note: The controller can be configured to apply a jitter to the interval inorder to distribute the load more evenly when multiple Kustomization objects areset up with the same interval. For more information, please refer to thekustomize-controller configuration options.

Retry interval

.spec.retryInterval is an optional field to specify the interval at which toretry a failed reconciliation. Unlike.spec.interval, this field isexclusively meant for failure retries. If not specified, it defaults to.spec.interval.

Path

.spec.path is an optional field to specify the path to the directory in theSource Artifact containing thekustomization.yaml file, or the set of plainYAMLs for which akustomization.yaml should be generated.It defaults to blank, which translates to the root of the Source Artifact.

For more details on the generation of the file, seegenerating akustomization.yaml file.

Target namespace

.spec.targetNamespace is an optional field to specify the target namespace forall the objects that are part of the Kustomization. It either configures oroverrides theKustomizenamespace.

While.spec.targetNamespace is optional, if this field is non-empty then theKubernetes namespace being pointed to must exist prior to the Kustomizationbeing applied or be defined by a manifest included in the Kustomization.kustomize-controller will not create the namespace automatically.

Suspend

.spec.suspend is an optional boolean field to suspend the reconciliation of theKustomization. When a Kustomization is suspended, new Source revisions are notapplied to the cluster and drift detection/correction is paused.To resume normal reconciliation, set it back tofalse or remove the field.

For more information, seesuspending and resuming.

Health checks

.spec.healthChecks is an optional list used to refer to resources for which thecontroller will perform health checks used to determine the rollout status ofdeployed workloadsand theReady status of custom resources.

A health check entry can reference one of the following types:

Kubernetes built-in kinds: Deployment, DaemonSet, StatefulSet,PersistentVolumeClaim, Pod, PodDisruptionBudget, Job, CronJob, Service,Secret, ConfigMap, CustomResourceDefinition
Flux kinds: HelmRelease, HelmRepository, GitRepository, etc.
Custom resources that are compatible withkstatus

Assuming the Kustomization source contains a Kubernetes Deployment namedbackend, a health check can be defined as follows:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:backendnamespace:defaultspec:interval:5mprune:truesourceRef:kind:GitRepositoryname:webapphealthChecks:-apiVersion:apps/v1kind:Deploymentname:backendnamespace:dev

After applying the kustomize build output, the controller verifies if therollout was completed successfully. If the deployment was successful, theKustomizationReady condition is marked asTrue, if the rollout failed,or if it takes more than the specified timeout to complete, then theKustomizationReady condition is set toFalse. If the deployment becomeshealthy on the next execution, then the Kustomization is marked as ready.

When a Kustomization contains HelmRelease objects, instead of checking theunderlying Deployments, you can define a health check that waits for theHelmReleases to be reconciled with:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:webappnamespace:defaultspec:interval:15mpath:"./releases/"prune:truesourceRef:kind:GitRepositoryname:webapphealthChecks:-apiVersion:helm.toolkit.fluxcd.io/v2kind:HelmReleasename:frontendnamespace:dev-apiVersion:helm.toolkit.fluxcd.io/v2kind:HelmReleasename:backendnamespace:devtimeout:5m

If all the HelmRelease objects are successfully installed or upgraded, thenthe Kustomization will be marked as ready.

Health check expressions

.spec.healthCheckExprs can be used to define custom logic for performinghealth checks on custom resources. This is done through Common ExpressionLanguage (CEL) expressions. This field accepts a list of objects with thefollowing fields:

apiVersion: The API version of the custom resource. Required.
kind: The kind of the custom resource. Required.
current: A required CEL expression that returnstrue if the resource is ready.
inProgress: An optional CEL expression that returnstrue if the resourceis still being reconciled.
failed: An optional CEL expression that returnstrue if the resourcefailed to reconcile.

The controller will evaluate the expressions in the following order:

inProgress if specified
failed if specified
current

The first expression that evaluates totrue will determine the healthstatus of the custom resource.

For example, to define a set of health check expressions for theSealedSecretcustom resource:

apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:sealed-secretsnamespace:flux-systemspec:interval:5mpath:./path/to/sealed/secretsprune:truesourceRef:kind:GitRepositoryname:flux-systemtimeout:1mwait:true# Tells the controller to wait for all resources to be ready by performing health checks.healthCheckExprs:-apiVersion:bitnami.com/v1alpha1kind:SealedSecretfailed:status.conditions.filter(e, e.type == 'Synced').all(e, e.status == 'False')current:status.conditions.filter(e, e.type == 'Synced').all(e, e.status == 'True')

A common error is writing expressions that reference fields that do notexist in the custom resource. This will cause the controller to waitfor the resource to be ready until the timeout is reached. To avoid this,make sure your CEL expressions are correct. TheCEL Playground is a useful resource forthis task. The input passed to each expression is the custom resourceobject itself. You can check for field existence with thehas(...) CEL macro,just be aware thathas(status) errors ifstatus does not (yet) existon the top level of the resource you are using.

It’s worth checking ifthe libraryhas expressions for the custom resources you are using.

Wait

.spec.wait is an optional boolean field to perform health checks forallreconciled resources as part of the Kustomization. If set totrue,.spec.healthChecks is ignored.

Timeout

.spec.timeout is an optional field to specify a timeout duration for anyoperation like building, applying, health checking, etc. performed during thereconciliation process.

Dependencies

.spec.dependsOn is an optional list used to refer to other Kustomizationobjects that the Kustomization depends on. If specified, then the Kustomizationis only applied after the referred Kustomizations are ready, i.e. have theReady condition marked asTrue. The readiness state of a Kustomization isdetermined by its last applied status condition.

This is helpful when there is a need to make sure other resources exist beforethe workloads defined in a Kustomization are deployed. For example, beforeinstalling objects of a certain custom resource kind, the CRDs and the relatedcontroller must exist in the cluster.

For example, assuming we have two Kustomizations:

cert-manager: reconciles the cert-manager CRDs and controller
certs: reconciles the cert-manager custom resources

You can instruct the controller to apply thecert-manager Kustomization beforecerts by defining adependsOn relationship between the two:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:cert-managernamespace:flux-systemspec:interval:5mpath:"./cert-manager/controller"prune:truesourceRef:kind:GitRepositoryname:flux-systemhealthChecks:-apiVersion:apps/v1kind:Deploymentname:cert-managernamespace:cert-manager---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:certsnamespace:flux-systemspec:dependsOn:-name:cert-managerinterval:5mpath:"./cert-manager/certs"prune:truesourceRef:kind:GitRepositoryname:flux-system

If.spec.healthChecks is non-empty or.spec.wait is set totrue, aKustomization will be applied after all its dependencies’ health checks havepassed. For example, this can be used to ensure a service mesh proxy injectoris running before deploying applications inside the mesh.

Note: Circular dependencies between Kustomizations must be avoided,otherwise the interdependent Kustomizations will never be applied on the cluster.

Service Account reference

.spec.serviceAccountName is an optional field used to specify theServiceAccount to be impersonated while reconciling the Kustomization. For moredetails, seeRole-based Access Control.

Common metadata

.spec.commonMetadata is an optional field used to specify any metadata thatshould be applied to all the Kustomization’s resources. It has two optional fields:

labels: A map used for settinglabelson an object. Any existing label will be overridden if it matches with a key inthis map.
annotations: A map used for settingannotationson an object. Any existing annotation will be overridden if it matches with a keyin this map.

Name Prefix and Suffix

.spec.namePrefix and.spec.nameSuffix are optional fields used to specify a prefix and suffixto be added to the names of all the resources in the Kustomization.

apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:appspec:# ...omitted for brevitynamePrefix:"prefix-"nameSuffix:"-suffix"

Patches

.spec.patches is an optional list used to specifyKustomizepatchesas inline YAML objects. This enables patching resources using either astrategic mergepatch or aJSON6902patch. A patch can target a single resource or multiple resources. Each item inthe list must have the two fields mentioned below:

patch: Patch contains an inline strategic merge patch or an inline JSON6902 patch with an array of operation objects.
target: Target points to the resources that the patch document should be applied to.

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:podinfonamespace:flux-systemspec:# ...omitted for brevitypatches:-patch:|-        apiVersion: apps/v1        kind: Deployment        metadata:          name: not-used        spec:          template:            metadata:              annotations:                cluster-autoscaler.kubernetes.io/safe-to-evict: "true"target:kind:DeploymentlabelSelector:"app.kubernetes.io/part-of=my-app"-patch:|        - op: add          path: /spec/template/spec/securityContext          value:            runAsUser: 10000            fsGroup: 1337        - op: add          path: /spec/template/spec/containers/0/securityContext          value:            readOnlyRootFilesystem: true            allowPrivilegeEscalation: false            runAsNonRoot: true            capabilities:              drop:                - ALLtarget:kind:Deploymentname:podinfonamespace:apps

Images

.spec.images is an optional list used to specifyKustomizeimages.This allows overwriting the name, tag or digest of container images without creating patches.

apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:podinfonamespace:flux-systemspec:# ...omitted for brevityimages:-name:podinfonewName:my-registry/podinfonewTag:v1-name:podinfonewTag:1.8.0-name:podinfonewName:my-podinfo-name:podinfodigest:sha256:24a0c4b4a4c0eb97a1aabb8e29f18e917d05abfe1b7a7c07857230879ce7d3d3

Components

.spec.components is an optional list used to specifyKustomizecomponents.This allows using reusable pieces of configuration logic that can be includedfrom multiple overlays.

apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:podinfonamespace:flux-systemspec:# ...omitted for brevitycomponents:- ../ingress- ../tls

Note: The components paths must be local and relative to the path specified by.spec.path.

Warning: Components are an alpha feature in Kustomize and are thereforeconsidered experimental in Flux. No guarantees are provided as the feature maybe modified in backwards incompatible ways or removed without warning.

Post build variable substitution

With.spec.postBuild.substitute you can provide a map of key-value pairsholding the variables to be substituted in the final YAML manifest, afterkustomize build.

With.spec.postBuild.substituteFrom you can provide a list of ConfigMaps andSecrets from which the variables are loaded. The ConfigMap and Secret data keysare used as the variable names.

The.spec.postBuild.substituteFrom.optional field indicates how thecontroller should handle a referenced ConfigMap or Secret being absentat reconciliation time. The controller’s default behavior ― withoptional unspecified or set tofalse ― it has failed reconciliation ifthe referenced object is missing. By setting theoptional field totrue, you can indicate that the controller should use the referencedobject if it’s there, but also tolerate its absence, treating thatabsence as if the object had been present but empty, defining novariables.

This offers basic templating for your manifests including supportforbash string replacement functions e.g.:

${var:=default}
${var:position}
${var:position:length}
${var/substring/replacement}

Note: The name of a variable can contain only alphanumeric and underscorecharacters. The controller validates the variable names using this regularexpression:^[_[:alpha:]][_[:alpha:][:digit:]]*$.

For example, assuming we have manifests with the following variables:

---apiVersion:v1kind:Namespacemetadata:name:appslabels:environment:${cluster_env:=dev}region:"${cluster_region}"

You can specify the variables and their values in the Kustomization definition using.spec.postBuild.substitute and/or.spec.postBuild.substituteFrom:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:appsspec:# ...omitted for brevitypostBuild:substitute:cluster_env:"prod"cluster_region:"eu-central-1"substituteFrom:-kind:ConfigMapname:cluster-vars# Use this ConfigMap if it exists, but proceed if it doesn't.optional:true-kind:Secretname:cluster-secret-vars# Fail if this Secret does not exist.

Note: For substituting variables in a secret,.spec.stringData field must be used i.e:

---apiVersion:v1kind:Secretmetadata:name:secretnamespace:flux-systemtype:OpaquestringData:token:${token}

Note: The var values which are specified in-line withsubstitutetake precedence over the ones derived fromsubstituteFrom.When var values for the same variable keys are derived from multipleConfigMaps orSecrets referenced in thesubstituteFrom list, then thefirst take precedence over the later values.

Note: If you want to avoid var substitutions in scripts embedded inConfigMaps or container commands, you must use the format$var instead of${var}. If you want to keep the curly braces you can use$${var} whichwill print out${var}.

All the undefined variables in the format${var} will be substituted with anempty string unless a default value is provided e.g.${var:=default}.

Note: It is recommended to set the--feature-gates=StrictPostBuildSubstitutions=truecontroller flag, so that the post-build substitutions will fail if avariable without a default value is declared in files but ismissing from the input vars.

You can disable the variable substitution for certain resources by eitherlabelling or annotating them with:

kustomize.toolkit.fluxcd.io/substitute:disabled

Substitution of variables only happens if at least a single variable or resourceto substitute from is defined. This may cause issues if you rely on expressionswhich should evaluate to a default value, even if no other variables areconfigured. To work around this, one can set an arbitrary key/value pair toenable the substitution of variables. For example:

apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:appsspec:# ...omitted for brevitypostBuild:substitute:var_substitution_enabled:"true"

Note: When using numbers or booleans as values for variables, they must beenclosed in double quotes vars to be treated as strings, for more information seesubstitution of numbers and booleans.

You can replicate the controller post-build substitutions locally usingkustomizeand the Flux CLI:

$exportcluster_region=eu-central-1$ kustomize build ./apps/ | flux envsubst --strict---apiVersion: v1kind: Namespacemetadata:  name: apps  labels:    environment: dev    region: eu-central-1

Force

.spec.force is an optional boolean field. If set totrue, the controllerwill replace the resources in-cluster if the patching fails due to immutablefield changes.

It can also be enabled for specific resources by labelling or annotating themwith:

kustomize.toolkit.fluxcd.io/force:enabled

KubeConfig reference

.spec.kubeConfig.secretRef.Name is an optional field to specify the name ofthe secret containing a KubeConfig. If specified, objects will be applied,health-checked, pruned, and deleted for the default cluster specified in thatKubeConfig instead of using the in-cluster ServiceAccount.

The secret defined in thekubeConfig.SecretRef must exist in the samenamespace as the Kustomization. On every reconciliation, the KubeConfig byteswill be loaded from the.secretRef.key key (default:value orvalue.yaml)of the Secret’s data , and the Secret can thus be regularly updated ifcluster-access-tokens have to rotate due to expiration.

---apiVersion:v1kind:Secretmetadata:name:prod-kubeconfigtype:OpaquestringData:value.yaml:|    apiVersion: v1    kind: Config    # ...omitted for brevity

Note: The KubeConfig should be self-contained and not rely on binaries,environment, or credential files from the kustomize-controller Pod.This matches the constraints of KubeConfigs from current Cluster API providers.KubeConfigs withcmd-path in them likely won’t work without a custom,per-provider installation of kustomize-controller.

When both.spec.kubeConfig and.spec.ServiceAccountName are specified,the controller will impersonate the service account on the target cluster.

For more information, seeremote clusters/Cluster-API.

Decryption

Storing Secrets in Git repositories in plain text or base64 is unsafe,regardless of the visibility or access restrictions of the repository.

In order to store Secrets safely in Git repositorioes you can use anencryption provider and the optional field.spec.decryption toconfigure decryption for Secrets that are a part of the Kustomization.

The only supported encryption provider isSOPS.With SOPS you can encrypt your secrets withageorOpenPGP keys, or with keys from Key Management Services(KMS), like AWS KMS, Azure Key Vault, GCP KMS or Hashicorp Vault.

Note: You must leavemetadata,kind orapiVersion in plain text.An easy way to do this is limiting the encrypted keys with the flag--encrypted-regex '^(data|stringData)$' in yoursops encrypt command.

The.spec.decryption field has the following subfields:

.provider: The secrets decryption provider to be used. This field is required andthe only supported value issops.
.secretRef.name: The name of the secret that contains the keys or cloud providerstatic credentials for KMS services to be used for decryption.
.serviceAccountName: The name of the service account used forsecret-less authentication with KMS services from cloud providers.

For a complete guide on how to set up authentication for KMS services fromcloud providers, see the integrationdocs.

If a static credential for a given cloud provider is defined inside the secretreferenced by.secretRef, that static credential takes priority over secret-lessauthentication for that provider. If no static credentials are defined for a givencloud provider inside the secret, secret-less authentication is attempted for thatprovider.

If.serviceAccountName is specified for secret-less authentication,it takes priority overcontroller global decryptionfor all cloud providers.

Example:

apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:sops-encryptednamespace:defaultspec:interval:5mpath:"./"sourceRef:kind:GitRepositoryname:repository-with-secretsdecryption:provider:sopsserviceAccountName:sops-identitysecretRef:name:sops-keys-and-credentials

The Secret’s.data section is expected to contain entries with decryptionkeys (for age and OpenPGP), or credentials (for any of the supported providerimplementations). The controller identifies the type of the entry by the suffixof the key (e.g..agekey), or a fixed key (e.g.sops.vault-token).

---apiVersion:v1kind:Secretmetadata:name:sops-keys-and-credentialsnamespace:defaultdata:# Exemplary age private keyidentity.agekey:<BASE64># Exemplary Hashicorp Vault tokensops.vault-token:<BASE64>

age Secret entry

To specify an age private key in a Kubernetes Secret, suffix the key of the.data entry with.agekey.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultdata:# Exemplary age private keyidentity.agekey:<BASE64>

OpenPGP Secret entry

To specify an OpenPGP (passwordless) keyring in armor format in a KubernetesSecret, suffix the key of the.data entry with.asc.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultdata:# Exemplary OpenPGP keyringidentity.asc:<BASE64>

AWS KMS Secret entry

To specify credentials for an AWS user account linked to the IAM role with accessto KMS, append a.data entry with a fixedsops.aws-kms key.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultdata:sops.aws-kms:|    aws_access_key_id: some-access-key-id    aws_secret_access_key: some-aws-secret-access-key    aws_session_token: some-aws-session-token # this field is optional

Azure Key Vault Secret entry

To specify credentials for Azure Key Vault in a Secret, append a.data entrywith a fixedsops.azure-kv key. The value can contain a variety of JSON orYAML formats depending on the authentication method you want to utilize.

Service Principal with Secret

To configure a Service Principal with Secret credentials to access the AzureKey Vault, a JSON or YAML object withtenantId,clientId andclientSecretfields must be configured as thesops.azure-kv value. It optionally supportsauthorityHost to configure an authority host other than the Azure Public Cloudendpoint.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultstringData:# Exemplary Azure Service Principal with Secretsops.azure-kv:|    tenantId: some-tenant-id    clientId: some-client-id    clientSecret: some-client-secret

Service Principal with Certificate

To configure a Service Principal with Certificate credentials to access theAzure Key Vault, a JSON or YAML object withtenantId,clientId andclientCertificate fields must be configured as thesops.azure-kv value.It optionally supportsclientCertificateSendChain andauthorityHost tocontrol the sending of the certificate chain, or to specify an authority hostother than the Azure Public Cloud endpoint.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultstringData:# Exemplary Azure Service Principal with Certificatesops.azure-kv:|    tenantId: some-tenant-id    clientId: some-client-id    clientCertificate: <certificate PEM>

`az` generated Service Principal

To configure a Service Principalgenerated usingaz,the output of the command can be directly used as asops.azure-kv value.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultstringData:# Exemplary Azure Service Principal generated with `az`sops.azure-kv:|    {      "appId": "559513bd-0c19-4c1a-87cd-851a26afd5fc",      "displayName": "myAKSClusterServicePrincipal",      "name": "http://myAKSClusterServicePrincipal",      "password": "e763725a-5eee-40e8-a466-dc88d980f415",      "tenant": "72f988bf-86f1-41af-91ab-2d7cd011db48"    }

Managed Identity with Client ID

To configure a Managed Identity making use of a Client ID, a JSON or YAMLobject with aclientId must be configured as thesops.azure-kv value. Itoptionally supportsauthorityHost to configure an authority host other thanthe Azure Public Cloud endpoint.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultstringData:# Exemplary Azure Managed Identity with Client IDsops.azure-kv:|    clientId: some-client-id

GCP KMS Secret entry

To specify credentials for GCP KMS in a Kubernetes Secret, append a.dataentry with a fixedsops.gcp-kms key and the service account keys as its value.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultstringData:# Exemplary GCP Service Account credentials filesops.gcp-kms:|    {      "type": "service_account",      "project_id": "<project-id>",      "private_key_id": "<private-key-id>",      "private_key": "<private-key>"    }

Hashicorp Vault Secret entry

To specify credentials for Hashicorp Vault in a Kubernetes Secret, append a.data entry with a fixedsops.vault-token key and the token as value.

---apiVersion:v1kind:Secretmetadata:name:sops-keysnamespace:defaultdata:# Exemplary Hashicorp Vault Secret tokensops.vault-token:<BASE64>

Working with Kustomizations

Recommended settings

When deploying applications to production environments, it is recommendedto configure the following fields, while adjusting them to your desires forresponsiveness:

apiVersion:source.toolkit.fluxcd.io/v1kind:GitRepositorymetadata:name:webappnamespace:appsspec:interval:1m0s# check for new commits every minute and apply changesurl:https://github.com/org/webapp# clone over HTTPSsecretRef:# use token authname:webapp-git-token# Flux user PAT (read-only access)ref:branch:mainignore:|    # exclude all    /*    # include deploy dir    !/deploy---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:webappnamespace:appsspec:interval:60m0s# detect drift and undo kubectl edits every hourwait:true# wait for all applied resources to become readytimeout:3m0s# give up waiting after three minutesretryInterval:2m0s# retry every two minutes on apply or waiting failuresprune:true# remove stale resources from clusterforce:false# enable this to recreate resources on immutable fields changestargetNamespace:apps# set the namespace for all resourcessourceRef:kind:GitRepositoryname:webappnamespace:appspath:"./deploy/production"

Generating a`kustomization.yaml` file

If your repository contains plain Kubernetes manifests without akustomization.yaml, the file is automatically generated for all theKubernetes manifests in the directory tree specified in.spec.path.

All YAML files present under that path must be valid Kubernetes manifests,unless they’re excluded either by way of the.sourceignorefile or the.spec.ignorefield on the corresponding Source object.

Example of excluding CI workflows and SOPS config files:

apiVersion:source.toolkit.fluxcd.io/v1kind:GitRepositorymetadata:name:podinfonamespace:defaultspec:interval:5murl:https://github.com/stefanprodan/podinfoignore:|    .github/    .sops.yaml    .gitlab-ci.yml

It is recommended to generate thekustomization.yaml on your own and store itin Git, this way you can validate your manifests in CI(example script).Assuming your manifests are insideapps/my-app, you can generate akustomization.yaml with:

cd apps/my-app# create kustomization.yamlkustomize create --autodetect --recursive

Controlling the apply behavior of resources

To change the apply behaviour for specific Kubernetes resources, you can annotate them with:

Annotation	Default	Values	Role
`kustomize.toolkit.fluxcd.io/ssa`	`Override`	-`Override` -`Merge` -`IfNotPresent` -`Ignore`	Apply policy
`kustomize.toolkit.fluxcd.io/force`	`Disabled`	-`Enabled` -`Disabled`	Recreate policy
`kustomize.toolkit.fluxcd.io/prune`	`Enabled`	-`Enabled` -`Disabled`	Delete policy

Note: These annotations should be set in the Kubernetes YAML manifests includedin the Flux Kustomization source (Git, OCI, Bucket).

`kustomize.toolkit.fluxcd.io/ssa`

Override

TheOverride policy instructs the controller to reconcile the Kubernetes resourceswith the desired state (YAML manifests) defined in the Flux source (Git, OCI, Bucket).

If you usekubectl to edit a Kubernetes resource managed by Flux, all changes will bereverted when the controller reconciles a Flux Kustomization containing that resource.In order to preserve fields added withkubectl, you have to specifya field manager namedflux-client-side-apply e.g.:

kubectl apply --field-manager=flux-client-side-apply

Merge

TheMerge policy instructs the controller to preserve the fields added by other tools to theKubernetes resources managed by Flux.

The fields defined in the manifests applied by the controller will always be overridden,theMerge policy works only for adding new fields that don’t overlap with the desiredstate.

For lists fields which are atomic (e.g..spec.tolerations in PodSpec), Kubernetesdoesn’t allow different managers for such fields, therefore any changes to thesefields will be reverted. For more context, please see the Kubernetes enhancement document:555-server-side-apply.

IfNotPresent

TheIfNotPresent policy instructs the controller to only apply the Kubernetes resourcesif they are not present on the cluster.

This policy can be used for Kubernetes Secrets and ValidatingWebhookConfigurations managed by cert-manager,where Flux creates the resources with fields that are later on mutated by other controllers.

Ignore

TheIgnore policy instructs the controller to skip applying Kubernetes resourceseven if they are included in a Flux source (Git, OCI, Bucket).

`kustomize.toolkit.fluxcd.io/force`

When set toEnabled, this policy instructs the controller to recreate the Kubernetes resourceswith changes to immutable fields.

This policy can be used for Kubernetes Jobs to rerun them when their container image changes.

Note: Using this policy for StatefulSets may result in potential data loss.

`kustomize.toolkit.fluxcd.io/prune`

When set toDisabled, this policy instructs the controller to skip the deletion ofthe Kubernetes resources subject togarbage collection.

This policy can be used to protect sensitive resources such as Namespaces, PVCs and PVsfrom accidental deletion.

Role-based access control

By default, a Kustomization apply runs under the cluster admin account and cancreate, modify and delete cluster level objects (namespaces, CRDs, etc) andnamespaced objects (deployments, ingresses, etc). For certain Kustomizations acluster admin may wish to control what types of Kubernetes objects can bereconciled and under which namespaces.To restrict a Kustomization, one can assign a service account under which thereconciliation is performed using.spec.serviceAccountName.

Assuming you want to restrict a group of Kustomizations to a single namespace,you can create an account with a role binding that grants access only to that namespace:

---apiVersion:v1kind:Namespacemetadata:name:webapp---apiVersion:v1kind:ServiceAccountmetadata:name:fluxnamespace:webapp---apiVersion:rbac.authorization.k8s.io/v1kind:Rolemetadata:name:webapp-reconcilernamespace:webapprules:-apiGroups:['*']resources:['*']verbs:['*']---apiVersion:rbac.authorization.k8s.io/v1kind:RoleBindingmetadata:name:webapp-reconcilernamespace:webapproleRef:apiGroup:rbac.authorization.k8s.iokind:Rolename:webapp-reconcilersubjects:-kind:ServiceAccountname:fluxnamespace:webapp

Note: The namespace, RBAC and service account manifests should beplaced in a Git source and applied with a Kustomization. The Kustomizations thatare running under that service account should depend on the one that contains the account.

Create a Kustomization that prevents altering the cluster state outside thewebapp namespace:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:backendnamespace:webappspec:serviceAccountName:fluxdependsOn:-name:rbacinterval:5mpath:"./webapp/backend/"prune:truesourceRef:kind:GitRepositoryname:webapp

When the controller reconciles thebackend Kustomization, it will impersonatetheflux ServiceAccount. If the Kustomization contains cluster level objectslike CRDs or objects belonging to a different namespace, the reconciliation willfail since the account it runs under has no permissions to alter objects outsidethewebapp namespace.

Enforcing impersonation

On multi-tenant clusters, platform admins can enforce impersonation with the--default-service-account flag.

When the flag is set, all Kustomizations which don’t have.spec.serviceAccountNamespecified will use the service account name provided by--default-service-account=<SA Name> in the namespace of the object.

Remote clusters/Cluster-API

With the.spec.kubeConfig field a Kustomization can be fullyreconciled on a remote cluster. This composes well with Cluster API bootstrapproviders such as CAPBK (kubeadm), CAPA (AWS) and others.

To reconcile a Kustomization to a CAPI controlled cluster, put theKustomization in the same namespace as yourCluster object, and set thekubeConfig.secretRef.name to<cluster-name>-kubeconfig:

apiVersion:cluster.x-k8s.io/v1alpha3kind:Clustermetadata:name:stage# the kubeconfig Secret will contain the Cluster namenamespace:capi-stagespec:clusterNetwork:pods:cidrBlocks:-10.100.0.0/16serviceDomain:stage-cluster.localservices:cidrBlocks:-10.200.0.0/12controlPlaneRef:apiVersion:controlplane.cluster.x-k8s.io/v1alpha3kind:KubeadmControlPlanename:stage-control-planenamespace:capi-stageinfrastructureRef:apiVersion:infrastructure.cluster.x-k8s.io/v1alpha3kind:DockerClustername:stagenamespace:capi-stage---# ... unrelated Cluster API objects omitted for brevity ...---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:cluster-addonsnamespace:capi-stagespec:interval:5mpath:"./config/addons/"prune:truesourceRef:kind:GitRepositoryname:cluster-addonskubeConfig:secretRef:name:stage-kubeconfig# Cluster API creates this for the matching Cluster

The Cluster and Kustomization can be created at the same time.The Kustomization will eventually reconcile once the cluster is available.

If you wish to target clusters created by other means than CAPI, you can createa ServiceAccount on the remote cluster, generate a KubeConfig for that accountand then create a secret on the cluster where kustomize-controller is running.For example:

kubectl create secret generic prod-kubeconfig\    --from-file=value.yaml=./kubeconfig

Controller global decryption

Other thanauthentication using a Secret reference,it is possible to specify global decryption settings on thekustomize-controller Pod. When the controller fails to find credentials on theKustomization object itself, it will fall back to these defaults.

AWS KMS

While making use of theIAM OIDC provideron your EKS cluster, you can create an IAM Role and Service Account with accessto AWS KMS (using at leastkms:Decrypt andkms:DescribeKey). Once these arecreated, you can annotate the kustomize-controller Service Account with theRole ARN, granting the controller permission to decrypt the Secrets. Please referto theSOPS guide for detailed steps.

kubectl -n flux-system annotate serviceaccount kustomize-controller\  --field-manager=flux-client-side-apply\  eks.amazonaws.com/role-arn='arn:aws:iam::<ACCOUNT_ID>:role/<KMS-ROLE-NAME>'

Furthermore, you can also use the usualenvironment variables used for specifying AWScredentials,by patching the kustomize-controller Deployment:

---apiVersion:apps/v1kind:Deploymentmetadata:name:kustomize-controllernamespace:flux-systemspec:template:spec:containers:-name:managerenv:-name:AWS_ACCESS_KEY_IDvalueFrom:secretKeyRef:name:aws-credskey:awsAccessKeyID-name:AWS_SECRET_ACCESS_KEYvalueFrom:secretKeyRef:name:aws-credskey:awsSecretAccessKey-name:AWS_SESSION_TOKENvalueFrom:secretKeyRef:name:aws-credskey:awsSessionToken

In addition to this, thegeneral SOPS documentation around KMS AWS applies,allowing you to specify e.g. aSOPS_KMS_ARN environment variable.

Note:: If you are mounting a secret containing the AWS credentials as afile in thekustomize-controller Pod, you need to specify an environmentvariable$HOME, since the AWS credentials file is expected to be present at~/.aws. For example:

env:-name:HOMEvalue:/home/{$USER}

Azure Key Vault

Workload Identity

If you have Workload Identity set up on your AKS cluster, you can establisha federated identity between the kustomize-controller ServiceAccount and anidentity that has “Decrypt” role on the Azure Key Vault. Once, this is doneyou can label and annotate the kustomize-controller ServiceAccount and Podwith the patch shown below:

apiVersion:kustomize.config.k8s.io/v1beta1kind:Kustomizationresources:- gotk-components.yaml- gotk-sync.yamlpatches:-patch:|-      apiVersion: v1      kind: ServiceAccount      metadata:        name: kustomize-controller        namespace: flux-system        annotations:          azure.workload.identity/client-id: <AZURE_CLIENT_ID>        labels:          azure.workload.identity/use: "true"-patch:|-      apiVersion: apps/v1      kind: Deployment      metadata:        name: kustomize-controller        namespace: flux-system        labels:          azure.workload.identity/use: "true"      spec:        template:          metadata:            labels:              azure.workload.identity/use: "true"

Kubelet Identity

If the kubelet managed identity hasDecrypt permissions on Azure Key Vault,no additional configuration is required for the kustomize-controller to decryptdata.

GCP KMS

While making use of Google Cloud Platform, theGOOGLE_APPLICATION_CREDENTIALSenvironment variableis automatically taken into account.Granting permissionsto the Service Account attached to this will therefore be sufficient to decryptdata. When running outside GCP, it is possible to manually patch thekustomize-controller Deployment with a valid set of (mounted) credentials.

---apiVersion:apps/v1kind:Deploymentmetadata:name:kustomize-controllernamespace:flux-systemspec:template:spec:containers:-name:managerenv:-name:GOOGLE_APPLICATION_CREDENTIALSvalue:/var/gcp/credentials.jsonvolumeMounts:-name:gcp-credentialsmountPath:/var/gcp/readOnly:truevolumes:-name:gcp-credentialssecret:secretName:mysecretitems:-key:credentialspath:credentials.json

Hashicorp Vault

To configure a global default for Hashicorp Vault, patch the controller’sDeployment with aVAULT_TOKEN environment variable.

---apiVersion:apps/v1kind:Deploymentmetadata:name:kustomize-controllernamespace:flux-systemspec:template:spec:containers:-name:managerenv:-name:VAULT_TOKENvalue:<token>

Kustomize secretGenerator

SOPS encrypted data can be stored as a base64 encoded Secret, which enables theuse ofKustomizesecretGeneratoras follows:

$echo"my-secret-token" | sops -e /dev/stdin > token.encrypted$ cat <<EOF > kustomization.yamlapiVersion: kustomize.config.k8s.io/v1beta1kind: KustomizationsecretGenerator: - name: token   files:   - token=token.encryptedEOF

Commit and pushtoken.encrypted andkustomization.yaml to Git.

The kustomize-controller scans the values of Kubernetes Secrets, and when itdetects that the values are SOPS encrypted, it decrypts them before applyingthem on the cluster.

For secrets in.json,.yaml.ini and.env format, make sure you specifythe input type when encrypting them with SOPS:

sops -e --input-type=json config.json > config.json.encryptedsops -e --input-type=yaml config.yaml > config.yaml.encryptedsops -e --input-type=env config.env > config.env.encrypted

For kustomize-controller to be able to decrypt a JSON config, you need to setthe file extension to.json:

kind:KustomizationsecretGenerator:-name:configfiles:- config.json=config.json.encrypted

For dotenv files, use theenvs directive:

kind:KustomizationsecretGenerator:-name:configenvs:- config.env.encrypted

For Docker config files, you need to specify both input and output type as JSON:

sops -e --input-type=json --output-type=json ghcr.dockerconfigjson > ghcr.dockerconfigjson.encrypted

To generate an image pull secret, use the.dockerconfigjson as the secret key:

kind:KustomizationsecretGenerator:-name:ghcr-authtype:kubernetes.io/dockerconfigjsonfiles:- .dockerconfigjson=ghcr.dockerconfigjson.encrypted

Post build substitution of numbers and booleans

When usingvariable substitution with valuesthat are numbers or booleans, the reconciliation may fail if the substitutionis for a field that must be of type string. To convert the number or booleanto a string, you can wrap the variable with a double quotes var:

apiVersion:v1kind:ServiceAccountmetadata:name:appannotations:id:${quote}${id}${quote}enabled:${quote}${enabled}${quote}

Then in the Flux Kustomization, define the variables as:

apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:appspec:postBuild:substitute:quote:'"'# double quote varid:"123"enabled:"true"

Triggering a reconcile

To manually tell the kustomize-controller to reconcile a Kustomization outsidethespecified interval window, it can be annotated withreconcile.fluxcd.io/requestedAt: <arbitrary value>. Annotating the resourcequeues the Kustomization for reconciliation if the<arbitrary-value> differsfrom the last value the controller acted on, as reported in.status.lastHandledReconcileAt.

Usingkubectl:

kubectl annotate --field-manager=flux-client-side-apply --overwrite kustomization/<kustomization-name> reconcile.fluxcd.io/requestedAt="$(date +%s)"

Usingflux:

flux reconcile kustomization <kustomization-name>

Waiting for`Ready`

When a change is applied, it is possible to wait for the Kustomization to reachaReady state usingkubectl:

kubectlwait kustomization/<kustomization-name> --for=condition=ready --timeout=1m

Suspending and resuming

When you find yourself in a situation where you temporarily want to pause thereconciliation of a Kustomization, you can suspend it using.spec.suspend.

To pause the reconciliation of a specific Kubernetes resource managed by a Flux Kustomization,you can annotate or label the resource in-cluster with:

kustomize.toolkit.fluxcd.io/reconcile:disabled

Note: When thekustomize.toolkit.fluxcd.io/reconcile annotation is set todisabled, the controller will no longer apply changes, norwill it prune the resource. To resume reconciliation, set the annotation toenabled in the source or remove it from the in-cluster object.

Suspend a Kustomization

In your YAML declaration:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:<kustomization-name>spec:suspend:true

Usingkubectl:

kubectl patch kustomization <kustomization-name> --field-manager=flux-client-side-apply -p'{\"spec\": {\"suspend\" : true }}'

Usingflux:

fluxsuspend kustomization <kustomization-name>

Resume a Kustomization

In your YAML declaration, comment out (or remove) the field:

---apiVersion:kustomize.toolkit.fluxcd.io/v1kind:Kustomizationmetadata:name:<kustomization-name>spec:# suspend: true

Note: Setting the field value tofalse has the same effect as removingit, but does not allow for “hot patching” using e.g.kubectl while practicingGitOps; as the manually applied patch would be overwritten by the declaredstate in Git.

Usingkubectl:

kubectl patch kustomization <kustomization-name> --field-manager=flux-client-side-apply -p'{\"spec\" : {\"suspend\" : false }}'

Usingflux:

flux resume kustomization <kustomization-name>

Debugging a Kustomization

There are several ways to gather information about a Kustomization fordebugging purposes.

Describe the Kustomization

Describing a Kustomization usingkubectl describe kustomization <kustomization-name>displays the latest recorded information for the resource in theStatus andEvents sections:

...Status:...  Conditions:    Last Transition Time:  2023-03-29T06:09:32Z    Message:               Fetching manifests for revision master/67e2c98a60dc92283531412a9e604dd4bae005a9 with a timeout of 4m30s    Observed Generation:   3    Reason:                ProgressingWithRetry    Status:                True    Type:                  Reconciling    Last Transition Time:  2023-03-29T06:09:32Z    Message:               kustomization path not found: stat /tmp/kustomization-1464362706/invalid: no such file or directory    Observed Generation:   3    Reason:                ArtifactFailed    Status:                False    Type:                  Ready  Last Applied Revision:    master/67e2c98a60dc92283531412a9e604dd4bae005a9  Last Attempted Revision:  master/67e2c98a60dc92283531412a9e604dd4bae005a9  Observed Generation:      2Events:  Type     Reason          Age   From                  Message  ----     ------          ----  ----                  -------  Warning  ArtifactFailed           2s   kustomize-controller  kustomization path not found: stat /tmp/kustomization-1464362706/invalid: no such file or directory

Trace emitted Events

To view events for specific Kustomization(s),kubectl events can be usedto list the Events for specific objects. For example, running

kubectl events -n flux-system --for kustomization/podinfo

lists

LAST SEEN   TYPE     REASON                    OBJECT                  MESSAGE31s         Warning  ArtifactFailed            kustomization/podinfo   kustomization path not found: stat /tmp/kustomization-3011588360/invalid: no such file or directory26s         Normal   ArtifactFailed            kustomization/podinfo   HorizontalPodAutoscaler/default/podinfo deleted...18s         Warning  ArtifactFailed            kustomization/podinfo   kustomization path not found: stat /tmp/kustomization-3336282420/invalid: no such file or directory9s          Normal   Progressing               kustomization/podinfo   Service/default/podinfo created...9s          Normal   ReconciliationSucceeded   kustomization/podinfo   Reconciliation finished in 75.190237ms, next run in 5m0s

You can also use theflux events command to view all events for aKustomization and its related Source. For example,

flux events --for Kustomization/podinfo

will list all events for thepodinfo Kustomization in theflux-systemnamespace and its related Source object, thepodinfo GitRepository.

LAST SEEN               TYPE    REASON                          OBJECT                  MESSAGE3m2s                    Warning ArtifactFailed                  Kustomization/podinfo   kustomization path not found: stat /tmp/kustomization-3336282420/invalid: no such file or directory2m53s                   Normal  ReconciliationSucceeded         Kustomization/podinfo   Reconciliation finished in 75.190237ms, next run in 5m0s2m53s                   Normal  Progressing                     Kustomization/podinfo   Service/default/podinfo created                                                                                        Deployment/default/podinfo created                                                                                        HorizontalPodAutoscaler/default/podinfo created19s (x17 over 8m24s)    Normal  GitOperationSucceeded           GitRepository/podinfo   no changes since last reconcilation: observed revision 'master/67e2c98a60dc92283531412a9e604dd4bae005a9'

Besides being reported in Events, the reconciliation errors are also logged bythe controller. The Flux CLI offer commands for filtering the logs for aspecific Kustomization, e.g.flux logs --level=error --kind=Kustomization --name=<kustomization-name>.

Kustomization Status

Conditions

A Kustomization enters various states during its lifecycle, reflected asKubernetes Conditions.It can bereconciling while applying the Kustomization on the cluster, it can beready, or it canfail duringreconciliation.

The Kustomization API is compatible with thekstatus specification,and reportsReconciling andStalled conditions where applicable toprovide better (timeout) support to solutions polling the Kustomization tobecomeReady.

Reconciling Kustomization

The kustomize-controller marks a Kustomization asreconciling when it startsthe reconciliation of the same. The Condition added to the Kustomization’s.status.conditions has the following attributes:

type: Reconciling
status: "True"
reason: Progressing |reason: ProgressingWithRetry

The Conditionmessage is updated during the course of the reconciliation toreport the action being performed at any particular moment such asbuilding manifests, detecting drift, etc.

TheReady Condition’sstatus is also marked asUnkown.

Ready Kustomization

The kustomize-controller marks a Kustomization asready when a Kustomizationis successfully reconciled, i.e. the source was fetched, the kustomization wasbuilt and applied on the cluster and all health checks are observed to be passing.

When the Kustomization is “ready”, the controller sets a Condition with thefollowing attributes in the Kustomization’s.status.conditions:

type: Ready
status: "True"
reason: ReconciliationSucceeded

Failed Kustomization

The kustomize-controller may get stuck trying to reconcile and apply aKustomization without completing. This can occur due to some of the following factors:

The Source object does not exist on the cluster.
The Source has not produced an Artifact yet.
The Kustomization’s dependencies aren’t ready yet.
The specified path does not exist in the Artifact.
Building the kustomization fails.
Garbage collection fails.
Running a health check failed.

When this happens, the controller sets theReady Condition status to Falseand adds a Condition with the following attributes to the Kustomization’s.status.conditions:

type: Ready | HealthyCondition
status: "False"
reason: PruneFailed | ArtifactFailed | BuildFailed | HealthCheckFailed | DependencyNotReady | ReconciliationFailed

Themessage field of the Condition will contain more information about whythe reconciliation failed.

While the Kustomization has one or more of these Conditions, the controllerwill continue to attempt a reconciliation of the Kustomization with anexponential backoff, until it succeeds and the Kustomization marked asready.

Note that a Kustomization can bereconcilingwhile failing at the same time, for example, due to a newly introducedconfiguration issue in the Kustomization spec. When a reconciliation fails, theReconciling Conditionreason would beProgressingWithRetry. When thereconciliation is performed again after the failure, thereason is updated toProgressing.

Inventory

In order to perform operations such as drift detection, garbage collection, etc.kustomize-controller needs to keep track of all Kubernetes objects that arereconciled as part of a Kustomization. To do this, it maintains an inventorycontaining the list of Kubernetes resource object references that have beensuccessfully applied and records it in.status.inventory. The inventoryrecords are in the format<namespace>_<name>_<group>_<kind>_<version>.

Status:  Inventory:    Entries:      Id: default_podinfo__Service      V:  v1      Id: default_podinfo_apps_Deployment      V:  v1      Id: default_podinfo_autoscaling_HorizontalPodAutoscaler      V:  v2

Last applied revision

.status.lastAppliedRevision is the last revision of the Artifact from thereferred Source object that was successfully applied to the cluster.

Last applied origin revision

status.lastAppliedOriginRevision is the last origin revision of the Artifactfrom the referred Source object that was successfully applied to the cluster.

This field is usually retrieved from the Metadata of the Artifact and dependson the Source type. For example, for OCI artifacts this is the value associatedwith the standard metadata keyorg.opencontainers.image.revision, which isused to track the revision of the source code that was used to build the OCIartifact.

The controller will forward this value when emitting events in the metadatakeyoriginRevision. The notification-controller will look for this key inthe event metadata when sendingcommit status update events to Git providers.

Last attempted revision

.status.lastAttemptedRevision is the last revision of the Artifact from thereferred Source object that was attempted to be applied to the cluster.

Observed Generation

The kustomize-controller reports anobserved generationin the Kustomization’s.status.observedGeneration. The observed generation isthe latest.metadata.generation which resulted in either aready state,or stalled due to an error it can not recover from without humanintervention.

Last Handled Reconcile At

The kustomize-controller reports the lastreconcile.fluxcd.io/requestedAtannotation value it acted on in the.status.lastHandledReconcileAt field.

For practical information about this field, seetriggering a reconcile.

Movatterモバイル変換

Kustomization

Example

Writing a Kustomization spec

Source reference

Cross-namespace references

Prune

Deletion policy

Interval

Retry interval

Path

Target namespace

Suspend

Health checks

Health check expressions

Wait

Timeout

Dependencies

Service Account reference

Common metadata

Name Prefix and Suffix

Patches

Images

Components

Post build variable substitution

Force

KubeConfig reference

Decryption

age Secret entry

OpenPGP Secret entry

AWS KMS Secret entry

Azure Key Vault Secret entry

Service Principal with Secret

Service Principal with Certificate

az generated Service Principal

Managed Identity with Client ID

GCP KMS Secret entry

Hashicorp Vault Secret entry

Working with Kustomizations

Recommended settings

Generating akustomization.yaml file

Controlling the apply behavior of resources

kustomize.toolkit.fluxcd.io/ssa

Override

Merge

IfNotPresent

Ignore

kustomize.toolkit.fluxcd.io/force

kustomize.toolkit.fluxcd.io/prune

Role-based access control

Enforcing impersonation

Remote clusters/Cluster-API

Controller global decryption

AWS KMS

Azure Key Vault

Workload Identity

Kubelet Identity

GCP KMS

Hashicorp Vault

Kustomize secretGenerator

Post build substitution of numbers and booleans

Triggering a reconcile

Waiting forReady

Suspending and resuming

Suspend a Kustomization

Resume a Kustomization

Debugging a Kustomization

Describe the Kustomization

Trace emitted Events

Kustomization Status

Conditions

Reconciling Kustomization

Ready Kustomization

Failed Kustomization

Inventory

Last applied revision

Last applied origin revision

Last attempted revision

Observed Generation

Last Handled Reconcile At

`az` generated Service Principal

Generating a`kustomization.yaml` file

`kustomize.toolkit.fluxcd.io/ssa`

`kustomize.toolkit.fluxcd.io/force`

`kustomize.toolkit.fluxcd.io/prune`

Waiting for`Ready`