Extend the Kubernetes API with CustomResourceDefinitions

This page shows how to install acustom resourceinto the Kubernetes API by creating aCustomResourceDefinition.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool mustbe configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have acluster, you can create one by usingminikubeor you can use one of these Kubernetes playgrounds:

• iximiuz Labs
• Killercoda
• KodeKloud
• Play with Kubernetes

To check the version, enterkubectl version.

Create a CustomResourceDefinition

When you create a new CustomResourceDefinition (CRD), the Kubernetes API Servercreates a new RESTful resource path for each version you specify. The customresource created from a CRD object can be either namespaced or cluster-scoped,as specified in the CRD'sspec.scope field. As with existing built-inobjects, deleting a namespace deletes all custom objects in that namespace.CustomResourceDefinitions themselves are non-namespaced and are available toall namespaces.

For example, if you save the following CustomResourceDefinition toresourcedefinition.yaml:

apiVersion:apiextensions.k8s.io/v1kind:CustomResourceDefinitionmetadata:# name must match the spec fields below, and be in the form: <plural>.<group>name:crontabs.stable.example.comspec:# group name to use for REST API: /apis/<group>/<version>group:stable.example.com# list of versions supported by this CustomResourceDefinitionversions:-name:v1# Each version can be enabled/disabled by Served flag.served:true# One and only one version must be marked as the storage version.storage:trueschema:openAPIV3Schema:type:objectproperties:spec:type:objectproperties:cronSpec:type:stringimage:type:stringreplicas:type:integer# either Namespaced or Clusterscope:Namespacednames:# plural name to be used in the URL: /apis/<group>/<version>/<plural>plural:crontabs# singular name to be used as an alias on the CLI and for displaysingular:crontab# kind is normally the CamelCased singular type. Your resource manifests use this.kind:CronTab# shortNames allow shorter string to match your resource on the CLIshortNames:- ct

and create it:

kubectl apply -f resourcedefinition.yaml

Then a new namespaced RESTful API endpoint is created at:

/apis/stable.example.com/v1/namespaces/*/crontabs/...

This endpoint URL can then be used to create and manage custom objects.Thekind of these objects will beCronTab from the spec of theCustomResourceDefinition object you created above.

It might take a few seconds for the endpoint to be created.You can watch theEstablished condition of your CustomResourceDefinitionto be true or watch the discovery information of the API server for yourresource to show up.

Create custom objects

After the CustomResourceDefinition object has been created, you can createcustom objects. Custom objects can contain custom fields. These fields cancontain arbitrary JSON.In the following example, thecronSpec andimage custom fields are set in acustom object of kindCronTab. The kindCronTab comes from the spec of theCustomResourceDefinition object you created above.

If you save the following YAML tomy-crontab.yaml:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:cronSpec:"* * * * */5"image:my-awesome-cron-image

and create it:

kubectl apply -f my-crontab.yaml

You can then manage your CronTab objects using kubectl. For example:

kubectl get crontab

Should print a list like this:

NAME                 AGEmy-new-cron-object   6s

Resource names are not case-sensitive when using kubectl, and you can use eitherthe singular or plural forms defined in the CRD, as well as any short names.

You can also view the raw YAML data:

kubectl get ct -o yaml

You should see that it contains the customcronSpec andimage fieldsfrom the YAML you used to create it:

apiVersion:v1items:-apiVersion:stable.example.com/v1kind:CronTabmetadata:annotations:kubectl.kubernetes.io/last-applied-configuration:|        {"apiVersion":"stable.example.com/v1","kind":"CronTab","metadata":{"annotations":{},"name":"my-new-cron-object","namespace":"default"},"spec":{"cronSpec":"* * * * */5","image":"my-awesome-cron-image"}}creationTimestamp:"2021-06-20T07:35:27Z"generation:1name:my-new-cron-objectnamespace:defaultresourceVersion:"1326"uid:9aab1d66-628e-41bb-a422-57b8b3b1f5a9spec:cronSpec:'* * * * */5'image:my-awesome-cron-imagekind:Listmetadata:resourceVersion:""selfLink:""

Delete a CustomResourceDefinition

When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpointand delete all custom objects stored in it.

kubectl delete -f resourcedefinition.yamlkubectl get crontabs

Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontabs"}: the server could notfind the requested resource (get crontabs.stable.example.com)

If you later recreate the same CustomResourceDefinition, it will start out empty.

Specifying a structural schema

CustomResources store structured data in custom fields (alongside the built-infieldsapiVersion,kind andmetadata, which the API server validatesimplicitly). WithOpenAPI v3.0 validation a schema can bespecified, which is validated during creation and updates, compare below fordetails and limits of such a schema.

Withapiextensions.k8s.io/v1 the definition of a structural schema ismandatory for CustomResourceDefinitions. In the beta version ofCustomResourceDefinition, the structural schema was optional.

A structural schema is anOpenAPI v3.0 validation schema which:

1. specifies a non-empty type (viatype in OpenAPI) for the root, for each specified field of an object node(viaproperties oradditionalProperties in OpenAPI) and for each item in an array node(viaitems in OpenAPI), with the exception of:
   • a node withx-kubernetes-int-or-string: true
   • a node withx-kubernetes-preserve-unknown-fields: true
2. for each field in an object and each item in an array which is specified within any ofallOf,anyOf,oneOf ornot, the schema also specifies the field/item outside of those logical junctors (compare example 1 and 2).
3. does not setdescription,type,default,additionalProperties,nullable within anallOf,anyOf,oneOf ornot, with the exception of the two pattern forx-kubernetes-int-or-string: true (see below).
4. ifmetadata is specified, then only restrictions onmetadata.name andmetadata.generateName are allowed.

Non-structural example 1:

allOf:-properties:foo:# ...

conflicts with rule 2. The following would be correct:

properties:foo:# ...allOf:-properties:foo:# ...

Non-structural example 2:

allOf:-items:properties:foo:# ...

conflicts with rule 2. The following would be correct:

items:properties:foo:# ...allOf:-items:properties:foo:# ...

Non-structural example 3:

properties:foo:pattern:"abc"metadata:type:objectproperties:name:type:stringpattern:"^a"finalizers:type:arrayitems:type:stringpattern:"my-finalizer"anyOf:-properties:bar:type:integerminimum:42required:["bar"]description:"foo bar object"

is not a structural schema because of the following violations:

• the type at the root is missing (rule 1).
• the type offoo is missing (rule 1).
• bar inside ofanyOf is not specified outside (rule 2).
• bar'stype is withinanyOf (rule 3).
• the description is set withinanyOf (rule 3).
• metadata.finalizers might not be restricted (rule 4).

In contrast, the following, corresponding schema is structural:

type:objectdescription:"foo bar object"properties:foo:type:stringpattern:"abc"bar:type:integermetadata:type:objectproperties:name:type:stringpattern:"^a"anyOf:-properties:bar:minimum:42required:["bar"]

Violations of the structural schema rules are reported in theNonStructural condition in theCustomResourceDefinition.

Field pruning

CustomResourceDefinitions store validated resource data in the cluster's persistence store,etcd.As with native Kubernetes resources such asConfigMap,if you specify a field that the API server does not recognize, the unknown field ispruned (removed) before being persisted.

CRDs converted fromapiextensions.k8s.io/v1beta1 toapiextensions.k8s.io/v1 might lackstructural schemas, andspec.preserveUnknownFields might betrue.

For legacy CustomResourceDefinition objects created asapiextensions.k8s.io/v1beta1 withspec.preserveUnknownFields set totrue, the following is also true:

• Pruning is not enabled.
• You can store arbitrary data.

For compatibility withapiextensions.k8s.io/v1, update your customresource definitions to:

1. Use a structural OpenAPI schema.
2. Setspec.preserveUnknownFields tofalse.

If you save the following YAML tomy-crontab.yaml:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:cronSpec:"* * * * */5"image:my-awesome-cron-imagesomeRandomField:42

and create it:

kubectl create --validate=false -f my-crontab.yaml -o yaml

Your output is similar to:

apiVersion:stable.example.com/v1kind:CronTabmetadata:creationTimestamp:2017-05-31T12:56:35Zgeneration:1name:my-new-cron-objectnamespace:defaultresourceVersion:"285"uid:9423255b-4600-11e7-af6a-28d2447dc82bspec:cronSpec:'* * * * */5'image:my-awesome-cron-image

Notice that the fieldsomeRandomField was pruned.

This example turned off client-side validation to demonstrate the API server's behavior, by addingthe--validate=false command line option.Because theOpenAPI validation schemas are also publishedto clients,kubectl also checks for unknown fields and rejects those objects well before theywould be sent to the API server.

Controlling pruning

By default, all unspecified fields for a custom resource, across all versions, are pruned. It is possible though toopt-out of that for specific sub-trees of fields by addingx-kubernetes-preserve-unknown-fields: true in thestructural OpenAPI v3 validation schema.

For example:

type:objectproperties:json:x-kubernetes-preserve-unknown-fields:true

The fieldjson can store any JSON value, without anything being pruned.

You can also partially specify the permitted JSON; for example:

type:objectproperties:json:x-kubernetes-preserve-unknown-fields:truetype:objectdescription:this is arbitrary JSON

With this, onlyobject type values are allowed.

Pruning is enabled again for each specified property (oradditionalProperties):

type:objectproperties:json:x-kubernetes-preserve-unknown-fields:truetype:objectproperties:spec:type:objectproperties:foo:type:stringbar:type:string

With this, the value:

json:spec:foo:abcbar:defsomething:xstatus:something:x

is pruned to:

json:spec:foo:abcbar:defstatus:something:x

This means that thesomething field in the specifiedspec object is pruned, but everything outside is not.

IntOrString

Nodes in a schema withx-kubernetes-int-or-string: true are excluded from rule 1, such that thefollowing is structural:

type:objectproperties:foo:x-kubernetes-int-or-string:true

Also those nodes are partially excluded from rule 3 in the sense that the following two patterns are allowed(exactly those, without variations in order to additional fields):

x-kubernetes-int-or-string:trueanyOf:-type:integer-type:string...

and

x-kubernetes-int-or-string:trueallOf:-anyOf:-type:integer-type:string-# ... zero or more...

With one of those specification, both an integer and a string validate.

InValidation Schema Publishing,x-kubernetes-int-or-string: true is unfolded to one of the two patterns shown above.

RawExtension

RawExtensions (as inruntime.RawExtension)holds complete Kubernetes objects, i.e. withapiVersion andkind fields.

It is possible to specify those embedded objects (both completely without constraints or partially specified)by settingx-kubernetes-embedded-resource: true. For example:

type:objectproperties:foo:x-kubernetes-embedded-resource:truex-kubernetes-preserve-unknown-fields:true

Here, the fieldfoo holds a complete object, e.g.:

foo:apiVersion:v1kind:Podspec:# ...

Becausex-kubernetes-preserve-unknown-fields: true is specified alongside, nothing is pruned.The use ofx-kubernetes-preserve-unknown-fields: true is optional though.

Withx-kubernetes-embedded-resource: true, theapiVersion,kind andmetadata are implicitly specified and validated.

Serving multiple versions of a CRD

SeeCustom resource definition versioningfor more information about serving multiple versions of yourCustomResourceDefinition and migrating your objects from one version to another.

Advanced topics

Finalizers

Finalizers allow controllers to implement asynchronous pre-delete hooks.Custom objects support finalizers similar to built-in objects.

You can add a finalizer to a custom object like this:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:finalizers:- stable.example.com/finalizer

Identifiers of custom finalizers consist of a domain name, a forward slash and the name ofthe finalizer. Any controller can add a finalizer to any object's list of finalizers.

The first delete request on an object with finalizers sets a value for themetadata.deletionTimestamp field but does not delete it. Once this value is set,entries in thefinalizers list can only be removed. While any finalizers remain it is alsoimpossible to force the deletion of an object.

When themetadata.deletionTimestamp field is set, controllers watching the object execute anyfinalizers they handle and remove the finalizer from the list after they are done. It is theresponsibility of each controller to remove its finalizer from the list.

The value ofmetadata.deletionGracePeriodSeconds controls the interval between polling updates.

Once the list of finalizers is empty, meaning all finalizers have been executed, the resource isdeleted by Kubernetes.

Validation

Custom resources are validated viaOpenAPI v3.0 schemas,by x-kubernetes-validations when theValidation Rules feature is enabled, and youcan add additional validation usingadmission webhooks.

Additionally, the following restrictions are applied to the schema:

• These fields cannot be set:

  • definitions,
  • dependencies,
  • deprecated,
  • discriminator,
  • id,
  • patternProperties,
  • readOnly,
  • writeOnly,
  • xml,
  • $ref.

• The fielduniqueItems cannot be set totrue.

• The fieldadditionalProperties cannot be set tofalse.

• The fieldadditionalProperties is mutually exclusive withproperties.

Thex-kubernetes-validations extension can be used to validate custom resources usingCommon Expression Language (CEL) expressions when theValidation rules feature is enabled and the CustomResourceDefinition schema is astructural schema.

Refer to thestructural schemas section for otherrestrictions and CustomResourceDefinition features.

The schema is defined in the CustomResourceDefinition. In the following example, theCustomResourceDefinition applies the following validations on the custom object:

• spec.cronSpec must be a string and must be of the form described by the regular expression.
• spec.replicas must be an integer and must have a minimum value of 1 and a maximum value of 10.

Save the CustomResourceDefinition toresourcedefinition.yaml:

apiVersion:apiextensions.k8s.io/v1kind:CustomResourceDefinitionmetadata:name:crontabs.stable.example.comspec:group:stable.example.comversions:-name:v1served:truestorage:trueschema:# openAPIV3Schema is the schema for validating custom objects.openAPIV3Schema:type:objectproperties:spec:type:objectproperties:cronSpec:type:stringpattern:'^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'image:type:stringreplicas:type:integerminimum:1maximum:10scope:Namespacednames:plural:crontabssingular:crontabkind:CronTabshortNames:- ct

and create it:

kubectl apply -f resourcedefinition.yaml

A request to create a custom object of kind CronTab is rejected if there are invalid values in its fields.In the following example, the custom object contains fields with invalid values:

• spec.cronSpec does not match the regular expression.
• spec.replicas is greater than 10.

If you save the following YAML tomy-crontab.yaml:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:cronSpec:"* * * *"image:my-awesome-cron-imagereplicas:15

and attempt to create it:

kubectl apply -f my-crontab.yaml

then you get an error:

The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:validation failure list:spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'spec.replicas in body should be less than or equal to 10

If the fields contain valid values, the object creation request is accepted.

Save the following YAML tomy-crontab.yaml:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:cronSpec:"* * * * */5"image:my-awesome-cron-imagereplicas:5

And create it:

kubectl apply -f my-crontab.yamlcrontab"my-new-cron-object" created

Validation ratcheting

If you are using a version of Kubernetes older than v1.30, you need to explicitlyenable theCRDValidationRatchetingfeature gate touse this behavior, which then applies to all CustomResourceDefinitions in yourcluster.

Provided you enabled the feature gate, Kubernetes implementsvalidation ratchetingfor CustomResourceDefinitions. The API server is willing to accept updates to resources thatare not valid after the update, provided that each part of the resource that failed to validatewas not changed by the update operation. In other words, any invalid part of the resourcethat remains invalid must have already been wrong.You cannot use this mechanism to update a valid resource so that it becomes invalid.

This feature allows authors of CRDs to confidently add new validations to theOpenAPIV3 schema under certain conditions. Users can update to the new schemasafely without bumping the version of the object or breaking workflows.

While most validations placed in the OpenAPIV3 schema of a CRD supportratcheting, there are a few exceptions. The following OpenAPIV3 schemavalidations are not supported by ratcheting under the implementation in Kubernetes1.34 and if violated will continue to throw an error as normally:

• Quantors

  • allOf
  • oneOf
  • anyOf
  • not
  • any validations in a descendent of one of these fields

• x-kubernetes-validationsFor Kubernetes 1.28, CRDvalidation rules are ignored byratcheting. Starting with Alpha 2 in Kubernetes 1.29,x-kubernetes-validationsare ratcheted only if they do not refer tooldSelf.

  Transition Rules are never ratcheted: only errors raised by rules that do notuseoldSelf will be automatically ratcheted if their values are unchanged.

  To write custom ratcheting logic for CEL expressions, check outoptionalOldSelf.

• x-kubernetes-list-typeErrors arising from changing the list type of a subschema will not beratcheted. For example addingset onto a list with duplicates will alwaysresult in an error.

• x-kubernetes-list-map-keysErrors arising from changing the map keys of a list schema will not beratcheted.

• requiredErrors arising from changing the list of required fields will not be ratcheted.

• propertiesAdding/removing/modifying the names of properties is not ratcheted, butchanges to validations in each properties' schemas and subschemas may be ratchetedif the name of the property stays the same.

• additionalPropertiesTo remove a previously specifiedadditionalProperties validation will not beratcheted.

• metadataErrors that come from Kubernetes' built-in validation of an object'smetadataare not ratcheted (such as object name, or characters in a label value).If you specify your own additional rules for the metadata of a custom resource,that additional validation will be ratcheted.

Validation rules

Validation rules use theCommon Expression Language (CEL)to validate custom resource values. Validation rules are included inCustomResourceDefinition schemas using thex-kubernetes-validations extension.

The Rule is scoped to the location of thex-kubernetes-validations extension in the schema.Andself variable in the CEL expression is bound to the scoped value.

All validation rules are scoped to the current object: no cross-object or stateful validationrules are supported.

For example:

# ...openAPIV3Schema:type:objectproperties:spec:type:objectx-kubernetes-validations:-rule:"self.minReplicas <= self.replicas"message:"replicas should be greater than or equal to minReplicas."-rule:"self.replicas <= self.maxReplicas"message:"replicas should be smaller than or equal to maxReplicas."properties:# ...minReplicas:type:integerreplicas:type:integermaxReplicas:type:integerrequired:- minReplicas- replicas- maxReplicas

will reject a request to create this custom resource:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:minReplicas:0replicas:20maxReplicas:10

with the response:

The CronTab "my-new-cron-object" is invalid:* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: replicas should be smaller than or equal to maxReplicas.

x-kubernetes-validations could have multiple rules.Therule underx-kubernetes-validations represents the expression which will be evaluated by CEL.Themessage represents the message displayed when validation fails. If message is unset, theabove response would be:

The CronTab "my-new-cron-object" is invalid:* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: failed rule: self.replicas <= self.maxReplicas

Validation rules are compiled when CRDs are created/updated.The request of CRDs create/update will fail if compilation of validation rules fail.Compilation process includes type checking as well.

The compilation failure:

• no_matching_overload: this function has no overload for the types of the arguments.

  For example, a rule likeself == true against a field of integer type will get error:

  Invalid value: apiextensions.ValidationRule{Rule:"self == true", Message:""}: compilation failed: ERROR: \<input>:1:6: found no matching overload for '_==_' applied to '(int, bool)'

• no_such_field: does not contain the desired field.

  For example, a rule likeself.nonExistingField > 0 against a non-existing field will returnthe following error:

  Invalid value: apiextensions.ValidationRule{Rule:"self.nonExistingField > 0", Message:""}: compilation failed: ERROR: \<input>:1:5: undefined field 'nonExistingField'

• invalid argument: invalid argument to macros.

  For example, a rule likehas(self) will return error:

  Invalid value: apiextensions.ValidationRule{Rule:"has(self)", Message:""}: compilation failed: ERROR: <input>:1:4: invalid argument to has() macro

Validation Rules Examples:

Xref:Supported evaluation on CEL

• If the Rule is scoped to the root of a resource, it may make field selection into any fieldsdeclared in the OpenAPIv3 schema of the CRD as well asapiVersion,kind,metadata.name andmetadata.generateName. This includes selection of fields in both thespec andstatus in thesame expression:

  # ...openAPIV3Schema:type:objectx-kubernetes-validations:-rule:"self.status.availableReplicas >= self.spec.minReplicas"properties:spec:type:objectproperties:minReplicas:type:integer# ...status:type:objectproperties:availableReplicas:type:integer

• If the Rule is scoped to an object with properties, the accessible properties of the object are field selectableviaself.field and field presence can be checked viahas(self.field). Null valued fields are treated asabsent fields in CEL expressions.

  # ...openAPIV3Schema:type:objectproperties:spec:type:objectx-kubernetes-validations:-rule:"has(self.foo)"properties:# ...foo:type:integer

• If the Rule is scoped to an object with additionalProperties (i.e. a map) the value of the mapare accessible viaself[mapKey], map containment can be checked viamapKey in self and allentries of the map are accessible via CEL macros and functions such asself.all(...).

  # ...openAPIV3Schema:type:objectproperties:spec:type:objectx-kubernetes-validations:-rule:"self['xyz'].foo > 0"additionalProperties:# ...type:objectproperties:foo:type:integer

• If the Rule is scoped to an array, the elements of the array are accessible viaself[i] andalso by macros and functions.

  # ...openAPIV3Schema:type:objectproperties:# ...foo:type:arrayx-kubernetes-validations:-rule:"size(self) == 1"items:type:string

• If the Rule is scoped to a scalar,self is bound to the scalar value.

  # ...openAPIV3Schema:type:objectproperties:spec:type:objectproperties:# ...foo:type:integerx-kubernetes-validations:-rule:"self > 0"

Examples:

TheapiVersion,kind,metadata.name andmetadata.generateName are always accessible fromthe root of the object and from anyx-kubernetes-embedded-resource annotated objects. No othermetadata properties are accessible.

Unknown data preserved in custom resources viax-kubernetes-preserve-unknown-fields is notaccessible in CEL expressions. This includes:

• Unknown field values that are preserved by object schemas withx-kubernetes-preserve-unknown-fields.

• Object properties where the property schema is of an "unknown type". An "unknown type" isrecursively defined as:

  • A schema with no type and x-kubernetes-preserve-unknown-fields set to true
  • An array where the items schema is of an "unknown type"
  • An object where the additionalProperties schema is of an "unknown type"

Only property names of the form[a-zA-Z_.-/][a-zA-Z0-9_.-/]* are accessible.Accessible property names are escaped according to the following rules when accessed in the expression:

Note: CEL RESERVED keyword needs to match the exact property name to be escaped (e.g. int in the word sprint would not be escaped).

Examples on escaping:

Equality on arrays withx-kubernetes-list-type ofset ormap ignores element order,i.e.,[1, 2] == [2, 1]. Concatenation on arrays with x-kubernetes-list-type use the semantics ofthe list type:

• set:X + Y performs a union where the array positions of all elements inX are preservedand non-intersecting elements inY are appended, retaining their partial order.

• map:X + Y performs a merge where the array positions of all keys inX are preserved butthe values are overwritten by values inY when the key sets ofX andY intersect. ElementsinY with non-intersecting keys are appended, retaining their partial order.

Here is the declarations type mapping between OpenAPIv3 and CEL type:

xref:CEL types,OpenAPI types,Kubernetes Structural Schemas.

The messageExpression field

Similar to themessage field, which defines the string reported for a validation rule failure,messageExpression allows you to use a CEL expression to construct the message string.This allows you to insert more descriptive information into the validation failure message.messageExpression must evaluate a string and may use the same variables that are available to therulefield. For example:

x-kubernetes-validations:-rule:"self.x <= self.maxLimit"messageExpression:'"x exceeded max limit of " + string(self.maxLimit)'

Keep in mind that CEL string concatenation (+ operator) does not auto-cast to string. Ifyou have a non-string scalar, use thestring(<value>) function to cast the scalar to a stringlike shown in the above example.

messageExpression must evaluate to a string, and this is checked while the CRD is being written. Note that it is possibleto setmessage andmessageExpression on the same rule, and if both are present,messageExpressionwill be used. However, ifmessageExpression evaluates to an error, the string defined inmessagewill be used instead, and themessageExpression error will be logged. This fallback will also occur ifthe CEL expression defined inmessageExpression generates an empty string, or a string containing linebreaks.

If one of the above conditions are met and nomessage has been set, then the default validation failuremessage will be used instead.

messageExpression is a CEL expression, so the restrictions listed inResource use by validation functions apply. If evaluation halts due to resource constraintsduringmessageExpression execution, then no further validation rules will be executed.

SettingmessageExpression is optional.

Themessage field

If you want to set a static message, you can supplymessage rather thanmessageExpression.The value ofmessage is used as an opaque error string if validation fails.

Settingmessage is optional.

Thereason field

You can add a machine-readable validation failure reason within avalidation, to be returnedwhenever a request fails this validation rule.

For example:

x-kubernetes-validations:-rule:"self.x <= self.maxLimit"reason:"FieldValueInvalid"

The HTTP status code returned to the caller will match the reason of the first failed validation rule.The currently supported reasons are: "FieldValueInvalid", "FieldValueForbidden", "FieldValueRequired", "FieldValueDuplicate".If not set or unknown reasons, default to use "FieldValueInvalid".

Settingreason is optional.

ThefieldPath field

You can specify the field path returned when the validation fails.

For example:

x-kubernetes-validations:-rule:"self.foo.test.x <= self.maxLimit"fieldPath:".foo.test.x"

In the example above, the validation checks the value of fieldx should be less than the value ofmaxLimit.If nofieldPath specified, when validation fails, the fieldPath would be default to whereverself scoped.WithfieldPath specified, the returned error will havefieldPath properly refer to the location of fieldx.

ThefieldPath value must be a relative JSON path that is scoped to the location of this x-kubernetes-validations extension in the schema.Additionally, it should refer to an existing field within the schema.For example when validation checks if a specific attributefoo under a maptestMap, you could setfieldPath to".testMap.foo" or.testMap['foo']'.If the validation requires checking for unique attributes in two lists, the fieldPath can be set to either of the lists.For example, it can be set to.testList1 or.testList2.It supports child operation to refer to an existing field currently.Refer toJSONPath support in Kubernetes for more info.ThefieldPath field does not support indexing arrays numerically.

SettingfieldPath is optional.

TheoptionalOldSelf field

If your cluster does not haveCRD validation ratcheting enabled,the CustomResourceDefinition API doesn't include this field, and trying to set it may resultin an error.

TheoptionalOldSelf field is a boolean field that alters the behavior ofTransition Rules describedbelow. Normally, a transition rule will not evaluate ifoldSelf cannot be determined:during object creation or when a new value is introduced in an update.

IfoptionalOldSelf is set to true, then transition rules will always beevaluated and the type ofoldSelf be changed to a CELOptional type.

optionalOldSelf is useful in cases where schema authors would like a morecontrol toolthan provided by the default equality based behavior ofto introduce newer, usually stricter constraints on new values, while stillallowing old values to be "grandfathered" or ratcheted using the older validation.

Example Usage:

Validation functions

Functions available include:

• CEL standard functions, defined in thelist of standard definitions
• CEL standardmacros
• CELextended string function library
• KubernetesCEL extension library

Transition rules

A rule that contains an expression referencing the identifieroldSelf is implicitly considered atransition rule. Transition rules allow schema authors to prevent certain transitions between twootherwise valid states. For example:

type:stringenum:["low","medium","high"]x-kubernetes-validations:-rule:"!(self == 'high' && oldSelf == 'low') && !(self == 'low' && oldSelf == 'high')"message:cannot transition directly between 'low' and 'high'

Unlike other rules, transition rules apply only to operations meeting the following criteria:

• The operation updates an existing object. Transition rules never apply to create operations.

• Both an old and a new value exist. It remains possible to check if a value has been added orremoved by placing a transition rule on the parent node. Transition rules are never applied tocustom resource creation. When placed on an optional field, a transition rule will not apply toupdate operations that set or unset the field.

• The path to the schema node being validated by a transition rule must resolve to a node that iscomparable between the old object and the new object. For example, list items and theirdescendants (spec.foo[10].bar) can't necessarily be correlated between an existing object and alater update to the same object.

Errors will be generated on CRD writes if a schema node contains a transition rule that can never beapplied, e.g. "oldSelf cannot be used on the uncorrelatable portion of the schema withinpath".

Transition rules are only allowed oncorrelatable portions of a schema.A portion of the schema is correlatable if allarray parent schemas are of typex-kubernetes-list-type=map;anysetoratomicarray parent schemas make it impossible to unambiguously correlate aself witholdSelf.

Here are some examples for transition rules:

Resource use by validation functions

When you create or update a CustomResourceDefinition that uses validation rules,the API server checks the likely impact of running those validation rules. If a rule isestimated to be prohibitively expensive to execute, the API server rejects the createor update operation, and returns an error message.A similar system is used at runtime that observes the actions the interpreter takes. If the interpreter executestoo many instructions, execution of the rule will be halted, and an error will result.Each CustomResourceDefinition is also allowed a certain amount of resources to finish executing all ofits validation rules. If the sum total of its rules are estimated at creation time to go over that limit,then a validation error will also occur.

You are unlikely to encounter issues with the resource budget for validation if you onlyspecify rules that always take the same amount of time regardless of how large their input is.For example, a rule that asserts thatself.foo == 1 does not by itself have anyrisk of rejection on validation resource budget groups.But iffoo is a string and you define a validation ruleself.foo.contains("someString"), that rule takeslonger to execute depending on how longfoo is.Another example would be iffoo were an array, and you specified a validation ruleself.foo.all(x, x > 5).The cost system always assumes the worst-case scenario if a limit on the length offoo is notgiven, and this will happen for anything that can be iterated over (lists, maps, etc.).

Because of this, it is considered best practice to put a limit viamaxItems,maxProperties, andmaxLength for anything that will be processed in a validation rule in order to prevent validationerrors during cost estimation. For example, given this schema with one rule:

openAPIV3Schema:type:objectproperties:foo:type:arrayitems:type:stringx-kubernetes-validations:-rule:"self.all(x, x.contains('a string'))"

then the API server rejects this rule on validation budget grounds with error:

spec.validation.openAPIV3Schema.properties[spec].properties[foo].x-kubernetes-validations[0].rule: Forbidden:CEL rule exceeded budget by more than 100x (try simplifying the rule, or adding maxItems, maxProperties, andmaxLength where arrays, maps, and strings are used)

The rejection happens becauseself.all implies callingcontains() on every string infoo,which in turn will check the given string to see if it contains'a string'. Without limits, thisis a very expensive rule.

If you do not specify any validation limit, the estimated cost of this rule will exceed theper-rule cost limit. But if you add limits in the appropriate places, the rule will be allowed:

openAPIV3Schema:type:objectproperties:foo:type:arraymaxItems:25items:type:stringmaxLength:10x-kubernetes-validations:-rule:"self.all(x, x.contains('a string'))"

The cost estimation system takes into account how many times the rule will be executed in addition to theestimated cost of the rule itself. For instance, the following rule will have the same estimated cost as theprevious example (despite the rule now being defined on the individual array items):

openAPIV3Schema:type:objectproperties:foo:type:arraymaxItems:25items:type:stringx-kubernetes-validations:-rule:"self.contains('a string'))"maxLength:10

If a list inside of a list has a validation rule that usesself.all, that is significantly more expensivethan a non-nested list with the same rule. A rule that would have been allowed on a non-nested list might needlower limits set on both nested lists in order to be allowed. For example, even without having limits set,the following rule is allowed:

openAPIV3Schema:type:objectproperties:foo:type:arrayitems:type:integerx-kubernetes-validations:-rule:"self.all(x, x == 5)"

But the same rule on the following schema (with a nested array added) produces a validation error:

openAPIV3Schema:type:objectproperties:foo:type:arrayitems:type:arrayitems:type:integerx-kubernetes-validations:-rule:"self.all(x, x == 5)"

This is because each item offoo is itself an array, and each subarray in turn callsself.all.Avoid nested lists and maps if possible where validation rules are used.

Defaulting

Defaulting allows to specify default values in theOpenAPI v3 validation schema:

apiVersion:apiextensions.k8s.io/v1kind:CustomResourceDefinitionmetadata:name:crontabs.stable.example.comspec:group:stable.example.comversions:-name:v1served:truestorage:trueschema:# openAPIV3Schema is the schema for validating custom objects.openAPIV3Schema:type:objectproperties:spec:type:objectproperties:cronSpec:type:stringpattern:'^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'default:"5 0 * * *"image:type:stringreplicas:type:integerminimum:1maximum:10default:1scope:Namespacednames:plural:crontabssingular:crontabkind:CronTabshortNames:- ct

With this bothcronSpec andreplicas are defaulted:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:image:my-awesome-cron-image

leads to

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:cronSpec:"5 0 * * *"image:my-awesome-cron-imagereplicas:1

Defaulting happens on the object

• in the request to the API server using the request version defaults,
• when reading from etcd using the storage version defaults,
• after mutating admission plugins with non-empty patches using the admission webhook object version defaults.

Defaults applied when reading data from etcd are not automatically written back to etcd.An update request via the API is required to persist those defaults back into etcd.

Default values for non-leaf fields must be pruned (with the exception of defaults formetadata fields) and mustvalidate against a provided schema. For example in the above example, a default of{"replicas": "foo", "badger": 1}for thespec field would be invalid, becausebadger is an unknown field, andreplicas is not a string.

Default values formetadata fields ofx-kubernetes-embedded-resources: true nodes (or parts ofa default value coveringmetadata) are not pruned during CustomResourceDefinition creation, butthrough the pruning step during handling of requests.

Defaulting and Nullable

Null values for fields that either don't specify the nullable flag, or give it afalse value, will be pruned before defaulting happens. If a default is present, it will beapplied. When nullable istrue, null values will be conserved and won't be defaulted.

For example, given the OpenAPI schema below:

type:objectproperties:spec:type:objectproperties:foo:type:stringnullable:falsedefault:"default"bar:type:stringnullable:truebaz:type:string

creating an object with null values forfoo andbar andbaz

spec:foo:nullbar:nullbaz:null

leads to

spec:foo:"default"bar:null

withfoo pruned and defaulted because the field is non-nullable,bar maintaining the nullvalue due tonullable: true, andbaz pruned because the field is non-nullable and has nodefault.

Publish Validation Schema in OpenAPI

CustomResourceDefinitionOpenAPI v3 validation schemas which arestructural andenable pruning are publishedasOpenAPI v3 andOpenAPI v2 from Kubernetes API server. It is recommended to use the OpenAPI v3 documentas it is a lossless representation of the CustomResourceDefinition OpenAPI v3 validation schemawhile OpenAPI v2 represents a lossy conversion.

Thekubectl command-line tool consumes the published schema to performclient-side validation (kubectl create andkubectl apply), schema explanation (kubectl explain)on custom resources. The published schema can be consumed for other purposes as well, like client generation or documentation.

Compatibility with OpenAPI V2

For compatibility with OpenAPI V2, the OpenAPI v3 validation schema performs a lossy conversionto the OpenAPI v2 schema. The schema show up indefinitions andpaths fields in theOpenAPI v2 spec.

The following modifications are applied during the conversion to keep backwards compatibility withkubectl in previous 1.13 version. These modifications prevent kubectl from being over-strict and rejectingvalid OpenAPI schemas that it doesn't understand. The conversion won't modify the validation schema defined in CRD,and therefore won't affectvalidation in the API server.

1. The following fields are removed as they aren't supported by OpenAPI v2.

   • The fieldsallOf,anyOf,oneOf andnot are removed

2. Ifnullable: true is set, we droptype,nullable,items andproperties because OpenAPI v2 isnot able to express nullable. To avoid kubectl to reject good objects, this is necessary.

Additional printer columns

The kubectl tool relies on server-side output formatting. Your cluster's API server decides whichcolumns are shown by thekubectl get command. You can customize these columns for aCustomResourceDefinition. The following example adds theSpec,Replicas, andAgecolumns.

Save the CustomResourceDefinition toresourcedefinition.yaml:

apiVersion:apiextensions.k8s.io/v1kind:CustomResourceDefinitionmetadata:name:crontabs.stable.example.comspec:group:stable.example.comscope:Namespacednames:plural:crontabssingular:crontabkind:CronTabshortNames:- ctversions:-name:v1served:truestorage:trueschema:openAPIV3Schema:type:objectproperties:spec:type:objectproperties:cronSpec:type:stringimage:type:stringreplicas:type:integeradditionalPrinterColumns:-name:Spectype:stringdescription:The cron spec defining the interval a CronJob is runjsonPath:.spec.cronSpec-name:Replicastype:integerdescription:The number of jobs launched by the CronJobjsonPath:.spec.replicas-name:Agetype:datejsonPath:.metadata.creationTimestamp

Create the CustomResourceDefinition:

kubectl apply -f resourcedefinition.yaml

Create an instance using themy-crontab.yaml from the previous section.

Invoke the server-side printing:

kubectl get crontab my-new-cron-object

Notice theNAME,SPEC,REPLICAS, andAGE columns in the output:

NAME                 SPEC        REPLICAS   AGEmy-new-cron-object   * * * * *   1          7s

Priority

Each column includes apriority field. Currently, the prioritydifferentiates between columns shown in standard view or wide view (using the-o wide flag).

• Columns with priority0 are shown in standard view.
• Columns with priority greater than0 are shown only in wide view.

Type

A column'stype field can be any of the following (compareOpenAPI v3 data types):

• integer – non-floating-point numbers
• number – floating point numbers
• string – strings
• boolean –true orfalse
• date – rendered differentially as time since this timestamp.

If the value inside a CustomResource does not match the type specified for the column,the value is omitted. Use CustomResource validation to ensure that the valuetypes are correct.

Format

A column'sformat field can be any of the following:

• int32
• int64
• float
• double
• byte
• date
• date-time
• password

The column'sformat controls the style used whenkubectl prints the value.

Field selectors

Field Selectorslet clients select custom resources based on the value of one or more resourcefields.

All custom resources support themetadata.name andmetadata.namespace fieldselectors.

Fields declared in aCustomResourceDefinitionmay also be used with field selectors when included in thespec.versions[*].selectableFields field of theCustomResourceDefinition.

Selectable fields for custom resources

Thespec.versions[*].selectableFields field of aCustomResourceDefinition may be used todeclare which other fields in a custom resource may be used in field selectorswith the feature ofCustomResourceFieldSelectorsfeature gate (This feature gate is enabled by default since Kubernetes v1.31).The following example adds the.spec.color and.spec.size fields asselectable fields.

Save the CustomResourceDefinition toshirt-resource-definition.yaml:

customresourcedefinition/shirt-resource-definition.yaml

apiVersion:apiextensions.k8s.io/v1kind:CustomResourceDefinitionmetadata:name:shirts.stable.example.comspec:group:stable.example.comscope:Namespacednames:plural:shirtssingular:shirtkind:Shirtversions:-name:v1served:truestorage:trueschema:openAPIV3Schema:type:objectproperties:spec:type:objectproperties:color:type:stringsize:type:stringselectableFields:-jsonPath:.spec.color-jsonPath:.spec.sizeadditionalPrinterColumns:-jsonPath:.spec.colorname:Colortype:string-jsonPath:.spec.sizename:Sizetype:string

Create the CustomResourceDefinition:

kubectl apply -f https://k8s.io/examples/customresourcedefinition/shirt-resource-definition.yaml

Define some Shirts by editingshirt-resources.yaml; for example:

customresourcedefinition/shirt-resources.yaml

---apiVersion:stable.example.com/v1kind:Shirtmetadata:name:example1spec:color:bluesize:S---apiVersion:stable.example.com/v1kind:Shirtmetadata:name:example2spec:color:bluesize:M---apiVersion:stable.example.com/v1kind:Shirtmetadata:name:example3spec:color:greensize:M

Create the custom resources:

kubectl apply -f https://k8s.io/examples/customresourcedefinition/shirt-resources.yaml

Get all the resources:

kubectl get shirts.stable.example.com

The output is:

NAME       COLOR  SIZEexample1   blue   Sexample2   blue   Mexample3   green  M

Fetch blue shirts (retrieve Shirts with acolor ofblue):

kubectl get shirts.stable.example.com --field-selector spec.color=blue

Should output:

NAME       COLOR  SIZEexample1   blue   Sexample2   blue   M

Get only resources with acolor ofgreen and asize ofM:

kubectl get shirts.stable.example.com --field-selector spec.color=green,spec.size=M

Should output:

NAME       COLOR  SIZEexample2   blue   M

Subresources

Custom resources support/status and/scale subresources.

The status and scale subresources can be optionally enabled bydefining them in the CustomResourceDefinition.

Status subresource

When the status subresource is enabled, the/status subresource for the custom resource is exposed.

• The status and the spec stanzas are represented by the.status and.spec JSONPathsrespectively inside of a custom resource.

• PUT requests to the/status subresource take a custom resource object and ignore changes toanything except the status stanza.

• PUT requests to the/status subresource only validate the status stanza of the customresource.

• PUT/POST/PATCH requests to the custom resource ignore changes to the status stanza.

• The.metadata.generation value is incremented for all changes, except for changes to.metadata or.status.

• Only the following constructs are allowed at the root of the CRD OpenAPI validation schema:

  • description
  • example
  • exclusiveMaximum
  • exclusiveMinimum
  • externalDocs
  • format
  • items
  • maximum
  • maxItems
  • maxLength
  • minimum
  • minItems
  • minLength
  • multipleOf
  • pattern
  • properties
  • required
  • title
  • type
  • uniqueItems

Scale subresource

When the scale subresource is enabled, the/scale subresource for the custom resource is exposed.Theautoscaling/v1.Scale object is sent as the payload for/scale.

To enable the scale subresource, the following fields are defined in the CustomResourceDefinition.

• specReplicasPath defines the JSONPath inside of a custom resource that corresponds toscale.spec.replicas.

  • It is a required value.
  • Only JSONPaths under.spec and with the dot notation are allowed.
  • If there is no value under thespecReplicasPath in the custom resource,the/scale subresource will return an error on GET.

• statusReplicasPath defines the JSONPath inside of a custom resource that corresponds toscale.status.replicas.

  • It is a required value.
  • Only JSONPaths under.status and with the dot notation are allowed.
  • If there is no value under thestatusReplicasPath in the custom resource,the status replica value in the/scale subresource will default to 0.

• labelSelectorPath defines the JSONPath inside of a custom resource that corresponds toScale.Status.Selector.

  • It is an optional value.
  • It must be set to work with HPA and VPA.
  • Only JSONPaths under.status or.spec and with the dot notation are allowed.
  • If there is no value under thelabelSelectorPath in the custom resource,the status selector value in the/scale subresource will default to the empty string.
  • The field pointed by this JSON path must be a string field (not a complex selector struct)which contains a serialized label selector in string form.

In the following example, both status and scale subresources are enabled.

Save the CustomResourceDefinition toresourcedefinition.yaml:

apiVersion:apiextensions.k8s.io/v1kind:CustomResourceDefinitionmetadata:name:crontabs.stable.example.comspec:group:stable.example.comversions:-name:v1served:truestorage:trueschema:openAPIV3Schema:type:objectproperties:spec:type:objectproperties:cronSpec:type:stringimage:type:stringreplicas:type:integerstatus:type:objectproperties:replicas:type:integerlabelSelector:type:string# subresources describes the subresources for custom resources.subresources:# status enables the status subresource.status:{}# scale enables the scale subresource.scale:# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.specReplicasPath:.spec.replicas# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.statusReplicasPath:.status.replicas# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.labelSelectorPath:.status.labelSelectorscope:Namespacednames:plural:crontabssingular:crontabkind:CronTabshortNames:- ct

And create it:

kubectl apply -f resourcedefinition.yaml

After the CustomResourceDefinition object has been created, you can create custom objects.

If you save the following YAML tomy-crontab.yaml:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:cronSpec:"* * * * */5"image:my-awesome-cron-imagereplicas:3

and create it:

kubectl apply -f my-crontab.yaml

Then new namespaced RESTful API endpoints are created at:

/apis/stable.example.com/v1/namespaces/*/crontabs/status

and

/apis/stable.example.com/v1/namespaces/*/crontabs/scale

A custom resource can be scaled using thekubectl scale command.For example, the following command sets.spec.replicas of thecustom resource created above to 5:

kubectl scale --replicas=5 crontabs/my-new-cron-objectcrontabs"my-new-cron-object" scaledkubectl get crontabs my-new-cron-object -ojsonpath='{.spec.replicas}'5

You can use aPodDisruptionBudget to protect customresources that have the scale subresource enabled.

Categories

Categories is a list of grouped resources the custom resource belongs to (eg.all).You can usekubectl get <category-name> to list the resources belonging to the category.

The following example addsall in the list of categories in the CustomResourceDefinitionand illustrates how to output the custom resource usingkubectl get all.

Save the following CustomResourceDefinition toresourcedefinition.yaml:

apiVersion:apiextensions.k8s.io/v1kind:CustomResourceDefinitionmetadata:name:crontabs.stable.example.comspec:group:stable.example.comversions:-name:v1served:truestorage:trueschema:openAPIV3Schema:type:objectproperties:spec:type:objectproperties:cronSpec:type:stringimage:type:stringreplicas:type:integerscope:Namespacednames:plural:crontabssingular:crontabkind:CronTabshortNames:- ct# categories is a list of grouped resources the custom resource belongs to.categories:- all

and create it:

kubectl apply -f resourcedefinition.yaml

After the CustomResourceDefinition object has been created, you can create custom objects.

Save the following YAML tomy-crontab.yaml:

apiVersion:"stable.example.com/v1"kind:CronTabmetadata:name:my-new-cron-objectspec:cronSpec:"* * * * */5"image:my-awesome-cron-image

and create it:

kubectl apply -f my-crontab.yaml

You can specify the category when usingkubectl get:

kubectl get all

and it will include the custom resources of kindCronTab:

NAME                          AGEcrontabs/my-new-cron-object   3s

What's next

• Read aboutcustom resources.

• SeeCustomResourceDefinition.

• Servemultiple versions of aCustomResourceDefinition.