Google Kubernetes Engine (GKE) clusters use containerdnodeimages withall worker nodes that run version 1.24 and later. The worker nodes use aspecific version of containerd, based on the operating system andGKE minor version:

• Container-Optimized OS and Ubuntu nodes (Linux):
  • Linux nodes that run GKE 1.32 or earlier, with containerdnode images, use containerd 1.7 or earlier versions.
  • Linux nodes that run GKE 1.33 use containerd 2.0.
• Windows Server nodes:
  • Windows Server nodes that run GKE 1.34 or earlier, withcontainerd node images, use containerd 1.7 or earlier versions.
  • Windows Server nodes that run GKE 1.35 use containerd2.0.

When GKE nodes are upgraded to the respective GKEminor version, the nodes migrate from using containerd 1.7 to the new majorversion, containerd 2.0. You can't change which containerd version aGKE version uses.

You can skip reading this page if you know that your workloads run as expectedon containerd 2.

How GKE is transitioning to containerd 2

Review the following timeline to understand how GKE istransitioning existing clusters to use containerd 2:

• For Linux nodes with 1.32 and Windows Server nodes with 1.34,GKE uses containerd 1.7. containerd1.7 deprecated both Docker Schema 1 images and the Container RuntimeInterface (CRI) v1alpha2 API. To learn about other features deprecated inearlier versions, seeDeprecated configproperties.
• For Linux nodes with 1.33 and Windows Server nodes with 1.35,GKE uses containerd 2.0, which removes support for DockerSchema 1 images and the CRI v1alpha2 API.
• The following containerd config properties in the CRI plugin are deprecatedand will be removed in containerd 2.2, with a GKE version yetto be announced:registry.auths,registry.configs,registry.mirrors.registry.configs.tls, however, was already removed in containerd 2.0.

For approximate timing of automatic upgrades to later minor versions, see theEstimated schedule for releasechannels.

Impact of the transition to containerd 2

Read the following section to understand the impact of this transition tocontainerd 2.

Paused automatic upgrades

GKE pauses automatic upgrades in the following ways, depending onthe cluster's current minor version and if the nodes in the cluster are Linuxnodes or Windows Server nodes.

Paused automatic upgrades for Linux nodes

GKEpauses automaticupgradesto 1.33 for clusters with Linux nodes when it detects that the cluster uses thedeprecated features. However, if your cluster nodes use these features, werecommend creating amaintenanceexclusionto prevent cluster upgrades. The maintenance exclusion ensures that your clusterisn't upgraded if GKE doesn't detect usage.

After you migrate from using these features, GKE resumesautomatic minor upgrades to 1.33 if the following are true:

• GKE hasn't detected usage of deprecated features in 14 days,or 3 days for deprecated CRIregistry.configs properties.
• 1.33 is an automatic upgrade target for your cluster.
• There are no other blocking factors. For more information, seeThe timingof automaticupgrades.

You can alsomanually upgrade thecluster.

Paused automatic upgrades for Windows Server nodes

GKE pauses automatic upgrades to 1.35 for any clusters withWindows Server nodes, regardless of whether the cluster uses the deprecatedfeatures. GKE can't detect whether Windows Server nodes areusing the deprecated features.

To upgrade your cluster with Windows Server nodes to 1.35, first follow theinstructions toMigrate from deprecated features. After you'vefollowed these instructions, you canmanually upgrade thecluster to 1.35.

End of support and the impact of failing to prepare for migration

GKE pauses automatic upgrades until theend of standardsupport. If your clusteris enrolled in theExtendedchannel, your nodescan remain on a version until theend of extendedsupport. For moredetails about automatic node upgrades at the end of support, seeAutomaticupgrades at the end ofsupport.

If you don't migrate from these features, when 1.32 (for Linux nodes) or 1.34(for Windows Server nodes) reaches the end of support, and your cluster nodesare automatically upgraded to 1.33 or 1.35, you could experience the followingissues with your clusters:

• Workloads using Docker Schema 1 images fail.
• Applications calling the CRI v1alpha2 API experience failures calling theAPI.

Identify affected clusters

GKE monitors your clusters and uses theRecommenderservice to deliver guidance through insights andrecommendations for identifying Linux nodes in your cluster that use thesedeprecated features. GKE doesn't detect usage from WindowsServer nodes.

Version requirements

Clusters receive these insights and recommendations if they're running thefollowing versions or later:

• 1.28.15-gke.1159000
• 1.29.9-gke.1541000
• 1.30.5-gke.1355000
• 1.31.1-gke.1621000

Get insights and recommendations

Follow the instructions toview insights andrecommendations.You can get insights using the Google Cloud console. You can also use theGoogle Cloud CLI or the Recommender API, by filtering with the followingsubtypes:

• DEPRECATION_CONTAINERD_V1_SCHEMA_IMAGES: Docker Schema 1 images
• DEPRECATION_CONTAINERD_V1ALPHA2_CRI_API: CRI v1alpha2 API
• DEPRECATION_CONTAINERD_V2_CONFIG_REGISTRY_CONFIGS: Deprecated CRIregistry.configs properties, includingregistry.configs.auth andregistry.configs.tls

Migrate from deprecated features

Review the following content to understand how to migrate from featuresdeprecated with containerd 2.

Migrate from Docker Schema 1 images

Identify workloads using images that must be migrated, then migrate thoseworkloads.

A Google-provided image affected by this issue isgcr.io/google-containers/startup-script.

• gcr.io/google-containers/startup-script:v1: uses the deprecated Schema 1format and can't be pulled on GKE 1.33 and later for Linuxnodes.
• gcr.io/google-containers/startup-script:v2: uses the supported Schema 2format and can be pulled successfully.

If you are usinggcr.io/google-containers/startup-script:v1 in any of yourworkloads (such as DaemonSets or Deployments), you must update the imagereference togcr.io/google-containers/startup-script:v2.

Find images to be migrated

You can use different tools to find images that must be migrated.

Use insights and recommendations or Cloud Logging

As explained in theIdentify affected clusters section,you can use insights and recommendations to find clusters with Linux nodes thatuse Docker Schema 1 images if your cluster is running a minimum version orlater. Additionally, you can use the following query in Cloud Logging tocheck containerd logs to find Docker Schema 1 images in your cluster:

jsonPayload.SYSLOG_IDENTIFIER="containerd""conversion from schema 1 images is deprecated"

If more than 30 days have passed since the image was pulled, you might not seelogs for an image.

Use thectr command directly on a node

To query a specific node to return all non-deleted images that were pulled asSchema 1, run the following command on a node:

ctr--namespacek8s.ioimageslist'labels."io.containerd.image/converted-docker-schema1"'

This command can be useful if, for example, you're troubleshooting a specificnode and you don't see log entries in Cloud Logging because it's been morethan 30 days since the image was pulled.

Use thecrane open-source tool

You can also use open-source tools such ascraneto check for images.

Run the followingcrane command to check the schema version for an image:

cranemanifest$tagged_image|jq.schemaVersion

Prepare workloads

To prepare workloads that run Docker Schema 1 images, you must migrate thoseworkloads to Schema 2 Docker images, or Open Container Initiative (OCI) images.Consider the following options for migrating:

• Find a replacement image: you might be able to find a publicly availableopen-source or vendor-provided image.
• Convert the existing image: if you can't find a replacement image, youcan convert existing Docker Schema 1 images to OCI images with the followingsteps:
  1. Pull the Docker image into containerd, which automatically converts it to an OCI image.
  2. Push the new OCI image to your registry.

Migrate from the CRI v1alpha2 API

The CRI v1alpha2 API wasremoved in Kubernetes1.26. You must identifyworkloads that access the containerd socket and update these applications to usethe v1 API.

Identify potentially affected workloads

You can use different techniques to identify workloads that might need to bemigrated. These techniques might generate false positives which you mustfurther investigate to determine that no action is needed.

Use insights and recommendations

You can use insights and recommendations to find clusters with Linux nodes thatuse the v1alpha2 API if your cluster is running a minimum version or later. Formore details, seeIdentify affected clusters.

When viewing insights in the Google Cloud console, see the sidebar panelMigrate your workloads off deprecated CRI v1alpha2 API. TheWorkloads toVerify table in this panel lists workloads thatmight be affected. This listincludes any workloads that are not managed by GKE that havehostPath volumes containing the containerd socket path (for example,/var/run/containerd/containerd.sock or/run/containerd/containerd.sock).

It's important to understand the following:

• The list of workloads to verify can contain false positives. Use it onlyfor investigation. A workload appearing in this list does notdefinitively mean it is using the deprecated API, and the presence of afalse positive willnot pause auto-upgrades. Pausing is based only ontheactually observed usage of the deprecated API.
• This list might be empty or incomplete. An empty or incomplete list canhappen if workloads that use the deprecated API were short-lived and notrunning when GKE performed its periodic check. The presenceof the recommendation itself means that CRI v1alpha2 API usage was detectedon at least one node in your cluster. Auto-upgrades resume after thedeprecated API usage has not been detected for 14 days.

Therefore, we recommend further investigation by using the following methods toconfirm actual API usage.

Check for affected third-party workloads

For third-party software deployed to your clusters, verify that these workloadsdon't use the CRI v1alpha2 API. You might need to contact the respective vendorsto verify which versions of their software are compatible.

Use kubectl

The following command helps you find potentially affected workloads by lookingfor those that access the containerd socket. It uses similar logic to the oneused for theWorkloads to Verify table in the Google Cloud consolerecommendation. It returns workloads not managed by GKE that havehostPath volumes including the socket's path. Like the recommendation, thisquery might return false positives or miss short-lived workloads.

Run the following command:

kubectlgetpods--all-namespaces-ojson|\jq-r'  [    "/", "/var", "/var/","/var/run", "/var/run/",    "/var/run/containerd", "/var/run/containerd/", "/var/run/containerd/containerd.sock",    "/run", "/run/", "/run/containerd", "/run/containerd/",    "/run/containerd/containerd.sock"  ] as $socket_paths |  [    "kube-system", "kube-node-lease", "istio-system", "asm-system",    "gatekeeper-system", "config-management-system", "config-management-monitoring",    "cnrm-system", "hnc-system", "gke-managed-system", "gke-gmp-system",    "gmp-system", "gke-managed-cim"  ] as $excluded_namespaces |  .items[] |  select(    (.spec.volumes[]?.hostPath.path as $p | $socket_paths | index($p))    and    ([.metadata.namespace] | inside($excluded_namespaces) | not)  ) |  .metadata.namespace + "/" + .metadata.name'

Use eBPF tracing to identify API callers

For a more definitive way to identify which workloads running on Linux nodescall the CRI v1alpha2 API, you can deploy two specialized DaemonSets:

• Thecontainerd-socket-tracer logs any process opening a connection to thecontainerd socket, along with the Pod and container details.
• Thecri-v1alpha2-api-deprecation-reporter logs the last time the CRIv1alpha2 API was called.

These tools useExtended Berkeley Packet Filter(eBPF) to trace connections to thecontainerd socket and correlate the connections with actual deprecated APIcalls.

You can't deploy these DaemonSets on Windows Server nodes.

By correlating the timestamps from these two tools, you can pinpoint the exactworkload making the deprecated API call. This method provides a higher degree ofconfidence than checking forhostPath volumes alone, because it observesactual socket connections and API usage.

For detailed instructions about how to deploy and use these tools, and how tointerpret their logs, seeTracing containerd Socket Connections.

If, after using these tools, you are still unable to identify the source of thedeprecated API calls but the recommendation remains active, seeGet support.

After you identify a workload that is using the CRI v1alpha2 API, either throughthe preceding methods or by inspecting your codebase, you must update its codeto use the v1 API.

Update application code

To update your application, remove where the application imports thek8s.io/cri-api/pkg/apis/runtime/v1alpha2 client library and modify the code touse thev1 version of the API. This step involves changing the import path andupdating how your code calls the API.

For example, see the following golang code, which uses the deprecated library:

packagemainimport(...runtimeapi"k8s.io/cri-api/pkg/apis/runtime/v1alpha2")funcfoo(){...client:=runtimeapi.NewRuntimeServiceClient(conn)version,err:=client.Version(ctx,&runtimeapi.VersionRequest{})...}

Here, the application imports the v1alpha2 library and uses it to issue RPCs. Ifthe RPCs use the connection to the containerd socket, then this application iscausing GKE to pause auto-upgrades for the cluster.

Do the following steps to search and update your application code:

1. Identify problematic golang applications by running the following command tosearch for the v1alpha2 import path:

   grep-r"k8s.io/cri-api/pkg/apis/runtime/v1alpha2"

   If the output of this command shows that the v1alpha2 library is used in thefile, you must update the file.

   For example, replace the following application code:

   runtimeapi"k8s.io/cri-api/pkg/apis/runtime/v1alpha2"

2. Update the code to use v1:

   runtimeapi"k8s.io/cri-api/pkg/apis/runtime/v1"

Migrate from deprecated containerd config properties

Theregistry.auths,registry.configs, andregistry.mirrors containerdconfig properties in the CRI plugin are deprecated and will be removed incontainerd 2.2, with a GKE version yet to be announced.registry.configs.tls, however, was already removed in containerd 2.0.

Identify workloads

You can use different techniques to identify workloads that must be migrated.

Use insights and recommendations

As an initial approach, you can use insights and recommendations to findclusters with Linux nodes that use the deprecated containerd config properties.This requires a minimum GKE version. For more information aboutthis approach, seeIdentify affected clusters.

When viewing insights in the Google Cloud console, see the sidebar panelMigrate your containerd configuration off deprecated CRI registry authsfield orMigrate your containerd configuration off deprecated CRI registry mirrorsfield. To find workloads that might access the containerd configuration, checktheWorkloads to Verify section.

Use kubectl

Alternatively, you can use kubectl to identify workloads.

Locate workloads that modify the containerd configuration by checking forworkloads with the following attributes:

• Workloads that contain ahostPath volume that includes the containerdconfig
• Workloads that have a container with privileged access(spec.containers.securityContext.privileged: true) and use the hostprocess ID (PID) namespace (spec.hostPID: true)

This command might return false positives because the workload might accessother files in these directories that aren't the containerd configuration. Or,this command might not return workloads which access the containerdconfiguration file in other, less common, ways.

Run the following command to check for the DaemonSets:

kubectlgetdaemonsets--all-namespaces-ojson|\jq-r'  [    "/", "/etc", "/etc/",    "/etc/containerd", "/etc/containerd/",    "/etc/containerd/config.toml"  ] as $host_paths |  [    "kube-system", "kube-node-lease", "istio-system", "asm-system",    "gatekeeper-system", "config-management-system", "config-management-monitoring",    "cnrm-system", "hnc-system", "gke-managed-system", "gke-gmp-system",    "gmp-system", "gke-managed-cim"  ] as $excluded_namespaces |  .items[] |  select(    ([.metadata.namespace] | inside($excluded_namespaces) | not)    and    (      (any(.spec.template.spec.volumes[]?.hostPath.path; IN($host_paths[])))      or      (        .spec.template.spec.hostPID == true and        any(.spec.template.spec.containers[]; .securityContext?.privileged == true)      )    )  ) |  .metadata.namespace + "/" + .metadata.name'

Migrate from the CRI registryauths orconfigs.auth properties

If your workloads use theauths orconfigs.auth properties in the containerdconfig to authenticate to a private registry for pulling container images, youmust migrate the workloads using those images to theimagePullSecrets fieldinstead. For more information, seePull an Image from a PrivateRegistry.

To identify and migrate workloads that use the deprecatedauths orconfigs.auth properties, review the following instructions.

Locate the authentication details for your registry

You can locate the authentication details for your registry in one of thefollowing ways:

• Review the CRI registryauths andconfigs.auth sections in the/etc/containerd/config.toml file by connecting to a GKEnode.
• Find the workload that modifies your containerd configuration file and seewhat authentication details are included using the previously describedmethods foridentifying workloads.GKE doesn't use these settings for its system workloads.

If you use theregistry.configs.auth property, the authentication detailsmight look like the following:

[plugins."io.containerd.grpc.v1.cri".registry.configs."$REGISTRY_DOMAIN".auth]username="example-user"password="example-password"

Collect these authentication details for each registry domain that's specifiedin your configuration.

Update your workload to use theimagePullSecrets field

1. Create a Secret with your authentication details from the previous sectionby following the instructions topull an image from a PrivateRegistry.

2. Identify which workloads need to be migrated to theimagePullSecrets fieldby running the following command:

   kubectlgetpods-A-ojson|jq-r".items[] |  select(.spec.containers[] |        .image | startswith(\"$REGISTRY_DOMAIN\")) |  .metadata.namespace + \"/\" + .metadata.name"

   You must create a Secret for each namespace that's used by workloads withimages from this registry domain.

3. Update your workloads to use theimagePullSecrets field with the Secretsthat you created in the previous step.

   Alternatively, if you need to migrate a large number of workloads, you canimplement aMutatingAdmissionWebhookto add theimagePullSecrets field.

Update your containerd config to stop setting registry auths

After your migrate your workloads to use theimagePullSecrets field, you mustupdate your workloads that modify your containerd configuration to stop settingregistry auths. For any workloads that you identified as modifying theconfiguration, modify the workloads to stop setting registry auths.

Test with a new node pool and migrate workloads to the new node pool

To mitigate the risk of causing issues with your workloads, do the following:

1. Create a new node pool.
2. Schedule the updated workload that modifies your containerd configuration tonodes in the new node pool.
3. Migrate your remaining workloads to the new node pool by following theinstructions tomigrate workloads between nodepools.

Migrate from the CRI registryconfigs.tls property

If your workloads use theregistry.configs.tls property, you must migratethose workloads to access private registries with private CA certificates.

Follow the instructions tomigrate from configuration DaemonSets. Thisprocess is done with the following steps:

1. Update your workloads that modify the containerd config to stop setting TLSdetails.
2. Store the certificates in Secret Manager.
3. Create a runtime configuration file that points to your certificates.
4. Create a new node pool and test that your workloads that use images hostedfrom the private registry work as expected.
5. Apply the configuration to a new cluster and start running the workloads onthat cluster, or apply the configuration to the existing cluster. Applyingthe configuration to the existing cluster could potentially disrupt otherexisting workloads. For more information about these two approaches, seeCreate a runtime configurationfile.

After you migrate, ensure that you stop applying any changes to yourregistry.configs field or you might experience issues with containerd.

Get support

If you still can't determine the source of the deprecated API calls, and therecommendations remain active, consider the following next step:

• If you can't find a solution to your problem in the documentation, seeGet support for further help,including advice on the following topics:

  • Opening a support case by contactingCloud Customer Care.
  • Getting support from the community byasking questions on StackOverflow and using thegoogle-kubernetes-engine tag to search for similarissues. You can also join the#kubernetes-engine Slack channel for more community support.
  • Opening bugs or feature requests by using thepublic issue tracker.