In this page, you learn how to create a Google Kubernetes Engine (GKE) clusterwithnode poolsrunning Microsoft Windows Server. With this cluster, you can use Windows Servercontainers. Microsoft Hyper-V containers are not currently supported.Similar to Linux containers, Windows Server containersprovide process and namespace isolation.

A Windows Server node requires more resources than a typicalLinux node. Windows Server nodes need the extra resources to run theWindows OS and for the Windows Server components that cannot run in containers. SinceWindows Server nodes require more resources, yourallocatable resourcesare lower than they would be with Linux nodes.

Creating a cluster using Windows Server node pools

In this section, you create a cluster that uses a Windows Server container.

To create this cluster you need to complete the following tasks:

1. Choose your Windows Server node image.
2. Update and configuregcloud.
3. Create a cluster and node pools.
4. Getkubectl credentials.
5. Wait for cluster initialization.

GKE uses IAM service accounts that are attached to your nodes to run system tasks like logging and monitoring. At a minimum, thesenode service accounts must have theKubernetes Engine Default Node Service Account (roles/container.defaultNodeServiceAccount) role on your project. By default, GKE uses theCompute Engine default service account, which is automatically created in your project, as the node service account.

To grant theroles/container.defaultNodeServiceAccount role to the Compute Engine default service account, complete the following steps:

Choose your Windows Server node image

To run on GKE, Windows Server container node images need to be builton Windows Server version 2019 (LTSC), Windows Server version 20H2 (SAC), orWindows Server version 2022 (LTSC). A single cluster can have multiple WindowsServer node pools using different Windows Server versions, but each individualnode pool can only use one Windows Server version.

Consider the following when choosing your node image:

• Support timing:
  • The support timing for a Windows Server node image is subject to thesupport timing provided by Microsoft, as described inSupport policy for OS images.You can find the support end date for GKE Windows nodeimages by using thegcloud container get-server-config command asdescribed in theMapping GKE and Windows versionssection.
  • SAC versions are only supported by Microsoft for 18 months after theirinitial release.If you choose SAC for the image type for your node pool,but do not upgrade your node pool to newer GKE versionsthat target newer SAC versions, you cannot create new nodes in your nodepool when the support lifecycle for the SAC version ends. Learn moreabout Google's support for theWindows Server operating system.We recommend using LTSC because of its longer support lifecycle.
  • Do not choose SAC if you enroll your GKE cluster in thestable release channel.Since SAC versions are only supported by Microsoft for 18 months, there isa risk of the SAC node pool image becoming unsupported while the stableGKE version is still available.
• Version compatibility and complexity:
  • Only choose SAC if you canupgrade your node pooland the containers running in it regularly. GKEperiodically updates the SAC version used for Windows node pools in newGKE releases, so choosing SAC for your node pool image typerequires you to rebuild your containers more often.
  • If you are unsure of which Windows Server image type to use, werecommend choosing Windows Server LTSC to avoid version incompatibilityproblems when upgrading your node pool. For additional information, seeWindows Server servicing channels: LTSC and SAC in Microsoft's documentation.
  • BothWindows Server Core andNano Server can be used as a base image for your containers.
  • Windows Server containers have important version compatibility requirements:
    • Windows Server containers built for LTSC do not run on SAC nodes, andvice-versa.
    • Windows Server containers built for a specific LTSC or SAC version donot run on other LTSC or SAC versions without being rebuilt to targetthe other version.
  • Building your Windows Server container images asmulti-arch imagesthat can target multiple Windows Server versions can help you manage thisversioning complexity.
• New features:
  • New Windows Server features are typically introduced into SAC versionsfirst. Because of this, new GKE Windows functionality mightbe introduced in SAC node pools first.
  • Consider SAC if you depend on features not yet available in the LTSCrelease.

• Container runtime:

  • For both the Windows Server LTSC and SAC node images, the containerruntime can be Docker or containerd. For GKE node version1.21.1-gke.2200 and later, we recommend using the containerd runtime. Formore information, seeNode images.

    Warning:In GKE version 1.24 and later, Docker-based node imagetypes are not supported. In GKE version 1.23, you also cannot create new node pools with Docker node image types. You must migrate to a containerd node image type. Tolearn more about this change, seeAbout the Docker node image deprecation.

Update and configuregcloud

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Ensure you have the correct permission to create clusters. At minimum, youshould be aKubernetes Engine Cluster Admin.

Create a cluster and node pools

To run Windows Server containers, your cluster must have at least one Windowsand one Linux node pool.You cannot create a cluster using only a WindowsServer node pool. The Linux node pool is required to run critical cluster add-ons.

Because of its importance, we recommend turning on autoscaling to ensure your Linux node pool has sufficient capacity to run cluster add-ons.

gcloudcontainerclusterscreateCLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--enable-ip-alias\--num-nodes=NUMBER_OF_NODES\--cluster-version=VERSION_NUMBER\--release-channelCHANNEL

--service-account=SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com

gcloudcontainernode-poolscreateNODE_POOL_NAME\--cluster=CLUSTER_NAME\--location=CONTROL_PLANE_LOCATION\--image-type=IMAGE_NAME\--no-enable-autoupgrade\--machine-type=MACHINE_TYPE_NAME\--windows-os-version=WINDOWS_OS_VERSION

gcloudcontainernode-poolscreatenode_pool_name\--cluster=cluster_name\--location=us-central1\--image-type=WINDOWS_LTSC_CONTAINERD\--windows-os-version=ltsc2022

gcloudcontainernode-poolscreatenode_pool_name\--cluster=cluster_name\--location=us-central1\--windows-os-version=ltsc2022

resource "google_container_cluster" "default" {  name     = "gke-standard-regional-cluster"  location = "us-west1"  initial_node_count = 1}resource "google_container_node_pool" "default" {  name     = "windows-node-pool"  cluster  = google_container_cluster.default.name  location = google_container_cluster.default.location  node_config {    image_type = "WINDOWS_LTSC_CONTAINERD"  }}

After you create a Windows Server node pool, the cluster goes into aRECONCILEstate for several minutes as the control plane is updated.

Get kubectl credentials

Use theget-credentials command to enablekubectl to work with the cluster youcreated.

gcloudcontainerclustersget-credentialsCLUSTER_NAME\--locationCONTROL_PLANE_LOCATION

For more information on theget-credentials command, see the SDKget-credentialsdocumentation.

Wait for cluster initialization

Before using the cluster, wait for several seconds untilwindows.config.common-webhooks.networking.gke.io is created. This webhook addsscheduling tolerations to Pods created with thekubernetes.io/os: windowsnode selector to ensure they are allowed to run on Windows Server nodes. It alsovalidates the Pod to ensure that it only uses features supported on Windows.

To ensure the webhook is created, run the following command:

kubectlgetmutatingwebhookconfigurations

The output should show the webhook running:

NAME                                              CREATED ATwindows.config.common-webhooks.networking.gke.io  2019-12-12T16:55:47Z

Now that you have a cluster with two node pools (one Linux and one Windows), youcandeploy a Windows application.

Mapping GKE and Windows versions

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

Microsoft releases new SAC versions approximately every six months and new LTSCversions every two to three years. These new versions are typically available innew GKE minor versions. Within aGKE minor versionthe LTSC and SAC versions usually remain fixed.

To see the version mapping between GKE versions and WindowsServer versions, use thegcloud beta container get-server-configcommand:

gcloudbetacontainerget-server-config

The version mapping is returned in thewindowsVersionMaps field of theresponse. To filter the response to see the version mapping for specificGKE versions in your cluster, perform the following steps in aLinux shell or in Cloud Shell.

1. Set the following variables:

   CLUSTER_NAME=CLUSTER_NAME\NODE_POOL_NAME=NODE_POOL_NAME\CONTROL_PLANE_LOCATION=CONTROL_PLANE_LOCATION

   Replace the following:

   • CLUSTER_NAME: the name of your cluster.
   • NODE_POOL_NAME: the name of the Windows Server node pool.
   • CONTROL_PLANE_LOCATION: the Compute Enginelocation of the control plane of yourcluster. Provide a region for regional clusters, or a zone for zonal clusters.

2. Obtain the node pool version and store it in theNODE_POOL_VERSIONvariable:

   NODE_POOL_VERSION=`gcloudcontainernode-poolsdescribe$NODE_POOL_NAME\--cluster=$CLUSTER_NAME\--location=$CONTROL_PLANE_LOCATION\--format="value(version)"`

3. Obtain the Windows Server versions forNODE_POOL_VERSION:

   gcloudbetacontainerget-server-config\--location=$CONTROL_PLANE_LOCATION\--format="yaml(windowsVersionMaps.\"$NODE_POOL_VERSION\")"

   The output is similar to the following:

   windowsVersionMaps:  1.18.6-gke.6601:    windowsVersions:    - imageType: WINDOWS_SAC      osVersion: 10.0.18363.1198      supportEndDate:        day: 10        month: 5        year: 2022    - imageType: WINDOWS_LTSC      osVersion: 10.0.17763.1577      supportEndDate:        day: 9        month: 1        year: 2024

4. Obtain the Windows Server version for theWINDOWS_SAC image type:

   gcloudbetacontainerget-server-config\--flatten=windowsVersionMaps.\"$NODE_POOL_VERSION\".windowsVersions\--filter="windowsVersionMaps.\"$NODE_POOL_VERSION\".windowsVersions.imageType=WINDOWS_SAC"\--format="value(windowsVersionMaps.\"$NODE_POOL_VERSION\".windowsVersions.osVersion)"

   The output is similar to the following:

   10.0.18363.1198

Upgrading Windows Server node pools

TheWindows Server container version compatibilityrequirements mean that yourcontainer images might need to be rebuilt to match the Windows Server versionfor a new GKE version before upgrading your node pools.

To ensure that your container images remain compatible with your nodes, werecommend that you checkthe version mapping and buildyour Windows Server container images asmulti-arch imagesthat can target multiple Windows Server versions. You can then update yourcontainer deployments to target the multi-arch images that will work on both thecurrent and the next GKE version before manually invoking aGKE node pool upgrade.Manual node pool upgradesmust be performed regularly because nodes cannot be more than two minor versionsbehind the control plane version.

We recommend that yousubscribe to upgrade notifications usingPub/Subto proactively receive updates about new GKE versions and theWindows OS versions they use.

We recommend enablingnode auto-upgradesonly if you continuously build multi-arch Windows Server container images thattarget the latest Windows Server versions, especially if you are using WindowsServer SAC as the node image type. Node auto-upgrades are less likely to causeproblems with the Windows Server LTSC node image type but there is still a riskof encountering version incompatibility issues.

Windows Updates

Windows Updates are disabled for Windows Server nodes. Automaticupdates can cause node restarts at unpredictable times, and any WindowsUpdates installed after a node starts would be lost when the node is recreatedby GKE. GKE makes Windows Updates available byperiodically updating the Windows Server node images used in newGKE releases. There can be a delay between when Windows Updatesare released by Microsoft and when they are available in GKE.When critical security updates are released, GKE updates theWindows Server node images as quickly as possible.

Control how Windows Pods and Services communicate

You can control how Windows Pods and Services communicate usingnetwork policies.

You can have a Windows Server container on clusters that havenetwork policy enabled in GKE versions 1.22.2 and later. Thisfeature is available for clusters that use theWINDOWS_LTSC orWINDOWS_LTSC_CONTAINERD node image types.

If your control planes or nodes are running earlier versions, you can migrateyour node pools to a version that supports network policy by upgrading your nodepools and your control plane to GKE version 1.22.2 or later.This option is only available if you created your cluster with the--enable-dataplane-v2 flag.

After you enable network policy, all previously configured policies, includingpolicies that did not work on Windows Server containers before you enabled thefeature, become active.

Some clusters cannot be used with Windows Server containers on clusters withnetwork policy enabled. See thelimitations section for moredetails.

Viewing and querying logs

Logging is enabled automatically in GKE clusters. You can viewthe logs of the containers and the logs from other services on the Windows Servernodes usingKubernetes Engine monitoring.

The following is an example of a filter to get the container log:

resource.type="k8s_container"resource.labels.cluster_name="your_cluster_name"resource.labels.namespace_name="your_namespace_id"resource.labels.container_name="your_container_name"resource.labels.Pod_name="your_Pod_name"

Accessing a Windows Server node using Remote Desktop Protocol (RDP)

You can connect to a Windows Server node in your cluster using RDP. For instructionson how to connect, seeConnecting to Windows instancesin the Compute Engine documentation.

Building multi-arch images

You can build the multi-arch images manually or use a Cloud Build builder.For instructions, seeBuilding Windows multi-arch images.

Using gMSA

The following steps show you how to use a Group Managed Service Account (gMSA)with your Windows Server node pools.

1. Configure Windows Server nodes in your cluster to automatically join your ADdomain. For instructions, seeConfigure Windows Server nodes to automatically join an Active Directory domain.

2. Create and grant a gMSA access to the security group automatically createdby the domain join service. This step needs to be done in a machine withadministrative access to your AD domain.

   $instanceGroupUri=gcloudcontainernode-poolsdescribeNODE_POOL_NAME--clusterCLUSTER_NAME--format="value(instanceGroupUrls)"$securityGroupName=([System.Uri]$instanceGroupUri).Segments[-1]$securityGroup=dsquerygroup-name$securityGroupName$gmsaName=GMSA_NAME$dnsHostName=DNS_HOST_NAMENew-ADServiceAccount-Name$gmsaName-DNSHostName$dnsHostName-PrincipalsAllowedToRetrieveManagedPassword$securityGroupGet-ADServiceAccount$gmsaNameTest-ADServiceAccount$gmsaName

   Replace the following:

   • NODE_POOL_NAME: the name of your Windows Servernode pool. The automatically created security group has the same name asyour Windows Server node pool.
   • CLUSTER_NAME: the name of your cluster.
   • GMSA_NAME: the name you choose for the new gMSA.
   • DNS_HOST_NAME: the Fully Qualified Domain Name(FQDN) of the service account you created. For example, ifGMSA_NAME iswebapp01 and the domain isexample.com, thenDNS_HOST_NAME iswebapp01.example.com.

3. Configure your gMSA by following the instructions in theConfigure GMSA for Windows Pods and containers tutorial.

Deleting Windows Server node pools

Delete a Windows Server node pool by usinggcloud or the Google Cloud console.

gcloudcontainernode-poolsdeleteNODE_POOL_NAME\--cluster=CLUSTER_NAME--location=CONTROL_PLANE_LOCATION

Limitations

There are some Kubernetes features that are not yet supported forWindows Server containers. In addition, some features are Linux-specific and do notwork for Windows. For the complete list of supported and unsupported Kubernetes features,see theKubernetes documentation.

In addition to the unsupported Kubernetes features, there are some GKEfeatures that are not supported.

For GKE clusters, the following features are not supported withWindows Server node pools:

• Cloud TPUs (--enable-tpu)
• Image streaming
• Intranode visibility(--enable-intra-node-visibility)
• IP masquerade agent
• Kubernetes alpha cluster (--enable-kubernetes-alpha)
• Node Local DNS cache
• Private use of Class E IP addresses
• Private use of public IP addresses
• Network policy logging
• Kubernetesservice.spec.sessionAffinity
• GPUs (--accelerator)
• Setting the maximum Pods per node greater than the default limit of 110
• Filestore CSI driver
• Docker-basedCloudSQL Auth proxy
• IPv4/IPv6 dual-stack networking IPv6 is not supported on Windows nodes.

Local External Traffic Policyon Windows node pool is only supported with GKEversion v1.23.4-gke.400 or later.

Other Google Cloud products that you want to use with GKE clustersmight not support Windows Server node pools. For specific limitations, refer tothe documentation of that product.

Troubleshooting

See the Kubernetes documentation for general guidance ondebugging Pods andServices.

Containerd node issues

For known issues using a Containerd node image, seeKnown issues.

Windows Pods fail to start

A version mismatch between the Windows Server container and the Windows nodethat is trying to run the container can result in your Windows Pods failing tostart.

If the version for your Windows node pool is 1.16.8-gke.8 or later, reviewMicrosoft's documentation for theFebruary 2020 Windows Server container incompatibility issue and build your container images withbase Windows images that include Windows Updates from March 2020. Container images built on earlierbase Windows images might fail to run on these Windows nodes and can also causethe node to fail with statusNotReady.

Image pull errors

Windows Server container images, and the individual layers they are composed of,can be quite large. Their size can cause Kubelet to timeout and fail when downloadingand extracting the container layers.

You might have encountered this problem if you see the "Failed to pull image" or "Imagepull context cancelled" error messages or anErrImagePull status for yourPods.

If the pull image occurs frequently, you should use node pools with a higher CPUspecification. Container extraction is executed in parallel across cores, somachine typeswith more cores reduces the overall pull time.

Try the following options to successfully pull your Windows Servercontainers:

• Break the application layers of the Windows Server container image intosmaller layers that can each be pulled and extracted more quickly. This canmake Docker's layer caching more effective and make image pull retries morelikely to succeed. To learn more about layers, see the Docker articleStorage drivers.

• Connect to your Windows Server nodes and manually use thedocker pull commandon your container images before creating your Pods.

• Set theimage-pull-progress-deadline flagfor thekubelet service to increase the timeout for pulling containerimages.

  Set the flag by connecting to your Windows nodes and running thefollowing PowerShell commands.

  Note: This procedure requires akubelet restart, which can disrupt Podsrunning on the node.

  1. Get the existing command line for the Kubelet service from the Windows registry.

     PSC:\>$regkey="HKLM\SYSTEM\CurrentControlSet\Services\kubelet"

     PSC:\>$name="ImagePath"

     PSC:\>$(regquery${regkey}/v${name}|Out-String)-match`"(?s)${name}.*(C:.*kubelet\.exe.*)"

     PSC:\>$kubelet_cmd=$Matches[1]-replace`"--image-pull-progress-deadline=.* ",""-replace"\r\n"," "

  2. Set a new command line for the Kubelet service, with an additional flag to increase the timeout.

     PSC:\>regadd${regkey}/f/v${name}/tREG_EXPAND_SZ/d"${kubelet_cmd} `--image-pull-progress-deadline=40m "

  3. Confirm that the change was successful.

     PSC:\>regquery${regkey}/v${name}

  4. Restart thekubelet service so the new flag takes effect.

     PSC:\>Restart-Servicekubelet

  5. Confirm that thekubelet service restarted successfully.

     PSC:\>Get-Servicekubelet# ensure state is Running

Image family reached end of life

When creating a node pool with a Windows image, you receive an error similar tothe following:

WINDOWS_SAC image family for 1.18.20-gke.501 has reached end of life, newer versions are still available.

To resolve this error, choose a Windows image that is available and supported.You can find the support end date for GKE Windows node images byusing thegcloud container get-server-config command as described in theMapping GKE and Windows versions section.

Timeout during node pool creation

Node pool creation can time out if you are creating a large number of nodes (forexample, 500) and it's the first node pool in the cluster using a Windows Serverimage.

To resolve this issue, reduce the number of nodes you are creating. You canincrease the number of nodes later.

Windows nodes becomeNotReady with error: "PLEG is not healthy"

This is aknown Kubernetes issue that happens when multiple Pods are started very rapidly on a single Windowsnode. To recover from this situation, restart the Windows Server node. Arecommended workaround to avoid this issue is to limit the rate at which WindowsPods are created to one Pod every 30 seconds.

Inconsistent TerminationGracePeriod

The Windows system timeout for the container might differ from the grace periodyou configure. This difference can cause Windows to force-terminate the containerbefore the end of the grace period passed to the runtime.

You can modify the Windows timeout by editing container-local registry keys atimage-build time. If you modify the Windows timeout, you mightalso need toadjust TerminationGracePeriodSeconds to match.

Network connectivity problems

If you experience network connectivity problems from your Windows Server containers,it might be because Windows Server container networking often assumes a network MTU of1500, which is incompatible with Google Cloud's MTU of1460.

Check that both the MTU of the network interface in the container and thenetwork interfaces of the Windows Server node itself are set to the same value(that is,1460 or less). For information on how to set the MTU, seeknown issues for Windows containers.

Node startup issues

If nodes fail to start in the cluster or fail to join the cluster successfully,review the diagnostic information provided in the node'sserial portoutput.

Run the following command to see the serial port output:

gcloudcomputeinstancesget-serial-port-outputNODE_NAME--zone=COMPUTE_ZONE

Replace the following:

• NODE_NAME: the name of the node.
• COMPUTE_ZONE: thecompute zonefor the specific node.

Intermittently unreachable Services in Windows nodes with cluster running 1.24 or earlier

When starting Windows nodes in Kubernetes clusters with a high number of HostNetwork Service Load Balancer rules, there is a delay in processing the rules.Services are intermittently unreachable during the delay, which lasts around 30seconds per rule, and the total delay can be significant if there are enoughrules. To learn more, see theoriginal issue in GitHub.

For GKE clusters running version 1.24 or earlier, with anyWindows nodes that had an event that restartedkube-proxy—for example, nodestartup, node upgrade, manual restart—any Services being reached by a Podrunning on that node will be unreachable until all rules are synced by thecomponent.

For GKE clusters running version 1.25 or later, this behavior issubstantially improved. For details on this improvement, see thepull request in GitHub.If you are experiencing this issue, we recommend upgrading your cluster'scontrol plane to 1.25 or later.

What's next

• Learn how todeploy a Windows application.
• Read Microsoft'sshort introduction on Windows containers.
• ReadMicrosoft's guidance on choosing the container base images.
• Read about Microsoft on Windowscontainer version compatibility.