You can verify the integrity of the Compute Engine virtual machine (VM) imagethat Google Kubernetes Engine (GKE) uses for control plane VMs. This page providesinstructions for a security team that monitors the control plane logs to verifythe following:

• The control plane VM booted with authentic firmware and other boot softwarethat was cryptographically verified by secure boot and integrity monitoring.
• The control plane VM booted from an authentic GKE OS image.

You can also perform this verification for the OS images and boot integrity ofyour nodes.

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, seeAbout GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describesoptional capabilities that give you more visibility or control over the GKE control plane.

About VM integrity verification

By default, all GKE control plane instances areShielded VMs, which are hardened VMsthat use security capabilities like secure and measured boot, a virtual trustedplatform module (vTPM), and UEFI firmware. All GKE nodes alsoenableintegrity monitoring,which validates the boot sequence of each Shielded VM against abaseline "good" boot sequence. This validation returns pass or fail results foreach boot sequence phase and adds those results to Cloud Logging. Integritymonitoring is enabled by default in all GKE clusters andvalidates the following phases:

• Early boot sequence: from when the UEFI firmware starts until thebootloader takes control. Added to the VM logs asearlyBootReportEvent.
• Late boot sequence: from when the bootloader takes control until theoperating system kernel takes control. Added to the VM logs aslateBootReportEvent.

GKE also adds control plane VM creation logs toLogging. These logs contain metadata that identifies the machineand includes details about the VM image and the boot sequence.Google Cloud publishes averification summary attestation (VSA)for each GKE control plane VM image in thegke-vsa repository on GitHub.The VSA uses thein-toto framework forattestations. You can validate the control plane VM logs for your clustersagainst the corresponding VSAs to verify that your control plane nodes bootedas expected.

Performing these validations can help you to achieve the following goals:

• Ensure that the software on the control plane is protected by secure bootand integrity monitoring, matches the intended source code, and is exactlythe same as the image that other Google Cloud customers use.
• Improve your confidence in how GKE secures the control plane.

Pricing

This feature is offered at no extra cost in GKE.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Enable the Cloud Logging API.

  Roles required to enable APIs

  To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

  Enable the API

• Ensure that you already have a GKE Autopilot mode orStandard mode cluster running version 1.29 or later.

Required roles

To get the permissions that you need to verify control plane VM integrity, ask your administrator to grant you the following IAM roles on your project:

• Create and interact with clusters:Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
• Access and process logs:Logs Viewer (roles/logging.viewer)

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Check for failed boot sequence phases

Integrity monitoring adds a log to Logging if a control planeVM fails or successfully completes a phase of the boot sequence. To viewfailed boot events, run the following commands:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. In theQuery field, specify the following query:

   jsonPayload.@type="type.googleapis.com/cloud_integrity.IntegrityEvent"jsonPayload.earlyBootReportEvent.policyEvaluationPassed="false" OR jsonPayload.lateBootReportEvent.policyEvaluationPassed="false"jsonPayload.metadata.isKubernetesControlPlaneVM="true"

   You can also check for successful boot events by replacingfalse withtrue in this query.

3. ClickRun query. If you don't see results, your control plane VMs passedall integrity monitoring checks. If you see an output, proceed to the nextstep to identify the corresponding cluster.

4. In the failed boot integrity log, copy the value in theresource.labels.instance_id field.

5. In theQuery field, specify the following query:

   protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"protoPayload.metadata.isKubernetesControlPlaneVM="true"resource.labels.instance_id="INSTANCE_ID"protoPayload.methodName="v1.compute.instances.insert"

   ReplaceINSTANCE_ID with the value of theinstance_id field from the previous step.

6. ClickRun query. The value in theprotoPayload.metadata.parentResource.parentResourceId field is theGKE cluster ID.

7. Find the name of the GKE cluster:

   gcloudassetquery\--organization=ORGANIZATION_ID\--statement="SELECT name FROM container_googleapis_com_Cluster WHERE resource.data.id='CLUSTER_ID';"

   Replace the following:

   • ORGANIZATION_ID: the numerical ID of yourGoogle Cloud organization.
   • CLUSTER_ID: the value of theprotoPayload.metadata.parentResource.parentResourceId field from theprevious step.

   The output is similar to the following:

   # lines omitted for clarity//container.googleapis.com/projects/PROJECT_ID/locations/LOCATION/clusters/CLUSTER_NAME

   This output has the following fields:

   • PROJECT_ID: your Google Cloud project ID.
   • LOCATION: the location of the cluster.
   • CLUSTER_NAME: the name of the cluster.

Find and inspect the control plane VM logs

The Compute Engine VM creation logs that correspond toGKE clusters are stored in the_Default log bucket.To find the creation logs for your cluster control plane VMs and retrieve thismetadata, do the following:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. In theQuery field, specify the following query:

   resource.type="gce_instance"protoPayload.methodName="v1.compute.instances.insert"protoPayload.metadata.isKubernetesControlPlaneVM="true"

3. ClickRun query. If you don't see results, check that you meet all ofthe requirements in theBefore you begin section.

4. In the query results, check themetadata field. The output is similar tothe following:

   # fields omitted for clarity"metadata": {  "usedResources": {    "attachedDisks": [      {        "sourceImageId": "9046093115864736653",        "sourceImage": "https://www.googleapis.com/compute/v1/projects/1234567890/global/images/gke-1302-gke1627000-cos-113-18244-85-49-c-pre",        "isBootDisk": true      }# fields omitted for clarity

   Themetadata field includes the following information:

   • usedResources: the list of resources used to create the VM.
   • attachedDisks: the boot disk for the VM.
   • sourceImageId: the unique ID of the VM image.
   • sourceImage: the URL of the source VM image. The syntax of the value inthis field ishttps://www.googleapis.com/compute/v1/projects/PROJECT_NUMBER/global/images/IMAGE_NAME,wherePROJECT_NUMBER is the number of theGoogle Cloud-owned project that hosts the control plane VMs andIMAGE_NAME is the name of the image that was usedto boot the VM.
   • isBootDisk: a boolean identifier for whether this disk was used as theboot disk for the VM.

Find and verify the VSA for control plane VM images

In this section, you find the VSA that corresponds to your controlplane VM image in the gke-vsa repository on GitHub. You then use a tool namedslsa-verifier provided by theSupply chain Levels for Software Artifacts (SLSA) frameworkto verify the VSA. You need the following data from the controlplane VM creation log:

• The VM image ID
• The project number of the Google Cloud-owned project that hosts the VMs
• The OS image name that was used to boot the VM

The file that corresponds to your control plane VM has the following filenameformat:

IMAGE_NAME:IMAGE_ID.intoto.jsonl

Replace the following:

• IMAGE_NAME: the VM image name, which is the stringafter/images/ in theattachedDisks.sourceImage field in the VM audit logfrom the previous section. For example,gke-1302-gke1627000-cos-113-18244-85-49-c-pre.
• IMAGE_ID: the VM image ID, which is the value of theattachedDisks.sourceImageId field in the VM audit log from the previoussection. For example,9046093115864736653.

To find and verify the VSA when you know the filename of your VSA file, performthe following steps:

1. Open thegke-vsa GitHub repository.
2. In the "gke-master-images" directory, find the file that corresponds to yourVM image. For example,https://github.com/GoogleCloudPlatform/gke-vsa/blob/main/gke-master-images:78064567238/IMAGE_NAME:IMAGE_ID.intoto.jsonl
3. Download the VSA file.
4. Install theslsa-verifier tool.

5. Save the public key for verifying the VSA to a file namedvsa_signing_public_key:

   -----BEGIN PUBLIC KEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEeGa6ZCZn0q6WpaUwJrSk+PPYEsca3Xkk3UrxvbQtoZzTmq0zIYq+4QQl0YBedSyy+XcwAMaUWTouTrB05WhYtg==-----END PUBLIC KEY-----

6. Verify the VSA:

   slsa-verifierverify-vsa\--attestation-path=PATH_TO_VSA_FILE\--resource-uri=gce_image://gke-master-images:IMAGE_NAME\--subject-digest=gce_image_id:IMAGE_ID\--verifier-id=https://bcid.corp.google.com/verifier/bcid_package_enforcer/v0.1\--verified-level=BCID_L1\--verified-level=SLSA_BUILD_LEVEL_2\--public-key-path=PATH_TO_PUBLIC_KEY_FILE\--public-key-id=keystore://76574:prod:vsa_signing_public_key

   Replace the following:

   • PATH_TO_VSA_FILE: the path to the VSA file thatyou downloaded.
   • IMAGE_NAME: the name of the VM image, likegke-1302-gke1627000-cos-113-18244-85-49-c-pre.
   • IMAGE_ID: the VM image ID, like9046093115864736653.

   If the VSA passes the verification checks, the output is thefollowing:

   Verifying VSA: PASSEDPASSED: SLSA verification passed

What's next

• Learn about what else you can view in the control plane.