This page describes how to verify connections made by Google personnel to yourGoogle Kubernetes Engine (GKE) cluster control plane by correlatingGKE logs with Access Transparency logs.

Access Transparency logs record the actions that Google personnel take whenaccessing your content. This guide is intended for security administrators whowant additional verification of the contents of Access Transparency logs and associatedAccess Approval approvals by correlating with additional loggingsources from GKE.This verification is entirely optional and isnot required to secure your control plane.

Ensure that you're familiar with the following concepts:

• Access Transparency
• Access Transparency logs
• Access Approval
• SSH connections in Compute Engine

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, seeAbout GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describesoptional capabilities that give you more visibility or control over the GKE control plane.

About Google access to cluster control plane instances

During troubleshooting sessions or for other justified business reasons, Googlepersonnel like site reliability engineers and Cloud Customer Care employees mightneedadministrative access to the Compute Engine instances that hostyour control plane. Depending on your Customer Care support package andconfiguration, Access Transparency provides detailed audit logging for thisadministrative access. Access Approval lets you require explicitapproval before any Google personnel can access your resources. To learn moreabout administrative access and the tooling that you can use to authorize accessand to record changes, seeAdministrative access for Google employees.

Control plane access logs

When you enable GKE control plane authority, GKEgeneratescontrol plane access logs that you can optionally use tocross-reference the audit logs generated by Access Transparency and byAccess Approval. GKE adds control plane accesslogs to the_Default bucket in Loggingto record incoming network connections and specific SSH eventsin your control plane instances. You must enableGKE control plane authority in your project to generate controlplane access logs for your clusters.

GKE generates the following access logs for the control plane:

• Control plane connection logs
• Control plane SSH logs

The volume of control plane connection logs depends on factors like the numberof nodes in the cluster, the number of control plane instances (regionalclusters have more control plane instances than zonal clusters), and how oftenyour workloads call the Kubernetes API server. The volume of SSH logs is smalland depends on the number of node restarts.

To verify the connections to your control plane, you find the control planeaccess logs for your cluster and match those logs to audit logs fromAccess Transparency and Access Approval. This lets you confirmthat all of the SSH connections to your control plane instances were as a resultof authorized administrative access by Google personnel. When you enableGKE control plane authority for your cluster, all SSH access by Googlepersonnel to your control plane isnon-interactive, which means that every SSHconnection runs a single command that you authorize.

Pricing

The following pricing considerations apply:

• Control plane access logs are subject toLogging pricing.
• Access Transparency is included with specific Customer Caresubscriptions. For details, seeAccess Transparency pricing.

Before you begin

Before you start, make sure that you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task,install and theninitialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running thegcloud components update command. Earlier gcloud CLI versions might not support running the commands in this document.Note: For existing gcloud CLI installations, make sure to set thecompute/regionproperty. If you use primarily zonal clusters, set thecompute/zone instead. By setting a default location, you can avoid errors in the gcloud CLI like the following:One of [--zone, --region] must be supplied: Please specify location. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

• Enable the Cloud Logging API.

  Roles required to enable APIs

  To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enable permission.Learn how to grant roles.

  Enable the API

• Enable Access Transparency for your organization. For instructions, seeEnabling Access Transparency.

• Optionally, enable Access Approval for your project and select theGKE service. For instructions, seeReview and approve access requests using the Google-managed signing key.

• Ensure that your environment is eligible to useGKE control plane authority features. To opt-in to thesefeatures, contact your Google Cloud sales team.

Requirements

Control plane access logs require GKE version 1.31.1-gke.1846000or later.

Required roles and permissions

To get the permissions that you need to enable log generation and to access and process logs, ask your administrator to grant you the following IAM roles:

• Enable control plane connection logging in your cluster:Kubernetes Engine Cluster Admin (roles/container.clusterAdmin) on your project
• Access logs and to use Logs Explorer and Log Analytics:Logs Viewer (roles/logging.viewer) on your project
• Enable Access Transparency for the organization:Access Transparency Admin (roles/axt.admin) on your organization

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Enable GKE cluster control plane access logs

You can enable control plane access log generation for Autopilot modeand Standard mode clusters by enabling the corresponding loggingcomponent. For more information about control plane log types, seeView GKE logs.

The supported logging component names for control plane access logs are asfollows:

• Control plane SSH logs:KCP_SSHD
• Control plane connection logs:KCP_CONNECTION

Enable control plane access logs on a new cluster

The following example creates an Autopilot mode cluster with both typesof control plane access logs enabled. To only enable one type of control planeaccess log, omit the corresponding component name from the command.

gcloudcontainerclusterscreate-autoCLUSTER_NAME\--location=LOCATION\--logging=SYSTEM,KCP_SSHD,KCP_CONNECTION

Replace the following:

• CLUSTER_NAME: the name for the new cluster.
• LOCATION: the location in which to create the cluster.

To specify logging components when you create a cluster using theGKE API, in theprojects.locations.clusters.createmethod, set the corresponding values in theLoggingConfig objectof theCluster resource.

Enable control plane access logs on an existing cluster

1. Check the existing logging configuration of the cluster:

   gcloudcontainerclustersdescribeCLUSTER_NAME\--location=LOCATION\--flatten=loggingConfig\--format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

   The output is similar to the following:

   SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER

2. Identify the gcloud CLI values for the--logging flag thatcorrespond to the logging component configuration from the output of thepreceding step. For a list of the gcloud CLI values thatcorrespond to specific logging components, see theAvailable logs table.

3. Update the logging configuration with control plane access logs:

   gcloudcontainerclustersupdateCLUSTER_NAME\--location=LOCATION\--logging=SYSTEM,EXISTING_LOGS,KCP_ACCESS_LOGS

   Replace the following:

   • EXISTING_LOGS: a comma-separated list of loggingcomponents that your cluster already uses. Ensure that you specify thegcloud CLI values that correspond to these loggingcomponents, taken from theAvailable logstable.

   • KCP_ACCESS_LOGS: a comma-separated list of thecontrol plane access log types to enable for the cluster, as follows:

     • For control plane SSH logs, specifyKCP_SSHD.
     • For control plane connection logs, specifyKCP_CONNECTION.

To specify logging components when you update a cluster using theGKE API, in theprojects.locations.clusters.updatemethod, set the existingand new logging component values in theLoggingConfig objectof theClusterUpdate resource.

Example cluster update to enable control plane access logs

Consider a cluster for which thegcloud container clusters describe commandhas the following logging configuration:

SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER

The following cluster update command enables both types of control plane accesslogs while retaining the existing log configuration for this example cluster:

gcloudcontainerclustersupdateexample-cluster\--location=us-central1\--logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION

Cross-reference control plane access logs with Access Transparency logs

To verify control plane access for a cluster, get the control plane connectionlogs, the control plane SSH logs, and the Access Transparency logs for thatcluster:

1. In the Google Cloud console, open theLogs Explorer page.

   Go to Logs Explorer

2. To get all of the logs for a specific cluster, including the control planeaccess logs and Access Transparency logs, run the following query:

   (logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection"resource.labels.cluster_name="CLUSTER_NAME"jsonPayload.connection.dest_port="22")OR(logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-sshd"resource.labels.cluster_name="CLUSTER_NAME")OR(logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Faccess_transparency"json_payload.accesses.methodName="GoogleInternal.SSH.Master"json_payload.accesses.resourceName="//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME")

The output should display all of the following types of logs for your cluster:

• Access Transparency log
• Control plane connection log
• SSH logs for each SSH session

Perform verification checks

Your primary verification check is whether you see all of the log types for anySSH connections when you run the Logging query from the precedingsection. Every Access Transparency log should have a corresponding control planeconnection log and one or more SSH logs. These logs are for actions that humansperform in your control plane instances, so the log volume should be small.

You can optionally perform the following additional checks of the log contents:

1. For each control plane SSH log, check whether an Access Transparency logexists in a 15-minute window before the timestamp of the SSH log. Thiswindow of time accounts for the final SSH session closure happening severalminutes after the initial connection was logged by Access Transparency.
2. For each control plane connection log, check whether an Access Transparencylog exists in a 5-minute window before the timestamp of the control planeconnection log.

3. If you use Access Approval for your cluster, check whethereach Access Transparency log has a correspondingaccessApprovals field.Cross-reference this field againstAccess Approval requestsfor your cluster.

   To get Access Approval requests for your project, seeView historical Access Approval requests.Access Approval might be subject toexclusions.

4. Optionally,validate the signatureof the signed Access Approval associated with theAccess Transparency log.

Control plane access log details

This section provides details and examples of the control plane access logs thatGKE generates when Google personnel connect to your control planeinstances.

Control plane connection logs

GKE adds acontrol plane connection log for every new incomingnetwork connection to a control plane instance. These logs include specificdetails like the following:

• Source and destination IP addresses and ports
• Connection direction and protocol

The following example is of a control plane connection log:

{insertId:"z1eq8wonio335a5h",jsonPayload:{instance:{vm_name:"gke-dee49f0d6fa34ce3a2ac-f513-d195-vm",zone:"us-central1-c"},cluster:{cluster_id:"CLUSTER_ID",cluster_urn:"//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1-c/clusters/CLUSTER_NAME"},connection:{state:"NEW",src_ip:"192.0.2.100",src_port:32774,dest_ip:"203.0.113.12",dest_port:22,direction:"INGRESS"protocol:"TCP"},}logName:"projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection",receiveTimestamp:"2024-04-11T04:08:01.883070399Z",resource:{labels:{cluster_name:"CLUSTER_NAME",location:"us-central1-c",project_id:"PROJECT_ID"}type:"gke_cluster",}severity:"NOTICE",timestamp:"2024-04-11T04:07:59.019330Z"}

The following fields in the log entry are relevant to verifying Google'sactions:

• cluster.cluster_urn: The fully-qualified resource identifier of the cluster.This identifier has the format//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME,with the following variables:

  • PROJECT_NUMBER: the numerical project number of yourclusters project.
  • LOCATION: the Google Cloud location of yourcluster.
  • CLUSTER_NAME: the name of your cluster.

• connection: Details about the connection attempt. This field has thefollowing information:

  • state: the state of the connection. For new connections, the value isNEW.
  • src_ip: the IP address of the connection source.
  • src_port: the port number of the connection source.
  • dest_ip: the internal IP address of your control plane VM.
  • dest_port: the destination port number.
  • direction: the connection direction. The value is alwaysINGRESS.
  • protocol: the IP protocol, likeTCP.

Control plane SSH logs

GKE adds control plane SSH logs for events related to SSHconnections to control plane instances. GKE records the followingevents:

• SSH key accepted for a user
• Session changed status from 0 to 1, indicating that the user successfully logged in
• SSH session opened
• SSH session closed
• Session changed status from 1 to 0, indicating that the user logged out
• SSH session failed

For example, the following control plane SSH log is for an SSH session beingopened:

{insertId:"8llczemdulwbbwpa",jsonPayload:{instance:{vm_name:"gke-06cb920c609941c0a5ce-6840-40e9-vm",zone:"us-central1-c"},cluster:{cluster_id:"891e6d12889747748c1ac16ffcc6cb7c0a96450b36864eb680917c119fd801d0",cluster_urn:"//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1/clusters/CLUSTER_NAME",},message:"pam_unix(sshd:session): session opened for user REDACTED by (uid=0)",},logName:"projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-ssh",receiveTimestamp:"2024-04-09T13:21:55.231436462Z"resource:{type:"gke_cluster",labels:{cluster_name:"CLUSTER_NAME",location:"us-central1",project_id:"PROJECT_ID"}},severity:"NOTICE",timestamp:"2024-04-09T13:21:50.742246Z"}

The following fields in the log entry are relevant to verifying Google'sactions:

• cluster.cluster_urn: The fully-qualified resource identifier of the cluster.This identifier has the format//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME,with the following variables:

  • PROJECT_NUMBER: the numerical project number of yourcluster project.
  • LOCATION: the Google Cloud location of yourcluster.
  • CLUSTER_NAME: the name of your cluster.

• message: Details about the SSH connection.

Disable control plane access logs

1. To view the specific log types that your cluster uses, run the followingcommand:

   gcloudcontainerclustersdescribeCLUSTER_NAME\--location=LOCATION\--flatten=loggingConfig\--format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

   The output is similar to the following:

   SYSTEM_COMPONENTS,WORKLOADS,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION

2. To disable the control plane access logs for a cluster, run the followingcommand:

   gcloudcontainerclustersupdateCLUSTER_NAME\--location=LOCATION\--logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER

In the--logging flag, specify the logging components from the output of theprevious command. This example command disables control plane access logs butkeeps other control plane component logs enabled.

To disable logging components using the GKE API, set thecorresponding values in theLoggingConfig object of theClusterUpdate resource intheprojects.locations.clusters.update method.

What's next

• Learn aboutcontrol plane security.
• Learn aboutadministrative access for Google employees.
• Learn how toconfigure logging and monitoring for GKE.
• Learn how toconfigure field-level access to logs.
• Learn aboutlogging usage limits.