This page shows you how to use logs to check the issuance and usage of Kubernetesidentities, in the form of certificates and service account tokens, by theGoogle Kubernetes Engine (GKE) cluster control plane.This verification is entirely optional and isnot required to secure your control plane.

This guide is intended for security administrators and platform owners whohave specific organizational compliance or policy requirements for control overcredential issuance and signing. You should already haveset up self-managed keys and CAswith GKE control plane authority.

You should already be familiar with the following concepts:

• CA Service logs
• Cloud KMS logs
• Kubernetes audit logging
• X.509 certificates

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, seeAbout GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describesoptional capabilities that give you more visibility or control over the GKE control plane.

About identity issuance logs

The GKE identity issuance logs arecontrol plane audit logsthat record when the control plane issues credentials and when those credentialsare used in the cluster. You can use these logs to track the lifetime of acredential, including issuance and usage, by correlating the identity issuancelogs with Cloud KMS, Certificate Authority Service, and Kubernetes APIlogs. GKE identity issuance logs are enabled when usingGKE control plane authority. These logs track the issuance andusage of the following types of credentials:

• X.509 certificates
• Cluster JSON Web Tokens (JWTs)

X.509 certificates

Kubernetes uses X.509 certificates forclient certificate authentication.To issue certificates, the Kubernetes control plane sends an approvedCertificateSigningRequestto a certificate authority (CA) in CA Service. The CA thenissues a certificate using the corresponding key in Cloud KMS to signthe certificate digest.

TheKubernetes API server logs contain certificate signature details for anyKubernetes API call that was authenticated with a certificate. The credential IDentry in the log has the following form:

"authentication.k8s.io/credential-id":"X509SHA256=CERTIFICATE_HASH"

TheCERTIFICATE_HASH value is the SHA256 hash for thecertificate, which you can use to trace the lifecycle of the certificate.

You can use Kubernetes API server certificate logs to trace the lifecycle of thecertificate by correlating logs from the following services:

• GKE identity issuance logs: Use theprotoPayload.metadata.credentialId query to find specific GKEidentity issuance logs based on the credential ID from theKubernetes API server logs. Then use theprotoPayload.metadata.certificateFingerprint field from theGKE identity issuance log to correlate the identity issuancelogs with the CA Service logs.
• CA Service logs: Find the certificate issuance log entry,which contains the following IDs:
  • cert_fingerprint.sha256_hash: the SHA256 hash of the signed certificate.Use this ID to match logs with GKE and Kubernetes API events.
  • tbs_certificate_digest: a hash of the certificate contents that weresent to be signed by a Cloud KMS key. Use this ID to match logswith Cloud KMS logs.
• Cloud KMS signing logs: Use thetbs_certificate_digest valuefrom the CA Service log to confirm that the expectedCloud KMS key signed the certificate.

JSON Web Tokens (JWTs)

Signed JWTs (JSON Web Tokens) are used asbearer tokens for entities in the cluster,like Kubernetes service accounts, when authenticating requests to the KubernetesAPI. When a Pod is created that uses a specific service account, Kubernetescreates a JWT and mounts it in the Pod. When you useGKE control plane authority to run your own keys and CAs, this JWT issigned and later verified using the service account signing key inCloud KMS.

The Kubernetes API server logs contain token signature details for anyKubernetes API call that was authenticated with a JWT. The token signature entryin the log has the following form:

"authentication.kubernetes.io/credential-id":"JTI=JWT_ID"

TheJWT_ID value is a string that identifies the JWT thatwas used in the Kubernetes API call.

You can use the JWT ID from Kubernetes API server logs to trace the lifecycle ofa JWT by correlating the following logs:

• GKE identity issuance logs: Use the JWT ID from theKubernetes API server logs to find specific JWT issuance entries. Each entryalso contains thetoBeSignedDigest field, the value of which can matchCloud KMS logs.
• Cloud KMS signing logs: Use the value of thetoBeSignedDigestfield from the GKE identity issuance logs to confirm that theexpected Cloud KMS key signed the JWT.

Pricing

In this document, you use the following billable components of Google Cloud:

• GKE
• Cloud KMS
• CA Service
• Cloud Logging

To generate a cost estimate based on your projected usage, use thepricing calculator.

Before you begin

1. Configure your GKE cluster control plane with self-managed CAs or keys.

2. Enable the following Cloud Audit Logging Service APIs:

   • ForCloud KMS, turn on Data Access audit logsof typeData Read.
   • ForCA Service, turn on Data Access audit logsof typesAdmin Read andData Write.

Required roles

To get the permissions that you need to access identity issuance logs, ask your administrator to grant you the following IAM roles on your project:

• Perform all actions in Logging:Logging Admin (roles/logging.admin).
• View logs:Private Logs Viewer (roles/logging.privateLogViewer).

For more information about granting roles, seeManage access to projects, folders, and organizations.

You might also be able to get the required permissions throughcustom roles or otherpredefined roles.

Requirements and limitations

The following requirements and limitations apply:

• You must use GKE version 1.31.1-gke.1846000 or later.
• Identity issuance logs are recorded as Cloud Audit Logs logs and have a setretention period of 400 days. The retention period is not configurable, butyou canroute your system event audit logs to other destinationsfor longer retention periods.

Verify certificates

You can use GKE control plane authority identity issuance logs to confirm that acertificate was successfully issued or used. You can use any of the followinglogs, or a combination of logs, to confirm information about certificateissuance and use:

Verify certificate usage with Kubernetes API logs

To find log entries for API calls that were authenticated using certificates,complete the following steps:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go toLogs Explorer

2. Paste the following expression into the query editor field:

   log_id("cloudaudit.googleapis.com/activity")resource.type="k8s_cluster"labels."authentication.kubernetes.io/credential-id":"X509SHA256="

3. ClickRun query.

   This query returns all Kubernetes API server logs that have an associated X.509certificate. Find specific log entries to investigate by using your securitytooling or by checking the logs manually.

To correlate these logs with other log types, find the following field:

"authentication.k8s.io/credential-id":"CREDENTIAL_ID"

TheCREDENTIAL_ID value is the identifier that you can use to correlate logs fromGKE, CA Service, and Cloud KMS. TheCREDENTIAL_ID has the form"X509SHA256=CERTIFICATE_HASH".

Verify certificate issuance with GKE logs

To find GKE log entries for certificate issuance events, complete the following steps:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. Paste the following expression into the query editor field:

   logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"resource.type="gke_cluster"protoPayload.serviceName="container.googleapis.com"protoPayload.metadata.credentialId="CREDENTIAL_ID"

   Replace the following:

   • PROJECT_ID: your project ID.
   • CREDENTIAL_ID: theCREDENTIAL_ID from theVerify certificate usage with Kubernetes API logssection.

3. ClickRun query.

Verify certificate issuance with CA Service logs

To find CA Service logs that match the GKE certificate issuance events, complete the following steps:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. Paste the following expression into the query editor field:

   resource.type="audited_resource"protoPayload.serviceName="privateca.googleapis.com"protoPayload.methodName="google.cloud.security.privateca.v1.CertificateAuthorityService.CreateCertificate"protoPayload.response.certificate_description.cert_fingerprint.sha256_hash="CERTIFICATE_HASH"

   ReplaceCERTIFICATE_HASH with theCERTIFICATE_HASHfrom theVerify certificate usage with Kubernetes API logssection. Ensure that you omit theX509SHA256= prefix from the hash.

3. ClickRun query.

   This query returns a log that contains atbs_certificate_digest:DIGEST_VALUEfield in the certificate description response section.

You can use theDIGEST_VALUE tomatch Cloud KMS signing logs for the certificate.

Verify certificate issuance with Cloud KMS signing logs

To find Cloud KMS signing events for the CA Service certificate issuance events, complete the following steps:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. Paste the following expression into the query editor field:

   protoPayload.request.digest.sha256="DIGEST_VALUE"protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"protoPayload.methodName="AsymmetricSign"protoPayload.serviceName="cloudkms.googleapis.com"

   ReplaceDIGEST_VALUE with the digest value from theVerify certificate issuance with CA Service logssection.

3. ClickRun query.

   This query returns the logs for certificate issuance events. Cloud KMSlogs don't distinguish between certificates and JWTs, so the log entries forboth are identical.

Verify tokens

You can use GKE control plane authority identity issuance logs andCloud KMS signing logs to confirm that a JSON Web Token (JWT) wassuccessfully issued.

Initiating the traceback of the token issuance event typically starts withmonitoring the Kubernetes API log for service account activity. After you identityan activity log needing further investigation, you can use thefollowing logs to confirm information about credential issuance and usage:

Verify token usage with Kubernetes API server logs

To find log entries for token usage events, complete the following steps:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. Paste the following expression into the query editor field:

   log_id("cloudaudit.googleapis.com/activity")resource.type="k8s_cluster"labels."authentication.kubernetes.io/credential-id":"JTI="

3. ClickRun query.

   This query returns all Kubernetes API server logs that have an associated JWT.Find specific log entries to investigate by using your security tooling or bychecking the logs manually.

To correlate these logs with other log types, find the following field:

"authentication.k8s.io/credential-id":"JTI=JWT_ID"

TheJWT_ID is the token identifier that you can use to correlate logs from GKEand Cloud KMS.

Verify token issuance with GKE logs

To find log entries for token issuance events, complete the following steps:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go to Logs Explorer

2. Paste the following expression into the query editor field:

   resource.type="gke_cluster"logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"protoPayload.methodName="google.cloud.gkeauth.v1.Auth.SignServiceAccountJWT"protoPayload.metadata.credentialId="JTI=JWT_ID"

   Replace the following:

   • PROJECT_ID: your project ID.
   • JWT_ID: the JWT ID from theVerify token usage with Kubernetes API server logssection.

3. ClickRun query.

   This query returns logs that contain atoBeSignedDigest field.

You can use thetoBeSignedDigest value to find Cloud KMS signing events.

Verify token issuance with Cloud KMS signing logs

To find log entries for signed digests, complete the following steps:

1. In the Google Cloud console, go to theLogs Explorer page:

   Go toLogs Explorer

2. Paste the following expression into the query editor field:

   protoPayload.request.digest.sha256="DIGEST_VALUE"protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"protoPayload.methodName="AsymmetricSign"protoPayload.serviceName="cloudkms.googleapis.com"

   ReplaceDIGEST_VALUE with the value in thetoBeSignedDigest field, from theVerify token issuance with GKE logs section.

3. ClickRun query.

   This query returns the logs for certificate issuance events. Cloud KMSlogs don't distinguish between certificates and JWTs, so the log entries forboth are identical.

What's next

• Learn aboutcontrol plane security.
• Learn aboutadministrative access for Google employees.
• Learn how toconfigure logging and monitoring for GKE.
• Learn how toconfigure field-level access to logs.
• Learn aboutlogging usage limits.