This document describes how to find and mitigate common CMEK configurationerrors, and it describes how to identify errors that occur when settingthe default resource location.

Troubleshoot setting the default resource location

You attempt to update the default storage location for anorganization or folder, but thecommand fails with an error similar to the following:

ERROR: (gcloud.logging.settings.update) INVALID_ARGUMENT: The KMS key location must match the storage location. Received KMS key location: us-central1, storage location: us-west1- '@type': type.googleapis.com/google.rpc.DebugInfo  detail: '[ORIGINAL ERROR] generic::invalid_argument: The KMS key location must match    the storage location. Received KMS key location: us-central1, storage location:    us-west1 [google.rpc.error_details_ext] { message: "The KMS key location must    match the storage location. Received KMS key location: us-central1, storage location:    us-west1" }'

To resolve this error, set the default storage location for theorganization or folder to match the location of theCloud Key Management Service key for the organization or folder.

Troubleshoot VPC Service Controls and domain restricted sharing

You configured CMEK as a default resource setting for your organization or fora folder, or you created a log bucket with CMEK enabled. You then configureVPC Service Controls. After you configure VPC Service Controls,you restrict access to Cloud Key Management Service in VPC Service Controls orenable domain restricted sharing.

At least one of the following occurs:

• You received a notification from Cloud Logging about CMEK access issues.

• You notice that CMEK isn't enabled for the_Default and_Requiredlog buckets when you create new Google Cloud projects in your organizationor in a folder.

• You receive errors when you read from log buckets with CMEK enabled. Theerrors you see are similar to the following error:

  ERROR: (gcloud.logging.read) FAILED_PRECONDITION: service account `cmek-PROJECT_ID@gcp-sa-logging.iam.gserviceaccount.com` must have both encrypt and decrypt access to the CMEK KMS key `projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY`

• You receive errors when you create or update log buckets with CMEK enabled.The errors you see are similar to the following error:

  ERROR: (gcloud.logging.buckets.create) service account `cmek-PROJECT_ID@gcp-sa-logging.iam.gserviceaccount.com` must have both encrypt and decrypt access to the CMEK KMS key `projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY`- '@type': type.googleapis.com/google.rpc.DebugInfo  detail: '[ORIGINAL ERROR] generic::permission_denied: Request is prohibited by  organization's policy. vpcServiceControlsUniqueIdentifier: <var>ERRORID</var>;'

To determine if these problems are due to the VPC Service Controls configuration,do the following:

1. Identify the Cloud Logging settings for the resource thatcontains the CMEK configuration. A resource can be aproject, a folder, or an organization. If you created log buckets withCMEK enabled, then select the PROJECT resource.

   PROJECT

   gcloud logging settings describe --project=PROJECT_ID

   Before you run the command, replacePROJECT_ID with the project ID thatcontains the log bucket.

   FOLDER

   gcloud logging settings describe --folder=FOLDER_ID

   Before you run the command, replaceFOLDER_ID with the ID ofthe folder.

   ORGANIZATION

   gcloud logging settings describe --organization=ORGANIZATION_ID

   Before you run the command, replaceORGANIZATION_ID with the ID ofthe organization.

   The previous command returns information similar to the following:

   kmsServiceAccountId:KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.comloggingServiceAccountId:SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com

   For organizations and folders, the following field is also returned:

   kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY

   The value of thekmsKeyName field include thes Google Cloud project that storesthe key.

2. Determine whether you need to migrate service accounts:

   • If the value of thekmsServiceAccountId field has the prefixservice-, then you don't need to migrate your service account.For information about CMEK configuration errors, see theTroubleshoot CMEK section of this document.

   • If the value ofkmsServiceAccountId has the prefixcmek-, thenproceed to the next step.

3. Confirm that you must migrate service accounts by disablingdomain restricted sharingor by removing Cloud Key Management Service from the list of VPC Service Controlsrestricted services.

   If the errors are resolved, then to resolve the failures you mustmigrate the affected resources to a new service account.For information about these steps, see the next section.

Migrate CMEK service accounts

The following process describes how to change the service account thatCloud Logging uses to access configured Cloud Key Management Service keys.The change in service account resolves aknown issue with VPC Service Controls and domain restricted sharing.

1. Identify theloggingServiceAccountId for your resource. A resource can bea project, a folder, or an organization. If you created log buckets withCMEK enabled, then select the PROJECT resource.

   PROJECT

   gcloud logging settings describe --project=PROJECT_ID

   Before you run the command, replacePROJECT_ID with the project ID thatcontains the log bucket.

   FOLDER

   gcloud logging settings describe --folder=FOLDER_ID

   Before you run the command, replaceFOLDER_ID with the ID ofthe folder.

   ORGANIZATION

   gcloud logging settings describe --organization=ORGANIZATION_ID

   Before you run the command, replaceORGANIZATION_ID with the ID ofthe organization.

   The previous command returns information similar to the following:

   kmsServiceAccountId:KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.comloggingServiceAccountId:SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com

   For organizations and folders, the following field is also returned:

   kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY

   The value of thekmsKeyName field include thes Google Cloud project that storesthe key.

2. If you configured a default resource setting for your organization or forfolders, then do the following:

   1. In theKMS_PROJECT_ID, grant theCloud Key Management Service CryptoKey Encrypter/Decrypterrole to the service account identified by theloggingServiceAccountIdfield.

   2. Run the following curl command, which changes theCloud Key Management Service service account used by the resource.

      PROJECT

      Not applicable.

      FOLDER

      curl -X PATCH -H "Authorization: Bearer \"$(gcloud auth print-access-token)\"" -H "Content-Type: application/json; charset=utf-8" -d '{"kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}' https://logging.googleapis.com/v2/folders/FOLDER_ID/settings?updateMask=kmsServiceAccountId

      Before you run the command, do the following:

      • ReplaceFOLDER_ID with the ID of the folder.
      • ReplaceSERVICE_ACCT_NAME with theloggingServiceAccountIdidentified earlier.

      ORGANIZATION

      curl -X PATCH -H "Authorization: Bearer \"$(gcloud auth print-access-token)\"" -H "Content-Type: application/json; charset=utf-8" -d '{"kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}' https://logging.googleapis.com/v2/organizations/ORGANIZATION_ID/settings?updateMask=kmsServiceAccountId

      Before you run the command, do the following:

      • ReplaceORGANIZATION_ID with the ID of the organization.
      • ReplaceSERVICE_ACCT_NAME with theloggingServiceAccountIdidentified earlier.

      The result of the previous command is similar to the following;

      {  "name": ".../settings",  "kmsKeyName": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY",  "kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com",  "storageLocation": "...",  "loggingServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}

   Note: The new service account is used for new resources created in theorganization or folder. The service account for existing resourcesisn't modified.

3. For each Google Cloud project or folder that contain existing log bucketsenabled with CMEK, do the following:

   1. In the project or folder, for each log bucket enabled with CMEK,do the following:

      1. Identify the Google Cloud project that stores the Cloud Key Management Service key:

         PROJECT

         gcloud logging buckets describeBUCKET_ID --location=LOCATION --project=PROJECT_ID

         Before you run the command, do the following:

         • ReplacePROJECT_ID with the project ID that containsthe log bucket.
         • ReplaceLOCATION with the location of the log bucket.

         FOLDER

         gcloud logging buckets describeBUCKET_ID --location=LOCATION --folder=FOLDER_ID

         Before you run the command, do the following:

         • ReplaceFOLDER_ID with the ID of the folder.
         • ReplaceLOCATION with the location of the log bucket.

         The result of the previous command is similar to the following:

         cmekSettings:  kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY  kmsKeyVersionName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY/cryptoKeyVersions/1  serviceAccountId:KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.comcreateTime: '2022-10-31T12:00:00.0000000Z'lifecycleState: ACTIVEname: projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_IDretentionDays: 30createTime: '2022-10-31T13:00:00.0000000Z'

      2. Go to the Google Cloud project that owns the Cloud Key Management Service key,KMS_PROJECT_ID, and grant theCloud Key Management Service CryptoKey Encrypter/Decrypterrole to the service account identified by theloggingServiceAccountIdfield.

   2. For the project, run the following curl command, which changes theCloud Key Management Service service account:

      PROJECT

      curl -X PATCH -H "Authorization: Bearer \"$(gcloud auth print-access-token)\"" -H "Content-Type: application/json; charset=utf-8" -d '{"kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}' https://logging.googleapis.com/v2/projects/PROJECT_ID/settings?updateMask=kmsServiceAccountId

      Before you run the command, do the following:

      • ReplacePROJECT_ID with the project ID that containsthe log bucket.
      • ReplaceSERVICE_ACCT_NAME with theloggingServiceAccountIdidentified earlier.

      FOLDER

      No action necessary, as you changed the Cloud Key Management Service service accountused by the folder in a previous step.

      The result of the previous command is similar to the following;

      {  "name": ".../settings",  "kmsServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com",  "loggingServiceAccountId": "SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com"}

4. For the each log bucket with CMEK enabled, do the following:

   1. Rotate the Cloud KMS key.

   2. Confirm the migration. The parent resource for the log bucket determineswhich Google Cloud CLI command to run. The parent might be aproject, folder, or organization.

      PROJECT

      gcloud logging buckets describeBUCKET_ID --location=LOCATION --project=PROJECT_ID

      Before you run the command, do the following:

      • ReplacePROJECT_ID with the project ID that containsthe log bucket.
      • ReplaceLOCATION with the location of the log bucket.

      FOLDER

      gcloud logging buckets describeBUCKET_ID --location=LOCATION --folder=FOLDER_ID

      Before you run the command, do the following:

      • ReplaceFOLDER_ID with the ID of the folder.
      • ReplaceLOCATION with the location of the log bucket.

      For a project, the result of the previous command is similar to thefollowing:

      cmekSettings:  kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY  kmsKeyVersionName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY/cryptoKeyVersions/1  serviceAccountId:SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.comcreateTime: '2022-10-31T12:00:00.0000000Z'lifecycleState: ACTIVEname: projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_IDretentionDays: 30createTime: '2022-10-31T13:00:00.0000000Z'

      Ensure thatserviceAccountId matches theloggingServiceAccountIdidentified earlier.

5. Wait at least 30 minutes before revoking permissions on the previous serviceaccount. If you experience any issues after revoking permissions on theprevious service account, then restore the permissions and contactCloud Support.

Troubleshoot CMEK

As you configure CMEK, the Google Cloud project containing theCloud KMS key is notified of related issues. For example, updates failwhen theKMS_KEY_NAME is invalid,when the associated service account doesn't have the requiredCloud Key Management Service CryptoKey Encrypter/Decrypter role,or when access to the key is disabled.

After you configure CMEK, at least on of the following occurs:

• You received a notification from Cloud Logging about CMEK access issues.

• You notice that CMEK isn't enabled for the_Default and_Requiredlog buckets when you create new Google Cloud projects in your organizationor in a folder.

• You receive errors when you read from log buckets with CMEK enabled, or ifyou try to create or update log buckets.

The notification provides information about the failure and it containsthat you can take to mitigate the issue:

Identify project that contains the Cloud KMS key

To identify the ID of the Google Cloud project that contains the encryption keyused by a log bucket, folder, or organization, do the following:

gcloud logging settings describe --project=PROJECT_ID

gcloud logging settings describe --folder=FOLDER_ID

gcloud logging settings describe --organization=ORGANIZATION_ID

The previous command returns information similar to the following:

kmsServiceAccountId:KMS_SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.comloggingServiceAccountId:SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com

For organizations and folders, the following field is also returned:

kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KMS_KEY_RING/cryptoKeys/KEY

The value of thekmsKeyName field include thes Google Cloud project that storesthe key.

Verify key usability

To verify the key's usability, run the following command to list all keys:

gcloud kms keys list \--location=KMS_KEY_LOCATION \--keyring=KMS_KEY_RING

This command returns information about each key in a tabular format. The firstline of the output is a list of column names:

NAME PURPOSE ...

Verify that the Cloud KMS key is listed in the command's output asENABLED, and that the purpose of the key is symmetric encryption: thePURPOSE column must containENCRYPT_DECRYPT and thePRIMARY_STATE columnmust containENABLED.

If necessary,create a new key.

Verify permissions configuration

Service accounts that are associated with the organization's CMEK settings musthave theCloud KMS CryptoKey Encrypter/Decrypterrole for the configured key.

To list the key's IAM policy, run the following command:

gcloud kms keys get-iam-policyKMS_KEY_NAME

If necessary, add the service account that contains theCloud KMS CryptoKey Encrypter/Decrypter role to the key.