You are currently viewing version 1.14 of the Apigee hybrid documentation. For more information, seeSupported versions.

Overview

This procedure describes rotating Cassandra credentials within Hashicorp Vault. For rotating credentials in Kubernetes secrets in your cluster, seeRotating Cassandra credentials in Kubernetes secrets.

This feature allows platform administrators to:

• Rotate Cassandra credentials in Hashicorp Vault.
• Roll back to the previous Cassandra credentials in Vault in case of any issues during password rotation.
• Rotate the Cassandra password for one region at a time, so that you can ensure minimal impact on service availability and maintain control over the rotation process.
• Track the start, progress, and completion of the rotation for a single region.

This feature is available in Apigee Hybrid 1.13.1 and later.

Note: Before using this feature, you must have Vault storage of Cassandra credentials set up. Follow the instructions inStoring Cassandra secrets in Hashicorp Vault

Before you begin

Before setting up credential rotation:

• Backup your Cassandra database. This backup is to ensure recovery is possible to pre-rotated credentials.
• Ensure the cluster is in a healthy state (i.e. all Apigee resources are running, no state changes are pending).

Note: You must have permission to runkubectl apply -f within the cluster to set up credential rotation.

Single region setup

1. Create a newSecretProviderClass Kubernetes resource in your Apigee namespace for the new Cassandra credentials.SeeStoring Cassandra secrets in Hashicorp Vault for a template to use.This allows a Vault role to access secrets within the Kubernetes namespaces.
2. Create a newSecretRotation custom resource using the following template:

   # rotation.yamlapiVersion: apigee.cloud.google.com/v1alpha1kind: SecretRotationmetadata:  name:ROTATION_PROCESS_NAME  namespace:APIGEE_NAMESPACEspec:  organizationId:ORG_NAME  rotationId:ROTATION_ID  timeoutMinutes: 480 # optional. overrides the default (480m == 8hr).                      # less than or equal to 0 means infinite timeout.  precheck: true  cassandra:    oldSecretProviderClass:OLD_SPC_NAME    newSecretProviderClass:NEW_SPC_NAME    jobType: ROTATE

   • ROTATION_PROCESS_NAME: A unique name for the rotation job. You will need to setmetadata.name to a unique value for the rotation precheck job and again for the rotation job. For examplesr-1-precheck followed bysr-1.
   • ROTATION_ID: Setspec.rotationId to a custom identifier, for examplerotation-1-precheck.
   • NEW_SPC_NAME: Setspec.cassandra.newSecretProviderClass to the new secret provider class name you created in the previous step.
   • OLD_SPC_NAME: Setspec.cassandra.oldSecretProviderClass to the SPC name currently being used by theApigeeDatastore.
3. Trigger the rotation precheck job by applying therotation.yaml file.

   kubectl -nAPIGEE_NAMESPACE apply -f rotation.yaml

4. Check the job status to verify when the precheck job is complete.

   kubectl -nAPIGEE_NAMESPACE get job sr-(rotationId)-(rotate|rollback|cleanup)-job

   Tip: You can also view the logs while the job is running.

   kubectl -nAPIGEE_NAMESPACE logs sr-(rotationId)-(rotate|rollback|cleanup)-job-(hash)

5. Once the rotation precheck job completes, change the value ofmetadata.name and setspec.precheck tofalse. Apply the file again to perform the rotation.

   kubectl -nAPIGEE_NAMESPACE apply -f rotation.yaml

6. After the rotation job completes and you have validated traffic is still flowing correctly, clean up the process with the following two steps:
   1. Update the value ofmetadata.name and setspec.cassandra.jobType toCLEANUP.
   2. Trigger the cleanup job by applying the file.

      kubectl -nAPIGEE_NAMESPACE apply -f rotation.yaml

      Note: Up until this step, you can trigger arollback job to revert the rotation process and leave the cluster in its original state.

   When the cleanup job is completed, the rotation process is complete.

7. Update your overrides file and setcassandra.auth.secretProviderClass to the new secret provider class (newSecretProviderClass).

   cassandra:  auth:    secretProviderClass:NEW_SPC_NAME

8. Backup your Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
9. Delete the old Cassandra credentials, role, and policy from Vault.

Multi-region setup

The multi-region setup procedures are divided into two sections: setup for the first region and setup for the remaining regions.

1. Complete the following steps in the first region before starting the subsequent regions.
   1. Create a newSecretProviderClass Kubernetes resource in theAPIGEE_NAMESPACE namespace for the new Cassandra credentials. SeeStoring Cassandra secrets in Hashicorp Vault for a template to use. This allows a Vault role to access secrets within the Kubernetes namespaces.
   2. Create a newSecretRotation custom resource using the following template:

      # rotation.yamlapiVersion: apigee.cloud.google.com/v1alpha1kind: SecretRotationmetadata:  name:ROTATION_PROCESS_NAME  namespace:APIGEE_NAMESPACEspec:  organizationId:ORG_NAME  rotationId:ROTATION_ID  timeoutMinutes: -1 # this value is required and should not be changed.  precheck: true  cassandra:    oldSecretProviderClass:OLD_SPC_NAME    newSecretProviderClass:NEW_SPC_NAME    jobType: ROTATE

      • ROTATION_PROCESS_NAME: A unique name for the rotation job. You will need to setmetadata.name to a unique value for the rotation precheck job and again for the rotation job. For examplesr-1-precheck followed bysr-1.
      • ROTATION_ID: Setspec.rotationId to a custom identifier, for examplerotation-1-precheck.
      • NEW_SPC_NAME: Setspec.cassandra.newSecretProviderClass to the new secret provider class name you created in the previous step.
      • OLD_SPC_NAME: Setspec.cassandra.oldSecretProviderClass to the SPC name currently being used by theApigeeDatastore.
   3. Trigger the rotation precheck job by applying therotation.yaml file.

      kubectl -nAPIGEE_NAMESPACE apply -f rotation.yaml

   4. Check the job status to verify when the precheck job is complete.

      kubectl -nAPIGEE_NAMESPACE get job sr-(rotationId)-(rotate|rollback|cleanup)-job

      Tip: You can also view the logs while the job is running.

      kubectl -nAPIGEE_NAMESPACE logs sr-(rotationId)-(rotate|rollback|cleanup)-job-(hash)

   5. Once the rotation precheck job completes:
      • Change the value ofmetadata.name, for example fromsr-1-precheck tosr-1.
      • Setspec.precheck tofalse to turn off the precheck and perform the rotation.
      • Setspec.rotationId to a new identifier, for examplerotation-1.
   6. Apply the file again to perform the rotation.

      kubectl -nAPIGEE_NAMESPACE apply -f rotation.yaml

   7. Check the state of theSecretRotation and wait until it iscomplete.

      kubectl -nAPIGEE_NAMESPACE get srSR_NAME

2. In each subsequent region, complete the following steps:
   1. Create a newSecretProviderClass Kubernetes resource in your Apigee namespace for the new Cassandra credentials.SeeStoring Cassandra secrets in Hashicorp Vault for a template to use.This should be the same definition as step 1a.
   2. Update youroverrides.yaml and setcassandra.auth.secretProviderClass to the match the value ofspec.cassandra.newSecretProviderClass in therotation.yaml file.

      cassandra:  auth:    secretProviderClass:NEW_SPC_NAME

   3. Apply the operator chart:

      helm upgrade operator apigee-operator/ \  --namespaceAPIGEE_NAMESPACE \  --atomic \  -fOVERRIDES_FILE

   4. A newReplicaSet will be created. Check that the new controller-manager pods are using the new SPC:

      export POD=NEW_CONTROLLER_MANAGER_POD_NAMEkubectl -nAPIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'

      The result should match the value you set forspec.cassandra.newSecretProviderClass inrotation.yaml, for example:

      kubectl -n apigee get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'my-new-spc

   5. Apply the datastore chart:

      helm upgrade datastore apigee-datastore/ \  --namespaceAPIGEE_NAMESPACE \  --atomic \  -fOVERRIDES_FILE

      Note: Applying the datastore chart will also trigger releases for the organization and environments.
   6. The datastore will go into a releasing state. Wait until the datastore has finished releasing and is in the running state.

      kubectl -nAPIGEE_NAMESPACE get apigeedatastoreDATASTORE_NAME

      DATASTORE_NAME isdefault in most installations.

   7. Check that the new datastore pods are using the new SPC:

      export POD=NEW_DATASTORE_POD_NAMEkubectl -nAPIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'

      The result should match the value you set forspec.cassandra.newSecretProviderClass inrotation.yaml, for example:

      kubectl -n apigee get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'my-new-spc

   8. Wait until the organization and environments are done releasing and have returned to the running state.

      kubectl -nAPIGEE_NAMESPACE get apigeeorgORG_NAMEkubectl -nAPIGEE_NAMESPACE get apigeeenvENV_NAME

   9. Check that the new MART, runtime, and synchronizer pods are using the new SPC:

      export POD=NEW_MART_POD_NAMEkubectl -nAPIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'export POD=NEW_RUNTIME_POD_NAMEkubectl -nAPIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'export POD=NEW_SYNCHRONIZER_POD_NAMEkubectl -nAPIGEE_NAMESPACE get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'

      The result should match the value you set forspec.cassandra.newSecretProviderClass inrotation.yaml, for example:

      kubectl -n apigee get pods $POD -o jsonpath='{.spec.volumes[?(@.name=="apigee-external-secrets")].csi.volumeAttributes.secretProviderClass}'my-new-spc

3. After completing the steps in every region and validate traffic is still flowing correctly, clean up the process in the first region with the following two steps:
   1. In the first region, update the value ofmetadata.name and setspec.cassandra.jobType toCLEANUP.
   2. Trigger the cleanup job by applying the file.

      kubectl -nAPIGEE_NAMESPACE apply -f rotation.yaml

   3. Check the job status and watch the job logs to verify when the cleanup job is complete.

   When the cleanup job is completed, the rotation process is complete.

4. Update your overrides file and setcassandra.auth.secretProviderClass to the new secret provider class (newSecretProviderClass).

   cassandra:  auth:    secretProviderClass:NEW_SPC_NAME

5. Backup your Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
6. Delete the old Cassandra credentials, role, and policy from Vault.

Rolling back a rotation

Note: You can only roll back a rotation before the cleanup job has been triggered.

For multi-region, perform the rollback in each region.

1. Create a new SecretRotation custom resource using the following template:

   # rollback-rotation.yamlapiVersion: apigee.cloud.google.com/v1alpha1kind: SecretRotationmetadata:  name:ROLLBACK_NAME  namespace:APIGEE_NAMESPACEspec:  organizationId:APIGEE_ORG  rotationId:ROTATION_ID # match the current rotation.  timeoutMinutes:TIMEOUT_MINUTES # optional.  precheck:false  cassandra:    oldSecretProviderClass:OLD_SPC_NAME # Must match the previous oldSecretProviderClass.    newSecretProviderClass:NEW_SPC_NAME # Must match the previous newSecretProviderClass.    jobType:ROLLBACK

   Where:

   • ROLLBACK_NAME: A name for the rollback job, for example:sr-1-rollback.
   • APIGEE_NAMESPACE: your Apigee namespace.
   • APIGEE_ORG: Your Apigee organization ID.
   • ROTATION_ID: The id of the current rotation that you are rolling back, for example:rot-1.
   • TIMEOUT_MINUTES: Optional. Overrides the default (480m == 8hr). <=0 means infinite timeout.
   • OLD_SPC_NAME: This must match the secret name foroldSecretProviderClass: in the rotation YAML file you used inSingle region setup orMulti-region setup procedure.
   • NEW_SPC_NAME: this must match the secret name fornewSecretProviderClass: in the rotation YAML file you used inSingle region setup orMulti-region setup procedure.
2. Apply the rollback:

   kubectl -nAPIGEE_NAMESPACE apply -fROLLBACK_YAML_FILE

3. Check the job status and wait for it to complete.

   kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME

4. When the rollback(s) complete, verify that traffic is still flowing correctly.
5. For multi-region installations, when the traffic is flowing correctly, repeat the rollback process in each region.
6. Once you have completed the rollback and verified that traffic is still flowing correctly in all regions, start the cleanup process.

   Make the following changes in the rotation YAML file:

   • Changemetadata.name to a name indicating this is a cleanup job, for example:sr-1-cleanup-rollback.
   • Changespec.cassandra.jobType toCLEANUP_ROLLBACK.
7. Apply the file to trigger the cleanup job:

   kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE

8. Check the job status and wait for it to complete.

   kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME

   When the cleanup job is completed, the rollback process is complete.

9. Update your overrides file and setcassandra.auth.secretProviderClass to the old secret provider class (oldSecretProviderClass).

   cassandra:  auth:    secretProviderClass:OLD_SPC_NAME