Rotating Cassandra credentials in Kubernetes secrets

Rotating Cassandra secrets in Kubernetes secrets

This procedure describes rotating Cassandra credentials in Kubernetes secrets in your cluster. For rotating credentials within Hashicorp Vault, seeRotating Cassandra credentials in Hashicorp Vault.

This procedure will enable you to rotate the Cassandra credentials used for production Apigee Hybrid services without causing downtime by automating and triggering three tasks:

Rotation of user credentials
Rollback of the rotation
Cleanup of old and new credentials

Rotate Cassandra secrets in Kubernetes secrets

There are six Cassandra credential pairs (username+password) that can be rotated:

Default *
Admin
DDL
DML
JMX
Jolokia

^*Note: You can only rotate the password for the default Cassandra user. The username cannot be changed.

Prepare for credential rotation

To perform these steps, you must have permission to runkubectl apply -f within the cluster.

Backup the Cassandra database. This backup is to ensure recovery is possible to pre-rotated credentials. See Cassandra backup overview.
Make sure the cluster is in a healthy state, that all Apigee resources are running and no state changes are pending.

Single region credential rotation

Create a new Kubernetes secret in your Apigee namespace which holds the Cassandra user credentials you want to rotate, followingSetting usernames and passwords in a Kubernetes secret.
Perform the rotation precheck.
1. Create a new SecretRotation custom resource using the following template:
```
# rotation-k8s-secrets.yamlapiVersion: apigee.cloud.google.com/v1alpha1kind: SecretRotationmetadata:  name:ROTATION_NAME  namespace:APIGEE_NAMESPACEspec:  organizationId:APIGEE_ORG  rotationId:ROTATION_ID  timeoutMinutes:TIMEOUT_MINUTES # optional. 480 recommended.  precheck: true  cassandra:    oldSecretRef:OLD_SECRET_REF    newSecretRef:NEW_SECRET_REF    jobType: ROTATE
```
  Where:
  - ROTATION_NAME: A name for the rotation job, for example:sr-1-precheck.
  - APIGEE_NAMESPACE: your Apigee namespace.
  - APIGEE_ORG: Your Apigee organization ID.
  - ROTATION_ID: A custom identifier, for example:rot-1-precheck.
  - TIMEOUT_MINUTES: Optional. Overrides the default (480m == 8hr). <=0 means infinite timeout.
  - PRECHECK: Triggers dry-run execution. When set to true, the rotation process goes through all the steps except actually applying the new credentials.
  - OLD_SECRET_REF: The secret name currently being used by theapigeedatastore. You can find this using the following command:
```
kubectl -nAPIGEE_NAMESPACE get apigeedatastore default -o jsonpath='{.spec.credentialRef}'
```
  - NEW_SECRET_REF: The secret name created in theprevious step.
2. Trigger the rotation precheck job by applying the rotation YAML file:
```
kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME
```
  When the job is finished, the output will contain:
```
Status:  Message:  rotation complete  State:    complete
```
  Tip: You can also view the logs while the job is running with the following command:
```
kubectl -nAPIGEE_NAMESPACE logs sr-ROTATION_ID-rotate-job-HASH
```
When the rotation precheck job completes, perform the rotation:
1. Make the following changes in the rotation YAML file:
  - Change themetadata.name to a name indicating this is not a precheck, for example:sr-1.
  - Setspec.precheck tofalse.
  - Changespec.rotationId to a new identifier, for example:rot-1.
2. Apply the file again to perform the rotation.
```
kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE
```
Once the rotation job completes, validate that traffic is still flowing correctly.
After validating your traffic, perform the cleanup of the old secrets.Caution: Once you trigger the cleanup job, you will not be able to perform a rollback to revert the rotation process and leave the cluster in its original state. SeeRolling back a rotation.
1. Make the following changes in the rotation YAML file:
  - Change themetadata.name to a name indicating this is a cleanup job, for example:sr-1-cleanup.
  - Setspec.cassandra.jobType toCLEANUP.
  - Ensurespec.precheck is set tofalse.
2. Apply the file again to perform the cleanup.
```
kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME
```
  When the cleanup job is completed, the rotation process is complete.
Update your overrides file and setcassandra.auth.secret to the new secret name (newSecretRef).
```
cassandra:  auth:    secret:NEW_SECRET_REF
```
Take a backup of the Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
Delete the old Cassandra credential secret.

Multi-region credential rotation

in each region, create a new Kubernetes secret in the your Apigee namespace which holds the Cassandra user credentials you want to rotate, followingSetting usernames and passwords in a Kubernetes secret.
The credentials must be the same in each region.
Perform the rotation precheck.
1. Create a new SecretRotation custom resource using the following template:
```
# rotation-k8s-secrets.yamlapiVersion: apigee.cloud.google.com/v1alpha1kind: SecretRotationmetadata:  name:ROTATION_NAME  namespace:APIGEE_NAMESPACEspec:  organizationId:APIGEE_ORG  rotationId:ROTATION_ID  timeoutMinutes:TIMEOUT_MINUTES # optional. 480 recommended.  precheck: true  cassandra:    oldSecretRef:OLD_SECRET_REF    newSecretRef:NEW_SECRET_REF    jobType: ROTATE
```
  Where:
  - ROTATION_NAME: A name for the rotation job, for example:sr-1-precheck.
  - APIGEE_NAMESPACE: your Apigee namespace.
  - APIGEE_ORG: Your Apigee organization ID.
  - ROTATION_ID: A custom identifier, for example:rot-1-precheck.
  - TIMEOUT_MINUTES: Optional. Overrides the default (480m == 8hr). <=0 means infinite timeout.
  - PRECHECK: Triggers dry-run execution. When set to true, the rotation process goes through all the steps except actually applying the new credentials.
  - OLD_SECRET_REF: The secret name currently being used by theapigeedatastore. You can find this using the following command:
```
kubectl -nAPIGEE_NAMESPACE get apigeedatastore default -o jsonpath='{.spec.credentialRef}'
```
  - NEW_SECRET_REF: The secret name created in theprevious step.
2. In the first region, trigger the rotation precheck job by applying the rotation YAML file:
```
kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME
```
  When the job is finished, the output will contain:
```
Status:  Message:  rotation complete  State:    complete
```
  Tip: You can also view the logs while the job is running with the following command:
```
kubectl -nAPIGEE_NAMESPACE logs sr-ROTATION_ID-rotate-job-HASH
```
When the rotation precheck job completes, perform the rotation in the same region:
1. Make the following changes in the rotation YAML file:
  - Change themetadata.name to a name indicating this is not a precheck, for example:sr-1.
  - Setspec.precheck tofalse.
  - Changespec.rotationId to a new identifier, for example:rot-1.
2. Apply the file again to perform the rotation.
```
kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE
```
Once the rotation job completes, validate that traffic is still flowing correctly.
Repeat the rotation steps for the remaining regions.
After validating your traffic, perform the cleanup of the old secrets. You only need to perform the cleanup in a single region. It does not matter which region you choose.Caution: Once you trigger the cleanup job, you will not be able to perform a rollback to revert the rotation process and leave the cluster in its original state. SeeRolling back a rotation.
1. Make the following changes in the rotation YAML file:
  - Change themetadata.name to a name indicating this is a cleanup job, for example:sr-1-cleanup.
  - Setspec.cassandra.jobType toCLEANUP.
  - Ensurespec.precheck is set tofalse.
2. Apply the file again to perform the cleanup.
```
kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE
```
3. Check the job status and wait for it to complete.
```
kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME
```
  When the cleanup job is completed, the rotation process is complete.
Update your overrides file and setcassandra.auth.secret to the new secret name (newSecretRef).
```
cassandra:  auth:    secret:NEW_SECRET_REF
```
Take a backup of the Cassandra database. This backup is to ensure recovery is possible to post-rotated credentials.
Delete the old Cassandra credential secret.

Rolling back a rotation

Note: You can only roll back a rotation before the cleanup job has been triggered.

Perform the following steps in each region.

Create a new SecretRotation custom resource using the following template:

# rollback-k8s-secrets.yamlapiVersion: apigee.cloud.google.com/v1alpha1kind: SecretRotationmetadata:  name:ROLLBACK_NAME  namespace:APIGEE_NAMESPACEspec:  organizationId:APIGEE_ORG  rotationId:ROTATION_ID # match the current rotation.  timeoutMinutes:TIMEOUT_MINUTES # optional.  precheck:false  cassandra:    oldSecretRef:OLD_SECRET_REF # Must match the previous oldSecretRef.    newSecretRef:NEW_SECRET_REF # Must match the previous newSecretRef.    jobType:ROLLBACK

Where:

ROLLBACK_NAME: A name for the rollback job, for example:sr-1-rollback.

APIGEE_NAMESPACE: your Apigee namespace.

APIGEE_ORG: Your Apigee organization ID.

ROTATION_ID: The id of the current rotation that you are rolling back, for example:rot-1.

TIMEOUT_MINUTES: Optional. Overrides the default (480m == 8hr). <=0 means infinite timeout.

OLD_SECRET_REF: This must match the secret name foroldSecretRef: in the rotation YAML file you used in Single region rotation orMulti-region rotation procedure.
NEW_SECRET_REF: this must match the secret name fornewSecretRef: in the rotation YAML file you used inSingle region rotation orMulti-region rotation procedure.

Apply the rollback:

kubectl -nAPIGEE_NAMESPACE apply -fROLLBACK_YAML_FILE

Check the job status and wait for it to complete.

kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME

When the rollback(s) complete, verify that traffic is still flowing correctly.
When the traffic is flowing correctly, repeat the rollback process in each remaining region.
Once you have completed the rollback and verified that traffic is still flowing correctly, start the cleanup process. You only need to perform the cleanup in a single region. It does not matter which region you choose.
Make the following changes in the rotation YAML file:
- Changemetadata.name to a name indicating this is a cleanup job, for example:sr-1-cleanup-rollback.
- Changespec.cassandra.jobType toCLEANUP_ROLLBACK.

Apply the file to trigger the cleanup job:

kubectl -nAPIGEE_NAMESPACE apply -fROTATION_YAML_FILE

Check the job status and wait for it to complete.
```
kubectl -nAPIGEE_NAMESPACE describe srROTATION_NAME
```
When the cleanup job is completed, the rollback process is complete.
Update your overrides file and setcassandra.auth.secret to the old secret name (oldSecretRef).
```
cassandra:  auth:    secret:OLD_SECRET_REF
```

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-02-18 UTC.

Movatterモバイル変換

Rotating Cassandra credentials in Kubernetes secrets Stay organized with collections Save and categorize content based on your preferences.