This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of theService Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see thelaunch stage descriptions.

Storing Cassandra secrets in Hashicorp Vault

This feature allows you to store Cassandra DB credentials for Apigee Hybrid inHashicorp Vault, an external secret manager. External secret managers allow you to manage how secrets are stored in Kubernetes, including managing data residency and fine grained access controls.

Before Apigee hybrid version 1.10, the only way to supply passwords for Cassandra users was to specify the password in overrides.yaml. These passwords are stored in Kubernetes secrets. For example:

cassandra:  auth:    default:      password: "********"    admin:      password: "********"    ddl:      password: "********"    dml:      password: "********"    jmx:      username: "jmxuser"      password: "********"    jolokia:      username: "apigee"      password: "********"

Using Hashicorp Vault, you can supply these passwords via the Kubernetes Secrets Store CSI Driver API (SecretProviderClass). This allows Kubernetes to mount multiple secrets, keys, and certs stored in an external Vault.

Cassandra users and passwords

You will need to create secrets for the following Cassandra users. Change the default values to meet your organization's security policies.

Seecassandra configuration property for more information.

Configure external secret integration

Setting up Vault integration for Apigee hybrid consists of the following procedures.

• In the first two procedures, you interact directly with Vault.
• In the third and fourth procedures, you apply the configurations to your Kubernetes cluster.

Use the following procedures to create the secrets in Vault and enable your hybrid installation to have access to them.

Create Vault secrets, policies, and roles

1. Verify that the current Kubernetes context is set to your cluster:

   kubectl config current-context

2. Use the Vault API, CLI, or UI to create the cassandra secrets. The secret values you create must match the Cassandra usernames and passwords currently used in your cluster.
   • Secret key: Any secret key (or combination of multiple keys) can be used, for example:

     secret/data/apigee/cassandra

   • Secret data: Apigee Hybrid expects username and password pairs for the following Cassandra users:

     Cassandra users
     Admin
     DDL
     Default
     DML
     JMX
     Jolokia

     These username and password values can be spread across any number of secret keys.
   • Vault CLI: The following command shows how to create a single secret containing all the required usernames and passwords:

     vault kv put secret/apigee/cassandra \    adminUsername="ADMIN_USERNAME" \    adminPassword="ADMIN_PASSWORD" \    ddlUsername="DDL_USERNAME" \    ddlPassword="DDL_PASSWORD" \    defaultUsername="cassandra" \    defaultPassword="DEFAULT_PASSWORD" \    dmlUsername="DML_USERNAME" \    dmlPassword="DML_PASSWORD" \    jmxUsername="JMX_USERNAME" \    jmxPassword="JMX_PASSWORD" \    jolokiaUsername="JOLOKIA_USERNAME" \    jolokiaPassword="JOLOKIA_PASSWORD"

     Note:data/ should not be included in the secret key when running this command. Vault automatically inserts it. The default usernames for each user are as follows:

     Cassandra user	Default value
     Admin	admin_user
     DDL	ddl_user
     Default	cassandra
     DML	dml_user
     JMX	jmxuser
     Jolokia	apigee

     Tip: You can see the default passwords in thevalues.yaml file in theapigee-datastore/ chart directory.
3. Within Vault, create a policy to grant access to the secret you just created.
   1. Create a policy file (suggested name:apigee-cassandra-auth.txt) with the following contents:

      path "secret/data/apigee/cassandra" {  capabilities = ["read"]}

      If you created multiple secrets, each secret must be added to the policy file:

      path "secret/data/apigee/cassandra/admin" {  capabilities = ["read"]}path "secret/data/apigee/cassandra/ddl" {  capabilities = ["read"]}

      Note: Do not forgetdata/ in the paths (e.g. secret/data/apigee/cassandra).
   2. Apply the policy to Vault:

      vault policy write apigee-cassandra-auth apigee-cassandra-auth.txt

      It is possible to create the policy using standard input instead of reading from a file:

      echo 'path "secret/data/apigee/cassandra" { capabilities = ["read"] }' | vault policy write apigee-cassandra-auth -

      Note:apigee-cassandra-auth is the policy name. Any value can be used, but this exact value must be specified when binding the policy to Kubernetes service accounts in the next step.
4. Bind the policy to the Apigee Cassandra Kubernetes service accounts.
   1. Define the following environmental variables:

      export ORG_NAME=APIGEE_ORG_NAMEexport ENVS_LIST=LIST_OF_APIGEE-ENVSexport APIGEE_NAMESPACE=YOUR_APIGEE_NAMESPACEexport NAMESPACES=apigee-system,${APIGEE_NAMESPACE}

      Where:

      • ORG_NAME is the name of your Apigee organization.
      • ENVS_LIST Is a comma separated list of your Apigee environments, for exampledev,prod.
      • APIGEE_NAMESPACE is your Apigee namespace. The default isapigee.
      • NAMESPACES is a comma-separated list of namespaces for Apigee,apigee-system and your Apigee namespace.
   2. Create a script with the following contents. The script can have any name. In the following example, the name of the script iscreate-vault-cassandra-role.sh:

      # create-vault-cassandra-role.shORG=ORG_NAME  # ORG nameENVS=ENVS_LIST # comma separated env names, for example: dev,prodorg_short_name=$(echo $ORG | head -c 15)encode=$(echo -n $ORG | shasum -a 256 | head -c 7)org_encode=$(echo "$org_short_name-$encode")names=apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-${org_encode},apigee-cassandra-schema-val-${org_encode},apigee-cassandra-user-setup-${org_encode},apigee-mart-${org_encode},apigee-mint-task-scheduler-${org_encode}for env in ${ENVS//,/ }do  env_short_name=$(echo $env | head -c 15)  encode=$(echo -n $ORG:$env | shasum -a 256 | head -c 7)  env_encode=$(echo "$org_short_name-$env_short_name-$encode")  names+=,apigee-synchronizer-${env_encode},apigee-runtime-${env_encode}doneecho $names

   3. Run the script and assign the output to theSERVICE_ACCOUNT_NAMES variable. This will create a comma-separated list of Kubernetes service account names.

      export SERVICE_ACCOUNT_NAMES=$(./create-vault-cassandra-role)

      Note: You may need tochmod your script before running it.

      Check that the variable was populated with the list:

      echo $SERVICE_ACCOUNT_NAMES

   4. Use the Vault CLI to create a role which binds the policy to Kubernetes service accounts:

      vault write auth/kubernetes/role/cassandra \    bound_service_account_names=${SERVICE_ACCOUNT_NAMES} \    bound_service_account_namespaces=${NAMESPACES} \    policies=apigee-cassandra-auth \    ttl=1m

      Note: The policy name must match the one created in the previous step (e.g.apigee-cassandra-auth).

Install CSI driver and Vault provider

Apigee hybrid v1.12.4 supports the following Helm chart versions:

1. Follow theSecrets Store CSI Driver installation instructions to Install the CSI driver on your cluster. The CSI driver has a Helm chart for installation.Note:On OpenShift installations, grant privileged access to thesecrets-store-csi-driver service account with the following command:

   oc adm policy add-scc-to-user privileged system:serviceaccount:"${NAMESPACE}":secrets-store-csi-driver

   WhereNAMESPACE is the namespace where you installed the CSI driver.

2. Follow the instructions inInstalling the Vault CSI provider to install the Vault CSI provider if you have not installed it already.Note:For installations on OpenShift, follow the procedures inInstallation on OpenShift.
   • ForKUBERNETES_APPLICATION_NAMESPACE provide the namespace where you installed Vault.
   • ForAPPLICATION_SERVICE_ACCOUNT provide the name of the Kubernetes service account for Vault. This is usuallyvault-csi-provider. You can check the name of the service account with the following command:

     kubectl -nKUBERNETES_APPLICATION_NAMESPACE get ds vault-csi-provider -oyaml | grep 'serviceAccountName'

           serviceAccountName: vault-csi-provider

Create SecretProviderClass object

TheSecretProviderClass resource tells the CSI driver what provider to communicate with when requesting secrets. The Cassandra users' credentials must be configured via this object. The following table shows the file names (objectNames) expected by Apigee Cassandra:

1. Create a YAML file for yourSecretProviderClass. The file name can be anything, for example:spc.yaml. Use the followingSecretProviderClass template to configure this resource:

   apiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata:  name: apigee-cassandra-auth-spcspec:  provider: vault  parameters:    roleName: apigee-cassandra-auth  # the roleName should match the vault role you created earlier in this procedure    # vaultAddress is the endpoint your Vault server is running at.    # If Vault is running in the same cluster as Apigee, the format will generally be:    # http://vault.<namespace>.svc.cluster.local:<vaultServicePort>    vaultAddress:VAULT_ADDRESS    # "objectName" is an alias used within the SecretProviderClass to reference    # that specific secret. This will also be the filename containing the secret.    # Apigee Cassandra expects these exact values so they must not be changed.    # "secretPath" is the path in Vault where the secret should be retrieved.    # "secretKey" is the key within the Vault secret response to extract a value from.    # For example, if the Vault secret is located at `secret/data/apigee/cassandra`    # and you want to specify the admin password, you would use the following:    # - objectName: "adminPassword"    #   secretPath: "secret/data/apigee/cassandra"    #   secretKey: "key within Vault secret specifying the admin password"    objects: |      - objectName: "adminUsername"        secretPath: ""        secretKey: ""      - objectName: "adminPassword"        secretPath: ""        secretKey: ""      - objectName: "defaultUsername"        secretPath: ""        secretKey: ""      - objectName: "defaultPassword"        secretPath: ""        secretKey: ""      - objectName: "ddlUsername"        secretPath: ""        secretKey: ""      - objectName: "ddlPassword"        secretPath: ""        secretKey: ""      - objectName: "dmlUsername"        secretPath: ""        secretKey: ""      - objectName: "dmlPassword"        secretPath: ""        secretKey: ""      - objectName: "jolokiaUsername"        secretPath: ""        secretKey: ""      - objectName: "jolokiaPassword"        secretPath: ""        secretKey: ""      - objectName: "jmxUsername"        secretPath: ""        secretKey: ""      - objectName: "jmxPassword"        secretPath: ""        secretKey: ""

2. Apply theSecretProviderClass to both yourapigee andapigee-system namespaces. In the following commands, the namespaces areapigee andapigee-system. Replace that values if you are using different namespaces:

   kubectl -napigee apply -fspc.yamlkubectl -napigee-system apply -fspc.yaml

Enable external secret for Cassandra

1. Within youroverrides.yaml, add the following configuration to enable external secret usage for Cassandra:

   cassandra:  auth:    secretProviderClass:apigee-cassandra-auth-spc  # The name of the SecretProviderClass created in spc.yaml.

   Note: The name you provide forcassandra.auth.secretProviderClass must match the name of the SecretProviderClass you created in thespc.yaml file in the previous steps.

   Seecassandra.auth.secretProviderClass.

2. Usehelm upgrade to apply the change to theapigee-operator andapigee-datastore components:
   • The datastore controller inapigee-operator takes part in Cassandra decommissioning and data replication during region expansion. These tasks require the JMX and Jolokia credentials.

     helm upgrade operator apigee-operator/ \  --namespace apigee-system \  --atomic \  -f overrides.yaml

   • apigee-datastore provides credentials that downstream components likeapigee-runtime, Synchronizer & MART use when connecting to Cassandra.

     helm upgrade datastore apigee-datastore/ \  --namespaceapigee \  --atomic \  -f overrides.yaml

3. Verify external secrets are being used. When external secrets are enabled, newVolumes,Volume Mounts , andEnvironment Variables, are added referencing the secrets.
   • Verify theapigee-controller-manager deployment.

     Check that aVolume namedapigee-external-secrets exists and references theSecretProviderClass created above:

     kubectl -n apigee-system get deployment apigee-controller-manager -o jsonpath='{.spec.template.spec.volumes[?(@.name=="apigee-external-secrets")]}'

     Example output:

     {  "csi": {    "driver": "secrets-store.csi.k8s.io",    "readOnly": true,    "volumeAttributes": {      "secretProviderClass": "apigee-cassandra-auth-spc"    }  },  "name": "apigee-external-secrets"}

     Check that aVolumeMount namedapigee-external-secrets exists:

     kubectl -n apigee-system get deployment apigee-controller-manager -o jsonpath='{.spec.template.spec.containers[?(@.name=="manager")].volumeMounts[?(@.name=="apigee-external-secrets")]}'

     Example output:

     {  "mountPath": "/opt/apigee/externalsecrets",  "name": "apigee-external-secrets",  "readOnly": true}

     Check thatEnvironment Variables exist that reference external secrets:

     kubectl -n apigee-system get deployment apigee-controller-manager -o jsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env}'

     Example output:

     [  ...  {    "name": "CASSANDRA_JOLOKIA_USERNAME_PATH",    "value": "/opt/apigee/externalsecrets/jolokiaUsername"  },  {    "name": "CASSANDRA_JOLOKIA_PASSWORD_PATH",    "value": "/opt/apigee/externalsecrets/jolokiaPassword"  }]

Rollback to K8s Secret

1. To revert back to non-external secrets, remove thesecretProviderClass configuration inoverrides.yaml and use the previous configuration:

   cassandra:      auth:        secretProviderClass: apigee-cassandra-auth-spc # remove this line

2. Usehelm upgrade to apply the change to theapigee-operator andapigee-datastore components:

   helm upgrade operator apigee-operator/ \  --namespaceapigee-system \  --atomic \  -f overrides.yaml

   helm upgrade datastore apigee-datastore/ \  --namespaceapigee \  --atomic \  -f overrides.yaml

Troubleshooting: Create a client container for debugging

If you are using Vault, this section replaces instructions in the troubleshooting section,Create a client container for debugging.

This section explains how to create a client container from which you can accessCassandra debugging utilities such ascqlsh. These utilities allow you to query Cassandra tables and can be useful for debugging purposes.

Create the client container

To create the client container, follow these steps:

1. The container uses the TLS certificate from theapigee-cassandra-user-setup pod. The first step is to fetch this certificate name:

   kubectl get secrets -n apigee --field-selector type=kubernetes.io/tls | grep apigee-cassandra-user-setup | awk '{print $1}'

   This command returns the certificate name. For example:apigee-cassandra-user-setup-rg-hybrid-b7d3b9c-tls.

2. Open a new file and paste the following pod spec into it:

   apiVersion:v1kind:Podmetadata:labels:name:CASSANDRA_CLIENT_NAME#Forexample:my-cassandra-clientnamespace:apigeespec:containers:-name:CASSANDRA_CLIENT_NAMEimage:"gcr.io/apigee-release/hybrid/apigee-hybrid-cassandra-client:1.12.4"imagePullPolicy:Alwayscommand:-sleep-"3600"env:-name:CASSANDRA_SEEDSvalue:apigee-cassandra-default.apigee.svc.cluster.local-name:APIGEE_DML_USERNAME_PATHvalue:/opt/apigee/externalsecrets/dmlUsername-name:APIGEE_DML_PASSWORD_PATHvalue:/opt/apigee/externalsecrets/dmlPasswordvolumeMounts:-mountPath:/opt/apigee/sslname:tls-volumereadOnly:true-name:apigee-external-secretsmountPath:/opt/apigee/externalsecretsreadOnly:truevolumes:-name:tls-volumesecret:defaultMode:420secretName:apigee-cassandra-user-setup-vaibhavhybridor-8b3e61d-tls-name:apigee-external-secretscsi:driver:secrets-store.csi.k8s.ioreadOnly:truevolumeAttributes:secretProviderClass:apigee-cass-passwordserviceAccount:apigee-cassandra-defaultserviceAccountName:apigee-cassandra-defaultrestartPolicy:Never

3. Save the file with a.yaml extension. For example:my-spec.yaml.
4. Apply the spec to your cluster:

   kubectl apply -fmy-spec.yaml -n apigee

5. Log in to the container:

   kubectl exec -nCASSANDRA_CLIENT_NAME -it -- bash

6. Connect to the Cassandracqlsh interface with the following commands. Enter the commands exactly as shown:

   APIGEE_DML_USER=$(cat "$APIGEE_DML_USERNAME_PATH")export APIGEE_DML_USERAPIGEE_DML_PASSWORD=$(cat "$APIGEE_DML_PASSNAME_PATH")export APIGEE_DML_PASSWORDcqlsh ${CASSANDRA_SEEDS} -u ${APIGEE_DML_USER} -p ${APIGEE_DML_PASSWORD} --ssl

Deleting the client pod

Use this command to delete the Cassandra client pod:

kubectl delete pods -n apigee cassandra-client