You are currently viewing version 1.15 of the Apigee hybrid documentation. For more information, seeSupported versions.

Storing data in an external secret manager

This guide tells you how to store and manage the following types of information in the external secret storage service,Hashicorp Vault, instead of in your overrides.yaml file.

• AX Hash Salt
• Redis password
• Encryption keys

To store other kinds of information in Vault, see:

• Storing Cassandra secrets in Hashicorp Vault
• Storing service account keys in Hashicorp Vault

Prerequisites

• AKubernetes secrets store CSI driver. You can install it by following instructions inSecrets Store CSI Driver: Installation. For supported versions, seeApigee hybrid supported platforms and versions: Secret Store CSI driver.
• TheVault CSI provider. You can install it by following instructions inInstalling the Vault CSI provider. For supported versions, seeApigee hybrid supported platforms and versions: Vault.

Procedure

1. Create the Vault secrets, policies, and roles.
   1. Use the Vault UI or APIs to create secrets and grant permissions for the Apigee Kubernetes service accounts to read those secrets as described here. The secrets need to consist of a key and one or more values as shown in the following table:

      Secret Key	Secret Data
      secret/data/apigee/axhashsalt	{ "ax-hash-salt": "AX_HASH_SALT_VALUE"}
      secret/data/apigee/redis	{ "redis-password": "REDIS_PASSWORD_VALUE"}
      secret/data/apigee/orgencryptionkeys	{ "kmsEncryptionKey": "KMS_ENCRYPTION_KEY_VALUE" "kvmEncryptionKey": "KVM_ENCRYPTION_KEY_VALUE" "contractEncryptionKey": "CONTRACT_ENCRYPTION_KEY_VALUE"} Important: ThecontractEncryptionKey value must be a base64 encoded string with a length of 16, 24, or 32 characters. SeeData encryption for more information.
      secret/data/apigee/envencryptionkeys	{ "cacheEncryptionKey": "CACHE_ENCRYPTION_KEY_VALUE" "kvmEncryptionKey": "KVM_ENCRYPTION_KEY_VALUE" "envKvmEncryptionKey": "ENV_KVM_ENCRYPTION_KEY_VALUE" "kmsEncryptionKey": "KMS_ENCRYPTION_KEY_VALUE"}

   2. Within Vault, create policies which grant access to the secrets:

      cat axhashsalt-auth-policy.txtpath "secret/data/apigee/axhashsalt" {  capabilities = ["read"]}cat redis-auth-policy.txtpath "secret/data/apigee/redis" {  capabilities = ["read"]}cat orgencryptionkeys-auth-policy.txtpath "secret/data/apigee/orgencryptionkeys" {  capabilities = ["read"]}cat envencryptionkeys-auth-policy.txtpath "secret/data/apigee/envencryptionkeys" {  capabilities = ["read"]}vault policy write apigee-axhashsalt-auth axhashsalt-auth-policy.txtvault policy write apigee-redis-auth redis-auth-policy.txtvault policy write apigee-orgencryptionkeys-auth orgencryptionkeys-auth-policy.txtvault policy write apigee-envencryptionkeys-auth envencryptionkeys-auth-policy.txt

   3. Create a script calledgenerate-encoded-sas.sh with the following contents:

      # generate-encoded-sas.shORG=$APIGEE_ORG            # Apigee organization nameENVS=$APIGEE_ENV_LIST      # comma separated env names, for example: dev,prodORG_SHORT_NAME=$(echo $ORG | head -c 15)ENCODE=$(echo -n $ORG | shasum -a 256 | head -c 7)ORG_ENCODE=$(echo "$ORG_SHORT_NAME-$ENCODE")NAMES="apigee-manager,apigee-redis-default,apigee-redis-envoy-default,apigee-mart-${ORG_ENCODE},apigee-mint-task-scheduler-${ORG_ENCODE}"for ENV in ${ENVS//,/ }do  ENV_SHORT_NAME=$(echo $ENV | head -c 15)  ENCODE=$(echo -n $ORG:$ENV | shasum -a 256 | head -c 7)  ENV_ENCODE=$(echo "$ORG_SHORT_NAME-$ENV_SHORT_NAME-$ENCODE")  NAMES+=,apigee-runtime-${ENV_ENCODE},apigee-synchronizer-${ENV_ENCODE}doneecho $NAMES

   4. Run the script to generate the service account name list to bind the policies to:

      chmod +x ./generate-encoded-sas.sh./generate-encoded-sas.sh

      The output should list the encoded service account names.

   5. Using the policies, create Vault roles which bind the required Apigee service accounts.

      vault write auth/kubernetes/role/apigee-axhashsalt \bound_service_account_names=BOUND_SA_NAMES \bound_service_account_namespaces=APIGEE_NAMESPACE \policies=apigee-axhashsalt-auth \ttl=1mvault write auth/kubernetes/role/apigee-redis \bound_service_account_names=BOUND_SA_NAMES \bound_service_account_namespaces=APIGEE_NAMESPACE \policies=apigee-redis-auth \ttl=1mvault write auth/kubernetes/role/apigee-orgencryptionkeys \bound_service_account_names=BOUND_SA_NAMES \bound_service_account_namespaces=APIGEE_NAMESPACE \policies=apigee-orgencryptionkeys-auth \ttl=1mvault write auth/kubernetes/role/apigee-envencryptionkeys \bound_service_account_names=BOUND_SA_NAMES \bound_service_account_namespaces=APIGEE_NAMESPACE \policies=apigee-envencryptionkeys-auth \ttl=1m

2. CreateSecretProviderClass objects.
   1. Add the following secrets via theSecretProviderClass resources. These resources tell the CSI driver what provider to communicate with when requesting secrets. The following table shows the file names (objectNames) expected by Apigee hybrid:

      Secret	Expected secret file names
      AX Hash Salt	ax-hash-salt
      Redis	redis-password
      Org Encryption Keys	kmsEncryptionKey kvmEncryptionKey contractEncryptionKey
      Env Encryption Keys	kmsEncryptionKey kvmEncryptionKey envKvmEncryptionKey cacheEncryptionKey

   2. Use the followingSecretProviderClass templates to configure these resources:

      # axhashsalt-spc.yamlapiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata:  name: apigee-axhashsalt-spcspec:  provider: vault  parameters:    roleName: apigee-axhashsalt    vaultAddress:VAULT_ADDRESS    # "objectName" is an alias used within the SecretProviderClass to reference    # that specific secret. This will also be the filename containing the secret.    # Apigee Hybrid expects these exact values so they must not be changed.    # "secretPath" is the path in Vault where the secret should be retrieved.    # "secretKey" is the key within the Vault secret response to extract a value from.    objects: |      - objectName: "ax-hash-salt"        secretPath: ""        secretKey: ""

      # redis-spc.yamlapiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata:  name: apigee-redis-spcspec:  provider: vault  parameters:    roleName: apigee-redis    vaultAddress:VAULT_ADDRESS    # "objectName" is an alias used within the SecretProviderClass to reference    # that specific secret. This will also be the filename containing the secret.    # Apigee Hybrid expects these exact values so they must not be changed.    # "secretPath" is the path in Vault where the secret should be retrieved.    # "secretKey" is the key within the Vault secret response to extract a value from.    objects: |      - objectName: "redis-password"        secretPath: ""        secretKey: ""

      # orgencryptionkeys-spc.yamlapiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata:  name: apigee-orgencryptionkeys-spcspec:  provider: vault  parameters:    roleName: apigee-orgencryptionkeys    vaultAddress:VAULT_ADDRESS    # "objectName" is an alias used within the SecretProviderClass to reference    # that specific secret. This will also be the filename containing the secret.    # Apigee Hybrid expects these exact values so they must not be changed.    # "secretPath" is the path in Vault where the secret should be retrieved.    # "secretKey" is the key within the Vault secret response to extract a value from.    objects: |      - objectName: "kmsEncryptionKey"        secretPath: ""        secretKey: ""      - objectName: "kvmEncryptionKey"        secretPath: ""        secretKey: ""      - objectName: "contractEncryptionKey"        secretPath: ""        secretKey: ""

      # envencryptionkeys-spc.yamlapiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata:  name: apigee-envencryptionkeys-spcspec:  provider: vault  parameters:    roleName: apigee-envencryptionkeys    vaultAddress:VAULT_ADDRESS    # "objectName" is an alias used within the SecretProviderClass to reference    # that specific secret. This will also be the filename containing the secret.    # Apigee Hybrid expects these exact values so they must not be changed.    # "secretPath" is the path in Vault where the secret should be retrieved.    # "secretKey" is the key within the Vault secret response to extract a value from.    objects: |      - objectName: "cacheEncryptionKey"        secretPath: ""        secretKey: ""      - objectName: "kvmEncryptionKey"        secretPath: ""        secretKey: ""      - objectName: "envKvmEncryptionKey"        secretPath: ""        secretKey: ""      - objectName: "kmsEncryptionKey"        secretPath: ""        secretKey: ""

      VAULT_ADDRESS is the endpoint where your Vault server is running. If Vault is running in the same cluster and namespace as Apigee, the format will generally behttp://vault.APIGEE_NAMESPACE.svc.cluster.local:VAULT_SERVICE_PORT.

   3. Apply the aboveSecretProviderClasses to theAPIGEE_NAMESPACE namespace:

      kubectl -nAPIGEE_NAMESPACE apply -f axhashsalt-spc.yamlkubectl -nAPIGEE_NAMESPACE apply -f redis-spc.yamlkubectl -nAPIGEE_NAMESPACE apply -f orgencryptionkeys-spc.yamlkubectl -nAPIGEE_NAMESPACE apply -f envencryptionkeys-spc.yaml

3. Enable the external secret for the AX Hash Salt.
   1. Within youroverrides.yaml file, add the following configuration to enable external secret usage for the AX Hash Salt:

      axHashSaltSecretProviderClass: apigee-axhashsalt-spc

   2. Apply the change by upgrading theorg Helm chart:

      helm upgrade org apigee-org/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yaml

4. Enable the external secret for the Redis password.
   1. Within youroverrides.yaml file, add the following configuration to enable external secret usage for the Redis password:

      redis:  auth:    secretProviderClass: apigee-redis-spc

   2. Then apply the changes by upgrading theoperator andredis charts in the following order:

      helm upgrade operator apigee-operator/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yamlhelm upgrade redis apigee-redis/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yaml

5. Enable the external secret for the encryption keys
   1. Within youroverrides.yaml file, add the following configuration to enable external secret usage for the org-level encryption keys:

      encryptionKeySecretProviderClass: apigee-orgencryptionkeys-spc

   2. Apply change by upgrading theorg Helm chart:

      helm upgrade org apigee-org/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yaml

   3. Within youroverrides.yaml file for each environment, add the following configuration for the env-specific encryption keys:

      envs:- name:ENV_NAME  encryptionKeySecretProviderClass: apigee-envencryptionkeys-spc

   4. Apply the change by upgrading theenv Helm chart once for each environment:

      helm upgradeENV_NAME apigee-env/ \  --namespaceAPIGEE_NAMESPACE \  --set env=ENV_NAME \  -f overrides.yaml

Note: After completing this procedure all secrets, including Cassandra backup service account secrets and Google service account keys, may still be present in the Kubernetes secrets. You can delete these secrets from Kubernetes once they have been securely stored in Vault.

Rollback

AX Hash Salt

1. Within youroverrides.yaml file, remove the configuration which enabled external secret usage for the AX Hash Salt:

   # Comment out or delete the following line:# axHashSaltSecretProviderClass: apigee-axhashsalt-spc

2. Apply the change by upgrading theorg Helm chart:

   helm upgrade org apigee-org/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yaml

Redis password

1. Within youroverrides.yaml file, remove the configuration which enabled external secret usage for the Redis password:

   redis:  auth:  # Comment out or delete the following line:  # secretProviderClass: apigee-redis-spc

2. Then apply the changes by upgrading theredis andoperator charts in the following order:

   helm upgrade redis apigee-redis/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yamlhelm upgrade operator apigee-operator/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yaml

Encryption keys

1. Within youroverrides.yaml file, remove the configuration which enabled external secret usage for the env encryption keys:

   envs:  - name:ENV_NAME  # Comment out or delete the following line:  # encryptionKeySecretProviderClass: apigee-envencryptionkeys-spc

2. Apply the change by upgrading theenv Helm chart once for each environment:

   helm upgradeENV_NAME apigee-env/ \  --namespaceAPIGEE_NAMESPACE \  --set env=ENV_NAME \  -f overrides.yaml

3. Within youroverrides.yaml file, remove the configuration which enabled external secret usage for the org encryption keys:

   # Comment out or delete the following line:# encryptionKeySecretProviderClass: apigee-orgencryptionkeys-spc

4. And then Upgrade theorg Helm chart:

   helm upgrade org apigee-org/ \  --namespaceAPIGEE_NAMESPACE \  -f overrides.yaml