- Notifications
You must be signed in to change notification settings - Fork36
Securely access AWS services from GKE cluster
License
doitintl/gtoken
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Ever wanted to access AWS services from Google Kubernetes cluster (GKE) without using AWS IAM credentials?
This solution can help you to get and exchange Google OIDC token for temporary AWS IAM security credentials are generated by AWS STS service. This approach allows you to access AWS services form a GKE cluster without pre-generated long-living AWS credentials.
Read more about this solution on DoiTSecurely Access AWS Services from Google Kubernetes Engine (GKE) blog post.
Thegtoken tool can get Google Cloud ID token when running with under GCP Service Account (for example, GKE Pod with Workload Identity).
NAME: gtoken - generate ID token with current Google Cloud service accountUSAGE: gtoken [global options] command [command options] [arguments...]COMMANDS: help, h Shows a list of commands or help for one commandGLOBAL OPTIONS: --refresh auto refresh ID token before it expires (default: true) --file value write ID token into file (stdout, if not specified) --help, -h show help (default: false) --version, -v print the versionThegtoken-webhook is a Kubernetes mutating admission webhook, that mutates any K8s Pod running under specially annotated Kubernetes Service Account (see details below).
Thegtoken-webhook injects agtokeninitContainer into a target Pod and an additionalgtoken sidekick container (to refresh an ID OIDC token a moment before expiration), mountstoken volume and injects three AWS-specific environment variables. Thegtoken container generates a valid GCP OIDC ID Token and writes it to thetoken volume.
Injected AWS environment variables:
AWS_WEB_IDENTITY_TOKEN_FILE- the path to the web identity token file (OIDC ID token)AWS_ROLE_ARN- the ARN of the role to assume by Pod containersAWS_ROLE_SESSION_NAME- the name applied to this assume-role session
The AWS SDK will automatically make the correspondingAssumeRoleWithWebIdentity calls to AWS STS on your behalf. It will handle in memory caching as well as refreshing credentials as needed.
Thegtoken-webhook can be configured to skip injection for all Pods in the specific Namespace by adding theadmission.gtoken/ignore label to the Namespace.
- Create a new
gtokennamespace:
kubectl create -f deployment/namespace.yaml
1. To deploy the `gtoken-webhook` server, we need to create a webhook service and a deployment in our Kubernetes cluster. It’s pretty straightforward, except one thing, which is the server’s TLS configuration. If you’d care to examine the [deployment.yaml](https://github.com/doitintl/gtoken/blob/master/deployment/deployment.yaml) file, you’ll find that the certificate and corresponding private key files are read from command line arguments, and that the path to these files comes from a volume mount that points to a Kubernetes secret:```yaml[...] args: [...] - --tls-cert-file=/etc/webhook/certs/cert.pem - --tls-private-key-file=/etc/webhook/certs/key.pem volumeMounts: - name: webhook-certs mountPath: /etc/webhook/certs readOnly: true[...] volumes: - name: webhook-certs secret: secretName: gtoken-webhook-certsThe most important thing to remember is to set the corresponding CA certificate later in the webhook configuration, so theapiserver will know that it should be accepted. For now, we’ll reuse the script originally written by the Istio team to generate a certificate signing request. Then we’ll send the request to the Kubernetes API, fetch the certificate, and create the required secret from the result.
First, runwebhook-create-signed-cert.sh script and check if the secret holding the certificate and key has been created:
./deployment/webhook-create-signed-cert.shcreating certs in tmpdir /var/folders/vl/gxsw2kf13jsf7s8xrqzcybb00000gp/T/tmp.xsatrckI71Generating RSA private key, 2048 bit long modulus.........................+++....................+++e is 65537 (0x10001)certificatesigningrequest.certificates.k8s.io/gtoken-webhook-svc.gtoken createdNAME AGE REQUESTOR CONDITIONgtoken-webhook-svc.gtoken 1s alexei@doit-intl.com Pendingcertificatesigningrequest.certificates.k8s.io/gtoken-webhook-svc.gtoken approvedsecret/gtoken-webhook-certs configuredNote For the GKE Autopilot, run thewebhook-create-self-signed-cert.sh script to generate a self-signed certificate.
Export CA Bundle as environment variable:
export CA_BUNDLE=[output value of the previous script"Encoded CA:"]
Then, we’ll create the webhook service and deployment.
First, create a Kubernetes Service Account to be used with thegtoken-webhook:
kubectl create -f deployment/service-account.yaml
Once the secret is created, we can create deployment and service. These are standard Kubernetes deployment and service resources. Up until this point we’ve produced nothing but an HTTP server that’s accepting requests through a service on port 443:
kubectl create -f deployment/deployment.yamlkubectl create -f deployment/service.yaml
Now that our webhook server is running, it can accept requests from theapiserver. However, we should create some configuration resources in Kubernetes first. Let’s start with our validating webhook, then we’ll configure the mutating webhook later. If you take a look at thewebhook configuration, you’ll notice that it contains a placeholder forCA_BUNDLE:
[...]service:name:gtoken-webhook-svcnamespace:gtokenpath:"/pods"caBundle:${CA_BUNDLE}[...]
There is asmall script that substitutes the CA_BUNDLE placeholder in the configuration with this CA. Run this command before creating the validating webhook configuration:
cat ./deployment/mutatingwebhook.yaml| ./deployment/webhook-patch-ca-bundle.sh> ./deployment/mutatingwebhook-bundle.yaml
Create mutating webhook configuration:
kubectl create -f deployment/mutatingwebhook-bundle.yaml
Define RBAC permission for webhook service account:
# create a cluster rolekubectl create -f deployment/clusterrole.yaml# define a cluster role bindingkubectl create -f deployment/clusterrolebinding.yaml
PROJECT_ID- GCP project IDCLUSTER_NAME- GKE cluster nameCLUSTER_ZONE- GKE cluster zoneGSA_NAME- Google Cloud Service Account name (choose any)GSA_ID- Google Cloud Service Account unique ID (generated by Google)KSA_NAME- Kubernetes Service Account name (choose any)KSA_NAMESPACE- Kubernetes namespaceAWS_ROLE_NAME- AWS IAM role name (choose any)AWS_POLICY_NAME- anexisting AWS IAM policy to assign to IAM roleAWS_ROLE_ARN- AWS IAM Role ARN identifier (generated by AWS)
Create a new GKE cluster withWorkload Identity enabled:
gcloud container clusters create${CLUSTER_NAME} \ --zone=${CLUSTER_ZONE} \ --workload-pool=${PROJECT_ID}.svc.id.goog
or update an existing cluster:
gcloud container clusters update${CLUSTER_NAME} \ --zone=${CLUSTER_ZONE} \ --workload-pool=${PROJECT_ID}.svc.id.goog
Create Google Cloud Service Account:
# create GCP Service Accountgcloud iam service-accounts create${GSA_NAME}# get GCP SA UID to be used for AWS Role with Google OIDC Web IdentityGSA_ID=$(gcloud iam service-accounts describe --format json${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com| jq -r'.uniqueId')
UpdateGSA_NAME Google Service Account with following roles:
roles/iam.workloadIdentityUser- impersonate service accounts from GKE Workloadsroles/iam.serviceAccountTokenCreator- impersonate service accounts to create OAuth2 access tokens, sign blobs, or sign JWTs
gcloud projects add-iam-policy-binding${PROJECT_ID} \ --member serviceAccount:${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com \ --role roles/iam.serviceAccountTokenCreatorgcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member"serviceAccount:${PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${KSA_NAME}]" \${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
# prepare role trust policy document for Google OIDC providercat> gcp-trust-policy.json<<EOF{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "accounts.google.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "accounts.google.com:sub": "${GSA_ID}" } } } ]}EOF# create AWS IAM Rome with Google Web Identityaws iam create-role --role-name${AWS_ROLE_NAME} --assume-role-policy-document file://gcp-trust-policy.json# assign AWS role desired policiesaws iam attach-role-policy --role-name${AWS_ROLE_NAME} --policy-arn arn:aws:iam::aws:policy/${AWS_POLICY_NAME}# get AWS Role ARN to be used in K8s SA annotationAWS_ROLE_ARN=$(aws iam get-role --role-name${AWS_ROLE_NAME} --query Role.Arn --output text)
Create K8s namespace:
kubectl create namespace${K8S_NAMESPACE}Create K8s Service Account:
kubectl create serviceaccount --namespace${K8S_NAMESPACE}${KSA_NAME}
Annotate K8s Service Account with GKE Workload Identity (GCP Service Account email)
kubectl annotate serviceaccount --namespace${K8S_NAMESPACE}${KSA_NAME} \ iam.gke.io/gcp-service-account=${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com
Annotate K8s Service Account with AWS Role ARN:
kubectl annotate serviceaccount --namespace${K8S_NAMESPACE}${KSA_NAME} \ amazonaws.com/role-arn=${AWS_ROLE_ARN}
Run a new K8s Pod with K8s ${KSA_NAME} Service Account:
# run a pod (with AWS CLI onboard) in interactive modcat<<EOF | kubectl apply -f -apiVersion: v1kind: Podmetadata: name: test-pod namespace:${K8S_NAMESPACE}spec: serviceAccountName:${KSA_NAME} containers: - name: test-pod image: mikesir87/aws-cli command: ["tail", "-f", "/dev/null"]EOF# in Pod shell: check AWS assumed roleaws sts get-caller-identity# the output should look similar to below{"UserId":"AROA9GB4GPRFFXVHNSLCK:gtoken-webhook-gyaashbbeeqhpvfw","Account":"906385953612","Arn":"arn:aws:sts::906385953612:assumed-role/bucket-full-gtoken/gtoken-webhook-gyaashbbeeqhpvfw"}
I've borrowed an initial mutating admission webhook code and deployment guide frombanzaicloud/admission-webhook-example repository. Big thanks to Banzai Cloud team!
About
Securely access AWS services from GKE cluster
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Contributors2
Uh oh!
There was an error while loading.Please reload this page.