- Notifications
You must be signed in to change notification settings - Fork4
Provides different Google Service Accounts and Scopes for pods running on Kubernetes
License
imduffy15/k8s-gke-service-account-assigner
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Provides Google Service Account Tokens to containers running inside a kubernetes cluster based on annotations.
Inspired by and heavily based offkube2iam
Service accounts are attached to instances and are accessible by services through the transparentusage by the google-cloud-sdk of the Google instance metadata API. When using the google-cloud-sdk,a call is made to the Google instance metadata API which provides temporary credentialsthat are then used to make calls to the Google service.
The problem is that in a multi-tenanted containers based world, multiple containers will be sharing the underlyingnodes. Given containers will share the same underlying nodes, they each get the same Google service account credentials.
The solution is to redirect the traffic that is going to the Google instance metadata API for docker containers to a containerrunning on each instance, make a call to the Google IAM Credentials API to retrieve temporary credentials and return these to the caller.Other calls will be proxied to the Google instance metadata API. This container will need to run with host networking enabledso that it can call the Google instance metadata API itself.
It is necessary to create an service account which has the roleroles/iam.serviceAccountTokenCreator so it get tokens for other service accounts and can assign it to each pod.
This service account should be associated to the kubernetes cluster and the kubernetes cluster should have the scopehttps://www.googleapis.com/auth/cloud-platform so that it can query theIAM credentials API
Additionally, its necessary to have enabled theGoogle IAM Credentials API for your project.
Run the k8s-gke-service-account-assigner container as a daemonset (so that it runs on each worker) withhostNetwork: true.The k8s-gke-service-account-assigner daemon and iptables rule (see below) need to run before all other pods that would requireaccess to Google resources.
apiVersion:extensions/v1beta1kind:DaemonSetmetadata:name:saassignerlabels:app:saassignerspec:template:metadata:labels:name:saassignerspec:hostNetwork:truecontainers: -image:imduffy15/k8s-gke-service-account-assigner:latestname:saassignerargs: -"--iptables=true" -"--host-ip=$(HOST_IP)" -"--node=$(NODE_NAME)" -"--default-service-account=default" -"--default-scopes=https://www.googleapis.com/auth/devstorage.read_write,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append"env: -name:HOST_IPvalueFrom:fieldRef:fieldPath:status.podIP -name:NODE_NAMEvalueFrom:fieldRef:fieldPath:spec.nodeNameports: -containerPort:8181hostPort:8181name:http
To prevent containers from directly accessing the Google instance metadata API and gaining unwanted access to Google resources,the traffic to169.254.169.254 must be proxied for docker containers.
iptables \ --append PREROUTING \ --protocol tcp \ --destination 169.254.169.254 \ --dport 80 \ --in-interface eth0 \ --jump DNAT \ --table nat \ --to-destination`curl http://169.254.169.254/computeMetadata/v1/instance/network-interfaces/0/ip -H"Metadata-Flavor: Google"`:8181This rule can be added automatically by setting--iptables=true, setting theHOST_IP environmentvariable, and running the container in a privileged security context (or more granularly withCAP_NET_ADMIN andCAP_NET_RAW).
Add anaccounts.google.com/service-account andaccounts.google.com/scopes annotation to your pods with the service accountand scopes that you want to used for this pod.
apiVersion:v1kind:Podmetadata:name:debug-shelllabels:name:debug-shellannotations:accounts.google.com/service-account:"<PROJECT-ID>-compute@developer.gserviceaccount.com"accounts.google.com/scopes:"https://www.googleapis.com/auth/cloud-platform"spec:restartPolicy:Nevercontainers: -image:imduffy15/docker-gcloudimagePullPolicy:Alwaysname:debug-shelltty:true
You can use--default-service-account and--default-scopes to set a fallback service account and scope to use when annotation is not set.
By using the flag --namespace-restrictions you can enable a mode in which the roles that pods can assume is restrictedby an annotation on the pod's namespace. This annotation should be in the form of a json array.
To allow the debug-shell pod specified above to run in the default namespace your namespace would look like the following.
apiVersion:v1kind:Namespacemetadata:annotations:accounts.google.com/allowed-service-accounts:| ["<PROJECT-ID>-compute@developer.gserviceaccount.com"]name:default
The entries in the array may also beglob patterns.For example, you could match any user-managed service account for a project using["*@<PROJECT-ID>.iam.gserviceaccount.com"].
This is the basic RBAC setup to get k8s-gke-service-account-assigner working correctly when your cluster is using rbac. Below is the bare minimum to get k8s-gke-service-account-assigner working.
First we need to make a service account.
---apiVersion:v1kind:ServiceAccountmetadata:name:k8s-gke-service-account-assignernamespace:kube-system
Next we need to setup roles and binding for the the process.
---apiVersion:v1items: -apiVersion:rbac.authorization.k8s.io/v1beta1kind:ClusterRolemetadata:name:k8s-gke-service-account-assignerrules: -apiGroups:[""]resources:["namespaces","pods"]verbs:["get","watch","list"] -apiVersion:rbac.authorization.k8s.io/v1beta1kind:ClusterRoleBindingmetadata:name:k8s-gke-service-account-assignersubjects: -kind:ServiceAccountname:k8s-gke-service-account-assignernamespace:kube-systemroleRef:kind:ClusterRolename:k8s-gke-service-account-assignerapiGroup:rbac.authorization.k8s.iokind:List
You will notice this lives in the kube-system namespace to allow for easier seperation between system services and other services.
By using the --debug flag you can enable some extra features making debugging easier:
/debug/storeendpoint enabled to dump knowledge of namespaces and service account association.
By default,k8s-gke-service-account-assigner will use the in-cluster method to connect to the kubernetes master, and use theaccounts.google.com/service-account andaccounts.google.com/scopes annotations to retrieve the serviceaccount and scopes for the container.
$ k8s-gke-service-account-assigner --helpUsage of ./k8s-gke-service-account-assigner: --api-server string Endpointfor the api server --api-token string Token to authenticate with the api server --app-port string Http port (default"8181") --backoff-max-elapsed-time duration Max elapsedtimefor backoff when queryingfor role. (default 2s) --backoff-max-interval duration Max intervalfor backoff when queryingfor role. (default 1s) --debug Enable debug features --default-scopes string Fallback scopes to use when annotation is notset --default-service-account string Fallback service account to use when annotation is notset --enable-metadata-proxy Send traffic to next-hop proxy --host-interface string Host interfacefor proxying google compute engine metadata (default"eth0") --host-ip string IP address of host --iam-role-key string Pod annotation key used to retrieve the IAM role (default"accounts.google.com/service-account") --insecure Kubernetes server should be accessed without verifying the TLS. Testing only --iptables Add iptables rule (also requires --host-ip) --log-format string Log format (text/json) (default"text") --log-level string Log level (default"info") --metadata-addr string Addressfor the google compute engine metadata (default"169.254.169.254") --metadata-proxy-addr string Addressfor the next-hop proxy, defaults to GKE's metadata-proxy location (default "127.0.0.1:988") --namespace-key string Namespace annotation key used to retrieve the service accounts allowed (value in annotation should be json array) (default "accounts.google.com/allowed-service-accounts") --namespace-restrictions Enable namespace restrictions --node string Name of the node where k8s-gke-service-account-assigner is running --verbose Verbose --version Print the version and exits
GKE's Metadata concealment protects some potentially sensitive system metadata from user workloads running on your cluster. It does so by deployingk8s-metadata-proxy and routing traffic to metadata service via this proxy. K8s-gke-service-account-assigner will take precedence, by putting its iptables rule before the k8s-metadata-proxy one, and therefore effectively bypassing it. If you want to use both, set--enable-metadata-proxy, in which case the traffic will we sent to the--metadata-proxy-addr (defaults to127.0.0.1:988, GKE's metadata-proxy default location).
- Create a Google Kubernetes Engine cluster
- Run skaffold dev
- Run make watch
- Create a container and exec onto it to make queries against the deployed k8s-gke-service-account-assigner instance
About
Provides different Google Service Accounts and Scopes for pods running on Kubernetes
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Contributors5
Uh oh!
There was an error while loading.Please reload this page.