This page contains the design of Role-based Access Control (RBAC) and Pod Security system that Toolforge Kubernetes cluster uses.

Roles are assigned at either the namespace level (rolebinding) or cluster level (clusterrolebinding) through bindings. A role binding links an API object to a user, serviceaccount or similar system object with one or more verbs. These verbs do not universally make sense for all API objects, and the documentation can be sparse outside of code-based, generated docs. In general, Toolforge user accounts are only permitted to act within their particular namespace, and therefore, they usually will have things applied via a rolebinding within the scope of their namespace.

As of today, we implement pod security controls usingkyverno policies.

The source of truth for the policy definition is in the maintain-kubeusers repository:

• https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/
• maintain_kubeusers resources/kyverno_pod_policy.py
• maintain_kubeusers resources/kyverno_pod_policy.yaml.tpl

Root on the controlplane can use the "cluster-admin" role by default. Not much else should be using that. Special roles should be defined for Toolforge services that offer the minimum required capabilities only. Toolforge users can all use the same role defined at the cluster level (a "ClusterRole") with a namespaced role binding.

The Toolforge users all share one cluster role that they can only use within their namespaces.

# RBAC minimum perms for toolforge users:# verbs for R/O# ["get", "list", "watch"]# verbs for R/W (there are some specific quirks like deletecollection)# ["get", "list", "watch", "create", "update", "patch", "delete"]apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRolemetadata:name:tools-userrules:-apiGroups:-""resources:-bindings-events-limitranges-namespaces-namespaces/status-persistentvolumeclaims-pods/log-pods/status-replicationcontrollers/status-resourcequotas-resourcequotas/statusverbs:-get-list-watch-apiGroups:-""resources:-configmaps-endpoints-pods-pods/attach-pods/exec-pods/portforward-pods/proxy-replicationcontrollers-replicationcontrollers/scale-secrets-services-services/proxyverbs:-get-list-watch-create-delete-deletecollection-patch-update-apiGroups:-appsresources:-controllerrevisions-daemonsetsverbs:-get-list-watch-apiGroups:-appsresources:-deployments-deployments/rollback-deployments/scale-replicasets-replicasets/scale-statefulsets-statefulsets/scaleverbs:-get-list-watch-create-delete-deletecollection-patch-update-apiGroups:-autoscalingresources:-horizontalpodautoscalersverbs:-get-list-watch-apiGroups:-batchresources:-cronjobs-jobsverbs:-get-list-watch-create-delete-deletecollection-patch-update-apiGroups:-extensionsresources:-daemonsetsverbs:-get-list-watch-apiGroups:-extensionsresources:-deployments-deployments/rollback-deployments/scale-ingresses-networkpolicies-replicasets-replicasets/scale-replicationcontrollers/scaleverbs:-get-list-watch-create-delete-deletecollection-patch-update-apiGroups:-networking.k8s.ioresources:-ingresses-networkpoliciesverbs:-get-list-watch-create-delete-deletecollection-patch-update-apiGroups:-policyresources:-poddisruptionbudgetsverbs:-get-list-watch

The easiest way to visualize all that is as a table.

The reason there is so much apparent repetition is because in various editions of Kubernetes, the same resources appear under multiple APIs as features are graduated from alpha/beta/extensions into core APIs or the Apps API. In later editions (1.16, for instance) many of the resources under extensions are only found under apps.

Most of this is likely not controversial, but there are some things to consider. Users can do nearly all of this in the current Toolforge. Something new is ingresses and networkpolicies. The reason they can launch ingresses is to be able to launch services that are accessible to the outside, and networkpolicies are, I think, required for ingresses to work properly. That last part about networkpolicies may be worth testing first. Each namespace should have quotas applied so scaling is not something I fear. "poddisruptionbudgets" are an HA feature that isn't something I think we should restrict, per se either. (seehttps://kubernetes.io/docs/concepts/workloads/pods/disruptions/). Another consideration is that we may want to restrict deletecollection in some cases, particularly in configmaps where deleting all configmaps in their namespace will recycle their x509 certs and secrets where they might be able to revoke their own service account credentials inadvertently (rendering Deployments non-functional).

One important note: for this and the PSP for Toolforge users to work right, it must be applied to both the toolforge user and the $namespace:default service account, which is what a replicationcontroller runs as (therefore the thing launching pods in a Deployment object). This last piece hasn't been included in maintain_users.py yet, but it will be before launch.

Some other interesting information related to this topic:

• Certificates in Toolforge k8s