Kubernetes is a complex system that relies on several components being configured correctly to have a working cluster.
Some solutions already exist to automate clusters' creation, upgrade, deletion, like for instanceClusterAPI. Such solutions require the existance of a management Kubernetes cluster from where tens of other Kubernetes clusters are then programmatically created and managed.
Managing the life-cycle of Kubernetes clusters is the first step. After a Kubernetes cluster is created, an handful number of features needs to be installed (CNI for instance).Kubernetes add-ons are used to extend the functionality of Kubernetes clusters.
So after cluster creation, the next problem that needs to be addressed is how to programmatically manage Kubernetes add-ons in each managed cluster from the management cluster. Luckily, even in this space, there are already existing solutions. For instance,Sveltos is an open source project to programmatically deploy Kubernetes add-ons in tens of Kubernetes clusters.
Summarizing, using a Kubernetes cluster to manage other clusters (creation/upgrade/deletion and add-ons deployment) provides several benefits, including:
- Centralized management: A cluster management cluster allows administrators to manage multiple clusters from a single place, making it easier to maintain consistency and reduce the risk of configuration issues.
- Scalability: A cluster management cluster can help organizations scale their infrastructure by making it easier to create, deploy, and manage multiple clusters.
- Better security: A cluster management cluster can be configured with security-related addons, such as network policies and secrets management, to help ensure that all managed clusters are secure.
- Increased automation: A cluster management cluster can be integrated with a continuous integration/continuous deployment (CI/CD) pipeline, making it easier to automate the deployment of new clusters and addons, and reducing the time and effort involved in managing the infrastructure.
Problem gets more complex when different organizations end up sharing the managed Kubernetes clusters. This scenario is commonly referred to asmulti-tenancy.
Common forms of multi-tenancy are:
- share a cluster between multiple teams within an organization, each of whom may operate one or more workloads;
- one (or more) cluster(s) fully reserved for an organization.
In this scenario we need to introduce two roles:
- platform admin: The role of a platform administrator is to manage and maintain the infrastructure of the managed Kubernetes clusters. This includes tasks such as setting up the cluster, managing nodes, installing and updating software components, and ensuring the overall health and security of the cluster. The platform administrator is also responsible for onboarding tenant admins and granting permissions to those;
- tenant admin: Has admin access to the clusters/namespaces assigned to them by the platform admin and manages tenant applications in those clusters/namespaces.
After cluster creation/upgrade/deletion and the addons deployment, the next problem to solve is how to allow platform admin to onboard tenants and programmatically grant them permissions from the management cluster.
Sveltos addons management solution
Before we deep dive into granting permissions to tenants, lets briefly summarizeSveltos solution to manage Kubernetes addons.
Sveltos introduces a single CRD calledClusterProfile for that.
By creating a ClusterProfile instance, we can tell Sveltos:
- where to deploy addons (i.e, in all managed Clusters matching theClusterSelector field);
- list of addons to deploy (either Helm charts or Kubernetes resource YAMLs contained in referenced ConfigMaps/Secrets).
apiVersion:config.projectsveltos.io/v1alpha1kind:ClusterProfilemetadata:name:deploy-kyvernospec:clusterSelector:env=fvsyncMode:ContinuoushelmCharts:-repositoryURL:https://kyverno.github.io/kyverno/repositoryName:kyvernochartName:kyverno/kyvernochartVersion:v2.6.0releaseName:kyverno-latestreleaseNamespace:kyvernohelmChartAction:InstallpolicyRefs:-name:disallow-latest-tag# referenced ConfigMap contains a Kyverno ClusterPolicynamespace:defaultkind:ConfigMap
WithSveltos configuration drift detection we don't have to worry about managed cluster states. Sveltos monitors the state of managed clusters and when it detects a configuration drift for one of the resource deployed because of a ClusterProfile, it re-syncs the cluster state back to the state described in the management cluster.
Sveltos multi-tenancy solution
Let's now go back to the multi-tenancy problem.
We want to:
- allow platform admin to programmatically grant permissions to tenant admins;
- allow tenant admins to manage applications from management cluster, making sure each tenant admin only deploys resources it has permissions to and only in the clusters/namespaces it has permission to.
RoleRequest is the CRD introduced by Sveltos to allow platform admin to grant permissions to various tenant admins.
apiVersion:lib.projectsveltos.io/v1alpha1kind:RoleRequestmetadata:name:full-accessspec:clusterSelector:env=pre-productionadmin:engroleRefs:-name:full-accessnamespace:defaultkind:ConfigMapwhere:
- ClusterSelector field is used to select a set of managed clusters where permissions will be granted;
- Admin field selects a tenant admin (later on we will cover how this identify a tenant admin);
- RoleRefs field references one of more ConfigMaps/Secrets. Each of those ConfigMaps/Secrets contains one or moreClusterRole/Role defining the permissions granted to the tenant admin by the platform admin.
Platform admin is the only admin in the management cluster with permissions to create/update/delete RoleRequest instances.
Tenant admin managing applications
Once RoleRequest instances have been created, tenant admins can use SveltosClusterProfile CRD to instruct Sveltos on what Kubernetes resources to deploy on which clusters.
Sveltos requires that following labelprojectsveltos.io/admin-name: <admin>
is set on a ClusterProfile instances created by a tenant admin.
How to achieve that, is up to platform admin.
For instance, if:
- Kyverno is present in the management cluster;
- tenant admin are represented by KubernetesServiceAccount;
following Kyverno ClusterPolicy is all that is needed for label to be automatically added to each instance of a ClusterProfile:
apiVersion:kyverno.io/v1kind:ClusterPolicymetadata:name:add-labelsannotations:policies.kyverno.io/title:Add Labelspolicies.kyverno.io/description:>-Adds projectsveltos.io/admin-name label on each ClusterProfilecreated by tenant admin. It assumes each tenant admin isrepresented in the management cluster by a ServiceAccount.spec:validationFailureAction:enforcebackground:falserules:-name:add-labelsmatch:resources:kinds:-ClusterProfilemutate:patchStrategicMerge:metadata:labels:projectsveltos.io/admin-name:"{{serviceAccountName}}"Examples
I have created management cluster on my laptop. It is aKind cluster.
In the management cluster, I have deployed both ClusterAPI (with Kind as infrastructure provider) and Sveltos.
kubectl get pods -ANAMESPACE NAME READY STATUS RESTARTS AGEcapd-system capd-controller-manager-59b7bdbcf9-zd4xq 1/1 Running 0 30mcapi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-68f975d954-9r7lm 1/1 Running 0 30mcapi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-8476fb8676-kpnkd 1/1 Running 0 30mcapi-system capi-controller-manager-69fd995599-67fh5 1/1 Running 0 30m...projectsveltos access-manager-569dd6cccd-m676t 2/2 Running 0 59sprojectsveltos classifier-manager-ddc8c756-bmxc9 2/2 Running 0 59sprojectsveltos fm-controller-manager-5964fb8bff-2j2gx 2/2 Running 0 59sprojectsveltos sveltoscluster-manager-75ddcf7cbd-q47hp 2/2 Running 0 58sI have one ClusterAPI powered cluster:
kubectl get clusters -ANAMESPACE NAME PHASE AGE VERSIONdefault sveltos-management-workload Provisioned 10m v1.25.2and I have one GKE clusterregistered with Sveltos
kubectl get sveltoscluster -ANAMESPACE NAME AGEgke production 6sCluster labels are set as follow:
| Cluster | Label |
|---|---|
| default/sveltos-management-workload | env:development |
| gke/production | env:production |
Fully reserve set of cluster to a tenant
apiVersion:v1kind:ConfigMapmetadata:name:full-accessnamespace:defaultdata:role.yaml:|apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:name: system-admin-full-accessrules:- apiGroups: ["*"]resources: ["*"]verbs: ["*"]---apiVersion:lib.projectsveltos.io/v1alpha1kind:RoleRequestmetadata:name:full-accessspec:clusterSelector:env=productionadmin:system-adminroleRefs:-name:full-accessnamespace:defaultkind:ConfigMapBy referencing ConfigMapdefault/full-access, above RoleRequest is reserving any cluster matching clusterSelectorenv=production to adminsystem-admin.
Usingsveltosctl we can verify Sveltos is aware system-admin has full access to managed cluster with label env:production
./bin/sveltosctl show admin-rbac +-------------------------------+--------------+-----------+------------+-----------+----------------+-------+| CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS |+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+| SveltosCluster:gke/production | system-admin |* |* |* |* |* |+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+If system-admin tries to deploy addons to such cluster, that operation will go through.
For instance, if system-admin posts following ClusterProfile
apiVersion:config.projectsveltos.io/v1alpha1kind:ClusterProfilemetadata:name:deploy-kyvernolabels:projectsveltos.io/admin-name:system-adminspec:clusterSelector:env=productionsyncMode:ContinuoushelmCharts:-repositoryURL:https://kyverno.github.io/kyverno/repositoryName:kyvernochartName:kyverno/kyvernochartVersion:v2.6.0releaseName:kyverno-latestreleaseNamespace:kyvernohelmChartAction:Installwe can verify Kyverno is deployed by Sveltos on behalf of system-admin:
./bin/sveltosctl show features +----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+| gke/production | helm chart | kyverno | kyverno-latest | 2.6.0 | 2023-02-03 14:06:48-0800 PST | deploy-kyverno |+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+If system-admin tries to deploy same add-ons on clusters matching clusterSelectorenv:development that operation will fail.
Share cluster between tenants
Platform admin here is granting following permissions in all clusters matching ClusterSelectorenv=development
- Admineng full access to namespace eng and some read permission in namespace shared-service-access;
- Adminhr full access to namespace human-resource.
# ConfigMap contains a Role which gives# service read access to namespace shared-servicesapiVersion:v1kind:ConfigMapmetadata:name:shared-service-accessnamespace:defaultdata:role.yaml:|apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata:name: service-read-rolenamespace: shared-servicesrules:- apiGroups: [""]resources: ["services"]verbs: ["get,list"]---# ConfigMap contains a Role which gives# full access to namespace engapiVersion:v1kind:ConfigMapmetadata:name:eng-full-accessnamespace:defaultdata:role.yaml:|apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata:name: edit-rolenamespace: engrules:- apiGroups: ["*"]resources: ["*"]verbs: ["*"]---# RoleRequest gives admin 'eng':# - full access to namespace eng# - some read permission in namespace shared-service-access# in all clusters matching the label# selector env=developmentapiVersion:lib.projectsveltos.io/v1alpha1kind:RoleRequestmetadata:name:eng-accessspec:clusterSelector:env=developmentadmin:engroleRefs:-name:eng-full-accessnamespace:defaultkind:ConfigMap-name:shared-service-accessnamespace:defaultkind:ConfigMap# ConfigMap contains a Role which gives# full access to namespace human-resourceapiVersion:v1kind:ConfigMapmetadata:name:hr-full-accessnamespace:defaultdata:role.yaml:|apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata:name: edit-rolenamespace: human-resourcerules:- apiGroups: ["*"]resources: ["*"]verbs: ["*"]---# RoleRequest gives admin 'hr' access to namespace# 'human-resource' in all clusters matching the label# selector env=developmentapiVersion:lib.projectsveltos.io/v1alpha1kind:RoleRequestmetadata:name:hr-accessspec:clusterSelector:env=developmentadmin:hrroleRefs:-name:hr-full-accessnamespace:defaultkind:ConfigMapAs before, we can use sveltosctl to verify granted permission:
./bin/sveltosctl show admin-rbac--namespace=default+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+| CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS |+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+| Cluster:default/sveltos-management-workload | eng | eng |* |* |* |* || Cluster:default/sveltos-management-workload | eng | shared-services | | services | services | get,list || Cluster:default/sveltos-management-workload | hr | human-resource |* |* |* |* |+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+If eng admin tries to deploy nginx in eng namespace, that will succeed.
apiVersion:v1kind:ConfigMapmetadata:name:nginxnamespace:defaultdata:nginx.yaml:|apiVersion: apps/v1kind: Deploymentmetadata:name: nginx-deploymentnamespace: englabels:app: nginxspec:replicas: 3selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: nginx:1.14.2ports:- containerPort: 80---apiVersion:config.projectsveltos.io/v1alpha1kind:ClusterProfilemetadata:name:deploy-nginxlabels:projectsveltos.io/admin-name:engspec:clusterSelector:env=developmentsyncMode:ContinuouspolicyRefs:-name:nginxnamespace:defaultkind:ConfigMapnginx is deployed
./bin/sveltosctl show features +-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+| default/sveltos-management-workload | apps:Deployment | eng | nginx-deployment | N/A | 2023-02-03 14:40:19-0800 PST | deploy-nginx |+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
Contribute
Sveltos is an open source project. Contributions are more welcome 🤗
If you would like to know more about it or would like to see a new feature added to sveltos, please reach to us onslack. Any feedback is very much appreciated ❤️
Documentation can be foundhere
Top comments(1)
- EducationPhD
- Workhttps://github.com/projectsveltos
- Joined
Sveltos is an open source project.
Contributions are more welcome 🤗
If you like golang and Kubernetes and would like to help, please contact us. We would love to have you part of the sveltos project.
Chat with us on the Slack in the#projectsveltos channel Slack
For further actions, you may consider blocking this person and/orreporting abuse
