Posted onJul 3, 2022 • Edited onJul 6, 2022

Space boxing user accounts with Kiosk

#kubernetes #multitenancy #tips

What

Kiosk is a CRD based approach to isolate namespaces with:

wrapper over namespace called "spaces" which are isolated and resoure quota applied
wrapper over users (and service-accounts) called "accounts" who operate within these spaces
configuration templates like AccountQuota and Template to generalize and simplify the management of above

Architecture

Credit:https://github.com/loft-sh/kiosk

Who can do it

All of kiosk's setup are to be done by ClusterAdmin (or users with sufficient RBAC permissions)

Check you are admin

❯ kubectl auth can-i"*""*"--all-namespaceskubectl auth can-i"*" namespacekubectl auth can-i"*" clusterrolekubectl auth can-i"*" crdyesyesyesyes

Setting up kiosk and impersonating users

Install kiosk

# Install kiosk with helm v3❯ kubectl create namespace kioskhelminstallkiosk--repo https://charts.devspace.sh/ kiosk--namespace kiosk--atomicnamespace/kiosk createdNAME: kiosk...Learn more about using kiosk here: https://github.com/loft-sh/kiosk#getting-started#verify❯ kubectl get pod-n kioskNAME                     READY   STATUS    RESTARTS   AGEkiosk-66dbfcf6db-5rfx2   1/1     Running   0          2m18s

Create some accounts

❯cataccount.yamlapiVersion: tenancy.kiosk.sh/v1alpha1kind: Accountmetadata:  name: ambers-accountspec:  subjects:  - kind: User    name: amber    apiGroup: rbac.authorization.k8s.io---apiVersion: tenancy.kiosk.sh/v1alpha1kind: Accountmetadata:  name: blues-accountspec:  subjects:  - kind: User    name: blue    apiGroup: rbac.authorization.k8s.io---❯ kubectl apply-f account.yamlaccount.tenancy.kiosk.sh/ambers-account createdaccount.tenancy.kiosk.sh/blues-account created# Check RBAC❯ kubectl get account.tenancy.kiosk.shNAME             SPACES   CREATED ATambers-account   0        2022-07-03T01:16:41Zblues-account    0        2022-07-03T01:16:41Z❯ kubectl get accounts--as=amberNAME             SPACES   CREATED ATambers-account   0        2022-07-03T01:16:41Z

Note
In the real world, users come through external systems and are usually authenticated through systems like Dex (alternatively, if you are in a public cloud, you may be able to use provider-specific solutions such as AWS IAM for EKS or GCP IAM for GKE).
For service-accounts instead of actual users checkthis

Working with Spaces

Spaces are wrappers over namespaces. Users are allowed to use and operate on spaces that they are alloted to while others
are denied and invisible to them.

❯catspaces.yamlapiVersion: tenancy.kiosk.sh/v1alpha1kind: Spacemetadata:  name: ambers-spacespec:# spec.account can be omitted if the current user only belongs to a single account  account: ambers-account---apiVersion: tenancy.kiosk.sh/v1alpha1kind: Spacemetadata:  name: blues-spacespec:# spec.account can be omitted if the current user only belongs to a single account  account: blues-account---❯ k create-f spaces.yamlspace.tenancy.kiosk.sh/ambers-space createdspace.tenancy.kiosk.sh/blues-space created# Verify❯ k get spaces--as=amberNAME           OWNER            CREATED ATambers-space   ambers-account   2022-07-03T01:22:44Z

Create resources

❯ k create deployment test-dep-001--image=nginx--as=amber--namespace=ambers-spacedeployment.apps/test-dep-001 created❯ k create deployment test-dep-002--image=nginx--as=amber--namespace=blues-spaceerror: failed to create deployment: deployments.apps is forbidden: User"amber" cannot create resource"deployments"inAPI group"apps"inthe namespace"blues-space"# Verify❯ k get deployments.apps-n ambers-spaceNAME           READY   UP-TO-DATE   AVAILABLE   AGEtest-dep-001   0/1     1            0           28s❯ k get deployments.apps-n blues-spaceNo resources foundinblues-space namespace.

Delete resources

❯ k delete space blues-space--as=blueError from server(Forbidden): spaces.tenancy.kiosk.sh"blues-space" is forbidden: User"blue" cannot delete resource"spaces"inAPI group"tenancy.kiosk.sh" at the cluster scope❯ k delete space blues-spacespace.tenancy.kiosk.sh"blues-space" deleted

Allow account-users to create and delete spaces using

Templates

Templates in kiosk are used to initialize namespaces and apply common resources across namespaces (e.g. secrets).
When creating a Space, kiosk will use these Templates to populate the newly created Namespace for this Space.
Templates:

can contain one or more kubernetes manifests or a helm invocation
administered and managed by TemplateInstance (in the respective namespace)
can be parametrized for customizability

Create a manifest

sample

Associate with an account

sample

Instantiate the template

sample

[Way more information on templates](https://github.com/loft-sh/kiosk#53-using-templates)

Miscellaneous

Account defaults

❯cataccount-default-space-metadata.yamlapiVersion: tenancy.kiosk.sh/v1alpha1kind: Accountmetadata:  name: alpha-space-default-metadataspec:  space:    clusterRole: kiosk-space-admin    spaceTemplate:      metadata:        labels:          some-label:"label-value"          other-label:"other-value"        annotations:"foo":"bar""department":"alpha"  subjects:  - kind: User    name: adam    apiGroup: rbac.authorization.k8s.io  - kind: User    name: brian    apiGroup: rbac.authorization.k8s.io❯ k create-f account-default-space-metadata.yamlaccount.tenancy.kiosk.sh/alpha-space-default-metadata created❯ k describe account alpha-space-default-metadata ...Name: alpha-space-default-metadataNamespace: nullSpec:  Space:    Cluster Role: kiosk-space-admin    Space Template:      Metadata:        Annotations:          Department: alpha          Foo: bar        Creation Timestamp: <nil>        Labels:          Other - Label: other-value          Some - Label: label-value  Subjects:    API Group: rbac.authorization.k8s.io    Kind: User    Name: brianStatus: null