Originally publishedon my blog

How to dynamically add nodes to a kind cluster

Kind allows you to run a Kubernetes cluster inside Docker. This is incredibly useful for developing Helm charts, Operators, or even just testing out different k8s features in a safe way.

I've recently been working on an operator (built using theoperator-sdk) that manages cluster node lifecycles. Kind allows you tospin up clusters with multiple nodes, using a Docker container per-node and joining them using a common Docker network. However, thekind executable does not allow you to modify an existing cluster by adding or removing a node.

I wanted to see if this was possible using a simple shell script, and it turns out that it's actually not too difficult!

Creating the node

Using my favorite diff tool,DiffMerge, anddocker inspect to compare an existing kind node's state to a new container's, I experimented with variousdocker run flags until I got something that's close enough to the kind node.

docker run\--restart on-failure\-v /lib/modules:/lib/modules:ro\--privileged\-h$NODE_NAME\-d\--network kind\--network-alias$NODE_NAME\--tmpfs /run\--tmpfs /tmp\--security-optseccomp=unconfined\--security-optapparmor=unconfined\--security-optlabel=disable\-v /var\--name$NODE_NAME\--label io.x-k8s.kind.cluster=kind\--label io.x-k8s.kind.role=worker\--env KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER\kindest/node:v1.25.2@sha256:9be91e9e9cdf116809841fc77ebdb8845443c4c72fe5218f3ae9eb57fdb4bace

Joining to the cluster

You can join new nodes to a k8s cluster by using thekubeadm join command. In this case, we can usedocker exec to execute this command on our node after its container has started up.

This command won't work out of the box because kind uses akubeadm.conf that does not exist in the node docker image. It is injected into the container by the kind executable.

Again, using my trusty DiffMerge tool, I compared two/kind/kubeadm.conf files in existing kind nodes and found very few differences. This allowed me to just grab one from any worker node to use as a template.

dockerexec--privileged kind-workercat /kind/kubeadm.conf>$LOCAL_KUBEADM

From here, I needed to set the node's unique IP in itskubeadm.conf. We can usedocker inspect to grab any node IP address we need. Since I'm working in bash, I just decided to use a simple sed replacement to replace the template node's IP address with my new node's IP in my local copy ofkubeadm.conf.

TEMPLATE_IP=$(docker inspect kind-worker | jq-r'.[0].NetworkSettings.Networks.kind.IPAddress')NODE_IP=$(docker inspect$NODE_NAME | jq-r'.[0].NetworkSettings.Networks.kind.IPAddress')ESCAPED_TEMPLATE_IP=$(echo$TEMPLATE_IP |sed's/\./\\./g')ESCAPED_NODE_IP=$(echo$NODE_IP |sed's/\./\\./g')sed-i.bkp"s/${ESCAPED_TEMPLATE_IP}/${ESCAPED_NODE_IP}/g"$LOCAL_KUBEADM

Now that ourkubeadm.conf is prepared, we need to copy it to the new node:

dockerexec--privileged-i$NODE_NAMEcp /dev/stdin /kind/kubeadm.conf <$LOCAL_KUBEADM

Finally, we can join our node to the cluster:

dockerexec--privileged$NODE_NAME kubeadmjoin--config /kind/kubeadm.conf--skip-phases=preflight--v=6

Node Tags

Since you have complete control of the new node'skubeadm.conf, it is possible to configure many of its properties for further testing. For example, to add additional labels to the new node, you can run something like this:

sed-i.bkp"s/node-labels:\"\"/node-labels:\"my-label-key=my-label-value\"/g"$LOCAL_KUBEADM

This will add themy-label-key=my-label-value label to the node once it joins the cluster.

Future Work

Based on this script, I believe it's possible to add akind create node subcommand to add a node to an existing cluster. Stay tuned for that...