Repository files navigation

This repo demonstrates how k8s deployment can be safe or unsafe depending onhow thekubectl apply command is used.

To jump into the demo as smoothly as possible, it's recommended that thefollowing are installed beforehand:

1. k8s itself (e.g. if using Docker Desktop, the built-in Kubernetes is enough)
2. yq - a library to parse yaml (.yml) files
3. ingress-nginx for k8s or an equivalence approach to help reach (e.g.curl) the k8s service

# run the initial deployment script./deploy.sh# expose the service via an ingress;# @NOTE this step is only needed if# `ingress-nginx` was used to access# the k8s service; if you use something# else, please expose your service in# accordance to your own setupkubectl apply -f ingress.yml

curl -s -I localhost/bazHTTP/1.1 200 OK

Assuming we now want to add a label calledversionto the k8s resources, and we've prepared themanifest filedeploy-version.yml, and now allthat's left is to run the deployment script. Let'ssee how it works out as we do it the UNSAFE and SAFEways.

# this basically calls `kubectl apply -f deploy-version.yml` in an unsafe way./deploy-version-unsafe.shnamespace/baz-service unchangedservice/baz-service configuredThe Deployment "baz-deployment" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"baz-service", "version":"2.0"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

What happened here was that: k8s reconfigured thebaz-service (to add the labelversion="2.0")but it then failed to reconfigure thebaz-deployment, due to thefield is immutable error.Why k8s behaves this way is beyond the scope of this demo, here we only care about the consequenceof this behavior.

If we curl again, we no longer get the successful 200 OK response:

curl -s -I localhost/bazHTTP/1.1 503 Service Temporarily Unavailable

This proves that our previous deployment script wasdestructive i.e. UNSAFE.

# run the initial deployment script again./deploy.sh# side-note: we don't need to deploy the `Ingress` resource again as it's not# part of `deploy-version.yml`, and thus was not "corrupted" by the previous# unsafe deployment script

Ensure everything works again

curl -s -I localhost/bazHTTP/1.1 200 OK

./deploy-version-safe.shnamespace/baz-service configured (dry run)service/baz-service configured (dry run)deployment.apps/baz-deployment configured (dry run)namespace/baz-service unchanged (server dry run)service/baz-service configured (server dry run)The Deployment "baz-deployment" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"baz-service", "version":"2.0"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

As you can see, the deployment still failed. The idea of this demo is not to fix this k8s error,but to fail safely. This time, the--dry-run argument did its job well: the script "failed early",so it exited before it had the chance to corrupt the running system.

Let's curl again to make sure things are still running well:

curl -s -I localhost/bazHTTP/1.1 200 OK

This demo reproduced a scenario where applying a collection of k8s manifest (.yml file)without any special validation mechanism may lead to service disruption (system downtime).

And one "quick win" is to make use of the--dry-run argument. It is literally free and effortlessto do so, while the peace of mind it brings is hugely valuable. No service disruption (financial loss)can ever justify the lack of properly applying an officially supported failsafe mechanism.