Posted onJun 17, 2020 • Edited onJul 14, 2021

Kubernetes Integration Testing

#kubernetes #integration #python #ruby

Summary

Infrastructure pipeline testing is essential for ensuring minimal regression, healthy systems, and faster mean time to recovery in patches. I have found it especially useful to perform integration tests to validate environments pre-deployment, to ensure they meet all the technical and compliance requirements.

Use Case

For the test, I have created a sample flask app, deployed on EKS, that will reach out to S3 and grab a file called "test". The flask-app manifest defines a readiness probe to ensure the /test endpoint (that grabs the file from s3) can fire successfully. Our inspec test suite will then ensure the pod deploys and starts. The pytest suite will validate k8s components like IAM authorization through our readiness probe, networking, and has the potential to expand to end to end tests in the future.

If deploying this code set, ensure to replace any <VAR> with an appropriate value for your environment.

Technologies

Inspec

Despite open source offerings, such asinspec-k8s, the backingsdk was written by a company who closed its doors in 2019. Depending on the backing runner for your cluster creation, you can also run into dependencies mismatch, creating a nonstarter execution, see dry typesPR. Ultimately, it is possible, but several workarounds are needed in getting started.

To fully utilize inspec, you will have to deploy your application configuration first, as well asinstall the Kubernetes train. Alternatively, running frombgeesaman's inspec-k8s-runner docker container gives you a nice starting place. Once installed, you can begin to test the various components within your namespace.

Below are some sample resources you can utilize to test against.

### Test your pods exists, and all are runningcontrol"k8s-app-validate"doimpact1.0title"Validate K8s test Application"desc"The k8s-app test app should exist and be running"### Test various namespaces existdescribek8sobject(api:'v1',type:'namespaces',name:'default')doit{shouldexist}end### Test for every pod in deployment the pod exists, and is runningk8sobjects(api:'v1',type:'pods',namespace:'default',labelSelector:'app=flask').items.eachdo|pod|describe"#{pod.namespace}/#{pod.name} pod"dosubject{k8sobject(api:'v1',type:'pods',namespace:pod.namespace,name:pod.name)}it{shouldexist}it{shouldbe_running}endend### Test your service existsdescribek8sobjects(api:'v1',type:'services',namespace:'default',labelSelector:'app=flask')doit{shouldexist}endend

Pytest

Vapor has a framework calledkubetest, which will deploy a configuration to your cluster in a unique namespace. This framework is meant for deploying and testing, and not necessarily testing existing infrastructure that exists. One of the advantages of this framework is it natively has readiness checks, whereas inspec did not.

Once deployed, you can utilize standard pytest execution to assert against those resources. Below walks through a few scenarios, which will use the configurations provided above.

There are some limitations in the API objects supported, seeAPI Resources.

fromboto3importclientfromtimeimporttime# Create And Test Service Accountdefcreate_sa(kube,modifier):"""    A helper function to create service account"""sa=kube.load_serviceaccount("configs/sa.yaml")account_id=client('sts').get_caller_identity()["Account"]role_arn=f"arn:aws:iam::{account_id}:role/test_role"sa.obj.metadata.annotations['eks.amazonaws.com/role-arn']=role_arnreturnsadeftest_create_sa(kube,modifier):"""    A function to test the creation of a service account    Goal: This will test the ability to interface with the k8s cli"""sa=create_sa(kube,modifier)kube.create(sa)assertsa.is_ready()

# Create and Test Deploymentdefcreate_deployment(kube,sa,modifier):"""    A helper function to create a deployment object"""account_id=client('sts').get_caller_identity()["Account"]deployment=kube.load_deployment('configs/deployment.yaml')repository=f"{account_id}.dkr.ecr.us-east-1.amazonaws.com/k8s-test"deployment.obj.spec.template.spec.containers[0].image=repositorydeployment.obj.spec.template.spec.service_account_name=sa.obj.metadata.namedeployment.obj.spec.template.spec.containers[0].env[1].value=f"quackenbush-test-bucket"returndeploymentdeftest_deployment(kube,modifier):"""    A function to test the creation of a deployment.    Goals: If a pod becomes ready, that means it can successful connect to SSM"""sa=create_sa(kube,modifier)kube.create(sa)deployment=create_deployment(kube,sa,modifier)kube.create(deployment)timeout=time()+60# 60 secondswhilenotdeployment.is_ready():iftime()>timeout:# Fail earlyassertdeployment.is_ready()pods=deployment.get_pods()forpodinpods:pod.wait_until_containers_start(timeout=60)timeout=time()+300# 5 Minuteswhilenotpod.is_ready():iftime()>timeout:breakassertpod.is_ready()

Go

Another popular framework that can be executed is terratest. This codebase could be extended to use the same workflows as before, where we have our container communicate with S3. However, for simplicity, I will demonstrate just spinning up a simple web based pod, and then tunneling traffic to verify health.

packagetestimport("crypto/tls""fmt""path/filepath""testing""time"http_helper"github.com/gruntwork-io/terratest/modules/http-helper""github.com/gruntwork-io/terratest/modules/k8s")funcTestKubernetesDeployment(t*testing.T){t.Run("test",func(t*testing.T){validateK8s(t)})}funcvalidateK8s(t*testing.T){kubeResourcePath,_:=filepath.Abs("./configs/")podName:="static-web"options:=k8s.NewKubectlOptions("","","default")k8s.KubectlApply(t,options,kubeResourcePath)deferk8s.KubectlDelete(t,options,kubeResourcePath)k8s.WaitUntilPodAvailable(t,options,podName,30,5*time.Second)// Create Tunnel and validate successtunnel:=k8s.NewTunnel(options,k8s.ResourceTypePod,podName,0,80)defertunnel.Close()tunnel.ForwardPort(t)tlsConfig:=tls.Config{}http_helper.HttpGetWithRetryWithCustomValidation(t,fmt.Sprintf("http://%s",tunnel.Endpoint()),&tlsConfig,10,5*time.Second,validateHealth,)}funcvalidateHealth(statusCodeint,bodystring)bool{ifstatusCode!=200{returnfalse}// Were going to ignore body, but we could do some test there tooreturntrue}

Top comments(1)

Alexey Melezhik

Sparrow - devops and automation tooling with Raku

Email
melezhik@gmail.com
Location
Woodlands, TX
Joined
Oct 8, 2017

• Aug 26 '21• Edited onAug 26• Edited

Copy link

Please add Sparrow (Written on awesome Raku language) to the collection, it has few plugins to validate k8s resources (sts, deployments, config maps and secrets ) -reddit.com/r/rakulang/comments/pc7...