If you develop APIs or microservices, especially in medium to large environments, you probably use Kubernetes.
Kubernetes is a project created by Google in mid-2015 that quickly became the standard for managing container execution. You can host it on your machines or use a solution delivered by one of the big cloud players likeAWS,Google, andDigitalOcean.
In this post, I want to talk about another functionality: the possibility of extending it to create new capabilities. Let's start with the essential concepts for understanding this article.
Resources and Controllers
One of the most fundamental concepts is that K8s manage resources. According to officialdocumentation,
A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a specific type; for example, the built-in "pods" resource contains a collection of Pod objects.
K8s manages these resources using another concept: controllers. When we use a k8s feature, we need to define, in a yaml file, what state we expect. For example:
apiVersion:apps/v1kind:Deploymentmetadata:name:nginx-deploymentspec:selector:matchLabels:app:nginxreplicas:2# tells deployment to run 2 pods that match the templatetemplate:metadata:labels:app:nginxspec:containers:-name:nginximage:nginx:1.14.2ports:-containerPort:80The information inside thespec key corresponds to the desired state of the resource.
What k8s does is ensure that the current state of the object contained in the cluster is equal to the desired one that was declared. In this case, two Nginx containers, version 1.14.2, are running on port 80. It does this using what is called acontrol loop:
It checks whether the current state of the resource differs from the desired state, and if so, it executes theReconcile function of the controller linked to the object. This way, we can define a controller like this:
A controller tracks at least one type of Kubernetes resource. These objects have a spec field that represents the desired state. This resource's controller(s) are responsible for bringing the current state closer to that desired state.
K8s has a series of built-in resources such asPod,Deployment,Service, and controllers that track the lifecycle of each of them. But in addition to them, we can create our resources throughCustom Resource Definitions (CRD). The combination of a CRD and a controller is what we call anoperator, and it is what we will explore in this text.
operator-sdk
To illustrate what we can do with an operator, I will create a proof of concept using operator-sdk. According to theofficial website::
The Operator SDK makes it easy to build Kubernetes-native applications, a process that can require deep, application-specific operational knowledge. This project is a component of theOperator Framework, an open-source toolkit for managing native Kubernetes applications called Operators in a practical, automated, and scalable way.
Creating an operator usingGo,Ansible orHelm is possible. In this article, I will use Go.
The first step is to install the SDK CLI on the machine. I used brew, but the other options are in thedocumentation.
brewinstalloperator-sdkThe next step is to use the CLI to generate the project scaffolding using the commands:
operator-sdk init--domain minetto.dev--repo github.com/eminetto/k8s-operator-talkoperator-sdk create api--version v1alpha1--kind Application--resource--controllerThe first command initializes the project by indicating the domain, information that k8s will use to identify the resource and the repository name used for the Go package name. The second command creates a newApplication resource in thealpha1 version and a controller skeleton.
Before we get into the code, it's essential to understand the purpose of the proof of concept. In its native form, getting an application running on K8s requires the developer to understand concepts such as Deployment, Pod, Service, etc. My goal is to reduce this cognitive load to just two resources: anamespace, where the Application will reside within the cluster, and anApplication, which will define the desired state of an application. For example, the team only needs to create the followingyaml :
apiVersion:v1kind:Namespacemetadata:name:application-sample---apiVersion:minetto.dev/v1alpha1kind:Applicationmetadata:name:application-samplenamespace:application-samplespec:image:nginx:latestreplicas:2port:80Apply it to the cluster using the command:
kubectl apply-f application.yamlAnd the rest will be created by our controller.
The first step is configuring our resource to have fields related tospec. To do this, you must change theapi/v1alpha/application_types.go file and add the fields to the struct:
typeApplicationSpecstruct{Imagestring`json:"image,omitempty"`Replicasint32`json:"replicas,omitempty"`Portint32`json:"port,omitempty"`}Later, we will use this information to generate the files necessary to install the CRD on our cluster. We will also use this structure to create the required resources.
The next step is to create the logic for our controller.operator-sdk made the controllers/application_controller.go file and the Reconcile function signature. This function is called by the control loop each time k8s detects a difference between the current state of the object and the desired state. In themain.go file that the SDK generated, we have the link between the Application resource and our controller, and we don't need to worry about it now. One of the advantages of operator-sdk is that it allows us to focus on the controller logic and abstracts all the massive details necessary for it to work.
TheReconcile function code and auxiliaries are below. I tried to document the most important excerpts:
func(r*ApplicationReconciler)Reconcile(ctxcontext.Context,reqctrl.Request)(ctrl.Result,error){l:=log.FromContext(ctx)varappminettodevv1alpha1.Application//recupera os detalhes do objeto sendo gerenciadoiferr:=r.Get(ctx,req.NamespacedName,&app);err!=nil{ifapierrors.IsNotFound(err){returnctrl.Result{},nil}l.Error(err,"unable to fetch Application")returnctrl.Result{},err}/* The finalizer is essential because it tells K8s we need control over object deletion. After all, how we will create other resources must be excluded together. Without the finalizer, there is no time for the K8s garbage collector to delete, and we risk having useless resources in the cluster. */if!controllerutil.ContainsFinalizer(&app,finalizer){l.Info("Adding Finalizer")controllerutil.AddFinalizer(&app,finalizer)returnctrl.Result{},r.Update(ctx,&app)}if!app.DeletionTimestamp.IsZero(){l.Info("Application is being deleted")returnr.reconcileDelete(ctx,&app)}l.Info("Application is being created")returnr.reconcileCreate(ctx,&app)}func(r*ApplicationReconciler)reconcileCreate(ctxcontext.Context,app*minettodevv1alpha1.Application)(ctrl.Result,error){l:=log.FromContext(ctx)l.Info("Creating deployment")err:=r.createOrUpdateDeployment(ctx,app)iferr!=nil{returnctrl.Result{},err}l.Info("Creating service")err=r.createService(ctx,app)iferr!=nil{returnctrl.Result{},err}returnctrl.Result{},nil}func(r*ApplicationReconciler)createOrUpdateDeployment(ctxcontext.Context,app*minettodevv1alpha1.Application)error{vardeplappsv1.DeploymentdeplName:=types.NamespacedName{Name:app.ObjectMeta.Name+"-deployment",Namespace:app.ObjectMeta.Name}iferr:=r.Get(ctx,deplName,&depl);err!=nil{if!apierrors.IsNotFound(err){returnfmt.Errorf("unable to fetch Deployment: %v",err)}/*If there is no Deployment, we will create it. An essential section in the definition is OwnerReferences, as it indicates to k8s that an Application is creating this resource. This is how k8s knows that when we remove an Application, it must also remove all the resources it created. Another important detail is that we use data from our Application to create the Deployment, such as image information, port, and replicas. */ifapierrors.IsNotFound(err){depl=appsv1.Deployment{ObjectMeta:metav1.ObjectMeta{Name:app.ObjectMeta.Name+"-deployment",Namespace:app.ObjectMeta.Name,Labels:map[string]string{"label":app.ObjectMeta.Name,"app":app.ObjectMeta.Name},Annotations:map[string]string{"imageregistry":"https://hub.docker.com/"},OwnerReferences:[]metav1.OwnerReference{{APIVersion:app.APIVersion,Kind:app.Kind,Name:app.Name,UID:app.UID,},},},Spec:appsv1.DeploymentSpec{Replicas:&app.Spec.Replicas,Selector:&metav1.LabelSelector{MatchLabels:map[string]string{"label":app.ObjectMeta.Name},},Template:v1.PodTemplateSpec{ObjectMeta:metav1.ObjectMeta{Labels:map[string]string{"label":app.ObjectMeta.Name,"app":app.ObjectMeta.Name},},Spec:v1.PodSpec{Containers:[]v1.Container{{Name:app.ObjectMeta.Name+"-container",Image:app.Spec.Image,Ports:[]v1.ContainerPort{{ContainerPort:app.Spec.Port,},},},},},},},}err=r.Create(ctx,&depl)iferr!=nil{returnfmt.Errorf("unable to create Deployment: %v",err)}returnnil}}/*The controller also needs to manage the update because if the dev changes any information in an existing Application, this must impact other resources.*/depl.Spec.Replicas=&app.Spec.Replicasdepl.Spec.Template.Spec.Containers[0].Image=app.Spec.Imagedepl.Spec.Template.Spec.Containers[0].Ports[0].ContainerPort=app.Spec.Porterr:=r.Update(ctx,&depl)iferr!=nil{returnfmt.Errorf("unable to update Deployment: %v",err)}returnnil}func(r*ApplicationReconciler)createService(ctxcontext.Context,app*minettodevv1alpha1.Application)error{srv:=v1.Service{ObjectMeta:metav1.ObjectMeta{Name:app.ObjectMeta.Name+"-service",Namespace:app.ObjectMeta.Name,Labels:map[string]string{"app":app.ObjectMeta.Name},OwnerReferences:[]metav1.OwnerReference{{APIVersion:app.APIVersion,Kind:app.Kind,Name:app.Name,UID:app.UID,},},},Spec:v1.ServiceSpec{Type:v1.ServiceTypeNodePort,ExternalTrafficPolicy:v1.ServiceExternalTrafficPolicyTypeLocal,Selector:map[string]string{"app":app.ObjectMeta.Name},Ports:[]v1.ServicePort{{Name:"http",Port:app.Spec.Port,Protocol:v1.ProtocolTCP,TargetPort:intstr.FromInt(int(app.Spec.Port)),},},},Status:v1.ServiceStatus{},}_,err:=controllerutil.CreateOrUpdate(ctx,r.Client,&srv,func()error{returnnil})iferr!=nil{returnfmt.Errorf("unable to create Service: %v",err)}returnnil}func(r*ApplicationReconciler)reconcileDelete(ctxcontext.Context,app*minettodevv1alpha1.Application)(ctrl.Result,error){l:=log.FromContext(ctx)l.Info("removing application")controllerutil.RemoveFinalizer(app,finalizer)err:=r.Update(ctx,app)iferr!=nil{returnctrl.Result{},fmt.Errorf("Error removing finalizer %v",err)}returnctrl.Result{},nil}To deploy our customized resource and its controller, the SDK provides commands in itsMakefile :
make manifestsmake docker-build docker-pushIMG=registry.hub.docker.com/eminetto/k8s-operator-talk:latestmake deployIMG=registry.hub.docker.com/eminetto/k8s-operator-talk:latestThe first command generates all the files necessary to create the CRD. The second generates a docker container and pushes it to the indicated repository. The last command installs the generated container on the cluster. Tip: You can automate controller generation and installation in your development environment using Tilt. This project'srepository has a Tiltfile that does all this work. To learn more about Tilt, check outmy post about the tool.
Now, apply theyaml with theApplication definition to the cluster, and the controller will generate theDeployment andService necessary for the Application to run.
We can check that the controller created the resources with the following commands.
kubectl-n application-sample get applicationsNAME AGEapplication-sample 18skubectl-n application-sample get deploymentsNAME READY UP-TO-DATE AVAILABLE AGEapplication-sample-deployment 2/2 2 2 41skubectl-n application-sample get podsNAME READY STATUS RESTARTS AGEapplication-sample-deployment-65b96554f8-8vv64 1/1 Running 0 56sapplication-sample-deployment-65b96554f8-v54gp 1/1 Running 0 56skubectl-n application-sample get servicesNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEapplication-sample-service NodePort 10.43.63.164 <none> 80:32591/TCP 66sThis post ended up being quite long, so there are other topics that I will leave for a future text, such as the testing part. But I hope I was able to spark interest in this subject. I'm very excited about it and believe it has incredible potential to help create automation that makes the lives of development and operations teams much more effortless.
Originally published athttps://eltonminetto.dev on September 8, 2023
Top comments(1)
For further actions, you may consider blocking this person and/orreporting abuse
