Repository files navigation

Cisco k8s operator and controller implementation for Application WAN Interface (AWI)

The kube-awi allows using k8s custom resources to interact with AWI project.

With kube-awi installed on the k8s cluster, the following actions can be done:

• creating requests to awi withkubectl apply
• getting information about instances, network domains etc. withkubectl get

Installation of kube-awi on the k8s cluster involves creating Custom ResourceDefinitions, namely:

• instances.awi.app-net-interface.io
• internetworkdomainappconnections.awi.app-net-interface.io
• internetworkdomainconnections.awi.app-net-interface.io
• networkdomains.awi.app-net-interface.io
• sites.awi.app-net-interface.io
• subnets.awi.app-net-interface.io
• vpcs.awi.app-net-interface.io
• vpns.awi.app-net-interface.io

They can be grouped into two use cases

1. Interacting with the awi

   Resourcesinternetworkdomainconnections andinternetworkdomainappconnections area way of creating requests to AWI system.

   For instance, applying the followinginternetworkdomainconnection.yaml manifest

   apiVersion:awi.app-net-interface.io/v1alpha1kind:InterNetworkDomainConnectionmetadata:name:my-internetworkdomainconnectionspec:metadata:name:example-namespec:destination:metadata:name:machine-learning-trainingdescription:"Description of the destination"networkDomain:selector:matchId:id:vpc-097e8ed349c13c004source:metadata:name:machine-learning-datasetdescription:"Description of the source"networkDomain:selector:matchId:id:vpc-04a1eaad3aa81310f

   will create a request to create Network Domain Connectionbetween Source VPCvpc-097e8ed349c13c004 and DestinationVPC vpc-04a1eaad3aa81310f.

2. Checking existing VPCs, Instances, Subnets etc.

   Resourcesinstance,network_domain,site,subnet,vpcandvpn represent resources that can be inspected using eitherAWI UI or AWI CLI.

   To check the list of instances, simply run

   // Get all instanceskubectl get instance -A

   and the kubectl will return a list of obtained instances fromsupported providers (AWS and GCP) just like getting list ofpods,deployments etc.

   To get details of a certain instance run

   kubectl get instance INSTANCE_ID -n awi-system -o yaml

   to see the details of the resource.

Both use cases described above are available thanks to the k8soperator.

Installation of the kube-awi on the k8s creates a special deploymentcalledkube-awi-controller-manager (kube-awi operator in the graph) which acts as a special processthat will:

• watch for updates ofinternetworkdomainconnections andinternetworkdomainappconnections custom resources and triggersactions defined incontrollers/RESOUCE_controller.go

• synchronizes other resources by periodically obtaining lists ofsubnets, instances etc. fromawi-grpc-catalyst-sdwan andcreating custom resources inside the cluster

The Kube-AWI operator consists of so called controllers, that definemethods to be triggered for certain events, syncer which is a simplegoroutine and awi client which implements necessary interfaces andspecifies an address of the actual server from which the informationwill be received and where connection requests will be forwarded.

The Kube-AWI operator also includes standard k8s operator managerresponsible for health checks and other useful resources.

Watching for updates and triggering certain actions is accomplishedwith so called controllers. To generate a controller go toAdding new object.

A Controller specifiesReconcile method will is triggered wheneverthere is an update of the certain Custom Resource. This method willbe called whenever a resource is created/updated/deleted.

Currently, kube-awi defines both resources in the following manner:

• removing Custom Resource usingkubectl delete triggers deletionof VPC Connection or App Connection

• other events trigger Connection Creation attempt.

Custom Resources specify two important sections:

1. Spec - the desired configuration for a resource

   Spec section is a user's input space. It accepts information aboutdesired settings and the Reconciler's job is to attempt accomplishingthem.

2. Status - the actual state of the object

   Status is a read-only information for the user about actual state ofthe resource. While state specifies the desired state, which may notbe accomplished yet, impossible to accomplish or be a high-leveldefinition of certain settings, the status section is updated by thereconciler and it is supposed to provide information about the presentstate and underlying low-level information that may be necessary forthe user.

Currently, we do not make much use of the status field, but it could beused for storing information about whether connection succeeded or not,what was the error etc.

Kube-awi operator runs a syncing goroutine which periodically callsawi-grpc-catalyst-sdwan to obtain resources from the AWI. Later, itcreates or updates Custom Resources associated with these resources.

Since these are read-only resources, they have no Controllers assignedto them, as the operator does not care about user's changes there.

Since these resources are updated by the periodic sync operation, theyare eventually consistent.

The kube-awi uses kubebuilder framework for automatic creation of:

• Custom Resource Definitions
• Operator's code for UPDATE actions

The input for both is a go fileapi/GROUP/VERSION/NAME_types.go.

It specifies the structure of the resource and relevant fields.The resouce can be marked with additional kubebuilder options tocustomize the structure and its interactions (for example additionalcolumns present when runningkubectl get instances -A)

To create a new kind such asinstance ornetwork_domain, run kubebuildercommand to initiate a new object

kubebuilder create api --group awi --version v1alpha1 --kind <NewObjectName>

This will generate a new file in the directoryapi/awi/v1alpha1/KIND_types.gowith placeholder golang structures for both structure itself and a listwrapper.

The CLI will ask if it should generate a controller for the resource. Replyingyes will add a note in thePROJECT filecontroller: true. This can bechanged manually later on.

To generate CRDs and operator code follow steps below.

To generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects run

make manifests

To generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations:

make generate

Make sure to use up-to-date version of awi-grpc repository.

Run:

• make install to install CRDs into the K8s cluster specified in ~/.kube/config.

Run:

• make docker-build to build controller image,
• (if you use kind cluster) change image pull policy forapp-net-interface.io/awi-controller:<TAG> image toimagePullPolicy: IfNotPresent to avoid pulling issues (TODO: automate this), required one time only,
• (if you use kind cluster)CLUSTER_NAME=<your-cluster-name> make kind-load,
• (if you use remote cluster)make docker-push,
• make deploy to update image in cluster controller deployment.

Currently, the kube-awi project gathers the entire logic in themain.go file which is an entry point for the k8s operator. Thisfile instantiates k8s operator manager, initializes kube awi client,registers existing reconcilers and runs syncing goroutine.

To make kube-awi more opened for other possible controllers, themain file requires some design decisions over how differentcontrollers should be implemented.

1. The kube-awi embeds all 3 GRPC interfaces (connection, app connection and cloud clients)into single client with a configurable address. It means that current implementationprevents user from specifying a different address for cloud interface responsible forobtaining information about existing subnets etc. and for connection/app connectionclients.

   It means that a new controller need to implement the entire logic - if we want towrite a new controller which defines new logic for connections or app connectionsbut we want to remain cloud connectivity from old controller, we would have tomake our new controller forward cloud requests to the old one.

2. Connection and App Connection interfaces are explicitely loaded in themain.gofile. If a new controller requires a different reconciling action, a new providershould be specified. Considering a scenario where all controller versions arespecified inside kube-awi repository, themain.go file should be changed todynamically load desired controllers based on the provided configuration.

3. Syncer goroutine is quite specific to awi project and the user may wish to usedifferent implementation, use different CRDs for that or to not use syncer atall. This topic leads to a further design discussion around making kube-awia library.

The project graph above shows the existing dependencies and potential points ofproviding an abstraction over pieces of code that can be turned into customizablemodules.

Here is the instruction how to test Kube AWI with locally createdcluster with Minikube.

Create minikube cluster

minikube start

Createawi-system namespace.

kubectl create ns awi-system

To avoid pushing images to registries, simply set docker registrycontext to use minikube's one:

eval $(minikube docker-env)

Now you can build images directly to minikube docker registry:

docker build --build-arg SSH_PRIVATE_KEY="$(cat PRIVATE_SSH_PATH)" -t IMAGE_NAME:IMAGE_TAG .

Replace PRIVATE_SSH_PATH, IMAGE_NAME and IMAGE_TAG with your values.

The private SSH key is necessary only for internal github access.After opensourcing the project, the Dockerfile will be adjusted properlyand no private key will be longer needed.The Dockerfile uses 2 stages for creating the image, the final stagedoesn't use the private SSH key so there is no need to worry aboutexposing your private SSH key.

Now generate CRDs and deploy Kube AWI:

IMG=IMAGE_NAME:IMAGE_TAG make installIMG=IMAGE_NAME:IMAGE_TAG make deploy

Again, replace IMAGE_NAME and IMAGE_TAG with your own values.

This should create CRDs and Manager pod in the cluster:

> kubectl get crds -A---NAME                                             CREATED ATinstances.awi.app-net-interface.io                          2024-02-09T04:09:55Zinternetworkdomainappconnections.awi.app-net-interface.io   2024-02-09T04:09:55Zinternetworkdomainconnections.awi.app-net-interface.io      2024-02-09T04:09:55Znetworkdomains.awi.app-net-interface.io                     2024-02-09T04:09:55Zsites.awi.app-net-interface.io                              2024-02-09T04:09:55Zsubnets.awi.app-net-interface.io                            2024-02-09T04:09:55Zvpcs.awi.app-net-interface.io                               2024-02-09T04:09:55Zvpns.awi.app-net-interface.io                               2024-02-09T04:09:55Z

> kubectl get pods -A---NAMESPACE     NAME                                          READY   STATUS    RESTARTS      AGEawi-system    kube-awi-controller-manager-9d4697db6-h9qmn   2/2     Running   0             24mkube-system   coredns-5dd5756b68-kz7zq                      1/1     Running   0             20hkube-system   etcd-minikube                                 1/1     Running   0             20hkube-system   kube-apiserver-minikube                       1/1     Running   0             20hkube-system   kube-controller-manager-minikube              1/1     Running   0             20hkube-system   kube-proxy-l6z4w                              1/1     Running   0             20hkube-system   kube-scheduler-minikube                       1/1     Running   0             20hkube-system   storage-provisioner                           1/1     Running   1 (20h ago)   20h

In order to makekube-awi-controller-manager working you need to modify the deployent:

kubectl edit deployment  kube-awi-controller-manager -n awi-system

Locate args and add--awi-catalyst-address pointing at the local processof awi-grpc-catalyst-sdwan

- args:    - --health-probe-bind-address=:8081    - --metrics-bind-address=127.0.0.1:8080    - --awi-catalyst-address=host.minikube.internal:50051    - --leader-elect

Thehost.minikube.internal address points to your host machine.The AWI GRPC Catalyst SDWAN needs to be started on0.0.0.0 ratherthan127.0.0.1 - otherwise it won't work.

If, for some reason, this won't be able to reach your host address(you will know that by the fact that manager logs will show only oneentry:connecting), try runningminikube tunnel in differentterminal.

After doing so, the pod should be restarted and initialized successfully,which you can inspect by seeing2/2 containers inget pods command andby seeing normal logs of your manager.

To create a connection, you can try running:

kubectl apply -f config/samples/awi_v1_internetworkdomain_vpc_to_vpc.yaml

Of course, the file needs to be modified to match your desired VPCs.

You can now see created connection:

> kubectl get internetworkdomainconnections---NAME                              AGEmy-internetworkdomainconnection   42s

This connection will most likely exist even if the actual connection was not created.Check your AWI GRPC Catalyst SDWAN logs to see how the creation went.

To destroy the connection, simply remove the CR:

kubectl delete internetworkdomainconnections my-internetworkdomainconnection

That's it :)

To create sample app connection, you can use the following file:

kubectl apply -f examples/gcp-vpc-vpc-label-app-connection.yaml

After your minikube cluster is no longer needed, you can remove it by running:

minikube delete

Thank you for interest in contributing! Please refer to ourcontributing guide.

kube-awi is released under the Apache 2.0 license. SeeLICENSE.

kube-awi is also made possible thanks tothird party open source projects.