Repository files navigation

Create a local Kubernetes development environment on macOS or Windows and WSL2.

Throughout this self-paced course, you'll gain hands-on experience with:

• Creating a local Kubernetes cluster withminikube
• Deploying applications in Kubernetes usingkubectl and this project's YAML files
• Serving applications securely via HTTPS withNGINX Ingress Controller (LoadBalancer) andcert-manager
• Managing Kubernetes resources as packages usingHelm v3
• Authenticating users withGoogle as an Identity Provider (IdP), implementing bothOAuth2 andOAuth2+OIDC usingoauth2-proxy
• Building the webserver applicationdocker-cats and deploying it to the Kubernetes cluster withkubectl rollout

• 05-Feb-2022: Updated to Kubernetes v1.21.2

• 04-May-2021: Added Challenges (Exercises): That is the first and last time you'll see the wordexercise in this repository. So from now on, I'll refer to it as achallenge. I've added thechallenges directory, which currently, contains two challenges, seechallenges/README.md. Feel free to post your feedback in theDiscussions section. I'll add more challenges, according to your feedback (yes,your feedback).

• 06-Apr-2021: Asynchronous Support And Discussions: If you have any questions, suggestions, ideas, or even if you want to show and tell about your work, feel free to create a discussion in theDiscussions section.

1. macOS/WSL2: Start the Docker daemon on your local machine; The simplest way is to runDocker For Desktop

2. macOS/WSL2: Create a Kubernetes cluster with minkube

   minikube start --driver=docker --kubernetes-version=v1.21.2# ...# 🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

3. macOS/WSL2: Check connectivity - HTTPS should work since we're usingca.crt

   MINIKUBE_EXPOSED_PORT="$(kubectl config view -o jsonpath='{.clusters[?(@.name == "minikube")].cluster.server}'| cut -d":" -f3)"&& \export MINIKUBE_EXPOSED_PORT=${MINIKUBE_EXPOSED_PORT}&& \curl -L --cacert~/.minikube/ca.crt"https://127.0.0.1:${MINIKUBE_EXPOSED_PORT}/version";echo# adds extra line

   A valid response

   {"major":"1","minor":"21","gitVersion":"v1.21.2","gitCommit":"092fbfbf53427de67cac1e9fa54aaa09a28371d7","gitTreeState":"clean","buildDate":"2021-06-16T12:53:14Z","goVersion":"go1.16.5","compiler":"gc","platform":"linux/amd64"}

The termHost refers to your machine (macOS/Windows). In this section, we're going to install CA certificates on the Host machine (macOS/Windows)

1. macOS/Windows: Check access to the cluster's endpoint by opening the browser inhttps://127.0.0.1:${MINIKUBE_EXPOSED_PORT}/version

   

Upon completing this course, you might askWhy does the Host need HTTPS access to minikube?. I've answered this question in theDicussions section.

"Picture this - You want to access the Kubernetes APIs, from the Host with HTTPS, how would you do that? SeeAccess Clusters Using the Kubernetes API. All the examples useHTTP, while with our setup we can useHTTPS, hooray for us 🎉"

IMPORTANT: In the latest version of minikube, v1.25.1 as of today, 26-Jan-2022, the rootCA is generated forlonger than 397 days; To overcome that, use version1.21.0.

1. macOS/Windows: LENS is so smart! Click on the Catalog icon (top left corner) and theminikube Kubernetes cluster will appear. LENS recognizes the kubernetes cluster context, according to the$HOME/.kube/config file.

   SelectAll namespaces

The main reasons why we deploy aKubernetes Ingress Controller

1. Load balancing traffic to services
2. A single endpoint that is exposed to the Host (macOS/Windows) and routes traffic to relevant services (apps)
3. Integrated HTTPS TLS termination, when appropriately configured 😉

An ingress controller is handy for exposing multiple services on the same set of ports (e.g., 80, 443). That is also a good practice for production environments where you hide your services in a private network and allow traffic only from a known external endpoint, such as a load balancer.

In this project, I chose to implement a Kubernetes Ingress Controller withNGINX's Ingress Controller. A great alternative isTraefik, though NGINX is probably the most popular.

• macOS/WSL2: Add the relevant Helm's repository and deploy the ingress controller

  helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx&& \helm repo update&& \helm upgrade --install nginx ingress-nginx/ingress-nginx --set controller.kind=DaemonSet# `upgrade --install` makes it idempotent

Helm is Kubernetes's package manager, which is similar to Python's package managerpip and Node's package managernpm. Though in Helm, "packages" are calledCharts.

Helm uses a packaging format calledcharts. A chart is a collection of files that describe a related set of Kubernetes resources. A single chart might be used to deploy something simple, like a memcached pod, or something complex, like a full web app stack with HTTP servers, databases, caches, and so on.Source

A chart usually contains a default set of values, and those values are defined in thevalues.yaml of the Helm chart.

For the sake of simplicity, when installing the NGINX's Helm Chart, I used the argument--set controller.kind=DaemonSet to override the default valuecontroller.kind=Deployment. Choosing theKubernetes DaemonSet kind means that NGINX'sKubernetes Pods are deployed to allKubernetes Nodes. I chose this setup for enabling High-Availability when adding more nodes to the cluster. High-Availability is probably irrelevant in a local development environment, but it undoubtedly helped me cover some core Kubernetes components. NGINX's default deployment kind is aKubernetes Deployment with a single replica. To see the default values check NGINX's Helm chartvalues.yaml file.

Another option for overriding the default values is to use a user-definedvalues.yaml file, a modified version of the originalvalues.yaml file.

1. macOS/WSL2: Copy thevalues.yaml file from the Chart's repository to your local machine

   curl -L -o values.yaml"https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/charts/ingress-nginx/values.yaml"

2. macOS/WSL2: Editvalues.yaml with your favorite text editor (Vim? 😃) and changekind: Deployment tokind: DaemonSet

   # Edit the file with your favotire text editor vimvim values.yaml

3. Install the helm chart and add the argument-f values.yaml

   # There's no need to execute this command as we already did it in the previous section with `--set`helm upgrade --install nginx ingress-nginx/ingress-nginx -f values.yaml

To access the NGINX Ingress Controller from the Host machine (macOS/Windows), we need to map its domain name to127.0.0.1, which will listen to ports 80 and 443.

1. Edit thehosts file
   • macOS: Edit/etc/hosts with your favotire editor

   sudo vim /etc/hosts

   • Windows: EditC:\Windows\System32\drivers\etc\hosts with Notepad orNotepad++ as Administrator
2. Add the following line

   127.0.0.1 baby.kubemaster.me green.kubemaster.me dark.kubemaster.me darker.kubemaster.me auth.kubemaster.me oidc.kubemaster.me

The downside is that you have to add any subdomain the application uses since wildcard domains such as*.mydomain.com are not allowed in thehosts file. The silver lining is you won't add all the subdomains that the application uses in production since the main goal is to test/develop only the necessary endpoints.

Deploy the1-baby.yaml app, a simple web application that serves static content and exposed to the Host (macOS/Windows) with aKubernetes Ingress.

1. macOS/WSL2: Clone this repo

   git clone https://github.com/unfor19/kubernetes-localdev.gitcd kubernetes-localdev

2. IMPORTANT: From now on, the working directory$PWD should be the cloned repository

3. macOS/WSL2: Deploy the application

   kubectl apply -f 1-baby.yaml

4. macOS/WSL2: Open anew terminal window and serve NGINX Ingress Controller on localhost (127.0.0.1), ports 80 and 443. That will provide thenginx-ingress-nginx-controllerKubernetes Service anExternal IP of the Windows host127.0.0.1.Keep it running in the background

   minikube tunnel# ❗  The service nginx-ingress-nginx-controller requires privileged ports to be exposed: [80 443]# 🔑  sudo permission will be asked for it.# 🏃  Starting tunnel for service nginx-ingress-nginx-controller

5. macOS/Windows: Check that minikube exposes NGINX Ingress Controller service on127.0.0.1

   

6. macOS/Windows: Open your browser in a new Incognito/Private window and navigate tohttp://baby.kubemaster.me/ (port 80). You should see a cute baby cat

   

IMPORTANT: The rest of this tutorial assumes thatminikube tunnel runs in the background in a separated terminal.

Create a localCertificate Authority certificate and key withmkcert and install it toTrusted Root Certificate Authorities on Windows, and toSystem Keychain on macOS. We'll use that certificate authority to create aTLS certificate for local development.

We're going to usecert-manager for issuing HTTPS/TLS certificates. Before we can do that, we need to create a Certificate Authority Certificate (rootCA.pem) and a Certificate Authority Key (rootCA-key.pem). Check out thethe difference between the two.

You can quickly generate a CA certificate and key withmkcert, which will also install both of them.There's no need to search for.crt files and install them.

We're usingcert-manager to issue TLS certificates per Ingress automatically. Adding the annotationcert-manager.io/cluster-issuer to the Ingress triggers the following process:

1. cert-manager spotted an ingress with thecert-manager.io/cluster-issuer annotation
2. cert-manager checks if thecluster-issuer exists and aborts if it doesn't
3. cert-manager checks if the ingress includesspec.tls[].secretName
   • Secretexists - aborts since it's a one-to-one relationship between a Kubernetes Ingress and a Kubernetes Secret when using thecluster-issuer annotation
   • Secretdoesn't exists - creates a secret in the Ingress's namespace. The secret's data is generated according tospec.tls[].hosts[] and the provided ClusterIssuer.

You can also create aCertificate (Kubernetes TLS Secret) and set itsspec.dnsNames[] to*.kubemaster.me, that will enable multiple ingresses to use the samesecretName. The downside of manually managing Certificates is that you have to create a Certificate per Kubernetes Namespace.

That is where thecluster-issuer annotation shines since it's "namespace independent". The Kubernetes Secret is created in the Ingress's namespaceautomatically.

Here's the snippet of2-green.yaml's Kubernetes Ingress

apiVersion:networking.k8s.io/v1beta1# NGINX Ingress Controller supports this versionkind:Ingressmetadata:name:greenannotations:cert-manager.io/cluster-issuer:tls-ca-issuer# Will be created in the next sectionspec:tls:  -hosts:    -green.kubemaster.mesecretName:green-tls-secret# Unique per ingress

We're going to create cert-manager'scustom resource definitions (CRDs), and then deploy cert-manager with Helm. Following that, we'll create the previously mentionedClusterIssuer.

Eventually, the NGINX Ingress controller will use the generated Kubernetes TLS Secret toterminate TLS connections (HTTPS --> HTTP).

1. macOS/WSL2: Add cert-manager to the Helm's repo, create cert-manager's CRDs and deploy cert-manager.

   helm repo add jetstack https://charts.jetstack.io&& \helm repo update&& \kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.2/cert-manager.crds.yaml&& \helm upgrade --install --wait cert-manager jetstack/cert-manager --namespace cert-manager --version v1.6.2

2. IMPORTANT: The ClusterIssuer will fail to create if cert-manager is not ready; see theTroubleshooting section if you experience any issues

3. macOS/WSL2: Create thecert-manager/clusterissuer.yaml and deploy the2-green.yaml application.

   # This issuer uses the TLS secret `kubemaster-me-ca-tls-secret` to create certificates for the ingresseskubectl apply -f cert-manager/clusterissuer.yaml&& \# Deploy sample appkubectl apply -f 2-green.yaml

4. macOS/Windows: Check connectivity to the deployedgreen app, open browser, and navigate tohttps://green.kubemaster.me (port 443). You should see a cat in a green scenery

   

We'll useoauth2-proxy to proxy requests toGoogle's OAuth 2.0 authentication service. Authenticated users are redirected to the initial URL that was requested (https://$host$escaped_request_uri).

Image Source:https://github.com/oauth2-proxy/oauth2-proxy

1. macOS/Windows:Google Developer Console >Create a New Project
   • Project Name:kubemaster
   • Organization: Leave empty
2. macOS/Windows:OAuth consent screen > SelectExternal > ClickCREATE
   • App name:kubemaster
   • User support email:your email address
   • Authorised domains > Add domain >kubemaster.me
   • Developer contact information:your email addressClickSAVE AND CONTINUE
3. macOS/Windows:Scopes > ClickSAVE AND CONTINTUE - there's no need for a scope, we don't plan on using Google APIs (authorization), we just need the authentication mechanism (OAuth2/OIDC)
4. (Optional)macOS/Windows:Test users > ClickSAVE AND CONTINTUE - it's irrelevant since either way we're allowing any Google user to login to the app since it's a local app
5. macOS/Windows:Summary > ClickBACK TO DASHBOARD
6. NOTE: There's no need toPUBLISH APP, keep it in sandbox mode
7. macOS/Windows:Credentials > ClickCREATE CREDENTIALS > SelectOAuth Client ID > Select Application typeWeb application
   • Name:kubemaster
   • Authorised JavaScript originsADD URI >https://auth.kubemaster.me
   • Authorised JavaScript originsADD URI >https://oidc.kubemaster.me (will use it later on)
   • Authorised redirect URIsADD URI >https://auth.kubemaster.me/oauth2/callback
   • Authorised redirect URIsADD URI >https://oidc.kubemaster.me/oauth2/callback (will use it later on)ClickCREATE
   • SaveYour Client ID andYour Client Secret in a safe place, we'll use them in the following section

1. macOS/WSL2:

   # Values from Google's Developer Console - the space at the beginning of the command is on purpose to keep it out from Bash's history OAUTH2_PROXY_CLIENT_ID="google_oauth2_project_client_id" OAUTH2_PROXY_CLIENT_SECRET="google_oauth2_project_client_secret"

   # Create the Kubernetes Secretkubectl -n default create secret generic google-credentials \    --from-literal=google_client_id="${OAUTH2_PROXY_CLIENT_ID}" \    --from-literal=google_client_secret="${OAUTH2_PROXY_CLIENT_SECRET}"

1. macOS/WSL2: Deploy3-oauth2-proxy.yaml and the sample3-dark.yaml application

   # Create a Kubernetes Secret for oauth2-proxy `cookie-secret`kubectl -n default create secret generic oauth2-proxy-cookie-secret --from-literal=oauth2_proxy_cookie_secret="$(docker run --rm python:3.9.1-alpine python -c'import os,base64; print(base64.urlsafe_b64encode(os.urandom(16)).decode())')"# Deploy oauth2-proxykubectl apply -f 3-oauth2-proxy.yaml# Deploy sample app `dark`, served via HTTPS and protected with Google authenticationkubectl apply -f 3-dark.yaml

2. macOS/Windows: Open a browser in a new Incognito/Private window and navigate tohttps://dark.kubemaster.me and login with your Google user. You should see a cat in a dark scenery.

   

OAuth2 was used for authentication in the previous step, though its primary purpose is forauthorization. Forauthentication, it is best to useOpen ID Connect (OIDC) whenever it's possible. The main benefit is that OIDC also provides the endpoint/userinfo, so the application can easily read aJSON Web Token (JWT) and get the user details such as full name and locale (preferred language).

As demonstrated in the below image, OIDCdoes not replace OAuth2. OIDC is a layer on top of OAuth2, which provides a better way to handle authentication.

Inspired by:https://developer.okta.com/blog/2018/11/26/spring-boot-2-dot-1-oidc-oauth2-reactive-apis

The deployment steps are same as before, though I do recommend viewing the files4-oauth2-proxy-oidc.yaml and4-darker.yaml, while comparing them to3-oauth2-proxy.yaml and3-dark.yaml.

The main difference is in theargs of oauth2-proxy's Deployment, where the provider is not using the default OAuth2 protocol for authentication; instead, it's using the OIDC protocol.

1. macOS/WSL2: Deploy4-oauth2-proxy-oidc.yaml and the sample4-darker.yaml application

   # Deploy oauth2-proxykubectl apply -f 4-oauth2-proxy-oidc.yaml# Deploy sample app `darker`, served via HTTPS and protected with Google authentication (OIDC)kubectl apply -f 4-darker.yaml

2. macOS/Windows: Open a browser in a new Incognito/Private window and navigate tohttps://darker.kubemaster.me and login with your Google user. You should see the same dark cat, but the message now contains your full name.

   • NOTE: If you have an existing browser window, even if it's incognito, then you might have already authenticated. You can verify it by checking if the cookie_oauth2_proxy exists. To get the entire flow, close all incognito windows and then open a new browser window in incognitohttps://darker.kubemaster.me

   • NOTE: If you've already authenticated when you navigated tohttps://dark.kubemaster.me (OAuth2), then you won't be prompted to be logged in when you navigate tohttps://darker.kubemaster.me (OIDC). Authentication occurs once, and thenoauth-proxy2 verifies the authenticated user with the secret cookie_oauth2_proxy for all subsequent requests. The cookie's domain is.kubemaster.me (includes any subdomain). That goes the other way around; if you've already authenticated onhttps://darker.kubemaster.me, you can also accesshttps://dark.kubemaster.me.

   • NOTE: Authenticating with OIDC (darker) provides more details about the authenticated user; therefore, it's possible to inject the user's name into the application. If you logged in with OAuth2 (dark), then your name won't be displayed in the message "Hello YOUR_GOOGLE_NAME". Google specifies the available user attributes in theID token's payload. Click the Expand/Collapse buttons to view the available attributes for OIDC and OAuth2.

     OIDC - ID token's payload - Expand/Collapse

     {/*    The Issuer Identifier for the Issuer of the response. Always https://accounts.google.com or accounts.google.com for Google ID tokens.    */"iss":"https://accounts.google.com",/*    The client_id of the authorized presenter. This claim is only needed when the party requesting the ID token is not the same as the audience of the ID token.    This may be the case at Google for hybrid apps where a web application and Android app have a different OAuth 2.0 client_id but share the same Google APIs project.    */"azp":"GOOGLE_CLIENT_ID",/*    The audience that this ID token is intended for. It must be one of the OAuth 2.0 client IDs of your application.    */"aud":"GOOGLE_CLIENT_ID",/*    An identifier for the user, unique among all Google accounts and never reused.    A Google account can have multiple email addresses at different points in time, but the sub value is never changed.    Use sub within your application as the unique-identifier key for the user. Maximum length of 255 case-sensitive ASCII characters.    */"sub":"USER_ID",/*    The user's email address. This value may not be unique to this user and is not suitable for use as a primary key.    Provided only if your scope included the email scope value.    */"email":"user@gmail.com",/*    True if the user's e-mail address has been verified; otherwise false.    */"email_verified":true,/*    Access token hash. Provides validation that the access token is tied to the identity token.    If the ID token is issued with an access_token value in the server flow, this claim is always included.    This claim can be used as an alternate mechanism to protect against cross-site request forgery attacks.    If you follow:    https://developers.google.com/identity/protocols/oauth2/openid-connect#createxsrftoken    and    https://developers.google.com/identity/protocols/oauth2/openid-connect#confirmxsrftoken    it is not necessary to verify the access token.    */"at_hash":"someNiceOverHere",/*    The user's full name, in a displayable form. Might be provided when:    The request scope included the string "profile"    The ID token is returned from a token refresh    When name claims are present, you can use them to update your app's user records. Note that this claim is never guaranteed to be present.    */"name":"Meir Gabay",/*    The URL of the user's profile picture. Might be provided when:    The request scope included the string "profile"    The ID token is returned from a token refresh    When picture claims are present, you can use them to update your app's user records. Note that this claim is never guaranteed to be present.    */"picture":"https://lh3.googleusercontent.com/a-/AOh14Gg2SJeDqusILfvvSG0boxvXX65QYrx5U3KK38xj-A=s96-c",/*    The user's given name(s) or first name(s). Might be provided when a name claim is present.    */"given_name":"Meir",/*    The user's surname(s) or last name(s). Might be provided when a name claim is present.    */"family_name":"Gabay",/*    The user's locale, represented by a BCP 47 language tag. Might be provided when a name claim is present.    */"locale":"en-GB",/*    The time the ID token was issued. Represented in Unix time (integer seconds).    */"iat":1618059677,/*    Expiration time on or after which the ID token must not be accepted. Represented in Unix time (integer seconds).    */"exp":1618063277}

     OAuth2 - ID token's payload - Expand/Collapse

     {/*    The Issuer Identifier for the Issuer of the response. Always https://accounts.google.com or accounts.google.com for Google ID tokens.    */"iss":"https://accounts.google.com",/*    The client_id of the authorized presenter. This claim is only needed when the party requesting the ID token is not the same as the audience of the ID token.    This may be the case at Google for hybrid apps where a web application and Android app have a different OAuth 2.0 client_id but share the same Google APIs project.    */"azp":"GOOGLE_CLIENT_ID",/*    The audience that this ID token is intended for. It must be one of the OAuth 2.0 client IDs of your application.    */"aud":"GOOGLE_CLIENT_ID",/*    An identifier for the user, unique among all Google accounts and never reused.    A Google account can have multiple email addresses at different points in time, but the sub value is never changed.    Use sub within your application as the unique-identifier key for the user. Maximum length of 255 case-sensitive ASCII characters.    */"sub":"USER_ID",/*    The user's email address. This value may not be unique to this user and is not suitable for use as a primary key.    Provided only if your scope included the email scope value.    */"email":"user@gmail.com",/*    True if the user's e-mail address has been verified; otherwise false.    */"email_verified":true,/*    Access token hash. Provides validation that the access token is tied to the identity token.    If the ID token is issued with an access_token value in the server flow, this claim is always included.    This claim can be used as an alternate mechanism to protect against cross-site request forgery attacks.    If you follow:    https://developers.google.com/identity/protocols/oauth2/openid-connect#createxsrftoken    and    https://developers.google.com/identity/protocols/oauth2/openid-connect#confirmxsrftoken    it is not necessary to verify the access token.    */"at_hash":"someNiceOverHere",/*    The time the ID token was issued. Represented in Unix time (integer seconds).    */"iat":1618059677,/*    Expiration time on or after which the ID token must not be accepted. Represented in Unix time (integer seconds).    */"exp":1618063277}

   

• I find it best to have a dedicated subdomain for Authentication services, as it allows using cookies with*.kubemaster.me and acts as an isolated service from the entire application
• TheAuthorised JavaScript origins andAuthorised redirect URIs in Google's Developer Console are used by oauth2-proxy. There's not a single time where Google tries to query your domains; this is why it's possible to make it work locally.
• Here's great 1 hour session about OAuth2 and OIDC -OAuth 2.0 and OpenID Connect (in plain English). I watched every bit of it, and it helped me to understand the whole flow.
• Using bare OAuth2 (without OIDC) means that if the app needs more details about the authenticated user, such asname, then the app will have to make another request from the backend to get this information. With OAuth2 + OIDC, you benefit from having extra details about the user in a single request.
• It's possible to access private resources by logging intohttps://auth.kubemaster.me andhttps://oidc.kubemaster.me since they both use the same Google's Credentials and COOKIE_SECRET (I think?)

We're running twoDocker Daemons, the first one runs on the Host machine (macOS/Windows), and the second one runs in minikube'sDocker Container. I find it very hard to understand this architecture, so I've created a diagram to visualize it.

Let's run some commands to see if it makes sense.

1. macOS/WSL2: Print the list of the running containers on theHost machine

   docker ps

   # Valid output - minikube's container name is `minikube`CONTAINER ID   IMAGE                                 COMMAND                  CREATED      STATUS       PORTS                                                                                                                                  NAMES9cab890fc446   gcr.io/k8s-minikube/kicbase:v0.0.18"/usr/local/bin/entr…"   5 days ago   Up 2 hours   127.0.0.1:63682->22/tcp, 127.0.0.      1:63683->2376/tcp, 127.0.0.1:63680->5000/tcp, 127.0.0.1:63681->8443/tcp, 127.0.0.1:63684->32443/tcp   minikube

2. macOS/WSL2: Usedocker exec to get into minikube's container

   dockerexec -it minikube bash

3. NOTE: For testing/debugging purposes, I prefer usingdocker exec overminikube ssh becausedocker exec allows logging asroot, whileminikube ssh as the non-root userdocker.

4. macOS/WSL2: Print the list of the running containers onminikube

   docker ps

   # Valid output# root@minikube:/# docker psCONTAINER ID   IMAGE                  COMMAND                  CREATED       STATUS       PORTS     NAMESd11f495c71a5   85069258b98a"/storage-provisioner"   2 hours ago   Up 2 hours             8s_storage-provisioner_storage-provisioner_kube-system_eb1bac83-2db5-41e8-9bdd-805e3969930b_3 ...5fbaf3683d33   k8s.gcr.io/pause:3.2"/pause"                 2 hours ago   Up 2 hours             k8s_POD_etcd-minikube_kube-system_c31fe6a5afdd142cf3450ac972274b36_1

Remember the sectionCreate a Kubernetes Cluster? We used the argument--driver=docker, which instructs minikube to use itsdocker driver. From the Kubernetes perspective, it's equivalent for choosing theDocker runtime as theContainer runtime. Eventually, this means thatKubernetes Workloads will run as Docker Containers.

As you can see from the last step, minikube's Docker Daemon runs containers that belong to theKubernetes Cluster. The Kubernetes object representing a group of containers, or a single container, is called aKubernetes Pod. If you're already familiar withDocker Compose, then writing aPod's YAML configuration is quite similar to writing adocker-compose.yaml file.

If you think about it, the Docker CLI sends API requests to theDocker Engine API, which is part of the Docker Daemon. Let's do a quick test to see if we cancurl the Host's (macOS/Windows) Docker Daemon and minikube's Docker Daemon.

1. macOS/WSL2: curl theinfo endpoint of theHost's Docker Daemon

   # Socket Request `--unix-socket`curl --unix-socket /var/run/docker.sock http://127.0.0.1/info

2. macOS/WSL2: curl theinfo endpoint ofminikube's Docker Daemon

   # HTTPS Requestcurl --cacert~/.minikube/certs/ca.pem \--key~/.minikube/certs/key.pem  --cert~/.minikube/certs/cert.pem \https://127.0.0.1:$(minikube docker-env| grep DOCKER_HOST| cut -d":" -f3| cut -d'"' -f1)/info

I haven't added the expected output since it's too long and can vary significantly between different Hosts. Search the attributeName in the output; for example, onWSL2, it'sdocker-desktop (Host) andminikube.

Initially, I've tried using a private local Docker repository, which was a nightmare (see myStackOverflow question). A more straightforward solution - using minikube's Docker Daemon, instead of the Host's (macOS/Windows) Docker Daemon for building Docker images.

1. macOS/WSL2: Setdocker command to use minikube's Docker Daemon

   eval`minikube docker-env`# from now on, the `docker` command refers to minikube's Docker Daemon# To undo the above command and use macOS/Windows's Docker Daemoneval`minikube docker-env --unset`

2. macOS/WSL2: Build the docker-cats application locally using minikube's Docker Daemon

   git clone https://github.com/unfor19/docker-cats.gitcd docker-catseval`minikube docker-env`# Using minikube's Docker Daemondocker build -t unfor19/docker-cats:latest.

We'll use the built-in kubectl commandrollout restart deployment/deployment-name. And of course, we'll probably create some Makefile or a bash script that runs bothbuild anddeploy.

1. macOS/WSL2:

   kubectl rollout restart deployment/baby deployment/green deployment/dark deployment/darker

IMPORTANT: Quit LENS before proceeding

• macOS/WSL2: Delete minikube's Kubernetes Cluster and CA certificates

  minikube delete --purge --all

• macOS/Windows: Uninstallmkcert's TLS certifictes, run PowerShell in elevated mode

  mkcert -uninstall# The local CA is now uninstalled from the system trust store(s)!

• Delete all relevant TLS certificates
  • macOS: Hit CMD+SPACE and runKeychain Access, delete allminikubeCA andminikube-user
  • Windows: Hit WINKEY+R and runcertmgr.msc
    1. Certificates - Current User > Trusted Root Certification Authorities > Certificates
    2. Delete all minikube's certificates -minikubeCA andminikube-user

1. Ingress: Make sure you expose the cluster to the Host withminikube tunnel before trying to access the application with the browser

   • ERR_CONNECTION_REFUSED

2. Ingress: Path-based ingresses issues, For exampleapp.kubemaster.me/baby would not work properly because the app serves static files in the root dir. The request to the HTML pageindex.html is successful, but subsequent requests toapp.kubemaster.me/baby/images/baby.png will fail since NGINX's upstream can't serve static content. It's best to use Path-based ingresses for serving APIs, for example,app.kubemaster.me/api/v1/get/something. Use bare (/) Host-based ingresses for serving static pages, just like I did in this project.

3. Ingress: version deprecation warning - ignore this warning; this is the latest version supported by the NGINX Ingress Controller

   Warning: networking.k8s.io/v1beta1 Ingress is deprecatedin v1.19+, unavailablein v1.22+; use networking.k8s.io/v1 Ingress

4. HTTPS: Certificate is invalid in the browser - Open your browser a new Incognito/Private window

   • ERR_CONNECTION_REFUSED

     

   • ERR_CERT_AUTHORITY_INVALID

     

5. cert-manager: Errors applying cert-manager resources

   • Delete the secretkubemaster-me-ca-tls-secret, re-create it and then re-applycert-manager/clusterissuer.yaml

     Error from server (NotFound): error when deleting "cert-manager/clusterissuer.yaml": clusterissuers.cert-manager.io "tls-ca-issuer" not found

   • Wait for cert-manager to be healthy, check all logs of the podscert-manager,cert-manager-cainjector, andcert-manager-webhook

     Error from server (InternalError): error when creating "cert-manager/clusterissuer.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.102.252.218:443: connect: connection refused

6. Authentication: After after a successful login you get redirected to/# 404 - Make sure the ingress annotation isoauth2/start?rd=https://$host$escaped_request_uri, for example

   nginx.ingress.kubernetes.io/auth-signin:https://auth.kubemaster.me/oauth2/start?rd=https://$host$escaped_request_uri

   NOTE: Even though you got to a 404 page, it's still possible to access private resources (dark and darker) check your Application cookies

7. LENS: Can't connect to cluster due to missing keys - Make sure you copiedclient.crt,client.key andca.crt from WSL2 to the Windows hostC:\Users\$HOST_USERNAME\.kube\certs

   error: unable to read client-key C:\Users\unfor19\.kube\certs\client.key for minikube due to open C:\Users\unfor19\.kube\certs\client.key: The system cannot find the file specified.

• How to Set Up an Nginx Ingress with Cert-Manager on DigitalOcean Kubernetes
• cert-manager docs
• minikube docker driver
• GitLab - Developing for Kubernetes with Minikube

• Docker Tips And Best Practices
• Troubleshooting the right way
• Boosting terminal productivity tips
• Parsing Command Line Arguments In Bash

• baby.jpg - sourcehttps://www.maxpixel.net/Animals-Stone-Kitten-Cats-Cat-Flower-Pet-Flowers-2536662
• green.jpg - sourcehttp://challengethestorm.org/cat-taught-love/ -img
• dark.jpg - sourcehttps://www.findcatnames.com/great-black-cat-names/ -img

• HackerBoy Emoticon -http://123emoji.com/wp-content/uploads/2016/08/5609229801667168449.png
• Helm -https://helm.sh/img/helm.svg
• NGINX -https://www.nginx.com/wp-content/uploads/2018/08/NGINX-logo-rgb-large.png
• Cert Manager -https://d33wubrfki0l68.cloudfront.net/f424659d40ec64748389d27b333510b3bd46f509/ed868/img/png/cert-manager-icon.png
• Kubernetes -https://kubernetes.io/images/kubernetes-horizontal-color.png
• OAuth2 Proxy -https://oauth2-proxy.github.io/oauth2-proxy/
• minikube -https://minikube.sigs.k8s.io/docs/

Created and maintained byMeir Gabay

This project is licensed under the MIT License - see theLICENSE file for details