In this tutorial, you set up a CI/CD solution usingGitOps with Flux v2 and Azure Arc-enabled Kubernetes or Azure Kubernetes Service (AKS) clusters. Using the sample Azure Vote app, you can:

If you don't have an Azure subscription, create afree account before you begin.

Prerequisites

• Complete theprevious tutorial to learn how to deploy GitOps for your CI/CD environment.

• Understand thebenefits and architecture of this feature.

• Verify you have:

  • Aconnected Azure Arc-enabled Kubernetes cluster namedarc-cicd-cluster.
  • A connected Azure Container Registry with eitherAKS integration ornon-AKS cluster authentication.

• Install the latest versions of these Azure Arc-enabled Kubernetes and Kubernetes Configuration CLI extensions:

  az extension add --name connectedk8saz extension add --name k8s-configuration

• Or to update these extensions to the latest version, run the following commands:

  az extension update --name connectedk8saz extension update --name k8s-configuration

Connect Azure Container Registry to Kubernetes

Enable your Kubernetes cluster to pull images from your Azure Container Registry. If it's private, authentication is required.

Connect Azure Container Registry to existing AKS clusters

Integrate an existing Azure Container Registry with existing AKS clusters using the following command:

az aks update -n arc-cicd-cluster -g myResourceGroup --attach-acr arc-demo-acr

Create an image pull secret

To connect non-AKS and local clusters to your Azure Container Registry, create an image pull secret. Kubernetes uses image pull secrets to store information needed to authenticate your registry.

Create an image pull secret with the followingkubectl command. Repeat for both thedev andstage namespaces.

kubectl create secret docker-registry <secret-name> \    --namespace <namespace> \    --docker-server=<container-registry-name>.azurecr.io \    --docker-username=<service-principal-ID> \    --docker-password=<service-principal-password>

To avoid having to set an imagePullSecret for every Pod, consider adding the imagePullSecret to the Service account in thedev andstage namespaces. For more information, see theKubernetes tutorial.

Depending on the CI/CD orchestrator you prefer, you can proceed with instructions either for Azure DevOps or for GitHub.

Implement CI/CD with Azure DevOps

This tutorial assumes familiarity with Azure DevOps, Azure Repos and Pipelines, and Azure CLI.

Make sure to complete the following steps first:

• Sign intoAzure DevOps Services.
• Verify you have "Build Admin" and "Project Admin" permissions forAzure Repos andAzure Pipelines.

Import application and GitOps repositories into Azure Repos

Import anapplication repository and aGitOps repository into Azure Repos. For this tutorial, use the following example repositories:

• arc-cicd-demo-src application repository

  • URL:https://github.com/Azure/arc-cicd-demo-src
  • Contains the example Azure Vote App to deploy using GitOps.
  • Import the repository with namearc-cicd-demo-src

• arc-cicd-demo-gitops GitOps repository

  • URL:https://github.com/Azure/arc-cicd-demo-gitops
  • Works as a base for your cluster resources that house the Azure Vote App.
  • Import the repository with namearc-cicd-demo-gitops

Learn more aboutimporting Git repositories.

Connect the GitOps repository

To continuously deploy your app, connect the GitOps repository to your cluster using GitOps. Yourarc-cicd-demo-gitops GitOps repository contains the basic resources to get your app up and running on yourarc-cicd-cluster cluster.

The initial GitOps repository contains only amanifest that creates thedev andstage namespaces corresponding to the deployment environments.

The GitOps connection that you create automatically syncs the manifests in the manifest directory and updates the cluster state.

The CI/CD workflow populates the manifest directory with extra manifests to deploy the app.

1. Create a new GitOps connection to your newly importedarc-cicd-demo-gitops repository in Azure Repos.

   az k8s-configuration flux create \   --name cluster-config \   --cluster-name arc-cicd-cluster \   --namespace flux-system \   --resource-group myResourceGroup \   -u https://dev.azure.com/<Your organization>/<Your project>/_git/arc-cicd-demo-gitops \   --https-user <Azure Repos username> \   --https-key <Azure Repos PAT token> \   --scope cluster \   --cluster-type connectedClusters \   --branch master \   --kustomization name=cluster-config prune=true path=arc-cicd-cluster/manifests

   Tip

   For an AKS cluster (rather than an Arc-enabled cluster), use-cluster-type managedClusters.

2. Check the state of the deployment in Azure portal.

   • If successful, you see bothdev andstage namespaces created in your cluster.
   • You can also confirm that on the Azure portal page of your cluster, a configurationcluster-config is created on theGitOps tab.

Import the CI/CD pipelines

Now that you synced a GitOps connection, you need to import the CI/CD pipelines that create the manifests.

The application repository contains a.pipeline folder with pipelines used for PRs, CI, and CD. Import and rename the three pipelines provided in the sample repository:

Connect Azure Container Registry to Azure DevOps

During the CI process, you deploy your application containers to a registry. Start by creating anAzure service connection:

1. In Azure DevOps, open theService connections page from the project settings page. In TFS, open theServices page from thesettings icon in the top menu bar.
2. Choose+ New service connection and select the type of service connection you need.
3. Fill in the parameters for the service connection. For this tutorial:
   • Name the service connectionarc-demo-acr.
   • SelectmyResourceGroup as the resource group.
4. Select theGrant access permission to all pipelines.
   • This option authorizes YAML pipeline files for service connections.
5. ChooseSave to create the connection.

Configure PR service connection

The CD pipeline manipulates pull requests (PRs) in the GitOps repository, which requires a service connection. To configure this connection:

1. In Azure DevOps, open theService connections page from the project settings page. In TFS, open theServices page from thesettings icon in the top menu bar.
2. Choose+ New service connection and selectGeneric type.
3. Fill in the parameters for the service connection. For this tutorial:
   • Server URLhttps://dev.azure.com/<Your organization>/<Your project>/_apis/git/repositories/arc-cicd-demo-gitops
   • Leave Username and Password blank.
   • Name the service connectionazdo-pr-connection.
4. Select theGrant access permission to all pipelines.
   • This option authorizes YAML pipeline files for service connections.
5. ChooseSave to create the connection.

Install GitOps Connector

1. Add GitOps Connector repository to Helm repositories:

      helm repo add gitops-connector https://azure.github.io/gitops-connector/

2. Install the connector to the cluster:

      helm upgrade -i gitops-connector gitops-connector/gitops-connector \      --namespace flux-system \      --set gitRepositoryType=AZDO \      --set ciCdOrchestratorType=AZDO \      --set gitOpsOperatorType=FLUX \      --set azdoGitOpsRepoName=arc-cicd-demo-gitops \      --set azdoOrgUrl=https://dev.azure.com/<Your organization>/<Your project> \      --set gitOpsAppURL=https://dev.azure.com/<Your organization>/<Your project>/_git/arc-cicd-demo-gitops \      --set orchestratorPAT=<Azure Repos PAT token>

   Note

   Azure Repos PAT token should haveBuild: Read & execute andCode: Full permissions.

3. Configure Flux to send notifications to GitOps connector:

   cat <<EOF | kubectl apply -f -apiVersion: notification.toolkit.fluxcd.io/v1beta2kind: Alertmetadata:  name: gitops-connector  namespace: flux-systemspec:  eventSeverity: info  eventSources:  - kind: GitRepository    name: cluster-config  - kind: Kustomization    name: cluster-config-cluster-config   providerRef:    name: gitops-connector---apiVersion: notification.toolkit.fluxcd.io/v1beta2kind: Providermetadata:  name: gitops-connector  namespace: flux-systemspec:  type: generic  address: http://gitops-connector:8080/gitopsphaseEOF

For details about installation, see theGitOps Connector repository.

Create environment variable groups

App repository variable group

Create avariable group namedaz-vote-app-dev. Set the following values:

Stage environment variable group

1. Clone theaz-vote-app-dev variable group.

2. Change the name toaz-vote-app-stage.

3. Ensure the following values for the corresponding variables:

   Variable	Value
   ENVIRONMENT_NAME	Stage
   TARGET_NAMESPACE	stage

You're now ready to deploy to thedev andstage environments.

Create environments

In your Azure DevOps project, createDev andStage environments. For details, seeCreate and target environments.

Give more permissions to the build service

The CD pipeline uses the security token of the running build to authenticate to the GitOps repository. More permissions are needed for the pipeline to create a new branch, push changes, and create PRs. To enable these permissions:

1. In Azure DevOps, openProject settings.
2. UnderRepositories, selectRepos.
3. SelectSecurity.
4. Find<Project Name> Build Service (<Organization Name>) andProject Collection Build Service (<Organization Name>) (use search if you don't see them), and allowContribute,Contribute to pull requests, andCreate branch.
5. UnderPipelines, selectSettings.
6. Turn off theProtect access to repositories in YAML pipelines option.

For more information, seeGrant version control permissions to the build service andManage build service account permissions.

Deploy the dev environment for the first time

With the CI and CD pipelines created, run the CI pipeline to deploy the app for the first time.

CI pipeline

During the initial CI pipeline run, if you see a resource authorization error in reading the service connection name, do the following:

1. Verify the variable being accessed is AZURE_SUBSCRIPTION.
2. Authorize the use.
3. Rerun the pipeline.

The CI pipeline:

• Ensures the application change passes all automated quality checks for deployment.
• Does any extra validation that couldn't be completed in the PR pipeline. Specific to GitOps, the pipeline also publishes the artifacts for the commit that will be deployed by the CD pipeline.
• Verifies the Docker image has changed and the new image is pushed.

CD pipeline

During the initial CD pipeline run, you need to give the pipeline access to the GitOps repository. SelectView when prompted that the pipeline needs permission to access a resource. Then, selectPermit to grant permission to use the GitOps repository for the current and future runs of the pipeline.

The successful CI pipeline run triggers the CD pipeline to complete the deployment process. You deploy to each environment incrementally.

Once the template and manifest changes to the GitOps repository are generated, the CD pipeline creates a commit, pushes it, and creates a PR for approval.

1. Find the PR created by the pipeline to the GitOps repository.

2. Verify the changes to the GitOps repository. You should see:

   • High-level Helm template changes.
   • Low-level Kubernetes manifests that show the underlying changes to the desired state. Flux deploys these manifests.

3. If everything looks good, approve and complete the PR.

4. After a few minutes, Flux picks up the change and starts the deployment.

5. Monitor thegit commit status on theCommit history tab. Once it issucceeded, the CD pipeline starts automated testing.

6. Forward the port locally usingkubectl and ensure the app works correctly using:

   kubectl port-forward -n dev svc/azure-vote-front 8080:80

7. View the Azure Vote app in your browser athttp://localhost:8080/.

8. Vote for your favorites and get ready to make some changes to the app.

Set up environment approvals

Upon app deployment, you can make changes to the code or templates, but you can also unintentionally put the cluster into a bad state.

If the dev environment reveals a break after deployment, enabling environment approvals helps keep the problem from later environments.

1. In your Azure DevOps project, go to the environment that needs to be protected.
2. Navigate toApprovals and Checks for the resource.
3. SelectCreate.
4. Provide the approvers and an optional message.
5. SelectCreate again to complete the addition of the manual approval check.

For more information, seeDefine approval and checks.

Next time the CD pipeline runs, the pipeline will pause after the GitOps PR creation. Verify that the change is properly synced and passes basic functionality. Approve the check from the pipeline to let the change flow to the next environment.

Make an application change

With this baseline set of templates and manifests representing the state on the cluster, you make a small change to the app.

1. In thearc-cicd-demo-src repository, editazure-vote/src/azure-vote-front/config_file.cfg file.

2. Since "Cats vs Dogs" isn't getting enough votes, change it to "Tabs vs Spaces" to drive up the vote count.

3. Commit the change in a new branch, push it, and create a pull request. This sequence of steps is the typical developer flow that starts the CI/CD lifecycle.

PR validation pipeline

The PR pipeline is the first line of defense against a faulty change. Usual application code quality checks include linting and static analysis. From a GitOps perspective, you also need to assure the same quality for the resulting infrastructure to be deployed.

The application's Dockerfile and Helm charts can use linting in a similar way to the application.

Errors found during linting range from incorrectly formatted YAML files, to best practice suggestions, such as setting CPU and Memory limits for your application.

Errors found during pipeline execution appear in the test results section of the run. From here, you can:

• Track the useful statistics on the error types.
• Find the first commit on which they were detected.
• Stack trace style links to the code sections that caused the error.

The pipeline run finishes, confirming the quality of the application code and the template that deploys it. You can now approve and complete the PR. The CI runs again, regenerating the templates and manifests, before triggering the CD pipeline.

CD process approvals

A successful CI pipeline run triggers the CD pipeline to complete the deployment process. This time, the pipeline requires you to approve each deployment environment.

1. Approve the deployment to thedev environment.
2. Once the template and manifest changes to the GitOps repository have been generated, the CD pipeline creates a commit, pushes it, and creates a PR for approval.
3. Verify the changes to the GitOps repository. You should see:
   • High-level Helm template changes.
   • Low-level Kubernetes manifests that show the underlying changes to the desired state.
4. If everything looks good, approve and complete the PR.
5. Wait for the deployment to complete.
6. As a basic smoke test, navigate to the application page and verify the voting app now displaysTabs vs Spaces.
   • Forward the port locally usingkubectl and ensure the app works correctly using:kubectl port-forward -n dev svc/azure-vote-front 8080:80
   • View the Azure Vote app in your browser athttp://localhost:8080/ and verify the voting choices have changed to Tabs vs Spaces.
7. Repeat steps 1-7 for thestage environment.

The deployment is now complete.

For a detailed overview of all the steps and techniques implemented in the CI/CD workflows used in this tutorial, see theAzure DevOps GitOps Flow diagram.

Implement CI/CD with GitHub

This tutorial assumes familiarity with GitHub, GitHub Actions.

Fork application and GitOps repositories

Fork anapplication repository and aGitOps repository. For this tutorial, use the following example repositories:

• arc-cicd-demo-src application repository

  • URL:https://github.com/Azure/arc-cicd-demo-src
  • Contains the example Azure Vote App to deploy using GitOps.

• arc-cicd-demo-gitops GitOps repository

  • URL:https://github.com/Azure/arc-cicd-demo-gitops
  • Works as a base for your cluster resources that house the Azure Vote App.

Connect the GitOps repository

To continuously deploy your app, connect the application repository to your cluster using GitOps. Yourarc-cicd-demo-gitops GitOps repository contains the basic resources to get your app up and running on yourarc-cicd-cluster cluster.

The initial GitOps repository contains only amanifest that creates thedev andstage namespaces corresponding to the deployment environments.

The GitOps connection that you create automatically:

• Sync the manifests in the manifest directory.
• Update the cluster state.

The CI/CD workflow populates the manifest directory with extra manifests to deploy the app.

1. Create a new GitOps connection to your newly forkedarc-cicd-demo-gitops repository in GitHub.

   az k8s-configuration flux create \   --name cluster-config \   --cluster-name arc-cicd-cluster \   --namespace cluster-config \   --resource-group myResourceGroup \   -u  https://github.com/<Your organization>/arc-cicd-demo-gitops.git \   --https-user <Azure Repos username> \   --https-key <Azure Repos PAT token> \   --scope cluster \   --cluster-type connectedClusters \   --branch master \   --kustomization name=cluster-config prune=true path=arc-cicd-cluster/manifests

2. Check the state of the deployment in Azure portal.

   • If successful, you see bothdev andstage namespaces created in your cluster.

Install GitOps Connector

1. Add GitOps Connector repository to Helm repositories:

      helm repo add gitops-connector https://azure.github.io/gitops-connector/

2. Install the connector to the cluster:

      helm upgrade -i gitops-connector gitops-connector/gitops-connector \      --namespace flux-system \      --set gitRepositoryType=GITHUB \      --set ciCdOrchestratorType=GITHUB \      --set gitOpsOperatorType=FLUX \      --set gitHubGitOpsRepoName=arc-cicd-demo-src \      --set gitHubGitOpsManifestsRepoName=arc-cicd-demo-gitops \      --set gitHubOrgUrl=https://api.github.com/repos/<Your organization> \      --set gitOpsAppURL=https://github.com/<Your organization>/arc-cicd-demo-gitops/commit \      --set orchestratorPAT=<GitHub PAT token>

3. Configure Flux to send notifications to GitOps connector:

   cat <<EOF | kubectl apply -f -apiVersion: notification.toolkit.fluxcd.io/v1beta2kind: Alertmetadata:  name: gitops-connector  namespace: flux-systemspec:  eventSeverity: info  eventSources:  - kind: GitRepository    name: cluster-config  - kind: Kustomization    name: cluster-config-cluster-config  providerRef:    name: gitops-connector---apiVersion: notification.toolkit.fluxcd.io/v1beta2kind: Providermetadata:  name: gitops-connector  namespace: flux-systemspec:  type: generic  address: http://gitops-connector:8080/gitopsphaseEOF

For the details on installation, refer to theGitOps Connector repository.

Create GitHub secrets

The next step is to create GitHub repository and environment secrets.

Create GitHub repository secrets

Use the following values for your GitHub repository secrets:

Create GitHub environment secrets

1. Createaz-vote-app-dev environment with the following secrets:

1. Createaz-vote-app-stage environment with the following secrets:

You're now ready to deploy to thedev andstage environments.

CI/CD Dev workflow

To start the CI/CD Dev workflow, change the source code. In the application repository, update values in.azure-vote/src/azure-vote-front/config_file.cfg file and push the changes to the repository.

The CI/CD Dev workflow:

• Ensures the application change passes all automated quality checks for deployment.
• Does any extra validation that couldn't be completed in the PR pipeline.
• Verifies the Docker image has changed and the new image is pushed.
• Publishes the artifacts (Docker image tags, Manifest templates, Utils) that are used by the following CD stages.
• Deploys the application to Dev environment.
  • Generates manifests to the GitOps repository.
  • Creates a PR to the GitOps repository for approval.

Once these steps are completed:

1. Find the PR created by the pipeline to the GitOps repository.

2. Verify the changes to the GitOps repository. You should see:

   • High-level Helm template changes.
   • Low-level Kubernetes manifests that show the underlying changes to the desired state. Flux deploys these manifests.

3. If everything looks good, approve and complete the PR.

4. After a few minutes, Flux picks up the change and starts the deployment.

5. Monitor thegit commit status on the Commit history tab. Once it issucceeded, theCD Stage workflow starts.

6. Forward the port locally usingkubectl and ensure the app works correctly using:

   kubectl port-forward -n dev svc/azure-vote-front 8080:80

7. View the Azure Vote app in your browser athttp://localhost:8080/.

8. Vote for your favorites and get ready to make some changes to the app.

CD Stage workflow

The CD Stage workflow starts automatically once Flux successfully deploys the application to dev environment and notifies GitHub actions via GitOps Connector.

The CD Stage workflow:

• Runs application smoke tests against Dev environment
• Deploys the application to Stage environment.
  • Generates manifests to the GitOps repository
  • Creates a PR to the GitOps repository for approval

Once the manifests PR to the Stage environment is merged and Flux successfully applies all the changes, the Git commit status is updated in the GitOps repository. The deployment is now complete.

For a detailed overview of all the steps and techniques implemented in the CI/CD workflows used in this tutorial, see theGitHub GitOps Flow diagram.

Clean up resources

If you're not going to continue to use this application, delete any resources with the following steps:

1. Delete the Azure Arc GitOps configuration connection:

   az k8s-configuration flux delete \      --name cluster-config \      --cluster-name arc-cicd-cluster \      --resource-group myResourceGroup \      -t connectedClusters --yes

2. Delete GitOps Connector:

   helm uninstall gitops-connector -n flux-systemkubectl delete alerts.notification.toolkit.fluxcd.io gitops-connector -n flux-systemkubectl delete providers.notification.toolkit.fluxcd.io  gitops-connector -n flux-system

Next steps

In this tutorial, you set up a full CI/CD workflow that implements DevOps from application development through deployment. Changes to the app automatically trigger validation and deployment, gated by manual approvals.

Advance to our conceptual article to learn more about GitOps and configurations with Azure Arc-enabled Kubernetes.