🕰️ In the previous episode

In the first part of this Dagger's series, I showed you what's Dagger.io, what's the features of it and it's benefits against others ci/cd solutions and finally the very basis of Dagger.

With this chapter, I will show you how we can overpower the CI/CD of any Pulumi project using Dagger.

🧰 Pulumi - An amazing IaC tool

First of all, I think that some of you may doesn't know what is Pulumi or evenIaC (Infrastructure as Code) concept, so I will quickly present to you these two points.

Infrastructure as Code

Nowadays, IT extends to many areas, and so new needs are emerging leadingly to a necessity to adapt infrastructures in order to be able to support all of this.
They also have seen their prerequisites evolve given their multiplication and their increasingly large sizes. As a result, companies started wanting to automate and simplify their infrastructures.

To address this problem, Amazon unveiled in 2006 the concept of Infrastructure As Code (orIaC) allowing, on Amazon Web Services, the configuration of instances using computer code.
It was a revolution for infrastructure management and, although limited at the time, this method was quickly adopted by the market.
This event also coincides with the date of appearance of the DevOps movement in which it's part.

Most of the infrastructure as code tools are based on the use of descriptor files to organize the code which avoids duplication between environments. Some advanced tools support variability, the use of outputs and even deployment to several providers simultaneously.

There are three types of infrastructure as code:

• Imperative: resources (instances, networks, etc.) are declared via a list of instructions in a defined order to obtain an expected result.
• Functional: unlike the imperative mode, the order of the instructions does not matter. The resources are defined in such a way that their final configuration is as expected.
• Based on the environment: the resources are declared in such a way that their state and their final configuration are consistent with the rest of the environment.

The advantages of IaC compared to traditional management are numerous, such as: cost reduction, the possibility of versioning the infrastructure, the speed of deployment and execution, the ability to collaborate, etc.
It also allows complete automation, in fact, once the process is launched, there is no longer any need for human intervention. This advantage not only limits the risks due to human error and therefore increases reliability, but also allows teams to focus on projects and less on the deployment of applications.

Pulumi

Pulumi is an open-source IaC tool. It can be used to create, manage and deploy an infrastructure on many cloud provider like AWS, GCP, Scaleway, etc.
It haves few strengths compared to others IaC solutions likeTerraform for example.

First of all, Pulumi support many programming languages like Go, TypeScript, Python, Java, F# and few others.
This is a great advantage for Pulumi and one of the reasons why we begin to use it atCamptocamp because the presence of a programming language like Go, rather than a configuration language like HCL (language used byTerraform) allows much more flexibility and adaptability.

Furthermore, Pulumi has new native providers like AWS, Google, and others in order to get new versions the same day they are released.
Additionally, Pulumi also supports Terraform providers to maintain compatibility with any infrastructure built using Terraform.

Another very interesting advantage is the support of what they call “Dynamic Providers” which allows to easily extend an existing provider with new types of personalized resources and that by directly programming new CRUD operations.
This can allow new resources to be added, for example, while adding complex migration or configuration logic.

Finally, there are still many advantages to Pulumi such as the ease of carrying out tests thanks to the native frameworks of the programming languages provided for this usage, the presence ofaliases allowing a resource to be renamed while maintaining compatibility with the state of the infrastructure, better integrations with mainstream IDEs like VSCode, etc.

⚡ Supercharge your Pulumi project thanks to Dagger

Now that you knowIaC,Pulumi and of courseDagger (if not, you can check the first part of this blog series), we will see how we can create CI/CD pipelines for any Pulumi project using Dagger and CUE and finally how can we run them.
For that, I will present to you the Dagger architecture we have built at Camptocamp, it was designed to be complete and reusable. It may be too complex for small projects but if you understand it you will normally be able to create your own !

Important note: Initially Dagger.io was developed usingCuelang, an amazing and modern declarative language. Cue was also the only way to do pipelines with it. However, Dagger's team have recently transformed Dagger to be language agnostic.
We now have different SDKs: Go (the main one), CUE, Node.js and Python.
As of today, I advice you to choose the Go SDK if you don't really know which one to take. For the CUE one, I only recommend it to you if:

• You like declarative language like YAML
• You know CUE or want to learn it

For the following chapters, I will present our implementation which is made in CUE. However, it will be easy to transpose it to other SDKs if you understand it.

📦 The Pulumi package

In order to make a reusable and powerful Dagger project, we decided to create a "Pulumi" package which embeds our objects definitions like:

• the Docker image
• the container definition
• the command object

As an exemple there is the command object definition:

// pulumi/command.cuepackage pulumiimport (    "dagger.io/dagger"    "universe.dagger.io/bash"    "universe.dagger.io/docker")#Command: self = {    image?: docker.#Image    name:   string    args: [...string]    env: [string]: string    source: dagger.#FS    input?: dagger.#FS    _forceRun: bash.#RunSimple & {        script: contents: "true"        always: true    }    _container: #Container & {        if self.image != _|_ {            image: self.image        }        source: self.source        command: {            name: self.name            args: self.args        }        env: self.env & {            FORCE_RUN_HACK: "\(_forceRun.success)"        }        if self.input != _|_ {            mounts: input: {                type:     "fs"                dest:     "/input"                contents: self.input            }        }        export: directories: "/output": _    }    output: _container.export.directories."/output"}

At the top of this file, you can see the package definition and all our imports.universe.dagger.io is a community repository where we can find pre-made package for Docker, bash, alpine, etc.
Just after that, we start defining our#Command object by adding public fields like command's name, args, custom Docker image (which is optional due to? character, etc.
We also define our unexported fields (which aren't accessible from outside the package) like the container definition which will run our command (the container object is defined in thecontainer.cue file in the same repository).

This is one of the few definitions that we have in this package, I will not show you directly all of them since it's really specific to our implementation and it's not very relevant to explain how it's working.
However, you can retrieve all of our source code here:https://github.com/camptocamp/dagger-pulumi

So one last file from this pulumi package that we will see is themain.cue one. It's where we defined all the Pulumi commands that we will use (using the#Command object seen just before).

// pulumi/main.cue[...]#Preview: self = {    stack: string    diff:  bool | *false    #Command & {        name: "preview"        args: [            "--stack",            stack,            "--save-plan",            "/output/plan.json",            if diff {                "--diff"            },        ]    }    _file: core.#ReadFile & {        input: self.output        path:  "plan.json"    }    plan: _file.contents}#Update: {    stack: string    diff:  bool | *false    plan:  string    _file: core.#WriteFile & {        input:    dagger.#Scratch        path:     "plan.json"        contents: planHere, you defined     }    #Command & {        name: "update"        args: [            "--stack",            stack,                        [...]            if diff {                "--diff"            },            "--skip-preview",        ]        input: _file.output    }}[...]

I voluntary remove some parts of the file and leave only the definition for Pulumi preview and update commands in order to keep it simple.
In the Pulumi context, preview command is used to compare the codebase which defines the desired infrastructure state with the actual one and preview all the potential changes if we apply the configuration.
The update one, as it name says, update the deployed infrastructure to match the desired state.

Using these definitions, we finally defined a#Pulumi object in themain.cue from the parentci package.
It's composed by few attributes used to set our project environment like, for example, the Pulumi stack (workspace), the env variables, the authorization to run destructive actions through this CI, etc.
We also have a list of all available commands construct using all#Command objects defined earlier.

Finally, the last step before we can be able to run our pipelines is to create the Dagger's plan.
This is the most important part where all jobs are defined but it's really specific to the project for which it's built.
I will present to you a plan from one of our projects where we used the pulumi package that I show you just before.

import (    "github.com/camptocamp/pulumi-aws-schweizmobil/ci"    "dagger.io/dagger")dagger.#Plan & {    client: {        env: {            PULUMI_GOMAXPROCS?: string            PULUMI_GOMEMLIMIT?: string        }        filesystem: {            sources: read: {                path:     "."                contents: dagger.#FS                exclude: [                    ".*",                ]            }            // TODO            // Simplify once https://github.com/dagger/dagger/issues/2909 is fixed            plan: read: {                path:     "plan.json"                contents: string            }            planWrite: write: {                path:     "plan.json"                contents: actions.preview.plan            }        }    }    #Pulumi: ci.#Pulumi & {        env: {            if client.env.PULUMI_GOMAXPROCS != _|_ {                GOMAXPROCS: client.env.PULUMI_GOMAXPROCS            }            if client.env.PULUMI_GOMEMLIMIT != _|_ {                GOMEMLIMIT: client.env.PULUMI_GOMEMLIMIT            }        }        source: client.filesystem.sources.read.contents        stack:  string        diff:   bool        update: plan: client.filesystem.plan.read.contents        enableDestructiveActions: bool    }    actions: {        #Pulumi.commands    }}

Firstly, you can see at the begin of this file the import of the homemadeci package (as a reminder, it is fully availablehere)
After, we defined thedagger.#Plan object which is composed by few parts:

• the client config: this is where we will be able to set some runtime elements like: env variables, filesystems with read/write permissions, the sources location...
• the actions: this is all the possible jobs that you will be able to run using Dagger CLI or in your CI/CD environment, in our case we just use local variable#Pulumi.commands which is built using the#Pulumi object defined in theci package.

🚀 Launch Dagger's jobs

Locally using CLI

As I told in the previous part of this blog series, Dagger can run any jobs locally thanks to his Dockerize design.
You must have installed the Dagger's CLI for Cue SDK (check the first part of this series orthe official documentation if it's not already done).

Once ready, you can open your favorite terminal client located inside your project directory (composed by your Pulumi project and your Dagger plan) and run:

• dagger --plan=ci.cue do --with '#Pulumi: stack: "<your_stack>"' preview -> it will run aPulumi preview on the desired stack
• dagger --plan=ci.cue do --with '#Pulumi: stack: "<your_stack>", #Pulumi: diff: true, #Pulumi: enableDestructiveActions: true' update -> it will run aPulumi update on the desired stack

Remotely using a CI/CD environment

With Dagger you can also run your pipelines on every CI/CD environments!
As a simple example, you can easily run these jobs on Github Actions:

name:pulumi-projecton:push:branches:-mainjobs:dagger:runs-on:ubuntu-lateststeps:-name:Clone repositoryuses:actions/checkout@v2-name:Install Dagger Enginerun:|cd /usr/localwget -O - https://dl.dagger.io/dagger/install.sh | sudo shcd --name:Run Pulumi update using Daggerrun:|dagger-cue --plan=ci.cue do update --log-format plain

Note than the--log-format plain flag is useful in order to have well formatted logs in the Github Action output.
With this action, apulumi update will be launched at every push on the main branch using Dagger.

🔚 Conclusion

If you have made it this far I thank you very much and I hope I have succeeded to give you a good overview of what is Dagger and how is it possible to exploit it in order to improve its practices in terms of CI/CD.

From my point of view, Dagger is a very promising solution. Right now we can still feel Dagger's young age, indeed, it's still not perfect.
In my opinion, each SDK should continue to develop and harmonize with each other. I also think that Dagger's execution performance and cache management should be improved (I'm basing myself on the 0.2 version of the engine since 0.3 is not yet available with the Cuelang SDK so maybe that's already better!).
Nevertheless, the advantages of Dagger are already essential for me, such as the possibility of running its pipelines locally or even being able to run them remotely on a remote VM using local sources.

To be followed very closely! 😉