Repository files navigation

Popper is a tool for defining and executing container-native testingworkflows in Docker. With Popper, you define a workflow in a YAMLfile, and then execute it with a single command. A workflow file lookslike this:

steps:-id:dev-inituses:docker://rikorose/gcc-cmake:gcc-9runs:[sh, -uexc]args:  -|    rm -rf build/    cmake -DCMAKE_BUILD_TYPE=Release -S . -B build-id:builduses:docker://rikorose/gcc-cmake:gcc-9runs:[cmake, --build, build/, --parallel, '4']-id:testuses:docker://rikorose/gcc-cmake:gcc-9dir:/workspace/build/runs:[ctest]

Assuming the above is stored in aci.yml file in the root of yourproject folder, this entire workflow gets executed by running:

popper run -f ci.yml

Running a single step:

popper run -f ci.yml build

Starting a shell inside thebuild step container:

popper run -f ci.yml build

Running on another engine (Podman):

popper run -f ci.yml -e podman build

See theexamples/ folder for examples for tests forother languages, as well as other types of tests (integration,regresssion, etc.).

To install or upgrade Popper, run the following in your terminal:

curl -sSfL https://raw.githubusercontent.com/getpopper/popper/master/install.sh| sh

Docker is required to run Popper and the installer willabort if thedocker command cannot be invoked from your shell. Forother installation options, including installing for use with theother supported engines (Singularity and Podman), or for setting up adeveloping environment for Popper,read the complete installationinstructions.

Once installed, you can get an overview and list of availablecommands:

popperhelp

Read theQuickstart Guide to learn the basics ofhow to use Popper. Or browse theOfficial documentation.

• Lightweight workflow and task automation syntax. Defining a list ofsteps is as simple as writing file in alightweight YAML syntax andinvokingpopper run (see demo above). If you're familiar withDocker Compose, you can think of Popper as Compose butfor end-to-end tasks instead of services.

• An abstraction over container runtimes. In addition to Docker,Popper can seamlessly execute workflows in other runtimes byinteracting with distinct container engines. Popper currentlysupportsDocker,Singularity andPodman.

• An abstraction over CI services. Define a pipeline once andthen instruct Popper to generate configuration files for distinctCI services, allowing users to run the exact same workflows theyrun locally on Travis, Jenkins, Gitlab, Circle and others. See theexamples/ folder for examples on how to automateCI tasks for multiple projects (Go, C++, Node, etc.).

• An abstraction over resource managers. Popper can also executeworkflows on a variety of resource managers and schedulers such asKubernetes and SLURM, without requiring any modifications to aworkflow YAML file. We currently support SLURM and are working onadding support for Kubernetes.

• Aid in workflow development. Aid in the implementation anddebugging of workflows, and provide with an extensivelist ofexampleworkflows that canserve as a starting point.

Popper is a container-native workflow execution and task automationengine. In practice, when we work following thecontainer-native paradigm, we end upinteractively executing multiple Docker commands (docker pull,docker build,docker run, etc.) so that we can build containers,compile code, test applications, deploy software, among others.Keeping track of whichdocker commands we have executed, in whichorder, and which flags were passed to each, can quickly becomeunmanageable, difficult to document (think of outdated READMEinstructions) and error prone.

On top of this, having the same workflow work in other environments(CI, K8S, etc.) is time-consuming and defeats the purpose of usingcontainers (portability). The goal of Popper is to bring order to thischaotic scenario by providing a framework for clearly and explicitlydefining container-native tasks. You can think of Popper as tool forwrapping all these manual tasks in a lightweight, machine-readable,self-documented format (YAML).

While this sounds simple at first, it has significant implications:results in time-savings, improves communication and in general unifiesdevelopment, testing and deployment workflows. As a developer or userof "Popperized" container-native projects, you only need to learn onetool, and leave the execution details to Popper, whether is to buildand tests applications locally, on a remote CI server or a Kubernetescluster.

Anyone is welcome to contribute to Popper! To get started, take a lookat ourcontributing guidelines, then dive in withourlist of good first issues.

Popper adheres to the code of conductposted in thisrepository. By participating or contributing toPopper, you're expected to uphold this code. If you encounter unacceptablebehavior, please immediatelyemail us.

Ivo Jimenez, Michael Sevilla, Noah Watkins, Carlos Maltzahn, JayLofstead, Kathryn Mohror, Andrea Arpaci-Dusseau and RemziArpaci-Dusseau.The Popper Convention: Making Reproducible SystemsEvaluation Practical. In 2017 IEEE International Parallel andDistributed Processing Symposium Workshops (IPDPSW), 1561–70, 2017.(https://doi.org/10.1109/IPDPSW.2017.157)

PDF for a pre-print versionavailable here.For BibTeX,click here.