Pumba: chaos testing tool for Docker

Prerequisites

Demo

Usage

$ pumba helpPumba version [VERSION](./blob/master/VERSION)NAME:   Pumba - Pumba is a resilience testing tool, that helps applications tolerate random Docker container failures: process, network and performance.USAGE:   pumba [global options] command [command options] containers (name, list of names, RE2 regex)VERSION:   [VERSION](./blob/master/VERSION) - `git rev-parse HEAD --short` and `build time`COMMANDS:   kill      kill specified containers   exec      exec specified containers   restart   restart specified containers   stop      stop containers   pause     pause all processes   rm        remove containers   stress    stress test a specified containers   netem     emulate the properties of wide area networks   iptables  apply IPv4 packet filter on incoming IP packets   help, h   Shows a list of commands or help for one commandGLOBAL OPTIONS:   --host value, -H value       daemon socket to connect to (default: "unix:///var/run/docker.sock") [$DOCKER_HOST]   --tls                        use TLS; implied by --tlsverify   --tlsverify                  use TLS and verify the remote [$DOCKER_TLS_VERIFY]   --tlscacert value            trust certs signed only by this CA (default: "/etc/ssl/docker/ca.pem")   --tlscert value              client certificate for TLS authentication (default: "/etc/ssl/docker/cert.pem")   --tlskey value               client key for TLS authentication (default: "/etc/ssl/docker/key.pem")   --log-level value, -l value  set log level (debug, info, warning(*), error, fatal, panic) (default: "warning") [$LOG_LEVEL]   --json, -j                   produce log in JSON format: Logstash and Splunk friendly [$LOG_JSON]   --slackhook value            web hook url; send Pumba log events to Slack   --slackchannel value         Slack channel (default #pumba) (default: "#pumba")   --interval value, -i value   recurrent interval for chaos command; use with optional unit suffix: 'ms/s/m/h' (default: 0s)   --label value                filter containers by labels, e.g '--label key=value' (multiple labels supported)   --random, -r                 randomly select single matching container from list of target containers   --dry-run                    dry run does not create chaos, only logs planned chaos commands [$DRY-RUN]   --skip-error                 skip chaos command error and retry to execute the command on next interval tick   --help, -h                   show help   --version, -v                print the version

Kill Container command

pumba kill -hNAME:   pumba kill - kill specified containersUSAGE:   pumba [global options] kill [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   send termination signal to the main process inside target container(s)OPTIONS:   --signal value, -s value  termination signal, that will be sent by Pumba to the main process inside target container(s) (default: "SIGKILL")   --limit value, -l value   limit to number of container to kill (0: kill all matching) (default: 0)

Pause Container command

pumba pause -hNAME:   pumba pause - pause all processesUSAGE:   pumba pause [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   pause all running processes within target containersOPTIONS:   --duration value, -d value  pause duration: should be smaller than recurrent interval; use with optional unit suffix: 'ms/s/m/h'

Stop Container command

pumba stop -hNAME:   pumba stop - stop containersUSAGE:   pumba stop [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   stop the main process inside target containers, sending  SIGTERM, and then SIGKILL after a grace periodOPTIONS:   --time value, -t value  seconds to wait for stop before killing container (default 10) (default: 10)

Remove (rm) Container command

pumba rm -hNAME:   pumba rm - remove containersUSAGE:   pumba rm [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   remove target containers, with links and voluemsOPTIONS:   --force, -f    force the removal of a running container (with SIGKILL, default: true)   --links, -l    remove container links (default: false)   --volumes, -v  remove volumes associated with the container (default: true)

Network Emulation (netem) command

pumba netem -hNAME:   Pumba netem - delay, loss, duplicate and re-order (run 'netem') packets, to emulate different network problemsUSAGE:   Pumba netem command [command options] [arguments...]COMMANDS:     delay      delay egress traffic     loss     duplicate     corrupt     rate       limit egress trafficOPTIONS:   --duration value, -d value   network emulation duration; should be smaller than recurrent interval; use with optional unit suffix: 'ms/s/m/h'   --interface value, -i value  network interface to apply delay on (default: "eth0")   --target value, -t value     target IP filter; comma separated. netem will impact only on traffic to target IP(s)   --tc-image value             Docker image with tc (iproute2 package); try 'ghcr.io/alexei-led/pumba-debian-nettools'   --help, -h                   show help

Network Emulation Delay sub-command

pumba netem delay -hNAME:   Pumba netem delay - delay egress trafficUSAGE:   Pumba netem delay [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   delay egress traffic for specified containers; networks show variability so it is possible to add random variation; delay variation isn't purely random, so to emulate that there is a correlationOPTIONS:   --time value, -t value          delay time; in milliseconds (default: 100)   --jitter value, -j value        random delay variation (jitter); in milliseconds; example: 100ms ± 10ms (default: 10)   --correlation value, -c value   delay correlation; in percentage (default: 20)   --distribution value, -d value  delay distribution, can be one of {<empty> | uniform | normal | pareto |  paretonormal}

Network Emulation Loss sub-commands

pumba netem loss -hNAME:   Pumba netem loss - adds packet lossesUSAGE:   Pumba netem loss [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   adds packet losses, based on independent (Bernoulli) probability model   see:  http://www.voiptroubleshooter.com/indepth/burstloss.htmlOPTIONS:   --percent value, -p value      packet loss percentage (default: 0)   --correlation value, -c value  loss correlation; in percentage (default: 0)

pumba netem loss-state -hNAME:   Pumba netem loss-state - adds packet losses, based on 4-state Markov probability modelUSAGE:   Pumba netem loss-state [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   adds a packet losses, based on 4-state Markov probability model     state (1) – packet received successfully     state (2) – packet received within a burst     state (3) – packet lost within a burst     state (4) – isolated packet lost within a gap   see: http://www.voiptroubleshooter.com/indepth/burstloss.htmlOPTIONS:   --p13 value  probability to go from state (1) to state (3) (default: 0)   --p31 value  probability to go from state (3) to state (1) (default: 100)   --p32 value  probability to go from state (3) to state (2) (default: 0)   --p23 value  probability to go from state (2) to state (3) (default: 100)   --p14 value  probability to go from state (1) to state (4) (default: 0)

pumba netem loss-gemodel -hNAME:   Pumba netem loss-gemodel - adds packet losses, according to the Gilbert-Elliot loss modelUSAGE:   Pumba netem loss-gemodel [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   adds packet losses, according to the Gilbert-Elliot loss model   see: http://www.voiptroubleshooter.com/indepth/burstloss.htmlOPTIONS:   --pg value, -p value  transition probability into the bad state (default: 0)   --pb value, -r value  transition probability into the good state (default: 100)   --one-h value         loss probability in the bad state (default: 100)   --one-k value         loss probability in the good state (default: 0)

pumba netem rate -hNAME:   Pumba netem rate - rate limit egress trafficUSAGE:   Pumba netem rate [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   rate limit egress traffic for specified containersOPTIONS:   --rate value, -r value            delay outgoing packets; in common units (default: "100kbit")   --packetoverhead value, -p value  per packet overhead; in bytes (default: 0)   --cellsize value, -s value        cell size of the simulated link layer scheme (default: 0)   --celloverhead value, -c value    per cell overhead; in bytes (default: 0)

Network Emulation Duplicate sub-commands

pumba netem duplicate -hNAME:   Pumba netem duplicate - adds duplicate packetsUSAGE:   Pumba netem duplicate [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   adds duplicate packets, based on independent (Bernoulli) probability model   see:  http://www.voiptroubleshooter.com/indepth/burstloss.htmlOPTIONS:   --percent value, -p value      packet duplicate percentage (default: 0)   --correlation value, -c value  duplicate correlation; in percentage (default: 0)

Network Emulation Corrupt sub-commands

pumba netem corrupt -hNAME:   Pumba netem corrupt - adds corrupt packetsUSAGE:   Pumba netem corrupt [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   adds corrupt packets, based on independent (Bernoulli) probability model   see:  http://www.voiptroubleshooter.com/indepth/burstloss.htmlOPTIONS:   --percent value, -p value      packet corrupt percentage (default: 0)   --correlation value, -c value  corrupt correlation; in percentage (default: 0)

Examples

# add 3 seconds delay for all outgoing packets on device `eth0` (default) of `mydb` Docker container for 5 minutespumba netem --duration 5m delay --time 3000 mydb

# add a delay of 3000ms ± 30ms, with the next random element depending 20% on the last one,# for all outgoing packets on device `eth1` of all Docker container, with name start with `test`# for 5 minutespumba netem --duration 5m --interface eth1 delay \      --time 3000 \      --jitter 30 \      --correlation 20 \    "re2:^test"

# add a delay of 3000ms ± 40ms, where variation in delay is described by `normal` distribution,# for all outgoing packets on device `eth0` of randomly chosen Docker container from the list# for 5 minutespumba --random netem --duration 5m \    delay \      --time 3000 \      --jitter 40 \      --distribution normal \    container1 container2 container3

# Corrupt 10% of the packets from the `mydb` Docker container for 5 minutespumba netem --duration 5m corrupt --percent 10 mydb

# Using the multi-arch nettools image explicitly# This is useful when you need to ensure both netem and iptables commands use the same imagepumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \    --duration 5m \    delay --time 1000 \    --jitter 100 \    myapp

Network Tools Images

Combined NetTools Images

# Use the same nettools image for both netem and iptables commandspumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest delay --time 100 mycontainerpumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest loss --probability 0.2 mycontainer

Architecture Support

Building Network Tools Images

# Build single-arch images for local testingmake build-local-nettools# Build multi-architecture images locally (doesn't push)make build-nettools-images# Build and push the multi-architecture images to GitHub Container Registrymake push-nettools-images

# Set your GitHub username and tokenexport GITHUB_USERNAME=your-github-usernameexport GITHUB_TOKEN=your-personal-access-token# Login to GitHub Container Registryecho$GITHUB_TOKEN| docker login ghcr.io -u$GITHUB_USERNAME --password-stdin# Run the make command with the environment variablesmake push-nettools-images

GITHUB_USERNAME=your-github-username GITHUB_TOKEN=your-personal-access-token make push-nettools-images

IPTables command

pumba iptables -hNAME:   Pumba iptables - emulate loss of incoming packets, all ports and address arguments will result in seperate rulesUSAGE:   Pumba iptables command [command options] containers (name, list of names, or RE2 regex if prefixed with "re2:"COMMANDS:   loss  adds iptables rules to generate packet loss on ingress trafficOPTIONS:   --duration value, -d value             network emulation duration; should be smaller than recurrent interval; use with optional unit suffix: 'ms/s/m/h' (default: 0s)   --interface value, -i value            network interface to apply input rules on (default: "eth0")   --protocol value, -p value             protocol to apply input rules on (any, udp, tcp or icmp) (default: "any")   --source value, --src value, -s value  source IP filter; supports multiple IPs; supports CIDR notation   --destination value, --dest value      destination IP filter; supports multiple IPs; supports CIDR notation   --src-port value, --sport value        source port filter; supports multiple ports (comma-separated)   --dst-port value, --dport value        destination port filter; supports multiple ports (comma-separated)   --iptables-image value                 Docker image with iptables and tc tools (default: "ghcr.io/alexei-led/pumba-alpine-nettools:latest")   --pull-image                           force pull iptables-image   --help, -h                             show help

IPTables loss command

pumba iptables loss -hNAME:   Pumba iptables loss - adds iptables rules to generate packet loss on ingress trafficUSAGE:   Pumba iptables loss [command options] containers (name, list of names, or RE2 regex if prefixed with "re2:"DESCRIPTION:   adds packet losses on ingress traffic by setting iptable statistic rules   see:  https://www.man7.org/linux/man-pages/man8/iptables-extensions.8.htmlOPTIONS:   --mode value         matching mode, supported modes are random and nth (default: "random")   --probability value  set the probability for a packet to me matched in random mode, between 0.0 and 1.0 (default: 0)   --every value        match one packet every nth packet, works only with nth mode (default: 0)   --packet value       set the initial counter value (0 <= packet <= n-1, default 0) for nth mode (default: 0)

Using theiptables Commands

Examples

# Drop 20% of incoming packets for a container named "web"pumba iptables loss --probability 0.2 web

# Drop every 5th packet coming from IP 192.168.1.100 to container "api" on port 8080pumba iptables loss --mode nth --every 5 --protocol tcp --source 192.168.1.100 --dst-port 8080 api

# Drop 15% of incoming ICMP packets (ping) for all containers with names matching "database"pumba iptables loss --probability 0.15 --protocol icmp"re2:database"

# Complex example: Drop 25% of TCP traffic coming to port 443 from a specific subnet, for 30 secondspumba iptables --duration 30s --protocol tcp --source 10.0.0.0/24 --dst-port 443 \    loss --probability 0.25 mycontainer

iptables Image Requirements

Advanced Network Chaos Scenarios

Asymmetric Network Conditions

# Add delay to outgoing traffic (slow uploads)pumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 5m delay --time 500 myapp&# Add packet loss to incoming traffic (unreliable downloads)pumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 5m loss --probability 0.1 myapp&

Combined Network Degradation

# Limit bandwidth and add packet corruptionpumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 10m rate --rate 1mbit myapp&# Add packet loss to incoming trafficpumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 10m loss --probability 0.05 myapp&

Testing Microservices Resilience

# Add high latency between service A and service Bpumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --target service-b-ip --duration 5m delay --time 2000 --jitter 500 service-a&# Add packet loss from service B to service Cpumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --source service-c-ip --duration 5m loss --probability 0.2 service-b&

Example Script

Stress testing Docker containers

NAME:   pumba stress - stress test a specified containersUSAGE:   pumba stress [command options] containers (name, list of names, or RE2 regex if prefixed with "re2:")DESCRIPTION:   stress test target container(s)OPTIONS:   --duration value, -d value  stress duration: must be shorter than recurrent interval; use with optional unit suffix: 'ms/s/m/h'   --stress-image value        Docker image with stress-ng tool, cgroup-bin and docker packages, and dockhack script (default: "alexeiled/stress-ng:latest-ubuntu")   --pull-image                pull stress-image form Docker registry   --stressors value           stress-ng stressors; see https://kernel.ubuntu.com/~cking/stress-ng/ (default: "--cpu 4 --timeout 60s")

stress-ng image requirements

Running inside Docker container

# run 10 Docker containers named test_(index)for i in `seq 1 10`; do docker run -d --name test_$i --rm alpine tail -f /dev/null; done# once in a 10 seconds, try to kill (with `SIGKILL` signal) all containers named **test(something)**# on same Docker host, where Pumba container is running$ docker run -it --rm  -v /var/run/docker.sock:/var/run/docker.sock gaiaadm/pumba --interval=10s --random --log-level=info kill --signal=SIGKILL "re2:^test"

Running Pumba on Kubernetes cluster

kubectl create -f deploy/pumba_kube.yml

"io.kubernetes.container.name":"test-container""io.kubernetes.pod.name":"test-pod""io.kubernetes.pod.namespace":"test-namespace"

Build instructions

Build using local Go environment

# create required foldercd$GOPATHmkdir github.com/alexei-led&&cd github.com/alexei-led# clone pumbagit clone git@github.com:alexei-led/pumba.gitcd pumba# build pumba binarymake# run tests and create HTML coverage reportmake test-coverage# create pumba binaries for multiple platformsmake release

Build using Docker

git clone git@github.com:alexei-led/pumba.gitcd pumba

DOCKER_BUILDKIT=1 docker build -t pumba -f docker/Dockerfile.

Exec Container command

pumba exec -hNAME:   pumba exec - exec specified containersUSAGE:   pumba [global options] exec [command options] containers (name, list of names, RE2 regex)DESCRIPTION:   send command to target container(s)OPTIONS:   --command value, -s value  shell command, that will be sent by Pumba to the target container(s) (default: "kill 1")   --args value, -a value     additional arguments for the command (can be repeated for multiple arguments)   --limit value, -l value    limit number of container to exec (0: exec all matching) (default: 0)

Examples

# Execute default command (kill 1) in container named webpumbaexec web

# Execute a custom command (echo) with a single argument in container named webpumbaexec --command"echo" --args"hello" web

# Execute ls with multiple arguments in all containers matching regex# Use repeated --args flags for multiple argumentspumbaexec --command"ls" --args"-la" --args"/etc""re2:^api.*"

# Limit execution to only 2 containers even if more matchpumbaexec --command"touch" --args"/tmp/test-file" --limit 2"re2:.*"

Network Tools Images

Combined NetTools Images

# Use the same nettools image for both netem and iptables commandspumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest delay --time 100 mycontainerpumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest loss --probability 0.2 mycontainer

Architecture Support

Building Network Tools Images

# Build single-arch images for local testingmake build-local-nettools# Build multi-architecture images locally (doesn't push)make build-nettools-images# Build and push the multi-architecture images to GitHub Container Registrymake push-nettools-images

# Set your GitHub username and tokenexport GITHUB_USERNAME=your-github-usernameexport GITHUB_TOKEN=your-personal-access-token# Login to GitHub Container Registryecho$GITHUB_TOKEN| docker login ghcr.io -u$GITHUB_USERNAME --password-stdin# Run the make command with the environment variablesmake push-nettools-images

GITHUB_USERNAME=your-github-username GITHUB_TOKEN=your-personal-access-token make push-nettools-images

IPTables command

pumba iptables -hNAME:   Pumba iptables - emulate loss of incoming packets, all ports and address arguments will result in seperate rulesUSAGE:   Pumba iptables command [command options] containers (name, list of names, or RE2 regex if prefixed with "re2:"COMMANDS:   loss  adds iptables rules to generate packet loss on ingress trafficOPTIONS:   --duration value, -d value             network emulation duration; should be smaller than recurrent interval; use with optional unit suffix: 'ms/s/m/h' (default: 0s)   --interface value, -i value            network interface to apply input rules on (default: "eth0")   --protocol value, -p value             protocol to apply input rules on (any, udp, tcp or icmp) (default: "any")   --source value, --src value, -s value  source IP filter; supports multiple IPs; supports CIDR notation   --destination value, --dest value      destination IP filter; supports multiple IPs; supports CIDR notation   --src-port value, --sport value        source port filter; supports multiple ports (comma-separated)   --dst-port value, --dport value        destination port filter; supports multiple ports (comma-separated)   --iptables-image value                 Docker image with iptables and tc tools (default: "ghcr.io/alexei-led/pumba-alpine-nettools:latest")   --pull-image                           force pull iptables-image   --help, -h                             show help

IPTables loss command

pumba iptables loss -hNAME:   Pumba iptables loss - adds iptables rules to generate packet loss on ingress trafficUSAGE:   Pumba iptables loss [command options] containers (name, list of names, or RE2 regex if prefixed with "re2:"DESCRIPTION:   adds packet losses on ingress traffic by setting iptable statistic rules   see:  https://www.man7.org/linux/man-pages/man8/iptables-extensions.8.htmlOPTIONS:   --mode value         matching mode, supported modes are random and nth (default: "random")   --probability value  set the probability for a packet to me matched in random mode, between 0.0 and 1.0 (default: 0)   --every value        match one packet every nth packet, works only with nth mode (default: 0)   --packet value       set the initial counter value (0 <= packet <= n-1, default 0) for nth mode (default: 0)

Using theiptables Commands

Examples

# Drop 20% of incoming packets for a container named "web"pumba iptables loss --probability 0.2 web

# Drop every 5th packet coming from IP 192.168.1.100 to container "api" on port 8080pumba iptables loss --mode nth --every 5 --protocol tcp --source 192.168.1.100 --dst-port 8080 api

# Drop 15% of incoming ICMP packets (ping) for all containers with names matching "database"pumba iptables loss --probability 0.15 --protocol icmp"re2:database"

# Complex example: Drop 25% of TCP traffic coming to port 443 from a specific subnet, for 30 secondspumba iptables --duration 30s --protocol tcp --source 10.0.0.0/24 --dst-port 443 \    loss --probability 0.25 mycontainer

iptables Image Requirements

Advanced Network Chaos Scenarios

Asymmetric Network Conditions

# Add delay to outgoing traffic (slow uploads)pumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 5m delay --time 500 myapp&# Add packet loss to incoming traffic (unreliable downloads)pumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 5m loss --probability 0.1 myapp&

Combined Network Degradation

# Limit bandwidth and add packet corruptionpumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 10m rate --rate 1mbit myapp&# Add packet loss to incoming trafficpumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --duration 10m loss --probability 0.05 myapp&

Testing Microservices Resilience

# Add high latency between service A and service Bpumba netem --tc-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --target service-b-ip --duration 5m delay --time 2000 --jitter 500 service-a&# Add packet loss from service B to service Cpumba iptables --iptables-image ghcr.io/alexei-led/pumba-alpine-nettools:latest \  --source service-c-ip --duration 5m loss --probability 0.2 service-b&

Example Script

Stress testing Docker containers

NAME:   pumba stress - stress test a specified containersUSAGE:   pumba stress [command options] containers (name, list of names, or RE2 regex if prefixed with "re2:")DESCRIPTION:   stress test target container(s)OPTIONS:   --duration value, -d value  stress duration: must be shorter than recurrent interval; use with optional unit suffix: 'ms/s/m/h'   --stress-image value        Docker image with stress-ng tool, cgroup-bin and docker packages, and dockhack script (default: "alexeiled/stress-ng:latest-ubuntu")   --pull-image                pull stress-image form Docker registry   --stressors value           stress-ng stressors; see https://kernel.ubuntu.com/~cking/stress-ng/ (default: "--cpu 4 --timeout 60s")

stress-ng image requirements

Running inside Docker container

# run 10 Docker containers named test_(index)for i in `seq 1 10`; do docker run -d --name test_$i --rm alpine tail -f /dev/null; done# once in a 10 seconds, try to kill (with `SIGKILL` signal) all containers named **test(something)**# on same Docker host, where Pumba container is running$ docker run -it --rm  -v /var/run/docker.sock:/var/run/docker.sock gaiaadm/pumba --interval=10s --random --log-level=info kill --signal=SIGKILL "re2:^test"

Running Pumba on Kubernetes cluster

kubectl create -f deploy/pumba_kube.yml

"io.kubernetes.container.name":"test-container""io.kubernetes.pod.name":"test-pod""io.kubernetes.pod.namespace":"test-namespace"

Build instructions

Build using local Go environment

# create required foldercd$GOPATHmkdir github.com/alexei-led&&cd github.com/alexei-led# clone pumbagit clone git@github.com:alexei-led/pumba.gitcd pumba# build pumba binarymake# run tests and create HTML coverage reportmake test-coverage# create pumba binaries for multiple platformsmake release

Build using Docker

git clone git@github.com:alexei-led/pumba.gitcd pumba

DOCKER_BUILDKIT=1 docker build -t pumba -f docker/Dockerfile.

License