Repository files navigation

SwarmKit is a toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.

Its main benefits are:

• Distributed:SwarmKit uses theRaft Consensus Algorithm in order to coordinate and does not rely on a single point of failure to perform decisions.
• Secure: Node communication and membership within aSwarm are secure out of the box.SwarmKit uses mutual TLS for nodeauthentication,role authorization andtransport encryption, automating both certificate issuance and rotation.
• Simple:SwarmKit is operationally simple and minimizes infrastructure dependencies. It does not need an external database to operate.

Machines runningSwarmKit can be grouped together in order to form aSwarm, coordinating tasks with each other.Once a machine joins, it becomes aSwarm Node. Nodes can either beworker nodes ormanager nodes.

• Worker Nodes are responsible for running Tasks using anExecutor.SwarmKit comes with a defaultDocker Container Executor that can be easily swapped out.
• Manager Nodes on the other hand accept specifications from the user and are responsible for reconciling the desired state with the actual cluster state.

An operator can dynamically update a Node's role by promoting a Worker to Manager or demoting a Manager to Worker.

Tasks are organized inServices. A service is a higher level abstraction that allows the user to declare the desired state of a group of tasks.Services define what type of task should be created as well as how to execute them (e.g. run this many replicas at all times) and how to update them (e.g. rolling updates).

Some ofSwarmKit's main features are:

• Orchestration

  • Desired State Reconciliation:SwarmKit constantly compares the desired state against the current cluster state and reconciles the two if necessary. For instance, if a node fails,SwarmKit reschedules its tasks onto a different node.

  • Service Types: There are different types of services. The project currently ships with two of them out of the box

    • Replicated Services are scaled to the desired number of replicas.
    • Global Services run one task on every available node in the cluster.

  • Configurable Updates: At any time, you can change the value of one or more fields for a service. After you make the update,SwarmKit reconciles the desired state by ensuring all tasks are using the desired settings. By default, it performs a lockstep update - that is, update all tasks at the same time. This can be configured through different knobs:

    • Parallelism defines how many updates can be performed at the same time.
    • Delay sets the minimum delay between updates.SwarmKit will start by shutting down the previous task, bring up a new one, wait for it to transition to theRUNNING statethen wait for the additional configured delay. Finally, it will move onto other tasks.

  • Restart Policies: The orchestration layer monitors tasks and reacts to failures based on the specified policy. The operator can define restart conditions, delays and limits (maximum number of attempts in a given time window).SwarmKit can decide to restart a task on a different machine. This means that faulty nodes will gradually be drained of their tasks.

• Scheduling

  • Resource Awareness:SwarmKit is aware of resources available on nodes and will place tasks accordingly.

  • Constraints: Operators can limit the set of nodes where a task can be scheduled by defining constraint expressions. Multiple constraints find nodes that satisfy every expression, i.e., anAND match. Constraints can match node attributes in the following table. Note thatengine.labels are collected from Docker Engine with information like operating system, drivers, etc.node.labels are added by cluster administrators for operational purpose. For example, some nodes have security compliant labels to run tasks with compliant requirements.

    node attribute	matches	example
    node.id	node's ID	node.id == 2ivku8v2gvtg4
    node.hostname	node's hostname	node.hostname != node-2
    node.ip	node's IP address	node.ip != 172.19.17.0/24
    node.role	node's manager or worker role	node.role == manager
    node.platform.os	node's operating system	node.platform.os == linux
    node.platform.arch	node's architecture	node.platform.arch == x86_64
    node.labels	node's labels added by cluster admins	node.labels.security == high
    engine.labels	Docker Engine's labels	engine.labels.operatingsystem == ubuntu 14.04

  • Strategies: The project currently ships with aspread strategy which will attempt to schedule tasks on the least loadednodes, provided they meet the constraints and resource requirements.

• Cluster Management

  • State Store: Manager nodes maintain a strongly consistent, replicated (Raft based) and extremely fast (in-memory reads) view of the cluster which allows them to make quick scheduling decisions while tolerating failures.
  • Topology Management: Node roles (Worker /Manager) can be dynamically changed through API/CLI calls.
  • Node Management: An operator can alter the desired availability of a node: Setting it toPaused will prevent any further tasks from being scheduled to it whileDrained will have the same effect while also re-scheduling its tasks somewhere else (mostly for maintenance scenarios).

• Security

  • Mutual TLS: All nodes communicate with each other using mutualTLS. Swarm managers act as aRoot Certificate Authority, issuing certificates to new nodes.
  • Token-based Join: All nodes require a cryptographic token to join the swarm, which defines that node's role. Tokens can be rotated as often as desired without affecting already-joined nodes.
  • Certificate Rotation: TLS Certificates are rotated and reloaded transparently on every node, allowing a user to set how frequently rotation should happen (the current default is 3 months, the minimum is 30 minutes).

Requirements:

• Go 1.6 or higher
• Aworking golang environment
• Protobuf 3.x or higher to regenerate protocol buffer files (e.g. usingmake generate)

SwarmKit is built in Go and leverages a standard project structure to work well with Go tooling.If you are new to Go, please seeBUILDING.md for a more detailed guide.

Once you haveSwarmKit checked out in your$GOPATH, theMakefile can be used for common tasks.

From the project root directory, run the following to buildswarmd andswarmctl:

$ make binaries

Before running tests for the first time, setup the tooling:

$ make setup

Then run:

$ make all

These instructions assume thatswarmd andswarmctl are in your PATH.

(Before starting, make sure/tmp/node-N don't exist)

Initialize the first node:

$ swarmd -d /tmp/node-1 --listen-control-api /tmp/node-1/swarm.sock --hostname node-1

Before joining cluster, the token should be fetched:

$ export SWARM_SOCKET=/tmp/node-1/swarm.sock  $ swarmctl cluster inspect default  ID          : 87d2ecpg12dfonxp3g562fru1Name        : defaultOrchestration settings:  Task history entries: 5Dispatcher settings:  Dispatcher heartbeat period: 5sCertificate Authority settings:  Certificate Validity Duration: 2160h0m0s  Join Tokens:    Worker: SWMTKN-1-3vi7ajem0jed8guusgvyl98nfg18ibg4pclify6wzac6ucrhg3-0117z3s2ytr6egmmnlr6gd37n    Manager: SWMTKN-1-3vi7ajem0jed8guusgvyl98nfg18ibg4pclify6wzac6ucrhg3-d1ohk84br3ph0njyexw0wdagx

In two additional terminals, join two nodes. From the example below, replace127.0.0.1:4242with the address of the first node, and use the<Worker Token> acquired above.In this example, the<Worker Token> isSWMTKN-1-3vi7ajem0jed8guusgvyl98nfg18ibg4pclify6wzac6ucrhg3-0117z3s2ytr6egmmnlr6gd37n.If the joining nodes run on the same host asnode-1, select a different remotelistening port, e.g.,--listen-remote-api 127.0.0.1:4343.

$ swarmd -d /tmp/node-2 --hostname node-2 --join-addr 127.0.0.1:4242 --join-token<Worker Token>$ swarmd -d /tmp/node-3 --hostname node-3 --join-addr 127.0.0.1:4242 --join-token<Worker Token>

If joining as a manager, also specify the listen-control-api.

$ swarmd -d /tmp/node-4 --hostname node-4 --join-addr 127.0.0.1:4242 --join-token<Manager Token> --listen-control-api /tmp/node-4/swarm.sock --listen-remote-api 127.0.0.1:4245

In a fourth terminal, useswarmctl to explore and control the cluster. Beforerunningswarmctl, set theSWARM_SOCKET environment variable to the path of themanager socket that was specified in--listen-control-api when starting themanager.

To list nodes:

$ export SWARM_SOCKET=/tmp/node-1/swarm.sock$ swarmctl node lsID                         Name    Membership  Status  Availability  Manager Status--                         ----    ----------  ------  ------------  --------------3x12fpoi36eujbdkgdnbvbi6r  node-2  ACCEPTED    READY   ACTIVE4spl3tyipofoa2iwqgabsdcve  node-1  ACCEPTED    READY   ACTIVE        REACHABLE *dknwk1uqxhnyyujq66ho0h54t  node-3  ACCEPTED    READY   ACTIVEzw3rwfawdasdewfq66ho34eaw  node-4  ACCEPTED    READY   ACTIVE        REACHABLE

Start aredis service:

$ swarmctl service create --name redis --image redis:3.0.508ecg7vc7cbf9k57qs722n2le

List the running services:

$ swarmctl service lsID                         Name   Image        Replicas--                         ----   -----        --------08ecg7vc7cbf9k57qs722n2le  redis  redis:3.0.5  1/1

Inspect the service:

$ swarmctl service inspect redisID                : 08ecg7vc7cbf9k57qs722n2leName              : redisReplicas          : 1/1Template Container  Image           : redis:3.0.5Task ID                      Service    Slot    Image          Desired State    Last State                Node-------                      -------    ----    -----          -------------    ----------                ----0xk1ir8wr85lbs8sqg0ug03vr    redis      1       redis:3.0.5    RUNNING          RUNNING 1 minutes ago    node-1

You can update any attribute of a service.

For example, you can scale the service by changing the instance count:

$ swarmctl service update redis --replicas 608ecg7vc7cbf9k57qs722n2le$ swarmctl service inspect redisID                : 08ecg7vc7cbf9k57qs722n2leName              : redisReplicas          : 6/6Template Container  Image           : redis:3.0.5Task ID                      Service    Slot    Image          Desired State    Last State                Node-------                      -------    ----    -----          -------------    ----------                ----0xk1ir8wr85lbs8sqg0ug03vr    redis      1       redis:3.0.5    RUNNING          RUNNING 3 minutes ago    node-125m48y9fevrnh77til1d09vqq    redis      2       redis:3.0.5    RUNNING          RUNNING 28 seconds ago    node-342vwc8z93c884anjgpkiatnx6    redis      3       redis:3.0.5    RUNNING          RUNNING 28 seconds ago    node-2d41f3wnf9dex3mk6jfqp4tdjw    redis      4       redis:3.0.5    RUNNING          RUNNING 28 seconds ago    node-266lefnooz63met6yfrsk6myvg    redis      5       redis:3.0.5    RUNNING          RUNNING 28 seconds ago    node-13a2sawtoyk19wqhmtuiq7z9pt    redis      6       redis:3.0.5    RUNNING          RUNNING 28 seconds ago    node-3

Changingreplicas from1 to6 forcedSwarmKit to create5 additional Tasks in order tocomply with the desired state.

Every other field can be changed as well, such as image, args, env, ...

Let's change the image fromredis:3.0.5 toredis:3.0.6 (e.g. upgrade):

$ swarmctl service update redis --image redis:3.0.608ecg7vc7cbf9k57qs722n2le$ swarmctl service inspect redisID                   : 08ecg7vc7cbf9k57qs722n2leName                 : redisReplicas             : 6/6Update Status State               : COMPLETED Started             : 3 minutes ago Completed           : 1 minute ago Message             : update completedTemplate Container  Image              : redis:3.0.6Task ID                      Service    Slot    Image          Desired State    Last State              Node-------                      -------    ----    -----          -------------    ----------              ----0udsjss61lmwz52pke5hd107g    redis      1       redis:3.0.6    RUNNING          RUNNING 1 minute ago    node-3b8o394v840thk10tamfqlwztb    redis      2       redis:3.0.6    RUNNING          RUNNING 1 minute ago    node-1efw7j66xqpoj3cn3zjkdrwff7    redis      3       redis:3.0.6    RUNNING          RUNNING 1 minute ago    node-38ajeipzvxucs3776e4z8gemey    redis      4       redis:3.0.6    RUNNING          RUNNING 1 minute ago    node-2f05f2lbqzk9fh4kstwpulygvu    redis      5       redis:3.0.6    RUNNING          RUNNING 1 minute ago    node-27sbpoy82deq7hu3q9cnucfin6    redis      6       redis:3.0.6    RUNNING          RUNNING 1 minute ago    node-1

By default, all tasks are updated at the same time.

This behavior can be changed by defining update options.

For instance, in order to update tasks 2 at a time and wait at least 10 seconds between updates:

$ swarmctl service update redis --image redis:3.0.7 --update-parallelism 2 --update-delay 10s$ watch -n1 "swarmctl service inspect redis"  # watch the update

This will update 2 tasks, wait for them to becomeRUNNING, then wait an additional 10 seconds before moving to other tasks.

Update options can be set at service creation and updated later on. If an update command doesn't specify update options, the last set of options will be used.

SwarmKit monitors node health. In the case of node failures, it re-schedules tasks to other nodes.

An operator can manually define theAvailability of a node and canPause andDrain nodes.

Let's putnode-1 into maintenance mode:

$ swarmctl node drain node-1$ swarmctl node lsID                         Name    Membership  Status  Availability  Manager Status--                         ----    ----------  ------  ------------  --------------3x12fpoi36eujbdkgdnbvbi6r  node-2  ACCEPTED    READY   ACTIVE4spl3tyipofoa2iwqgabsdcve  node-1  ACCEPTED    READY   DRAIN         REACHABLE *dknwk1uqxhnyyujq66ho0h54t  node-3  ACCEPTED    READY   ACTIVE$ swarmctl service inspect redisID                   : 08ecg7vc7cbf9k57qs722n2leName                 : redisReplicas             : 6/6Update Status State               : COMPLETED Started             : 2 minutes ago Completed           : 1 minute ago Message             : update completedTemplate Container  Image              : redis:3.0.7Task ID                      Service    Slot    Image          Desired State    Last State                Node-------                      -------    ----    -----          -------------    ----------                ----8uy2fy8dqbwmlvw5iya802tj0    redis      1       redis:3.0.7    RUNNING          RUNNING 23 seconds ago    node-27h9lgvidypcr7q1k3lfgohb42    redis      2       redis:3.0.7    RUNNING          RUNNING 2 minutes ago     node-3ae4dl0chk3gtwm1100t5yeged    redis      3       redis:3.0.7    RUNNING          RUNNING 23 seconds ago    node-39fz7fxbg0igypstwliyameobs    redis      4       redis:3.0.7    RUNNING          RUNNING 2 minutes ago     node-3drzndxnjz3c8iujdewzaplgr6    redis      5       redis:3.0.7    RUNNING          RUNNING 23 seconds ago    node-27rcgciqhs4239quraw7evttyf    redis      6       redis:3.0.7    RUNNING          RUNNING 2 minutes ago     node-2

As you can see, every Task running onnode-1 was rebalanced to eithernode-2 ornode-3 by the reconciliation loop.