synctool © 2024 Walter de Jong <walter@heiho.net>
synctool comes withno warranty. synctool isfree software.
synctool is distributed under terms describedin theGNU General PublicLicense.
In synctool terminology, anode is a host, a computer in a groupof computers. A group of computers is called acluster.
synctool depends on a number of (fairly standard) programs:
ping, or you can configurefping laterIf you got all that, it’s on to the next section.
synctool requires passwordless SSH from the master node to each cluster nodeas root. If you need more information on how to set this up, please see theSSH documentation or just google around. I like to give you these tips:
PermitRootLogin without-password insshd_config on the nodesssh-keyscan to create/etc/ssh/ssh_known_hostssshd only the internal network interface to secure your system;configureListenAddress appropriatelyIf you want extra security, use a passphrase on the keypair and employssh-agent. Usessh-add with a timeout.For sites with extra tight security, it is possible to configuressh torun only specific (synctool) commands, or maybe you want to changethessh_cmd in synctool’s configuration so that it runs a different command,one that does suit your security needs.
When passwordless SSH as root works, proceed to installing the software.
To install synctool on the master node, runsetup.sh like so:
# ./setup.sh --installdir=/opt/synctoolThe default location is/opt/synctool, which is a good place to put it.Note that synctool requires an ‘installdir’ directory of its own. Theinstalldir is not the same as a prefix; whatever you do, donot installsynctool directly under/usr or/usr/local. Use/usr/synctool or/usr/local/synctool instead, or better, stick with the default location.The rest of the documentation assumes the default/opt/synctool.
setup.sh creates the following directory structure:
/opt/synctool/bin/ synctool commands/opt/synctool/sbin/ 'system' programs/opt/synctool/etc/ configuration files/opt/synctool/lib/ libraries, modules/opt/synctool/lib/synctool//opt/synctool/lib/synctool/main//opt/synctool/lib/synctool/pkg//opt/synctool/doc/ documentation/opt/synctool/scripts/ place to store your scripts/opt/synctool/var/ repository directory/opt/synctool/var/overlay//opt/synctool/var/delete//opt/synctool/var/purge/Thedoc/ directory contains a copy of this documentation.You may build the HTML documentation from the plain text sourcesby runningsetup.sh with--build-docs.
The following synctool commands will be made available in/opt/synctool/bin/:
synctool Main commanddsh Run remote commandsdsh-pkg Upgrade or install packagesdsh-ping Check whether nodes are updsh-cp Copy files to nodessynctool-client Only run on target nodessynctool-client-pkg Only run on target nodessynctool-config Inspect the configurationsynctool-template Useful command for .post scriptsTip: Add
/opt/synctool/binto yourPATH.
Copy thesynctool.conf.example file to/opt/synctool/etc/synctool.conf.Editsynctool.conf, adjusting it as needed.
The filesynctool.conf describes what your cluster looks like;what nodes have what roles, and how synctool can contact them.Think a bit about what role each machine has. There is no need to go intogreat depth right now; you can always adjust the configuration later.
node n1 group1 group2 ipaddress:machine-n01The nodename is the ‘synctool name that the node has.’ It is in general theshort hostname of the host, but in fact it can be anything you like.The nodename has nothing to do with hostnames or DNS entries.Theipaddress specifier tells synctool how to contact the node; this can bean IP address or a DNS name of the host you wish to contact. In clusters,there is often a management network interface — configure its IP addresshere. Theipaddress specifier isoptional and only needed if the nodenamedoes not exactly match the DNS name for contacting the remote host.
Directly following the node name, you can list groups. synctool uses theterm ‘group’, but you can also think of them as node properties. You can makeup as many different properties as you like. You can split long lines byending them with a backslash:
node n101 workernode plasma mathworks solar \ fsmounted backup debian ipaddress:if0-n101Mind that in practice, synctool repositories are generally easiestmaintainable with as few groups as possible. Make sure to uselogical names for logical groups, and use a top-down group structure.Make things easy on yourself.
If you have many nodes that all share the same long list of groups, thegroups may be abbreviated by defining acompound group. This compoundgroup must be defined before defining the nodes:
group wn workernode plasma mathworks solar \ fsmounted backupnode n101 wn debian ipaddress:if0-n101You have to add a node definition for each and every node in your cluster.If your nodes are neatly numbered (and for large clusters, they often are),you can make use of node ranges and IP address sequences, like so:
node n[001-100] wn debian ipaddress:if0-n[001]node n[101-200] wn debian ipaddress:192.168.1.[20]If you do have the luxury of a high performance shared filesystem on yourcluster, you may put/opt/synctool/ on there and addrsync:no to the nodedefinition lines in the config file to tell synctool not to runrsync.Mind that there are certain security implications with having a sharedfilesystem between management and production nodes.
Next, you have to tell synctool which node is the master management node.This is done by settingmaster to the fqdn (fully qualified domain name)of the management host.
master n1.mycluster.orgIf you don’t know what the fqdn is, you can get it by running the command:
synctool-config --fqdnIf you want to manage the master node itself with synctool, you should alsodefine it as a node. It is a matter of taste, but it is maybe betternotto do so. If you choose not to manage the master node, it may be omittedfrom the configuration. You may also explicitly exclude it:
node n1 master hostname:n1.mycluster.orgignore_node n1Beside a master node, you may also define slave nodes.Slaves are cold standby’s that get full copies of the synctool repository.A slave may be used as a failback in case your management host breaks down.Since there can be only one master node in a synctool cluster, slaves mustbe enabled ‘by hand’ by editing the config file and changing the masterdefinition.
Previous versions of synctool had a
masterdirsetting.It no longer exists; the overlay directory now must reside underthe synctool root, under/opt/synctool/var/.
You can test yoursynctool.conf with the commandsynctool-config.It’s more exciting however to test withdsh and actually run commandson the cluster.
After filling in a couple of nodes insynctool.conf, try the commanddsh-ping to see if the nodes are ‘up’. If they are, try running thecommandsdsh hostname,dsh uptime, ordsh date.If you correctly set up passwordless SSH,dsh should run the commands onevery node without problems or needed manual intervention. It is importantthat this works before proceeding.
Some (mostly IBM) systems already have a
dshcommand.Be mindful to start the correctdshcommand.
See section 3.15 for a trick that greatly speeds up synctool and dsh usingOpenSSH’s multiplexed connections capability.
Now that you have a rough setup on the master node, try runningsynctoolto a single node:
synctool -n nodenameThere should be some output message sayingDRY RUN.This means that synctool is now working. You can try running synctool acrossall nodes:
synctoolCheck that every node responds. If it doesn’t, go back to the step wherewe tested the setup usingdsh.When synctool to every node works, the basic setup is done and you can startfilling your repository with useful files.
As you may have noticed, we never installed any client software on the nodes.There is no client installation step; the master node automaticallyupdates synctool on the client nodes. The binaries needed for this arelocated under/opt/synctool/sbin/, and this directory gets synced to thenodes withrsync every time you runsynctool.