Programmatic Private Keys

By default,parallel-ssh will attempt to use loaded keys in an available SSH agent as well as default identity files under the user’s home directory.

SeeIDENTITIES inSSHClient for the list of default identity files.

A private key can also be provided programmatically.

frompssh.clientsimportParallelSSHClientclient=ParallelSSHClient(hosts,pkey="~/.ssh/my_key")

Wheremy_key is a private key file under.ssh in the user’s home directory.

In-Memory Private Keys

Private key data can also be provided as bytes for authentication from in-memory private keys.

frompssh.clientsimportParallelSSHClientpkey_data=b"""-----BEGIN RSA PRIVATE KEY-----<key data>-----END RSA PRIVATE KEY-----"""client=ParallelSSHClient(hosts,pkey=pkey_data)

Private key data provided this waymust be in bytes. This is supported by all parallel and single host clients.

ssh2-python (libssh2)

The default client inparallel-ssh is based onssh2-python (libssh2). It is a native client, offering C level performance with an easy to use Python API.

Seethis post for a performance comparison of the available clients in the1.x.x series.

frompssh.clientsimportParallelSSHClient,SSHClienthosts=['my_host','my_other_host']client=ParallelSSHClient(hosts)output=client.run_command('uname')forhost_outinoutput:forlineinhost_out.stdout:print(line)

New in 1.2.0

ssh-python (libssh) Client

A set of alternative clients based onlibssh viassh-python are also provided.

The API is similar to the default client, whilessh-python offers more supported authentication methods compared to the default client, such as certificate and GSS API authentication.

On the other hand, these clients lack SCP, SFTP and proxy functionality.

frompssh.clients.sshimportParallelSSHClient,SSHClienthosts=['localhost','localhost']client=ParallelSSHClient(hosts)output=client.run_command('uname')client.join(output)forhost_outinoutput:forlineinhost_out.stdout:print(line)

New in 1.12.0

frompssh.clients.sshimportParallelSSHClientclient=ParallelSSHClient(hosts,gssapi_auth=True,gssapi_server_identity='gss_server_id')output=client.run_command('id')client.join(output)forhost_outinoutput:forlineinoutput.stdout:print(line)

frompssh.clients.sshimportParallelSSHClientclient=ParallelSSHClient(hosts,pkey='id_rsa',cert_file='id_rsa-cert.pub')

Per Host Proxy Configuration

Proxy host can be configured in Per-Host Configuration:

hosts=[<..>]host_config=[HostConfig(proxy_host='127.0.0.1'),HostConfig(proxy_host='127.0.0.2'),HostConfig(proxy_host='127.0.0.3'),HostConfig(proxy_host='127.0.0.4'),]client=ParallelSSHClient(hosts,host_config=host_config)output=client.run_command('echo me')

SeeHostConfig for all possible configuration.

Reading Output from Partially Finished Commands

Timeout exception when callingjoin has finished and unfinished commands as arguments.

This can be used to handle sets of commands that have finished and those that have not separately, for example to only gather output on finished commands to avoid blocking.

output=client.run_command(..)try:client.join(output,timeout=5)exceptTimeoutasex:# Some commands timed outfinished_output=ex.args[2]unfinished_output=ex.args[3]else:# No timeout, all commands finished within five secondsfinished_output=outputunfinished_output=Noneforhost_outinfinished_output:forlineinhost_out.stdout:print(line)ifunfinished_outputisnotNone:<handleunfinishedoutput>

In the above example, output is printed only for those commands which have completed within the five second timeout.

Client code may choose to then join again only on the unfinished output if some commands have failed in order to gather remaining output.

Reading Partial Output of Commands That Do Not Terminate

In some cases, such as when the remote command never terminates unless interrupted, it is necessary to use PTY and to close the channel to force the process to be terminated before ajoin sans timeout can complete. For example:

output=client.run_command('while true; do echo a line; sleep .1; done',use_pty=True,read_timeout=1)# Read as many lines of output as hosts have sent before the timeoutstdout=[]forhost_outinoutput:try:forlineinhost_out.stdout:stdout.append(line)exceptTimeout:pass# Closing channel which has PTY has the effect of terminating# any running processes started on that channel.forhost_outinoutput:host_out.client.close_channel(host_out.channel)# Join is not strictly needed here as channel has already been closed and# command has finished, but is safe to use regardless.client.join(output)# Can now read output up to when the channel was closed without blocking.rest_of_stdout=list(output[0].stdout)

Without a PTY, ajoin call with a timeout will complete with timeout exception raised but the remote process will be left running as per SSH protocol specifications.

Run with sudo

parallel-ssh can be instructed to run its commands undersudo:

client=<..>output=client.run_command(<..>,sudo=True)client.join(output)

While not best practice and password-lesssudo is best configured for a limited set of commands, a sudo password may be provided via the stdin channel:

client=<..>output=client.run_command(<..>,sudo=True)forhost_outinoutput:host_out.stdin.write('my_password\n')host_out.stdin.flush()client.join(output)

Run with configurable shell

By default the client will use the login user’s shell to execute commands per the SSH protocol.

Shell to use is configurable:

client=<..>output=client.run_command(<..>,shell='zsh -c')forhost_outinoutput;forlineinhost_out.stdout:print(line)

Commands will be run under thezsh shell in the above example. The command string syntax of the shell must be used, typically<shell>-c.

Output And Command Encoding

By default, command string and output are encoded asUTF-8. This can be configured with theencoding keyword argument torun_command andopen_shell.

client=ParallelSSHClient(<..>)cmd=b"echo\xbc".decode('latin-1')output=client.run_command(cmd,encoding='latin-1')stdout=list(output[0].stdout)

Contents ofstdout arelatin-1 decoded.

cmd string is alsolatin-1 encoded when running command or writing to interactive shell.

Output encoding can also be changed by adjustingHostOutput.encoding.

client=ParallelSSHClient(<..>)output=client.run_command('echo me')output[0].encoding='utf-16'stdout=list(output[0].stdout)

Contents ofstdout areutf-16 decoded.

Enabling use of pseudo terminal emulation

Pseudo Terminal Emulation (PTY) can be enabled when running commands, defaults to off.

Enabling it has some side effects on the output and behaviour of commands such as combining stdout and stderr output - seebash man page for more information.

All output, including stderr, is sent to thestdout channel with PTY enabled.

client=<..>output=client.run_command("echo 'asdf' >&2",use_pty=True)forlineinoutput[0].stdout:print(line)

Note output is from thestdout channel while it was written tostderr.

Output:

asdf

Stderr is empty:

forlineinoutput[0].stderr:print(line)

No output fromstderr.

Copying files to remote hosts in parallel

To copy the local file with relative path../test to the remote relative pathtest_dir/test - remote directory will be created if it does not exist, permissions allowing.raise_error=True instructsjoinall to raise any exceptions thrown by the greenlets.

frompssh.clientsimportParallelSSHClientfromgeventimportjoinallclient=ParallelSSHClient(hosts)cmds=client.copy_file('../test','test_dir/test')joinall(cmds,raise_error=True)

To recursively copy directory structures, enable therecurse flag:

cmds=client.copy_file('my_dir','my_dir',recurse=True)joinall(cmds,raise_error=True)

Copying files from remote hosts in parallel

Copying remote files in parallel requires that file names are de-duplicated otherwise they will overwrite each other.copy_remote_file names local files as<local_file><suffix_separator><host>, suffixing each file with the host name it came from, separated by a configurable character or string.

frompssh.pssh_clientimportParallelSSHClientfromgeventimportjoinallclient=ParallelSSHClient(hosts)cmds=client.copy_remote_file('remote.file','local.file')joinall(cmds,raise_error=True)

The above will create fileslocal.file_host1 wherehost1 is the host name the file was copied from.

Configurable per host Filenames

File name arguments, for both local and remote files and for copying to and from remote hosts, can be configured on a per-host basis similarly tohost arguments inrun_command.

Example shown applies to all file copy functionality, all ofscp_send,scp_recv,copy_file andcopy_remote_file.

For example, to copy the local files['local_file_1','local_file_2'] as remote files['remote_file_1','remote_file_2'] on the two hosts['host1','host2']

hosts=['host1','host2']client=ParallelSSHClient(hosts)copy_args=[{'local_file':'local_file_1','remote_file':'remote_file_1',},{'local_file':'local_file_2','remote_file':'remote_file_2',}]cmds=client.copy_file('%(local_file)s','%(remote_file)s',copy_args=copy_args)joinall(cmds)

The client will copylocal_file_1 tohost1 asremote_file_1 andlocal_file_2 tohost2 asremote_file_2.

Each item incopy_args list should be a dictionary as shown above. Number of items incopy_args must match length ofclient.hosts if provided or exception will be raised.

copy_remote_file,scp_send andscp_recv may all be used in the same manner to configure remote and local file names per host.

Single host copy

If wanting to copy a file from a single remote host and retain the original filename, can use the single hostSSHClient and itscopy_remote_file directly.

frompssh.clientsimportSSHClientclient=SSHClient('localhost')client.copy_remote_file('remote_filename','local_filename')client.scp_recv('remote_filename','local_filename')

Running Commands On Shells

Command to run can be multi-line, a single command or a list of commands.

Shells provided are used for all commands, reusing the channel opened byopen_shell.

client=ParallelSSHClient(<..>)cmd="""echo meecho me too"""shells=client.open_shell()client.run_shell_commands(shells,cmd)

cmd='echo me three'client.run_shell_commands(shells,cmd)cmd=['echo me also','echo and as well me','exit 1']client.run_shell_commands(shells,cmd)

Waiting For Completion

Joining shells waits for running commands to complete and closes shells.

This allows output to be read up to the last command executed without blocking.

client.join_shells(shells)

Joined on shells are closed and may not run any further commands.

Trying to use the same shells afterjoin_shells will raisepssh.exceptions.ShellError.

Reading Shell Output

Output for each shell includes all commands executed.

forshellinshells:stdout=list(shell.stdout)exit_code=shell.exit_code

Exit code is for thelast executed command only and can be retrieved whenrun_shell_commands has been used at least once.

Each shell also has ashell.output which is aHostOutput object.shell.stdout et al are the same asshell.output.stdout.

shells=client.open_shell(read_timeout=1)client.run_shell_commands(shells,['echo me'])# Times out after one secondforlineinshells[0].stdout:print(line)

Join Timeouts

Timeouts onjoin_shells can be done similarly tojoin.

cmds=["echo me","sleep 1.2"]shells=client.open_shell()client.run_shell_commands(shells,cmds)client.join_shells(shells,timeout=1)

Single Clients

On single clients shells can be used as a context manager to join and close the shell on exit.

client=SSHClient(<..>)cmd='echo me'withclient.open_shell()asshell:shell.run(cmd)print(list(shell.stdout))print(shell.exit_code)

Or explicitly:

cmd='echo me'shell=client.open_shell()shell.run(cmd)shell.close()

Closing a shell also waits for commands to complete.

Iterators and filtering

Any type of iterator may be used as hosts list, including generator and list comprehension expressions.

List comprehension:

hosts=['dc1.myhost1','dc2.myhost2']client=ParallelSSHClient([hforhinhostsifh.find('dc1')])

Generator:

hosts=['dc1.myhost1','dc2.myhost2']client=ParallelSSHClient((hforhinhostsifh.find('dc1')))

Filter:

hosts=['dc1.myhost1','dc2.myhost2']client=ParallelSSHClient(filter(lambdah:h.find('dc1'),hosts))client.run_command(<..>)

Overriding hosts list

Hosts list can be modified in place.

A call torun_command will create new connections as necessary and output will only be returned for the hostsrun_command executed on.

Clients for hosts that are no longer on the host list are removed on host list assignment. Reading output from hosts removed from host list is feasible, as long as their output objects or interactive shells are in scope.

client=<..>client.hosts=['otherhost']print(client.run_command('exit 0'))<..>host='otherhost'exit_code=None<..>

When reassigning host list frequently, it is best to sort or otherwise ensure order is maintained to avoid reconnections on hosts that are still in the host list but in a different position.

For example, the following will cause reconnections on both hosts, though both are still in the list.

client.hosts=['host1','host2']client.hosts=['host2','host1']

In such cases it would be best to maintain order to avoid reconnections. This is also true when adding or removing hosts in host list.

No change in clients occurs in the following case.

client.hosts=sorted(['host1','host2'])client.hosts=sorted(['host2','host1'])

Clients for hosts that would be removed by a reassignment can be calculated with:

set(enumerate(client.hosts)).difference(set(enumerate(new_hosts)))