Repository files navigation

Asynchronous parallel SSH client library.

Run SSH commands over many - hundreds/hundreds of thousands - number of servers asynchronously and with minimal system load on the client host.

Native code based clients with extremely high performance, making use of C libraries.

pip install parallel-ssh

An update to pip may be needed to be able to install binary wheels.

pip install -U pippip install parallel-ssh

See documentation onread the docs for more complete examples.

Rununame on two hosts in parallel.

frompssh.clientsimportParallelSSHClienthosts= ['localhost','localhost']client=ParallelSSHClient(hosts)output=client.run_command('uname')forhost_outputinoutput:forlineinhost_output.stdout:print(line)exit_code=host_output.exit_code

LinuxLinux

Single host client with similar API can be used if parallel functionality is not needed.

frompssh.clientsimportSSHClienthost='localhost'cmd='uname'client=SSHClient(host)host_out=client.run_command(cmd)forlineinhost_out.stdout:print(line)exit_code=host_out.exit_code

Github discussions can be used to discuss, ask questions and share ideas regarding the use of parallel-ssh.

The default client inparallel-ssh is a native client based onssh2-python -libssh2 C library - which offers much greater performance and reduced overhead compared to other Python SSH libraries.

Seethis post for a performance comparison of different Python SSH libraries.

Alternative clients based onssh-python (libssh) are also available underpssh.clients.ssh. Seeclient documentation for a feature comparison of the available clients in the library.

parallel-ssh makes use of clients and an event loop solely based on C libraries providing native code levels of performance and stability with an easy to use Python API.

• Highest performance and least overhead of any Python SSH library
• Thread safe - makes use of native threads for CPU bound calls like authentication
• Natively asynchronous utilising C libraries implementing the SSH protocol
• Significantly reduced overhead in CPU and memory usage

Because other options are either immature, unstable, lacking in performance or all of the aforementioned.

Certain other self-proclaimedleading Python SSH libraries leave a lot to be desired from a performance and stability point of view, as well as suffering from a lack of maintenance with hundreds of open issues, unresolved pull requests and inherent design flaws.

The SSH librariesparallel-ssh uses are, on the other hand, long standing mature C libraries inlibssh2 andlibssh that have been in production use for decades and are part of some of the most widely distributed software available today - Git itself, OpenSSH, Curl and many others.

These low level libraries are far better placed to provide the maturity, stability and performance needed from an SSH client for production use.

parallel-ssh provides easy to use SSH clients that hide the complexity, while offering stability and native code levels of performance and as well as the ability to scale to hundreds or more concurrent hosts.

Seealternatives for a more complete comparison of alternative SSH libraries, as well asperformance comparisons mentioned previously.

The client'sjoin function can be used to wait for all commands in output to finish.

Afterjoin returns, commands have finished and all output can be read without blocking.

Onceeither standard output is iterated onto completion, orclient.join() is called, exit codes become available in host output.

Iteration endsonly when remote command has completed, though it may be interrupted and resumed at any point - seejoin and output timeouts documentation.

HostOutput.exit_code is a dynamic property and will returnNone when exit code is not ready, meaning command has not finished, or unavailable due to error.

Once all output has been gathered exit codes become available even without callingjoin as per previous examples.

output=client.run_command('uname')client.join()forhost_outinoutput:forlineinhost_out.stdout:print(line)print(host_out.exit_code)

Linux0Linux0

Similarly, exit codes are available afterclient.join() without reading output.

output=client.run_command('uname')client.join()forhost_outputinoutput:print(host_out.exit_code)

00

There is also a built in host logger that can be enabled to log output from remote hosts for both stdout and stderr. The helper functionpssh.utils.enable_host_logger will enable host logging to stdout.

To log output without having to iterate over output generators, theconsume_output flagmust be enabled - for example:

frompssh.utilsimportenable_host_loggerenable_host_logger()client.run_command('uname')client.join(consume_output=True)

[localhost]       Linux

SCP is supported - native client only - and provides the best performance for file copying.

Unlike with the SFTP functionality, remote files that already exist arenot overwritten and an exception is raised instead.

Note that enabling recursion with SCP requires server SFTP support for creating remote directories.

To copy a local file to remote hosts in parallel with SCP:

frompssh.clientsimportParallelSSHClientfromgeventimportjoinallhosts= ['myhost1','myhost2']client=ParallelSSHClient(hosts)cmds=client.scp_send('../test','test_dir/test')joinall(cmds,raise_error=True)

SeeSFTP and SCP documentation for more examples.

SFTP is supported in the native client.

To copy a local file to remote hosts in parallel:

frompssh.clientsimportParallelSSHClientfrompssh.utilsimportenable_logger,loggerfromgeventimportjoinallenable_logger(logger)hosts= ['myhost1','myhost2']client=ParallelSSHClient(hosts)cmds=client.copy_file('../test','test_dir/test')joinall(cmds,raise_error=True)

Copiedlocalfile ../testtoremotedestinationmyhost1:test_dir/testCopiedlocalfile ../testtoremotedestinationmyhost2:test_dir/test

There is similar capability to copy remote files to local ones with configurable file names via thecopy_remote_file function.

In addition, per-host configurable file name functionality is provided for both SFTP and SCP - seedocumentation.

Directory recursion is supported in both cases via therecurse parameter - defaults to off.

SeeSFTP and SCP documentation for more examples.