Repository files navigation

StarCluster is a utility for creating and managing computing clusters hosted onAmazon's Elastic Compute Cloud (EC2). StarCluster utilizes Amazon's EC2 webservice to create and destroy clusters of Linux virtual machines on demand.

All that's needed to create your own cluster(s) on Amazon EC2 is an AWS accountand StarCluster. StarCluster features:

• Simple configuration - with examples ready to go out-of-the-box
• Create/Manage Clusters - simplestart command to automatically launchand configure one or more clusters on EC2
• Automated Cluster Setup - includes NFS-sharing, Open Grid Schedulerqueuing system, Condor, password-less ssh between machines, and more
• Scientific Computing AMI - comes with Ubuntu 11.10-based EBS-backed AMIthat contains Hadoop, OpenMPI, ATLAS, LAPACK, NumPy, SciPy, IPython, andother useful libraries
• EBS Volume Sharing - easily NFS-share Amazon Elastic Block Storage (EBS)volumes across a cluster for persistent storage
• EBS-Backed Clusters - start and stop EBS-backed clusters on EC2
• Cluster Compute Instances - support for "cluster compute" instance types
• Expand/Shrink Clusters - scale a cluster by adding or removing nodes
• Elastic Load Balancing - automatically shrink or expand a cluster basedon Open Grid Scheduler queue statistics
• Plugin Support - allows users to run additional setup routines on thecluster after StarCluster's defaults. Comes with plugins for IPythonparallel+notebook, Condor, Hadoop, MPICH2, MySQL cluster, installing Ubuntupackages, and more.

Interested? See thegetting started section for more details.

Install StarCluster using easy_install:

$ easy_install StarCluster

or using pip:

$ pip install StarCluster

or manually:

$ (Download StarCluster from http://star.mit.edu/cluster/downloads.html)$ tar xvzf starcluster-X.X.X.tar.gz  (where x.x.x is a version number)$ cd starcluster-X.X.X$ sudo python setup.py install

After the software has been installed, the next step is to setup theconfiguration file:

$ starcluster helpStarCluster - (http://star.mit.edu/cluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to starcluster@mit.edu!!! ERROR - config file /home/user/.starcluster/config does not existOptions:--------[1] Show the StarCluster config template[2] Write config template to /home/user/.starcluster/config[q] QuitPlease enter your selection:

Select the second option by typing2 and pressing enter. This will give you atemplate to use to create a configuration file containing your AWS credentials,cluster settings, etc. The next step is to customize this file using yourfavorite text-editor:

$ vi ~/.starcluster/config

This file is commented with example "cluster templates". A cluster templatedefines a set of configuration settings used to start a new cluster. Theexample config provides asmallcluster template that is ready to goout-of-the-box. However, first, you must fill in your AWS credentials andkeypair info:

[aws info]aws_access_key_id = #your aws access key id hereaws_secret_access_key = #your secret aws access key hereaws_user_id = #your 12-digit aws user id here

The next step is to fill in your keypair information. If you don't already havea keypair you can create one from StarCluster using:

$ starcluster createkey mykey -o ~/.ssh/mykey.rsa

This will create a keypair calledmykey on Amazon EC2 and save the privatekey to ~/.ssh/mykey.rsa. Once you have a key the next step is to fill-in yourkeypair info in the StarCluster config file:

[key key-name-here]key_location = /path/to/your/keypair.rsa

For example, the section for the keypair created above using thecreatekeycommand would look like:

[key mykey]key_location = ~/.ssh/mykey.rsa

After defining your keypair in the config, the next step is to update thedefault cluster templatesmallcluster with the name of your keypair on EC2:

[cluster smallcluster]keyname = key-name-here

For example, thesmallcluster template would be updated to look like:

[cluster smallcluster]keyname = mykey

Now that the config file has been set up we're ready to start usingStarCluster. Next we start a cluster named "mycluster" using the defaultcluster templatesmallcluster in the example config:

$ starcluster start mycluster

Thedefault_template setting in the[global] section of the configspecifies the default cluster template and is automatically set tosmallcluster in the example config.

After thestart command completes you should now have a working cluster.You can login to the master node as root by running:

$ starcluster sshmaster mycluster

You can also copy files to/from the cluster using theput andgetcommands. To copy a file or entire directory from your local computer to thecluster:

$ starcluster put mycluster /path/to/local/file/or/dir /remote/path/

To copy a file or an entire directory from the cluster to your local computer:

$ starcluster get mycluster /path/to/remote/file/or/dir /local/path/

Once you've finished using the cluster and wish to stop paying for it:

$ starcluster terminate mycluster

Have a look at the rest of StarCluster's available commands:

$ starcluster --help

• Amazon AWS Account
• Python 2.6+
• Boto 2.23.0+
• Paramiko 1.12.1+
• WorkerPool 0.9.2
• Jinja2 2.7
• decorator 3.4.0+
• iptools 0.6.1+
• optcomplete 1.2-devel+
• PyCrypto 2.5+
• scp 0.7.1+
• iso8601 0.1.8+

Watch an ~8 minute screencast @http://star.mit.edu/cluster

To learn more have a look at the documentation:http://star.mit.edu/cluster/docs/latest

StarCluster has a mailing list for users and developers:

http://star.mit.edu/cluster/mailinglist.html

Join our IRC channel #starcluster on freenode. If you do not have an IRC clientyou can join the #starcluster channel using your web browser:

http://webchat.freenode.net/?channels=starcluster

StarCluster is licensed under the LGPLv3See COPYING.LESSER (LGPL) and COPYING (GPL) for LICENSE details