Repository files navigation

Please note: spark-ec2 isno longer under active development and the project has been archived. All the existing code, PRs and issues are still accessible but are now read-only. If you're looking for a similar tool that is under active development, we recommend you take a look atFlintrock.

spark-ec2 allows youto launch, manage and shut downApache Spark [1] clusterson Amazon EC2. It automatically sets up Apache Spark andHDFSon the cluster for you. This guide describeshow to usespark-ec2 to launch clusters, how to run jobs on them, and howto shut them down. It assumes you've already signed up for an EC2 accounton theAmazon Web Services site.

spark-ec2 is designed to manage multiple named clusters. You canlaunch a new cluster (telling the script its size and giving it a name),shutdown an existing cluster, or log into a cluster. Each cluster isidentified by placing its machines into EC2 security groups whose namesare derived from the name of the cluster. For example, a cluster namedtest will contain a master node in a security group calledtest-master, and a number of slave nodes in a security group calledtest-slaves. Thespark-ec2 script will create these security groupsfor you based on the cluster name you request. You can also use them toidentify machines belonging to each cluster in the Amazon EC2 Console.

[1] Apache,Apache Spark, and Spark are trademarks of the Apache Software Foundation.

• Create an Amazon EC2 key pair for yourself. This can be done bylogging into your Amazon Web Services account through theAWSconsole, clicking Key Pairs on theleft sidebar, and creating and downloading a key. Make sure that youset the permissions for the private key file to600 (i.e. only youcan read and write it) so thatssh will work.
• Whenever you want to use thespark-ec2 script, set the environmentvariablesAWS_ACCESS_KEY_ID andAWS_SECRET_ACCESS_KEY to yourAmazon EC2 access key ID and secret access key. These can beobtained from theAWS homepage by clickingAccount > Security Credentials > Access Credentials.

• Go into theec2 directory in the release of Apache Spark you downloaded.

• Run./spark-ec2 -k <keypair> -i <key-file> -s <num-slaves> launch <cluster-name>,where<keypair> is the name of your EC2 key pair (that you gave itwhen you created it),<key-file> is the private key file for yourkey pair,<num-slaves> is the number of slave nodes to launch (try1 at first), and<cluster-name> is the name to give to yourcluster.

  For example:

  export AWS_SECRET_ACCESS_KEY=AaBbCcDdEeFGgHhIiJjKkLlMmNnOoPpQqRrSsTtUexport AWS_ACCESS_KEY_ID=ABCDEFG1234567890123./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a launch my-spark-cluster

• After everything launches, check that the cluster scheduler is up and seesall the slaves by going to its web UI, which will be printed at the end ofthe script (typicallyhttp://<master-hostname>:8080).

You can also run./spark-ec2 --help to see more usage options. Thefollowing options are worth pointing out:

• --instance-type=<instance-type> can be used to specify an EC2instance type to use. For now, the script only supports 64-bit instancetypes, and the default type ism3.large (which has 2 cores and 7.5 GBRAM). Refer to the Amazon pages aboutEC2 instancetypes andEC2pricing for information about otherinstance types.
• --region=<ec2-region> specifies an EC2 region in which to launchinstances. The default region isus-east-1.
• --zone=<ec2-zone> can be used to specify an EC2 availability zoneto launch instances in. Sometimes, you will get an error because thereis not enough capacity in one zone, and you should try to launch inanother.
• --ebs-vol-size=<GB> will attach an EBS volume with a given amountof space to each node so that you can have a persistent HDFS clusteron your nodes across cluster restarts (see below).
• --spot-price=<price> will launch the worker nodes asSpot Instances,bidding for the given maximum price (in dollars).
• --spark-version=<version> will pre-load the cluster with thespecified version of Spark. The<version> can be a version number(e.g. "0.7.3") or a specific git hash. By default, a recentversion will be used.
• --spark-git-repo=<repository url> will let you run a custom version ofSpark that is built from the given git repository. By default, theApache Github mirror will be used.When using a custom Spark version,--spark-version must be set to gitcommit hash, such as 317e114, instead of a version number.
• If one of your launches fails due to e.g. not having the rightpermissions on your private key file, you can runlaunch with the--resume option to restart the setup process on an existing cluster.

• Run./spark-ec2 -k <keypair> -i <key-file> -s <num-slaves> --vpc-id=<vpc-id> --subnet-id=<subnet-id> launch <cluster-name>,where<keypair> is the name of your EC2 key pair (that you gave itwhen you created it),<key-file> is the private key file for yourkey pair,<num-slaves> is the number of slave nodes to launch (try1 at first),<vpc-id> is the name of your VPC,<subnet-id> is thename of your subnet, and<cluster-name> is the name to give to yourcluster.

  For example:

  export AWS_SECRET_ACCESS_KEY=AaBbCcDdEeFGgHhIiJjKkLlMmNnOoPpQqRrSsTtUexport AWS_ACCESS_KEY_ID=ABCDEFG1234567890123./spark-ec2 --key-pair=awskey --identity-file=awskey.pem --region=us-west-1 --zone=us-west-1a --vpc-id=vpc-a28d24c7 --subnet-id=subnet-4eb27b39 --spark-version=1.1.0 launch my-spark-cluster

• Go into theec2 directory in the release of Spark you downloaded.
• Run./spark-ec2 -k <keypair> -i <key-file> login <cluster-name> toSSH into the cluster, where<keypair> and<key-file> are asabove. (This is just for convenience; you could also usethe EC2 console.)
• To deploy code or data within your cluster, you can log in and use theprovided script~/spark-ec2/copy-dir, which,given a directory path, RSYNCs it to the same location on all the slaves.
• If your application needs to access large datasets, the fastest way to dothat is to load them from Amazon S3 or an Amazon EBS device into aninstance of the Hadoop Distributed File System (HDFS) on your nodes.Thespark-ec2 script already sets up a HDFS instance for you. It'sinstalled in/root/ephemeral-hdfs, and can be accessed using thebin/hadoop script in that directory. Note that the data in thisHDFS goes away when you stop and restart a machine.
• There is also apersistent HDFS instance in/root/persistent-hdfs that will keep data across cluster restarts.Typically each node has relatively little space of persistent data(about 3 GB), but you can use the--ebs-vol-size option tospark-ec2 to attach a persistent EBS volume to each node forstoring the persistent HDFS.
• Finally, if you get errors while running your application, look at the slave's logsfor that application inside of the scheduler work directory (/root/spark/work). You canalso view the status of the cluster using the web UI:http://<master-hostname>:8080.

You can edit/root/spark/conf/spark-env.sh on each machine to set Spark configuration options, suchas JVM options. This file needs to be copied toevery machine to reflect the change. The easiest way todo this is to use a script we provide calledcopy-dir. First edit yourspark-env.sh file on the master,then run~/spark-ec2/copy-dir /root/spark/conf to RSYNC it to all the workers.

Theconfiguration guide describes the available configuration options.

Note that there is no way to recover data on EC2 nodes after shuttingthem down! Make sure you have copied everything important off the nodesbefore stopping them.

• Go into theec2 directory in the release of Spark you downloaded.
• Run./spark-ec2 destroy <cluster-name>.

Thespark-ec2 script also supports pausing a cluster. In this case,the VMs are stopped but not terminated, so theylose all data on ephemeral disks but keep the data in theirroot partitions and theirpersistent-hdfs. Stopped machines will notcost you any EC2 cycles, butwill continue to cost money for EBSstorage.

• To stop one of your clusters, go into theec2 directory and run./spark-ec2 --region=<ec2-region> stop <cluster-name>.
• To restart it later, run./spark-ec2 -i <key-file> --region=<ec2-region> start <cluster-name>.
• To ultimately destroy the cluster and stop consuming EBS space, run./spark-ec2 --region=<ec2-region> destroy <cluster-name> as described in the previoussection.

• Support for "cluster compute" nodes is limited -- there's no way to specify alocality group. However, you can launch slave nodes in your<clusterName>-slaves group manually and then usespark-ec2 launch --resume to start a cluster with them.

If you have a patch or suggestion for one of these limitations, feel free tocontribute it!

Spark's file interface allows it to process data in Amazon S3 using the same URI formats that are supported for Hadoop. You can specify a path in S3 as input through a URI of the forms3n://<bucket>/path. To provide AWS credentials for S3 access, launch the Spark cluster with the option--copy-aws-credentials. Full instructions on S3 access using the Hadoop input libraries can be found on theHadoop S3 page.

In addition to using a single input file, you can also use a directory of files as input by simply giving the path to the directory.

This repository contains the set of scripts used to setup a Spark cluster onEC2. These scripts are intended to be used by the default Spark AMI and isnotexpected to work on other AMIs. If you wish to start a cluster using Spark,please refer tohttp://spark-project.org/docs/latest/ec2-scripts.html

The Spark cluster setup is guided by the values set inec2-variables.sh.setup.shfirst performs basic operations like enabling ssh across machines, mounting ephemeraldrives and also creates files named/root/spark-ec2/masters, and/root/spark-ec2/slaves.Following that every module listed inMODULES is initialized.

To add a new module, you will need to do the following:

1. Create a directory with the module's name.

2. Optionally add a file namedinit.sh. This is called before templates are configuredand can be used to install any pre-requisites.

3. Add any files that need to be configured based on the cluster setup totemplates/.The path of the file determines where the configured file will be copied to. Right nowthe set of variables that can be used in a template are:

   {{master_list}}{{active_master}}{{slave_list}}{{zoo_list}}{{cluster_url}}{{hdfs_data_dirs}}{{mapred_local_dirs}}{{spark_local_dirs}}{{spark_worker_mem}}{{spark_worker_instances}}{{spark_worker_cores}}{{spark_master_opts}}

You can add new variables by modifyingdeploy_templates.py.

4. Add a file namedsetup.sh to launch any services on the master/slaves. This is calledafter the templates have been configured. You can use the environment variables$SLAVES toget a list of slave hostnames and/root/spark-ec2/copy-dir to sync a directory across machines.

5. Modifyspark_ec2.py to add your module to the list of enabled modules.