Installation

conda

conda create --name ec2-setup-sos --channel conda-forge python=3 pyyaml boto3 sos blackgit clone https://github.com/lucasrla/ec2-setup-sos-workflowcd ec2-setup-sos-workflowconda activate ec2-setup-sos

poetry or pip

git clone https://github.com/lucasrla/ec2-setup-sos-workflowcd ec2-setup-sos-workflow# create and activate a virtualenv, for example:pyenv virtualenv ec2-setup-sos&& pyenvlocal ec2-setup-sos# install the dependencies, either with:poetry install# or:pip install -r requirements.txt# note, requirements.txt in this repository were generated via:# poetry export --without-hashes -f requirements.txt -o requirements.txt

Configuration

# edit config.TEMPLATE.yml to match your needsvim config.TEMPLATE.yml# and then save it as config.yml# edit hosts.TEMPLATE.yml to match your needsvim hosts.TEMPLATE.yml# save it as ~/.sos/hosts.yml# make sure your ~/.aws/config is properly set# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.htmlcat~/.aws/config# make sure the [default] is correct within ~/.aws/credentials# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.htmlcat~/.aws/credentials

Usage

Init

sos run -c config.yml init.sos -v4# this will:# launch a new ec2 instance, create a new ebs volume, attach the volume to the instance,# add the instance's public dns/ip to your local ssh known_hosts, update the yum packages,# install conda and sos, format the ebs volume, and mount it to the instance

Your pipeline/workflow

Teardown

sos run -c config.yml teardown.sos -v4# this will undo the steps done with `init.sos`:# umount the ebs volume (device), detach it, terminate the instance, delete metadata files,# optionally save a snapshot of the ebs volume, and delete the volume

Troubleshooting

# lists all local tasks# tasks live in ~/.sos/tasks/sos status# check the details of a specific task/job# -v4 (max verbosity level) is great for debuggingsos status TASK_HASH_ID -v4# for example:sos status c23ad54f11df17b6    c23ad54f11df17b65214df0f1e51390c ebs_umount ec2-setup-sos-workflowRanfor 0s    failedsos status c23ad54f11df17b6 -v4    ...# check the connection to the ec2 instance# which is also named ec2 in our config YAMLsos remotetest ec2 -v4# if substeps are being skipped because of saved signatures# remove these signatures with:sos remove -s# more information on signatures at https://vatlab.github.io/sos-docs/doc/user_guide/signature.html# eventually, you may also want to clean up ~/.sos/tasks/rm -R~/.sos/tasks/

License