- Notifications
You must be signed in to change notification settings - Fork1
License
NotificationsYou must be signed in to change notification settings
miniwdl-ext/miniwdl-slurm
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Extends miniwdl to run workflows on SLURM clusters in singularity containers.
ThisSLURM backend plugin forminiwdl runs WDL task containersby creating a job script that is submitted to a SLURM cluster. In case the jobdescription has a container, singularity will be used as container runtime.
For the latest stable version:
pip install miniwdl-slurm
For the development version:
pip install git+https://github.com/miniwdl-ext/miniwdl-slurm.git
The followingminiwdl configurationexample can be used to use miniwdl on a SLURM cluster:
[scheduler]container_backend=slurm_singularity# task_concurrency defaults to the number of processors on the system.# since we submit the jobs to SLURM this is not necessary.# higher numbers means miniwdl has to monitor more processes simultaneously# which might impact performance.task_concurrency=200# This setting allows running tasks to continue, even if one other tasks fails.# Useful in combination with call caching. Prevents wasting resources by# cancelling jobs half-way that would probably succeed.fail_fast = false[call_cache]# The following settings create a call cache under the current directory.# This prevents wasting unnecessary resources on the cluster by rerunning# jobs that have already succeeded.put = trueget = truedir ="$PWD/miniwdl_call_cache"[task_runtime]# Setting a 'maxRetries' default allows jobs that fail due to intermittent# errors on the cluster to be retried.defaults = {"maxRetries": 2,"docker":"ubuntu:20.04" }[singularity]# This plugin wraps the singularity backend. Make sure the settings are# appropriate for your cluster.exe = ["singularity"]# the miniwdl default options contain options to run as a fake root, which# is not available on most clusters.run_options = ["--containall" ]# Location of the singularity images (optional). The miniwdl-slurm plugin# will set it to a directory inside $PWD. This location must be reachable# for the submit nodes.image_cache ="$PWD/miniwdl_singularity_cache"[slurm]# extra arguments passed to the srun command (optional).extra_args="--partition heavy_users,gpu --comment 'run with miniwdl'"
About
No description, website, or topics provided.
Resources
License
Stars
Watchers
Forks
Packages0
No packages published