Repository files navigation

Package website:release |dev

As a successor of the packagesBatchJobs andBatchExperiments,batchtools provides a parallel implementation of Map for highperformance computing systems managed by schedulers like Slurm, Sun GridEngine, OpenLava, TORQUE/OpenPBS, Load Sharing Facility (LSF) or DockerSwarm (see the setup section in thevignette).

Main features:

• Convenience: All relevant batch system operations (submitting,listing, killing) are either handled internally or abstracted viasimple R functions
• Portability: With a well-defined interface, the source is independentfrom the underlying batch system - prototype locally, deploy on anyhigh performance cluster
• Reproducibility: Every computational part has an associated seedstored in a data base which ensures reproducibility even when theunderlying batch system changes
• Abstraction: The code layers for algorithms, experiment definitionsand execution are cleanly separated and allow to write readable andmaintainable code to manage large scale computer experiments

Install the stable version from CRAN:

install.packages("batchtools")

For the development version, usedevtools:

devtools::install_github("mlr-org/batchtools")

Next, you need to setupbatchtools for your HPC (it will runsequentially otherwise). See thevignette forinstructions.

The development ofBatchJobs andBatchExperiments isdiscontinued for the following reasons:

• Maintainability: The packagesBatchJobs andBatchExperiments aretightly connected which makes maintenance difficult. Changes have tobe synchronized and tested against the current CRAN versions forcompatibility. Furthermore, BatchExperiments violates CRAN policies bycalling internal functions of BatchJobs.
• Data base issues: Although we invested weeks to mitigate issues withlocks of the SQLite data base or file system (staged queries, filesystem timeouts, …),BatchJobs kept working unreliable on somesystems with high latency under certain conditions. This madeBatchJobs unusable for many users.

BatchJobs andBatchExperiments willremain on CRAN, but new features are unlikely to be ported back. Thevignettecontains a section comparing the packages.

• Function reference
• Vignette
• JOSS Paper: Short paper onbatchtools. Please cite this if you use batchtools.
• Paper onBatchJobs/BatchExperiments: Thedescribed concept still holds for batchtools and most examples workanalogously (see thevignettefor differences between the packages).

Please cite theJOSS paper usingthe following BibTeX entry:

@article{,  doi = {10.21105/joss.00135},  url = {https://doi.org/10.21105/joss.00135},  year  = {2017},  month = {feb},  publisher = {The Open Journal},  volume = {2},  number = {10},  author = {Michel Lang and Bernd Bischl and Dirk Surmann},  title = {batchtools: Tools for R to work on batch systems},  journal = {The Journal of Open Source Software}}

• TheHigh Performance Computing TaskView liststhe most relevant packages for scientific computing with R.
• clustermq is a similarapproach which also supports multiple schedulers. Uses the ZeroMQnetwork protocol for communication, and shines if you have millions offast jobs.
• batch assists in splittingand submitting jobs to LSF and MOSIX clusters.
• flowr supports LSF, Slurm,TORQUE and Moab and provides a scatter-gather approach to definecomputational jobs.
• future.batchtoolsimplementsbatchtools as backend forfuture.
• doFuture together withfuture.batchtoolsconnectsbatchtools toforeach.
• drake uses graphs todefine computational jobs.batchtools is used as a backend viafuture.batchtools.

This R package is licensed under theLGPL-3. If youencounter problems using this software (lack of documentation,misleading or wrong documentation, unexpected behaviour, bugs, …) orjust want to suggest features, please open an issue in theissuetracker. Pull requestsare welcome and will be included at the discretion of the author. If youhave customized a template file for your (larger) computing site, pleaseshare it: fork the repository, place your template ininst/templatesand send a pull request.