- Notifications
You must be signed in to change notification settings - Fork3
mlr-org/mlr3batchmark
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A connector betweenmlr3 andbatchtools. This allows to runlarge-scale benchmark experiments on scheduled high-performancecomputing clusters.
The package comes with two core functions for switching betweenmlr3andbatchtools to perform a benchmark:
- After creating a
designobject (as required formlr3’sbenchmark()function), instead ofbenchmark()callbatchmark()which populates anExperimentRegistryfor the computational jobsof the benchmark. You are now in the world ofbatchtoolswhere youcan selectively submit jobs with different resources, monitor theprogress or resubmit as needed. - After the computations are finished, collect the results with
reduceResultsBatchmark()to return tomlr3. The resulting objectis a regularBenchmarkResult.
library("mlr3")library("batchtools")library("mlr3batchmark")tasks= tsks(c("iris","sonar"))learners= lrns(c("classif.featureless","classif.rpart"))resamplings= rsmp("cv",folds=3)design= benchmark_grid(tasks=tasks,learners=learners,resamplings=resamplings)reg= makeExperimentRegistry(NA)
## No readable configuration file found## Created registry in '/tmp/Rtmp8DlMZQ/registry704553adf7a88' using cluster functions 'Interactive'ids= batchmark(design,reg=reg)
## Adding algorithm 'run_learner'## Adding problem 'b39ef23a66b1f1ee'## Exporting new objects: '5ec484de3f93431b' ...## Exporting new objects: '7c35d835f3dfae37' ...## Exporting new objects: '70dd22724e5c724d' ...## Adding 6 experiments ('b39ef23a66b1f1ee'[1] x 'run_learner'[2] x repls[3]) ...## Adding problem '76c4fc7a533d41b7'## Exporting new objects: 'b209de197d6cbe75' ...## Adding 6 experiments ('76c4fc7a533d41b7'[1] x 'run_learner'[2] x repls[3]) ...submitJobs()
## Submitting 12 jobs in 12 chunks using cluster functions 'Interactive' ...getStatus()
## Status for 12 jobs at 2023-11-13 19:32:20:## Submitted : 12 (100.0%)## -- Queued : 0 ( 0.0%)## -- Started : 12 (100.0%)## ---- Running : 0 ( 0.0%)## ---- Done : 12 (100.0%)## ---- Error : 0 ( 0.0%)## ---- Expired : 0 ( 0.0%)reduceResultsBatchmark()
## <BenchmarkResult> of 12 rows with 4 resampling runs## nr task_id learner_id resampling_id iters warnings errors## 1 iris classif.featureless cv 3 0 0## 2 iris classif.rpart cv 3 0 0## 3 sonar classif.featureless cv 3 0 0## 4 sonar classif.rpart cv 3 0 0- TheLarge-Scale Benchmarking chapter of themlr3book
About
Connector between mlr3 and batchtools