Movatterモバイル変換

whirl: Log Execution of Scripts

Description

Logging of scripts suitable for clinical trials using 'Quarto' to create nice human readable logs. 'whirl' enables execution of scripts in batch, while simultaneously creating logs for the execution of each script, and providing an overview summary log of the entire batch execution.

Author(s)

Maintainer: Aksel Thomsenoath@novonordisk.com

Authors:

• Lovemore Gakavalvgk@novonordisk.com

• Cervan Girardcgid@novonordisk.com

• Kristian Troejelsgaardktqn@novonordisk.com

• Steffen Falgreen Larsensffl@novonordisk.com

• Vladimir Obucinavlob@novonordisk.com

• Michael Svingelwymz@novonordisk.com

• Skander Muldersktf@novonordisk.com

• Oliver Lundsgaardovlr@novonordisk.com

Other contributors:

• Novo Nordisk A/S [copyright holder]

See Also

Useful links:

• https://novonordisk-opensource.github.io/whirl/

• https://github.com/novonordisk-opensource/whirl

• Report bugs athttps://github.com/NovoNordisk-OpenSource/whirl/issues

Helper function to log custom messages

Description

Useful for e.g. read and write operations on databases etc.that are not automatically captured.

Usage

log_read(file, log = Sys.getenv("WHIRL_LOG_MSG"))log_write(file, log = Sys.getenv("WHIRL_LOG_MSG"))log_delete(file, log = Sys.getenv("WHIRL_LOG_MSG"))

Arguments

Details

The default environment variableWHIRL_LOG_MSG is set in the session usedto log scripts, and input is automatically captured in the resulting log.

If run outside of whirl, meaning when the above environment variable isunset, the operations are streamed tostdout(). By default the console.

Examples

# Stream logs to console since `WHIRL_LOG_MSG` is not set:log_read("my/folder/input.txt")log_write("my/folder/output.txt")log_delete("my/folder/old_output.txt")

Execute single or multiple R, R Markdown, and Quarto scripts

Description

Executes and logs the execution of the scripts.Logs for each script are stored in the same folder as the script.

The way the execution is logged is configurable through several options fore.g. the verbosity of the logs.Seewhirl-options on how to configure these.

Usage

run(  input = "_whirl.yml",  steps = NULL,  summary_file = "summary.html",  n_workers = zephyr::get_option("n_workers", "whirl"),  check_renv = zephyr::get_option("check_renv", "whirl"),  track_files = zephyr::get_option("track_files", "whirl"),  out_formats = zephyr::get_option("out_formats", "whirl"),  log_dir = zephyr::get_option("log_dir", "whirl"))

Arguments

Value

A tibble containing the execution results for all the scripts.

Examples

# Copy example scripts:file.copy(  from = system.file("examples", c("success.R", "warning.R", "error.R"),    package = "whirl"  ),  to = tempdir())# Run a single script and create log:run(file.path(tempdir(), "success.R"))# Run several scripts in parallel on up to 2 workers:run(  input = file.path(tempdir(), c("success.R", "warning.R", "error.R")),  n_workers = 2)# Run several scripts in two steps by providing them as list elements:run(  list(    file.path(tempdir(), c("success.R", "warning.R")),    file.path(tempdir(), "error.R")  ))# Re-directing the logs to a sub-folder by utilizing the log_dir argument in# run(). This will require that the sub-folder exists.# Specifying the path using a manually defined characterrun(file.path(tempdir(), "success.R"), log_dir = tempdir())# Specifying the path with a generic function that can handle the scripts# individually.run(  input = file.path(tempdir(), "success.R"),  log_dir = function(x) {paste0(dirname(x), "/logs")})

Use whirl to create biocompute logs

Description

Utility function to setup execution with whirl in your project suitable forcreating biocompute logs withwrite_biocompute():

1. Creates configuration file (default⁠_whirl.yml⁠) with default values for thebiocompute metadata.

2. Updates.gitignore to not include log files

Seevignette("whirl") for how to specify paths inside theconfiguration file.

Usage

use_biocompute(  config_file = "_whirl.yml",  parametrics_file = "_parametrics.yml")

Arguments

Use whirl

Description

Utility function to setup execution with whirl in your project:

1. Creates configuration file (default⁠_whirl.yml⁠)

2. Updates.gitignore to not include log files

Seevignette("whirl") for how to specify paths inside theconfiguration file.

Usage

use_whirl(config_file = "_whirl.yml")

Arguments

Options for whirl

Description

verbosity_level

Verbosity level for functions in whirl.Seezephyr::verbosity_level for details.

• Default:NA_character_

• Option:whirl.verbosity_level

• Environment:R_WHIRL_VERBOSITY_LEVEL

out_formats

Which log format(s) to produce. Possibilities arehtml,json, and markdown formats:gfm,commonmark, andmarkua.

• Default:"html"

• Option:whirl.out_formats

• Environment:R_WHIRL_OUT_FORMATS

track_files

Should files read and written be tracked?Currently only supported on Linux.

• Default:FALSE

• Option:whirl.track_files

• Environment:R_WHIRL_TRACK_FILES

check_renv

Should the projects renv status be checked?

• Default:FALSE

• Option:whirl.check_renv

• Environment:R_WHIRL_CHECK_RENV

track_files_discards

List of file naming patterns not be tracked when track_files = TRUE

• Default:c("^/lib", "^/etc", "^/lib64", "^/usr", "^/var", "^/opt", "^/sys", "^/proc", "^/tmp", "^/null", "^/urandom", "^/.cache")

• Option:whirl.track_files_discards

• Environment:R_WHIRL_TRACK_FILES_DISCARDS

track_files_keep

List of file naming patterns always to be tracked whentrack_files = TRUE

• Default:NULL

• Option:whirl.track_files_keep

• Environment:R_WHIRL_TRACK_FILES_KEEP

approved_packages

List of approved R packages and their version in the format: {name}@{version}

• Default:NULL

• Option:whirl.approved_packages

• Environment:R_WHIRL_APPROVED_PACKAGES

approved_python_packages

List of approved Python packages and their version in the format: {name}@{version}

• Default:NULL

• Option:whirl.approved_python_packages

• Environment:R_WHIRL_APPROVED_PYTHON_PACKAGES

n_workers

Number of simultaneous workers used in the run function.A maximum of 128 workers is allowed.

• Default:1

• Option:whirl.n_workers

• Environment:R_WHIRL_N_WORKERS

log_dir

The output directory of the log files. Default is the folder ofthe executed script. log_dir can be a path as a character or it can be afunction that takes the script path as input and returns the log directory.For more information see the examples ofrun() orvignette('whirl').

• Default:function (x) dirname(x)

• Option:whirl.log_dir

• Environment:R_WHIRL_LOG_DIR

execute_dir

The working directory of the process executing each script.Default us to execute R files from the working directory when callingrun()and all other functions from the directory of the script. To change providea character path (used for all scripts) or a function that takes the scriptas input and returns the execution directory.

• Default:NULL

• Option:whirl.execute_dir

• Environment:R_WHIRL_EXECUTE_DIR

wait_timeout

Timeout for waiting for the R process from callr::r_session tostart, in milliseconds.

• Default:9000

• Option:whirl.wait_timeout

• Environment:R_WHIRL_WAIT_TIMEOUT

environment_secrets

Secret environment variable patterns.Any variables matching will not be included in the logs.

• Default:c("BASH_FUNC", "_SSL_CERT", "_KEY", "_PAT", "_TOKEN")

• Option:whirl.environment_secrets

• Environment:R_WHIRL_ENVIRONMENT_SECRETS

Internal parameters for reuse in functions

Description

Internal parameters for reuse in functions

Arguments

Details

Seewhirl-options for more information.

Create biocompute logs

Description

BioCompute is a standard for logs of programs for for Bioinformatics Computational Analyses.

The BioCompute object is ajson log that can be created based on the output ofrun().

Usage

write_biocompute(queue = run("_whirl.yml"), path = "bco.json", ...)

Arguments

Details

The object consists of the following domains:

• Specifications:

  • spec_version: Version of BioCompute used ('https://w3id.org/biocompute/1.3.0/“)

  • object_id: Unique project id

  • type: Your project type

  • etag: Youretag id from the BioCompute Object Portal

• Provenance Domain

  • This is used to track the history of the BCO. Review and signatures go here.

• Usability Domain

  • This is used to improve searchability by allowing a free-text description of the BCO.

  • Provide external document.

• Extension Domain

  • This is used to add any additional structured information that is not directly covered by the BCO.

• Description Domain

  • Contains a structured field for the description of external references, the pipeline steps,and the relationship of I/O objects.

  • Provide external document.

  • Note: Use ofkeywords andExternal_Reference entries are not yet implemented.To use fill out the entries manually after creating the BioCompute object.'

• Execution Domain

  • Contains fields for the execution of the BCO.

  • Note: Use ofexternal_data_endpoints not implemented. Fill out manually afterwards if needed.

• Parametric Domain

  • Represents the list of parameters customizing the computational flow which can affectthe output of the calculations.

• IO Domain

  • Represents the list of global input and output files created by the computational workflow.

• Error Domain

  • Defines the empirical and algorithmic limits and error sources of the BCO.

  • Note: Use of this domain is not clearly defined.It is therefore always left empty in the current implementation.If you want to add content do so manually after creating the BCO.

See theBioCompute Object Portal and theBioCompute Objects Wiki for more information.

Value

(invisible)list of the biocompute domains and their content.