Repository files navigation

This work is now published in Wellcome Open Research, please cite as:

Smith RA, Samyshkin Y, Mohammed W et al. assertHE: an R package toimprove quality assurance of HTA models. Wellcome Open Res 2024,9:701.https://doi.org/10.12688/wellcomeopenres.23180.1

The goal ofassertHE is to help modellers build and review healtheconomic models in R. The package provides functions which can beincluded within models to check that the objects created conform tostandard rules (e.g. probabilities between 0 and 1). It also providesfunctions to review the structure of the model, showing the network offunctions color coded by test coverage. Users can click on the nodes tosee function and test source code, test coverage and create an AIgenerated summary of the function.

Rob outlined the package at R-HTA 2024 with avideo andslidespublicly available for those interested in finding out more.

We are continuing to work to improve the package and welcomecontributions. To get involved, please see theContributionguide.For more context about the aims of the wider project please readthewiki.

You can install the CRAN version of assertHE fromCRAN with:

install.packages("assertHE")library(assertHE)

Alternatively the development version of assertHE can be installed fromGitHub with:

# install.packages("devtools")devtools::install_github("dark-peak-analytics/assertHE")library(assertHE)

The below code creates a visual representation of the model structurefor a given project. The user must provide a path to the project folder,the location of functions (typically “R”) and the location of tests(typically “tests/testthat”).

visualise_project(project_path="path_to_project_directory",foo_path="R",test_path="tests/testthat",run_coverage=TRUE)

The result is a visual representation of the model functions. This givessome indication of how to review the model since each function can bechecked in isolation and in combination. It may also reveal redundantcode.

The below is an example of using the function on the cdx2cea model. Thered nodes are the ones without tests, the green nodes are the ones withtests. When hovering over a function we can see more informationincluding where it is defined (file and line number) and where the test(if any) resides. The coverage % of the function is also provided. Testswith coverage <20% are in red, between 20-80% in orange, and above 80%in green. These are arbitrary cut-points, reviewers should assesssufficiency of testing.

To use the LLM function summary tool follow the guidehere.

The package has a series of functions to be usedwithin models tocheck that the objects created conform to standard rules(e.g. probabilities between 0 and 1).

The code below shows a basic example which shows you how to usecheck_trans_prob_array to ensure that the time dependent transitionprobability array is balanced.

library(assertHE)# create a transition probability arrayn_t<-1000# number of cyclesv_hs_names<- c("H","S","D")# health statesn_hs<- length(v_hs_names)# create array of transition probabilitiesa_P<-array(data=0,dim= c(n_hs,n_hs,n_t),dimnames=list(v_hs_names,v_hs_names,1:n_t))# fill in transition probabilities for main transitions.a_P["H","S",]<-0.3a_P["H","D",]<-0.01a_P["S","D",]<-0.1a_P["S","H",]<-0.5# Fill in the proportion remaining in health state in each slice.# This is the remainder after all other transitions are accounted for.for(xin1:n_t){ diag(a_P[,,x])<-1- rowSums(a_P[,,x])}# Use the function from the package.# This check should return no error, the array is square, numeric, values# are between 0 and 1 and all rows sum to 1.# Note: stop_if_not = FALSE returns warnings, stop_if_not = TRUE returns errors.check_trans_prob_array(a_P=a_P,stop_if_not=TRUE)# We can introduce an error to see the output# In this case, we set the first 10 cycles of transition from H to S to 0.# This means that the rows don't sum to 1 for the H row for 1:10 cycle.a_P["H","S",1:10]<-0check_trans_prob_array(a_P=a_P,stop_if_not=FALSE)# The output looks like this:# Warning message:# In check_array_rows_balanced(a_P, stop_if_not = stop_if_not) :#   Not valid transition probabilities#    Transition probabilities not valid from Health States:# 1                                           H; at cycle 1# 2                                           H; at cycle 2# 3                                           H; at cycle 3# 4                                           H; at cycle 4# 5                                           H; at cycle 5# 6                                           H; at cycle 6# 7                                           H; at cycle 7# 8                                           H; at cycle 8# 9                                           H; at cycle 9# 10                                         H; at cycle 10

Please get in contact if you would like to use the package to helpreview a model in R.

The following models have been visualized using the package, as testcases:

• NICE RCC Model
• sicksickerPackteaching model contained in a package.
• cdx2cea as described inAlarid-Escudero etal. 2022
• DOACs-AF-Economic-modeldeveloped by Bristol University
• The CGD AMR Cost model - in press.
• Embedding EconomicsAnalysisDiabetes Microsimulation model described in (in press).
• Several internal models at Dark Peak Analytics.
• Several internal models at Maple Health.
• The National Institute for Health and Care Excellence (NICE) havebuilt theassertHE visualiser into their template for model reviews.

Once the model has been generated, it is possible to share the HTML forthe interactive network. In the visualisation tab click the downwardarrow on the ‘export’ button and then click ‘save as web page’.

The visualisation for the HTML file may take a while to load for largenetworks. However, all the funtionality from the HTML version (not theshiny version with the links) should be there.

To get in contact about this project or other collaborations please feelfree to email me atrsmith@darkpeakanalytics.com.