Repository files navigation

This repo is a simulation study of statistical properties for hypothesistests in LRTesteR. Each row is an experiment where data are generatedfrom random number generators and hypothesis test are done. The firstfive rows look like

#> # A tibble: 450,000 × 8#>   test                      mu variance   stat pvalue alt       CI_LB CI_UB#>   <chr>                  <dbl>    <dbl>  <dbl>  <dbl> <chr>     <dbl> <dbl>#> 1 gaussian_mu_one_sample    -4        1 0.251   0.617 two.sided -4.07 -3.89#> 2 gaussian_mu_one_sample    -4        1 1.79    0.181 two.sided -4.03 -3.85#> 3 gaussian_mu_one_sample    -4        1 1.30    0.255 two.sided -4.04 -3.86#> 4 gaussian_mu_one_sample    -4        1 0.453   0.501 two.sided -4.11 -3.94#> 5 gaussian_mu_one_sample    -4        1 0.0143  0.905 two.sided -4.09 -3.92#> # ℹ 449,995 more rows

For each simulated experiment, both the true hypothesis and the outcomeof the test are known. Multiple experiments are aggregated to calculatetype I error rates.

#> # A tibble: 30 × 4#> # Groups:   test, mu [10]#>   test                      mu variance Type_I_Error#>   <chr>                  <dbl>    <dbl>        <dbl>#> 1 gaussian_mu_one_sample    -4        1         0.05#> 2 gaussian_mu_one_sample    -4        3         0.05#> 3 gaussian_mu_one_sample    -4        5         0.05#> 4 gaussian_mu_one_sample    -2        1         0.05#> 5 gaussian_mu_one_sample    -2        3         0.05#> # ℹ 25 more rows

Each simulated experiment is based on a sample size of 500. Eachcombination of true hypothesis, parameter value, and test is repeated5,000 times.

Detailed statistical analysis can be found in the type I and type IIfolders. Code to run the simulation is in the R Code folder.

Most tests have a type I error rate of 5%. Likelihood ratio tests havesimilar error rates to exact tests.

Compared to the one sample tests, type I error rates are further fromthe .05 target. The worst performing tests are within half a percentagepoint.

All tests achieve near 0% type II error for a large enough effect size.

Similar to above, all one way tests have near 0% type II error rate forlarge effect sizes.

Most functions have a coverage rate of 95%. The worst performingconfidence intervals are within one percentage point.