Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design
- PMID:9750888
- DOI: 10.1016/s1076-6332(98)80294-8
Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design
Abstract
Rationale and objectives: The authors conducted a series of null-case Monte Carlo simulations to evaluate the Dorfman-Berbaum-Metz (DBM) method for comparing modalities with multireader receiver operating characteristic (ROC) discrete rating data.
Materials and methods: Monte Carlo simulations were performed by using discrete ratings on fully crossed factorial designs with two modalities and three, five, and 10 hypothetical readers. The null hypothesis was true for all simulations. The population ROC areas, latent variable structures, case sample sizes, and normal/abnormal case sample ratios used in another study were used in these simulations.
Results: For equal allocation ratios and small (Az = 0.702) and moderate (Az = 0.855) ROC areas, the empirical type I error rate closely matched the nominal alpha level. For very large ROC areas (Az = 0.961), however, the empirical type I error rate was somewhat smaller than the nominal alpha level. This conservatism increased with decreasing case sample size and asymmetric normal/abnormal case allocation ratio. The empirical type I error rate was sometimes slightly larger than the nominal alpha level with many cases and few readers, where there was large residual, relatively small treatment-by-case interaction and relatively large treatment-by-reader interaction.
Conclusion: The results suggest that the DBM method provides trustworthy alpha levels with discrete ratings when the ROC area is not too large and case and reader sample sizes are not too small. In other situations, the test tends to be somewhat conservative or slightly liberal.
Similar articles
- Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification.Hillis SL, Berbaum KS.Hillis SL, et al.Acad Radiol. 2005 Dec;12(12):1534-41. doi: 10.1016/j.acra.2005.07.012.Acad Radiol. 2005.PMID:16321742Free PMC article.
- Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation.Roe CA, Metz CE.Roe CA, et al.Acad Radiol. 1997 Apr;4(4):298-303. doi: 10.1016/s1076-6332(97)80032-3.Acad Radiol. 1997.PMID:9110028
- Prediction accuracy of a sample-size estimation method for ROC studies.Chakraborty DP.Chakraborty DP.Acad Radiol. 2010 May;17(5):628-38. doi: 10.1016/j.acra.2010.01.007.Acad Radiol. 2010.PMID:20380980Free PMC article.
- A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data.Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS.Hillis SL, et al.Stat Med. 2005 May 30;24(10):1579-607. doi: 10.1002/sim.2024.Stat Med. 2005.PMID:15685718
- "Binary" and "non-binary" detection tasks: are current performance measures optimal?Gur D, Rockette HE, Bandos AI.Gur D, et al.Acad Radiol. 2007 Jul;14(7):871-6. doi: 10.1016/j.acra.2007.03.014.Acad Radiol. 2007.PMID:17626312Review.
Cited by
- MR diagnosis of vertebral artery dissection: value of 3D time-of-flight and true fast imaging with steady-state precession fusion imaging.Kidoh M, Nakaura T, Takashima H, Yoshikawa M, Uemura S, Harada K, Hirai T, Yamashita Y.Kidoh M, et al.Insights Imaging. 2013 Feb;4(1):135-42. doi: 10.1007/s13244-012-0204-x. Epub 2012 Dec 2.Insights Imaging. 2013.PMID:23203816Free PMC article.
- Satisfaction of search in multitrauma patients: severity of detected fractures.Berbaum KS, El-Khoury GY, Ohashi K, Schartz KM, Caldwell RT, Madsen M, Franken EA Jr.Berbaum KS, et al.Acad Radiol. 2007 Jun;14(6):711-22. doi: 10.1016/j.acra.2007.02.016.Acad Radiol. 2007.PMID:17502261Free PMC article.
- A large web-based observer reliability study of early ischaemic signs on computed tomography. The Acute Cerebral CT Evaluation of Stroke Study (ACCESS).Wardlaw JM, von Kummer R, Farrall AJ, Chappell FM, Hill M, Perry D.Wardlaw JM, et al.PLoS One. 2010 Dec 30;5(12):e15757. doi: 10.1371/journal.pone.0015757.PLoS One. 2010.PMID:21209901Free PMC article.
- A comparison of denominator degrees of freedom methods for multiple observer ROC analysis.Hillis SL.Hillis SL.Stat Med. 2007 Feb 10;26(3):596-619. doi: 10.1002/sim.2532.Stat Med. 2007.PMID:16538699Free PMC article.
- Comparison between differently priced devices for digital capture of X-ray films using computed tomography as a gold standard: a multireader-multicase receiver operating characteristic curve study.Salazar AJ, Camacho JC, Aguirre DA.Salazar AJ, et al.Telemed J E Health. 2011 May;17(4):275-82. doi: 10.1089/tmj.2010.0189. Epub 2011 Apr 1.Telemed J E Health. 2011.PMID:21457011Free PMC article.
Publication types
MeSH terms
Grants and funding
LinkOut - more resources
Medical