Computer Science > Computer Vision and Pattern Recognition
arXiv:2302.01790 (cs)
[Submitted on 3 Feb 2023 (v1), last revised 23 Feb 2024 (this version, v4)]
Title:Understanding metric-related pitfalls in image analysis validation
Authors:Annika Reinke,Minu D. Tizabi,Michael Baumgartner,Matthias Eisenmann,Doreen Heckmann-Nötzel,A. Emre Kavur,Tim Rädsch,Carole H. Sudre,Laura Acion,Michela Antonelli,Tal Arbel,Spyridon Bakas,Arriel Benis,Matthew Blaschko,Florian Buettner,M. Jorge Cardoso,Veronika Cheplygina,Jianxu Chen,Evangelia Christodoulou,Beth A. Cimini,Gary S. Collins,Keyvan Farahani,Luciana Ferrer,Adrian Galdran,Bram van Ginneken,Ben Glocker,Patrick Godau,Robert Haase,Daniel A. Hashimoto,Michael M. Hoffman,Merel Huisman,Fabian Isensee,Pierre Jannin,Charles E. Kahn,Dagmar Kainmueller,Bernhard Kainz,Alexandros Karargyris,Alan Karthikesalingam,Hannes Kenngott,Jens Kleesiek,Florian Kofler,Thijs Kooi,Annette Kopp-Schneider,Michal Kozubek,Anna Kreshuk,Tahsin Kurc,Bennett A. Landman,Geert Litjens,Amin Madani,Klaus Maier-Hein,Anne L. Martel,Peter Mattson,Erik Meijering,Bjoern Menze,Karel G.M. Moons,Henning Müller,Brennan Nichyporuk,Felix Nickel,Jens Petersen,Susanne M. Rafelski,Nasir Rajpoot,Mauricio Reyes,Michael A. Riegler,Nicola Rieke,Julio Saez-Rodriguez,Clara I. Sánchez,Shravya Shetty,Maarten van Smeden,Ronald M. Summers,Abdel A. Taha,Aleksei Tiulpin,Sotirios A. Tsaftaris,Ben Van Calster,Gaël Varoquaux,Manuel Wiesenfarth,Ziv R. Yaniv,Paul F. Jäger,Lena Maier-Hein
View a PDF of the paper titled Understanding metric-related pitfalls in image analysis validation, by Annika Reinke and 77 other authors
View PDFHTML (experimental)Abstract:Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.
| Comments: | Shared first authors: Annika Reinke and Minu D. Tizabi; shared senior authors: Lena Maier-Hein and Paul F. Jäger. Published in Nature Methods. arXiv admin note: text overlap witharXiv:2206.01653 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2302.01790 [cs.CV] |
| (orarXiv:2302.01790v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2302.01790 arXiv-issued DOI via DataCite | |
| Journal reference: | Nature methods, 1-13 (2024) |
| Related DOI: | https://doi.org/10.1038/s41592-023-02150-0 DOI(s) linking to related resources |
Submission history
From: Annika Reinke [view email][v1] Fri, 3 Feb 2023 14:57:40 UTC (34,409 KB)
[v2] Thu, 9 Feb 2023 16:00:45 UTC (34,178 KB)
[v3] Mon, 25 Sep 2023 12:55:05 UTC (31,224 KB)
[v4] Fri, 23 Feb 2024 13:37:33 UTC (31,216 KB)
Full-text links:
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
- Other Formats
View a PDF of the paper titled Understanding metric-related pitfalls in image analysis validation, by Annika Reinke and 77 other authors
References & Citations
Bookmark
Bibliographic and Citation Tools
Bibliographic Explorer(What is the Explorer?)
Connected Papers(What is Connected Papers?)
Litmaps(What is Litmaps?)
scite Smart Citations(What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv(What is alphaXiv?)
CatalyzeX Code Finder for Papers(What is CatalyzeX?)
DagsHub(What is DagsHub?)
Gotit.pub(What is GotitPub?)
Hugging Face(What is Huggingface?)
Papers with Code(What is Papers with Code?)
ScienceCast(What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower(What are Influence Flowers?)
CORE Recommender(What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community?Learn more about arXivLabs.