Skip to content Skip to navigation
University of Warwick
  • Study
  • |
  • Research
  • |
  • Business
  • |
  • Alumni
  • |
  • News
  • |
  • About

University of Warwick
Publications service & WRAP

Highlight your research

  • WRAP
    • Home
    • Search WRAP
    • Browse by Warwick Author
    • Browse WRAP by Year
    • Browse WRAP by Subject
    • Browse WRAP by Department
    • Browse WRAP by Funder
    • Browse Theses by Department
  • Publications Service
    • Home
    • Search Publications Service
    • Browse by Warwick Author
    • Browse Publications service by Year
    • Browse Publications service by Subject
    • Browse Publications service by Department
    • Browse Publications service by Funder
  • Help & Advice
University of Warwick

The Library

  • Login
  • Admin

Why rankings of biomedical image analysis competitions should be interpreted with care

Tools
- Tools
+ Tools

Maier-Hein, Lena, Eisenmann, Matthias, Reinke, Annika, Onogur, Sinan, Stankovic, Marko, Scholz, Patrick, Arbel, Tal, Bogunovic, Hrvoje, Bradley, Andrew P., Carass, Aaron et al.
(2018) Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications, 9 (1). 5217. doi:10.1038/s41467-018-07619-7 ISSN 2041-1723.

[img]
Preview
PDF
WRAP-why-rankings-biomedical-image-interpreted-care-Rajpoot-2018.pdf - Published Version - Requires a PDF viewer.
Available under License Creative Commons Attribution 4.0.

Download (813Kb) | Preview
Official URL: http://dx.doi.org/10.1038/s41467-018-07619-7

Request Changes to record.

Abstract

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Item Type: Journal Article
Subjects: R Medicine > RC Internal medicine
Divisions: Faculty of Science, Engineering and Medicine > Science > Computer Science
Library of Congress Subject Headings (LCSH): Diagnostic imaging, Imaging systems in medicine, Diagnostic imaging -- Competitions -- Evaluation
Journal or Publication Title: Nature Communications
Publisher: Nature Publishing Group
ISSN: 2041-1723
Official Date: 6 December 2018
Dates:
DateEvent
6 December 2018Published
7 November 2018Accepted
Volume: 9
Number: 1
Article Number: 5217
DOI: 10.1038/s41467-018-07619-7
Status: Peer Reviewed
Publication Status: Published
Access rights to Published version: Open Access (Creative Commons)
Date of first compliant deposit: 7 December 2018
Date of first compliant Open Access: 11 December 2018
RIOXX Funder/Project Grant:
Project/Grant IDRIOXX Funder NameFunder ID
ERC-2015-StG-37960H2020 European Research Councilhttp://dx.doi.org/10.13039/100010663
318068Seventh Framework Programmehttp://dx.doi.org/10.13039/100011102
MA 6340/10-1[DFG] Deutsche Forschungsgemeinschafthttp://dx.doi.org/10.13039/501100001659
MA 6340/12-1[DFG] Deutsche Forschungsgemeinschafthttp://dx.doi.org/10.13039/501100001659
106-3114-8-011-002Ministry of Science and Technology, Taiwanhttp://dx.doi.org/10.13039/501100004663
106-2622-8-011-001-TE2Ministry of Science and Technology, Taiwanhttp://dx.doi.org/10.13039/501100004663
105-2221-E-011-121-MY2Ministry of Science and Technology, Taiwanhttp://dx.doi.org/10.13039/501100004663
R01-NS070906National Institutes of Healthhttp://dx.doi.org/10.13039/100000002
RG-1507-05243National Institutes of Healthhttp://dx.doi.org/10.13039/100000002
R01-EB017230National Institutes of Healthhttp://dx.doi.org/10.13039/100000002
DP140102794[ARC] Australian Research Councilhttp://dx.doi.org/10.13039/501100000923
FT110100623[ARC] Australian Research Councilhttp://dx.doi.org/10.13039/501100000923
205321_157207[SNSF] Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschunghttp://dx.doi.org/10.13039/501100001711
P302/12/G157Grantová Agentura České Republikyhttp://dx.doi.org/10.13039/501100001824
LTC17016Ministerstvo Školství, Mládeže a Tělovýchovyhttp://dx.doi.org/10.13039/501100001823
EP/N026993/1[EPSRC] Engineering and Physical Sciences Research Councilhttp://dx.doi.org/10.13039/501100000266
EP/P012841/1[EPSRC] Engineering and Physical Sciences Research Councilhttp://dx.doi.org/10.13039/501100000266
NS/A000050/1Wellcome Trusthttp://dx.doi.org/10.13039/100010269
RGPIN-2015-05471[NSERC] Natural Sciences and Engineering Research Council of Canadahttp://dx.doi.org/10.13039/501100000038
MR/P015476/1[MRC] Medical Research Councilhttp://dx.doi.org/10.13039/501100000265
Heidelberg Collaboratory for Image ProcessingUniversität Heidelberghttp://dx.doi.org/10.13039/501100001661

Request changes or add full text files to a record

Repository staff actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics

twitter

Email us: wrap@warwick.ac.uk
Contact Details
About Us