Calibration and Uncertainty for multiRater Volume Assessment in Banner
Join

The Testing Phase are now open!


  New dataset version uploaded in Zenodo: https://zenodo.org/records/12687192

Find our evaluation and baseline code in our Github: https://github.com/SYCAI-Technologies/curvas-challenge

Visit our website: https://www.sycaimedical.com/challenge


Clinical problem

In medical imaging, DL models are often tasked with delineating structures or abnormalities within complex anatomical structures, such as tumors, blood vessels, or organs. Uncertainty arises from the inherent complexity and variability of these structures, leading to challenges in precisely defining their boundaries. This uncertainty is further compounded by interrater variability, as different medical experts may have varying opinions on where the true boundaries lie. DL models must grapple with these discrepancies, leading to inconsistencies in segmentation results across different annotators and potentially impacting diagnosis and treatment decisions. Addressing interrater variability in DL for medical segmentation involves the development of robust algorithms capable of capturing and quantifying uncertainty, as well as standardizing annotation practices and promoting collaboration among medical experts to reduce variability and improve the reliability of DL-based medical image analysis. Interrater variability poses significant challenges in the field of DL for medical image segmentation.

Furthermore, achieving model calibration, a fundamental aspect of reliable predictions, becomes notably challenging when dealing with multiple classes and raters. Calibration is pivotal for ensuring that predicted probabilities align with the true likelihood of events, enhancing the model's reliability. It must be considered that, even if not clearly, having multiple classes account for uncertainties arising from their interactions. Moreover, incorporating annotations from multiple raters adds another layer of complexity, as differing expert opinions may contribute to a broader spectrum of variability and computational complexity.

Consequently, the development of robust algorithms capable of effectively capturing and quantifying variability and uncertainty, while also accommodating the nuances of multi-class and multi-rater scenarios, becomes imperative. Striking a balance between model calibration, accurate segmentation and handling variability in medical annotations is crucial for the success and reliability of DL-based medical image analysis.

CURVAS Challenge Goal

Because of all the previously stated reasons, we have created a challenge that considers all of the above. In this challenge, we will work with abdominal CT scans. Each of them will have three different annotations obtained from different experts and each of the annotations will have three classes: pancreas, kidney and liver.

The main idea is to be able to evaluate the results considering the multi rater information. There will be three separate evaluations: firstly, a classical dice score evaluation together with an uncertainty study will be performed; secondly, a volumetric assessment to give relecant clinical information will take place; finally, this part will consist on studying whether the model is calibrated or not. All of these evaluations will be performed considering all three different annotations.

More details will be specified in the Metrics section.

Winners

Top five performing methods will be announced publicly. The CURVAS consortium will extend invitations to the top the corresponding teams to join its ranks. These teams will earn recognition as consortium authors in an upcoming influential journal publication that will contain the contributions from this challenge.

Furthermore, winners will be invited to present their methods and results in the challenge event hosted in MICCAI 2024.

Finally, there will be cash prices for the top three methods:


The participating teams may publish their own results separately only after the organizer has published a challenge paper and always mentioning the organizer's challenge paper.





This work was supported by the Catalan Government inside the program ”Doctorats Industrials” and by the company Sycai Technologies SL. Mertixell Riera i Marín is supported by the industrial doctorate of the AGAUR 2021-063.  

     

The challenge has been co-funded by Proyectos de Colaboración Público-Privada (CPP2021-008364), funded by MCIN/AEI, and the European Union through the NextGenerationEU/PRTR.