MIM-HeMIS: Self-supervised masked image modeling with heterogeneous modality fusion for brain tumor segmentation under missing MRI modalities

Abstract

Accurate segmentation of regions of interest from MRI to diagnose brain tumors is crucial. MRI is the standard technique used in imaging the brain, but manual segmentation of tumor regions is time-consuming, manual and prone to human error. Although deep learning has been able to significantly enhance segmentation

performance, it requires big data sets of labeled images which is difficult in medical applications. In this study, we introduce a hybrid CNN-Transformer framework that integrates the Masked Image Modeling (MIM) based self-supervised pretraining and the HeMIS statistical abstraction mechanism. A hybrid model is pretrained on the BraTS 2021 dataset and finetuned for 4-class brain tumor segmentation for all 4 MRI modalities following the proposed framework. A major advantage is that the modalities are robust, meaning that there is no need to train a new model for each of the 15 possible combinations of modalities. When modalities are missing, the variance computed by the HeMIS abstraction layer captures feature disagreement

across available inputs, providing useful information for identifying cases where pre- dictions may be less reliable. Though this variance has not been calibrated as a

clinical confidence score. The model was tested on BraTS 2021 with all 15 modality combinations with excellent and clinically applicable results: Whole Tumor Dice of 90.2%, Tumor Core Dice of 85.9%, and Enhancing Tumor Dice of 79.1% for all four modalities.