C-MAT: Cross-modal aligned transformer for four-class differential diagnosis of neurodegenerative diseases using structural MRI and resting-state EEG

Abstract

Neurodegenerative diseases, including Alzheimer disease (AD), frontotemporal dementia (FTD), and Parkinson disease (PD), affect more than 57 million people globally and remain difficult to distinguish because symptoms overlap and MRI/EEG biomarkers are modality dependent. This thesis presents C-MAT (Cross-Modal Aligned Transformer), a unified multimodal framework for four-class classification of AD, FTD, PD, and healthy controls using T1-weighted structural MRI and resting state EEG. The model combines a shared pretrained image encoder, modality specific attention pooling, a dual-path EEG branch that fuses pseudo-image and raw-feature representations, and a sigmoid-gated fusion mechanism with learnable null tokens for missing modalities. C-MAT was developed through three iterative pipeline versions over nine months, with forensic debugging used to expose cross-dataset subject-ID collisions, EEG reprocessing reuse, and pretrained-weight configuration errors. The final system was evaluated on 1,102 subjects from 10 open datasets spanning seven countries, using subject-level stratified five-fold cross-validation and dataset-prefixed identifiers. ConvNeXt-Tiny achieved a macro F1 of 0.611 ± 0.029, and DeiT-Tiny achieved 0.590 ± 0.044, outperforming classical baselines such as PSD+SVM (0.309) and Random Forest (0.453). On the strictly paired 25-subject MRI+EEG subset, CMAT DeiT reached a mean macro F1 of 0.702, exceeding late-fusion baselines such as ResNet50 Fusion (0.482) and DeiT Fusion (0.296). Class-balanced focal loss improved macro F1 by +0.048 over cross-entropy. GradCAM and fusion-gate analysis showed clinically plausible patterns, including hippocampal and frontal emphasis for AD/FTD and greater EEG reliance for PD. To the best of our knowledge, CMAT is among the first MRI+EEG systems for four-class neurodegenerative disease classification evaluated under subject-level multi-site validation