Uncertainty-aware federated dual-branch multimodal fusion for privacy-preserving skin lesion diagnosis

Abstract

Skin lesion diagnosis is well-suited to data-driven methods, yet its real-world use is often hindered by strict privacy rules, fragmented data ownership, and inconsistent imaging conditions. To overcome these barriers, our work introduces a robust privacy-preserving framework that combines multimodal representation learning with decentralized training which reduces the need to centralize sensitive clinical data. We propose a unified dual-branch framework that supports (i) task-level multitask learning by coupling lesion classification with lesion segmentation using a shared Swin Transformer encoder and an Attention U-Net decoder, and (ii) modality-level learning by fusing image features with auxiliary patient metadata through configurable fusion strategies. To address client heterogeneity under federated learning, we introduce a Dynamic Uncertainty-Aware Divide2Conquer (DUA-D2C) aggregation strategy that adaptively weights client updates based on predictive performance and uncertainty, improving robustness in non-IID settings. Extensive evaluations on HAM10000 and MILK10K demonstrate that multimodal fusion improves over image-only baselines, and that DUA-D2C preserves or improves performance on HAM10000 while constraining degradation on the more heterogeneous MILK10K setting.