Lightweight visual question answering (VQA) model for skin disease detection

Abstract

Visual Question Answering (VQA) is an area of artificial intelligence that combines image analysis with natural language understanding to generate context-aware answers to visual queries. Its application in the medical field, particularly in dermatology, holds transformative potential by enabling accessible, interpretable, and efficient diagnostic support. However, there is a lack of structured VQA dataset for skin diseases that can be used for training as well as benchmarking models. Moreover, existing VQA model are all extremely heavy weight and require specialized hardware to train and run. Hence, we developed a custom dataset of 1,038 images annotated by a medical expert for 11 disease classes, with seven question-answer pairs per image. The entire dataset is split into three sections: Train set with 833 images, Test set with 103 images and Validation set with 102 images. In addition, this research proposes a lightweight VQA model pipeline capable of identifying common skin diseases from images and responding to clinically relevant questions related to disease name, severity, causes, diagnostic approach, prevention, contagiousness, and cancer risk. The model uses a modular architecture that integrates a Vision Transformer (ViT) with 86 million parameters for image encoding and MiniLM, a transformer-based text encoder with 22 million parameter, ensuring high accuracy while minimizing computational requirements. We have also trained state-of-theart vision language models such as Gemma-3, QwenVL-2.5, LLaVA-1.5, and BLIP-2 using our dataset for comparison. We achieved high semantic similarity scores on these models having 93.52%, 94.54%, 93.78%, and 94.47% BertScore on Gemma-3, QwenVL-2.5, LLaVA-1.5, and BLIP-2. Unlike these models, our solution is optimized for performance in low-resource settings. The system provides an intuitive interface for users to interact and receive actionable insights, benefiting both patients and healthcare professionals by reducing diagnostic delays and improving decisionmaking with accuracy of 94.87% with a total parameter count of only 108 million, rivaling those with billions of parameters. The results demonstrate that lightweight, domain-adapted VQA models can effectively bridge the healthcare accessibility gap through accurate and interpretable AI assistance.