A multimodal AI-based agricultural assistance system: Enhancing farmer decision-making through visual question answering (VQA) in Bangladesh

Abstract

Agriculture plays a vital role in Bangladesh, employing approximately 43% of the population, yet many farmers lack formal education and expert guidance, leading to uninformed decision-making. This research explores the development of a multimodal Visual Question Answering (VQA)-based agricultural assistance system that integrates computer vision, natural language processing (NLP), and speech processing to provide real-time, voice-based responses in Bangla, reducing the immediate need for agricultural experts. Instead of relying solely on CNN-based classification, the system employs Vision–Language Models (VLMs) with visual encoders for crop disease understanding, a Large Language Model (LLM) for agricultural query answering, and speech models for Bangla voice interaction. The system offers an intuitive, offline-compatible mobile solution tailored for farmers in low-resource environments. A conversational agricultural diagnostic dataset was developed containing 96,003 expert-style diagnostic entries with 9,138 rice leaf images across six classes. A structured dataset covering Bangladesh’s diverse crops, pests, and diseases ensures AI model optimization and practical usability. A comparative evaluation assesses the multimodal AI system (Image + Text + Voice) against traditional single-input models (Image-only, Text-only) based on diagnostic accuracy, response relevance, and user satisfaction. In the final offline pipeline, MobileNetV3-Small achieved 95.35% disease classification accuracy with stable mobile CPU latency ( 12– 18 ms), Gemma 3–1B produced the strongest overall response quality among tested lightweight LLMs (METEOR = 0.1095, cosine similarity = 0.539, and BERTScore F1 ￿ 0.70, BertScore Precision: 0.72, BertScore Recall: 0.69), and Whisper-Base enabled faster-than-real-time Bangla speech recognition (RTF = 0.7859, WER = 0.353). This approach minimizes dependence on agricultural experts, saving time and costs while positively impacting the economy. A farmer field survey (n = 30) using five evaluation parameters indicates strong real-world usability: 92% rated the system helpful/very helpful, 88% found it easy to use, 84% reported successful offline usage, 86% expressed satisfaction with response clarity, and 82% reported improved confidence in decision-making. Ultimately, this research seeks to bridge the digital divide in agriculture, empowering farmers with an AI-driven solution for real-time problem-solving and enhanced food security.