Image processing and LLM-based VQA framework for crop dsease diagnosis and solutions leveraging contextual analysis

Abstract

Early detection of crop diseases is very important to reduce losses in yield and to improve food security but the farmers have to overcome many obstacles like lack of sufficient experts and language barrier that prevents them to make decisions in time. To overcome these complications, this study introduces a single system, which is a combination of crop disease classification, Image captioning, and Visual Question Answering (VQA) into a bilingual web-based assistant to operate in Bangla and English. The system utilizes a tuned dataset containing crop images and labels with disease conditions and the Swin Transformer as the backbone classifier with a test accuracy of 96.37 and macro precision, recall and F1 of 0.9526, 0.9643, and 0.9571 respectively, with an average prediction confidence of 0.9249. The BLIP based captioning module complements the classifier producing descriptive textual summaries of disease symptoms, and a combined context of classification and captioning aids the VQA component, allowing interactive, natural language queries on the part of users. The VQA analysis had a macro faithfulness of 0.6380, a macro context precision value of 0.7512, macro context recall value of 0.8796, and macro answer relevance value of 0.7842 which implied a high contextual integration but also showed a need to improve the process of adjusting the responses to user queries. This framework goes beyond standard classification systems by combining both technical precision and accessibility to provide a practical, interactive and inclusive framework that allows farmers to diagnose crop diseases and make effective management decisions which enhances agricultural productivity as well as sustainability.