ViTDeBERTaNews: A comparative study of single-modal, multimodal, and LLM techniques for detecting fake news

Abstract

The spread of fake news online poses a significant challenge, particularly as social media increasingly combines images and text. To address this, we present the ViT+DeBERTaNews model, which effectively merges visual and textual information for fake news detection. This model utilizes ViT for detailed visual feature extraction and DeBERTa for deep textual understanding. Our experiments demonstrate its effectiveness, achieving 94.8% accuracy on the Weibo dataset and 92.6% on Twitter. The model's precision, recall, and F1 scores for fake news detection were 0.967, 0.945, and 0.956 on Weibo, and 0.929, 0.930, and 0.933 on Twitter, respectively. For real news, it scored 0.968, 0.944, and 0.956 on Weibo, and 0.925, 0.944, and 0.956 on Twitter. In contrast, text-based models like GPT-2 Epoch 3, while strong in precision and recall, are limited by their text-only approach. GPT-4 also faced challenges in recall on the Weibo dataset, indicating the need for task-specific optimizations. These findings underscore the necessity of advanced multimodal models for effective fake news detection.