Evaluating the performance of open-source LLMs in sarcasm detection: a comparative analysis

Abstract

Detecting sarcasm is a nuanced and challenging task in natural language processing (NLP), as it requires understanding subtle contextual cues, symbolic gestures, and timeline of dataset as sarcasm constantly changes through time. This study tends to evaluate the performance of open-source large language models in sarcasm detection, exploring each model’s effectiveness in identifying sarcasm across varied contexts. In doing so, it tries to apply various prompt engineering techniques towards establishing an optimized environment; where these open source models will detect sarcasm based on the hyper parameters like context, and the actual comment corresponding to the context. Among the evaluated models, Llama 3.1 8B achieved the highest overall accuracy (69.07%) while DeepSeek R1 14B demonstrated the most balanced performance (accuracy = 66%, precision = 64.29%, recall = 72%). On sarcasm-device level evaluation, the models showed varying strengths: Llama 3.1 8B performed best on Contextual Incongruity (63.15%), Mistral 7B on both Contradiction and Contextual Incongruity (50%), Gemma 3 12B on Contradiction (80%), Qwen 3 8B on Contradiction (50%), and DeepSeek R1 14B on Hyperbole (50%). The findings highlight that while current open-source LLMs exhibit promising capabilities in detecting sarcastic intent, their performance varies by sarcasm device, revealing architectural biases and limitations. This study thus provides benchmark insights and interpretive evidence for advancing future NLP research in sarcasm modeling, architectural evaluation, and pragmatic language understanding.