Zero-shot detection of jailbreaking attempts in LLMs

Abstract

The widespread deployment of Large Language Models (LLMs) has introduced significant safety challenges, notably the emergence of sophisticated ‘jailbreak’ attacks designed to bypass alignment measures and elicit harmful responses. While existing defenses often fail to generalize novel, zero-day attacks. we investigate the hypothesis that a classifier trained to a known distribution of attack patterns can achieve superior detection performance on entirely unseen adversarial prompts. We demonstrate that training on a specialized corpus of engineered safe prompts data that mirrors the structure and tonality of attacks—enhances the model’s ability to recognize conceptually similar yet novel threat vectors. When evaluated on a completely unseen challenge dataset of prompts confirmed to jailbreak state-of-theart models (including Grok-4, Grok-4 Heavy, and Gemini-2.5-Pro), our specialized detector improves accuracy from a baseline of 62.22% to 73.33%. These results, achieved with a compact training set, suggest that for rapidly evolving security threats like jailbreaking, targeted training with high-fidelity engineered data offers a more effective and resource-efficient defense mechanism than reliance on generalized, large-scale datasets.