In my previous article about the shutdown of Fable 5 and Mythos 5, I mentioned that I would write in more detail about prompt jailbreaking — the technique that allows someone to trick an AI model into ignoring its own safety restrictions. The time has come. Understanding what a jailbreak is, how it works, and why it is so difficult to eliminate is key to following the regulation and AI security debate that lies ahead.
What is a jailbreak in an LLM?
A jailbreak is any technique that causes a large language model (LLM) to generate content that its safety guidelines — its alignment — should prevent. Modern models like GPT, Claude, or Gemini are trained not only to be helpful, but also to refuse to produce, for example, instructions for making weapons, illegal content, or disinformation. A jailbreak is the attempt — often successful — to circumvent those barriers.
Academic research on this topic has grown explosively. On arXiv alone, the Cornell preprint platform, there are now more than 1,000 scientific papers published on LLM jailbreaks, with dozens of new works every week.
Main types of jailbreak
There is no single method. The research community has catalogued several families of attacks:
1. Manual jailbreak or prompt engineering
These are the most publicly known. They involve reformulating the request creatively: asking the model to “role-play” an unrestricted character, building a science fiction narrative that includes the forbidden information, or using the famous “DAN” (Do Anything Now) prompt. They are easy to apply, but also the easiest to patch with model updates.
2. Automatic adversarial attacks (GCG)
The seminal paper in this field is “Universal and Transferable Adversarial Attacks on Aligned Language Models” (Zou et al., 2023, Carnegie Mellon University / Center for AI Safety). Its authors demonstrated that it is possible to automatically generate text suffixes — strings of characters that appear meaningless — that, when appended to any request, cause the model to respond affirmatively instead of refusing. Most concerningly, these suffixes are transferable: they work not only on the model they were generated against, but also on ChatGPT, Claude, and other black-box models.
3. Multi-turn jailbreaks
Rather than attacking in a single message, the attacker gradually steers the conversation. Recent research such as MT-JailBench (Zhang et al., 2026) and D-Judge (Gong et al., ICML 2026) shows that multi-turn attacks are especially difficult to detect because the accumulated conversational context can “convince” the model that the request is legitimate.
4. Multilingual jailbreaks
A disturbing result that research has repeatedly confirmed: models are far more fragile in low-resource languages. The paper “Multilingual jailbreaking of LLMs using low-resource languages” (Marx and Dunaiski, 2026) demonstrates that rephrasing a request in minority languages can bypass filters that would block the same request in English. The explanation is that training data is much sparser in those languages, and therefore safety alignment is weaker.
5. Code-based attacks
Directly related to the Fable 5 case: researchers at Peking University recently published “Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code” (Zhang et al., June 2026), demonstrating that restricting model generation to formal grammars — common in programming assistants — can be used as an attack vector to obtain malicious code.
Why is it so hard to solve?
Here lies the core of the problem, and the reason why Anthropic was honest in admitting that “perfect resistance is probably not possible today for any provider.”
LLMs learn to generate useful and coherent text from vast amounts of human data. Safety alignment is, to some extent, an attempt to impose restrictions after the fact on a system not designed from scratch with security as a central principle. The research “Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks” (Kuo et al., ICML 2026) shows that even the most recent defenses for open-weight models can be broken with simple attacks.
There is also what some researchers call the safety paradox: making a model too safety-aware can create new vulnerabilities. The paper “Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack” (Hoang et al., 2026) argues that models with greater awareness of their own restrictions are more susceptible to certain targeted attacks.
The industry reference framework: OWASP Top 10 for LLMs
The OWASP Foundation — the nonprofit organization of reference in application security — has published the OWASP Top 10 for Large Language Model Applications, which includes jailbreaking and prompt injection as the most critical vulnerabilities in LLM-based systems. This list is the de facto standard that companies and developers use to assess the risk of their AI systems.
What defenders are doing
The defensive response is as active as the attacks. Some of the most promising lines of research include:
- Detection based on latent space trajectories: the work “Defending Jailbreak Attacks via Manifold Trajectory Kinetics” (Zhang et al., USENIX Security 2026) proposes detecting jailbreak attempts by analyzing how the model’s internal representation moves during prompt processing.
- Self-evolving safety memories: the Membrane system (Choi et al., 2026) builds a contrastive memory that the model updates with each newly detected attack.
- Re-triggering safeguards: Re-Triggering Safeguards within LLMs (Lin et al., 2026) proposes a method for the model itself to detect when it is being manipulated and reactivate its defense mechanisms.
- Standardized benchmarks: initiatives like Gate AI (Goehausen and Sousa, 2026) work to create rigorous evaluation methodologies that allow reproducible comparison of the robustness of different models and defenses.
What the Fable 5 case tells us
Returning to the beginning: the U.S. government ordered the shutdown of Fable 5 and Mythos 5 because someone demonstrated a jailbreak technique. Anthropic argues it was a narrow, non-universal jailbreak, and that the same result can be obtained with other models without tricks. They may be technically correct. But the precedent is set: for the first time, a documented jailbreak has led to a government-ordered shutdown of a commercial model used by hundreds of millions of users.
The debate ahead is not just technical. It is political and philosophical: who has the authority to decide what a language model can and cannot do? How is risk calibrated when the same technique that allows extraction of sensitive information is also what developers use to find bugs in their code? There are no easy answers. But to participate in that debate with judgment, one must first understand how jailbreaking works. I hope this article has helped.
References and verified sources
- Zou, A. et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043. Carnegie Mellon University / Center for AI Safety.
- OWASP Foundation. OWASP Top 10 for Large Language Model Applications. Version 2025.
- Zhang, X. et al. (2026). MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks. arXiv:2605.11002.
- Gong, H. et al. (2026). D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting. ICML 2026. arXiv:2606.02640.
- Marx, D. and Dunaiski, M. (2026). Multilingual jailbreaking of LLMs using low-resource languages. arXiv:2605.18239.
- Zhang, Y., Lu, S. and Li, J. (2026). Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code. arXiv:2606.11817.
- Hoang, L.P. et al. (2026). Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack. arXiv:2606.05614.
- Kuo, K., Yadav, C. and Smith, V. (2026). Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks. ICML 2026. arXiv:2605.26526.
- Zhang, H. et al. (2026). Defending Jailbreak Attacks on Large Language Models via Manifold Trajectory Kinetics. USENIX Security 2026. arXiv:2606.07335.
- Choi, M. et al. (2026). Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defense. arXiv:2606.05743.
- Goehausen, R. and Sousa, M. (2026). Gate AI: LLM Security Benchmark Evaluation Methodology and Results. arXiv:2606.02959.