Black October, explainedAppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Jailbreak

Also called jailbreaking, DAN prompt

A jailbreak is a trick that gets a chatbot to break its own safety rules, usually by disguising a request it would normally refuse.

Safety and societyEveryday

In plain words

Chatbots are trained to refuse certain requests, such as help with weapons or hacking. A jailbreak is any wording that gets around that refusal. The name comes from jailbreaking phones, which means removing the maker's restrictions.

Most jailbreaks work by disguise. The request is wrapped in a pretend story, framed as a hypothetical, split into harmless-looking pieces, or written in an unusual language or code. The model's safety training learned to spot the plain version, so the disguised one can slip past.

AI makers treat jailbreaks as bugs. They collect them, retrain models to recognise the trick, and add separate filters that check requests and answers. Attackers then find new disguises, so it is an ongoing contest rather than a solved problem.

Everyday example

Talking past the society guard

A housing society guard is told: no delivery agents upstairs. Someone says, "I'm not delivering, I'm just visiting my cousin on floor 4," while holding a parcel. The guard checks for uniforms, not intentions, so he lets them in. The society then updates the rule: anyone with a parcel waits at the gate. Jailbreaks and their fixes follow the same back-and-forth.

Where the example breaks: A guard can use common sense and ask questions. A chatbot only matches patterns it learned in training, which is why oddly worded requests can fool it in ways that would not fool a person.

Go deeper

How it actually works

We describe the kinds of jailbreak here, never working examples. Researchers study them so they can be fixed.

  1. Why safety training can be bypassed

    A 2023 study named two root causes. Competing objectives: the model is trained both to be helpful and to be safe, and a clever request sets one against the other. Mismatched generalisation: the model's skills reach places its safety training never covered, like rare languages or encoded text.

  2. Hand-written tricks

    Early jailbreaks spread on forums from late 2022, such as the "DAN" (do anything now) role-play prompts. A 2023 study collected 1,405 jailbreak prompts from 131 online communities. Common patterns: role-play, fake authority, hypotheticals and splitting a task into steps.

  3. Automated attacks

    Researchers showed in 2023 that a computer search could find strings of odd characters that, added to a request, made several chatbots comply. Some strings worked across models from different companies.

  4. Defence: training and filters

    Makers retrain models on known jailbreaks and add guardrails: separate classifiers that check the request and the answer. In a 2025 test of one such system, Anthropic reported jailbreak success falling from 86% to 4.4%, with refusals of normal requests rising by 0.38%.

  5. Defence: testing and monitoring

    Red teams hunt for new jailbreaks before launch, bug bounty programmes pay outsiders who find them, and companies watch for abuse after launch. The 2026 International AI Safety Report notes safeguards still sometimes fail when a request is rephrased or split up.

Became widespread
December 2022, soon after ChatGPT launched
Related attack
Prompt injection. Security group OWASP counts jailbreaking as a form of it
A public stress test
In 2025, 183 people spent over 3,000 hours trying to break Anthropic's prototype filter without finding a universal jailbreak. In a later week-long public demo, four people passed all eight levels, one with a universal jailbreak
Open models
With open-weight models, people can retrain away safety behaviour entirely, so jailbreak defences apply mainly to hosted chatbots

Questions people ask

Is jailbreaking ChatGPT illegal?

Usually it is not a crime by itself in most countries, but it typically breaks the service's terms of use and can get an account banned. Using the output to cause harm can be illegal. We are not lawyers; rules vary by country.

What is the difference between a jailbreak and prompt injection?

A jailbreak is usually the user tricking the model into breaking its safety rules. Prompt injection is hidden instructions, often planted in a web page or document, that hijack an AI app working for someone else. The terms overlap and are often mixed up.

Why can't AI companies just block all jailbreaks?

Because language is flexible. There are endless ways to phrase or disguise a request, and the model's understanding reaches further than its safety training. Each fix closes known gaps, but new ones keep being found.

What was the DAN prompt?

"DAN", short for do anything now, was a family of role-play prompts shared online from late 2022 that asked ChatGPT to pretend it had no rules. Versions kept changing as OpenAI patched them. Today it is mainly a historical example.

Do jailbreaks still work?

Sometimes. Major chatbots resist the well-known tricks much better than in 2023, and many use extra filters. But researchers and the 2026 International AI Safety Report say determined attackers still find ways through.

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked October 2026.