AppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Content moderation

Also called ai moderation

Content moderation is checking what people post, or what an AI writes, against a set of rules, and removing, labelling or blocking what breaks them.

Safety and societyEveryday

In plain words

Every big app has rules: no threats, no scams, no sexual content involving children, and many more. Content moderation is the work of enforcing those rules. Billions of posts arrive every day, far more than people could ever read, so most of the first check is done by AI.

A moderation model reads a post or looks at an image and gives it a score for each kind of harm. Clear cases are handled automatically. Unclear ones go to trained human reviewers, who make the call and whose decisions help train the next model.

Chatbots use the same idea twice: once on what you type, and again on what the model writes back, before you see it.

Mistakes go both ways. Too strict, and jokes, news and harmless posts disappear. Too loose, and harm spreads. Every platform is choosing where to draw that line.

Everyday example

A school exam with a strict invigilator

In a big exam hall, a camera system flags anyone who looks at a neighbour's paper. Most flags are quickly cleared by the invigilator: the student was just stretching. A few are real, and those papers are taken. The camera does the watching at scale, a person makes the hard calls, and the rulebook decides what counts as cheating.

Where the example breaks: Exam rules are short and the same everywhere. Platform rules are long, change often, and must work across languages, cultures and laws, so the same post can be fine in one country and illegal in another.

Go deeper

How it actually works

Modern moderation is a pipeline of classifiers, rules and people, tuned for speed on clear cases and care on hard ones.

  1. Write the policy

    Everything starts with written rules: categories such as hate, harassment, self-harm, violence and sexual content, with examples of what is and is not allowed. A model can only be as clear as the policy it was trained on.

  2. Score every item

    A classifier, often a fine-tuned language model, gives each post, image or chat message a score per category. OpenAI's free moderation model, for example, scores text and images across 13 categories. Meta's Llama Guard checks both the user's prompt and the chatbot's reply.

  3. Act by threshold

    High scores trigger automatic removal or blocking. Middle scores may be hidden, labelled, down-ranked or sent for review. Where each threshold sits is a choice between catching more harm and wrongly removing more harmless posts, the same trade-off as precision and recall.

  4. Humans review the hard cases

    Reviewers judge context the model misses: satire, news reporting, reclaimed slang. Their decisions feed back as labelled data. This work can be harmful to the reviewers: in 2024, 81% of 144 Facebook moderators assessed in Kenya were classed as having severe PTSD, in medical reports filed in a lawsuit against Meta and a contractor.

  5. Appeals and audits

    Users can usually appeal. Meta said in January 2025 that one to two in every 10 of its content removals in December 2024 may have been mistakes, and it began using language models as a second opinion before acting.

Two places it runs
On platforms (posts, comments, ads) and inside AI tools (guardrails on prompts and replies)
India
Amended IT Rules in force from 20 February 2026 require platforms to act on government or court orders within 3 hours (earlier 36) and to label AI-generated content
Meta's 2025 shift
Automated systems focused on illegal and high-severity posts; US fact-checking replaced by Community Notes (January 2025)
Hardest part
Context: the same words can be a threat, a quote, a joke or a news report

Moderation decides what is allowed on a service. Whether that is too much or too little is a public debate, not a technical one.

Should platforms moderate more, or less?

One view

More: harm spreads fast

Scams, harassment and abuse material cause real damage within hours. Platforms profit from attention, so rules and strong automated filters are needed to protect users, especially children.

The other view

Less: mistakes silence people

Automated filters wrongly remove lawful speech at huge scale, and fast takedown deadlines push platforms to delete first and check later. That can silence journalists, minorities and critics.

Most likely: Both sides agree illegal content like child abuse material must go. The real dispute is about the large grey zone of legal but harmful posts, and about who decides: companies, governments or users. Different countries are settling it differently.

Questions people ask

Is content moderation done by AI or by humans?

Both. AI handles most of the volume, scoring and removing clear cases in milliseconds. Humans write the rules, review unclear cases and appeals, and label examples that train the next model. Removing humans entirely tends to raise mistakes.

Why was my post removed when it broke no rules?

Automated filters judge words and images without full context, so sarcasm, news or a quote can look like a violation. Most platforms let you appeal, and a human or a second model takes another look.

How do chatbots like ChatGPT block harmful requests?

The model itself is trained to refuse some requests, and separate moderation classifiers check both the prompt and the reply. Together these form the tool's guardrails. Determined users sometimes get around them with a jailbreak.

Is content moderation the same as censorship?

Not exactly. Censorship usually means a government suppressing speech. Moderation is a service enforcing its own rules, sometimes because a law requires it. The line blurs when governments order removals, which is why takedown laws are debated.

What are India's rules on removing content?

Under the IT Rules as amended from 20 February 2026, platforms must act on government or court orders within three hours and label AI-generated content. As of November 2026 these rules apply alongside the IT Act.

Was this clear?

1 is confusing, 5 is crystal clear. One tap, no account needed.

Spotted something wrong, or know a better example? Suggest a change

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked November 2026.