BERT in AI: what it is, explained simply

In plain words

Think of the word "bank". In "I opened a bank account" it means money. In "we sat on the river bank" it means land. You only know which by reading around it.

BERT was built to do exactly that. It is a transformer model that reads a whole sentence at once, and for every word it looks both left and right. Earlier models mostly read in one direction, so they had only half the clues.

It learned by playing a fill-in-the-blank game on billions of words: hide a word, guess it from both sides. That gave it a deep feel for how words fit together.

BERT is a reader, not a writer. It does not chat. It is used to understand text: ranking search results, spotting names, answering questions from a passage, judging whether a review is positive. In 2019 Google started using it in Search.

Everyday example

Filling the blank in a school worksheet

A worksheet says: "Two cups of ____ and a plate of samosas, please." You don't just look at "Two cups of". You also read "samosas, please", and the answer, chai, jumps out. BERT trained on millions of blanks like this, using the words before and after each gap.

Where the example breaks: You know what chai and samosas are. BERT only knows which words tend to appear together in the text it trained on. It has no taste of either.

How it actually works

The name stands for Bidirectional Encoder Representations from Transformers. It is the encoder half of the original transformer, used on its own.

Pre-train on fill-in-the-blank

BERT picks 15% of the tokens in each passage. Most are replaced with a [MASK] symbol, some with a random word, some left as they are, and the model must predict the original word. This is self-supervised learning: the text supplies its own answers.

Learn how sentences connect

It also practised guessing whether sentence B really followed sentence A or was a random sentence. Training used about 3.3 billion words of English Wikipedia and books.

Fine-tune for a job

After pre-training, BERT is adapted with one small extra layer and a modest set of labelled examples: question answering, spotting names, classifying reviews. This is fine-tuning.

Encoder only, so it does not write

Every word sees every other word, in both directions. That is great for understanding, but there is no left-to-right writing step. Chatbot models like GPT are decoder-only and see only earlier words.

Released

Paper October 2018, code and models open-sourced by Google in November 2018

Who

Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google AI Language

Sizes

BERT-Base: 110 million parameters. BERT-Large: 340 million

In Google Search

October 2019: helped with about one in 10 English searches in the US, said Google

Questions people ask

What does BERT stand for?

Bidirectional Encoder Representations from Transformers. Bidirectional means it reads both ways around each word. Encoder means it is the reading half of a transformer. Representations means it turns words into numbers that capture their meaning in context.

What is the difference between BERT and GPT?

BERT reads, GPT writes. BERT looks at the whole sentence at once to understand it, so it suits search and classification. GPT reads left to right and predicts the next word, so it can generate text and power chatbots.

Is BERT a large language model?

By today's standards it is small. BERT-Large has 340 million parameters, while many modern large language models have billions. It is often called an early pre-trained language model, and it cannot hold a conversation.

Does Google still use BERT?

In 2019 Google said BERT was helping it understand about one in 10 English searches in the US. Google does not publish a full current list of models behind Search, so we cannot say exactly how it is used today.

Is BERT open source?

Yes. Google released the code and pre-trained models for free in November 2018, which is a big reason it spread so fast. Anyone can download it and fine-tune it for their own task.