Encoder-decoder models: what they are, simply

In plain words

Think of an interpreter at a meeting. First they listen to the whole sentence. Then they speak it in the other language, glancing back at their notes as they go.

An encoder-decoder model works the same way. The encoder reads the entire input, say an English sentence or a recording, and turns it into a set of rich notes made of numbers. The decoder then writes the output one word at a time, and at every step it looks back at those notes.

This design was made for jobs that turn one sequence into another: translating, summarising, turning speech into text. Many of today's chatbots keep only the decoder half, and treat your message and their reply as one long text.

Everyday example

Ordering at a South Indian café

A waiter listens to our whole order: two masala dosas, one without onion, and filter coffee. That is the encoder: take in everything and understand it. In the kitchen, the cook makes dish after dish, checking the order slip each time. That is the decoder: produce the output piece by piece, always looking back at the notes.

Where the example breaks: The waiter's slip is a few words. The encoder's notes are long lists of numbers, one set per word, and the decoder does not read them like a slip: it uses attention to weigh which parts matter for each word it writes.

How it actually works

The idea came from machine translation in 2014 and became the shape of the original transformer in 2017.

2014: two recurrent networks

Two research groups described pairs of recurrent networks: one reads the source sentence into a single vector, the other writes the translation from it. Google's version was called sequence to sequence, or seq2seq.

The bottleneck and attention

Squeezing a whole sentence into one fixed vector lost detail on long sentences. Attention, added the same year, let the decoder look back at every encoder state instead of one summary.

The 2017 transformer

The original transformer had a stack of 6 encoder layers and 6 decoder layers. The encoder lets every input word look at every other, in both directions. The decoder looks only at words it has already written, plus the encoder's output.

Cross-attention links the halves

In each decoder layer, a cross-attention step takes its questions (queries) from the decoder and its keys and values from the encoder's output. This is how each new word checks the input.

Three families today

Encoder-only models such as BERT (2018) are good at understanding and classifying text, for example in search. Decoder-only models such as GPT power most chatbots. Full encoder-decoders such as T5 and OpenAI's Whisper speech model are still used for translation, summaries and speech recognition.

Named

"RNN Encoder-Decoder", Kyunghyun Cho and colleagues, June 2014

Seq2seq

Ilya Sutskever, Oriol Vinyals and Quoc Le at Google, September 2014

Original transformer

6 encoder plus 6 decoder layers, built for translation (Google, 2017)

A modern example

Whisper (OpenAI, 2022): an encoder-decoder transformer trained on 680,000 hours of audio

Questions people ask

Is ChatGPT an encoder-decoder?

No. The GPT models behind ChatGPT are decoder-only: they read your message and write the reply as one continuous text, predicting the next token each time. OpenAI's Whisper speech model, by contrast, is a full encoder-decoder.

What is the difference between an encoder and a decoder?

The encoder reads the input and builds understanding; every word can look at every other word. The decoder generates output one word at a time and can only look at what it has already written, plus the encoder's notes.

What is BERT, and why is it encoder-only?

BERT is a 2018 Google model whose name stands for Bidirectional Encoder Representations from Transformers. It keeps only the encoder, so it reads text in both directions to understand it, but it does not write long text. It suits search and classification.

What does seq2seq mean?

Sequence to sequence: turning one sequence, like a sentence in Hindi, into another, like the same sentence in English. Encoder-decoder models were invented for exactly this kind of job.

Why did chatbots drop the encoder?

For open-ended chat there is no clear line between input and output, so one decoder that keeps predicting the next word is simpler and scales well. Encoder-decoders still shine when the input and output are clearly separate, such as audio in and text out.