In plain words
Many useful jobs have the same shape: something goes in as a sequence, something else comes out as a sequence. A Hindi sentence in, an English sentence out. A long article in, a three-line summary out. Speech in, text out.
The tricky part is that the two sides rarely line up word for word. "Main chai peeta hoon" is four words. "I drink tea" is three, in a different order. Input and output can differ in length and order.
A sequence-to-sequence model, or seq2seq, handles this in two parts. One part reads the whole input and builds an understanding of it. The other part writes the output one word at a time, from that understanding, until it decides to stop. This two-part build is called an encoder-decoder.
Everyday example
A court interpreter
A witness speaks a long sentence in Tamil. The interpreter listens to the whole thing first, then says it in English, in English word order, sometimes with more words, sometimes fewer. They don't translate word by word as it comes. Seq2seq works the same way: read everything, then write the new version.
Where the example breaks: An interpreter understands the witness. A seq2seq model learns which output patterns tend to follow which inputs, from millions of examples, so it can produce fluent sentences that are wrong.
How it actually works
The idea took off with a 2014 Google paper that showed one neural network could learn translation end to end, with no hand-built rules.
Encode the input
In the 2014 design, an encoder, a type of recurrent neural network called an LSTM, read the source sentence word by word and squeezed it into a single list of 8,000 numbers.
Decode the output
A second LSTM started from those numbers and wrote the translation one word at a time. Each output sentence ends with a special end-of-sentence token, which lets the model produce outputs of any length.
Fix the bottleneck with attention
Squeezing a long sentence into one fixed list loses detail. The same year, attention let the decoder look back at every input word while writing, which worked much better on long sentences.
Transformers take over
In 2017 the transformer replaced recurrent networks with attention everywhere. It was first built as an encoder-decoder for translation. Models like Google's T5 later treated every text task as text in, text out.
- Key paper
- "Sequence to Sequence Learning with Neural Networks", Sutskever, Vinyals and Le, Google, September 2014
- Built at the same time
- An RNN encoder-decoder by Cho, Bengio and colleagues, June 2014
- Trick that helped
- Feeding the source sentence in reverse word order made the 2014 model learn much better
- Into products
- Google Translate switched Chinese to English to a neural seq2seq system with attention in September 2016
Questions people ask
What is seq2seq used for?
Jobs where text or sound goes in and different text comes out: translation, summarising, speech to text, answering questions, and fixing grammar. The output length is not fixed in advance, which is the point.
Is seq2seq the same as an encoder-decoder?
Nearly. Seq2seq names the job: map one sequence to another. Encoder-decoder names the usual build for it: one part reads, one part writes. People often use the two terms interchangeably.
Is ChatGPT a sequence-to-sequence model?
Are seq2seq models still used?
Yes, in transformer form. OpenAI's Whisper, a speech-to-text model from 2022, is an encoder-decoder transformer, and Google's T5 treats tasks like summarising as text in, text out. The older LSTM versions are mostly historical now.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change
Further reading
- Sequence to Sequence Learning with Neural Networks (Sutskever, Vinyals, Le)arXiv, 2014
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (Cho et al.)arXiv, 2014
- A Neural Network for Machine Translation, at Production ScaleGoogle Research, 2016
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)arXiv, 2019