In plain words
Language, speech and stock prices all come as sequences: order matters. An ordinary neural network looks at one input in isolation. A recurrent neural network, or RNN, is built for sequences.
It reads the first word, updates a small internal memory, then reads the next word together with that memory, and so on. The same network is used at every step, passing its memory forward like a note to itself. By the end, the memory holds a summary of the whole sentence.
The weakness is that the note gets smudged. Early words fade as more arrive, so long passages are hard. Improved versions like the LSTM held on longer, and for years RNNs powered translation and voice assistants. Since 2017 the transformer, which reads all words at once, has largely replaced them for language.
Everyday example
Listening to an address over the phone
A friend reads out a long address while you listen: flat number, building, lane, landmark, area, PIN code. You keep a running picture in your head and update it with each bit. By the PIN code the flat number is getting hazy. An RNN works like that, one piece at a time, with a memory that fades for early parts.
Where the example breaks: You can ask your friend to repeat. A basic RNN only sees each piece once, in order, and its memory is a fixed-size list of numbers, so it cannot go back to check an early word.
How it actually works
Google's glossary describes an RNN as a network that is run many times, where hidden layers from each run feed into the same hidden layer in the next run.
One step at a time
At each step the network takes the current input, such as a token, plus its previous hidden state, and produces a new hidden state:
new memory = f(current word, old memory). The same weights are reused at every step.Training through time
To train it, the network is unrolled across the sequence and backpropagation runs back through every step. Over long sequences the error signal shrinks or explodes, the vanishing and exploding gradient problems.
LSTM and GRU
Long short-term memory (LSTM) adds a separate memory cell with gates that decide what to keep, forget and output, which lets information survive far longer. The GRU is a simpler cousin.
Where RNNs worked
Through the 2010s, LSTMs powered machine translation, speech recognition, handwriting recognition and keyboard suggestions, including in Google Translate in 2016.
Why transformers took over
An RNN must process words in order, which is slow to train on modern GPUs. The transformer uses attention to look at all words at once, trains in parallel and keeps long-range context better. Some newer designs revisit recurrence for efficiency.
- Influential early designs
- Jordan network (1986) and Elman network (1990)
- LSTM
- Sepp Hochreiter and Jürgen Schmidhuber; standard reference published in Neural Computation, 1997
- Good for
- Sequences: text, speech, time series, handwriting
- Main limit
- Fading memory over long sequences and slow, step-by-step training
Questions people ask
What is an RNN in simple terms?
A neural network that reads a sequence one step at a time and keeps a running memory, so each step is understood in light of what came before. It is like reading a sentence word by word while holding the gist in mind.
What is the difference between an RNN and a transformer?
An RNN reads words one after another and squeezes the past into a single memory, so early words fade. A transformer looks at all the words at once and uses attention to link any word to any other, which handles long text better and trains much faster.
What is an LSTM?
Long short-term memory, an improved RNN cell with gates that decide what to remember, what to forget and what to pass on. It keeps information over much longer sequences than a plain RNN and was the standard for language tasks before transformers.
Are RNNs still used?
Less for language than before, but still in some speech, time series and on-device tasks where processing a stream step by step is useful. Researchers also keep exploring new recurrent designs that aim to match transformers while using less memory.