Black October, explainedAppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Positional encoding

Also called position embedding, RoPE

Positional encoding stamps each word with its place in the sentence, so an AI that reads all words at once still knows their order.

Inside language modelsTechnical

In plain words

"Dog bites man" and "man bites dog" use exactly the same three words. Only the order tells us who is in trouble.

A transformer reads every word at the same moment rather than left to right. That makes it fast, but on its own it would see a sentence like a bag of word tiles: it would know which words are there, not where they sit.

Positional encoding fixes this. Before the model starts thinking, each word gets a small extra signal that says "I am word 1", "I am word 2", and so on. Now the same word in a different spot looks slightly different to the model, and order is back.

Everyday example

Seat numbers on a train ticket

A group of friends boards a train with tickets that carry only names. Everyone crowds in, and nobody knows who sits where. Add seat numbers, and the same friends now have fixed places: Riya in 12, Arjun in 13. Positional encoding gives every word a seat number, so the model knows not just who came, but in what order.

Where the example breaks: A seat number is a plain label. In a model, position is mixed into the word's numbers themselves, and modern methods mostly tell the model how far apart two words are rather than their exact seat.

Go deeper

How it actually works

Attention compares every word with every other, but by itself it ignores order: shuffle the words and each word's result stays the same. Position has to be added.

  1. The 2017 recipe: waves added to each word

    The original transformer added a position vector to each word's embedding. Each number in it comes from a sine or cosine wave of a different length, some changing fast from word to word, some slowly. Together they give every position a unique pattern.

  2. Why waves

    The authors chose waves because the pattern for "5 words later" can be reached from the pattern for now by the same simple, fixed transformation (a rotation), wherever in the text we are. They hoped this would let the model learn relative distance easily, and might let it handle longer text than it saw in training.

  3. Learned positions

    BERT (Google, 2018) instead learned one position vector per slot, up to 512. It works well, but there is simply no vector for slot 513, so the model cannot go past its trained length.

  4. Rotary position embedding (RoPE)

    Introduced in 2021, RoPE rotates each word's query and key vectors by an angle that grows with position. When two words are compared, only the difference in their angles survives, so attention naturally depends on how far apart they are. Meta's Llama 3 models use it.

  5. Stretching to longer texts

    Other methods such as ALiBi (2021) penalise attention between distant words instead. Researchers also found that squeezing new, longer positions into the range a RoPE model already knows lets it read far longer text after brief extra training. Tricks like these help today's long context windows.

First used in transformers
June 2017, "Attention Is All You Need" (Google): sine and cosine waves added to the input
Learned vs fixed
The 2017 paper tried both and found they "produced nearly identical results"
RoPE
Jianlin Su and colleagues, April 2021, in the RoFormer paper
Context stretching
Position interpolation (Meta, 2023) extended Llama models to 32,768 tokens with under 1,000 steps of fine-tuning

Questions people ask

Why does a transformer need positional encoding?

Because it reads all words in parallel and attention on its own treats the input as an unordered set. Without a position signal, "dog bites man" and "man bites dog" would look the same. Older recurrent networks got order for free by reading one word at a time.

What is RoPE in simple terms?

Rotary position embedding. Picture each word's numbers as a clock hand that is turned a little more for each later position. When the model compares two words, what matters is the angle between their hands, which tells it how far apart they are.

Is positional encoding the same as an embedding?

No. The embedding says what a word means. The positional encoding says where it sits. In the original design the two are simply added together into one vector. In RoPE, position is applied inside attention instead.

Does positional encoding limit how long a text can be?

It can. Learned positions stop at the last slot trained. Wave and rotary methods can in principle go further, but models often do poorly on positions far beyond their training, which is why extra tricks and training are used to extend context windows.

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked October 2026.