In plain words
Computers can't do maths on words. word2vec gives every word a position on a huge map, written as a list of numbers. Words with similar meanings land near each other: "chai" near "coffee", "Monday" near "Tuesday".
How does it know? It never reads a dictionary. It reads billions of words of ordinary text and learns from one simple clue: words that keep the same company tend to mean similar things. "Chai" and "coffee" both appear next to "cup", "hot" and "morning".
The surprise was that directions on the map carry meaning too. The step from "man" to "woman" is roughly the same as the step from "king" to "queen". So king minus man plus woman lands closest to queen.
Everyday example
Guessing a stranger's job from their friends
At a wedding you meet someone and hear who they hang out with: the stethoscope crowd, the night-shift group, the hospital canteen regulars. You can guess they work in a hospital without being told. word2vec places each word the same way, by the words it keeps appearing with, until similar words share a neighbourhood.
Where the example breaks: You can ask the stranger to check your guess. word2vec can't: it only sees the text, so it also soaks up the biases in that text, and it gives a word one position even when it has several meanings.
How it actually works
word2vec is a small, shallow neural network trained on a guessing game. The game itself is thrown away. What we keep is the numbers it learned for each word, now called word embeddings.
Slide a window over text
Take a sentence and look at each word with a few neighbours on either side. Every window gives a small training example. Wider windows gave better vectors in the original paper, but took longer to train.
Play one of two guessing games
In CBOW (continuous bag of words), the network guesses the middle word from its neighbours. In skip-gram, it does the reverse: guesses the neighbours from the middle word.
Nudge the numbers
Each word starts with random numbers, often 300 of them. After every guess, the numbers are adjusted slightly by gradient descent so the right answer becomes likelier. Shortcuts like negative sampling, and skipping some very common words, made training much faster.
Use the map
Words that appear in similar contexts end up with similar numbers. That can power "more like this" search, recommendations and features for other models. Simple sums on the vectors showed patterns like Paris minus France plus Italy is close to Rome.
- Published
- January and October 2013, by Tomas Mikolov and colleagues at Google
- Speed
- The first paper learned good vectors from 1.6 billion words in under a day
- Honour
- The October 2013 paper won the NeurIPS Test of Time award in 2023
- Known flaw
- A 2016 study found vectors trained on Google News encoded gender stereotypes about jobs
word2vec gives "bank" one position, whether it means money or a river. Later models like BERT give each word a different set of numbers in every sentence, based on context.
Questions people ask
What does word2vec do in simple terms?
It turns words into lists of numbers so a computer can measure how similar they are. Words that appear in similar sentences get similar numbers, so "doctor" and "nurse" sit closer than "doctor" and "mango".
Is word2vec the same as an embedding?
word2vec is one way to make embeddings, for single words. An embedding is the general idea: any list of numbers that captures meaning. Modern AI makes embeddings for whole sentences, images and more.
Is king minus man plus woman really queen?
Roughly. The original paper reported that the result lands closest to the vector for queen. Not every analogy works this cleanly, and results depend on the training text, but it showed that directions on the map can carry meaning.
Is word2vec still used?
It still appears in teaching and in some lightweight systems. For most language tasks, context-aware models have taken over, because they give a word different numbers in different sentences and so handle words with several meanings.
Is word2vec a large language model?
No. It is a small, shallow network that only learns one fixed vector per word. It cannot write or answer questions. Large language models are far bigger and read words in context, but they still begin by turning tokens into embeddings.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change
Further reading
- Efficient Estimation of Word Representations in Vector Space (Mikolov et al.)arXiv, 2013
- Distributed Representations of Words and Phrases and their Compositionality (Mikolov et al.)arXiv, 2013
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (Bolukbasi et al.)arXiv, 2016
- Test of Time Award: Distributed Representations of Words and Phrases and their CompositionalityNeurIPS, 2023