In plain words
When an app says "you might also like", or a feed seems to know your taste, a recommendation system is at work. People often just call it "the algorithm". Its job is to pick a handful of things for you out of millions.
It learns from behaviour: what you watched to the end, skipped, bought, liked or searched. Two simple ideas do much of the work. One: suggest things similar to what you already enjoyed, such as more cooking videos if you watch cooking videos. Two: find people whose choices overlap with yours and suggest what they liked that you have not tried yet.
Big services usually work in stages. First they quickly narrow a huge catalogue down to a few hundred candidates, then a more careful machine learning model scores and ranks them, then final rules mix things up so the list is not all the same.
Everyday example
The kirana shopkeeper
Your local shopkeeper notices you buy bread and eggs every Sunday, so he keeps them ready. He also knows that customers who buy what you buy often pick up a new brand of peanut butter, so he mentions it. One idea uses your own history; the other uses people like you. A recommendation system does both, for millions of customers at once.
Where the example breaks: The shopkeeper knows you as a person and can ask what you want. A recommendation system only sees clicks, views and purchases, so it can mistake curiosity for interest and keep pushing more of something you clicked once.
How it actually works
Google's course puts the goal simply: "A recommendation system helps users find compelling content in a large corpus", including things "users might not have thought to search for".
Content-based filtering
Describe each item by its features (genre, topic, maker, price) and recommend items similar to the ones a user liked. It needs only that user's own history.
Collaborative filtering
Use everyone's behaviour at once. Arrange it as a big table of users and items, mostly empty. The system learns an embedding, a list of numbers, for each user and item so that matching pairs score high. It can then suggest to you what a similar user enjoyed.
Candidate generation
Narrow the whole catalogue to a small set quickly. Google's course notes YouTube's candidate generator reduces billions of videos to hundreds or thousands.
Scoring and re-ranking
A heavier model ranks the candidates, for example by how likely you are to click or watch, and picks around ten to show. A final pass removes things you disliked and adds variety and freshness.
Known problems
New users and new items have no history, the cold start problem. A list driven only by predicted engagement can turn repetitive, which is why Google's course describes re-ranking for diversity, freshness and fairness.
- Two classic approaches
- Content-based filtering (similar items) and collaborative filtering (similar users)
- Typical stages
- Candidate generation, scoring, re-ranking
- An early system
- Wikipedia credits Grundy, built by Elaine Rich in 1979, as the first recommender system
- Famous contest
- The Netflix Prize (2006 to 2009) offered US$1 million for a 10% better film recommender, using about 100 million ratings
Questions people ask
What is a recommendation system in simple terms?
Software that picks what to show you, such as videos, songs, products or posts, by learning from what you did before and what people with similar tastes chose. It is what people usually mean by "the algorithm" in an app.
How does YouTube or Instagram decide what to show me?
In broad terms, recommendation systems narrow millions of items to a few hundred candidates, then rank them by how likely you are to watch, like or engage, using your history and similar users' behaviour. Each company's exact signals change and are only partly public.
What is the difference between content-based and collaborative filtering?
Content-based filtering recommends items similar to ones you liked, using the items' features. Collaborative filtering recommends what similar users liked, using everyone's behaviour, so it can suggest something that looks nothing like your past choices.
Why do I keep seeing the same kind of content?
Systems learn from what you engage with. If you watch a few videos on one topic, they predict you want more and show more, which leads to more watching. Searching for other topics, or marking things "not interested", gives the system different signals.
Is a recommendation system AI?
Yes. Modern ones are machine learning systems trained on huge amounts of behaviour data, often using neural networks and embeddings. Simple versions can also be hand-written rules, such as "bestsellers in this category".