AppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Optimizer (Adam)

Also called adam optimizer, optimiser

An optimizer is the rule that decides how a model adjusts itself after each mistake. Adam, a common default, gives every setting its own step size.

How machines learnTechnical

In plain words

Training a model means nudging millions of internal numbers, a little at a time, so its mistakes shrink. Gradient descent tells us which direction to nudge. The optimizer is the rule that decides how far to move each number on each step.

The simplest rule moves every number by the same fixed fraction of its slope. That works, but it wobbles. Where the error surface is steep in one direction and gentle in another, it zig-zags across the steep part and creeps along the gentle part.

Adam fixes this with two memories. It keeps a running average of recent directions, so steps build momentum and stop wobbling. And it tracks how big each number's recent slopes have been, so numbers with huge, jumpy slopes take smaller steps and quiet ones take bigger steps. Every number gets its own step size.

Because it usually works well without much fiddling, Adam and its close cousin AdamW are the default choice for training most neural networks, including large language models.

Everyday example

Walking down a hill in fog

You are walking down a hill in thick fog and can only feel the slope under your feet. A careless walker takes the same size stride every time and keeps stumbling side to side in a ditch. A smarter walker remembers which way the last few steps went, keeps going that way, and takes shorter strides on jumpy ground. That walker is Adam.

Where the example breaks: A walker has two legs and one direction. A model has millions or billions of numbers, and Adam keeps a separate memory for every single one, which is why it needs extra memory while training.

Go deeper

How it actually works

Adam stands for adaptive moment estimation. It was published by Diederik Kingma and Jimmy Ba in December 2014 and designed to combine the strengths of two earlier methods, AdaGrad and RMSProp.

  1. Start from the gradient

    On each training step, backpropagation works out the gradient: for every parameter, how much the loss would change if that parameter moved. Plain gradient descent subtracts a fixed fraction of it.

  2. First memory: momentum

    Adam keeps an exponential moving average of past gradients, called the first moment. With the usual setting of 0.9, it is mostly the last ten or so steps. Directions that keep agreeing add up; directions that flip back and forth cancel out.

  3. Second memory: how jumpy each parameter is

    It also keeps a moving average of the squared gradients, the second moment, usually with a setting of 0.999. The step for each parameter is divided by the square root of this, so parameters with large gradients take smaller steps and rarely updated ones take larger steps.

  4. Correct the cold start

    Both averages start at zero, so early on they are too small. Adam divides each by a correction factor that fades away as training goes on. A tiny number, epsilon, is added to the divisor to avoid dividing by zero.

  5. AdamW: the version most models use

    In 2017 Ilya Loshchilov and Frank Hutter showed that the usual way of adding weight decay, a pull that keeps parameters small, does not behave as intended inside Adam. Their fix, AdamW, applies the decay separately and is now common for large models: Meta trained Llama 2 with it.

Published
December 2014 on arXiv, at the ICLR conference in 2015
Who
Diederik P. Kingma and Jimmy Ba
Usual settings
Learning rate 0.001, beta1 0.9, beta2 0.999, epsilon 0.00000001 (PyTorch defaults)
Memory cost
Two extra numbers stored for every parameter: the two moving averages
Family
Designed to combine AdaGrad and RMSProp; variants include AdaMax and AdamW

Stanford's CS231n course puts the practical advice plainly: Adam is recommended as the default, with SGD plus Nesterov momentum worth trying as an alternative.

Questions people ask

What is an optimizer in machine learning?

The rule that turns gradients into actual changes to a model's parameters. Gradient descent says which way to move; the optimizer decides how far, and whether to use memory of past steps. SGD, momentum, RMSProp and Adam are all optimizers.

What does Adam stand for?

Adaptive moment estimation. "Moments" are the two running averages it keeps: the average gradient and the average squared gradient. "Adaptive" means each parameter gets a step size adapted to its own history.

What is the difference between Adam and AdamW?

They differ only in how weight decay is applied. Adam usually folds it into the gradient, where Adam's scaling weakens it. AdamW applies it as a separate step, which is what was intended and tends to generalise better. Many large language models, such as Meta's Llama 2, were trained with AdamW.

Is Adam better than SGD?

Not always. Adam usually trains faster with less tuning, which is why it is the common default. Plain SGD with momentum, carefully tuned, can match or beat it on some tasks, especially image classification. Many teams try both.

Does Adam still need a learning rate?

Yes. Adam adapts step sizes per parameter, but the overall learning rate still matters. The usual starting value is 0.001, and large models often use a schedule that warms it up and then lowers it, as Llama 2 did.

Was this clear?

1 is confusing, 5 is crystal clear. One tap, no account needed.

Spotted something wrong, or know a better example? Suggest a change

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked November 2026.