Dropout in neural networks, explained simply

In plain words

A neural network is made of many small units, artificial neurons, wired together. During training, units can form cosy partnerships: one learns to fix another's particular mistakes. That works on the training examples but often fails on new ones, which is overfitting.

Dropout breaks those partnerships. At every training step, a random set of units is switched off, often half of them in hidden layers, and the network has to get the answer right without them. Next step, a different random set is off.

Because any unit might vanish at any moment, each one has to be useful on its own. When the finished network is actually used, every unit is switched back on, with their signals scaled so the totals match what the network saw in training.

Everyday example

Practice without your star player

A cricket coach notices the team falls apart whenever the star batter is out early. So in practice matches, he randomly benches a few players each time, never the same ones. Everyone learns to step up and cover different roles. On match day the full team plays, and it is stronger because nobody depends on one person.

Where the example breaks: A coach chooses who sits out. Dropout is purely random, it happens at every one of thousands of training steps, and the units being switched off are just numbers, not players with skills of their own.

How it actually works

Dropout is a form of regularisation introduced by Geoffrey Hinton and colleagues in 2012 and set out in full by Srivastava and colleagues in 2014.

Drop at random

For each training example, every unit in a dropout layer is kept with some probability and otherwise set to zero for that step. Goodfellow et al. give typical keep rates of 0.8 for inputs and 0.5 for hidden units.

Train a crowd of thinned networks

Each random pattern is effectively a different, smaller network, and they all share the same weights. Training with dropout is like training a huge ensemble of these thinned networks at once.

Use the whole network

At prediction time nothing is dropped. The weights are scaled to make up for it, which approximates averaging the predictions of all those thinned networks. Goodfellow et al. describe dropout as making bagging, the ensemble trick behind random forests, practical for very large networks.

Why it helps

Units cannot rely on particular partners being present, so the original paper describes it as preventing "co-adaptation" of feature detectors. The result is less overfitting.

Tuning it

The dropout rate is a hyperparameter. Too much dropout slows learning and can cause underfitting; too little leaves overfitting in place.

Introduced

Hinton, Srivastava, Krizhevsky, Sutskever and Salakhutdinov, arXiv paper, July 2012

Full paper

"Dropout: A Simple Way to Prevent Neural Networks from Overfitting", Journal of Machine Learning Research, 2014

Typical rates

Keep about 80% of inputs and 50% of hidden units (Goodfellow et al.)

Reported gains

Vision, speech recognition, document classification and computational biology tasks

Questions people ask

What is dropout in simple terms?

A training trick for neural networks: at each step, randomly switch off some units so the network must work without them. It stops units depending too much on each other and reduces memorising of the training data.

Is dropout used when the model is making predictions?

Normally no. Dropout only happens during training. When the model is used, all units are on, and the signals are scaled so their overall size matches what the network got used to in training.

What is a good dropout rate?

It depends on the network and data, so it is tuned like other settings. Goodfellow et al. give typical keep probabilities of 0.8 for input units and 0.5 for hidden units, meaning 20% and 50% are dropped.

Is dropout a type of regularization?

Yes. Regularisation means any technique that reduces overfitting. Dropout does it by adding randomness during training rather than by adding a penalty on the weights.