In plain words
When a model trains, it adjusts millions of internal numbers on its own. Those are its parameters. But some choices have to be made by a person before training even begins: how big each adjustment should be, how many examples to look at at once, how many times to go through the data. Those choices are hyperparameters.
The name just means settings about the learning, one level above the parameters. The model cannot learn them from the data in the normal way, because they control how that learning happens.
Good settings matter. Too large a step and training goes haywire; too small and it crawls. So people try several combinations, train a model with each, and keep the one that scores best on data held back for checking. This search is called hyperparameter tuning.
Everyday example
Pressure cooker settings
Cooking dal, you decide the flame level, the amount of water and the number of whistles before you start. Those are your hyperparameters. What happens inside the cooker, the lentils softening, is the learning, and you do not control it directly. If the dal comes out hard, you change the settings next time: one more whistle, a little more water. Then you compare.
Where the example breaks: In a kitchen you judge by taste. In machine learning each new combination of settings means training a model again, which can take a long time, so people search cleverly rather than trying everything.
How it actually works
Google's crash course puts it simply: "hyperparameters are values that you control; parameters are values that the model calculates during training."
Common examples
For a neural network: the learning rate, the batch size, the number of epochs, the number of layers, and the dropout rate. For a decision tree: its maximum depth. For gradient boosting: the number of trees and their learning rate.
Grid search
List a few values for each setting and train every combination. Simple, but the number of runs multiplies quickly: four values for each of three settings is already 64 runs.
Random search
Pick combinations at random within sensible ranges. Bergstra and Bengio (2012) showed this is usually more efficient than a grid, because only a few settings really matter for a given problem, and random tries explore those more thoroughly.
Smarter searches
Adaptive methods use the results so far to choose the next combination to try, rather than picking blindly. Libraries such as scikit-learn automate grid and random search.
Keep the test honest
Compare settings on a validation set or with cross-validation, never on the final test set. Otherwise the settings are tuned to that test, and the final score looks better than the model really is.
- Set by
- People (or an automated search), before training
- Versus parameters
- Parameters, such as weights, are learned during training
- Three classics
- Learning rate, batch size and number of epochs (Google crash course)
- Search methods
- Grid search, random search, adaptive search
Questions people ask
What is the difference between a parameter and a hyperparameter?
Parameters, like the weights in a neural network, are learned by the model from data during training. Hyperparameters, like the learning rate, are chosen by people before training and control how that learning happens.
What is hyperparameter tuning?
Trying different combinations of settings, training a model with each, and keeping the combination that scores best on held-back validation data. It can be done by hand, by grid search, by random search, or by adaptive automated methods.
Is the learning rate a hyperparameter?
Yes, one of the most important. The learning rate sets how big each adjustment is during training. Too high and training becomes unstable; too low and it is slow or gets stuck.
Is temperature in ChatGPT a hyperparameter?
Not in the training sense. Temperature is a setting used when the model generates text, after training is finished. People sometimes loosely call it a hyperparameter, but it does not change what the model has learned.