What is gradient descent?

Gradient descent is a method for reducing error by using derivatives to decide which way to change a model’s parameters. It treats the error function, also called a loss function, as a surface: higher places mean worse predictions, lower places mean better ones. At the current position, the derivative tells the algorithm which direction makes the error rise fastest. Gradient descent moves the other way, then repeats until further changes no longer reduce the error in a useful way. Imagine rolling a ball down a hill until it settles at the bottom. That is gradient descent in action, except the “hill” is mathematical and the movement is guided by calculus.

What the gradient tells you

A derivative measures how a function changes when its input changes. In gradient descent, the algorithm uses derivatives to read the slope of the error function.

For a model with many adjustable parameters, the gradient collects those slope readings together. It shows how the error would change if each parameter moved slightly. The gradient points uphill, towards a sharper increase in error. To reduce error, the algorithm moves downhill, in the opposite direction.

This is why gradient descent is useful. It does not try every possible setting. It asks a more focused question: from where the model is now, which small change is most likely to lower the error?

Steps, learning rate, and settling

Each update has a direction and a size. The gradient gives the direction. The learning rate controls the size of the step.

If the learning rate is too large, the algorithm can jump past the low point and keep bouncing around it. If it is too small, progress can become slow because each update changes little. The right setting depends on the shape of the error function and how the model and data are scaled.

When the updates keep reducing the error, the process is said to converge. Convergence does not always mean the algorithm has found the absolute lowest point anywhere on the whole surface. Some error surfaces have several valleys. Gradient descent may settle in a nearby low point rather than the global lowest point. In many practical cases, a low enough error is still useful.

Why derivatives matter

Without derivatives, reducing error would involve guesswork or broad search. Derivatives give the algorithm local information: they show what happens to the error when the model changes from its current position.

That local information is enough to guide a long sequence of improvements. Each step is modest, but the repeated effect can turn a poorly tuned model into a better one. This is the core idea behind training many machine learning models: define an error function, measure its gradient, adjust the parameters, and repeat.

The method also explains why data, model design, and scaling matter. Gradient descent can only follow the surface it is given. If the error function is shaped awkwardly, or if the inputs make some directions dominate others, the path down can be harder to follow.

Where gradient descent is used

Gradient descent appears wherever a system needs to minimise an error function with adjustable parameters. In machine learning, that includes fitting models to data, training neural networks, and tuning systems that make predictions or classifications.

Its strength is not magic. It is a disciplined way to improve a model by turning “make the error smaller” into a calculable sequence of steps. Derivatives provide the direction, the learning rate controls the movement, and the error function defines what “better” means.