Exploring the shared language of gradient descent and Newton’s movement equations

I bear in mind the primary course on Machine Studying I took throughout undergrad as a physics scholar within the college of engineering. In different phrases, I used to be an outsider. Whereas the professor defined the backpropagation algorithm through gradient descent, I had this considerably obscure query in my head: “Is gradient descent a random algorithm?” Earlier than elevating my hand to ask the professor, the non-familiar setting made me assume twice; I shrunk slightly bit. Out of the blue, the reply struck me.

This is what I assumed.

To say what gradient descent is, first we have to outline the issue of coaching a neural community, and we will do that with an summary of how machines be taught.

Overview of a neural community coaching

In all supervised neural community duties, we have now a prediction and the true worth. The bigger the distinction between the prediction and the true worth, the more severe our neural community is with regards to predicting the values. Therefore, we create a operate referred to as the loss operate, often denoted as L, that quantifies how a lot distinction there may be between the precise worth and the expected worth. The duty of coaching the neural community is to replace the weights and biases (for brief, the parameters) to reduce the loss operate. That is the large image of coaching a neural community, and “studying” is solely updating the parameters to suit precise information greatest, i.e., minimizing the loss operate.

Optimizing through gradient descent

Gradient descent is likely one of the optimization strategies used to calculate these new parameters. Since our activity is to decide on the parameters to reduce the loss operate, we’d like a criterion for such a alternative. The loss operate that we are attempting to reduce is a operate of the neural community output, so mathematically we specific it as L = L(y_nn, y). However the neural community output y_nn additionally relies on its parameters, so y_nn = y_nn(θ), the place θ is a vector containing all of the parameters of our neural community. In different phrases, the loss operate itself is a operate of the neural networks’ parameters.

Borrowing some ideas from vector calculus, we all know that to reduce a operate, you could go towards its gradient, because the gradient factors within the route of the quickest enhance of the operate. To realize some instinct, let’s check out what L(θ) would possibly appear like in Fig. 1.

Right here, we have now a transparent instinct of what’s fascinating and what’s not when coaching a neural community: we wish the values of the loss operate to be smaller, so if we begin with parameters w1 and w2 that lead to a loss operate within the yellow/orange area, we wish to slide right down to the floor within the route of the purple area.

This “sliding down” movement is achieved via the gradient descent technique. If we’re positioned on the brightest area on the floor, the gradient will proceed to level up, since it is the route of maximally quick enhance. Then, stepping into the wrong way (therefore, gradient descent) creates a movement onto the area of maximally quick lower.

To see this, we will plot the gradient descent vector, as displayed in Fig. 2. On this determine, we have now a contour plot that exhibits the identical area and the identical operate displayed in Fig. 1, however the values of the loss operate at the moment are encoded into the colour: the brighter, the bigger.

We will see that if we decide a degree within the yellow/orange area, the gradient descent vector factors within the route that arrives the quickest within the purple area.

A pleasant disclaimer is that often a neural community might comprise an arbitrarily giant variety of parameters (GPT 3 has over 100 billion parameters!), which implies that these good visualizations are utterly unpractical in real-life functions, and parameter optimization in neural networks is often a really high-dimensional downside.

Mathematically, the gradient descent algorithm is then given by

Right here, θ_(n+1) are the up to date parameters (the results of sliding down the floor of Fig. 1); θ_(n) are the parameters that we began with; ρ known as the training fee (how massive is the step in direction of the route the place the gradient descent is pointing); and ∇L is the gradient of the loss operate calculated on the preliminary level θ_(n). What provides the identify descent right here is the minus check in entrance of it.

Arithmetic is vital right here as a result of we’ll see that Newton’s Second Legislation of Movement has the identical mathematical formulation because the gradient descent equation.

Newton’s second regulation of movement might be probably the most essential ideas in classical mechanics because it tells how drive, mass, and acceleration are tied collectively. Everyone is aware of the highschool formulation of Newton’s second regulation:

the place F is the drive, m is the mass, and a the acceleration. Nevertheless, Newton’s unique formulation was by way of a deeper amount: momentum. Momentum is the product between the mass and the speed of a physique:

and will be interpreted because the amount of motion of a physique. The concept behind Newton’s second regulation is that to vary the momentum of a physique, you could disturb it one way or the other, and this disturbance known as a drive. Therefore, a neat formulation of Newton’s second regulation is

This formulation works for each drive you’ll be able to consider, however we wish slightly extra construction in our dialogue and, to realize construction, we have to constrain our area of potentialities. Let’s speak about conservative forces and potentials.

Conservative forces and potentials

A conservative drive is a drive that doesn’t dissipate power. It implies that, after we are in a system with solely conservative forces concerned, the entire power is a continuing. This should sound very restrictive, however in actuality, probably the most elementary forces in nature are conservative, comparable to gravity and electrical drive.

For every conservative drive, we affiliate one thing referred to as a potential. This potential is said to the drive through the equation

in a single dimension. If we take a more in-depth take a look at the final two equations introduced, we arrive on the second regulation of movement for conservative fields:

Since derivatives are type of difficult to take care of, and in laptop sciences we approximate derivatives as finite variations anyway, let’s change d’s with Δ’s:

We all know that Δ means “take the up to date worth and subtract by the present worth”. Therefore, we will re-write the system above as

This already seems fairly just like the gradient descent equation proven in some traces above. To make it much more comparable, we simply have to take a look at it in three dimensions, the place the gradient arises naturally:

We see a transparent correspondence between the gradient descent and the formulation proven above, which utterly derives from Newtonian physics. The momentum of a physique (and you’ll learn this as velocity in case you want) will all the time level towards the route the place the potential decreases the quickest, with a step dimension given by Δt.

So we will relate the potential, throughout the Newtonian formulation, to the loss operate in machine studying. The momentum vector is just like the parameter vector, which we are attempting to optimize, and the time step fixed is the training fee, i.e., how briskly we’re transferring in direction of the minimal of the loss operate. Therefore, the same mathematical formulation exhibits that these ideas are tied collectively and current a pleasant, unified approach of them.

If you’re questioning, the reply to my query at first is “no”. There isn’t any randomness within the gradient descent algorithm because it replicates what nature does day by day: the bodily trajectory of a particle all the time tries to discover a option to relaxation within the lowest doable potential round it. When you let a ball fall from a sure hight, it should all the time have the identical trajectory, no randomness. While you see somebody on a skateboard sliding down a steep ramp, bear in mind: that is actually nature making use of the gradient descent algorithm.

The way in which we see an issue might affect its resolution. On this article, I’ve not proven you something new by way of laptop science or physics (certainly, the physics introduced right here is ~400 years previous), however shifting the angle and tying (apparently) non-related ideas collectively might create new hyperlinks and intuitions a few matter.

[1] Robert Kwiatkowski, Gradient Descent Algorithm — a deep dive, 2021.

[2] Nivaldo A. Lemos, Analytical Mechanics, Cambridge College Press, 2018.