Taylor Series Refresher: Descent Directions and Contour Tangents

September 2026 · 5 min read

Review the 1D and multivariate Taylor expansions, and use the Taylor formula to prove why the negative gradient is a descent direction and why the gradient is perpendicular to the contour tangent.

Every gradient-based optimizer in deep learning rests on one local approximation: if you nudge the parameters $w$ by a small step $\Delta w$, how does the loss $L(w+\Delta w)$ change? The Taylor series expansion answers that question and directly proves two geometric facts used across the entire series: why $-\nabla L(w)$ is a descent direction, and why $\nabla L(w)$ meets every constant-loss contour at a $90^\circ$ angle.

1. The 1D Taylor expansion: local height, slope, and curvature

For a smooth single-variable loss $L(w)$, the Taylor expansion around the current weight $w$ approximates the loss after a step $\Delta w$ as a sum of powers of $\Delta w$:

Subtracting $L(w)$ from both sides isolates the change in loss $\Delta L = L(w+\Delta w)-L(w)$:

When the step $\Delta w$ is small (for example, $|\Delta w|=10^{-3}$), its square $(\Delta w)^2=10^{-6}$ is a thousand times smaller than $|\Delta w|$. Grouping the squared and higher powers into $\mathcal{O}\bigl((\Delta w)^2\bigr)$ leaves the first-order Taylor approximation:

Worked 1D check: Let $L(w)=w^2$ at $w=3$ (so $L(3)=9$, slope $L'(3)=2(3)=6$, curvature $L''(3)=2$) and take a small step $\Delta w=-0.1$ to $w+\Delta w=2.9$:

2. Multivariate Taylor expansion for $d$ parameters

In a neural network, $w=(w_1,\dots,w_d)^\top\in\mathbb{R}^d$ is a column vector of $d$ parameters and $\Delta w=(\Delta w_1,\dots,\Delta w_d)^\top\in\mathbb{R}^d$ is the parameter update vector:

• The first derivative $L'(w)$ becomes the gradient vector $\nabla L(w)=\left[\frac{\partial L}{\partial w_1},\dots,\frac{\partial L}{\partial w_d}\right]^\top\in\mathbb{R}^d$.
• The second derivative $L''(w)$ becomes the symmetric $d\times d$ Hessian matrix $H=\nabla^2 L(w)$ with entries $H_{ij}=\frac{\partial^2 L}{\partial w_i\,\partial w_j}$.

Expanding $L(w+\Delta w)$ around $w$ adds each coordinate’s linear slope contribution $\frac{\partial L}{\partial w_j}\Delta w_j$, which writes compactly as the dot product $\nabla L(w)^\top\Delta w$:

Subtracting $L(w)$ gives the fundamental multivariate Taylor change formula:

3. Why the negative gradient $-\nabla L(w)$ is a descent direction

A direction vector $d\in\mathbb{R}^d$ is called a descent direction at $w$ if moving a small step $\Delta w=\eta d$ (with learning rate $\eta>0$) strictly decreases the loss: $L(w+\eta d)<L(w)$, or $\Delta L<0$.

Plugging $\Delta w=\eta d$ into the Taylor expansion formula gives:

When $\eta>0$ is sufficiently small, $\mathcal{O}(\eta^2)$ is negligible compared to $\eta$, so the sign of $\Delta L$ is completely determined by the dot product $\nabla L(w)^\top d$:

Now substitute the negative gradient $d=-\nabla L(w)$ (so $\Delta w=-\eta\nabla L(w)$) into the Taylor expansion. We get two results at once—an algebraic proof that $-\nabla L(w)$ is always a descent direction whenever $\nabla L(w)\ne 0$, and a geometric proof via the angle $\theta$ between $\nabla L(w)$ and $d$ that it is the steepest descent direction:

• Why $\Delta L<0$ is guaranteed: Every squared partial derivative $(\partial L/\partial w_j)^2\ge 0$. As long as $\nabla L(w)\ne 0$, at least one component is non-zero, so $\|\nabla L(w)\|^2>0$ and $-\eta\|\nabla L(w)\|^2<0$.
• Why $-\nabla L(w)$ is the steepest descent direction: Any direction $d$ making an obtuse angle $90^\circ<\theta\le 180^\circ$ with $\nabla L(w)$ has $\cos\theta<0$ and is a valid descent direction. Because $\cos\theta\in[-1,1]$ hits its most negative value $-1$ at $\theta=180^\circ$, pointing $d$ directly opposite to $\nabla L(w)$ produces the largest drop in loss per unit step.
• Why a large learning rate $\eta$ can still increase the loss: Keeping the second-order Taylor term for $\Delta w=-\eta g$ (with $g=\nabla L(w)$) gives
  $$L(w-\eta g)-L(w)\approx -\eta\|g\|^2+\frac{1}{2}\eta^2 g^\top H g.$$
  When $\eta$ is too large along a positively curved direction ($g^\top H g>0$), the quadratic term $+\frac{1}{2}\eta^2 g^\top H g$ grows faster than $-\eta\|g\|^2$ and makes $\Delta L>0$.

4. Using the Taylor formula to prove $\nabla L(w)$ is perpendicular to the contour tangent

A contour line (or level set) of the loss is the set of parameter points that share the exact same loss value:

Let $t\in\mathbb{R}^d$ be a unit tangent vector to the contour at $w$. If we take an infinitesimal step $\Delta w=\epsilon t$ of length $\epsilon$ along the contour tangent, we remain on the contour to first order, which means the loss does not change ($\Delta L=0$). Plugging $\Delta w=\epsilon t$ into the multivariate Taylor expansion proves that $\nabla L(w)$ and $t$ must meet at $90^\circ$:

Because $\nabla L(w)^\top t=0$ for every tangent vector $t$ to the contour at $w$, both the gradient $\nabla L(w)$ (pointing uphill, $\theta=0^\circ$) and the negative gradient $-\nabla L(w)$ (pointing downhill, $\theta=180^\circ$) are always perpendicular ($90^\circ$) to the contour tangent $t$.

Scroll the figure horizontally on small screens.

Worked 2D check: Consider the elliptical bowl $L(w_1,w_2)=w_1^2+2w_2^2$ at the point $w=(2,1)^\top$, which lies on the contour $L(w)=2^2+2(1)^2=6$. Parameterize a curve $w(\epsilon)=(2-\epsilon,\,1+\epsilon)^\top$ passing through $(2,1)^\top$ at $\epsilon=0$ with tangent vector $t=w'(0)=[-1,\,1]^\top$: