Newton and Gauss–Newton: Use Curvature Without Forming an Inverse

September 2026 · 4 min read

Before you start: Gradient geometry and matrix multiplication.

Gradient descent chooses a direction from a slope. Newton’s method also asks how that slope changes. For a quadratic objective, that extra information can identify the minimizer in one exact step. For a nonlinear or nonconvex objective, the local quadratic is only an approximation, so damping and step acceptance matter.

Minimize a local quadratic

Here g is the gradient and H is the Hessian at the current point. If H is positive definite, the quadratic has a unique minimizer. For f(x,y)=(x²+10y²)/2, g=(x,10y) and H=diag(1,10), so Δ=(−x,−y) reaches the minimum directly. The inverse curvature cancels the coordinate scale difference.

An indefinite Hessian can make the local quadratic unbounded below, and a singular Hessian may not define a unique step. A raw Newton direction is then not guaranteed to decrease the objective. Line searches, trust regions, and damping provide ways to control the mismatch between the local model and the actual objective.

Gauss–Newton exploits residual structure

For nonlinear least squares, write $f(\theta)=\tfrac12\sum_i r_i(\theta)^2=\tfrac12\|r(\theta)\|^2$. Let J contain residual derivatives, $J_{ij}=\partial r_i/\partial\theta_j$. Linearize the residual: $r(\theta+\Delta)\approx r+J\Delta$. Minimizing that approximate squared norm gives the Gauss–Newton system.

The exact Hessian is $H=J^\top J+\sum_i r_i\nabla^2r_i$. Gauss–Newton drops the second term. It is especially plausible when residuals are small or residual functions are nearly linear, but those are conditions to assess rather than universal guarantees. $J^\top J$ is positive semidefinite; it need not be invertible.

One nonlinear scalar example

Let r₁(θ)=θ²−4 and r₂(θ)=θ−2. At θ=1, residuals are [−3,−1] and J=[2,1]ᵀ. Thus Jᵀr=−7 and JᵀJ=5, giving Δ=1.4. The new point θ=2.4 has objective 1.6288, down from 5, but it overshoots the exact zero-residual solution θ=2. This is a useful step, not a proof that one linearization solves the nonlinear problem.

At θ=0, the first residual has zero derivative even though its value is −4. A local linearization cannot see its immediate improvement direction. This illustrates a structural limitation: curvature approximations cannot recover information the local model discards.

Damping interpolates toward smaller steps

In the example with λ=2, Δ=7/7=1, landing at θ=2. That convenient outcome is specific to the numbers. Generally, increasing λ reduces the step and improves conditioning; for large λ the direction approaches a scaled negative gradient. An adaptive rule should compare actual reduction with the local model’s predicted reduction, increasing damping after poor agreement.

import numpy as np

def damped_gauss_newton_step(residuals, jacobian, damping):
    if damping <= 0:
        raise ValueError("Use positive damping in this example")
    j = np.asarray(jacobian, dtype=float)
    r = np.asarray(residuals, dtype=float)
    system = j.T @ j + damping * np.eye(j.shape[1])
    return np.linalg.solve(system, -(j.T @ r))

step = damped_gauss_newton_step([-3, -1], [[2], [1]], 2)
assert np.allclose(step, [1])

Large models need products, not dense curvature matrices

A model with P parameters cannot generally store a P×P Hessian. Matrix-free methods compute products such as Jv and Jᵀu using automatic differentiation and solve iteratively. Approximate solves add another error source; monitor residuals and total work. A small number of powerful curvature steps can still cost more than many first-order steps.

Generalized Gauss–Newton extends the idea beyond squared residuals using the curvature of an outer loss with respect to model outputs. Do not equate every gradient outer product with the full Gauss–Newton matrix. The next chapter carefully distinguishes a paper’s rank-one game preconditioner from this standard least-squares construction.

Sources and further reading

• SciPy nonlinear least-squares documentation — Trust regions, Jacobians, and nonlinear least-squares methods.
• Nocedal & Wright: Numerical Optimization — Local quadratic models, line searches, and numerical optimization foundations.