RMSprop: Adapting to Recent Gradients
Normalize each coordinate by a decaying root-mean-square of recent gradients so step sizes adapt without stalling.
When some parameter coordinates consistently see much larger gradients than others, a single scalar learning rate $\eta$ either oscillates along steep coordinates or crawls along flat ones. RMSprop gives each coordinate its own adaptive step size by dividing the gradient by the root-mean-square (RMS) of its recent gradients.
1. The RMSprop update rule
Starting from $v_0=0$ with decay rate $0<\rho<1$, RMSprop maintains an element-wise exponential moving average $v_t$ of squared gradients and divides the step by $\sqrt{v_t}+\epsilon$:
Each coordinate $i$ therefore receives an effective learning rate $\eta_{t,i}=\frac{\eta}{\sqrt{v_{t,i}}+\epsilon}$: coordinates with large recent gradients get damped, while coordinates with small gradients get amplified. Because every term $g_k^2\ge 0$ is nonnegative, opposite signs do not cancel—$v_t$ estimates the uncentered second moment $\mathbb E[g^2]=(\mathbb E[g])^2+\operatorname{Var}(g)$, not the centered variance alone.
2. How RMSprop differs from AdaGrad
AdaGrad accumulates all past squared gradients with equal weight forever, whereas RMSprop decays older squared gradients by $\rho$ on every step:
| Feature | AdaGrad | RMSprop |
|---|---|---|
| Accumulator update | $G_t = G_{t-1} + g_t^2$ (lifetime sum) | $v_t = \rho v_{t-1} + (1-\rho)g_t^2$ (exponential moving average) |
| Weight on past $g_k^2$ | $1$ forever (never forgets) | $(1-\rho)\rho^{t-k}$ (geometrically decays) |
| Denominator over time | Monotonically non-decreasing ($G_t \to \infty$) | Grows and shrinks with recent gradient magnitude |
| Steady gradient $g_k=g\neq 0$ | $G_t=t\,g^2 \implies \Delta w_t \approx -\frac{\eta}{\sqrt{t}}\operatorname{sign}(g) \to 0$ | $v_t=(1-\rho^t)g^2 \to g^2 \implies \Delta w_t \to -\eta\operatorname{sign}(g)$ |
3. Lab: watch $v_t$ respond to a gradient spike
This lab feeds a controlled gradient sequence into RMSprop so you can watch $v_t$ and the step $\Delta w_t$ respond to a sudden spike or sign reversal.
RMSprop memory microscope
Step through the sequence or inspect all 40 steps at once. Change the decay $\rho$ to see how fast $v_t$ forgets a spike.
- Spike damping and recovery: a sudden gradient jump inflates $v_t$ immediately, shrinking the effective learning rate $\eta/(\sqrt{v_t}+\epsilon)$ for several subsequent steps until $\rho^{t-k}$ decays the spike.
- Decay $\rho$: larger $\rho$ (closer to $1$) smooths $v_t$ over a longer horizon $1/(1-\rho)$ but slows recovery after a spike.
- Sign reversal: flipping the sign of $g_t$ leaves $g_t^2$ and $v_t$ unchanged while immediately reversing $\Delta w_t$.
4. Implementation, centered variant, and state memory
Centered RMSprop additionally tracks the running mean gradient $q_t=\rho q_{t-1}+(1-\rho)g_t$ and normalizes by the estimated standard deviation $\sqrt{v_t-q_t^2}+\epsilon$:
# Basic uncentered RMSprop (PyTorch names the decay parameter `alpha`)
v = rho * v + (1 - rho) * grad.square()
parameter -= lr * grad / (v.sqrt() + eps)
optimizer = torch.optim.RMSprop(
model.parameters(), lr=0.01, alpha=0.99,
eps=1e-8, momentum=0, centered=False, weight_decay=0
)Optimizer state memory: Uncentered RMSprop without momentum stores one scalar per parameter—the squared-gradient average $v_{t,i}$ ($+4\text{ bytes/parameter}$ in FP32, matching AdaGrad and SGD with Momentum, and half of Adam’s $+8\text{ bytes/parameter}$). Enabling centered=True or momentum>0 adds a second state buffer per parameter.
5. Questions worth answering
Because $v_t$ averages $g_t^2$ without subtracting $(\mathbb E[g_t])^2$. Since $\mathbb E[g^2]=(\mathbb E[g])^2+\operatorname{Var}(g)$, a large $v_t$ can come from a large steady gradient, high noise, or both.
AdaGrad sums $g_t^2$ without decay ($G_t=G_{t-1}+g_t^2$); RMSprop uses a decaying average $v_t=\rho v_{t-1}+(1-\rho)g_t^2$ with scalar $\eta$; Adadelta replaces $\eta$ with the running RMS of past parameter updates $\sqrt{u_{t-1}+\epsilon}$; and Adam adds a signed momentum buffer $m_t$ plus $1-\beta^t$ bias correction on both $m_t$ and $v_t$.
Sources
Duchi, Hazan, and Singer, Adaptive Subgradient Methods (AdaGrad); Tieleman and Hinton, RMSprop (CSC321 Lecture 6); PyTorch RMSprop.