Watching an optimiser move

Most of what is hard about training is invisible: a number goes down, or it doesn't. Two dimensions are enough to see the dynamics directly. This placeholder post has an optimiser you can push around and a short video, so the site has somewhere to show both kinds of motion.

Momentum

Plain gradient descent takes a step against the gradient. Heavy-ball momentum keeps a running velocity:

$$v_{t+1} = \beta v_t - \eta \nabla f(\theta_t), \qquad \theta_{t+1} = \theta_t + v_{t+1}.$$

With $\beta = 0$ this is ordinary gradient descent. As $\beta \to 1$ the optimiser remembers more of its past, which carries it along narrow valleys and, sometimes, straight past the minimum.¹

The surface is

$$\begin{aligned} f(x, y) ={}& 0.12(x-1)^2 + 1.4(y - 0.35x)^2 \\ &- 1.2\, \exp\!\left(-\tfrac{(x+1.6)^2 + (y+0.9)^2}{0.5}\right) + \epsilon(x, y), \end{aligned}$$

a tilted quadratic valley with a Gaussian well cut into it and a little ripple $\epsilon$ to make it less polite.

Noise to structure

A diffusion model learns to undo the process that turns data into noise. Run backwards from pure Gaussian noise, the sample is pulled step by step onto the data distribution. The film below shows that for a two-dimensional point cloud whose data is a spiral.

The video uses the closed form rather than a trained network, so every point lands exactly on the spiral. A real model gets close; how close is the whole game.

betas = np.linspace(1e-4, 0.02, 1000)
alpha_bar = np.cumprod(1 - betas)

for t in reversed(range(1000)):
    a = alpha_bar[t]
    x_t = np.sqrt(a) * x0 + np.sqrt(1 - a) * noise
    draw(x_t)

@article{sahu2026watching,
  author = {Surya Kant Sahu},
  title  = {Watching an optimiser move},
  year   = {2026},
  url    = {https://ojus1.github.io/posts/watching-an-optimiser-move/}
}