A VISUAL INTRODUCTION
Stable Diffusion, and every model like it, rests on one idea small enough to fit on a napkin: if you know which direction makes noisy data look more like real data, you can walk there. Everything else — the U-Net, the text conditioning, the latent space — is engineering built around that one idea. This page derives it, visually, from scratch.
↓ every chart below is real math, not a stock illustration — drag things
nine gaussian clusters, forward-diffused to noise and back, on loop
01 THE FORWARD PROCESS
Start with a toy dataset: 1,600 points sampled from nine 2-D Gaussian clusters arranged in a spiral. This is a stand-in for "images" — instead of a 3-million-pixel photo, it's a 2-number point, but the math is identical.
The forward process repeatedly mixes in a little Gaussian noise. After enough steps, the spiral is indistinguishable from pure noise. Critically, this doesn't require simulating every step — there's a closed form for jumping straight to step t:
where ᾱt ("alpha-bar") is a number that glides from 1 down to ~0 as t runs from 0 to T. At ᾱt=1 you get the original data back; at ᾱt≈0 the signal term vanishes and only noise remains.
Drag the slider. Notice the spiral doesn't blur or fade — each point jitters and drifts until its origin is unrecoverable. That's what "destroying information" looks like at the level of a single sample.
02 THE NOISE SCHEDULE
The sequence ᾱ0, ᾱ1, …, ᾱT is called the noise schedule. A linear schedule (older, used in the original DDPM paper) spends too much of its budget destroying fine detail early and wastes late steps on data that's already nearly pure noise. A cosine schedule holds signal longer in the middle of the trajectory, giving the model more useful intermediate states to learn from — a few lines of scheduling code that measurably improve sample quality.
03 THE SCORE FUNCTION
Reversing the forward process means answering one question at every noisy point: which direction increases the probability of this being real data? That direction is the score, ∇x log pt(x) — the gradient of the log-density at noise level t. It always points uphill, toward the nearest mode of the data distribution, weighted by how much that mode currently "claims" the point.
For our toy Gaussian-mixture spiral, the score has a clean closed form, so the field below isn't approximated by a neural network — it's computed exactly. In Stable Diffusion, a U-Net is trained to approximate this same quantity for the distribution of natural images, which has no closed form.
Move your cursor over the canvas — the bright arrow is the exact score at that point: a responsibility-weighted pull toward every nearby cluster center.
04 REVERSE SAMPLING
This is the whole algorithm. Start from pure Gaussian noise. At each step, nudge every point along the score — toward higher density — and add back a little fresh noise (this keeps the process from collapsing onto a single mode; it's what makes the output a sample rather than an average). Repeat T times, annealing from high noise to low.
This is the exact DDPM ancestral sampler, run here with the true analytic score instead of a learned approximation. A trained U-Net doing this in a 4×64×64 latent space, conditioned on a text embedding, is Stable Diffusion's sampling loop.
05 FROM POINTS TO PIXELS
Everything above operates directly on data. Doing that with 512×512 pixel images is correct but wasteful — most of that iterative denoising effort would go into imperceptible pixel-level texture. Stable Diffusion's contribution was to run the entire process in a much smaller latent space instead, learned by a separate autoencoder, cutting compute by roughly an order of magnitude with little quality loss.
The text prompt enters through cross-attention inside the U-Net: a CLIP text encoder turns the prompt into a sequence of embeddings, and at every denoising step the U-Net attends over them — effectively asking "given this noisy latent and this description, which direction does the score point?" No architectural change was needed to add conditioning; you're just handing the network more context to compute the same quantity.
Classifier-free guidance pushes this further: the model is run twice per step, once with the prompt and once without, and the sample is pulled harder in the direction the prompt's score differs from the unconditional one. It's the same "follow the arrow" logic, exaggerated.
06 WHY THIS IS ELEGANT
TRAINING
The training objective is: given a noised sample and its noise level, predict the noise that was added. That's a plain L2 regression loss on a known target — no adversarial minimax game, no discriminator to balance, no mode collapse. GANs generate in one shot and are notoriously unstable to train; diffusion models trade a single hard problem for thousands of easy ones.
THEORY
Denoising Diffusion Probabilistic Models (Ho et al.) and score-based generative modeling with Langevin dynamics (Song & Ermon) were developed on separate tracks — one from variational inference, one from score matching. Song et al.'s SDE framework showed both are discretizations of the same continuous-time stochastic differential equation, run forward or backward in time. The forward-process demo and the score-field demo above are two views of that one equation.
DENSITY ESTIMATION
A probability density p(x) = exp(f(x)) / Z requires knowing Z, an integral over all of image space that's never tractable. The score ∇x log p(x) doesn't depend on Z at all — the constant vanishes under differentiation. Diffusion models never estimate a density; they only ever need its gradient.
COMPUTE
Because sampling is T repeated small steps, you can trade steps for speed: fewer steps, faster and rougher; more steps (or a smarter ODE solver), slower and cleaner. A GAN's generator has one fixed forward pass with no equivalent knob.
CONDITIONING
Every conditioning trick — class labels, text prompts, inpainting masks, ControlNet's pose and depth inputs — reduces to the same move: bias the score in a useful direction at each step. Because the model is already just "which way should this point move," bolting on a new source of "which way" is additive rather than architectural.