Stable Diffusion is a latent diffusion model: instead of denoising in pixel space, it runs the diffusion process in a compressed latent space, which is…
See the problem latent diffusion was built to solve.
Running diffusion directly on pixels is expensive. A high-resolution image has millions of pixels, and denoising all of them over dozens of steps demands enormous compute and memory — enough to keep quality image generation on big research clusters.
The insight behind Stable Diffusion is that most of those pixels are redundant detail. If you could do the hard denoising work on a much smaller representation that still captures the image's essence, generation would get dramatically cheaper — cheap enough to run on a single consumer GPU.
Understand diffusing in a compressed space via the VAE.
Latent diffusion moves the whole process into a compressed latent space. A variational autoencoder (VAE) has two halves: an encoder that squeezes an image into a small latent representation, and a decoder that reconstructs a full image from a latent. The latent is far smaller than the pixels but keeps the meaningful content.
So generation runs entirely on these small latents — all the noisy denoising steps happen there — and only at the very end does the VAE decoder turn the finished latent back into a full-resolution image. That's the efficiency win: the expensive loop operates on something tiny.
Meet the denoiser and how the prompt steers it.
The U-Net is the workhorse: at each step it looks at the current noisy latent and predicts the noise to remove, gradually cleaning the latent toward a coherent image. It's the component that actually 'draws' during the diffusion loop.
To follow a prompt, a text encoder (in early Stable Diffusion, from CLIP) turns the prompt into an embedding, which is fed into the U-Net at every step through attention. This conditions the denoising so the latent evolves toward the described image rather than a random one. Prompt in, guidance at every step.
Trace the full pipeline and note practical implications.
End to end: the text encoder turns the prompt into an embedding; generation starts from random noise in latent space; the U-Net denoises the latent over many steps, guided by the prompt embedding; and finally the VAE decoder converts the clean latent into a full-resolution image. For image-to-image, you start from a real image's latent instead of pure noise.
Because the model is open and efficient, an ecosystem grew around these parts — fine-tunes, LoRA adapters that tweak the U-Net, and control methods — all leveraging the same modular architecture.
Consequences of the design: generation is fast and runs locally; the number of denoising steps and the guidance strength trade quality against speed and prompt-adherence; and the modular parts (VAE, U-Net, text encoder) can be swapped or fine-tuned independently, which is why community models proliferate.
Stable Diffusion is a latent diffusion model: it runs diffusion in a compressed latent space instead of on pixels, making high-resolution generation efficient. A VAE encodes images to latents and decodes them back; a U-Net denoises the latent over many steps; and a text encoder turns the prompt into an embedding that conditions the U-Net at each step. The full flow is prompt to embedding, noise to denoised latent, latent to image — and its modular, open parts spawned a large fine-tuning ecosystem.
Explain to a colleague why Stable Diffusion can run on a laptop GPU when earlier image models needed clusters. Walk through where compression happens, what the U-Net and VAE each do, and how the prompt steers the result.
What is the key idea of Stable Diffusion as a latent diffusion model?
Diffusing in a small latent space, not millions of pixels, is the efficiency breakthrough behind Stable Diffusion.
What are the two jobs of the VAE in Stable Diffusion?
The VAE bridges pixel space and latent space, letting the expensive diffusion loop operate on small latents.
What does the U-Net do?
The U-Net is the denoiser that 'draws' during diffusion, conditioned by the prompt embedding via attention.
How does the prompt influence the generated image?
Text conditioning guides each denoising step so the latent evolves toward the prompt, enabling text-to-image.