Plain-English explanation
A raw image is a large grid of pixel values, and running a multi-step denoising process on every one of those pixels, over and over, is expensive. Latent diffusion sidesteps most of that cost by first shrinking the image down into a compact, abstract representation (the latent space) doing the actual generation work there, and only expanding the result back into a full, viewable image at the end.
This one architectural choice, introduced in a 2022 paper by researchers who went on to help create Stable Diffusion, is a large part of why realistic AI image generation became something a home computer with a decent GPU could do, rather than something that required a data-center cluster.
How it works
A separate small network (the encoder half of a VAE) compresses an input image down to a much smaller latent grid; in Stable Diffusion's case, roughly an eight-times reduction in each spatial dimension, with several channels of information per position rather than the original three (red, green, blue). The diffusion process itself (noising during training, iterative denoising during generation) happens entirely on that compressed latent, using a smaller and faster network than pixel-space diffusion would need. Once the reverse denoising process finishes, the VAE's decoder half converts the finished latent back into a normal pixel image. Text conditioning works the same way regardless of where diffusion happens, via a text embedding injected into the network at each step.
Adult-AI use
Latent diffusion is the specific technical reason local generation (running an unrestricted image model entirely on a consumer GPU at home, with no hosted platform's content policy in the loop) is realistic at all. Stable Diffusion and the checkpoints, LoRAs, and communities built around it are latent diffusion models; without the compute savings latent-space compression provides, that entire self-hosted ecosystem, and the privacy and control it offers over a hosted alternative, wouldn't be practical on consumer hardware.
How it differs from a pixel-space diffusion model
Latent diffusion is a diffusion model: it's not a separate category, just a specific answer to where the denoising math runs. “Diffusion model” describes the general noise-removal algorithm; “latent diffusion” specifies that it happens inside a compressed representation rather than on raw pixels. Nearly every popular text-to-image and text-to-video tool today (the Stable Diffusion family, Flux, most video diffusion models) is latent, though a handful of specialized pixel-space diffusion models still exist for particular use cases.
Common uses
- Stable Diffusion 1.5, SDXL, SD3, and similar checkpoints run through local tools like ComfyUI or Automatic1111.
- Flux and other modern open-weight image models.
- Most hosted image and video generation platforms, which typically run a latent diffusion model on their own servers.
Limitations and misconceptions
- The compression step is lossy: the VAE approximates rather than perfectly reconstructs the original pixels, which can introduce subtle artifacts like blurred fine text or, in earlier models, distorted hands and eyes.
- “Latent” doesn't mean hidden or secret in a privacy sense: it's a purely mathematical term for a compressed representation, unrelated to data privacy or security.
- Latent diffusion doesn't automatically make generation feel faster to an end user: the compute savings are in the underlying model; perceived speed still depends on hardware, step count, and resolution.
Technical context
See the pipeline dossier for the complete component path and the SDXL dossier for a documented model-family implementation.