Common floating-point formats
FP32 uses 32 bits per floating-point value and remains a common reference format. FP16 and BF16 use 16 bits, roughly halving weight storage and memory bandwidth for supported operations. FP16 provides more fraction detail but a narrower numerical range than BF16; BF16 keeps an exponent range similar to FP32 with fewer fraction bits.
Hardware and software determine which formats are fast and stable. A loader accepting an FP16 checkpoint does not guarantee that every operation executes in FP16 or that every device implements it efficiently.
Mixed precision and inference
Mixed precision uses different formats for different operations. Frameworks can run matrix multiplications or convolutions in a lower precision while retaining selected reductions, normalization, or accumulation in FP32. During training, gradient scaling can reduce FP16 underflow. During inference, the main practical goals are usually lower memory use and faster supported kernels.
Reduced precision can change rounding and execution order. PyTorch does not promise bitwise-identical floating-point results across platforms or releases even when random inputs are controlled. Small numerical changes may become visible in iterative image generation.
Precision versus quantization
Quantization also uses lower-bit representations, but the term usually refers to a conversion scheme with scales, groups, integer or specialized low-bit formats, and loader-specific kernels. Loading a model in FP16 is reduced floating-point precision; converting selected weights to an 8-bit or 4-bit scheme is quantization.
The boundaries can overlap in user interfaces because both are memory controls. Record the actual data type or quantization format instead of describing every reduced-memory model as “half precision.”
Where users encounter it
The Local Diffusion Inference Stacks dossier explains how precision interacts with model loading, VRAM, offloading, attention kernels, and VAE decoding. SDXL commonly runs with 16-bit weights locally, while larger transformer pipelines may mix BF16, quantized components, and CPU-offloaded modules.
LLM runtimes likewise distinguish model-weight formats from KV-cache precision and computation type. A reported “4-bit model” does not reveal the precision used for every activation or cache.
Practical implications
- Memory: lower-precision weights occupy less VRAM or RAM before other tensors and caches are counted.
- Speed: gains require hardware kernels that accelerate the chosen format.
- Stability: unsupported operations or limited range can produce overflow, underflow, NaNs, or visible output changes.
- Reproducibility: precision, backend, and framework version belong in a saved workflow when repeatability matters.
Technical context
The local-stack dossiers place FP16, BF16, mixed precision, quantization, caches, and offloading in their architecture-specific memory context.