This document explains the Variational Autoencoder (VAE) system and latent space representation in ComfyUI. It covers how images are encoded into compressed latent representations, decoded back to pixels, the format specifications for different model types, tiled processing mechanisms, the LatentFormat system for handling multiple model families, and previewing latents efficiently via TAESD.
The VAE system in ComfyUI acts as the bridge between the high-dimensional pixel space and the lower-dimensional latent space on which diffusion models operate. It performs compression and decompression of images, enabling efficient diffusion by working on compact latent tensors.
Key Responsibilities:
Sources: comfy/sd.py439-470 comfy/latent_formats.py3-19 nodes.py320-383
The VAE class is the main interface for variational autoencoding logic. It encapsulates various architectures, from standard KL autoencoders to specialized video and audio VAEs.
During initialization, the VAE class inspects the state dict to automatically select the appropriate internal architecture comfy/sd.py471-828:
| Key Pattern | VAE Type | File Reference |
|---|---|---|
decoder.mid.block_1.mix_factor | Video VAE (SVD) | comfy/sd.py560 |
taesd_decoder.1.weight | TAESD | comfy/sd.py656 |
vquantizer.codebook.weight | StageA (Cascade) | comfy/sd.py614 |
decoder.layers.1.layers.0.beta | Audio VAE | comfy/sd.py734 |
encoder.blocks.0.0.v_linear.weight | Wan Video VAE | comfy/sd.py756 |
Sources: comfy/sd.py439-828 comfy/latent_formats.py1-19
The LatentFormat class defines per-model scaling, shifting, and channel dimensions required to translate between raw VAE outputs and the tensors expected by diffusion models.
scale_factor: Normalizes latent magnitudes comfy/latent_formats.py4latent_channels: Number of channels (e.g., 4 for SD1.5, 16 for Flux, 256 for Stable Audio 3) comfy/latent_formats.py5-156spacial_downscale_ratio: Usually 8 for standard VAEs, but can be 16 for Flux2 comfy/latent_formats.py11-194process_in / process_out: Methods to apply format-specific transformations like the shift factor in SD3/Flux or mean/std normalization in Playground 2.5 comfy/latent_formats.py62-191Sources: comfy/latent_formats.py1-230 comfy/model_base.py156-160
To handle high-resolution images or long videos without exceeding VRAM, ComfyUI implements tiled encoding and decoding via the VAEEncodeTiled and VAEDecodeTiled nodes.
VAEEncodeTiled)Input images are split into overlapping tiles, encoded individually, and then reconstructed in latent space nodes.py353-383 This is critical for 4K+ image generation on consumer hardware.
VAEDecodeTiled)Latent tensors are decoded in tiles. The system calculates the appropriate tile size based on the VAE's spacial_downscale_ratio and temporal_downscale_ratio to ensure seamless reconstruction nodes.py320-351 For video models, this also involves handling temporal dimensions comfy/sd.py1010-1030
Sources: nodes.py320-383 comfy/sd.py945-1105
TAESD (Tiny AutoEncoder for Stable Diffusion) provides near-instant decoding for previews during the sampling loop by using highly optimized, lightweight decoders latent_preview.py4-110
taesd_decoder, taesdxl_decoder, etc.) for high-quality previews latent_preview.py84-105latent_rgb_factors defined in the LatentFormat. This is the fallback if no TAESD model is found latent_preview.py52-110Sources: latent_preview.py31-111 comfy/taesd/taesd.py1-20 comfy/latent_formats.py7-10
| Model Family | Channels | Downscale | Scale Factor | TAESD Key |
|---|---|---|---|---|
| SD 1.5 | 4 | 8x | 0.18215 | taesd_decoder |
| SDXL | 4 | 8x | 0.13025 | taesdxl_decoder |
| SD3 | 16 | 8x | 1.5305 | taesd3_decoder |
| Flux | 16 | 8x | 0.3611 | taef1_decoder |
| Flux2 (Video) | 128 | 16x | 1.0 | None |
| Stable Audio 3 | 256 | 4096x (T) | 1.0 | None |
Sources: comfy/latent_formats.py20-195 comfy/supported_models.py61-172
Refresh this wiki