Contents · Overview
TL;DR
What does diffusion generate at each noise level? We answer this quantitatively through feature information dynamics, using optimal denoising losses to trace when features emerge and in what order. This order differs between diffusion models trained in different representation spaces: only the diffusion model in RAE space exhibits an abstract-to-concrete hierarchy in our comparison, and it also converges fastest. Finding the optimal generation order and representation for each modality is an interesting future direction.
An autoregressive view of diffusion
Autoregression and diffusion make generation tractable through smaller prediction tasks: one predicts successive sequence elements, the other progressively removes noise. What implicit generation order does diffusion follow?
Building on the spectral analysis of Rissanen et al. (2022)1, Sander Dieleman’s blog2 frames pixel-space diffusion as approximate spectral autoregression: low frequencies emerge before high frequencies. This account relies on the approximately power-law spectrum of natural images: low frequencies carry more signal power, so under white Gaussian noise they have higher effective SNR and emerge earlier during denoising. Learned latent representations need not preserve this spectral structure, so the same argument does not directly apply. Their markedly different convergence speeds make understanding what happens in each representation essential.
Across arbitrary representations, which features are generated at each SNR? We quantify how information about a feature—class, mask, or Canny edges—is distributed along diffusion, then study the order in which multiple features emerge.
Feature information density
Let \(X_\gamma\coloneqq\sqrt{\gamma}X+N\) be the noisy observation at SNR \(\gamma\), with independent standard Gaussian noise. Mutual information \(I(Y;X_\gamma)\) measures how much it contains about feature \(Y\). We define feature information density as its rate of change along log-SNR; I-MMSE3 connects this rate to optimal denoising losses:
Here \(m_S(\gamma)\) is the minimum x-prediction loss (MMSE) given condition \(S\). The unconditional denoiser predicts the clean image from the noisy image alone; the class-conditional denoiser also receives its class label. Their optimal losses are \(m_{\varnothing}\) and \(m_Y\), respectively.
The interactive example below takes MNIST images as \(X\) and digit class as the feature \(Y\), tracing how class information emerges during diffusion. The digit’s class becomes visually clear around \(\log_{10}\gamma \approx 1\), where accumulated class information also saturates. Feature information density further reveals that most class information emerges within \(\log_{10}\gamma \in [-2,0]\).
\(m_{\varnothing},\;m_Y\)
━ Unconditional
━ Class-conditional
Both curves fall with SNR; their shaded separation is the conditioning gain.
\(\Delta_Y\coloneqq m_{\varnothing}-m_Y\)
Subtract the two optimal losses.
\(D_Y^{(\log\gamma)}=\frac{\gamma}{2}\Delta_Y\)
The rate of class information gain along log-SNR. Its peak locates the fastest emergence.
\(\begin{aligned}&\int_{-\infty}^{\log\gamma}D_Y^{(\log\gamma)}(u)\,du\\&\qquad=I(Y;X_\gamma)\end{aligned}\)
Integrate feature information density over log-SNR up to the selected SNR.
Drag the slider or click a curve to explore. Images show one digit; curves average the full test set. Measured losses estimate MMSE.
§3.1–3.4 · MMSE, information density, and estimation · Run the notebook
Chained information decomposition
Features can share information. Our chained information decomposition measures what each feature adds beyond those already given. The conditional mutual information \(I(X_\gamma;Y_k\mid Y_{\lt k})\) measures how much the noisy observation reveals about the next feature once the preceding features are known. We define its rate of change along log-SNR as the chained feature information density:
Here \(m_{k-1}(\gamma)\) and \(m_k(\gamma)\) are the optimal denoising losses given the first \(k-1\) and \(k\) features, respectively. Their difference measures how much adding the \(k\)-th feature reduces the optimal denoising loss.
We apply this decomposition to MNIST images, using frequency bands ordered from low to high as features. Chaining makes spectral autoregression visible quantitatively. Independently measured frequency bands overlap; after conditioning on lower-frequency bands, their incremental information peaks in low-to-high frequency order. This quantifies when each band is generated along SNR.

§4 · Chained information decomposition · §5.1 · Spectral autoregression
Representations change generation order
When drawing an image, people naturally first decide what to draw, then sketch its overall shape, and finally add local details. This motivates examining features at different levels of abstraction: class, mask, and Canny. Does diffusion generate them in an abstract-to-concrete order—and does that order change across representations?
We compare diffusion models in four representation spaces: pixels; SDVAE, an autoencoder trained for reconstruction; VAVAE, which additionally aligns its latent representation with a visual foundation model (VFM)4; and RAE, where diffusion models operate directly in a VFM’s latent space5. On ImageNet-256, we apply chained information decomposition to class → mask → Canny. By training the corresponding denoisers with corresponding standard recipes6,7,4,5, we estimate their chained feature information densities, shown below.
Representation choice changes the order in which diffusion generates features at different levels of abstraction. Only the diffusion model in RAE space exhibits an abstract-to-concrete order in our comparison.
Does each modality have an optimal generation order?
AO-GPT8 finds that left-to-right language modeling converges much faster than a fixed random order, attributing this advantage to language’s sequential structure.
Our observation is suggestive: diffusion in RAE space generates class, mask, and Canny in that order, and diffusion models trained in this space converge faster than those trained in pixel, SDVAE, and VAVAE spaces. We conjecture that images favor generation from abstract to concrete.
How can we discover the optimal generation order for each modality? Could those orders guide the next generation of generative models?
I am currently researching these questions. If you are interested, get in touch.
References
- Rissanen, Heinonen, and Solin. Generative Modelling with Inverse Heat Dissipation. 2022; ICLR 2023.
- Sander Dieleman. Diffusion is spectral autoregression. Blog, 2024.
- Guo, Shamai, and Verdú. Mutual Information and Minimum Mean-Square Error in Gaussian Channels. IEEE Transactions on Information Theory, 2005.
- Yao, Yang, and Wang. Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models. CVPR 2025.
- Zheng et al. Diffusion Transformers with Representation Autoencoders. ICLR 2026.
- Li and He. Back to Basics: Let Denoising Generative Models Denoise. CVPR 2026.
- Ma et al. SiT: Exploring Flow and Diffusion-Based Generative Models with Scalable Interpolant Transformers. ECCV 2024.
- Xue et al. Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture. ICML 2026.
Citation
@article{pan2026feature,
title={Feature Information Dynamics in Diffusion},
author={Pan, Jia-Shu and Zhang, Tao and Huang, Yufei and Sheng, Yanjun and Wu, Tailin},
year={2026},
eprint={2610.08626},
archivePrefix={arXiv},
primaryClass={stat.ML},
url={https://arxiv.org/abs/2610.08626}
}