Colorado State Fair’s fine art competition, 1st prize in digital art category Generated by NovelAI その他,たんぱく質の生成 (Baker’s lab (https://www.bakerlab.org/2023/07/11/diffusion- model-for-protein-design/)),音声合成など.
et al.: Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML2015] [Ho et al.: Denoising Diffusion Probabilistic Models. NeurIPS2020] [Song et al.: Score-Based Generative Modeling through Stochastic Differential Equations. ICLR2021] 徐々にノイズを添加する ノイズを除去する過程を近似
Score-based Generative Modeling in Latent Space. arXiv:2106.05931] (𝑌𝑡 ∼ 𝑋 𝑇−𝑡 ) [Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song et al., 2020; Ho et al., 2020; Vahdat et al., 2021]
ICLR2021. 2. Karas et al.: Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS2022 3. Lu et al.: DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. NeurIPS2022. 4. Liu et al.: Pseudo Numerical Methods for Diffusion Models on Manifolds. ICLR2022. 5. Dockhorn, Vahdat, Kreis: GENIE: Higher-Order Denoising Diffusion Solvers. NeurIPS2022. 様々な解法が提案されている. • ナイーブに実装すると時間離散化誤差が強く影響 [2]. • 拡散モデル用に実装を工夫する必要がある [3,4,5]. ➢ 線形多段法 [4],Heun法 [2],変形exp-Runge-Kutta法 [3],高次漸近展開 [5] • スコアの推定誤差には鋭敏かもしれない. ← 計算を工夫したODE型の方法は ステップ数を減らしても誤差が発 散しにくい.
flow ODE is provably fast. 2023. ➢ Li et al.: Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models. 2023. SDE手法:O(1/𝑇) ODE手法:O(1/𝑇2) (𝑇は離散化後のステップ数)
bound via Girsanov’s theorem: Chen et al. (2022) • Error bound with LSI: Lee et al. (2022a) ➢ With smoothness: Chen et al. (2022) and Lee et al. (2022b) • Error propagation with manifold assumption: Pidstrigach (2022) [Generalization analysis] • Wasserstein dist bound (𝑛−1/𝑑) with manifold assumption: De Bortoli (2022) 70
Taiji Suzuki: Diffusion Models are Minimax Optimal Distribution Estimators. ICML2023 (oral), arXiv:2303.01861] Kazusato Oko (The University of Tokyo) Shunta Akiyama (The University of Tokyo)
Forward process Backward process どちらも(ほぼ)ミニマックス最適 [Yang & Barron, 1999; Niles-Weed & Berthet, 2022]. 経験スコアマッチング推定量: (for any 𝛿 > 0). 定理 Let 𝑌 be the r.v. generated by the backward process w.r.t. Ƹ 𝑠, then (Estimator for 𝑊1 distance requires some modification) (𝑠: 密度関数の滑らかさ) [Kazusato Oko, Shunta Akiyama, Taiji Suzuki: Diffusion Models are Minimax Optimal Distribution Estimators. ICML2023]
setting 73 Assumption 1 The true distribution 𝑝0 is supported on −1,1 𝑑 and with 𝑠 > Τ 1 𝑝 − Τ 1 2 + as a density function on −1,1 𝑑. Assumption2 Very smooth Besov space Besov space (𝐵𝑝,𝑞 𝑠 (Ω)) Smoothness Spatial inhomogeneity
setting 74 Assumption 1 The true distribution 𝑝0 is supported on −1,1 𝑑 and with 𝑠 > Τ 1 𝑝 − Τ 1 2 + as a density function on −1,1 𝑑. Assumption2 Very smooth Besov space Besov space (𝐵𝑝,𝑞 𝑠 (Ω)) Smoothness Spatial inhomogeneity Intuition Smoothness Uniformity of smoothness
𝑇 = 𝑛−𝑂(1), 𝑇 = 𝑂(log(𝑛)). Then, the empirical risk minimizer Ƹ 𝑠 in DNN satisfies This is minimax optimal, that is, the worst case error is lower bounded as Although Ƹ 𝑠(𝑥, 𝑡) is a function with 𝑑 + 1-dimensional input, there appears “𝑑” in the bound instead of 𝑑 + 1. This is because Gaussian convolution makes the density smoother. 𝑇 𝑇
𝑇 𝑡∗ ത 𝑇 • Bound by diffused B-spline approximation • A tighter bound on the smooth part (𝑡 > 𝑡∗ ) (take 𝑘 = 𝑠 + 1) ➢ Similar argument is applied to 𝛻𝑝𝑡 : - Useful for W1 bound. - Smoothness around the edge (A2) is not requires.
For any fixed 𝛿 > 0, by slightly changing the estimator, the empirical risk minimizer Ƹ 𝑠 in DNN satisfies This is also known as minimax optimal (up to 𝛿) [Niles-Weed & Berthet (2022)]. • 𝑑′ appears instead of 𝑑: Diffusion model can avoid curse of dimensionality. • The minimax rate of Wasserstein distance is faster than that of TV distance, which makes it difficult to establish the bound. ➢ We need more precise estimate of the score around 𝑡 = 0. (TV) (W1)
= 𝑣𝑡 𝑇𝑡 𝑤 • ある𝜙𝑡 を用いて𝑣𝑡 = 𝛻𝜙𝑡 と書けるとする. 89 この時,以下が成り立つ: 定理 詳細は以下を参照: Ambrosio, Gigli, and Savaré. Gradient Flows in Metric Spaces and in the Space of Probability Measures. Lectures in Mathematics. ETH Zürich. Birkhäuser Basel, 2008. 𝑇𝑡 𝑣𝑡