VC'23 チュートリアル講演 『三次元メッシュと深層学習』

VC'23 (https://visualcomputing.jp/vc2023/) で行ったチュートリアル講演のスライドです.

Transcript

1. Visual Computing 2023
2. CG Polygon Mesh (c) Autodesk, Inc. Simulation Rendering Modeling (c) Autodesk, Inc. 5
3. ... SIGGRAPH ( ) Photoshop : People are looking at a presentation about very difficult math, particularly about geometry. There are about ten people, but only some of them can understand the presentation. So, there are three people in front of the white board discussing eagerly about the presentation, while the other people get bored. ( ) 10 3 6
4. Gartner , Hype Cycle for Emerging Technologies 2023 , ( ) AI 2-5 . ( ) → . ( ) l l l l , etc. 7
5. , . 8 Polygon Mesh NeRF Deep Implicits Point Sets
6. , . , CG ... 9
7. ... → Poisson Surface Reconstruction ... , alpha shape(*1) ball pivoting(*2) OK! *1) Delauney , *2) 3 10
8. (Polygonal Mesh Processing ) l Remeshing ( ) l Simplification ( ) l Parameterization ( ) l Smoothing ( ) l Model Repair ( ) 12 , , ... , .
9. l Differentiable Surface Triangulation [Rakotosaona et al., TOG 2021] l Neural Mesh Simplification [Potamias et al., CVPR 2022] l Learning Direction Fields for Quadrangulation [Dielen et al., SGP 2021] 13 Remeshing & Simplification [Rakotosaona et al., TOG 2021] [Dielen et al., SGP 2021]
10. l Neural Subdivision [Liu et al., TOG 2020] l Neural Progressive Meshes [Chen et al., TOG 2023] 14 Remeshing & Simplification [Chen et al., TOG 2023]
11. : 1: 2: ([Ohtake et al., VMV 2022] ) , 線 15 Smoothing & Denoising l NN , ◦ DNF-Net [Li et al., TVCG 2020] ◦ GCN-Denoiser [Shen et al., TOG 2022]) l ◦ GeoBi-GNN [Zhang et al., CAD 2022] l Deep Image Prior [Ulyanov et al., CVPR 2018] , . End-to-End . ◦ Dual-DMP [Hattori et al., ECCV 2022]) [Hattori et al., ECCV 2022]
12. l Functional Map , ◦ Deep Functional Maps [Litany et al., ICCV 2017] ◦ Weakly Supervised DFM [Sharma and Ovsjanikov 2020], etc. Parameterization 16 [Litany et al., ICCV 2017] l Poincare disk 0, 1 ! ◦ AtlasNet [Groueix et al., CVPR 2018] ◦ Neural Shape Maps [Morreale et al., CVPR 2021] ◦ DA Wand [Liu et al., CVPR 2023], etc. [Morreale et al., CVPR 2021]
13. l , . ( ) l , DMP-Inpaint [Hattori et al., arXiv 2023] Model Repair 17 Hattori et al., “Learning Self-Prior for Mesh Inpainting Using Self-Supervised GCNs,” arXiv, 2023.
14. l l l 3 . 18 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]
15. l CG CAD l , 19
16. l l , 20
17. Progressive Meshes . [Hoppe et al., SIGGRAPH 1996] ( ) l → l → , 21
18. Pooling , . , ( , ). 22 : Graph Pooling l Gated Global Pooling [Li et al., ICLR 2015] softmax global pooling l gPool Layer [Gao and Ji, ICML 2019] top-k , , l diffpool Layer , softmax l Eigen Pooling [Ma et al., SIGKDD 2019] , [Ma et al., SIGKDD 2019]
19. l l , l → l → l CG → 23
20. l l l 3 . 24 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]
21. l l l 3 . 25 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]
22. Quiz: Hint: The Graph Neural Network Model 1. 26 Answer: 2008 SIGGRAPH, CVPR Neural Network 算 (CVPR Network Flow, Bayesian Network 1 )
23. “The Graph Neural Network Model” [Scarselli et al., IEEE Trans. NN, 2008] Feed-Forward Network (FFN) Almeida-Pinda Generalization of Back-Propagation to Recurrent Neural Networks (1987 ). Hinton Backpropagation . 1. 27 g: f: ...Almeida-Pinda 𝜃 𝑓(𝑥; 𝜃) 𝑥 = 𝑓(𝑥; 𝜃) 𝜃 = 𝑓(𝜃; 𝑥) 𝜃
24. , : 𝐡" # 𝐡$ # 𝐡$%& # 𝐡$%! # 𝐡" #%& = 1 𝑁 ' $∈𝒩(") 𝜎 𝐖𝐡$ # + 𝐛 𝐡" # 𝜎 𝐖, 𝐛 , ( ) 1. 28
25. l : 𝑀# l : 𝑈# Message Passing Neural Network [Gilmer et al., ICML 2017] 𝐡" # 𝐡$ # 𝐡$%& # 𝐡$%! # 𝐦! "#$ = # %∈𝒩(!) 𝑀" (𝐡! ", 𝐡% ", 𝑒!% ) 𝐡! "#$ = 𝑈" (𝐡! ", 𝐦! ") 1. 29
26. GraphSAGE [Hamilton et al., NeurIPS 2017] l l Mean, LSTM, Max-pooling aggregators : 1. 30 Graph Attention Network [Veličković et al., ICLR 2017] l Attention . Heterogeneous Graph Transformer [Hu et al., WWW 2020] l , , Transformer GAT [Hu et al., WWW 2020] [Hamilton et al., NIPS 2017]
27. l RNN l → Walk Attention AttWalk [Izhak et al., WACV 2022] 1. 31 MeshWalker [Lahav and Tal, TOG 2020] ( , )
28. : MPNN , . → 1. 32 l ACNN [Boscaini et al., NeurIPS 2016], GCNN [Masci et al., 3DRR 2015], MoNet [Monti et al., CVPR 2017], FeaStNet [Verma et al., CVPR 2018], etc. l ZerNet [Sun et al., CGF 2020] l Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021] l Gauge Equivalent Transformer [He et al., NeurIPS 2021], etc. [Monti et al., CVPR 2017]
29. l , Exponential Map , ( ) l Zernike ( ) 1. 33 ZerNet [Sun et al., CGF 2020] Zernike 𝑍! (𝑟, 𝜃) 𝑓 𝑟, 𝜃 = # !*$ + 𝛼! 𝑍! (𝑟, 𝜃) Zernike . . 𝑓 ∗ 𝑔 𝑝 = # !*$ + 𝛼! ,𝛼! -
30. Group Equivalent CNN (GE-CNN) [Cohen & Welling, NeurIPS 2016] 1. 34 Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021] [Linmans et al., arXiv 2018] GE-CNN: ( ) CNN . Group (cyclic group) . 2 (= ) .
31. GEM-CNN : 1. 35 Gauge Equivalent Mesh CNNs [de Haan et al., ICLR 2021] Gauge equivalent (gauge) . , (Lagrangian) . , (≒ ) . , 𝐾 𝑝 , 𝑔.→0 , 𝑞 𝑝 . → ,
32. l Zernike , ( ) l , l 1. 36 : [Glimer et al., ICML 2017] MPNN [Pfaff et al., ICLR 2021]: MPNN
33. l l l 3 . 37 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]
34. l , cf.) = l ( ) : 2. 38
35. = 𝐾 ∗ 𝑓 = ℱ!"(ℱ 𝐾 ⋅ ℱ 𝑓 ) 𝑓# = ℱ!"(Θ ∘ ℱ 𝑓 ) , [Hammond et al., 2011] Hammond et al., “Wavelets on graphs via spectral graph theory,” Applied ComputationalHarmonic Analysis, 2011. 2. 39
36. 0 1 2 3 𝐖 = 0 1 1 0 1 0 1 0 1 1 0 1 0 0 1 0 : 𝐋 = 𝐈 − 𝐃! " $𝐖𝐃! " $ 𝐷%% = 3 & 𝑊%& (𝐃 ) 𝐋: 2. 40
37. ( ) l 𝐋 = 𝐔𝚲𝐔5 (𝐔 , 𝚲 ) , 2. 41
38. = ( ) ℱ 𝜔 = 2 ,- - 𝑓 𝑡 𝑒,!."/#𝑑𝑡 ( 𝜔 , cos 2𝜋𝜔𝑡 + 𝑖 sin 2𝜋𝜔𝑡 ) 𝐟 = (𝑓$ , 𝑓1 , … , 𝑓2 ) , . ℱ 𝐟 = 𝐔(𝐟 2. 42
39. l (or ) ... 𝐖 = 0 1 1 0 1 0 1 0 1 1 0 1 0 0 1 0 𝐋 = 𝐈 − 𝐃! " $𝐖𝐃! " $ l (= ) l , (+ ) ℱ 𝐟 = 𝐔(𝐟 ℱ!" 7 𝐟 = 𝐔 7 𝐟 : : : : ( , 𝐔 ) 2. 43
40. ( ) ( ) 1 → − 0 − 0 2. 44
41. Spectral CNN [Bruna et al., ICLR 2013] l l f 𝑔 ( ) ... 7 𝐟 = ℱ 𝐟 = 𝐔(𝐟, 𝐟 ∗ 𝐠 = ℱ!" 7 𝐟 ⊙ ; 𝐠 = 𝐔< 𝐆𝐔(𝐟 ; 𝐠 = ℱ 𝐠 = 𝐔(𝐠 , E 𝐆 G 𝐠 8 𝐠 2. 45
42. 1. l 𝑁 , 𝑁×𝑁 𝐔 ( : 𝒪 𝑁! ) Spectral CNN 𝐟 ∗ 𝐠 = ℱ!" 7 𝐟 ⊙ ; 𝐠 = 𝐔< 𝐆𝐔(𝐟 2. l l , , 2. 46
43. l 1 , ChebNet [Defferrard et al., NeurIPS 2016] l → l 𝐋 𝐔 𝐟0 = ' 123 4,& 𝜃1 𝐋1𝐟 = 𝐔 ' 123 4,& 𝜃1 𝚲1 𝐔5𝐟 (𝜃3 ) 2. 47
44. Chebyshev l , 𝐋 → Chebyshev ChebNet [Defferrard et al., NeurIPS 2016] Chebyshev 𝐟0 = ' 123 4,& 𝜃1𝑇1 𝐋 𝐟 = 𝐔 ' 123 4,& 𝜃1𝑇1(𝚲) 𝐔5𝐟 𝑇) 𝜆 = 2𝜆𝑇)!" 𝜆 − 𝑇)!$ 𝜆 𝑇) 𝜆 = cos(𝑛𝜆) 𝐋 : 2. 48
45. ChebNet l , → [Kipf & Welling 2017] = ChebNet , Graph Convolutional Network [Kipf & Welling, ICLR 2017] GCN , (PyTorch Geometric ) 2. 49
46. Graph Convolutional Network [Kipf & Welling, ICLR 2017] GCN l Chebyshev 𝐾 = 2 , 𝐋 𝜆678 = 2 𝐟0 = 2𝜃&𝑇& F 𝐋 𝐟 − 𝜃3𝑇3 F 𝐋 𝐟 = 2𝜃& F 𝐋𝐟 − 𝜃3𝐟 = 2𝜃& 2 ⁄ 𝐋 2 − 𝐈 𝐟 − 𝜃3𝐟 = −𝜃(𝐋 − 𝐈)𝐟 + 𝜃𝐟 = 𝜃𝐃, & !𝐖𝐃, & !𝐟 + 𝜃𝐟 = 𝜃 𝐈 + 𝐃, & !𝐖𝐃, & ! 𝐟 → = 𝜃& 0 𝐋 − 𝐈 𝐟 + 𝜃3 0𝐟 l 𝜃 = 𝜃3 0 = −𝜃& 0 2. 50
47. Graph Convolutional Network [Kipf & Welling, ICLR 2017] Rescaling Trick 𝐟0 = 𝜃 𝐈 + 𝐃,! "𝐖𝐃,! " 𝐟 , 𝐈 + 𝐃,! "𝐖𝐃,! " [0, 2] → , K 𝐖 = 𝐈 + 𝐖 K 𝐷"" = ' $ K 𝑊"$ (* 𝐃 ) , . 𝐟0 = 𝜃K 𝐃, & ! K 𝐖K 𝐃, & !𝐟 * 𝐃!" # * 𝐖* 𝐃!" # [-1, 1] → GCN 1 2. 51
48. → : Kipf-Welling’s GCN 2. 52 2. Chebyshev Spectral CNN (ChebNet) - [Defferrad et al., NeurIPS 2016] 2 Chebyshev . . 3. Graph Convolutional Network - [Kipf and Welling, ICLR 2017] ChebNet . , . 1. Spectral CNN - [Bruna et al., ICLR 2013] . 2 . ( ) ,
49. Laplacian l CayleyNets [Levie et al., IEEE Trans SP 2018] l ChebNet II [He et al., NeurIPS 2022] l HodgeNet [Smirnov & Solomon, TOG 2021], etc. : 2. 53 l DiffusionNet [Sharp et al., TOG 2022] l Laplacian Mesh Transformer [Li et al., ECCV 2022] l Laplacian Pooling Network [Qiao et al., TVCG 2022] l Laplacian2Mesh [Dong et al., TVCG 2023]
50. Hodge 2. 54 HodgeNet [Smirnov & Solomon, TOG 2021] Hodge , , , , , ( ⋆! , ⋆" , ⋆# ). 𝐋 =⋆3 ,& 𝑑5 ⋆& 𝑑 : Hodge Laplacian (𝑑 ＝) → Laplacian
51. 2. 55 HodgeNet [Smirnov & Solomon, TOG 2021] Hodge : Hodge ⋆4 (𝐹): 𝐹 𝑓5 : ℝ67 → ℝ NN . , , . ⋆$ (𝐹): , 4 𝑔5 : ℝ87 → ℝ NN . [Meyer et al., 2003] Meyer et al., “Discrete Differential Geometry Operators for Triangulated 2-Manifolds,” Visualization and Mathematics III, 2003. ※ Laplacian , 𝑓$ , 𝑔$ → Laplacian ,
52. Laplacian Spectral Clustering , . 2. 56 Laplacian Pooling Network [Qiao et al., TVCG 2022] super-patch . Pooling/Unpooling . [Qiao et al., TVCG 2023] Spectral Clustering , ( ) [Qiao et al., TVCG 2023]
53. Laplacian , (= Pooling / Unpooling) . , . 2. 57 Laplacian2Mesh [Dong et al., TVCG 2023] : Spectral Compression of Mesh Geometry [Karni & Gotsman, SIGGRAPH 2000] [Zhang et al., CGF 2010]
54. NN , (= ) SE-ResNet Block [Hu et al., PAMI 2020] Pooling / Unpooling Laplacian 2. 58 Laplacian2Mesh [Dong et al., TVCG 2023]
55. , 2. 59 DiffusionNet [Sharp et al., TOG 2022] ℎ# 𝑢 = 𝑀 + 𝛿𝑡 ⋅ 𝐿 ,&𝑀𝑢 (∆= 𝑀9$𝐿 ∆= ΦΛΦ: ) ℎ# 𝑢 = Φ diag[𝑒,<##, 𝑒,<!#, … ] Φ5𝑢 l l (𝑀 , )
56. : l 𝑢 𝑡 : ℎ#(𝑢) l 1-ring : 𝑧= = 𝐺𝑢 l 𝑤> , 𝐴 . , 𝑤> , : Re b 𝑤> ⊙ 𝐴𝑤> 2. 60 DiffusionNet [Sharp et al., TOG 2022]
57. : , , DiffusionNet , DiffusionNet , , . 2. 61 DiffusionNet [Sharp et al., TOG 2022] : DiffusionNet , , , , .
58. l , . l , , ... ( ...) 2. 62 :
59. Heat Kernel Signature [Sun et al., SGP 2009] , . 2. 63 Hand-Crafted Feature ( ) [Sun et al., SGP 2009] Heat Kernel Signature (HKS) . , ( ) . l HKS-based Structural Encoding (HKSSE) [Wong, CVPR 2023] → , HKS MLP . l Mesh-MLP [Dong, arXiv Jun. 8th, 2023] (CVPR ) → HKS , MLP , . ( Concurrent ) [Wong, CVPR 2023] HKS Laplacian , .
60. l l l 3 . 64 Agenda [Hanocka et al., TOG 2019] [Sharp et al., TOG 2022] [Sun et al., CGF 2022]
61. MeshNet [Feng et al., AAAI 2018] l , 3 : MeshCNN [Hanocka et al., TOG 2019] l Progressive Meshes 3. 65
62. MeshCNN [Hanocka et al., TOG 2019] & l l 4 , l (= ) [Hanocka et al., TOG 2019] 3. 66
63. (edge collapse) Progressive Meshes [Hoppe, SIGGRAPH 1996] l l 3. 67
64. MeshCNN MeshCNN l → l → → , [Hanocka et al. 2019] 3. 68
65. MeshCNN l Primal-Dual Mesh CNN [Milano et al., NeurIPS 2020] l HalfedgeCNN [Ludwig et al., CGF 2023] l PicassoNet++ [Lei et al., IEEE Trans. NN, 2023] l Vertex/Face-based MeshCNN [Perez et al., TVCG 2023] : 3. 69 [Milano et al., NeurIPS 2020] Subdivision l SubdivNet [Hu et al., TOG 2022] l SPMM-Net [Shi et al., CAD 2023]
66. Picasso . 2 l : l : 3. 70 PicassoNet++ [Lei et al., IEEE Trans. NN 2023] QEM , , QEM disjoint . QEM . → [Lei et al., 2023]
67. Loop Subdivision → 4 1 3. 71 SubdivNet [Hu et al., TOG 2022] : Loop Subdivision : Loop Subdivision [Hu et al., 2022] [Hu et al., 2022] ※ MAPS [Lee et al., SIGGRAPH 1998] Neural Subdivision [Liu et al., TOG 2020]
68. SubdivNet → Pooling 3. 72 SPMM-Net [Shi et al., CAD 2023] SubdivNet [Hu et al., TOG 2022] QEM , Pooling . , Hourglass HRNet [Wang et al., PAMI 2019] . , , .
69. l MeshCNN , , , . l , , . SubdivNet CAE . l Task-specific Mesh Decimation (PicassoNet++ QEM ) 3. 73 : [Ludwig et al., CGF 2023] Convolution/Pooling
70. l (MPNN , GCN ) → GNN CG 74 C G DS l Remeshing Smoothing , Model Repair Parameterization l ( ) , → ,
71. 1: , . 頂 . 75 Take Home Message: ( , ) . , 1 . NeRF AI , [Sahillioğlu & Horsman, TOG 2022] 2: , .