3D Point Cloud Generation Using Adversarial Training for Large Scale Outdoor Scene

IGARSS2021
Title: 3D POINT CLOUD GENERATION USING ADVERSARIAL TRAINING FOR LARGE-SCALE OUTDOOR SCENE
Authors: Takayuki Shinohara, Haoyi Xiu, Masashi Matsuoka, Tokyo Institute of Technology, Japan

Abstract: Three-dimensional (3D) point clouds are becoming an important part of the geospatial domain. During research on 3D point clouds, deep-learning models have been widely used for the classification and segmentation of 3D point clouds observed by airborne LiDAR. However, most previous studies used discriminative models, whereas few studies used generative models. Specifically, one unsolved problem is the synthesis of large-scale 3D point clouds, such as those observed in outdoor scenes, because of the 3D point clouds’ complex geometric structure.
In this paper, we propose a generative model for generating large-scale 3D point clouds observed from airborne LiDAR. Generally, because the training process of the famous generative model called generative adversarial network (GAN) is unstable, we combine a variational autoencoder and GAN to generate a suitable 3D point cloud. We experimentally demonstrate that our framework can generate high-density 3D point clouds by using data from the 2018 IEEE GRSS Data Fusion Contest.

Transcript

1. 1 3D Point Cloud Generation Using Adversarial Training for Large Scale Outdoor Scene Takayuki Shinohara, Haoyi Xiu, and Masashi Matsuoka Tokyo Institute of Technology 15/July/2021 Online, IGARSS2021
2. 2 Tokyo Tech Outline 1. Background and Objective 2. Proposed Method 3. Experimental Result 4. Conclusion
3. 3 Tokyo Tech 1. Background and Objectives
4. 4 Tokyo Tech Deep Learning (DL) on Point Cloud lDiscriminative Model n Classification n Semantic Segmentation n Object Detection lGenerative Model n Point Cloud Generation Many Researches We need to research generative models for airborne Point Cloud. Generative models for Point Cloud observed by airborne LiDAR (airborne Point Cloud) are not studied enough.
5. 5 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target latent 3d points (Achlioptas et al.) tree GAN (Shu et al.) Source: Shu et al.
6. 6 Tokyo Tech Generative model for Point Cloud lGenerative Adversarial Network Previous papers only generate simple objects Airborne Point Cloud has more complex objects than previous target source: https://udayton.edu/engineering/research/centers/vision_lab/research/was_data_analysis_and_processing/dale.php Building Vegetation Ground Road
7. 7 Tokyo Tech DL-based Generative Models l VAEs[Kingma&Welling, 2014] n pros😄 l Clear objective function l Suitable training n cons😭 l Blurred results l Simple distribution l GANs[Goodfellow et al., 2014] n pros😄 l Clear results l Complex generation n cons😭 l Not clear object function l Hard to train We propose VAE and GAN-based airborne Point Cloud generation with suitable training and clear results. Complex objects generation needs suitable training and clear results
8. 8 Tokyo Tech 2. Proposed Method
9. 9 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake Data G(𝑧) Input Data 𝑥 𝑧 E G D Sampled from Real Data 𝑥~𝑅𝑒𝑎𝑙 Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator
10. 10 Tokyo Tech Network: Encoder lPointNet++[Qi et al. 2017]-based 𝑧 = 𝒩(𝜇(𝑥), 𝜎(𝑥)) 3 N Input Data 𝑥 𝜇(𝑥) 8,192 4,096 2,048 1DCNN Downsampling( ) Sampling Grouping Fully Connected 𝜎(𝑥) 𝑧 is sampled from estimated gaussian distribution Encoder estimates latent distribution Dow nsam pling and Convolution
11. 11 Tokyo Tech Network: Generator lTreeGAN[Shu et al. 2017]-based 𝑧 ∈ ℝ!" Upsampling and Convolution Fake Data latent vector from Encoder Reconstructed data 𝐺 𝑧 = 𝑝# ∈ ℝ$×& Generator reconstructs input data from latent vector extracted by Encoder 𝑝' ∈ ℝ'×& 𝑝( ∈ ℝ(×& 𝑝& ∈ ℝ)×& 𝑝#*' ∈ ℝ ( $×& ⋯ seed point Gradually Upsampling
12. 12 Tokyo Tech Network: Discriminator lPointNet[Qi.et.al 2017]-based 𝐺(𝑧) ́ 𝑥~𝑅𝑒𝑎𝑙 Fake or Real Input … Discriminator judges Fake data or Real data and minimizes distribution between Fake data and Real data Fake Data Real Data MLP local features Max Pooling global features Randomly sampled from all of training data. Input MLP
13. 13 Tokyo Tech Optimization lVAE n Reconstruction and Regularization 𝐿!"# = 𝐿$%& + 𝐿$%' 𝐿$%& = 𝐿$%' = KL Divergence(𝑞(𝑧|𝑥) 𝑝 𝑧 lGAN n Wasserstein Gradient Penurity loss Gen: 𝐿( = −𝔼) 𝐷 𝐺 𝑧 Disc: 𝐿* = 𝔼) 𝐷 𝐺 𝑧 - 𝔼 ́ ,~ℝ 𝐷 ́ 𝑥 + 𝜆'/𝔼0 ,[( ∇0 ,𝐷 B 𝑥 1−1)1] Chamfer Distance Earth Mover Distance Penurity Close to Normal Distribution with 𝜇 = 0, 𝜎 = 1 " !∈#! min $∈#" 𝑥 − 𝑦 % % + " !∈#" min $∈#! 𝑥 − 𝑦 % % + min #!→#" " ': !∈#! 1 2 𝑥 − 𝜙(𝑥) % %
14. 14 Tokyo Tech 3. Experimental Results
15. 15 Tokyo Tech Experimental Data lGRSS Data Fusion Contest 2018 n Airborne LiDAR observation l Target Area l urban area l Building l Vegetation l Road l Training Patch l 25 m2 l 2,048 points l 1,000 patches Training Patch side view top view 25 m 25 m Target Area
16. 16 Tokyo Tech Generation Phase 𝑧 Fake data 𝐺(𝑧) ∈ ℝ#×% 1. Random vector from estimated distribution 𝑧 = 𝒩(𝜇, 𝜎), 𝜇 and 𝜎 are deRined in training process. 2. Generating Fake data 𝐺(𝑧) via trained generator Fake data 𝐺(𝑧) ∈ ℝ#×% Typical Fake data Typical Fake data Typical Fake data Typical Fake data 3. Pick up generated Fake data including typical objects such as buildings and vegetation. Pick up
17. 17 Tokyo Tech Generated Point Cloud Proposed GAN and VAE method generated better results than raw GAN model.
18. 18 Tokyo Tech 4. Conclusion and Future Work
19. 19 Tokyo Tech Conclusion and Future Work lConclusion n We propose a generative model for Point Cloud observed by airborne LiDAR using VAE and GAN. n Our trained Generator was able to make fake point clouds clearly. lFuture work n Only Qualitative evaluation => Quantitative evaluation n Sparse fixed points => Dense point cloud generation n Traditional method => Change generator into recent architecture
20. 20 Tokyo Tech Network: Generator lTree structure convolution latent vector Generated points Upsam pling with convolution Source: Shu et al. 𝑧 ∈ ℝ!"
21. 22 Tokyo Tech Overview of Our Method lVAE+GAN Reconstructed Fake Data G(𝑧) Input Data 𝑥 𝑧 E G D Sampled from Real Data 𝑥~𝑅𝑒𝑎𝑙 Real/Fake PointNet++ TreeGAN PointNet VAE: Encoder and Generator GAN judges Real or Reconstructed Fake data latent vector VAE encodes input points into latent vector 𝑧 and reconstructs input data from latent vector GAN: Discriminator