[Paper Summary] Image Inpainting for Irregular Holes Using Partial Convolution

This slide is the summary of "Image Inpainting for Irregular Holes Using Partial Convolution".
Original Paper is http://openaccess.thecvf.com/content_ECCV_2018/papers/Guilin_Liu_Image_Inpainting_for_ECCV_2018_paper.pdf .
My Re-implementation is here ( https://github.com/tanimutomo/partialconv ).

Transcript

1. 2019/05/21 Image Inpainting 
 for Irregular Holes 
 using Partial Convolution Tomoki Tanimura, B4 d-hacks, Jin Nakazawa Lab, Keio University
2. 2019/05/21 Information • Title: • Image Inpainting for Irregular Holes Using Partial Convolution • Authors: • Guilin Liu Fitsum A. Reda Kevin J. Shih Ting-Chun Wang Andrew Tao Bryan Catanzaro • Affiliation: • NVIDIA Corporation • Accepted: • ECCV2018 !2
3. 2019/05/21 Overview • Propose a new image inpainting method • Enable to deal with the irregular holes • Predict the hole pixels based on valid (no-hole) pixels • Outperforms other methods for irregular holes !3
4. 2019/05/21 Background • Existing Methods • Initialization • Hole pixels are initialized by the mean of the entire pixels • => blurriness and color discrepancy • Rely on Post-process • Reduce such blurriness • But, Expensive or may fail • Our Method (Inpainting with Partial Conv) • In partial convolution, the hole pixels is inpainted based on the valid (non hole) pixels • Hole pixels are inpainted gradually as well as mask become small gradually • Initialization and Post-process are NOT needed !4
5. 2019/05/21 Inpainting Network • Base Network is the U-Net-like architecture • Typical convolution is replaced with the Partial convolution !5
6. 2019/05/21 Partial Convolutional Layer • Partial Convolution + Mask Update function • Partial Convolution • • Mask Update function • !6
7. 2019/05/21 Partial Convolutional Layer !7 Next Input Renormalization
8. 2019/05/21 Loss function • Definition • Iin: Input Image / M(0 for holes): mask image / Iout: predicted image • Igt: Grand Truth Image / Icomp: Iout with non-hole’GT • Total Loss function • • Valid Loss • • Hole Loss • • Total Variation Loss • !8
9. 2019/05/21 Loss function • Perceptual Loss • • Style Loss • !9 VGG16
10. 2019/05/21 Perceptual Loss • L1 distance between the following image’s feature maps • the generated image • the ground truth image • Feature maps is created by VGG16 !10 VGG16
11. 2019/05/21 Style Loss • L1 distance between the gram matrices of the feature maps • The feature maps is same as the one used in perceptual loss • Gram Matrix !11
12. 2019/05/21 Qualitative Comparison !12
13. 2019/05/21 Quantitative Comparison !13 Score (Method) Hole ratio
14. 2019/05/21 User Study • Which result is more realistic than the other? • 50% means two methods are equal !14
15. 2019/05/21 Conclusion • Propose the Partial Convolution (in other paper) and Mask update mechanism • Enable to deal with the mask which is any shape, size, location • This method is weak • when the image has sparse structure • when the mask has large hole (as well as all other methods) !15
16. 2019/05/21 Difficulty in Implementation • Data is too huge (Places2 - 512px is 0.5TB) • Training time is too crazy long (10 days described in the paper) • Whether the gradient of VGG network’s parameters should be turned on • Where the VGG’s normalization should be applied • How to calculate the Total Variation Loss • How to update the mask for partial con • How to create the irregular mask !16