Upgrade to Pro — share decks privately, control downloads, hide ads and more …

EdgeConnect: Generative Image Inpainting with A...

tanimutomo
March 29, 2019

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

[DL-paper] 2019年1月にarxivに投稿されたEdgeConnectというInpainting手法に関する論文.論文(https://arxiv.org/abs/1901.00212) , Code (https://github.com/knazeri/edge-connect)

tanimutomo

March 29, 2019
Tweet

More Decks by tanimutomo

Other Decks in Research

Transcript

  1. w λΠτϧ w &EHF$POOFDU(FOFSBUJWF*NBHF*OQBJOUJOHXJUI"EWFSTBSJBM&EHF-FBSOJOH w ஶऀ w ,BNZBS/B[FSJ &SJD/H 5POZ+PTFQI

    'BJTBM;2VSFTIJ .ISBO&CSBIJNJ w $BOBEBͷ0OUBSJPେֶ w ౤ߘ w ݄ʹ"SYJWʹ౤ߘ͞Εͨ΋ͷ ॻࢽ৘ใ !2
  2. w ֦ࢄϕʔε %J⒎VTJPOCBTFE  w ύονϕʔε 1BUDICBTFE  w ֶशϕʔε

    -FBSOJOHCBTFE  w ωοτϫʔΫͷ޻෉ طଘख๏ !7
  3. w ಛ௃ w ϚεΫྖҬपลʹྨࣅ͍ͯ͠ΔྖҬΛಉ͡ը૾ͳ͍͔Βݕࡧ͠షΓ෇͚Δ w ϚεΫྖҬʹྨࣅ͢ΔྖҬ͕ը૾಺ʹଘࡏ͢Δ͜ͱ͕લఏ w ݶք w إը૾ʹ͓͍ͯޱΛ࠶ߏ੒͢ΔέʔεͷΑ͏ʹ


    ը૾ʹͳ͍΋ͷΛ࠶ߏ੒͢Δ͜ͱ͸Ͱ͖ͳ͍ w ग़ݱස౓͕௿͍΋ͷΛ࠶ߏ੒͕೉͍͠ w 1BUDI.BUDI w ྨࣅύονͷݕࡧΛϥϯμϜʹߦ͏ΞϧΰϦζϜΛఏҊ͠ɼ
 ܭࢉ࣌ؒΛѹ౗తʹ୹ॖͨ͠ ύονϕʔε !9 PatchMatch
  4. w ಛ௃ w جຊ͸ֶशϕʔεͷख๏Ͱɼ༷ʑͳ$7ܥͷλεΫͰ࢖ΘΕΔςΫχοΫΛ੝ΓࠐΜͩ w ݶք w ͦ΋ͦ΋$PBSTFSFTVMU͕ਖ਼͍͠ߏ଄Λ͍࣋ͬͯΔͱ͸
 ݶΒͳ͍ w

    ϚεΫͷܗ͕ݶΒΕΔ w $" XJUI$POUFYUVBM"UUFOUJPO  w $PBSTFUP'JOFߏ଄ͷಋೖ w $PBSTF3FTVMUͱݩʑͷೖྗը૾ͱͷ
 ྨࣅ౓Λܭࢉ͢Δ"UUFOUJPOػߏΛಋೖ ωοτϫʔΫͷ޻෉ !11 CA (Generative Image Inpainting with Contextual Attention)
  5. w ( &EHF(FOFSBUPS ͱ( *NBHF$PNQMFUJPO/FUXPSL Ͱߏ੒͞ΕΔ w &EHF(FOFSBUFS ( 

    w ೖྗάϨʔεέʔϧը૾ɼ&EHFը૾ɼϚεΫը૾ w ग़ྗ࠶ߏ੒͞Εͨ&EHFը૾ w *NBHF$PNQMFUJPO/FUXPSL (  w ೖྗݩը૾ɼ࠶ߏ੒͞Εͨ&EHFը૾ w ग़ྗ࠶ߏ੒ը૾ &EHF$POOFDU !12 Edge Generater Image Completion Net ֶश • G1ͱG2ΛɼD1, D2Λ༻͍ͯ
 ͦΕͧΕผʑʹֶश • D1Λൈ͍ͯɼG1ͱG2Λ
 End-to-EndͰֶश
  6. w (FOFSBUFS w %JMBUFE3FTJEVBM#MPDLT w ֤૚$POWPMVUJPO4QFDUSBM/PSN
 *OTUBODF/PSN3F-6 w -PTT w

    "EWFSTBSJBM-PTT 'FBUVSF.BUDIJOH-PTT w "EWFSTBSJBM-PTTʢ4/("/  w 'FBUVSF.BUDIJOH-PTT &EHF(FOFSBUPS !13
  7. w (FOFSBUFS w ϊΠζ͔ΒԿΒ͔ͷը૾Λੜ੒͢Δ w %JTDSJNJOBUFS w ੜ੒͞Εͨը૾ͱσʔληοτʹ͋ΔϦΞϧ ͳը૾ΛݟͯɼͲͪΒ͕ຊ෺ͷը૾͔Λݟ෼ ͚Δ

    w Πϝʔδ w %JTDSJNJOBUFSͷݟ෼͚Δೳྗ͕ߴ·Δ w (FOFSBUFS͸ݟ෼͚Δೳྗ͕ߴ·ͬͨ %JTDSJNJOBUFSΛͩ·͢Α͏ʹֶश͢Δ w ݁Ռతʹ(FOFSBUFS͕ຊ෺ͷΑ͏ͳը૾Λੜ ੒Ͱ͖ΔΑ͏ʹͳΔ "EWFSTBSJBM-PTT !14
  8. w ೖྗը૾ͱɼೖྗը૾ʹϚεΫΛ͔͚ͨը૾ Λ༻ҙ w (FOFSBUFS w ϚεΫΛؚΉೖྗը૾͔ΒϚεΫΛ࠶ߏ੒ ͨ͠ը૾Λੜ੒ w %JTDSJNJOBUFS

    w ݩʑͷೖྗը૾ͱϚεΫ෦෼͕࠶ߏ੒͞Ε ͨೖྗը૾Λࣝผ͢ w ໨తؔ਺ w ɹ w $$PNQMFUJPO/FUXPSL (FOFSBUFS 
 %%JTDSJNJOBUFS ..BTL *NBHF*OQBJOUJOHʹ͓͚Δఢରతֶश !15  
  9. w (FOFSBUFS͸&EHF(FOFSBUFSͱಉ͡ w -PTT w --PTT "EWFSTBSJBM-PTT 1FSDFQUVBM-PTT 4UZMF-PTT w

    1FSDFQUVBM-PTT w 4UZMF-PTT w "EWFSTBSJBM-PTT XJUIPVU4/  w -PTTؔ਺ʹΑΔ੍໿͕ଟ͍ͨΊɼ4/ΛೖΕΔͱֶश͕ۃ୺ʹ஗͘ͳΔͨΊɼൈ͍ͨ *NBHF$PNQMFUJPO/FUXPSL !18
  10. w ੜ੒ը૾ͱϦΞϧը૾ͷελΠϧʢ৭ɼςΫενϟɼ ࢹ֮తͳύλʔϯʣ͕ࣅΔΑ͏ʹֶश w 1FSDFQUVBM-PTTͰར༻͢Δ'FBUVSF.BQͷϖΞʢੜ ੒ͱϦΞϧʣͷάϥϜϚτϦοΫεͷࠩ෼ w 4UZMF-PTT !20 9

    Style Lossͷ࿦จͷਤ ੜ੒ը૾ͷ Feature Mapͷ Gram Matrix ϦΞϧը૾ͷ Feature Mapͷ Gram Matirx Gram Matrix νϟϯωϧcͱc’ʹ͓͚ΔFeature Mapͷཁૉ੮ͷ࿨
  11. w &EHFͷݕग़ w ਖ਼ղσʔλͷ࡞੒΋ؚΊɼ$BOOZΤοδݕग़Λར༻ͨ͠ w ଞͷ&EHFݕग़ख๏Λ࢖༻ͨ࣌͠ͷਫ਼౓ൺֱ΋ߦͳ͍ͬͯΔ w .BTLը૾ w 3FHVMBS)PMFͱ*SSFHVMBSIPMFͲͪΒ΋࢖༻࣮ͯ͠ݧͨ͠

    w ը૾σʔληοτ w $FMFC" 1MBDFT 1BSJT4USFFU7JFX w αΠζY ࣮ݧ !21 Regular Hole Irregular Hole CelebA Places2 Paris Street View
  12. w ఆੑධՁ w *OQBJOUJOHͷ݁Ռͱ&EHFը૾ͷ*OQBJOUJOHͷ݁Ռ w (-$*$ $"ͷ݁Ռͱͷൺֱ w ఆྔධՁ w

    - 14/3 44*. '*% w ϢʔβʔධՁ w ը૾ΛݟͤͯϦΞϧ͔൱͔Λ౴͑Δςετ w ϦΞϧը૾ͱ࠶ߏ੒݁Ռͷը૾ΛݟͤͯͲͪΒ͕ϦΞϧ͔Λ౴͑Δςετ w &EHFΛར༻͢Δ͜ͱʹΑΔਫ਼౓޲্ʹؔ͢Δݕূ ࣮ݧ಺༰ !22
  13. w 14/3ʢϐʔΫ৴߸ϊΠζൺʣ w ͭͷը૾ؒͷϊΠζൺ཰Λද͢ɽ w 44*.ʢ4USVDUVBM4JNJMBSJUZʣ w ըૉ஋ͷมԽɼίϯτϥετͷมԽɼߏ଄ͷมԽΛද͢ ఆྔධՁࢦඪ !25

    PSNR = 10 ⇥ log10 MAX2 I MSE = 20 ⇥ log10 MAXI p MSE <latexit sha1_base64="6cdW3QK+V7H5Ue5NeKZnMNCX958=">AAACU3ichVFNS8MwGE7r93Q69eglOBRPo52CXoSpCHpQ5sd0sM6RZukWl36YvBVG6X8UwYN/xIsHTesO6gRfCDw8H+TNEzcSXIFlvRrmxOTU9MzsXGF+obi4VFpeuVFhLClr0FCEsukSxQQPWAM4CNaMJCO+K9itOzjK9NtHJhUPg2sYRqztk17APU4JaKpTuq9fnV/iTbyPbQs7wH2mcOJIH4uwl+acJwlNzg6ad9XOaZqcXR2n2HEKWaL6TyLzO+pBQp5KO6WyVbHywePAHoEyGk29U3p2uiGNfRYAFUSplm1F0E6IBE4FSwtOrFhE6ID0WEvDgOhV2kneSYo3NNPFXij1CQDn7PdEQnylhr6rnT6BvvqtZeRfWisGb6+d8CCKgQX06yIvFhhCnBWMu1wyCmKoAaGS610x7RNdCehvKOgS7N9PHgc31Yq9Xale7JRrh6M6ZtEaWkdbyEa7qIZOUB01EEVP6A19GMh4Md5N05z8sprGKLOKfoxZ/AS+e69x</latexit> SSIM = (2µxµy + c1)(2 xy + c2) (µ2 x + µ2 y + c1)( 2 x + 2 y + c2) <latexit sha1_base64="te08wqiCIK9nNox9fuCmoQRUS/U=">AAACW3icbVFdS+swGE7r15xfU/HKmxeHUBFGWwW9OTD05ngheNCpsM6SZukMJm1J0sNK2Z/0ynNx/srBdN3g+PFCypPngyRPo4wzpV33j2UvLC4trzRWm2vrG5tbre2de5XmktAeSXkqHyOsKGcJ7WmmOX3MJMUi4vQherms9IffVCqWJne6yOhA4FHCYkawNlTYkre3V9fwA4JYYlI6fiDycAzVt4BjIKF3BA74ECg2Ejgsx8Wkov2jSekY05NvzMdQo3nAqc1zbb6pZZMMW223404HvgJvBtpoNjdh6zUYpiQXNNGEY6X6npvpQYmlZoTTSTPIFc0wecEj2jcwwYKqQTntZgKHhhlCnEqzEg1T9v9EiYVShYiMU2D9rD5rFfmd1s91fD4oWZLlmiakPijOOegUqqJhyCQlmhcGYCKZuSuQZ2xa1uZ3NE0J3ucnfwX3fsc76fi/Ttvdi1kdDbSPDpCDPHSGuugnukE9RNAb+metWA3rr71gN+312mpbs8wu+jD23js2Ya63</latexit>