Deep Learning Image Manipulation

Illustrated guide to some image manipulation methods, with demonstration.

Transcript

1. %FFQ
   -FBSOJOH
   *NBHF
   .BOJQVMBUJPO BO
   JMMVTUSBUFE
   HVJEF .-,JUDIFO
   
   
   
2. "CPVU
   NF w -FT[FL
   3ZCJDLJ
   w HJUIVC
   
   
   
   
   
   
   !MVOBSEPH
   w CPSO
   JO
   1PMBOE
   w .-
   3FTFBSDIFS
   BU
   $PPLQBE
   w *
   MJLF
   OBUUP
3. DBSFFST!DPPLQBEDPN 8BOU
   UP
   XPSL
   XJUI
   VT
4. $POWPMVUJPOBM
   "SJUINFUJD OCIKE
5. *NBHFT
   UP
   GFBUVSFT
6. $POWPMVUJPO http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html input output input output kernel
7. 4USJEF http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html 2px 2px 2px 2px
8. 1BEEJOH http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html 2px 2px
9. 4USJEF
   
   QBEEJOH http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html
10. 5SBOTQPTFE http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html simulated here with padding also called “deconvolution” “fractional stride”
11. %PXOTBNQMJOH features or small resolution image convolutional layer or layers RGB image input output
12. 6QTBNQMJOH upsampling CNN layer or layers RGB image features or small resolution image input output
13. &ODPEFS%FDPEFS D E image in Decoder Encoder image out feature space
14. 'VMMZ$POOFDUFE
    $MBTTJpFS approve loan reject class data or features also called “Dense” layer
15. $//
    $MBTTJpFS food person plant other AlexNet, LeNet, VGG…
16. 'PPE/FU ™ food not food
17. @teenybiscuit
19. @teenybiscuit
20. @teenybiscuit
21. @teenybiscuit
22. @teenybiscuit
23. @teenybiscuit
24. (FOFSBUJWF
    "EWFSTBSJBM
    /FUXPSLT
25. Generator Discriminator https://speakerdeck.com/lunardog/deep-convolutional-voight-kampf-test “Couple of bots studying for the Turing Test”
26. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford, Luke Metz, Soumith Chintala (Submitted on 19 Nov 2015 (v1), last revised 7 Jan 2016 (this version, v2)) https://arxiv.org/abs/1511.06434
27. Generator Discriminator G MPPLT
    MFHJU
    UPUBMMZ
    TIPQQFE D
28. G SFBM
    GBLF D D(G(noise)) ˠ real (FOFSBUPS
    USBJOJOH Discriminator acts as the teacher
29. G SFBM
    GBLF D SFBM
    GBLF D D(G(noise)) ˠ fake D(photo) ˠ real %JTDSJNJOBUPS
    USBJOJOH Generator provides negative examples
31. https://www.youtube.com/watch?v=rs3aI7bACGc ©Yota Ishida
32. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford, Luke Metz, Soumith Chintala (Submitted on 19 Nov 2015 (v1), last revised 7 Jan 2016 (this version, v2)) https://arxiv.org/abs/1511.06434
33. $POEJUJPOBM
    ("/T
34. G NBMF
    GFNBMF
    DIJME
    FMEFSMZ G(noise | conditions) $POEJUJPOBM
    (FOFSBUPS
35. SJHIU
    XSPOH NBMF
    GFNBMF
    DIJME
    FMEFSMZ D $POEJUJPOBM
    %JTDSJNJOBUPS
36. SJHIU
    XSPOH NBMF
    GFNBMF
    DIJME
    FMEFSMZ D SJHIU
    XSPOH NBMF
    GFNBMF
    DIJME
    FMEFSMZ SJHIU
    XSPOH NBMF
    GFNBMF
    DIJME
    FMEFSMZ D D
37. SJHIU
    XSPOH D $POEJUJPOBM
    ("/ https://arxiv.org/abs/1411.1784 Conditional Generative Adversarial Nets Mehdi Mirza, Simon Osindero (Submitted on 6 Nov 2014) Generator Discriminator NBMF
    GFNBMF
    DIJME
    FMEFSMZ G NBMF
    GFNBMF
    DIJME
    FMEFSMZ same condition
38. G NBMF
    GFNBMF
    DIJME
    FMEFSMZ SJHIU
    XSPOH NBMF
    GFNBMF
    DIJME
    FMEFSMZ D $POEJUJPOBM
    ("/ Discriminator Generator
39. https://www.faceapp.com/ Disclaimer: FaceApp authors don’t disclose their method. This is only my guess. It may have nothing to do with GANs. original
40. original https://www.faceapp.com/
41. https://www.faceapp.com/ original
42. "SUJTUJD
    4UZMF
    5SBOTGFS Improved!
43. https://prisma-ai.com/
44. https://prisma-ai.com/ https://prisma-ai.com/
45. https://prisma-ai.com/ https://prisma-ai.com/
46. https://prisma-ai.com/ https://prisma-ai.com/
47. https://arxiv.org/abs/1603.08155 transformation network loss network Gram matrices in feature space pre-trained content image style image
48. “Gram matrices in feature space” https://en.wikipedia.org/wiki/Gramian_matrix
49. https://www.youtube.com/watch?v=xVJwwWQlQ1o
50. $ZDMF
    ("/
51. https://github.com/junyanz/CycleGAN
52. https://github.com/junyanz/CycleGAN
53. https://github.com/junyanz/CycleGAN
54. (FOFSBUPS transformation network https://arxiv.org/abs/1603.08155
55. GBLF
    IPSTF GBLF
    IPSTF … %JTDSJNJOBUPS fully convolutional judges patches 
 of the input image https://arxiv.org/abs/1603.08155
56. "EWFSTBSJBM
    -PTT X F G Y GBLF
    [FCSB GBLF
    [FCSB … GBLF
    IPSTF GBLF
    IPSTF … X(F(horse)) ˠ classify as zebra Y(F(zebra)) ˠ classify as horse
57. $ZDMF
    -PTT G F G(F(image))ˠ the same image F G F(G(image))ˠ the same image
58. https://www.youtube.com/watch?v=9reHvktowLY
59. 5IF
    &OE