CompML: 次元削減によるDNN特徴量空間の可視化

CompMLでの発表資料です。次元削減の概要とDNNが生成する特徴量空間の可視化について説明しています。

Transcript

1. CompML ࣍ݩ࡟ݮʹΑΔDNNಛ௃ྔۭؒͷ ՄࢹԽ Ryo Okuda (@RyoOkudaJ)
2. CompML TL;DR • ࣍ݩ࡟ݮͷ֓ཁʹ͍ͭͯ঺հ • ࣍ݩ࡟ݮΛ༻͍ͨDNNಛ௃ྔۭؒͷՄࢹԽʹ͍ͭͯ঺հ 2
3. CompML ࣍ݩ࡟ݮͷ֓ཁ
4. CompML ࣍ݩ࡟ݮͱ͸ ߴ࣍ݩ࠲ඪ͕༩͑ΒΕͨΦϒδΣΫτͷ௿࣍ݩ࠲ඪΛٻΊΔ͜ͱͰɺ
 ՄࢹԽʹΑΔ෼ੳ΍ϊΠζͷ࡟আΛߦ͏ख๏ɻ 4 ΦϒδΣΫτͷ෼෍ ߴ࣍ݩ࠲ඪ ௿࣍ݩ࠲ඪ
5. CompML ࣍ݩ࡟ݮͷ໰୊ઃఆ ΦϒδΣΫτ ʹߴ࣍ݩ࠲ඪ ͕༩͑ΒΕͨσʔλू߹ ͔Β ΦϒδΣΫτ ʹରԠ͢Δ௿࣍ݩ࠲ඪ ΛٻΊΔɻͨͩ͠ɺ ྲྀΕ͸ओʹҎԼ̎ͭ 1. ௿࣍ݩ࠲ඪΛ༩͑ͨσʔλू߹ Λ௚઀ܭࢉ 2. ͱ͍͏ؔ܎Λ༩͑ΔࣹӨΛٻΊɺ௿࣍ݩ࠲ඪΛܭࢉ oi xi D = {(oi , xi ) |(1 ≤ i ≤ N)} oi yi xi ∈ RD, yi ∈ Rd, D > > d D′ = {(oi , yi ) |(1 ≤ i ≤ n)} yi = f(xi ) 5
6. CompML ࣍ݩ࡟ݮͷྫ: Locally Linear Embedding [Sam et al., 2000[1]] ΦϒδΣΫτ ͷ࠲ඪ Λۙ๣ͷΦϒδΣΫτ ͷ࠲ඪ ͷઢܗ࿨Ͱۙࣅ Λ༻͍ͯ Λֶशͨ͠ޙʹɺ Λ༻͍ͯ Λֶश ߴ࣍ݩ࠲ඪ: ௿࣍ݩ࠲ඪ: oi xi oij xij xi wij wij yi ε(W ) = ∑ i |xi − ∑ j wij xij |2 ϕ(y) = ∑ i |yi − ∑ j wij yij |2 6
7. CompML ࣍ݩ࡟ݮͷྫ: SNE [Hinton et al., 2002[2]] ΦϒδΣΫτ ͱ ͷۙ๣ (৚݅෇͖) ֬཰Λ࠲ඪʹج͖ͮܭࢉ͠ɺ
 ߴ࣍ݩۭؒͰͷ֬཰෼෍ͱ௿࣍ݩۭؒͰͷ֬཰෼෍ͱͷڑ཭Λখ͘͢͞Δ ߴ࣍ݩ࠲ඪ: ௿࣍ݩ࠲ඪ: oi oj pj|i = exp( − ||xi − xj ||2 /2σ2 i ) ∑ k≠i exp( − ||xi − xk ||2 /2σ2 i ) qj|i = exp( − ||yi − yj ||2 /2σ2 i ) ∑ k≠i exp( − ||yi − yk ||2 /2σ2 i ) 7 MNISTͷՄࢹԽ
8. CompML ࣍ݩ࡟ݮͷྫ: SNE [Hinton et al., 2002[2]] ΦϒδΣΫτ ͱ ͷۙ๣ (৚݅෇͖) ֬཰Λ࠲ඪʹج͖ͮܭࢉ͠ɺ
 ߴ࣍ݩۭؒͰͷ֬཰෼෍ͱ௿࣍ݩۭؒͰͷ֬཰෼෍ͱͷڑ཭Λখ͘͢͞Δ ڑ཭: ߋ৽ࣜ: oi oj C = ∑ i KL(Pi ||Qi ) = ∑ i,j pj|i log pj|i qj|i δC δyi = 2∑ j (pj|i − qj|i + pi|j − qi|j )(yi − yj ) 8 MNISTͷՄࢹԽ
9. CompML ࣍ݩ࡟ݮͷྫ: SNE [Hinton et al., 2002[2]] ໰୊఺. 1
 ߋ৽͕ࣜෳࡶͰܭࢉ͕஗͍ δC δyi = 2∑ j (pj|i − qj|i + pi|j − qi|j )(yi − yj ) 9
10. CompML ໰୊఺. 2
 Crowding Problem • ࣍ݩ਺ + 1 ·Ͱ͔͠ΦϒδΣΫτΛۭؒʹ౳ִؒʹ഑ஔͰ͖ͳ͍ • ௿࣍ݩۭؒͰখ͞ͳڑ཭Λਖ਼֬ʹදݱ͠Α͏ͱ͢Δͱɺߴ࣍ݩۭؒͰҰఆҎ্ͷڑ཭͕ ͋ͬͨΦϒδΣΫτಉ࢜͸௿࣍ݩۭؒͰඇৗʹԕ͘ʹ཭ΕΔ • ԕํʹ͋Δଟ਺ͷΦϒδΣΫτ͕ۭؒͷத৺ͰͷΫϥελܗ੒Λ્֐͢Δ 10 ࣍ݩ࡟ݮͷྫ: SNE [Hinton et al., 2002[2]]
11. CompML ࣍ݩ࡟ݮͷྫ: t-SNE [Laurens et al., 2008[3]] ख๏. 1 Symmetric SNE ৚݅෇͖෼෍ͷ୅ΘΓʹಉ࣌෼෍Λ༻͍ͯଛࣦؔ਺Λର৅Խͤ͞ɺෳࡶ͞Λղফ ߴ࣍ݩ࠲ඪ:
 pj|i = exp( − ||xi − xj ||2 /2σ2 i ) ∑ k≠i exp( − ||xi − xk ||2 /2σ2 i ) pij = pi|j + pj|i 2N 11 MNISTͷՄࢹԽ
12. CompML ࣍ݩ࡟ݮͷྫ: t-SNE [Laurens et al., 2008[3]] ख๏. 1 Symmetric SNE ৚݅෇͖෼෍ͷ୅ΘΓʹಉ࣌෼෍Λ༻͍ͯଛࣦؔ਺Λର৅Խͤ͞ɺෳࡶ͞Λղফ ڑ཭: ߋ৽ࣜ: C = ∑ i KL(Pi ||Qi ) = ∑ i,j pij log pij qij δC δyi = 4∑ j (pij − qij )(yi − yj ) 12 MNISTͷՄࢹԽ
13. CompML ࣍ݩ࡟ݮͷྫ: t-SNE [Laurens et al., 2008[3]] ख๏. 2 t-Distributed SNE ௿࣍ݩۭؒͰͷ෼෍ͷܭࢉʹɺΨ΢ε෼෍ΑΓ΋੄໺͕ߴ͍ t ෼෍Λ༻͍ͯ
 Crowding ProblemΛܰݮ ௿࣍ݩ࠲ඪ: 
 qij = (1 + ||yi − yj ||2 )−1 ∑ k≠l (1 + ||yk − yl ||2 )−1 13 MNISTͷՄࢹԽ
14. CompML ࣍ݩ࡟ݮͷྫ: t-SNE [Laurens et al., 2008[3]] ख๏. 2 t-Distributed SNE ௿࣍ݩۭؒͰͷ෼෍ͷܭࢉʹɺΨ΢ε෼෍ΑΓ΋੄໺͕ߴ͍ t ෼෍Λ༻͍ͯ
 Crowding ProblemΛܰݮ ڑ཭: ߋ৽ࣜ: C = ∑ i KL(Pi ||Qi ) = ∑ i,j pij log pij qij δC δyi = 4∑ j (pij − qij )(yi − yj )(1 + ||yi − yj ||2 )−1 14 MNISTͷՄࢹԽ
15. CompML ࣍ݩ࡟ݮ͕ѻ͏σʔλ ࠓճ͸ҎԼೋͭΛ঺հ 1. (લड़) ߴ࣍ݩͷ࠲ඪ΍ΦϒδΣΫτಉ࢜ͷྨࣅ౓͕༩͑ΒΕ͍ͯΔσʔλ 2. (ࠓ͔Β঺հ) 1ͷ࠲ඪ΍ྨࣅ౓͕࣌ࠁຖʹ༩͑ΒΕ͍ͯΔσʔλ 1 ͸͋Δ࣌ࠁͰͷεφοϓγϣοτͰ 2 ͸࣌ܥྻσʔλͷΑ͏ͳ΋ͷ 15
16. CompML ࣌ܥྻΛѻ͏࣍ݩ࡟ݮ Ϟνϕʔγϣϯ • ຖ࣌ࠁͷσʔλʹผʑʹ࣍ݩ࡟ݮΛద༻͢Δͱ
 ௿࣍ݩ࠲ඪʹҰ؏ੑ͕ͳ͍ • ߴ࣍ݩͷσʔλ෼෍͕େ͖͘มԽͯ͠ͳͯ͘΋
 ௿࣍ݩ࠲ඪͷ෼෍͕มԽ͢Δ 16 Ϋϥεؒڑ཭͕େ͖͘ͳΔ
 ଟ࣍ݩΨ΢ε෼෍ͷՄࢹԽ
17. CompML Dynamic t-SNE [Paulo et al., 2016[4]] ख๏ ௿࣍ݩۭؒͰ࣌ࠁؒͷ෼෍ؒڑ཭ʹਖ਼ଇԽ߲ΛՃ͑Δ C(t) = N ∑ i=1 KL(Pi (t)||Qi (t)) = N ∑ i,j=1 pij (t)log pij (t) qij (t) C = T ∑ t=1 C(t) + λ 2N N−1 ∑ i=1 T−1 ∑ t=1 ||Qi (t) − Qi (t + 1)||2 17 Ϋϥεؒڑ཭͕େ͖͘ͳΔ
 ଟ࣍ݩΨ΢ε෼෍ͷՄࢹԽ
18. CompML Dynamic t-SNE [Paulo et al., 2016[4]] DNNಛ௃ྔۭؒͷՄࢹԽ SVHN Λ CNN Ͱֶशͤ͞ɺCNN ͷ࠷ޙͷӅΕ૚
 ͕ܗ੒͢Δಛ௃ྔۭؒ ( ) ΛՄࢹԽ D = 512 18
19. CompML Dynamic t-SNEͷॴײ numpy౳ͷ࣮૷ͩͱɺ࣮ߦ͕஗͍ • ਺෼ఔ౓Ͱॲཧ͕ऴΘΒͳ͍ͱ࣮ݧαΠΫϧʹ૊ΈࠐΊͳ͍ • ͦ΋ͦ΋ t-SNE ͕஗্͍ʹɺ࣌ܥྻσʔλͷ਺͕ଟ͍ͷͰ͔ͳΓ஗͍ • Dynamic t-SNE ͷ CUDA Խ΍ t-SNE ΑΓߴ଎ͳख๏ͷ࣌ܥྻσʔλ΁ͷ֦ு͕ඞཁ 19
20. CompML DNNಛ௃ྔۭؒͷՄࢹԽ
21. CompML DNNಛ௃ྔۭؒʹ͍ͭͯ ݱঢ়ͷ໰୊ઃఆ ՄࢹԽۭͯؒ͠ͷܗ੒্͕ख͍͍ͬͯ͘ΔͷΛ֬ೝ (ݸਓతʹ෺଍Γͳ͍) • t-SNE ʹΑͬͯυϝΠϯؒͷΫϥελߏ଄ΛՄࢹԽ • Dynamic t-SNEʹΑͬͯ൑ผ෼ੳʹదۭ͕ͨؒ͠ܗ੒͞ΕΔաఔΛՄࢹԽ 21
22. CompML ՄࢹԽͷϞνϕʔγϣϯ Ϟσϧ։ൃ͕ద੾ͳํ޲ʹ޲͔͍ͬͯΔ͔ௐ΂Δख๏Λ࡞Γ͍ͨ • learning curve౳ͷεΧϥʔͷมԽΛݟΔҎ֎ʹ΋ɺϞσϧͷσόοάํ๏͕ཉ͍͠ (ֶश͕ࣦഊ͢ΔݪҼΛՄࢹԽͰ͍ࣔࠦͨ͠) • ӅΕ૚ͷಛ௃ྔۭؒͰͷσʔλ෼෍ͷՄࢹԽ͸ t-SNE ͕༻͍ΒΕ΍͍͕͢ɺผͷख๏ ʹΑͬͯΫϥελߏ଄Ҏ֎ʹ΋நग़Ͱ͖Δ৘ใ͕͋Δ͔΋͠Εͳ͍ 22
23. CompML ༧ଌ͞ΕΔ໰୊఺ 1. ࣮ߦ࣌ؒ: શ࣌ࠁͷ৘ใΛࢀরͯ͠࠲ඪΛܭࢉ͢Δͱ஗͘ɺ࣮ݧαΠΫϧʹ૊ΈࠐΊͳ͍
 
 2. σʔλ਺: DNNͰ༻͍Δσʔλ਺͸ଟ͘ɺద੾ͳ෦෼ू߹Λબ୒ͯ͠ՄࢹԽ͢Δඞཁ༗Γ C(t) = N ∑ i=1 KL(Pi (t)||Qi (t)) = N ∑ i,j=1 pij (t)log pij (t) qij (t) C = T ∑ t=1 C(t) + λ 2N N−1 ∑ i=1 T−1 ∑ t=1 ||Qi (t) − Qi (t + 1)||2 23
24. CompML ࣮ݧ (ਐߦத) ҎԼͷ৚݅Ͱ࣮ݧ༧ఆ • ֶश্͕ख͍͘͘৔߹ • ֶश͕ࣦഊ or े෼ͳਫ਼౓͕ग़ͳ͍৔߹ • mixup౳ͷಛ௃ྔۭؒʹհࡏ͢Δख๏Λ༻͍ͨ৔߹ • VAE౳ͷۭؒʹಛఆͷߏ଄ΛԾఆͰ͖Δ৔߹ • Domain Adaptation౳ͷෳ਺ͷσʔλ෼෍͕ଘࡏ͢Δ৔߹ 24
25. CompML ՄࢹԽ݁Ռͷղऍ ಉ͡σʔλͰ΋ɺ࢖༻͢Δ࣍ݩ࡟ݮͷख๏ʹΑͬͯՄࢹԽ݁Ռ͸શ͘ҟͳΔ [ࢀߟ] ·ͨɺՄࢹԽۭͨؒ͠ͷڑ཭ͱݩͷۭؒͷڑ཭ͷҙຯ͕ҧ͏৔߹΋͋Δ e.g. t-SNE: ΫϥελੑΛநग़͢Δ͕ɺҰఆҎ্ͷڑ཭ͷ৘ใ΍֯౓ͷ৘ใʹҙຯ͸ແ͍ ݩʑͷಛ௃ྔۭؒͷߏ଄ʹదͨ࣍͠ݩ࡟ݮͷख๏Λબ୒͢Δඞཁ͕͋Δ 25
26. CompML ࢀߟจݙ • [1] Roweis, Sam T and Saul, Lawrence K. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323–2326, 2000. • [2] G.E. Hinton and S.T. Roweis. Stochastic Neighbor Embedding. In Advances in Neural Information Processing Systems, volume 15, pages 833–840, Cambridge, MA, USA, 2002. The MIT Press. • [3] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, Vol. 9, pp. 2579–2605, 2008. • [4] Paulo E. Rauber, Alexandre X. Falcão, and Alexandru C. Telea. Visualizing time- dependent data using dynamic t-SNE. Proc. EuroVis Short Papers, 2(5), 2016. 26