Poincare Embeddings で遊んでみた

CompML の発表資料です。

Transcript

1. CompML Poincare Embeddings Ryo Okuda (@RyoOkudaJ), 2021-05-14
2. CompML TL; DR • Embedding ͷ֓ཁʹ͍ͭͯ঺հ • Poincare Embeddings ͷ֓ཁʹ͍ͭͯ঺հ • ݪஶͱ͸ҟͳΔσʔληοτͰͷ࣮ݧ݁ՌΛ঺հ 2
3. CompML Embedding ͷ֓ཁ
4. CompML ΦϒδΣΫτͱΦϒδΣΫτಉ࢜ͷؔ܎ (e.g. ڞىؔ܎) ͕༩͑ΒΕͨͱ͖ʹɺ 
 ֤ΦϒδΣΫτʹ࠲ඪΛ༩͑Δ͜ͱͰ਺஋తͳѻ͍ΛՄೳʹ͢Δख๏ɻ ྨࣅͨ͠ΦϒδΣΫτಉ࢜Λۙ๣࠲ඪʹ෍ஔ͢Δ৔߹͕ଟ͍ɻ Embedding ͱ͸ (1/2) 4 Embedding Space ࣅͯΔʂʂ
5. CompML Embedding ͱ͸ (2/2) • ೖྗ: • ΦϒδΣΫτͷू߹: • ೋͭͷΦϒδΣΫτ ͷڞىؔ܎΍਌ࢠؔ܎Λࣔ͢σʔλ: • ग़ྗ: • ֤ΦϒδΣΫτͷ࠲ඪू߹: X = {xi |1 ≤ i ≤ N} xi , xj ∈ X D Y = {yi |1 ≤ i ≤ N} 5
6. CompML Embedding ͷྫ: Word2Vec (1/2) Skip-Gram [Mikolov et al., 2013[1]]: ֓ཁ • ฏۉର਺໬౓Λ࠷େԽ͢Δ͜ͱͰ จষ ͔Β 
 ୯ޠϕΫτϧΛܭࢉՄೳ 
 
 1 T T ∑ t=1 ∑ −c≤j≤c,j≠0 logp(wt+j |wt ) p(wO |wI ) = exp(v′  T wO vwI ) ∑W w=1 exp(v′  T w vwI ) 6
7. CompML Embedding ͷྫ: Word2Vec (2/2) Skip-Gram [Mikolov et al., 2013[1]]: ࣮ݧ • ʮDNN ͷ embedding ʹར༻ͯ͠λεΫͷੑೳΛධՁʯͱ͍͏ྲྀΕͰ͸ͳ͍ • ఆੑධՁͷ໘ന͞ͱֶश଎౓ΛΞϐʔϧ 7
8. CompML Poincare Embeddings ͷ֓ཁ
9. CompML Poincare Embeddings ͱ͸ (1/4) Poincare Space • ࠲ඪ ؒͷڑ཭͕ҎԼͰఆ·Δ૒ۂۭؒ 
 • த৺͔Β཭ΕΔ΄Ͳ఺ͷ਺͕૿͑Δٿঢ়ͷۭؒͰɺ 
 ֊૚ੑͷ͋ΔΦϒδΣΫτΛ embed ͠΍͍͢ u, v d(u, v) = arcosh(1 + 2 ||u − v||2 (1 − ||u||2 )(1 − ||v||2 ) ) 9
10. CompML Poincare Embeddings ͱ͸ (2/4) Embed ݁Ռͷྫ [Nickel et al., 2017[2]] ಈ෺ͷ֊૚ੑΛ Wordnet ͔Βநग़ • Mammal -> Rodent • Rodent -> Squirrel 10
11. CompML Poincare Embeddings ͱ͸ (3/4) ֶशํ๏ (WordNet ͷ৔߹) 1. ਌ࢠؔ܎ͷ͋ΔΦϒδΣΫτೋͭͷ૊ͷू߹ Λੜ੒ 2. ͔ΒωΨςΟϒαϯϓϧ Λੜ੒ 3. ҎԼͷଛࣦؔ਺Λ࠷దԽ֤ͯ͠ΦϒδΣΫτʹ࠲ඪΛ༩͑Δ D = {(u, v)|u ∈ v} D N( ⋅ ) 11
12. CompML Poincare Embeddings ͱ͸ (4/4) ࣮ݧ: 3 छྨͷσʔληοτʹର͠ఆྔධՁ • Network Reconstruction, Link Prediction (DNN ͸ؔ܎ͳ͘ɺ 
 Poincare Space ͰΦϒδΣΫτͷۙ๣ؔ܎Λอ͍ͯͯΔ͔ධՁ) • ௿࣍ݩͰߴ͍ੑೳΛग़ͤΔ͜ͱΛΞϐʔϧ 12
13. CompML Poincare Embeddings Ͱ ༡ΜͰΈͨ
14. CompML Poincare Embeddings ʹ͍ͭͯͷٙ໰ ҎԼࡾͭͷσʔληοτҎ֎ʹ΋ Citation Network Λ࢖͑ͦ͏ • WordNet: ਌ࢠؔ܎͕ࣗ໌ͳ༗޲άϥϑ • ໦ߏ଄ͷ਌ࢠؔ܎Λͦͷ··ೖྗʹ࢖༻Մೳ • Co-author NetWork: ਌ࢠؔ܎͕ඇࣗ໌ͳແ޲άϥϑ • ڞஶάϥϑ͔ΒΦϒδΣΫτؒͷۙ๣֬཰Λܭࢉ • Lexical Entailment: ΦϒδΣΫτؒͷؚҙؔ܎Λࣔ͢ू߹ • X ͕ Y ʹଐ͢Δఔ౓Λࣔ͢஋͔ΒείΞ(ͱ Spearman’s ρ) Λܭࢉ 14
15. CompML Citation Network ͱ͸ • ֓ཁ • ࿦จͷҾ༻ / ඃҾ༻ͷؔ܎Λࣔͨ͠άϥϑ • Ҿ༻ -> ࢠ, ඃҾ༻ -> ਌ ʹม׵͢Ε͹ Poincare Embeddings Λద༻Ͱ͖ͦ͏ 15
16. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (1/5) • ࣮ݧઃఆ • ֶश • Ҿ༻ -> ࢠ, ඃҾ༻ -> ਌ ʹม׵͠ೖྗσʔλੜ੒ • ೖྗσʔλʹର͠ Poincare Embeddings • ධՁ • ՄࢹԽʹΑΔఆੑධՁ • Network Reconstruction Error ʹΑΔఆྔධՁ • Ծઆ: WordNet ͱҟͳΓෳ਺ͷ໦ߏ଄͕ଘࡏ͢ΔͷͰධՁ͸Լ͕Δ 16
17. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (2/5) ఆੑධՁ dim = 2 ͷ Embed ݁ՌΛՄࢹԽ • σʔλ͕ଟ͍ͷͰີू͍ͯ͠Δ • ֊૚͝ͱʹ෼཭͍ͯ͠ͳ͍ 17
18. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (3/5) ఆྔධՁ (1/2) MAP (Mean Average Precision) (ߴ͍΄Ͳྑ͍) • dim = 2 ͱͦͷଞͰ͕ࠩ͋Δ • Network ͰͷੑೳΑΓ΋ѱ͍ 18
19. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (4/5) ఆྔධՁ (2/2): Mean Rank (௿͍΄Ͳྑ͍) • dim = 2 ͱͦͷଞͰ͕ࠩ͋Δ • WordNet ͰͷੑೳΑΓ΋ѱ͍ 
 (Network Ͱͷ Mean Rank ͸ݪஶະهࡌ) 19
20. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (5/5) ࣮ݧͷ·ͱΊ • Citation Network ΛφΠʔϒʹֶशͤͯ͞΋࿦จ΄Ͳͷੑೳ͸ग़ͳ͔ͬͨ • ਌ͱͯ͠Χ΢ϯτ͞Ε͍ͯΔΦϒδΣΫτͷݸ਺౳Λ෼ੳͯ͠σʔλͷҧ͍ Λௐ΂Δඞཁ͋Γ • ࠓճ͸ Citation Network ͱ gensim ͷ WordNet ·Ͱ͸෼ੳࡁΈ • gensim ͱ Facebook Research Ͱσʔληοτ͕ҟͳΔͷͰɺ·ͩޙऀ ͷσʔλΛ෼ੳͰ͖ͯͳ͍ 20
21. CompML ࢀߟจݙ [1] Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg, and Dean, Jeffrey. Distributed representations of phrases and their compositionality. In Advances on Neural Information Processing Systems, 2013. [2] Maximillian Nickel and Douwe Kiela. Poincare embeddings for learning hierarchical representations. In Advances in Neural Information Processing Systems, 2017. 21