Share
Embed
Isotropy, Clusters, and Classifiers
2024-08-25,26: 第16回 最先端NLP勉強会
https://sites.google.com/view/snlp-jp/home/2024
Hayato Tsukagoshi
August 19, 2024
Transcript
36 slides
-
Isotropy, Clusters, and Classi fi ers D2, Graduate School of
Informatics, Nagoya University, Japan Hayato Tsukagoshi Timothee Mickus, Stig-Arne Grönroos, Joseph Attieh ACL 2024 https://aclanthology.org/2024.acl-short.7/ -
•TL;DR: ຒΊࠐΈදݱͷํੑ্ • ҙຯతྨࣅΛଌఆ͢ΔλεΫ (e.g. STS)ͷੑೳʹͱͬͯ༗ӹ • ΫϥελϦϯάλεΫͷੑೳʹͱͬͯෆརӹ •ߴ࣭ͳຒΊࠐΈදݱͷ֫ಘʹํੑ͕ॏཁͱ৴͡ΒΕ͖ͯͨ •
ํੑ: ຒΊࠐΈදݱ͕ۭؒதʹࢄΒ͍ͬͯͯ͠Δ͔ʁ •ຊݚڀඞͣ͠ํੑͷ্͕λεΫੑೳ্ʹد༩͠ͳ͍͜ͱΛࢦఠ • ͔ͳΓఏىدΓͷจ •࣮ࡍʹԼྲྀλεΫͰΫϥελϦϯάͱͯ͠ͷྑ͞ͱํੑʹͱͬͯͷྑ͞ͷ ૬ݱΛ֬ೝͨ͠ ֓ཁ 2 -
•ํత: ຒΊࠐΈදݱ͕ۭؒதʹࢄΒ͍ͬͯͯ͠Δ͜ͱ • ֶతఆٛ: ࢄڞࢄߦྻ͕୯Ґߦྻʹൺྫ͍ͯ͠Δ͜ͱ •ҟํత: ຒΊࠐΈදݱ͕ۭؒதʹภ͍ͬͯͯ͠Δ͜ͱ Isotropy (ํੑ) ͱ
Anisotropy (ҟํੑ) 5 ਤ Rudman et al. 2021 ΑΓҾ༻ ҟํతͳຒΊࠐΈ ͪΐͬͱํతʹͳͬͨຒΊࠐΈ -
•ํత: ຒΊࠐΈදݱ͕ۭؒதʹࢄΒ͍ͬͯͯ͠Δ͜ͱ • ֶతఆٛ: ࢄڞࢄߦྻ͕୯Ґߦྻʹൺྫ͍ͯ͠Δ͜ͱ •ҟํత: ຒΊࠐΈදݱ͕ۭؒதʹภ͍ͬͯͯ͠Δ͜ͱ Isotropy (ํੑ) ͱ
Anisotropy (ҟํੑ) 6 ਤ Rudman et al. 2021 ΑΓҾ༻ ҟํతͳຒΊࠐΈ ͪΐͬͱํతʹͳͬͨຒΊࠐΈ ;ΜΘΓ Isotropy ͔ͬͪΓ Isotropy -
•Word2VecGloVeͳͲͷֶशࡁΈ੩త୯ޠຒΊࠐΈҟํతʹ [1] •BERTGPT-2ͳͲͷจ຺Խ୯ޠຒΊࠐΈҟํతʹ [2] •→ʮํੑΛ্ͤ͞Δ͜ͱ͕ੑೳ্ʹͭͳ͕Δͣʯͱ͍͏৴ڼͷੜ •ํੑ্ͷఆੑతޮՌ (ಛʹന৭Խ) • ੩త୯ޠຒΊࠐΈʹରͯ͠: ୯ޠසʹΑΔόΠΞεΛআڈ͢Δ
[3] • ಈత୯ޠຒΊࠐΈʹରͯ͠: ͳΜΒ͔ଞͷόΠΞεΛআڈ͢Δ •ํੑΛ্ͤͭͭ͞දݱֶश͢Δख๏ͱͯ͠ରরֶश͕಄ [1] Mu et al., All-but-the-Top: Simple and E ff ective Postprocessing for Word Representations, arXiv 2017 [2] Ethayaraja, How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings, arXiv 2019 [3] Sasaki et al., Examining the e ff ect of whitening on static and contextualized word embeddings, Information Processing & Management 2023 ຒΊࠐΈදݱͱํੑ 7 [ -
•Word2VecGloVeͳͲͷֶशࡁΈ੩త୯ޠຒΊࠐΈҟํతʹ [1] •BERTGPT-2ͳͲͷจ຺Խ୯ޠຒΊࠐΈҟํతʹ [2] •→ʮํੑΛ্ͤ͞Δ͜ͱ͕ੑೳ্ʹͭͳ͕Δͣʯͱ͍͏৴ڼͷੜ •ํੑ্ͷఆੑతޮՌ (ಛʹന৭Խ) • ੩త୯ޠຒΊࠐΈʹରͯ͠: ୯ޠසʹΑΔόΠΞεΛআڈ͢Δ
[3] • ಈత୯ޠຒΊࠐΈʹରͯ͠: ͳΜΒ͔ଞͷόΠΞεΛআڈ͢Δ •ํੑΛ্ͤͭͭ͞දݱֶश͢Δख๏ͱͯ͠ରরֶश͕಄ [1] Mu et al., All-but-the-Top: Simple and E ff ective Postprocessing for Word Representations, arXiv 2017 [2] Ethayaraja, How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings, arXiv 2019 [3] Sasaki et al., Examining the e ff ect of whitening on static and contextualized word embeddings, Information Processing & Management 2023 ຒΊࠐΈදݱͱํੑ 8 [ -
•දݱֶश (representation learning) ͷख๏ͷҰͭ •ਖ਼ྫಉ͕࢜ۙͮ͘Α͏ʹɺ͔ͭɺෛྫಉ͕࢜ΕΔΑ͏ʹֶश͢Δ • ਖ਼ྫಉ࢜ͷྨࣅ࠷େԽ & ෛྫಉ࢜ͷྨࣅ࠷খԽ ଛࣦ
(InfoNCE [4]) ͷܭࢉ •ਖ਼ྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅΛٻΊΔ •ෛྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅΛٻΊΔ •ྨࣅΛฒͯԹύϥϝʔλΛద༻͢Δ •SoftmaxؔΛద༻ͯ֬͠ͱΈͳ͢ •ਖ਼ྫʹ͚ͩ1ཱ͕ͭʹ͚ۙͮΔ [4] Oord et al., Representation Learning with Contrastive Predictive Coding, arXiv 2018 ରরֶश 9 -
•දݱֶश (representation learning) ͷख๏ͷҰͭ •ਖ਼ྫಉ͕࢜ۙͮ͘Α͏ʹɺ͔ͭɺෛྫಉ͕࢜ΕΔΑ͏ʹֶश͢Δ • ਖ਼ྫಉ࢜ͷྨࣅ࠷େԽ & ෛྫಉ࢜ͷྨࣅ࠷খԽ ଛࣦ
(InfoNCE [4]) ͷܭࢉ •ਖ਼ྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅΛٻΊΔ •ෛྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅΛٻΊΔ •ྨࣅΛฒͯԹύϥϝʔλΛద༻͢Δ •SoftmaxؔΛద༻ͯ֬͠ͱΈͳ͢ •ਖ਼ྫʹ͚ͩ1ཱ͕ͭʹ͚ۙͮΔ [4] Oord et al., Representation Learning with Contrastive Predictive Coding, arXiv 2018 ରরֶश 10 -
•ରরֶशʹ͓͚Δදݱͷ “ྑ͞” ͷࢦඪ [5] • ٿ໘্ʹຒΊࠐΈ͕Ͳ͏͍ͯ͠Δ͔ʁ Alignment •ਖ਼ྫϖΞͷຒΊࠐΈ͕े͍͍ۙͮͯΔ͔ʁ Uniformity •ຒΊࠐΈදݱͷฏۉೋؒڑ
•ํੑͷධՁʹར༻͞ΕΔ͜ͱ͕͋Δ (e.g. SimCSE) • ݫີʹํੑଌΕ͍ͯͳ͍ (ࢄڞࢄߦྻΛݟ͍ͯͳ͍) [5] Wang et al., Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020 ํੑͷଌΓํ: Alignment & Uniformity 11 -
•ରরֶशʹ͓͚Δදݱͷ “ྑ͞” ͷࢦඪ [5] • ٿ໘্ʹຒΊࠐΈ͕Ͳ͏͍ͯ͠Δ͔ʁ Alignment •ਖ਼ྫϖΞͷຒΊࠐΈ͕े͍͍ۙͮͯΔ͔ʁ Uniformity •ຒΊࠐΈදݱͷฏۉೋؒڑ
•ํੑͷධՁʹར༻͞ΕΔ͜ͱ͕͋Δ (e.g. SimCSE) • ݫີʹํੑଌΕ͍ͯͳ͍ (ࢄڞࢄߦྻΛݟ͍ͯͳ͍) [5] Wang et al., Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML 2020 ํੑͷଌΓํ: Alignment & Uniformity 12 -
•ࢦඪͷཧղ: ۭؒதͷͲΕ͘Β͍ͷ࣍ݩΛۉʹར༻͍ͯ͠Δ͔ͷׂ߹ •ܭࢉํ๏ͷେࡶͳཧղ: • ຒΊࠐΈͷू߹ΛPCAͯ࣍͠ݩ͝ͱແ૬ؔԽ • ࢄϕΫτϧΛͦͷϊϧϜͰׂͬͯਖ਼نԽˠ 1ϕΫτϧͱͷζϨΛଌΔ [6] Rudman
et al., IsoScore: Measuring the Uniformity of Embedding Space Utilization, ACL 2022 fi ndings ํੑͷଌΓํ: IsoScore [6] 13 0.9996 0.6105 0.0281 2࣍ݩΨεʹ͓͚Δܗঢ়ͱIsoScoreͷؔ -
•ࢦඪͷཧղ: ۭؒதͷͲΕ͘Β͍ͷ࣍ݩΛۉʹར༻͍ͯ͠Δ͔ͷׂ߹ •ܭࢉํ๏ͷେࡶͳཧղ: • ຒΊࠐΈͷू߹ΛPCAͯ࣍͠ݩ͝ͱແ૬ؔԽ • ࢄϕΫτϧΛͦͷϊϧϜͰׂͬͯਖ਼نԽˠ 1ϕΫτϧͱͷζϨΛଌΔ [6] Rudman
et al., IsoScore: Measuring the Uniformity of Embedding Space Utilization, ACL 2022 fi ndings ํੑͷଌΓํ: IsoScore [6] 14 0.9996 0.6105 0.0281 2࣍ݩΨεʹ͓͚Δܗঢ়ͱIsoScoreͷؔ -
•ΫϥελϦϯάʹ͓͚Δྑ͞ΛଌΔͨΊͷࢦඪ [7] • ಉΫϥε͕ۙ͘ɺผΫϥε͕ԕ͍͔ʁ [7] Rousseeuw, Silhouettes: A graphical aid
to the interpretation and validation of cluster analysis, Journal of Computational and Applied Mathematics 1987. Silhouette score: γϧΤοτείΞ 16 ∈ [−1,1] -
•ΫϥελϦϯάʹ͓͚Δྑ͞ΛଌΔͨΊͷࢦඪ [7] • ಉΫϥε͕ۙ͘ɺผΫϥε͕ԕ͍͔ʁ [7] Rousseeuw, Silhouettes: A graphical aid
to the interpretation and validation of cluster analysis, Journal of Computational and Applied Mathematics 1987. Silhouette score: γϧΤοτείΞ 17 ∈ [−1,1] -
•ΫϥελϦϯάʹ͓͚Δྑ͞ΛଌΔͨΊͷࢦඪ [7] • ಉΫϥε͕ۙ͘ɺผΫϥε͕ԕ͍͔ʁ [7] Rousseeuw, Silhouettes: A graphical aid
to the interpretation and validation of cluster analysis, Journal of Computational and Applied Mathematics 1987. Silhouette score: γϧΤοτείΞ 18 ͋Δࣄྫɾ͋Δू߹ͷ ฏۉϢʔΫϦουϊϧϜ ∈ [−1,1] -
•ΫϥελϦϯάʹ͓͚Δྑ͞ΛଌΔͨΊͷࢦඪ [7] • ಉΫϥε͕ۙ͘ɺผΫϥε͕ԕ͍͔ʁ [7] Rousseeuw, Silhouettes: A graphical aid
to the interpretation and validation of cluster analysis, Journal of Computational and Applied Mathematics 1987. Silhouette score: γϧΤοτείΞ 19 ͋Δࣄྫɾ͋Δू߹ͷ ฏۉϢʔΫϦουϊϧϜ ͋ΔࣄྫͱಉΫϥεͷࣄྫ ͱͷίετ (Intra-cluster) ∈ [−1,1] -
•ΫϥελϦϯάʹ͓͚Δྑ͞ΛଌΔͨΊͷࢦඪ [7] • ಉΫϥε͕ۙ͘ɺผΫϥε͕ԕ͍͔ʁ [7] Rousseeuw, Silhouettes: A graphical aid
to the interpretation and validation of cluster analysis, Journal of Computational and Applied Mathematics 1987. Silhouette score: γϧΤοτείΞ 20 ͋Δࣄྫɾ͋Δू߹ͷ ฏۉϢʔΫϦουϊϧϜ ͋ΔࣄྫͱಉΫϥεͷࣄྫ ͱͷίετ (Intra-cluster) ͋ΔࣄྫͱผΫϥεͷࣄྫ ͱͷ࠷খίετ (Inter-cluster) ∈ [−1,1] -
•ΫϥελϦϯάʹ͓͚Δྑ͞ΛଌΔͨΊͷࢦඪ [7] • ಉΫϥε͕ۙ͘ɺผΫϥε͕ԕ͍͔ʁ [7] Rousseeuw, Silhouettes: A graphical aid
to the interpretation and validation of cluster analysis, Journal of Computational and Applied Mathematics 1987. Silhouette score: γϧΤοτείΞ 21 ͋Δࣄྫɾ͋Δू߹ͷ ฏۉϢʔΫϦουϊϧϜ ͋ΔࣄྫͱಉΫϥεͷࣄྫ ͱͷίετ (Intra-cluster) ͋ΔࣄྫͱผΫϥεͷࣄྫ ͱͷ࠷খίετ (Inter-cluster) େ͖͍΄Ͳ͍͍ ∈ [−1,1] -
•Silhouette scoreͱIsoScoreΛతؔͱͯ͠ఆࣜԽ •݁Ռ: Silhouette scoreͱIsoScoreͷతؔ૬͍ͯ͠Δ Silhouette scoreͱIsoScoreͷؔ 22 Silhouette IsoScore
ϥϕϧґଘͰڑͷ ࠷େԽ or ࠷খԽ ϥϕϧඇґଘͰڑͷ ࠷খԽ -
•Silhouette scoreͱIsoScoreΛతؔͱͯ͠ఆࣜԽ •݁Ռ: Silhouette scoreͱIsoScoreͷతؔ૬͍ͯ͠Δ Silhouette scoreͱIsoScoreͷؔ 23 Silhouette IsoScore
ϥϕϧґଘͰڑͷ ࠷େԽ or ࠷খԽ ϥϕϧඇґଘͰڑͷ ࠷খԽ -
•Silhouette scoreͱIsoScoreΛతؔͱͯ͠ఆࣜԽ •݁Ռ: Silhouette scoreͱIsoScoreͷతؔ૬͍ͯ͠Δ Silhouette scoreͱIsoScoreͷؔ 24 Silhouette IsoScore
ϥϕϧґଘͰڑͷ ࠷େԽ or ࠷খԽ ϥϕϧඇґଘͰڑͷ ࠷খԽ -
•Silhouette scoreͱIsoScoreΛతؔͱͯ͠ఆࣜԽ •݁Ռ: Silhouette scoreͱIsoScoreͷతؔ૬͍ͯ͠Δ Silhouette scoreͱIsoScoreͷؔ 25 Silhouette IsoScore
ϥϕϧґଘͰڑͷ ࠷େԽ or ࠷খԽ ϥϕϧඇґଘͰڑͷ ࠷খԽ -
•Silhouette scoreͱIsoScoreΛతؔͱͯ͠ఆࣜԽ •݁Ռ: Silhouette scoreͱIsoScoreͷతؔ૬͍ͯ͠Δ Silhouette scoreͱIsoScoreͷؔ 26 Silhouette IsoScore
ϥϕϧґଘͰڑͷ ࠷େԽ or ࠷খԽ ϥϕϧඇґଘͰڑͷ ࠷খԽ -
•ઢܗྨثΛ܇࿅ͨ͠ࡍͷIsoScore, Silhouette scoreͷมԽΛ؍ •λεΫ • SBERTͷຒΊࠐΈ͔Βۃੑྨ • SBERTͷຒΊࠐΈ͔Βࣗવݴޠਪ (2จؒͷҙຯؔ༧ଌ) •
Word2VecͷຒΊࠐΈ͔ΒPOS-tagging • Word2VecͷຒΊࠐΈ͔ΒWordNetͷsupersense༧ଌ ࣮ݧ 30 SBERTΛ͏λεΫʹ͍ͭͯຊจதʹ “we directly optimize the output embeddings of the SBERT model rather than update the parameters of the SBERT model” ͱ͋Δ͕ɺ͜Ε͔ͳΓมͳઃఆͱ͍͏ؾ͕͢Δ…? SBERTͷग़ྗຒΊࠐΈΛnn.Parameterʹͯ͠࠷దԽ͍ͯ͠ΔΑ͏ (SBERTͷग़ྗจຒΊࠐΈͷू߹ΛWord2VecͷΑ͏ʹѻ͍ͬͯΔ?) SBERTͷύϥϝʔλݻఆ -
•ֶशͷաఔͰ • Silhouette score (ΫϥελϦϯάʹ͓͍ͯͷྑ͞) վળ • IsoScore (ํੑʹ͓͍ͯͷྑ͞) ѱԽ
•ֶश͕ਐΉʹͭΕͯຒΊࠐΈ͕ҟํతʹ͢ΔΑ͏มԽ ݁Ռ: ֶशதͷSilhouetteείΞɾIsoScoreͷભҠ 31 Silhouette score IsoScore -
•ຒΊࠐΈදݱͷํੑ্ • ҙຯతྨࣅΛଌఆ͢ΔλεΫ (e.g. STS) ͷੑೳʹͱͬͯ༗ӹ • ΫϥελϦϯάλεΫͷੑೳʹͱͬͯෆརӹ •ຒΊࠐΈͷઢܗྨͷతؔΫϥελϦϯάͷྑ͞ͱ΄΅Ձ ॴײ
•શମతʹͤΜͳͱ͍͏ײ͡ͷจ • ຒΊࠐΈͬͯΔਓͳΜͱͳ͘ײͯͨ͡ҙࣝΛͪΌΜͱ໌ݴͯ͠Έ ·ͨ͠ͱ͍͏งғؾ •୯ҰϞσϧͰཱ྆Ͱ͖ͳ͍͔ʁʹ͍ͭͯະ ·ͱΊ 33 -
•࠷ۙͷςΩετຒΊࠐΈଟ༷ͳλεΫͰͷฏۉੑೳͰଌΒΕΔ͜ͱ͕ଟ͍ • Clustering, ྨ, จॻݕࡧ, ҙຯతྨࣅఆ… •λεΫͷੑ࣭͝ͱʹʮڧݻͳΫϥελΛܗ͢ΔʯຒΊࠐΈ͔ʮํతʹ ͢ΔʯຒΊࠐΈΛར༻͢Δ͔Λߟ͑Δඞཁ͕͋Δ •ۙςΩετຒΊࠐΈʹ͓͍ͯʮࢦࣔʹै͍ςΩετͷຒΊࠐΈํΛม Խͤ͞Δख๏ʯ͕಄͖͍ͯͯ͠Δ
• e.g. InstructOR, E5-Mistral, LLM2Vec, NV-Embed, … •͜ΕΒͷϞσϧࢦࣔʹैͬͯʮҟํతͳຒΊࠐΈʯʮํతͳຒΊࠐΈʯ Λग़͚͠Δ͜ͱ͕Ͱ͖Δ͔ʁ ༨ஊ: ࢦࣔʹै͏ຒΊࠐΈϞσϧ 34 -
•ۙͷ൚༻ςΩετຒΊࠐΈϞσϧ(E5GTE)ରরֶशʹ͓͚ΔԹύϥ ϝʔλΛখ͍ͯ͘͞͠Δ (τ = 0.01) • ςΩετຒΊࠐΈ+ରরֶशͷ͚తଘࡏͰ͋ΔSimCSE0.05 •ରরֶशͷԹύϥϝʔλ্͛Δ΄ͲڧݻͳΫϥελΛ͢ [8] •
େنֶशͷࡍϊΠζ༝དྷͷṖΫϥελ͕Ͱ͖ͳ͍Α͏ԹΛԼ͛Δʁ [8] Wang et al., Understanding the Behaviour of Contrastive Loss, CVPR 2021 ༨ஊ: ରরֶशͷԹύϥϝʔλ 35 -
•ຊจʹΑΕΫϥελϦϯάྨλεΫҟํతͰ͋Δ΄ͲΑ͍ • ௨ৗͷin-batch negativesΛར༻͢Δରরֶशଛࣦ͜Εʹ͢Δ •ςΩετຒΊࠐΈϞσϧͷֶशʹ͓͍ͯɺྨɾΫϥελϦϯάλεΫͷ fi ne-tuningΛ͢Δ߹ʹɺରরֶशΛ෦తʹΊΔ͜ͱ͕ੑೳʹد༩͢Δ ͱ͍͏ใࠂ͕࠷ۙग़͖ͯͨ [9] •తʹରরֶशଛࣦΛ͏࣌ͦΖͦΖऴΘΓʁ
Salesforce, SFR-Embedding-Mistral: Enhance Text Retrieval with Transfer Learning, 2024 ༨ஊ: ΫϥελϦϯάλεΫʹ͓͚Δରরֶशଛࣦͷมભ 36