Share
Embed
WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings
対照学習と白色化処理を組み合わせた文埋め込み手法 WhitenedCSE について解説した資料です。
2023-09-28: ACL2023読み会@名大
http://cr.fvcrc.i.nagoya-u.ac.jp/~sasano/acl2023nagoya/
Hayato Tsukagoshi
September 26, 2023
Transcript
32 slides
-
WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings D1, Graduate School
of Informatics, Nagoya University, Japan Hayato Tsukagoshi Wenjie Zhuo, Yifan Sun, Xiaohan Wang, Linchao Zhu, Yi Yang ACL 2023 https://aclanthology.org/2023.acl-long.677/ -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 3 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ
͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 4 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ
͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] ҙຯతʹྨࣅ ͍ۙҙຯΛ࣋ͭจ ۙ͘ʹ ϕΫτϧؒͷڑ͕ ҙຯతͳؔΛදݱ -
ಋೖ: Contrastive Learning / ରরֶश •ਖ਼ྫͱෛྫͷಛදݱΛϞσϧ͔Βग़ྗ •ਖ਼ྫಉ࢜ͷྨࣅ͕ߴ͘ͳΔΑ͏ʹֶशΛߦ͏ •Computer VisionͰେਓؾɺNLPͰྲྀߦத SimCLR
•ಉ͡ը૾ʹରͯ͠ҟͳΔ data augmentationΛͨ͠ ը૾ಉ࢜Λਖ਼ྫʹ͢Δ •ޙஈͷը૾ྨλεΫͳͲ Ͱߴ͍ੑೳ •CVʹ͓͚ΔදݱֶशͷͨΊ ͷpre-trainingͱͯ͠༗ޮ 5 ը૾ϒϩά[16]ΑΓҾ༻ Oord+: Representation Learning with Contrastive Predictive Coding, arXiv ‘18 Chen+: A Simple Framework for Contrastive Learning of Visual Representations, ICML ’20 Advancing Self-Supervised and Semi-Supervised Learning with SimCLR, ’20 Chen+: Big Self-Supervised Models are Strong Semi-Supervised Learners, NeurIPS ’20 -
•ྨࣅߦྻΛܭࢉɺର֯(ਖ਼ྫಉ࢜ͷྨࣅ)Λਖ਼ղͱ͢Δ • ର֯ͷྨࣅ࠷େԽ == ਖ਼ྫಉ࢜ͷྨࣅ࠷େԽ in-batch negatives •mini-batchͷ͋Δࣄྫʹ͍ͭͯɺଞͷࣄྫΛෛྫͱͯ͠ߟ͑Δ ଛࣦؔ (InfoNCE)
Contrastive Learning / ֶशखॱ 6 ℒi = − log esim(hi ,h+ i )/τ ∑N j=1 esim(hi ,h+ j )/τ -
•ಛදݱͷྑ͞ΛଌΔ (ඍՄೳͳ) ࢦඪ (ଛࣦؔ) • ͜ΕΒ͕ྑ͍(͍)΄Ͳྑ͍දݱ(ͱ͞Ε͍ͯΔ) Alignment •ࣅͨαϯϓϧ͕ಛ্ۭؒͰۙ͘ʹͯ͘͠ΕΔ͔ Uniformity •ಛදݱ͕୯Ґٿ໘্ʹҰ༷͢Δ͔
Wang+: Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML ’20 Alignment / Uniformity 7 -
•จຒΊࠐΈ+ରরֶशͷύΠΦχΞతݚڀ •Unsupervised SimCSE:ʮಉ͡จΛ2ճຒΊࠐΜͰରরֶशʯ •Supervised SimCSE: ʮؚҙؔʹ͋ΔจΛਖ਼ྫͱͯ͠ରরֶशʯ Gao+: SimCSE: Simple Contrastive
Learning of Sentence Embeddings, EMNLP 2021 ઌߦݚڀ: SimCSE 8 -
•σʔλಉ͕࢜ٿ໘্ʹҰ༷͢ΔΑ͏ม • ฏۉ: 0 • ࢄڞࢄߦྻ: ୯Ґߦྻ ന৭Խʹ͏ख๏ •Principal Component
Analysis (PCA): σʔλߦྻΛݻ༗ղ •Zero-phase Component Analysis (ZCA): PCA + ճసଧͪফ͠ ҟํੑ (anisotropy) •σʔλ͕ߴ࣍ݩ্ۭؒͷ࣍ݩۭؒͷΈʹͯ͠͠·͏ੑ࣭ • ന৭Խҟํੑͷղফʹ༗༻ (ํతʹͨ͠ํ͕ੑೳ͕ྑ͍͜ͱ͕ଟ͍) ന৭Խ 10 H = WZ HHT = I WZ(WZ)T = WZZTWT ZZT = UΛUT WPCA = Λ−1/2UT WZCA = UΛ−1/2UT -
•SimCSEͰalignment / uniformity্͍ͯ͠Δ͕… • ରরֶशͩͱෛྫಉ࢜Λͤͳ͍ Shu ff l ed Group
Whitening (SGW) •γϟοϑϧͯ͠άϧʔϓ͝ͱʹന৭Խ •ݩͷॱ൪ʹͯ͠ग़ྗຒΊࠐΈͱ͢Δ Multi-Positive Contrastive Loss •SGWΛෳճ܁Γฦͯ͠ਖ਼ྫΛਫ૿͠ •ଟ༷ͳਖ਼ྫͰֶशͰ͖ؤ݈ੑ্ WhitenedCSE 11 -
Shu ffl ed Group Whitening: ٙࣅίʔυ 15 https://github.com/SupstarZh/WhitenedCSE/blob/master/whitenedcse/shu ffl ed_group_whitening.py
(G, d, B) x (G, B, d) → (G, d, d) -
λεΫ: จຒΊࠐΈͷඪ४తͳϕϯνϚʔΫ •Semantic Textual Similarity (STS) •SentEval ՃͷධՁࢦඪ •Uniformity, Alignment
܇࿅ઃఆ •ӳޠWikipedia͔ΒϥϯμϜʹαϯϓϦϯάͨ͠100ສจ (ϥϕϧͳ͠) •BERT, RoBERTaΛ fi ne-tuning •ന৭Խόον͝ͱʹɺ384άϧʔϓ(1άϧʔϓ͋ͨΓ2࣍ݩ) (BERT) ࣮ݧ 18 -
•จຒΊࠐΈϞσϧͷҙຯΛଊ͑Δ ೳྗΛਓؒධՁͱͷ૬ؔͰධՁ •จϖΞʹҙຯతྨࣅ͕ਓखͰ༩ •ਓखධՁͱϞσϧ͕ܭࢉͨ͠ྨࣅ ͷ૬ؔͰධՁ • Pearsonͷ(ੵ)૬ؔ • SpearmanͷॱҐ૬ؔ •จຒΊࠐΈධՁͰڭࢣͳ͠ઃఆ
• STSσʔλΛ༻ֶ͍ͨश͠ͳ͍ • ࣄલʹ܇࿅͞ΕͨϞσϧΛධՁ Semantic Textual Similarity (STS) 19 -
ڭࢣͳ͠STSͷධՁखॱ ᶃ ύϥϝʔλΛݻఆͨ͠ จຒΊࠐΈϞσϧΛ༻ҙ ᶄ จϖΞͦΕͧΕΛจຒΊࠐΈʹม ᶅ จ“ϕΫτϧ”ϖΞͷྨࣅΛܭࢉ • ίαΠϯྨࣅ͕Α͘༻͍ΒΕΔ
ᶆ ਓؒධՁͱͷ(ॱҐ)૬ؔΛܭࢉ •૬͕ؔߴ͍ํ͕“ྑ͍จຒΊࠐΈ” ڭࢣͳ͠ (Unsupervised) STSλεΫ 20 จA จB ᶄ ᶅ ᶆ ᶃ จຒΊࠐΈϞσϧ ਓखධՁͱͷ ૬ؔͰධՁ จྨࣅ -
•ςΩετྨͳͲͷԼྲྀλεΫ͕ू·ͬͨtoolkit •จຒΊࠐΈΛೖྗͱ͢ΔྨثΛ܇࿅ɺྨੑೳ͔ΒจຒΊࠐΈͷ࣭ΛධՁ Conneau+: SentEval: An Evaluation Toolkit for Universal Sentence
Representations, LREC ‘18 SentEval 21 Task ྨର Ϋϥε ྫจ MR өըϨϏϡʔͷpos/neg 2 Too slow for a younger crowd, too shallow for an older one. CR ϨϏϡʔͷpos/neg 2 We tried it out christmas night and it worked great. SUBJ өը/͋Β͢͡ͷओ؍ੑ 2 A movie that doesn’t aim too high, but doesn’t need to. MPQA ϑϨʔζͷۃੑ 2 would like to tell SST-2 өըϨϏϡʔͷpos/neg 2 Audrey Tautou has a knack for picking roles that magnify her [..] TREC ࣭ͷछผ 6 What are the twin cities? MRPC 2จ͕ݴ͍͔͑Ͳ͏͔ 2 The procedure is generally performed in the second or third trimester. & The technique is used during the second and, occasionally, third trimester of pregnancy. -
SentEvalͷධՁखॱ ᶃ ύϥϝʔλΛݻఆͨ͠ จຒΊࠐΈϞσϧΛ༻ҙ ᶄ ֤จΛจຒΊࠐΈʹม ᶅ จຒΊࠐΈΛೖྗͱ͢ΔྨثΛ܇࿅ ᶆ ྨثͷੑೳ͔ΒจຒΊࠐΈ
ͷ࣭ΛධՁ •ྨੑೳ͕ߴ͍ํ͕“ྑ͍จຒΊࠐΈ” •ྨثϩδεςΟοΫճؼྨث͕ଟ͍ SentEval 22 จ ᶄ ᶃ ྨੑೳ͔Β จຒΊࠐΈͷ࣭ΛධՁ ᶅ จຒΊࠐΈϞσϧ ྨث ᶆ -
BERT- fl ow: ҟํతͳBERTͷจຒΊࠐΈۭ͔ؒΒํతͳજࡏۭؒͷࣸ૾Λֶश BERT-whitening: จຒΊࠐΈͷฏۉ͕0ɼڞࢄߦྻ͕୯ҐߦྻʹͳΔΑ͏ʹઢܗม (+࣍ݩݮ) IS-BERT: จຒΊࠐΈͱจதͷn-gramͷຒΊࠐΈͷ૬ޓใྔΛ࠷େԽ͢ΔΑ͏ʹֶश BERT-CT:
ҟͳΔೋͭͷಉ͡Ϟσϧͷಉ͡จʹର͢ΔຒΊࠐΈಉ࢜ͷੵ͕େ͖͘ͳΔΑ͏ʹֶश SimCSE: ҟͳΔDropoutΛద༻ͨ͠ಉ͡จΛਖ਼ྫ or ؚҙؔͷจϖΞΛਖ਼ྫͱͨ͠ରরֶश MixCSE: ҟͳΔจΛࠞͥͨจΛhard negativeͱͯ͠ڭࢣͳ͠ରরֶश ArcCSE: ؚҙϖΞจຒΊࠐΈͷmargin͖֯࠷খԽ+DAͨ͠จΛෛྫʹ͢ΔTriplet Lossͷ༥߹ DCLR: ΨγΞϯϊΠζΛෛྫͱͯ͠Ճ + ࣄྫ͝ͱॏΈ͚ͯ͠Unsup-SimCSE MoCoSE: ϞʔϝϯλϜΤϯίʔμͷ࠷దͳෛྫੳ+FGSMʹΑΔσʔλ֦ுͰରরֶश Li+: On the Sentence Embeddings from Pre-trained Language Models, EMNLP '20 Su+: Whitening Sentence Representations for Better Semantics and Faster Retrieval, arXiv ’21 Zhang+: An Unsupervised Sentence Embedding Method by Mutual Information Maximization, EMNLP ’20 Carlsson+: Semantic Re-tuning with Contrastive Tension, ICLR ’21 Gao+: SimCSE: Simple Contrastive Learning of Sentence Embeddings, EMNLP ’21 Zhang+: Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives, AAAI ’22 Zhang+: A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space, ACL ’22 Zhou+: Debiased Contrastive Learning of Unsupervised Sentence Representations, ACL 2022 Cao+: Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding, ACL fi ndings ’22 ൺֱख๏ 23 -
•ֶशॳظ͔ΒUniformityͷ͕͍͍ (lower is better) • ന৭ԽʹΑֶͬͯशॳظ͔ΒຒΊࠐΈ͕Ұ༷͢ΔͨΊ Alignment / Uniformity 27
Alignment Uniformity SimCSE WhitenedCSE -
•ରরֶशͱന৭ԽॲཧΛΈ߹Θͤͨ จຒΊࠐΈֶशख๏ΛఏҊ • ຒΊࠐΈΛάϧʔϓʹׂͯ͠ന৭Խ (SGW) • ෳͷਖ਼ྫΛ༻͍ͨରরֶश ײ •ͿͬͪΌ͚ධՁ͕গ͠ո͍͠ •άϧʔϓԽͤͣʹന৭Խͨ͠߹ͷੑೳ͕݁ߏ͍
• ಛදݱ͕drasticʹมԽ͗͢͠ΔͨΊʁ • άϧʔϓ͕খ͍͞ͷͰϚΠϧυͳന৭ԽΛ͍ͯ͠Δ •ϚΠϧυന৭ԽͰྑ͍ͳΒന৭Խૢ࡞ࣗମෆཁ͔ʁ • ࢄڞࢄߦྻ͕IdenticalʹͳΔΑ͏ͳଛࣦʁ ·ͱΊ 32 ന৭Խͷํ͕ ؾʹͳΓ͗͢Δ