Share
Embed
[輪講資料] Language-agnostic BERT Sentence Embedding
多言語文埋め込み手法であるLanguage-agnostic BERT Sentence Embedding (LaBSE)の論文について解説した資料です。
Hayato Tsukagoshi
May 24, 2022
Transcript
85 slides
-
Language-agnostic BERT Sentence Embedding M2, Graduate School of Informatics, Nagoya
University, Japan ൃදऀ: ௩ӽॣ / Hayato TSUKAGOSHI Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, Wei Wang ACL 2022 URL: https://arxiv.org/abs/2007.01852 -
•109Ҏ্ͷݴޠʹద༻ՄೳͳଟݴޠจຒΊࠐΈϞσϧLaBSEΛఏҊ •ଟݴޠࣄલֶशˠ༁ίʔύεΛ༻͍ͨจຒΊࠐΈͷରরֶश • MLM + Translation Language Modeling → Additive
Margin Softmax •छʑͷධՁ࣮ݧ • ݴޠԣஅݕࡧͷੑೳΛେ෯ʹվળ • ಛʹগࢿݯݴޠͰߴ͍ੑೳ • ୯ݴޠSTS/SentEvalߴ͘ͳ͍ จબఆཧ༝ •จຒΊࠐΈؔ࿈ͷ/ධՁ͕๛ • ڭҭతͳ(ଟݴޠ)จຒΊࠐΈͷจ LaBSE: Language-agnostic BERT Sentence Embedding 2 https://arxiv.org/abs/2007.01852 -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 4 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ
͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 5 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ
͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ ಋೖ: จຒΊࠐΈ / Sentence embedding 6 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ
͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] ҙຯతʹྨࣅ ͍ۙҙຯΛ࣋ͭจ ۙ͘ʹ ϕΫτϧؒͷڑ͕ ҙຯతͳؔΛදݱ -
•ຒΊࠐΈ (embedding) ͱ͍͏໊લͷ༝དྷ • ୯ޠྻޠኮͷߴ࣍ݩϕΫτϧྻ • ΑΓ࣍ݩͷϕΫτϧͰจΛද͢ • ଟ༷ମؔ࿈ͷ༻ޠΒ͍͠ ಋೖ:
จຒΊࠐΈ / Sentence embedding 7 -
•ຒΊࠐΈ (embedding) ͱ͍͏໊લͷ༝དྷ • ୯ޠྻޠኮͷߴ࣍ݩϕΫτϧྻ • ΑΓ࣍ݩͷϕΫτϧͰจΛද͢ • ଟ༷ମؔ࿈ͷ༻ޠΒ͍͠ ༗༻ੑɾԠ༻ઌ
•ྨࣅจ(ॻ)ݕࡧɾΫϥελϦϯά •ܰྔͳจ(ॻ)ྨɾಛநग़ (ଞλεΫͰԉ༻) •ີϕΫτϧݕࡧ (Dense Passage Retrieval)ʹΑΔ࣭Ԡ •༁ϝϞϦΛ༻͍ͨࣄྫϕʔε༁ɺࣄྫϕʔεػցֶश •Ԡ༻ඇৗʹൣ ಋೖ: จຒΊࠐΈ / Sentence embedding 8 -
BERTҎલ •୯ޠຒΊࠐΈͷ(ॏΈ͚)ฏۉͳͲͰจຒΊࠐΈΛ֫ಘ • p-mean [01], SWEM [02], DynaMax [03], SIF
[04], uSIF [05], etc. •จຒΊࠐΈઐ༻ͷϞσϧΛߏங • Skip-Thought [06], SCDV [07], InferSent [08], USE [09], etc. [01] Ru ̈ckle ́+: Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations, arXiv ’18 [02] Shen+: Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms, ACL ’18 [03] Zhelezniak+: Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors, ICLR ’19 [04] Arora+: A Simple but Tough-to-Beat Baseline for Sentence Embeddings, ICLR '17 [05] Ethayarajh: Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline, Rep4NLP ’18 [06] Kiros+: Skip-Thought Vectors, NIPS ’15 [07] Mekala+: SCDV : Sparse Composite Document Vectors using soft clustering over distributional representations, ACL ’17 [08] Conneau+: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data, EMNLP '17 [09] Cer+: Universal Sentence Encoder, arXiv, Mar 2018 ಋೖ: จຒΊࠐΈͷදతͳख๏ 9 -
Sentence-BERT [10] •ࣗવݴޠਪ (Natural Language Inference; NLI) ͰBERTΛ fi ne-tuning
• ࣄલֶशࡁΈݴޠϞσϧ (Pre-trained Language Model; PLM) Λ༻͍ͨ จຒΊࠐΈϞσϧͷ͚తݚڀ • BERTͰInferSentΛΔ •࣌େ෯ʹSOTA (state-of-the-art) Λߋ৽ • ޙड़͢ΔSimCSE͕΄΅্Ґޓʹͳͬͯ ͠·ͬͨͷͰɺࠓޙ͋·ΓΘΕͳͦ͞͏? [10] Reimers+: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP '19 ಋೖ: จຒΊࠐΈͷදతͳख๏ 10 ਤ֘จΑΓҾ༻ -
SimCSE [11] •ରরֶश(Contrastive Learning)Λ༻͍ͯBERTΛ fi ne-tuning • Unsupervised SimCSE:ʮಉ͡จΛ2ճຒΊࠐΜͰରরֶशʯ •
Supervised SimCSE: ʮؚҙؔʹ͋ΔจΛਖ਼ྫͱͯ͠ରরֶशʯ •Ϳͬͪ͗ΓͷSOTAɺੜݚڀଓʑ [11] Gao+: SimCSE: Simple Contrastive Learning of Sentence Embeddings, EMNLP ’21 ಋೖ: จຒΊࠐΈͷදతͳख๏ 11 ਤ֘จΑΓҾ༻ɻҎલ࣮ࢪͨ͠SimCSEͷྠߨࢿྉͪ͜Β -
•จຒΊࠐΈͷ“ྑ͞”ͲͷΑ͏ʹධՁ͢ΕΑ͍͔ʁ • ࣮͕ٞਚ͘͞Ε͍ͯͳ͍(ͱࢥ͏) • ͲͷΑ͏ͳจຒΊࠐΈ͕࡞ΒΕΔ͖ͩΖ͏͔ʁͱ͍͏͍ະղܾ •ͱ͍͑ԿΒ͔ͷج४ͰධՁ͠ͳ͍ͱ͍͚ͳ͍ ධՁࢦඪ •Semantic Textual Similarity
(STS): ਓؒɾϞσϧ͕ଌͬͨจྨࣅͷ૬ؔ •SentEval: ςΩετྨͳͲԼྲྀλεΫͰͷੑೳ [12, 13] •SentGLUE: GLUE [14]ΛจຒΊࠐΈΛ༻͍ͨղ͖ํʹ੍ݶ [15] •ΫϥελϦϯάɺςΩετݕࡧ [12] Conneau+: SentEval: An Evaluation Toolkit for Universal Sentence Representations, LREC ’18 [13] Conneau+: What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties, ACL ’18 [14] Wang+: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, EMNLP Workshop BlackboxNLP ’18 [15] Ni+: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models, CoRR ’21 ಋೖ: จຒΊࠐΈͷධՁ 12 -
•จຒΊࠐΈͷ“ྑ͞”ͲͷΑ͏ʹධՁ͢ΕΑ͍͔ʁ • ࣮͕ٞਚ͘͞Ε͍ͯͳ͍(ͱࢥ͏) • ͲͷΑ͏ͳจຒΊࠐΈ͕࡞ΒΕΔ͖ͩΖ͏͔ʁͱ͍͏͍ະղܾ •ͱ͍͑ԿΒ͔ͷج४ͰධՁ͠ͳ͍ͱ͍͚ͳ͍ ධՁࢦඪ •Semantic Textual Similarity
(STS): ਓؒɾϞσϧ͕ଌͬͨจྨࣅͷ૬ؔ •SentEval: ςΩετྨͳͲԼྲྀλεΫͰͷੑೳ [12, 13] •SentGLUE: GLUE [14]ΛจຒΊࠐΈΛ༻͍ͨղ͖ํʹ੍ݶ [15] •ΫϥελϦϯάɺςΩετݕࡧ [12] Conneau+: SentEval: An Evaluation Toolkit for Universal Sentence Representations, LREC ’18 [13] Conneau+: What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties, ACL ’18 [14] Wang+: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, EMNLP Workshop BlackboxNLP ’18 [15] Ni+: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models, CoRR ’21 ಋೖ: จຒΊࠐΈͷධՁ 13 -
•จຒΊࠐΈͷ“ྑ͞”ͲͷΑ͏ʹධՁ͢ΕΑ͍͔ʁ • ࣮͕ٞਚ͘͞Ε͍ͯͳ͍(ͱࢥ͏) • ͲͷΑ͏ͳจຒΊࠐΈ͕࡞ΒΕΔ͖ͩΖ͏͔ʁͱ͍͏͍ະղܾ •ͱ͍͑ԿΒ͔ͷج४ͰධՁ͠ͳ͍ͱ͍͚ͳ͍ ධՁࢦඪ •Semantic Textual Similarity
(STS): ਓؒɾϞσϧ͕ଌͬͨจྨࣅͷ૬ؔ •SentEval: ςΩετྨͳͲԼྲྀλεΫͰͷੑೳ [12, 13] •SentGLUE: GLUE [14]ΛจຒΊࠐΈΛ༻͍ͨղ͖ํʹ੍ݶ [15] •ΫϥελϦϯάɺςΩετݕࡧ [12] Conneau+: SentEval: An Evaluation Toolkit for Universal Sentence Representations, LREC ’18 [13] Conneau+: What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties, ACL ’18 [14] Wang+: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, EMNLP Workshop BlackboxNLP ’18 [15] Ni+: Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models, CoRR ’21 ಋೖ: จຒΊࠐΈͷධՁ 14 STSͱSentEval͕࠷Α͘ΘΕΔ -
•จຒΊࠐΈϞσϧͷҙຯΛଊ͑Δ ೳྗΛਓؒධՁͱͷ૬ؔͰධՁ •จϖΞʹҙຯతྨࣅ͕ਓखͰ༩ •ਓखධՁͱϞσϧ͕ܭࢉͨ͠ྨࣅ ͷ૬ؔͰධՁ • Pearsonͷ(ੵ)૬ؔ • SpearmanͷॱҐ૬ؔ •จຒΊࠐΈධՁͰڭࢣͳ͠ઃఆ
• STSσʔλΛ༻ֶ͍ͨश͠ͳ͍ • ࣄલʹ܇࿅͞ΕͨϞσϧΛධՁ ಋೖ: Semantic Textual Similarity (STS) 17 -
ڭࢣͳ͠STSͷධՁखॱ ᶃ จϖΞσʔληοτΛ༻ҙ ᶄ จຒΊࠐΈϞσϧΛ༻ҙ ᶅ จϖΞͦΕͧΕΛจϕΫτϧʹ ಋೖ: ڭࢣͳ͠ (Unsupervised)
STS 18 จຒΊࠐΈϞσϧ จA จB ᶄ ᶅ ᶃ -
ڭࢣͳ͠STSͷධՁखॱ ᶃ จϖΞσʔληοτΛ༻ҙ ᶄ จຒΊࠐΈϞσϧΛ༻ҙ ᶅ จϖΞͦΕͧΕΛจϕΫτϧʹ ᶆ จ“ϕΫτϧ”ϖΞͷྨࣅΛܭࢉ •
ίαΠϯྨࣅ͕Α͘༻͍ΒΕΔ ᶇ ਓؒධՁͱͷ(ॱҐ)૬ؔΛܭࢉ ಋೖ: ڭࢣͳ͠ (Unsupervised) STS 19 จຒΊࠐΈϞσϧ จA จB ᶄ ᶅ ਓखධՁ ʹΑΔྨࣅ ᶆ Ϟσϧ ʹΑΔྨࣅ ᶇ ૬ؔ ᶃ -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 20 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 21 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 22 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 1 1 4 3 2 2 3 4 5 5 1 2 3 4 5 -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 23 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 1 1 4 3 2 2 3 4 5 5 1 2 3 4 5 จϖΞͷྨࣅॱҐ -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 24 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 1 1 4 3 2 2 3 4 5 5 1 2 3 4 5 r1 = 1 − 6 5(52 − 1) {(1−1)2 + (2−4)2 + (3−2)2 + (4−5)2 + (5−5)2} = 1 − 6 120 (0 + 4 + 1 + 1 + 0) -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 25 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 1 1 4 3 2 2 3 4 5 5 1 2 3 4 5 r1 = 1 − 6 5(52 − 1) {(1−1)2 + (2−4)2 + (3−2)2 + (4−5)2 + (5−5)2} = 1 − 6 120 (0 + 4 + 1 + 1 + 0) ਖ਼ղͷॱҐͱ༧ଌʹΑΔ ॱҐͷ૬ؔΛܭࢉ(ެࣜʹಥͬࠐΉ) -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 26 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 1 1 4 3 2 2 3 4 5 5 1 2 3 4 5 r1 = 1 − 6 5(52 − 1) {(1−1)2 + (2−4)2 + (3−2)2 + (4−5)2 + (5−5)2} = 1 − 6 120 (0 + 4 + 1 + 1 + 0) r1 = 0.7 r2 = 0.9 ਖ਼ղͷॱҐͱ༧ଌʹΑΔ ॱҐͷ૬ؔΛܭࢉ(ެࣜʹಥͬࠐΉ) -
ಋೖ: STSʹ͓͚ΔSpearmanͷॱҐ૬ؔͷܭࢉ 27 จA จB ਓؒධՁ Model 1 Model 2
A man is playing a guitar. The man is playing the guitar. 4.909 0.985 0.978 A man is playing a guitar. A guy is playing an instrument. 3.800 0.646 0.895 A man is playing a guitar. A man is playing a guitar and singing. 3.200 0.874 0.977 A man is playing a guitar. The girl is playing the guitar. 2.250 0.747 0.831 A man is playing a guitar. A woman is cutting vegetable. 0.000 0.290 0.595 1 1 4 3 2 2 3 4 5 5 1 2 3 4 5 r1 = 1 − 6 5(52 − 1) {(1−1)2 + (2−4)2 + (3−2)2 + (4−5)2 + (5−5)2} = 1 − 6 120 (0 + 4 + 1 + 1 + 0) r1 = 0.7 r2 = 0.9 Model 2ͷ΄͏͕༏Ε͍ͯΔ -
ӳޠσʔληοτ •STS12, 13, 14, 15, 16 [16, 17, 18, 19,
20] •STS Benchmark (test set) [21] •SICK-R [22] ຊޠσʔληοτ •JSICK [23] •JSTS [24] [16] Agirre+: SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity, *SEM ’12 [17] Agirre+: *SEM 2013 shared task: Semantic Textual Similarity, *SEM ‘13 [18] Agirre+: SemEval-2014 Task 10: Multilingual Semantic Textual Similarity, SemEval ‘14 [19] Agirre+: SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability, SemEval ’15 [20] Agirre+: SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation, SemEval ’16 [21] Cer+: SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation, SemEval ’17 [22] Marelli+: A SICK cure for the evaluation of compositional distributional semantic models, LREC ’14 [23] ୩த+: JSICK: ຊޠߏతਪɾྨࣅσʔληοτͷߏங, ਓೳֶձ ୈ35ճશࠃେձ (2021) [24] ܀ݪ+: JGLUE: ຊޠݴޠཧղϕϯνϚʔΫ, ݴޠॲཧֶձ ୈ28ճ࣍େձ (2022) ಋೖ: STSͷධՁ༻σʔληοτ 28 -
ӳޠσʔληοτ •STS12, 13, 14, 15, 16 [16, 17, 18, 19,
20] •STS Benchmark (test set) [21] •SICK-R [22] ຊޠσʔληοτ •JSICK [23] •JSTS [24] [16] Agirre+: SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity, *SEM ’12 [17] Agirre+: *SEM 2013 shared task: Semantic Textual Similarity, *SEM ‘13 [18] Agirre+: SemEval-2014 Task 10: Multilingual Semantic Textual Similarity, SemEval ‘14 [19] Agirre+: SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability, SemEval ’15 [20] Agirre+: SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation, SemEval ’16 [21] Cer+: SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation, SemEval ’17 [22] Marelli+: A SICK cure for the evaluation of compositional distributional semantic models, LREC ’14 [23] ୩த+: JSICK: ຊޠߏతਪɾྨࣅσʔληοτͷߏங, ਓೳֶձ ୈ35ճશࠃେձ (2021) [24] ܀ݪ+: JGLUE: ຊޠݴޠཧղϕϯνϚʔΫ, ݴޠॲཧֶձ ୈ28ճ࣍େձ (2022) ಋೖ: STSͷධՁ༻σʔληοτ 29 STS12-16ͦΕͧΕখ͍͞σʔληοτͷू߹ ௨ৗɺ“αϒ”σʔληοτΛࠞͥͯ૬ؔΛܭࢉ STS12-16, STS Benchmark, SICK-RͷείΞͷ ฏۉͰ࠷ऴతͳධՁ͕͞ΕΔ͜ͱ͕ଟ͍ -
•ςΩετྨͳͲͷԼྲྀ (Downstream) λεΫ͕ू·ͬͨtoolkit •จຒΊࠐΈΛೖྗͱ͢ΔྨثΛ܇࿅ɺྨੑೳ͔ΒจຒΊࠐΈͷ࣭ΛධՁ [25] Conneau+: SentEval: An Evaluation Toolkit
for Universal Sentence Representations, LREC ‘18 ಋೖ: SentEval [25] 30 Task Type #train #test #class MR movie review 11,000 11,000 2 CR product review 4,000 4,000 2 SUBJ subjectivity status 10,000 10,000 2 MPQA opinion-polarity 11,000 11,000 2 SST-2 binary sentiment analysis 67,000 1,800 2 TREC question-type classi fi cation 6,000 500 6 MRPC paraphrase detection 4,100 1,700 2 λεΫҰཡ -
•ςΩετྨͳͲͷԼྲྀ (Downstream) λεΫ͕ू·ͬͨtoolkit •จຒΊࠐΈΛೖྗͱ͢ΔྨثΛ܇࿅ɺྨੑೳ͔ΒจຒΊࠐΈͷ࣭ΛධՁ [25] Conneau+: SentEval: An Evaluation Toolkit
for Universal Sentence Representations, LREC ‘18 ಋೖ: SentEval [25] 31 Task Type #train #test #class MR movie review 11,000 11,000 2 CR product review 4,000 4,000 2 SUBJ subjectivity status 10,000 10,000 2 MPQA opinion-polarity 11,000 11,000 2 SST-2 binary sentiment analysis 67,000 1,800 2 TREC question-type classi fi cation 6,000 500 6 MRPC paraphrase detection 4,100 1,700 2 λεΫҰཡ -
SentEvalͷධՁखॱ ᶃ ύϥϝʔλΛݻఆͨ͠ จຒΊࠐΈϞσϧΛ༻ҙ ᶄ จຒΊࠐΈΛೖྗͱ͢ΔྨثΛ܇࿅ ᶅ ྨثͷੑೳ͔ΒจຒΊࠐΈ ͷ࣭ΛධՁ •ྨੑೳ͕ߴ͍ํ͕“ྑ͍จຒΊࠐΈ”ͱ͍͏Ծఆ
•ྨثϩδεςΟοΫճؼྨث͕ଟ͍ • i.e. จຒΊࠐΈͷ֤࣍ݩͷॏΈ͖Ͱྨ •“ࣄલ܇࿅ࡁΈͷ”จຒΊࠐΈϞσϧͷੑೳΛධՁ ಋೖ: SentEval 32 จຒΊࠐΈϞσϧ จ ᶄ ᶃ ྨث ྨੑೳ͔Β จຒΊࠐΈͷ࣭ΛධՁ ᶅ -
•109Ҏ্ͷݴޠʹద༻ՄೳͳଟݴޠจຒΊࠐΈϞσϧLaBSEΛఏҊ •ଟݴޠࣄલֶशˠ༁ίʔύεΛ༻͍ͨจຒΊࠐΈͷରরֶश • MLM + Translation Language Modeling → Additive
Margin Softmax •छʑͷධՁ࣮ݧ • ݴޠԣஅݕࡧͷੑೳΛେ෯ʹվળ • ಛʹগࢿݯݴޠͰߴ͍ੑೳ • ୯ݴޠSTS/SentEvalߴ͘ͳ͍ จબఆཧ༝ •จຒΊࠐΈؔ࿈ͷ/ධՁ͕๛ • ڭҭతͳ(ଟݴޠ)จຒΊࠐΈͷจ LaBSE: Language-agnostic BERT Sentence Embedding 34 https://arxiv.org/abs/2007.01852 -
•طଘݚڀͱͷൺֱ •LaBSEͷߏཁૉ • Dual-encoderΞʔΩςΫνϟ • Translation ranking task • MLM
and TLM Pre-training •ଟݴޠจຒΊࠐΈͷؔ࿈ݚڀ •࣮ݧઃఆɾֶशͷɾධՁख๏ •࣮ݧ݁Ռ •ੳ •Ճతͳ࣮ݧ ࣍ 35 -
•ଟݴޠจຒΊࠐΈϞσϧΛ࡞ΔͨΊʹࣄલֶशˠ fi ne-tuning •গࢿݯݴޠͰߴ͍ੑೳ • ֶशσʔλʹແ͍ݴޠϖΞͰͷੑೳߴ͍ •ݸʑͷख๏ɾςΫχοΫ΄ͱΜͲطଘݚڀͷͷ • ྑ͍ײ͡ͷΈ߹Θͤ +
େنσʔλ + େنֶशͷͨΊͷ ͕͜ͷݚڀͷߩݙ LaBSE: طଘݚڀͱͷൺֱ 36 -
Dual-encoderΞʔΩςΫνϟ •܇࿅ํ๏ (Training strategy)ͷҰͭɺจຒΊࠐΈख๏ͰҰൠత •Sentence-BERT, SimCSEͳͲ͜ͷํࣜ Translation ranking task •༁จϖΞͷྨࣅΛɺͦͷଞͷจϖΞͷྨࣅΑΓߴ͘͢ΔΑ͏ʹֶश
•Additive margin softmaxΛ༻ֶ͍ͯशΛ MLM and TLM Pre-training •Masked Language Modeling (MLM) •Translation Language Modeling (TLM) LaBSEͷߏཁૉ 37 -
•2ͭͷEncoderͰจຒΊࠐΈදݱΛߏ • ଟ͘ͷ߹EncoderॏΈΛڞ༗ (ʹಉ͡Ϟσϧ) • Siamese network (γϟϜωοτϫʔΫ)ͱݺΕΔ LaBSEͷߏཁૉ: Dual-encoderΞʔΩςΫνϟ
38 Encoder Decoder Encoder-Decoder Encoder Encoder Dual-Encoder ଛࣦܭࢉ -
•2ͭͷEncoderͰจຒΊࠐΈදݱΛߏ • ଟ͘ͷ߹EncoderॏΈΛڞ༗ (ʹಉ͡Ϟσϧ) • Siamese network (γϟϜωοτϫʔΫ)ͱݺΕΔ LaBSEͷߏཁૉ: Dual-encoderΞʔΩςΫνϟ
39 Encoder Decoder Encoder-Decoder Encoder Encoder Dual-Encoder ଛࣦܭࢉ EncDecͷλεΫ • લޙจੜ • ༁จੜ • Denoising AE ॏΈڞ༗ Dual-EncoderͷλεΫ • ؚҙؔೝࣝ • ରরֶश -
•GuoΒ[26] ͕ఏҊ •༁จϖΞͷྨࣅΛɺͦͷଞͷจϖΞͷྨࣅΑΓߴ͘͢ΔΑ͏ʹֶश • ༁จϖΞΛਖ਼ྫͱͨ͠ରরֶश (Contrastive Learning) •“ranking”ͱݴ͍ͭͭͬͯΔͷਖ਼ྫ(ਖ਼͍͠༁จϖΞ)ͷྨࣅ࠷େԽ [26] Guo+:
E ff ective Parallel Corpus Mining using Bilingual Sentence Embeddings, WMT ‘18 LaBSEͷߏཁૉ: Translation ranking task 40 զഐೣͰ͋Δɻ Ja I am a cat. En Nice to meet you. En ਖ਼ྫ ෛྫ -
•GuoΒ[26] ͕ఏҊ •༁จϖΞͷྨࣅΛɺͦͷଞͷจϖΞͷྨࣅΑΓߴ͘͢ΔΑ͏ʹֶश • ༁จϖΞΛਖ਼ྫͱͨ͠ରরֶश (Contrastive Learning) •“ranking”ͱݴ͍ͭͭͬͯΔͷਖ਼ྫ(ਖ਼͍͠༁จϖΞ)ͷྨࣅ࠷େԽ [26] Guo+:
E ff ective Parallel Corpus Mining using Bilingual Sentence Embeddings, WMT ‘18 LaBSEͷߏཁૉ: Translation ranking task 41 զഐೣͰ͋Δɻ Ja I am a cat. En Nice to meet you. En ͚ۙͮΔ ԕ͚͟Δ ਖ਼ྫ ෛྫ -
•GuoΒ[26] ͕ఏҊ •༁จϖΞͷྨࣅΛɺͦͷଞͷจϖΞͷྨࣅΑΓߴ͘͢ΔΑ͏ʹֶश • ༁จϖΞΛਖ਼ྫͱͨ͠ରরֶश (Contrastive Learning) •“ranking”ͱݴ͍ͭͭͬͯΔͷਖ਼ྫ(ਖ਼͍͠༁จϖΞ)ͷྨࣅ࠷େԽ [26] Guo+:
E ff ective Parallel Corpus Mining using Bilingual Sentence Embeddings, WMT ‘18 LaBSEͷߏཁૉ: Translation ranking task 42 զഐೣͰ͋Δɻ Ja I am a cat. En Nice to meet you. En ͚ۙͮΔ ԕ͚͟Δ ਖ਼ྫ ෛྫ ྨࣅ࠷େԽ ྨࣅ࠷খԽ -
•·ͣ༁จϖΞΛจຒΊࠐΈʹ •ਖ਼ྫɾෛྫͯ͢ͷΈ߹Θͤʹ ͍ͭͯྨࣅΛܭࢉ • ྨࣅͷߦྻ͕Ͱ͖Δ LaBSEͷߏཁૉ: Translation ranking task 43
ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. ਖ਼ྫ զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ -
•·ͣ༁จϖΞΛจຒΊࠐΈʹ •ਖ਼ྫɾෛྫͯ͢ͷΈ߹Θͤʹ ͍ͭͯྨࣅΛܭࢉ • ྨࣅͷߦྻ͕Ͱ͖Δ •ਖ਼ྫͷྨࣅΛ࠷େԽ͢Δ • ʹྨࣅߦྻͷର֯ઢ͕ਖ਼ղ LaBSEͷߏཁૉ: Translation
ranking task 44 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. ਖ਼ྫ զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ ྨࣅ0.98… 0.24… -
•·ͣ༁จϖΞΛจຒΊࠐΈʹ •ਖ਼ྫɾෛྫͯ͢ͷΈ߹Θͤʹ ͍ͭͯྨࣅΛܭࢉ • ྨࣅͷߦྻ͕Ͱ͖Δ •ਖ਼ྫͷྨࣅΛ࠷େԽ͢Δ • ʹྨࣅߦྻͷର֯ઢ͕ਖ਼ղ • ߦํ(→)ʹSoftmaxͯ͠ਖ਼نԽ
• 1ରNΛNճ܁Γฦ͢Πϝʔδ LaBSEͷߏཁૉ: Translation ranking task 45 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. ਖ਼ྫ զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ 0.24… ྨࣅ0.98… -
•·ͣ༁จϖΞΛจຒΊࠐΈʹ •ਖ਼ྫɾෛྫͯ͢ͷΈ߹Θͤʹ ͍ͭͯྨࣅΛܭࢉ • ྨࣅͷߦྻ͕Ͱ͖Δ •ਖ਼ྫͷྨࣅΛ࠷େԽ͢Δ • ʹྨࣅߦྻͷର֯ઢ͕ਖ਼ղ • ߦํ(→)ʹSoftmaxͯ͠ਖ਼نԽ
• 1ରNΛNճ܁Γฦ͢Πϝʔδ •ଛࣦؔˠ • ຒΊࠐΈͷੵ ϕ LaBSEͷߏཁૉ: Translation ranking task 46 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. ਖ਼ྫ զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ 0.24… ྨࣅ0.98… -
•ಉ͡όονͷผͷϖΞΛෛྫʹ͢Δ • in-batch negativesͱݺΕΔ • ྨࣅߦྻ (batch_size x batch_size) ͷਖ਼ํߦྻʹͳΔ
• (ෛྫΛߋʹ૿͢͜ͱՄೳ) LaBSEͷߏཁૉ: Translation ranking task 47 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. ਖ਼ྫ զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ -
•ಉ͡όονͷผͷϖΞΛෛྫʹ͢Δ • in-batch negativesͱݺΕΔ • ྨࣅߦྻ (batch_size x batch_size) ͷਖ਼ํߦྻʹͳΔ
• (ෛྫΛߋʹ૿͢͜ͱՄೳ) LaBSEͷߏཁૉ: Translation ranking task 48 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. batch_size ਖ਼ྫ batch_size զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ -
•ಉ͡όονͷผͷϖΞΛෛྫʹ͢Δ • in-batch negativesͱݺΕΔ • ྨࣅߦྻ (batch_size x batch_size) ͷਖ਼ํߦྻʹͳΔ
• (ෛྫΛߋʹ૿͢͜ͱՄೳ) •softmaxͰଛࣦ͕ඇରশੑʹ LaBSEͷߏཁૉ: Translation ranking task 49 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. batch_size ਖ਼ྫ batch_size զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ -
•ಉ͡όονͷผͷϖΞΛෛྫʹ͢Δ • in-batch negativesͱݺΕΔ • ྨࣅߦྻ (batch_size x batch_size) ͷਖ਼ํߦྻʹͳΔ
• (ෛྫΛߋʹ૿͢͜ͱՄೳ) •softmaxͰଛࣦ͕ඇରশੑʹ LaBSEͷߏཁૉ: Translation ranking task 50 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. batch_size ਖ਼ྫ batch_size զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ -
•ಉ͡όονͷผͷϖΞΛෛྫʹ͢Δ • in-batch negativesͱݺΕΔ • ྨࣅߦྻ (batch_size x batch_size) ͷਖ਼ํߦྻʹͳΔ
• (ෛྫΛߋʹ૿͢͜ͱՄೳ) •softmaxͰଛࣦ͕ඇରশੑʹ • ղফ͢ΔͨΊɺ2ํ(→↓)ͷଛࣦΛ͠߹ΘͤΔ LaBSEͷߏཁૉ: Translation ranking task 51 ࢲϖϯͰ͢ɻ I am a pen. I’m a cat. Nice to m eet you. Sentence em bedding I’m a perfect hum an. batch_size ਖ਼ྫ batch_size զഐೣͰ͋Δɻ ͡Ί·ͯ͠ɻ จຒΊࠐΈ ࢲᘳͳਓؒͰ͢ɻ -
•ྨࣅؔ ΛmarginΛಋೖͨ͠ ʹஔ͖͑ • marginਖ਼ྫʹ͚ͩద༻ •ਖ਼ྫΑΓू·ΓɺෛྫΑΓΕΔ ϕ ϕ′  [27]
Yang+: Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax, IJCAI ‘19 LaBSEͷߏཁૉ: Additive Margin Softmax (AMS) [27] 52 -
•ྨࣅؔ ΛmarginΛಋೖͨ͠ ʹஔ͖͑ • marginਖ਼ྫʹ͚ͩద༻ •ਖ਼ྫΑΓू·ΓɺෛྫΑΓΕΔ •มߋޙͷଛࣦؔ͜͏↓ ϕ ϕ′ 
[27] Yang+: Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax, IJCAI ‘19 LaBSEͷߏཁૉ: Additive Margin Softmax (AMS) [27] 53 -
•ྨࣅؔ ΛmarginΛಋೖͨ͠ ʹஔ͖͑ • marginਖ਼ྫʹ͚ͩద༻ •ਖ਼ྫΑΓू·ΓɺෛྫΑΓΕΔ •มߋޙͷଛࣦؔ͜͏↓ ϕ ϕ′ 
[27] Yang+: Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax, IJCAI ‘19 LaBSEͷߏཁૉ: Additive Margin Softmax (AMS) [27] 54 -
Masked Language Modeling (MLM) •ϚεΫ͞Εͨ෦ʹͯ·Δ୯ޠΛ༧ଌ͢Δ͜ͱͰϞσϧΛ܇࿅ • ࣗݾڭࢣ͋ΓֶशͰ൚༻తͳݴޠࣝΛ֫ಘ •BERT [00]Ͱ͓ೃછΈ LaBSEͷߏཁૉ:
MLM and TLM Pre-training 55 զഐ [MASK] Ͱ͋Δ ɻ [CLS] [SEP] BERT ೣ -
Translation Language Modeling (TLM) [28] •༁จϖΞΛ࿈݁ͯ͠MLM • ೋݴޠؒͷରԠ (alignment) ͷֶशΛظ
[28] Conneau+: Cross-lingual Language Model Pretraining, NeurIPS ‘19 LaBSEͷߏཁૉ: MLM and TLM Pre-training 56 զഐ [MASK] Ͱ͋Δ ɻ [/s] [/s] Transformer ೣ [MASK] am a [/s] I cat -
•TLMMLMͷ֦ு • ଟগͷมߋͰMLMͱಉ͡Α͏ʹֶश͕Ͱ͖Δ •LaBSEͰMLMͱTLM ΛΈ߹Θֶͤͯश [28] Conneau+: Cross-lingual Language Model
Pretraining, NeurIPS ‘19 LaBSEͷߏཁૉ: MLM and TLM Pre-training 57 ͜ͷݚڀͰଟݴޠೳྗ (multilinguality)Λ্ͤ͞ΔͨΊ Language embeddings Λ͍ͬͯͳ͍ -
•BiLSTMΛ༻͍ͨseq2seqͷ༁λεΫΛղ͖จຒΊࠐΈϞσϧΛ֫ಘ • Encoder͕ݴޠඇґଘͷจදݱΛநग़ • DecoderʹจຒΊࠐΈͱݴޠIDΛิॿతʹೖྗͯ͠จੜ •ݴޠԣஅNLI (XNLI)ଟݴޠݕࡧλεΫͰߴ͍ੑೳɺ97ݴޠʹରԠ [29] Artetxe+: Massively
Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond, TACL ‘18 ؔ࿈ݚڀ: LASER [29] 58 -
•ϚϧνλεΫֶशʹΑͬͯߴੑೳͳଟݴޠจຒΊࠐΈϞσϧΛ֫ಘ • Translation ranking taskֶ͕शλεΫʹؚ·Ε͍ͯΔ • ࣗݾڭࢣ͋ΓֶशͳͲʹΑΔࣄલֶश(ڪΒ͘)͍ͯ͠ͳ͍ [30] Yang+: Multilingual
Universal Sentence Encoder for Semantic Retrieval, ACL: System Demonstrations ‘20 ؔ࿈ݚڀ: m-USE [30] 59 -
Monolingual Data •CommonCrawl, Wikipedia͔Βऩूɺ17B (170ԯ) sentences •લॲཧࡁΈɺࣄલֶश(MLM)ʹͷΈར༻ Bilingual Translation Pairs
•Webϖʔδͷ༁จϚΠχϯά(bitext mining)Ͱσʔλऩूɺ6B (60ԯ) pairs •σʔλෆۉߧରࡦͱͯ͠ɺ֤ݴޠͷจ͕100MҎԼʹͳΔΑ͏੍ݶ •αϒηοτͷਓखධՁΛ༻͍࣭ͨσʔλͷϑΟϧλϦϯά •ࣄલֶश(MLM & TLM)ͱdual-encoderͷ܇࿅ʹར༻ ֶशσʔλ 60 -
United Nations (UN) •ӳޠ͔ΒରԠ͢ΔผݴޠͷจॻΛݕࡧ (Precision@1ʹaccuracy) •en-fr, en-es, en-ru, en-ar, en-zhͷ5ݴޠରɺ86,000จ
Tatoeba •ӳޠҎ֎ͷݴޠ͔ΒରԠ͢Δӳ༁Λݕࡧ (Average accuracy) •https://tatoeba.org ͔ΒྫจͱͦΕʹඥͮ͘ର༁Λऩूͨ͠ίʔύε •112ݴޠɺ֤ݴޠʹ͖ͭ1000จͱରԠ͢Δӳ༁͕ଘࡏ •طଘݚڀʹ฿͍ɺ36ݴޠͷΈʹߜͬͨαϒηοτͰͷධՁ࣮ࢪ BUCC •୯ݴޠίʔύε͔Β༁จϖΞΛݟ͚ͭΔ (Precision, Recall, F1) •fr-en, de-en, ru-en, zh-enͷ4ݴޠର ධՁλεΫ: bitext retrieval 62 -
•ޠኮαΠζ • mBERT Vocab: multilingual BERT (mBERT)ͱಉ͡(119,547) • Customized Vocab:
ݴޠ͝ͱͷσʔλෆۉߧରࡦΛͯ͠ॳΊ͔Β࡞ (501,153) •ࣄલֶश (PT) • MLM+TLMʹΑΔࣄલֶशΛΔ͔Ͳ͏͔ • Βͳ͍߹Translation ranking taskͷΈΛߦ͏ •Additive Margin Softmax (AMS) • Translation ranking taskʹmarginΛ͏͔Ͳ͏͔ ࣮ݧ݅ 64 -
•จຒΊࠐΈʹ [CLS] ΛL2ਖ਼نԽͯ͠ར༻ •optimizer = AdamW, learning rate = 1e-3,
seq length = 128 Pre-training •batch size: 8192 Translation ranking task •batch size: 4096 •w/ Pre-training: 50k steps, w/o Pre-training: 500k steps •margin value: 0.3 ࣮ݧઃఆ 65 -
•LaBSE (Customized Vocab + AMS + PT) ʹSOTAߋ৽ • Yang
et al. bilingual modelͰ͋Γɺ֤ݴޠ͝ͱʹϞσϧ͕ඞཁ • LaBSEҰͭͷϞσϧͰ109Ҏ্ͷݴޠʹରԠՄೳ ࣮ݧ݁Ռ: United Nations (UN) & Tatoeba 66 ͳ͔ͥBase w/ Customized Vocabͷ݁Ռ͕ଘࡏ͠ͳ͍ -
•maringͷʹΑΔUNͰͷ ੑೳมԽΛ؍ • margin͕ੑೳ্ʹد༩ •0.3ఔͰੑೳมԽ͕ऩଋ • ͦ͜·Ͱconsistentʹ্ • શͯͷϞσϧͰੑೳ্͕ ੳ:
Additive Margin Softmax 72 marginΛ0.4ΑΓେ͖͍ͯͬͨ࣌͘͠ͷมԽ͕ऩଋͨ͠··ͳͷ͔ؾʹͳΔ͕… -
•PTʹΑͬͯશମతͳੑೳ্ •PT͋ΓͷϞσϧ50K stepsͷ ܇࿅Ͱطʹੑೳ͕ऩଋ • 50K steps = 200M examples
• ର༁σʔλ͕গͳ͍͍ͯ͘ •PTੑೳ্ͱऩଋ্ (=܇࿅ࣄྫݮ)ʹཱͭ ੳ: ࣄલֶशͷ༗༻ੑ 73 -
•Tatoebaͷগࢿݯݴޠʹ ର͢ΔੑೳΛੳ • LaBSEগσʔλͳݴޠ ͕ࠞͬͯ͟ੑೳ͕ߴ͍ •en-xxͳݴޠରͷσʔλ͕ ଘࡏ͠ͳ͍ݴޠʹ͍ͭͯͷ TatoebaͰͷੑೳΛੳ • 3ׂ͕75%͑ɺ݁ߏߴ͍
• ݴޠಉ࢜ͷྨࣅͷଞʹɺ େنσʔλͰͷ܇࿅ͷ ޮՌͩͱߟ͑ΒΕΔ ੳ: গࢿݯݴޠʹର͢Δੑೳ 74 -
•LaBSEͷӳޠSTSʹ͓͚Δ ੑೳΛੳ •NLIͰֶश͍ͯ͠Δm-USE ͕ඇৗʹߴ͍ੑೳ • SBERTΑΓߴ͍ʁ •LaBSEΊͷ •༁จϖΞʹΑΔֶश • ҙຯͷՁੑͷݕग़ʹ༏ΕΔ͕
• ҙຯ͕ͲΕ͘Β͍ҟͳ͍ͬͯΔ͔Λࡉ͔۠͘ผ͢Δ͜ͱͰ͖ͳ͍ ੳ: STS 75 -
•CommonCrawl͔Βର༁σʔλΛऩू͢Δ࣮ݧΛ࣮ࢪ •ऩूͨ͠ର༁ίʔύεͰػց༁ϞσϧΛ܇࿅ɺੑೳධՁ • ଟݴޠจຒΊࠐΈϞσϧͷԠ༻ɾ༗༻ੑධՁ •ӳޠɺதࠃޠɺυΠπޠͷCommonCrawlΛલॲཧ • ·ͣશͯͷจΛຒΊࠐΈදݱʹ • ͯ͢ͷඇӳޠͷจʹରͯ͠ɺຒΊࠐΈʹ͓͚Δ࠷ۙΛରԠ͚ͮ •
ྨࣅ͕0.6ະຬͷͷআڈ •WMTͷϕϯνϚʔΫσʔληοτΛ༻͍ͯɺBLEUͰ༁ੑೳධՁ Ճతͳ࣮ݧ: Mining Parallel Text from CommonCrawl 76 -
•109Ҏ্ͷݴޠʹద༻ՄೳͳଟݴޠจຒΊࠐΈϞσϧLaBSEΛఏҊ •ଟݴޠࣄલֶशˠ༁ίʔύεΛ༻͍ͨจຒΊࠐΈͷରরֶश • MLM + Translation Language Modeling → Additive
Margin Softmax •छʑͷධՁ࣮ݧ • ݴޠԣஅݕࡧͷੑೳΛେ෯ʹվળ • ಛʹগࢿݯݴޠͰߴ͍ੑೳ • ୯ݴޠSTS/SentEvalߴ͘ͳ͍ • ࣄલֶशʹΑͬͯର༁σʔλྔΛݮ • AMS͕ੑೳʹେ͖ͳӨڹ •ࣄલֶशࡁΈϞσϧ͕ެ։͞Ε͍ͯΔ ·ͱΊ: Language-agnostic BERT Sentence Embedding 78 https://arxiv.org/abs/2007.01852 -
•(ॳग़2020ͷݚڀ͚ͩͲ)ͪΐͬͱݹ͍ • ൺֱରͱͯ͠SimCSEDistilCSE͕ΘΕ͍ͯͳ͍ •Typo͕ͪΐͬͱଟ͍ʢ ; ; ʣ •ରরֶशΛ༻͍ͨจຒΊࠐΈख๏ͷఆੴ͕ΘΕ͍ͯͳ͍ • จྨࣅͱͯ͠ੵΛ͍ͬͯΔ
• ԹύϥϝʔλΛ͍ͬͯͳ͍ • ͱࢥͬͨΒL2 normalization→scaling͍ͯͨ͠ͷͰຊ࣭తʹಉͩͬͨ͡ ͪΐͬͱؾʹͳΔ 80 -
•࠷ۙPearsonͰͳ͘SpearmanͰධՁ͢Δ͜ͱ͕ଟ͍ • Pearson͋Μ·ΓධՁࢦඪͱͯ͠ྑ͘ͳ͍ΑͶͱ͍͏͕͋Δ [31] •SpearmanͰධՁ͢ΔݶΓSTSʮॱҐ͚λεΫʯͳ͜ͱʹҙ •࠷ۙSTS Benchmarkͷdev setͰϋΠύϥௐ͢Δͷ͕ྲྀߦ͍ͬͯΔ • 250
step͝ͱʹdevͰධՁͯ͠࠷ߴͷcheckpointͰtestͰධՁ (SimCSE) • STSʹաద߹ͦ͠͏ͳͷͰ͋·Γྑ͍ํʹࢥ͑ͳ͍͕… • ʮֶशʹ͑ͳ͍͚Ͳdevͱͯ͑͠·͢ʂʯྑ͍ઃఆʁ •STSλεΫධՁख๏͕จ͝ͱʹҟͳΔ͕࣌͋Γɺҙ͕ඞཁ • ධՁࢦඪධՁखॱ͕όϥ͍͍ͭͯΔ͜ͱ͕͋ͬͨ (࠷ۙ౷Ұ͞ΕͯΔ) • SimCSEจ [11] ͷAppendix.Bʹهड़͕͋ΔͷͰҰಡΛਪ [31] Reimers+: Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity, COLING ‘16 ಋೖ: STSʹ·ͭΘΔখൌ 81 -
•࣮จͷҙຯҎ֎ΛຒΊࠐΜͰ͍͍ • ༻୯ޠɺελΠϧ(ϑΥʔϚϧ͞ɺܟޠ)ɺ࣭ͱ͑ͷۙ͞ɺetc. •จຒΊࠐΈۭؒจͷԿΛ͚ۙͮΔ͔Ͱಛ͚ͮΒΕΔ • ͲͷΑ͏ʹڑΛఆٛ͢Δ͔͕จຒΊࠐΈͷੑ࣭ΛܾΊΔ •܇࿅Ͱ͚ۙͮΔจͱͦΕʹΑͬͯදݱ͞ΕΔ“ڑ”ͷରԠ (චऀͷ༧ଌ) • ؚҙؔʹ͋Δจ:
จͷදతྨࣅΑΓҙຯʹ • ࣭ͱճ: จࣗମͷҙຯΑΓ࣭ͱճ͕ද͢༰ʹ • ༁จϖΞ: Ͳͷݴޠ͔Λແࢹͯ͠จͷҙຯʹ ಋೖ: จຒΊࠐΈ͕ຒΊࠐΉͷ(ൃදऀͷ͓ؾ࣋ͪ) 82 -
[08] Conneau+: Supervised Learning of Universal Sentence Representations from Natural
Language Inference Data, EMNLP ’17 [10] Reimers+: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP ’19 [11] Gao+: SimCSE: Simple Contrastive Learning of Sentence Embeddings, EMNLP ’21 [32] Hill+: Learning Distributed Representations of Sentences from Unlabelled Data, NAACL ’16 [33] Wang+:, TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning, EMNLP fi ndings ’21 [34] Li+: OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space, EMNLP ’20 ಋೖ: จຒΊࠐΈ͕ຒΊࠐΉͷ(ൃදऀͷ͓ؾ࣋ͪ) 83 จͷҙຯΛຒΊࠐΉख๏ •InferSent: dual-encoder (Siamese) ߏͰLSTMΛNLIྨͰֶश [08] •Sentence-BERT: dual-encoderߏͰBERTΛNLIྨͰ fi ne-tuning [10] •Supervised SimCSE: NLIͷؚҙؔͷจϖΞΛਖ਼ྫͱͨ͠ରরֶश[11] จΛ࠶ߏஙͰ͖ΔΑ͏ʹจͷใΛຒΊࠐΉख๏ •SDAE: ೖྗจͷϊΠζΛআڈͭͭ͠࠶ߏஙͯ͠LSTMΛֶश [32] •TSDAE: ೖྗจͷϊΠζΛআڈͭͭ͠࠶ߏஙͯ͠TransformerΛֶश [33] •Optimus: Ͱ͔͍VAE [34] -
[06] Kiros+: Skip-Thought Vectors, NIPS ’15 [09] Cer+: Universal Sentence
Encoder, arXiv, Mar 2018 [35] Tsukagoshi+: DefSent: Sentence Embeddings using De fi nition Sentences, ACL ’21 [36] Wu+: DistilCSE: E ff ective Knowledge Distillation For Contrastive Sentence Embeddings, ARR ’22 [37] Wu+: DisCo: E ff ective Knowledge Distillation For Contrastive Learning of Sentence Embeddings, arXiv ’21 ([35]ͱಉ༰) ಋೖ: จຒΊࠐΈ͕ຒΊࠐΉͷ(ൃදऀͷ͓ؾ࣋ͪ) 84 લޙͷจͷใΛຒΊࠐΉख๏ •Skip-Thought: લޙͷจΛ࠶ߏ͢ΔΑ͏ʹੜతʹֶश [06] •USE: Skip-Thoughtͷڭࢣͳֶ͠श + ྨʹΑΔڭࢣ͋Γֶश [09] ఆٛจ͔Β୯ޠͷҙຯΛߏ͢ΔΑ͏ʹจͷҙຯΛຒΊࠐΉख๏ •DefSent: લޙͷจΛ࠶ߏ͢ΔΑ͏ʹੜతʹֶश [35] Α͘Θ͔Βͳ͍ख๏ •Unsupervised SimCSE: ҟͳΔdropoutΛద༻ͨ͠จΛਖ਼ྫʹରরֶश [11] •DistilCSE: ڭࢣͱੜెͷจຒΊࠐΈΛਖ਼ྫͱͨ͠ରরֶशʹΑΔৠཹ [36, 37] -
•୯ݴޠจຒΊࠐΈϞσϧ͔ΒͷࣝৠཹͰଟݴޠจຒΊࠐΈϞσϧΛ܇࿅ • ༁ίʔύεΛͬͯҟͳΔݴޠͷจຒΊࠐΈΛ͚ۙͮΔ •ҟݴޠؒSTSͰΑ͍ੑೳ •୯ݴޠSTSͰੑೳ͕ߴ͍ • NLIΛ͍ͬͯΔͷͰͨΓલʁ •ݴޠόΠΞε͕LaBSEΑΓখ͍͞ •Teacher modelͷจຒΊࠐΈۭؒͱࣅͨߏΛ࣋ͭଟݴޠจຒΊࠐΈϞσ
ϧΛ࡞ΕΔͷ͕ྑ͍ͱ͜Ζ • ҰͭͷϞσϧͰsequentialʹΔͱۭ͕ؒյΕΔ [38] Reimers+: Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation, EMNLP ‘20 ͕ؔ͋Δจ: Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation [38] 85