[輪講資料] Text Embeddings by Weakly-Supervised Contrastive Pre-training

大規模な弱教師あり対照学習によって訓練された強力なテキスト埋め込みモデルE5について解説した輪講資料です。

元論文: https://arxiv.org/abs/2212.03533

Transcript

1. Text Embeddings by Weakly-Supervised Contrastive Pre-training Graduate School of Informatics, Nagoya University, Japan. ൃදऀ: Hayato Tsukagoshi Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei
 https://arxiv.org/abs/2212.03533
2. •େن໛ͳࣄલରরֶशʹΑΓߏங͞Εͨ
 ςΩετຒΊࠐΈϞσϧE5ΛఏҊ • ൒ߏ଄ԽσʔλͱϑΟϧλϦϯάΛ
 ༻͍ͨऑڭࢣ͋ΓσʔληοτΛߏங • όοναΠζ32000Ͱͷpre-training • hard negativeͱCross-Encoder͔Βͷ
 ஌ࣝৠཹΛ׆༻ͨ͠ fi ne-tuning •ධՁͷ݁ՌछʑͷϕϯνϚʔΫͰ
 ฏۉͯ͠طଘϞσϧΛ্ճΔ ֓ཁ 2 #Layers hidden size #params E5-small 12 384 33M E5-base 12 768 110M E5-large 24 1024 330M
3. •ۙ೥࠷΋Α͘ར༻͞Ε͍ͯΔจຒΊࠐΈϞσϧͷҰ͔ͭͩΒ •จຒΊࠐΈݚڀͷࠓޙͷํ޲ੑΛཧղ͢Δ্ͰࢀߟʹͳΔ࿦จ͔ͩΒ • Cross-Encoder͔Βͷ஌ࣝৠཹ • ଟஈ֊Ͱͷରরֶश ໔੹ࣄ߲ •εϥΠυதͷਤද͸֤εϥΠυͰݴٴ͞Ε͍ͯΔ࿦จ͔ΒͷҾ༻Ͱ͢ •࿦จதͷ਺ࣜͱ͸ҟͳΔจࣈΛ࢖͍ͬͯΔ৔߹͕͋Γ·͢ બఆཧ༝ 3
4. ࣄલ஌ࣝ
5. •ۙ೥ͷϞσϧͷଟ͘͸஫ҙػߏ(Attention Mechanism)ʹجͮ͘ TransformerͰߏ੒ •͍Ζ͍Ζͳछྨ͕ଘࡏ ݴޠϞσϧ: Language Models 5 ଞʹ΋ݴޠϞσϧʹ͸͞·͟·ͳछྨ͕ଘࡏɻྫ: XLNet, ELECTRA, UL2, … BERTͷ֓ཁਤ
6. •ۙ೥ͷϞσϧͷଟ͘͸஫ҙػߏ(Attention Mechanism)ʹجͮ͘ TransformerͰߏ੒ •͍Ζ͍Ζͳछྨ͕ଘࡏ ࣗݾճؼܕݴޠϞσϧ (Causal LM) •ࠨ͔Βӈʹ୯ޠΛ༧ଌͯ͠܇࿅ •ྫ: GPT, GPT-2, GPT-3, Llama 2, … ݴޠϞσϧ: Language Models 6 ଞʹ΋ݴޠϞσϧʹ͸͞·͟·ͳछྨ͕ଘࡏɻྫ: XLNet, ELECTRA, UL2, … BERTͷ֓ཁਤ
7. •ۙ೥ͷϞσϧͷଟ͘͸஫ҙػߏ(Attention Mechanism)ʹجͮ͘ TransformerͰߏ੒ •͍Ζ͍Ζͳछྨ͕ଘࡏ ࣗݾճؼܕݴޠϞσϧ (Causal LM) •ࠨ͔Βӈʹ୯ޠΛ༧ଌͯ͠܇࿅ •ྫ: GPT, GPT-2, GPT-3, Llama 2, … ϚεΫݴޠϞσϧ (Masked LM) •จதͷҰ෦ΛӅ͢ɾ༧ଌͯ͠܇࿅ •ྫ: BERT, RoBERTa, DeBERTa, … ݴޠϞσϧ: Language Models 7 ଞʹ΋ݴޠϞσϧʹ͸͞·͟·ͳछྨ͕ଘࡏɻྫ: XLNet, ELECTRA, UL2, … BERTͷ֓ཁਤ
8. •ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ •ೖྗΛQ (Query), K (Key), V (Value)ʹ෼͚ͯܭࢉ • K, V: nݸͷd࣍ݩϕΫτϧ • Q: mݸͷd࣍ݩϕΫτϧ ஫ҙػߏ (Attention Mechanism) 8 ਤ͸ Jaegle et al., Perceiver IO: A General Architecture for Structured Inputs & Outputs, ICLR 2022. ΑΓҾ༻ Θ͔Γ΍͍͢ղઆ: ʲਂ૚ֶशʳAttention - શྖҬʹԠ༻͞Ε࠷ߴਫ਼౓Λୟ͖ग़͢஫ҙػߏͷ࢓૊ΈʲσΟʔϓϥʔχϯάͷੈք vol. 24ʳ
9. •ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ •ೖྗΛQ (Query), K (Key), V (Value)ʹ෼͚ͯܭࢉ • K, V: nݸͷd࣍ݩϕΫτϧ • Q: mݸͷd࣍ݩϕΫτϧ •Qʹର͢ΔVͷॏཁ౓ΛQͱKͷ಺ੵˠSoftmaxͰܭࢉ • Attention Weights: ܭࢉͷ݁ՌಘΒΕΔ(m × n)ߦྻ ஫ҙػߏ (Attention Mechanism) 9 ਤ͸ Jaegle et al., Perceiver IO: A General Architecture for Structured Inputs & Outputs, ICLR 2022. ΑΓҾ༻ Θ͔Γ΍͍͢ղઆ: ʲਂ૚ֶशʳAttention - શྖҬʹԠ༻͞Ε࠷ߴਫ਼౓Λୟ͖ग़͢஫ҙػߏͷ࢓૊ΈʲσΟʔϓϥʔχϯάͷੈք vol. 24ʳ
10. •ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ •ೖྗΛQ (Query), K (Key), V (Value)ʹ෼͚ͯܭࢉ • K, V: nݸͷd࣍ݩϕΫτϧ • Q: mݸͷd࣍ݩϕΫτϧ •Qʹର͢ΔVͷॏཁ౓ΛQͱKͷ಺ੵˠSoftmaxͰܭࢉ • Attention Weights: ܭࢉͷ݁ՌಘΒΕΔ(m × n)ߦྻ •Self-Attention (ࣗݾ஫ҙػߏ): Q, K, VΛಉ͡ϕΫτϧྻ͔Βߏ੒ (i.e. n=m) •Cross-Attention: ʮQʯͱʮK, VʯΛҟͳΔϕΫτϧྻ͔Βߏ੒ ஫ҙػߏ (Attention Mechanism) 10 ਤ͸ Jaegle et al., Perceiver IO: A General Architecture for Structured Inputs & Outputs, ICLR 2022. ΑΓҾ༻ Θ͔Γ΍͍͢ղઆ: ʲਂ૚ֶशʳAttention - શྖҬʹԠ༻͞Ε࠷ߴਫ਼౓Λୟ͖ग़͢஫ҙػߏͷ࢓૊ΈʲσΟʔϓϥʔχϯάͷੈք vol. 24ʳ
11. •஫ҙػߏͷΈͰߏ੒͞ΕͨϞσϧߏ଄ • ͦΕ·ͰNLPͰΑ͘ར༻͞Ε͍ͯͨ
 RNN, LSTM΍CNNΛഉআ • Transformer 11 Vaswani etl al., Attention Is All You Need, NeurIPS 2017. Θ͔Γ΍͍͢ղઆ: ʲਂ૚ֶशʳTransformer - Multi-Head AttentionΛཧղͯ͠΍Ζ͏͡Όͳ͍ͷʲσΟʔϓϥʔχϯάͷੈքvol.28ʳ ֓ཁਤ Encoder Decoder
12. •஫ҙػߏͷΈͰߏ੒͞ΕͨϞσϧߏ଄ • ͦΕ·ͰNLPͰΑ͘ར༻͞Ε͍ͯͨ
 RNN, LSTM΍CNNΛഉআ •ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ • ೖྗϕΫτϧಉ࢜ͷ૬ޓ࡞༻Λߟྀ Transformer 12 Vaswani etl al., Attention Is All You Need, NeurIPS 2017. Θ͔Γ΍͍͢ղઆ: ʲਂ૚ֶशʳTransformer - Multi-Head AttentionΛཧղͯ͠΍Ζ͏͡Όͳ͍ͷʲσΟʔϓϥʔχϯάͷੈքvol.28ʳ ֓ཁਤ Encoder Decoder
13. •஫ҙػߏͷΈͰߏ੒͞ΕͨϞσϧߏ଄ • ͦΕ·ͰNLPͰΑ͘ར༻͞Ε͍ͯͨ
 RNN, LSTM΍CNNΛഉআ •ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ • ೖྗϕΫτϧಉ࢜ͷ૬ޓ࡞༻Λߟྀ •EncoderͱDecoderͷೋछྨ͕ଘࡏ • EncoderͷΈ: BERT, LUKE, … • DecoderͷΈ: GPT, GPT-2, GPT-3, … • Encoder-Decoder: BART, T5, UL2, … Transformer 13 Vaswani etl al., Attention Is All You Need, NeurIPS 2017. Θ͔Γ΍͍͢ղઆ: ʲਂ૚ֶशʳTransformer - Multi-Head AttentionΛཧղͯ͠΍Ζ͏͡Όͳ͍ͷʲσΟʔϓϥʔχϯάͷੈքvol.28ʳ ֓ཁਤ Encoder Decoder
14. •Transformer EncoderΛෳ਺૚ॏͶͯେن໛ʹࣄલֶशͨ͠Ϟσϧ • base͸12૚ (1ԯύϥϝʔλ)ɺlarge͸24૚ (3.3ԯύϥϝʔλ) •ࣄલֶश (pre-training) → ඍௐ੔ ( fi ne-tuning) ͱ͍͏ύϥμΠϜ͕ීٴ BERT: Bidirectional Encoder Representations from Transformers 14 Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019.
15. •ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ཭͕จͷҙຯͷۙ͞Λදݱ จຒΊࠐΈ: Sentence Embedding 15 ͜Ͳ΋͕Ոʹ޲͔͍ͬͯΔɻ ͜Ͳ΋ֶ͕ߍ͔ΒՈʹ޲͔͍ͬͯΔɻ ͜Ͳ΋͕ਤॻؗʹ͍Δɻ ͜Ͳ΋͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...]
16. •ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ཭͕จͷҙຯͷۙ͞Λදݱ จຒΊࠐΈ: Sentence Embedding 16 ͜Ͳ΋͕Ոʹ޲͔͍ͬͯΔɻ ͜Ͳ΋ֶ͕ߍ͔ΒՈʹ޲͔͍ͬͯΔɻ ͜Ͳ΋͕ਤॻؗʹ͍Δɻ ͜Ͳ΋͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] ҙຯతʹྨࣅ ͍ۙҙຯΛ࣋ͭจ͸ ۙ͘ʹ෼෍ ϕΫτϧؒͷڑ཭͕
 ҙຯతͳؔ܎Λදݱ
17. ॳظ (~2018) •੩త୯ޠຒΊࠐΈ(Word2Vec, GloVe)͔ΒจຒΊࠐΈΛߏ੒͢Δख๏͕ओྲྀ • SIF, uSIF, All-but-the-Top, … •LSTM౳Λར༻ͯ͠from scratchʹֶश͢Δख๏΋͍͔ͭ͘ଘࡏ • SkipThought, InferSent, Universal Sentence Encoder (USE), … ࣄલֶशϞσϧඍௐ੔ख๏ͷ୆಄ (2019~2021) •BERTͷ fi ne-tuningʹΑΓจຒΊࠐΈϞσϧΛಘΔख๏͕૿Ճ • BERT- fl ow, Sentence-BERT (SBERT), … จຒΊࠐΈݚڀͷมભ 17 ஫ҙ: ೥୅͸ͬ͘͟ΓͰ͢
18. ࣗવݴޠਪ࿦ (Natural Language Inference; NLI) •จϖΞ (લఏจɾԾઆจ) ʹϥϕϧ (ؚҙɾໃ६ɾதཱ) ͕෇༩ •จϖΞͷҙຯؔ܎Λ༧ଌ͢ΔλεΫ NLIσʔληοτ 18 લఏจ Ծઆจ ϥϕϧ A man playing an electric guitar on stage. A man playing guitar on stage. ؚҙ A man playing an electric guitar on stage. A man playing banjo on the fl oor. ໃ६ A man playing an electric guitar on stage. A man is performing for cash. தཱ
19. ରরֶशོ੝ظ (2021~) •ը૾෼໺Ͱྲྀߦ͍ͯͨ͠ରরֶशख๏͕จຒΊࠐΈʹ΋ •ಛʹ SimCSE ͕୅දతͳख๏ʹ • ڭࢣ͋Γɾڭࢣͳ͠ͷೋͭͷख๏ΛఏҊ Unsupervised SimCSE 1. ಉ͡ೖྗʹର͠ҟͳΔdropout maskͰforward 2. ಘΒΕͨʮಉ͡ೖྗʹର͢ΔҟͳΔग़ྗʯಉ࢜Λਖ਼ྫʹରরֶश Supervised SimCSE • NLIσʔληοτதͷʮؚҙʯؔ܎ʹ͋ΔจϖΞΛਖ਼ྫʹରরֶश จຒΊࠐΈݚڀͷมભ 19 ೔ຊޠSimCSEͷςΫχΧϧϨϙʔτ͸ͪ͜Β
20. •දݱֶश (representation learning) ͷख๏ͷҰͭ •ਖ਼ྫಉ͕࢜ۙͮ͘Α͏ʹɺ͔ͭɺෛྫಉ͕࢜཭ΕΔΑ͏ʹֶश͢Δ • ਖ਼ྫಉ࢜ͷྨࣅ౓࠷େԽ & ෛྫಉ࢜ͷྨࣅ౓࠷খԽ ରরֶश 20
21. •දݱֶश (representation learning) ͷख๏ͷҰͭ •ਖ਼ྫಉ͕࢜ۙͮ͘Α͏ʹɺ͔ͭɺෛྫಉ͕࢜཭ΕΔΑ͏ʹֶश͢Δ • ਖ਼ྫಉ࢜ͷྨࣅ౓࠷େԽ & ෛྫಉ࢜ͷྨࣅ౓࠷খԽ ଛࣦ (InfoNCE) ͷܭࢉ •ਖ਼ྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅ౓ΛٻΊΔ •ෛྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅ౓ΛٻΊΔ •ྨࣅ౓Λฒ΂ͯԹ౓ύϥϝʔλΛద༻͢Δ •Softmaxؔ਺Λద༻ͯ֬͠཰෼෍ͱΈͳ͢ •ਖ਼ྫʹ͚ͩ1ཱ͕ͭ෼෍ʹ͚ۙͮΔ ରরֶश 21
22. Unsupervised SimCSE:ʮਖ਼ଇԽ+จຒΊࠐΈಉ࢜Λ཭͢ʯ Supervised SimCSE:ʮҙຯతʹ͍ۙจຒΊࠐΈΛ͚ۙͮΔ+ͦͷଞͷจຒΊࠐΈಉ࢜Λ཭͢ʯ SimCSE: ֓ཁਤ 22
23. εέʔϦϯάظ (2022~) •܇࿅ͷେن໛Խ͕ੵۃతʹߦΘΕΔΑ͏ʹ • σʔλྔͱόοναΠζͷ૿େʹΑΔରরֶशͷεέʔϦϯά • ϞσϧύϥϝʔλͷεέʔϦϯά •multi-stage contrastive learningͷಋೖ • ऑڭࢣσʔλΛ༻͍ͨࣄલֶशˠڭࢣ͋ΓֶशʹΑΔFine-tuning • Ϟσϧྫ: E5, GTE, BGE, … •େن໛ݴޠϞσϧ(LLM)Λ༻͍ͨςΩετຒΊࠐΈͷݚڀ΋ൃలத • PromptEOL, E5-Mistral, LLM2Vec, … จຒΊࠐΈݚڀͷมભ 23 ೔ຊޠSimCSEͷςΫχΧϧϨϙʔτ͸ͪ͜Β
24. E5
25. •େن໛ͳࣄલରরֶशʹΑΓߏங͞Εͨ
 ςΩετຒΊࠐΈϞσϧE5ΛఏҊ • ൒ߏ଄ԽσʔλͱϑΟϧλϦϯάΛ
 ༻͍ͨऑڭࢣ͋ΓσʔληοτΛߏங • όοναΠζ32000Ͱͷpre-training • hard negativeͱCross-Encoder͔Βͷ
 ஌ࣝৠཹΛ׆༻ͨ͠ fi ne-tuning •ධՁͷ݁ՌछʑͷϕϯνϚʔΫͰ
 ฏۉͯ͠طଘϞσϧΛ্ճΔ ֓ཁ: ࠶ܝ 25 #Layers hidden size #params E5-small 12 384 33M E5-base 12 768 110M E5-large 24 1024 330M
26. E5ͷߏ੒ཁૉ •େن໛ͳςΩετϖΞσʔληοτͷߏங •ରরֶशʹΑΔେن໛ͳࣄલ܇࿅ •஌ࣝৠཹΛซ༻ͨ͠ϥϕϧ෇͖ߴ඼࣭σʔληοτͰͷ fi ne-tuning •Ϟσϧͷೖྗʹpre fi xΛ෇Ճɺσʔλͷඇରশͳؔ܎Λଊ͑ΒΕΔΑ͏ֶश E5: શମ૾ 26
27. •ਂ૚ֶशϞσϧͷ܇࿅Ͱ͸σʔλͷ඼࣭ͱଟ༷ੑ͕ੑೳΛେ͖͘ࠨӈ •͔͠͠ςΩετຒΊࠐΈϞσϧֶशͷͨΊͷσʔληοτ͸গ਺ • طଘݚڀ͸ Stanford NLI ΍ MS-MARCO ͳͲਓखখن໛σʔλΛར༻ •େن໛ͳςΩετຒΊࠐΈϞσϧ܇࿅༻σʔληοτΛߏங͢Δ • ൒ߏ଄Խ͞Εͨσʔλ͔ΒςΩετϖΞΛऩू (ϑΟϧλલ: 1.3B pairs) CCPairs: ࣄલରরֶशͷͨΊͷେن໛σʔληοτ 27 Source Query Passage Size Wikipedia entity + section title passage 24M Reddit post upvoted comment 60M Common Crawl title passage 69M ࠷ऴతʹऩू͞Εͨσʔλͱܗࣜͷྫ
28. •σʔλͷ඼࣭޲্ɾ܇࿅ίετ࡟ݮͷͨΊϑΟϧλϦϯάΛ࣮ࢪ • ࠷ऴతʹ270M·Ͱ࡟ݮ •ਂ૚ֶशϞσϧͷʮnoisyͳσʔλதͷ៉ྷͳࣄྫ͔Β֮͑ΔʯڍಈΛར༻ CCPairs: Consistency fi lterʹΑΔϊΠζআڈ 28 ޙଓͷGTEͰ͸ fi ltering͸͞Ε͍ͯͳ͍ Ϟσϧ͸Ԛ͍σʔλதͷ៉ྷͳσʔλ͔Β͍֮͑ͯ͘ →֮͑ΒΕͳ͔ͬͨσʔλ͸ʮԚ͍ʯ
29. •σʔλͷ඼࣭޲্ɾ܇࿅ίετ࡟ݮͷͨΊϑΟϧλϦϯάΛ࣮ࢪ • ࠷ऴతʹ270M·Ͱ࡟ݮ •ਂ૚ֶशϞσϧͷʮnoisyͳσʔλதͷ៉ྷͳࣄྫ͔Β֮͑ΔʯڍಈΛར༻ Consistency-based data fi ltering 1. 1.3BͷnoisyͳσʔληοτͰϞσϧΛ܇࿅ 2. 1MͷจষΛϥϯμϜʹநग़ͯ͠༻ҙ 3. ͋ΔΫΤϦʹର͠ਖ਼ྫจষͱϥϯμϜநग़͞Ε֤ͨจষͱͷྨࣅ౓Λ1ͷ ϞσϧΛ࢖ͬͯܭࢉ 4. ਖ਼ྫจষͷྨࣅ౓ॱҐ͕2Ҏ্ͷࣄྫͷΈ࢒͢ CCPairs: Consistency fi lterʹΑΔϊΠζআڈ 29 ޙଓͷGTEͰ͸ fi ltering͸͞Ε͍ͯͳ͍
30. •2ஈ֊ͷֶशख๏Λ࠾༻ 1. Contrastive Pre-training •௨ৗͷରরֶशଛࣦͱCCPairsΛ༻͍ͯڊେόοναΠζͰ܇࿅ • σʔλ͕noisyͳ৔߹΄ͲόοναΠζ͸େ͖ͨ͘͠΄͏͕ྑͦ͞͏ •ೖྗʹଐੑ৘ใΛද͢pre fi xΛ෇Ճ 2. Fine-tuning •ਓखͰ࡞੒͞Εͨϥϕϧ෇͖σʔληοτͰ fi ne-tuning • ରরֶशଛࣦͷଞʹ஌ࣝৠཹଛࣦ΋༻͍Δ ֶशख๏ 30 “query:” ͱ “passage:” ͷೋͭ
31. •ΫΤϦɾจষͷຒΊࠐΈಉ࢜Ͱྨࣅ౓Λܭࢉ • ΫΤϦ—ෛྫͷྨࣅ౓ΑΓΫΤϦ—ਖ਼ྫͷྨࣅ౓͕ߴ͘ͳΔΑ͏ֶश • ҙ༁: ΫΤϦͱจষͱͷྨࣅ౓ߦྻʹ͓͚Δର֯੒෼ͷ࠷େԽ •ಉ͡όον಺ͷଞͷࣄྫΛෛྫʹ͢Δ: in-batch negatives ֶशख๏: ରরֶश / Contrastive Pre-training 31
32. •ΫΤϦɾจষͷຒΊࠐΈಉ࢜Ͱྨࣅ౓Λܭࢉ • ΫΤϦ—ෛྫͷྨࣅ౓ΑΓΫΤϦ—ਖ਼ྫͷྨࣅ౓͕ߴ͘ͳΔΑ͏ֶश • ҙ༁: ΫΤϦͱจষͱͷྨࣅ౓ߦྻʹ͓͚Δର֯੒෼ͷ࠷େԽ •ಉ͡όον಺ͷଞͷࣄྫΛෛྫʹ͢Δ: in-batch negatives ֶशख๏: ରরֶश / Contrastive Pre-training 32 ΫΤϦ ਖ਼ྫจষ Model Model ਖ਼ྫͷຒΊࠐΈΛ͚ۙͮΔ యܕతʹ͸cosྨࣅ౓
33. •ΫΤϦɾจষͷຒΊࠐΈಉ࢜Ͱྨࣅ౓Λܭࢉ • ΫΤϦ—ෛྫͷྨࣅ౓ΑΓΫΤϦ—ਖ਼ྫͷྨࣅ౓͕ߴ͘ͳΔΑ͏ֶश • ҙ༁: ΫΤϦͱจষͱͷྨࣅ౓ߦྻʹ͓͚Δର֯੒෼ͷ࠷େԽ •ಉ͡όον಺ͷଞͷࣄྫΛෛྫʹ͢Δ: in-batch negatives ֶशख๏: ରরֶश / Contrastive Pre-training 33 ΫΤϦ ਖ਼ྫจষ Model Model ਖ਼ྫͷຒΊࠐΈΛ͚ۙͮΔ batch size batch size ΫΤϦͱจষΛผʑʹ encode͢ΔͷͰDual- Encoderͱݺ͹ΕΔ ॏΈ͸ڞ༗
 (ಉ͡Ϟσϧ) యܕతʹ͸cosྨࣅ౓
34. •ਖ਼ྫ͚ͩ֬཰1ͷ֬཰෼෍Λ໨ࢦ͢ •ଛࣦΛԼ͛ΔͨΊʹ͸… • ਖ਼ྫͷྨࣅ౓Λେ͖͘͢Δ • ෛྫͷྨࣅ౓Λখ͘͢͞Δ ଛࣦؔ਺ͷ௚ײతཧղ: ରরֶशଛࣦ 34 ಺ੵʹsoftmaxΛ͔͚ΔͷͰ֬཰෼෍ͱΈͳͤΔ Pcl 1 Pstu … in-batch negatives one-hotͳ෼෍ʹ͚ۙͮΔ hard negatives Query Passage
35. •ϥϕϧ෇͖σʔλͰͷ fi ne-tuningͰ͸σʔλͷ඼࣭͕ॏཁ 1. hard negativesͷར༻ •ͺͬͱݟ͸Θ͔Βͳ͍೉͍͠ࣄྫ • ϞσϧͷදݱྗΛߴΊΔɺඍࡉͳ৘ใΛଊ͑ΒΕΔΑ͏ʹ͢ΔޮՌ •MS-MARCO΍Natural Questions (NQ)Ͱ͸ෛྫΛminingͯ͠ར༻ 2. ରরֶशͱ஌ࣝৠཹͷ૊Έ߹Θͤ •ڭࢣ৴߸ΛΑΓϦονʹ͢ΔͨΊCross-Encoderͷग़ྗΛڭࢣͱͯ͠ར༻ •ରরֶशଛࣦͱ஌ࣝৠཹଛࣦΛ૊Έ߹ΘͤͨϚϧνλεΫֶश ֶशख๏: Fine-tuning 35 ஌ࣝৠཹଛࣦ ରরֶशଛࣦ
36. •Dual-Encoder (DE) ͸ΫΤϦɾจষؒͷ૬ޓ࡞༻ΛߟྀͰ͖ͳ͍ •Cross-Encoder (CE) ͸૬ޓ࡞༻ΛݟΕΔ͕ඇޮ཰ •CEΛ໛฿Ͱ͖ΔΑ͏ʹDEΛֶश͢Ε͹ղܾͰ͸ʁ🧐 •CEͷྨࣅ౓෼෍ʹDEͷग़ྗྨࣅ౓෼෍Λ͚ۙͮΔ ֶशख๏: Cross-Encoder͔Βͷ஌ࣝৠཹ 36 ΫΤϦ+จষ Model ྨࣅ౓είΞ ΫΤϦ จষ Model Model ྨࣅ౓ Dual-Encoder Cross-Encoder
37. •CEͷྨࣅ౓ͷ෼෍ʹDEͷ෼෍Λ͚ۙͮΔ •ରরֶशͱҧ͍ྨࣅ౓ͷ্͛Լ͕͛ෆఆ • negativeͰ΋্͛ΔΑ͏ʹֶश͞Ε͏Δ • ણࡉͳྨࣅ౓ͷؔ܎ΛDEʹڭ͑Δ ଛࣦؔ਺ͷ௚ײతཧղ: ஌ࣝৠཹଛࣦ 37 ಺ੵʹsoftmaxΛ͔͚ΔͷͰ֬཰෼෍ͱΈͳͤΔ ༧ଌ෼෍ CEͷ෼෍ CEͷ෼෍ʹ͚ۙͮΔ Query Passage
38. •ରরֶशͰ͸in-batch negativesΛར༻͢Δ͕false negative͕ൃੜ͠͏Δ • ຊ౰͸ਖ਼ྫ(͚͍ۙͮͨ)ͳͷʹෛྫͱͯ͠ॲཧͯ͠͠·͏ݱ৅ ஌ࣝৠཹͷར఺: false negativeͷ཈੍ 38 1 … ຊ౰͸ਖ਼ղ͕ͩ… ରরֶशଛࣦͷ৔߹ ྨࣅ౓Λա౓ʹԼ͛Δํ޲
 ΁ֶश͞Εͯ͠·͏
39. •ରরֶशͰ͸in-batch negativesΛར༻͢Δ͕false negative͕ൃੜ͠͏Δ • ຊ౰͸ਖ਼ྫ(͚͍ۙͮͨ)ͳͷʹෛྫͱͯ͠ॲཧͯ͠͠·͏ݱ৅ •஌ࣝৠཹͰ͸Cross-Encoderͷग़ྗΛٖࣅతͳڭࢣϥϕϧͱͯ͠ར༻ • false negativeͷ໰୊Λ؇࿨ ஌ࣝৠཹͷར఺: false negativeͷ཈੍ 39 ຊ౰͸ਖ਼ղ͕ͩ
 ྨࣅ౓͕খ͍͞ false negativeͷ໰୊؇࿨
 ྨࣅ౓Λ্͛ΔΑ͏ʹֶशՄೳ ஌ࣝৠཹʹΑΔଛࣦͷ৔߹
40. ଛࣦؔ਺ͷ௚ײతཧղ: ͚͔ۙͮͨͷൺֱ 40 ༧ଌ෼෍ ਖ਼ղ෼෍ 1 … CEͷ෼෍ʹ͚ۙͮΔ DKL one-hotͳ෼෍ʹ͚ۙͮΔ ରরֶशଛࣦ ΄΅͚ۙͮΔ෼෍͕ҟͳΔ͚ͩ
41. Summary: E5ͷ࡞Γํ 41 Un fi ltered
 Corpus Consistency-based
 Filtering CCPairs Masked LM E5-PT Contrastive
 Pre-training E5 Contrastive
 Fine-tuning Labeled Data Knowledge Distillation Reranker encoder-only
42. Summary: E5ͷ࡞Γํ 42 rerankerͷ࡞੒खॱ͸ݪஶ࿦จʹ͸શવॻ͍͍ͯͳ͍ Un fi ltered
 Corpus Consistency-based
 Filtering CCPairs Masked LM E5-PT Contrastive
 Pre-training E5 Contrastive
 Fine-tuning Labeled Data Knowledge Distillation Reranker Retriever 1 Retriever 2 ৄࡉ͸ઌߦݚڀͷSimLM࿦จΛࢀরͷ͜ͱ encoder-only ৭ʑΊͪΌͪ͘Όؤு͍ͬͯΔ
43. •Poolingख๏: Average Pooling (ग़ྗຒΊࠐΈͷฏۉΛऔΔ) • Transformerͷग़ྗ͸ϕΫτϧྻɺ୯ҰϕΫτϧʹ͢ΔͨΊͷૢ࡞͕Pooling Ϟσϧઃఆɾ܇࿅ৄࡉ 43 E5-large pre-training fi ne-tuning #GPUs (V100) 64 8 batch size 32000 256 max length 128 192 #iteration 20000 steps 3 epochs Թ౓ύϥϝʔλ (τ) 0.01 0.01 ଛࣦͷॏΈ (α) N/A 0.2 #hard negatives N/A 7 Dataset CCPairs MS-MARCO, NQ, NLI
44. •Poolingख๏: Average Pooling (ग़ྗຒΊࠐΈͷฏۉΛऔΔ) • Transformerͷग़ྗ͸ϕΫτϧྻɺ୯ҰϕΫτϧʹ͢ΔͨΊͷૢ࡞͕Pooling Ϟσϧઃఆɾ܇࿅ৄࡉ 44 E5-large pre-training fi ne-tuning #GPUs (V100) 64 8 batch size 32000 256 max length 128 192 #iteration 20000 steps 3 epochs Թ౓ύϥϝʔλ (τ) 0.01 0.01 ଛࣦͷॏΈ (α) N/A 0.2 #hard negatives N/A 7 Dataset CCPairs MS-MARCO, NQ, NLI SimCSEͷ0.05ΑΓখ͍͞ Թ౓ύϥϝʔλ͕খ͍͞
 →ͦ͜·Ͱྨࣅ౓෼෍Λ
 ઑΒͤͳͯ͘΋͍͍ ଟ༷ͳσʔλΛֶश
 ͢Δނͷ഑ྀʁ
 (ແཧʹྨࣅ౓ΛߴΊΑ͏
 ͱ͠ͳ͍Α͏ʹ)
45. •ରরֶशͰ࠷΋ॏཁͳϋΠύϥͷҰͭ •Softmaxલͷ஋ΛՃ޻ͯ͠Softmaxޙͷ
 ෼෍ͷܗঢ়ΛมԽͤ͞Δ Թ౓ύϥϝʔλͷิ଍: Πϝʔδ 45 ༧ଌ෼෍ ߴԹ౓ύϥϝʔλ
 ྫ: 10 ௿Թ౓ύϥϝʔλ
 ྫ: 0.01 ෼෍͕ฏୱʹ ෼෍͕ٸफ़ʹ
46. •ରরֶशͰ࠷΋ॏཁͳϋΠύϥͷҰͭ •Softmaxલͷ஋ΛՃ޻ͯ͠Softmaxޙͷ
 ෼෍ͷܗঢ়ΛมԽͤ͞Δ Թ౓ύϥϝʔλͷิ଍: Πϝʔδ 46 ༧ଌ෼෍ ߴԹ౓ύϥϝʔλ
 ྫ: 10 ௿Թ౓ύϥϝʔλ
 ྫ: 0.01 ෼෍͕ฏୱʹ ෼෍͕ٸफ़ʹ Ϟσϧ͕ؤுͬͯ෼෍Λ
 ઑΒͤΔඞཁ͋Γ Ϟσϧ͕ؤுΒͳͯ͘΋
 ෼෍͕ઑΔ
47. •ςΩετຒΊࠐΈධՁ༻ͷϕϯνϚʔΫΛར༻ BEIR •19ݸͷσʔληοτΛؚΉ৘ใݕࡧλεΫʹಛԽͨ͠ϕϯνϚʔΫ •nDCG@10ͰධՁ MTEB •6λεΫɾ56σʔληοτ͔ΒͳΔ൚༻ϕϯνϚʔΫ •ϦʔμʔϘʔυ΋੔උ͞Ε͓ͯΓۙ೥׆ൃʹར༻͞Ε͍ͯΔ ධՁ࣮ݧ 47
48. •SimCSE΍Contrieverͱ͍ͬͨطଘख๏Λ্ճΔੑೳ • E5͸͜ΕΒͷख๏ΑΓσʔληοτ࡞੒
 Λؤு͍ͬͯΔ •Contrastive Pre-trainingͷΈ΋͔ͳΓڧ͍ • ςΩετຒΊࠐΈͷͨΊͷࣄલֶश͕ޮՌత •E5ͷ fi ne-tuningͷσʔληοτ͸ݶఆత • طଘख๏ʹෛ͚ͯΔλεΫ΋ׂͱ͋Δ • ଟ༷Խɾେن໛Խ͢Δ͜ͱͰੑೳ޲্Λ
 ໨ࢦͤͦ͏ ࣮ݧ݁Ռ: BEIR🍺 48 ද͸ॾʑলུͨ݁͠Ռɺৄࡉͳ࣮ݧ݁Ռ͸ݪஶ࿦จΛࢀরͷ͜ͱ Avg. BM25 41.7 SimCSEbase 20.3 Contrieverunsup 36.0 E5-PTlarge 44.2 Contrieversup 46.6 ColBERT 44.4 E5large 50.0 ڭࢣͳ͠ ڭࢣ͋Γ
49. •ଟ༷ͳσʔληοτͰฏۉͯ͠ߴ͍ੑೳ ࣮ݧ݁Ռ: MTEB 49
50. •Contrastive Pre-trainingʹ͓͚ΔόοναΠζΛม͑ͯBEIRͰੑೳධՁ •όοναΠζΛେ͖͘͢Δ΄Ͳੑೳ޲্ •🧐16kʹϐʔΫ͕͋ΔՄೳੑ͕ͳ͘΋ͳ͍ • ޙଓݚڀͷGTEͰ͸8k͔16k͕࠷ળ ෼ੳ: όοναΠζ͕ٴ΅͢Өڹ 50
51. • fi ne-tuningʹ࢖͏σʔληοτΛม͑ͯMTEBͰੑೳධՁ •Contrastive Pre-training͚ͩΑΓ fi ne-tuningͨ͠ํ͕ฏۉੑೳ͸ߴ͍ • ͕ɺNLIσʔληοτ͚ͩͰ fi ne-tuning͢Δͱݕࡧੑೳ͸Ή͠Ζ௿Լ • ݕࡧ+QAͰੑೳ͕͔ͳΓ޲্͢Δ͕ɺSTSͷੑೳ࠷େԽʹ͸NLI͕ඞཁ •શͯΛࠞͥͯ࢖͏͜ͱͰฏۉͯ͠࠷ߴੑೳɺଟ༷Խ͕େࣄ ෼ੳ: fi ne-tuningσʔληοτͷଟ༷ੑ 51 NLIͰtuning͞ΕͨจຒΊࠐΈϞσϧ͸ݕࡧ༻్ʹ͸޲͔ͳ͍Ͱ͢(ࢲݟ)
52. •ϑΟϧλϦϯάͷ༗ແΛม͑ͯੑೳൺֱ • σʔλن໛ʹΑΔӨڹ΋ݟΔͨΊখن໛σʔλͰͷ࣮ݧ΋࣮ࢪ •σʔλن໛ʹΑΒͣϑΟϧλϦϯάʹΑΓੑೳ޲্ • ಛʹখن໛σʔλͰϑΟϧλϦϯάͷޮՌ͕େ͖͍ͱݴ͑Δ ෼ੳ: ϑΟϧλϦϯάͷॏཁੑ 52
53. •େن໛ͳࣄલରরֶशʹΑΓߏங͞ΕͨςΩετຒΊࠐΈϞσϧE5ΛఏҊ •൒ߏ଄ԽσʔλͱϑΟϧλϦϯάΛ༻͍ͨऑڭࢣ͋ΓσʔληοτΛߏங •2ஈ֊ͷֶशख๏Λ࠾༻ 1. Contrastive Pre-training 2. ରরֶशͱ஌ࣝৠཹΛ૊Έ߹ΘͤͨFine-tuning ؾʹͳͬͨ͜ͱ •͔ͳΓςΫχΧϧϨϙʔτ෩ͳ࿦จͰResearch Question͕͋·Γͳ͍ • ஌ࣝৠཹͷ༗༻ੑͷݕূ͸ͳ͠ɾhard negativeΛͲ͏࡞Δ͔΋ᐆດ •pre fi x͸༗༻ͦ͏͕ͩޮՌ͕ݕূ͞Ε͍ͯͳ͍ ·ͱΊ 53
54. •E5͸ઌߦݚڀʹ͋ͨΔSimLMͷ஌ࣝΛલఏͱ͍ͯ͠Δ෦෼͕͋Δ • ಛʹRerankerͷ࡞Γํ΍negative miningʹ͍ͭͯ͸SimLM࿦จ͕ৄࡉ •ޙଓݚڀͷGTE΋ษڧʹͳΔͷͰ߹ΘͤͯͲ͏ͧ •pre-trainingʹrerankerΛ࢖Θͳ͍ͷ͸ίετͷ໰୊ͬΆ͍ • ಉ༷ʹcontrastive pre-trainingͰ͸hard negative mining΋͍ͯ͠ͳ͍ •E5ͷଟݴޠ൛Ͱ͋ΔMultilingual E5͸ඇৗʹڧ͘ϕʔεϥΠϯͱͯ͠ྑ޷ • ೔ຊޠʹ͓͚Δݕࡧੑೳɾݕࡧ֦ுੜ੒ͰͷධՁ࿦จ͸ͪ͜Β ิ଍ 54