Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

NLP2024 招待論文セッション: 定義文を用いた文埋め込み構成法

Hayato Tsukagoshi
September 29, 2024
34

NLP2024 招待論文セッション: 定義文を用いた文埋め込み構成法

言語処理学会第30回年次大会(NLP2024)の招待論文セッションでの発表「定義文を用いた文埋め込み構成法」の発表資料です。

https://www.anlp.jp/proceedings/annual_meeting/2024/#invitedpapers

Hayato Tsukagoshi

September 29, 2024
Tweet

Transcript

  1. • ࣗવݴޠจͷີϕΫτϧදݱ • ϕΫτϧͷڑ཭͕จͷҙຯͷۙ͞Λදݱ จຒΊࠐΈ / Sentence Embedding 2 ͜Ͳ΋͕Ոʹ޲͔͍ͬͯΔɻ

    ͜Ͳ΋ֶ͕ߍ͔ΒՈʹ޲͔͍ͬͯΔɻ ͜Ͳ΋͕ਤॻؗʹ͍Δɻ ͜Ͳ΋͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ
  2. ࣗવݴޠਪ࿦ (NLI) λεΫ • จϖΞͷҙຯؔ܎Λ༧ଌ • ؚҙɾໃ६ɾͦͷଞͷࡾ஋෼ྨ SBERT • NLI෼ྨ༻ͷ૚ΛBERTʹ௥Ճ

    • BERTΛ100ສจϖΞͰ fi ne-tuning ୅දతͳطଘख๏: Sentence-BERT (SBERT) 3 จB จA BERT BERT ໃ६ ؚҙ ͦͷଞ ϥϕϧ༧ଌ૚ Pooling Pooling
  3. ఆٛจ w|V| w1 w2 w3 ... BERT ୯ޠ༧ଌ૚ Pooling •

    ఆٛจˠ୯ޠͷ༧ଌλεΫͰ܇࿅ • จͷߏ੒తͳҙຯΛཧղ • ఆٛจˠ୯ޠ༧ଌ૚ʹ͸
 ࣄલֶश(MLM)࣌ͷ૚Λར༻ • ࣄલֶशͰ֫ಘ͞ΕͨҙຯۭؒΛ׆༻ DefSent: ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ 6 ৽نύϥϝʔλ͕ͳ͍ →ֶश΋ޮ཰త
  4. • Top10ਖ਼ղ཰50%Ҏ্Ͱ༧ଌՄೳʹ ఆٛจˠ୯ޠ༧ଌλεΫ: ධՁ࣮ݧ 9 Ϟσϧ Pooling MRR Top1 Top3

    Top10 BERT-
 base CLS 30.1 20.4 35.7 53.2 Mean 29.3 19.5 35.0 52.6 Max 27.4 17.6 32.5 50.4 RoBERTa-
 base CLS 32.3 21.8 38.4 56.8 Mean 31.8 21.4 37.8 56.4 Max 29.5 19.8 34.9 53.0
  5. • ॳݟͷ(ఆٛ)จʹ͍ͭͯ΋ଥ౰ͳ୯ޠΛ༧ଌ ఆٛจˠ୯ޠ༧ଌλεΫ: ఆੑධՁ 10 ਖ਼ղ୯ޠ ఆٛจ ༧ଌ୯ޠ cost be

    expensive for (someone) cost charge pay preserve prevent (food) from rotting preserve keep spoil chief a person who is in charge leader boss master - not good bad poor wrong
  6. • ʮϞσϧ͕ܭࢉͨ͠ྨࣅ౓ʯͱ
 ʮਓؒධՁʯͱͷ૬ؔ܎਺ΛධՁ • ҙຯΛଊ͑ΔೳྗΛଌΔ • จຒΊࠐΈධՁͰ͸ڭࢣͳ͠ઃఆ • STSσʔληοτͰͷ܇࿅ͳ͠ •

    ૬ؔ܎਺͕ߴ͍ˠྑ͍จຒΊࠐΈ STS (Semantic Textual Similarity) λεΫ 11 จA จB จຒΊࠐΈϞσϧ ਓखධՁͱͷ
 ૬ؔ܎਺ͰධՁ จྨࣅ౓
  7. • SBERTͱಉ౳ఔ౓ͷੑೳ • ܇࿅σʔλྔ͸SBERTͷ1/20ఔ౓ (܇࿅࣌ؒ5෼΄Ͳ) STS: ࣮ݧ݁Ռ / BERT-base 12

    Ϟσϧ STS12 STS13 STS14 STS15 STS16 STS-B SICK Avg. GloVe 55.1 70.7 59.7 68.3 63.7 58.0 53.8 61.3 FTͳ͠ 21.5 32.1 21.3 37.9 44.3 20.3 42.4 31.4 SBERT 69.8 72.5 70.4 78.0 73.5 76.0 72.3 73.2 DefSent 67.3 81.8 71.8 78.2 76.9 77.0 73.5 75.2
  8. • ͪ͜Β΋SBERTͱಉ౳ఔ౓ͷੑೳ SentEval: ࣮ݧ݁Ռ / BERT-base 14 Ϟσϧ MR CR

    SUBJ MPQA SST-2 TREC MRPC Avg. GloVe 77.3 78.3 91.2 87.9 80.2 83.0 72.9 81.5 FTͳ͠ 81.8 87.9 95.5 88.2 86.5 91.0 72.3 86.2 SBERT 82.7 89.4 93.4 89.7 88.2 85.9 76.2 86.5 DefSent 81.8 88.0 94.9 89.9 86.3 90.1 75.4 86.6
  9. ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳ 17 Semantic Textual Similarity (STS) ᶃ จͷιʔε ᶄ จϖΞͷද૚తྨࣅ౓

    ൺֱ؍఺ SentEval ᶅ ԼྲྀλεΫ͝ͱͷੑೳ ᶆ ݴޠֶత৘ใͷ෼ྨੑೳ ൺֱ؍఺ ද૚తྨࣅ౓ SBERT DefSent •܇࿅σʔλʹ͍ۙจͷํ͕
 ͏·͘ྨࣅ౓ΛଌΕΔ •SBERT͸ද૚త৘ใ͕গͳ͍ •DefSent͸੍࣌ͳͲද૚త৘ใ͕ ଟ͘ɺϑϨʔζͷߏ੒͕ಘҙ ߴ ௿
  10. • DefSentͱSBERTΛ
 ૊Έ߹ΘͤͯධՁ • ϚϧνλεΫֶश΍
 ຒΊࠐΈͷฏۉɾ݁߹ • SBERT→DefSent͸ڧ͘
 DefSent→SBERT͸ऑ͍ •

    ഁ໓త๨٫ͷӨڹΛࣔࠦ • Average͕୯७&ߴੑೳ DefSentͱSBERTͷ౷߹ 18 BERT-base STS SentEval SBERT 73.2 86.5 DefSent 75.2 86.6 SBERT→DefSent 78.5 86.8 DefSent→SBERT 72.9 86.1 ϚϧνλεΫ 72.9 86.2 Average 77.8 87.5 Concat 76.0 87.9
  11. • ఆٛจˠ୯ޠ༧ଌʹΑΔ
 จຒΊࠐΈख๏ DefSent ΛఏҊ • SBERTͷ1/20ఔ౓ͷσʔλͰಉ౳ੑೳ • ෼ੳͷ݁Ռੑ࣭ͷҧ͍Λ໌Β͔ʹ •

    e.g. ද૚తྨࣅ౓ʹΑΔӨڹ • ౷߹: ୯७ͳฏۉ͕ߴੑೳ ͓ΘΓʹ 19 ఆٛจ w|V| w1 w2 w3 ... BERT ୯ޠ༧ଌ૚ Pooling