NLP2024 招待論文セッション: 定義文を用いた文埋め込み構成法

言語処理学会第30回年次大会(NLP2024)の招待論文セッションでの発表「定義文を用いた文埋め込み構成法」の発表資料です。

https://www.anlp.jp/proceedings/annual_meeting/2024/#invitedpapers

Transcript

1. ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ ௩ӽॣɹ࡫໺ྒྷฏɹ෢ాߒҰ ݴޠॲཧֶձୈ30ճ೥࣍େձ (NLP 2024) ট଴࿦จ 2024/03/11 ໊ݹ԰େֶେֶӃ৘ใֶݚڀՊ
2. • ࣗવݴޠจͷີϕΫτϧදݱ • ϕΫτϧͷڑ཭͕จͷҙຯͷۙ͞Λදݱ จຒΊࠐΈ / Sentence Embedding 2 ͜Ͳ΋͕Ոʹ޲͔͍ͬͯΔɻ ͜Ͳ΋ֶ͕ߍ͔ΒՈʹ޲͔͍ͬͯΔɻ ͜Ͳ΋͕ਤॻؗʹ͍Δɻ ͜Ͳ΋͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ
3. ࣗવݴޠਪ࿦ (NLI) λεΫ • จϖΞͷҙຯؔ܎Λ༧ଌ • ؚҙɾໃ६ɾͦͷଞͷࡾ஋෼ྨ SBERT • NLI෼ྨ༻ͷ૚ΛBERTʹ௥Ճ • BERTΛ100ສจϖΞͰ fi ne-tuning ୅දతͳطଘख๏: Sentence-BERT (SBERT) 3 จB จA BERT BERT ໃ६ ؚҙ ͦͷଞ ϥϕϧ༧ଌ૚ Pooling Pooling
4. • NLIσʔληοτ͸ଟ͕͘ӳޠ • NLIϕʔεख๏͸ӳޠҎ֎ͷݴޠ΁ͷద༻͕೉͍͠ • طଘͷݴޠࢿݯͰจຒΊࠐΈϞσϧΛ֫ಘͰ͖ͳ͍͔ʁ → ଟ͘ͷݴޠͰ੔උ͞Ε͍ͯΔ୯ޠࣙॻʹண໨ • ࣙॻͰ͸୯ޠͱͦͷఆٛจ͕ಉ͡ҙຯΛද͢ → ୯ޠͱఆٛจͷҙຯ(ϕΫτϧ)͕ରԠ͢ΔΑ͏ʹ܇࿅ طଘख๏ͷ໰୊఺ɾղܾࡦ 4
5. • จຒΊࠐΈ • DefSent: ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ • ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳ ໨࣍ 5
6. ఆٛจ w|V| w1 w2 w3 ... BERT ୯ޠ༧ଌ૚ Pooling • ఆٛจˠ୯ޠͷ༧ଌλεΫͰ܇࿅ • จͷߏ੒తͳҙຯΛཧղ • ఆٛจˠ୯ޠ༧ଌ૚ʹ͸
 ࣄલֶश(MLM)࣌ͷ૚Λར༻ • ࣄલֶशͰ֫ಘ͞ΕͨҙຯۭؒΛ׆༻ DefSent: ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ 6 ৽نύϥϝʔλ͕ͳ͍ →ֶश΋ޮ཰త
7. • ఆٛจˠ୯ޠ༧ଌλεΫ • STS (Semantic Textual Similarity) λεΫ • SentEval ධՁ࣮ݧ 7
8. σʔληοτ • Oxford Dictionaryͷ୯ޠ—ఆٛจͷϖΞ5.4ສ݅ • Ϟσϧͷޠኮʹؚ·ΕΔ୯ޠ(ͱఆٛจ)ͷΈར༻ ධՁࢦඪ • ฏۉٯॱҐ (Mean Reciprocal Rank: MRR) • Top1, Top3, Top10 ਖ਼ղ཰ ఆٛจˠ୯ޠ༧ଌλεΫ: ධՁ࣮ݧ 8
9. • Top10ਖ਼ղ཰50%Ҏ্Ͱ༧ଌՄೳʹ ఆٛจˠ୯ޠ༧ଌλεΫ: ධՁ࣮ݧ 9 Ϟσϧ Pooling MRR Top1 Top3 Top10 BERT-
 base CLS 30.1 20.4 35.7 53.2 Mean 29.3 19.5 35.0 52.6 Max 27.4 17.6 32.5 50.4 RoBERTa-
 base CLS 32.3 21.8 38.4 56.8 Mean 31.8 21.4 37.8 56.4 Max 29.5 19.8 34.9 53.0
10. • ॳݟͷ(ఆٛ)จʹ͍ͭͯ΋ଥ౰ͳ୯ޠΛ༧ଌ ఆٛจˠ୯ޠ༧ଌλεΫ: ఆੑධՁ 10 ਖ਼ղ୯ޠ ఆٛจ ༧ଌ୯ޠ cost be expensive for (someone) cost charge pay preserve prevent (food) from rotting preserve keep spoil chief a person who is in charge leader boss master - not good bad poor wrong
11. • ʮϞσϧ͕ܭࢉͨ͠ྨࣅ౓ʯͱ
 ʮਓؒධՁʯͱͷ૬ؔ܎਺ΛධՁ • ҙຯΛଊ͑ΔೳྗΛଌΔ • จຒΊࠐΈධՁͰ͸ڭࢣͳ͠ઃఆ • STSσʔληοτͰͷ܇࿅ͳ͠ • ૬ؔ܎਺͕ߴ͍ˠྑ͍จຒΊࠐΈ STS (Semantic Textual Similarity) λεΫ 11 จA จB จຒΊࠐΈϞσϧ ਓखධՁͱͷ
 ૬ؔ܎਺ͰධՁ จྨࣅ౓
12. • SBERTͱಉ౳ఔ౓ͷੑೳ • ܇࿅σʔλྔ͸SBERTͷ1/20ఔ౓ (܇࿅࣌ؒ5෼΄Ͳ) STS: ࣮ݧ݁Ռ / BERT-base 12 Ϟσϧ STS12 STS13 STS14 STS15 STS16 STS-B SICK Avg. GloVe 55.1 70.7 59.7 68.3 63.7 58.0 53.8 61.3 FTͳ͠ 21.5 32.1 21.3 37.9 44.3 20.3 42.4 31.4 SBERT 69.8 72.5 70.4 78.0 73.5 76.0 72.3 73.2 DefSent 67.3 81.8 71.8 78.2 76.9 77.0 73.5 75.2
13. • ςΩετ෼ྨͳͲͷλεΫ͕
 ू·ͬͨϕϯνϚʔΫ • จຒΊࠐΈʹجͮ͘෼ྨثΛ܇࿅
 →จຒΊࠐΈͷ࣭Λ෼ྨੑೳͰධՁ • จຒΊࠐΈϞσϧͷύϥϝʔλ͸ݻఆ • ෼ྨੑೳ͕ߴ͍ˠྑ͍จຒΊࠐΈ SentEval 13 จ ෼ྨੑೳ͔Β
 จຒΊࠐΈͷ඼࣭ΛධՁ จຒΊࠐΈϞσϧ ෼ྨث
14. • ͪ͜Β΋SBERTͱಉ౳ఔ౓ͷੑೳ SentEval: ࣮ݧ݁Ռ / BERT-base 14 Ϟσϧ MR CR SUBJ MPQA SST-2 TREC MRPC Avg. GloVe 77.3 78.3 91.2 87.9 80.2 83.0 72.9 81.5 FTͳ͠ 81.8 87.9 95.5 88.2 86.5 91.0 72.3 86.2 SBERT 82.7 89.4 93.4 89.7 88.2 85.9 76.2 86.5 DefSent 81.8 88.0 94.9 89.9 86.3 90.1 75.4 86.6
15. • จຒΊࠐΈ • DefSent: ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ • ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳɾ౷߹ ໨࣍ 15
16. • ػցֶशϞσϧ͸ֶशσʔλʹΑΓৼΔ෣͍Λม͑Δ • ಉ౳ੑೳͷϞσϧͰ΋ಘҙɾෆಘҙ͕͋Δ͸ͣ • ҟͳΔϞσϧͷ౷߹͸ੑೳ޲্΋ૂ͑Δ (Ξϯαϯϒϧ) • DefSent͕SBERT͸Ϟσϧ͕ಉ͡ɾڭࢣ৴߸͕ҟͳΔ • จຒΊࠐΈͷੑ࣭ʹ͸ҧ͍͕͋Δͷ͔ʁ • ૊Έ߹ΘͤΔ͜ͱͰ͞ΒʹੑೳվળͰ͖Δ͔ʁ ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳɾ౷߹ 16
17. ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳ 17 Semantic Textual Similarity (STS) ᶃ จͷιʔε ᶄ จϖΞͷද૚తྨࣅ౓ ൺֱ؍఺ SentEval ᶅ ԼྲྀλεΫ͝ͱͷੑೳ ᶆ ݴޠֶత৘ใͷ෼ྨੑೳ ൺֱ؍఺ ද૚తྨࣅ౓ SBERT DefSent •܇࿅σʔλʹ͍ۙจͷํ͕
 ͏·͘ྨࣅ౓ΛଌΕΔ •SBERT͸ද૚త৘ใ͕গͳ͍ •DefSent͸੍࣌ͳͲද૚త৘ใ͕ ଟ͘ɺϑϨʔζͷߏ੒͕ಘҙ ߴ ௿
18. • DefSentͱSBERTΛ
 ૊Έ߹ΘͤͯධՁ • ϚϧνλεΫֶश΍
 ຒΊࠐΈͷฏۉɾ݁߹ • SBERT→DefSent͸ڧ͘
 DefSent→SBERT͸ऑ͍ • ഁ໓త๨٫ͷӨڹΛࣔࠦ • Average͕୯७&ߴੑೳ DefSentͱSBERTͷ౷߹ 18 BERT-base STS SentEval SBERT 73.2 86.5 DefSent 75.2 86.6 SBERT→DefSent 78.5 86.8 DefSent→SBERT 72.9 86.1 ϚϧνλεΫ 72.9 86.2 Average 77.8 87.5 Concat 76.0 87.9
19. • ఆٛจˠ୯ޠ༧ଌʹΑΔ
 จຒΊࠐΈख๏ DefSent ΛఏҊ • SBERTͷ1/20ఔ౓ͷσʔλͰಉ౳ੑೳ • ෼ੳͷ݁Ռੑ࣭ͷҧ͍Λ໌Β͔ʹ • e.g. ද૚తྨࣅ౓ʹΑΔӨڹ • ౷߹: ୯७ͳฏۉ͕ߴੑೳ ͓ΘΓʹ 19 ఆٛจ w|V| w1 w2 w3 ... BERT ୯ޠ༧ଌ૚ Pooling