SNLP2019

第11回最先端NLP勉強会 発表資料

1. Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, Hinrich Schutze ACL2019 11 NLP  2019/09/28 
      <ayana.niwa@nlp.c.titech.ac.jp>
2. Outline 2 H9P=O8 O8 "!PC   
    
    
    $@K !.S%,'3 F5T&
   *'$:I P=O8$.S%,'PV ! R<72$PC H/+%)- >#"!.S O8! H9E69NR<72;BJ?Q1L  >#".S(rare senses)P=O8APM0 " 4D('% UG
3. Background 3 # " !Word2VecGrove  #"
   NLPIR* - ELMoBERT  $'+
   ) %" full-space , % &   L  "$( "   #
    &!
4. Background 4 # 
    "@!Semantic class
   >*( S-class , "@=6
   :-&C?4=6  B $8 SEMCAT, HyperLex
   … =1;+=62%A → 0# .9/ - WIKI-PSE(WIKIpedia-based resource for Probing Semantics in word Embeddings) - "@=1;+sense embedding'7) “Lamb” Food3< Living-thing5< "@ !
5. Background 5 #   Apple Apple
    Apple
    + Word embeddingsense embedding  Arora et al. (2018) Arora et al. (2018) Word embedding sparse coding WIKI-PSE
   sense embedding  Word embedding Sense embedding Sense embedding
6. WIKI-PSE Resource 6 
   Wikipedia .!2,$#  .! ()   S-class 113 FIGER types%+134"10 person/author
   person/politician… à person
   -S-class  
    
   
    *'.!/.! & - S-class0
7. WIKI-PSE Resource 7
    "$%  %2& +:! 343,000-6.-S-class#
   5'5000;-!% 75,.6,.83 +*1! /4   44,250- 
    
   -S-class
     - S-class )0organization (9food @apple@ – food @apple@ – organization 
    
8.  
    8 Word embedding (word) @apple@ WIKI-PSE   word embedding Sense embedding @apple@ @apple@ - food @apple@ - organization  @  word/S-class Uniform sum (unifΣ) !"
    Weighted sum (wghtΣ) !"
     # = % " !" #&" #
   Aggregated word embedding #&"
   Sense (word/S-class) embeddings word, unifΣ, wghtΣ
9. Experiments 9 Problem settings   SkipGram Structured SkipGram 
    WIKI-PSE    (LR) # 
   (MLP k*)KNN) 1. word % word embedding 2. unifΣ sense embedding  " 3. wghtΣ sense embedding ! $'& (" Word embedding
10. Experiments 10 Probing Task 1
    S-class Prediction  S-class   @apple@ +food ∩ +organization ∩ -event S-class  
11. Experiments 11 MLP > LR
    KNN 8&!S-class ."7;5     KNN )#, *:3,
    <(3-/ unifΣ > word > wghtΣ 'rare sense 0$  97< à Rare sense42%+6 unifΣ Probing Task 1S-class Prediction
    F11 4
12. Experiments 12 Probing Task 1S-class Prediction  
       unifΣ  $
        #rare sense    (13,000)
    F1! "
13. Experiments 13 Probing Task 1
    S-class Prediction * #4(!  Recall ) S-class&  sense embedding+%3 Rare S-class/"12 %"0, .$- Recall     Dominance S-class'
    Recall 
    
14. Experiments 14 Probing Task 1
    S-class Prediction /&)5. !"$'  ,(20-    3+1(#41(       Personlocation,(20* S-class !"6 20% Recall Recall #4 ,(20 Typicality
    Recall
    
15. Experiments 15  "    Probing Task 2
    Ambiguity Prediction ! L2 SSKIP$" LR / KNN / MLP ! ! unifΣ > word > wghtΣ KNN    à    %
    FREQUENCY(Baseline)  # &LR
16. NLP Application Experiments 16 wghtΣ > word >(=) unifΣ <-- Probing task #!! -the U.S. Attorney’s Office announced Friday → location Common S-classtime Rare S-class location Friday mountain unifΣ  ( !+  entity mention  MC ), CR
    MR $)& SUBJ 
    %*' MRPC "   S-class 
17. Summary & Conclusion 17 * $2"(3 WIKI-PSE'.
    1)/& 0+$2/& 1, 1)/&%  # 0+$2- a a e Rare sense e harder NLP Rare sense !# e