Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SNLP2019

Ayana Niwa
September 25, 2019

 SNLP2019

第11回最先端NLP勉強会 発表資料

Ayana Niwa

September 25, 2019
Tweet

More Decks by Ayana Niwa

Other Decks in Research

Transcript

  1. Probing for Semantic Classes: Diagnosing the Meaning Content of Word

    Embeddings Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, Hinrich Schutze ACL2019 11 NLP  2019/09/28    <[email protected]>
  2. Outline 2 H9P=O8 O8 "!PC     

       $@K !.S%,'3 F5T&*'$:I P=O8$.S%,'PV ! R<72$PC H/+%)- >#"!.S O8! H9E69NR<72;BJ?Q1L  >#".S(rare senses)P=O8APM0 " 4D('% UG
  3. Background 3 # " !Word2VecGrove  #"NLPIR* - ELMoBERT 

    $'+ ) %" full-space , % &   L  "$( "   # &!
  4. Background 4 #  "@!Semantic class>*( S-class , "@=6:-&C?4=6 

    B $8 SEMCAT, HyperLex… =1;+=62%A → 0# .9/ - WIKI-PSE(WIKIpedia-based resource for Probing Semantics in word Embeddings) - "@=1;+sense embedding'7) “Lamb” Food3< Living-thing5< "@ !
  5. Background 5 #   Apple Apple  Apple 

    + Word embeddingsense embedding  Arora et al. (2018) Arora et al. (2018) Word embedding sparse coding WIKI-PSEsense embedding  Word embedding Sense embedding Sense embedding
  6. WIKI-PSE Resource 6 Wikipedia .!2,$#  .! ()  

    S-class 113 FIGER types%+134"10 person/authorperson/politician… à person -S-class      *'.!/.! & - S-class0
  7. WIKI-PSE Resource 7  "$%  %2& +:! 343,000-6.-S-class#5'5000;-!% 75,.6,.83

    +*1! /4   44,250-   -S-class  - S-class )0organization (9food @apple@ – food @apple@ – organization   
  8.   8 Word embedding (word) @apple@ WIKI-PSE  

    word embedding Sense embedding @apple@ @apple@ - food @apple@ - organization  @  word/S-class Uniform sum (unifΣ) !" Weighted sum (wghtΣ) !"  # = % " !" #&" # Aggregated word embedding #&"  Sense (word/S-class) embeddings word, unifΣ, wghtΣ
  9. Experiments 9 Problem settings   SkipGram Structured SkipGram 

    WIKI-PSE    (LR) # (MLP k*)KNN) 1. word % word embedding 2. unifΣ sense embedding  " 3. wghtΣ sense embedding ! $'& (" Word embedding
  10. Experiments 10 Probing Task 1S-class Prediction  S-class  

    @apple@ +food ∩ +organization ∩ -event S-class  
  11. Experiments 11 MLP > LRKNN 8&!S-class ."7;5   

     KNN )#, *:3, <(3-/ unifΣ > word > wghtΣ 'rare sense 0$  97< à Rare sense42%+6 unifΣ Probing Task 1S-class Prediction F11 4
  12. Experiments 12 Probing Task 1S-class Prediction    

     unifΣ  $    #rare sense    (13,000) F1! "
  13. Experiments 13 Probing Task 1S-class Prediction * #4(!  Recall

    ) S-class&  sense embedding+%3 Rare S-class/"12 %"0, .$- Recall     Dominance S-class' Recall 
  14. Experiments 14 Probing Task 1S-class Prediction /&)5. !"$'  ,(20-

       3+1(#41(       Personlocation,(20* S-class !"6 20% Recall Recall #4 ,(20 Typicality Recall 
  15. Experiments 15  "    Probing Task 2Ambiguity

    Prediction ! L2 SSKIP$" LR / KNN / MLP ! ! unifΣ > word > wghtΣ KNN    à    % FREQUENCY(Baseline)  # &LR
  16. NLP Application Experiments 16 wghtΣ > word >(=) unifΣ <--

    Probing task #!! -the U.S. Attorney’s Office announced Friday → location Common S-classtime Rare S-class location Friday mountain unifΣ  ( !+  entity mention  MC ), CRMR $)& SUBJ  %*' MRPC "   S-class 
  17. Summary & Conclusion 17 * $2"(3 WIKI-PSE'.  1)/& 0+$2/&

    1, 1)/&%  # 0+$2- a a e Rare sense e harder NLP Rare sense !# e