Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SNLP2019
Search
Ayana Niwa
September 25, 2019
Research
1
460
SNLP2019
第11回最先端NLP勉強会 発表資料
Ayana Niwa
September 25, 2019
Tweet
Share
More Decks by Ayana Niwa
See All by Ayana Niwa
A Quick Overview to Unlock the Potential of LLMs through Prompt Engineering
ayaniwa
0
100
Learning To Retrieve Prompts for In-Context Learning
ayaniwa
0
960
UnNatural Language Inference
ayaniwa
0
380
Trends in Natural Language Processing at NeurIPS 2019.
ayaniwa
8
4.2k
Other Decks in Research
See All in Research
機械学習でヒトの行動を変える
hiromu1996
1
200
機械学習による言語パフォーマンスの評価
langstat
6
680
論文紹介/Expectations over Unspoken Alternatives Predict Pragmatic Inferences
chemical_tree
1
250
LLM based AI Agents Overview -What, Why, How-
masatoto
2
590
言語と数理の交差点:テキストの埋め込みと構造のモデル化 (IBIS 2024 チュートリアル)
yukiar
2
190
Practical The One Person Framework
asonas
1
1.4k
Tietovuoto Social Design Agency (SDA) -trollitehtaasta
hponka
0
2.1k
TransformerによるBEV Perception
hf149
1
380
SNLP2024:Planning Like Human: A Dual-process Framework for Dialogue Planning
yukizenimoto
1
310
LLM時代の半導体・集積回路
kentaroy47
1
470
Generative Predictive Model for Autonomous Driving 第61回 コンピュータビジョン勉強会@関東 (後編)
kentosasaki
0
200
授業評価アンケートのテキストマイニング
langstat
1
350
Featured
See All Featured
Designing the Hi-DPI Web
ddemaree
280
34k
GraphQLとの向き合い方2022年版
quramy
43
13k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.1k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.2k
The Cost Of JavaScript in 2023
addyosmani
45
6.6k
Building Adaptive Systems
keathley
38
2.2k
The Power of CSS Pseudo Elements
geoffreycrofte
72
5.3k
The Cult of Friendly URLs
andyhume
78
6k
4 Signs Your Business is Dying
shpigford
180
21k
What's in a price? How to price your products and services
michaelherold
243
12k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
RailsConf 2023
tenderlove
29
880
Transcript
Probing for Semantic Classes: Diagnosing the Meaning Content of Word
Embeddings Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, Hinrich Schutze ACL2019 11 NLP 2019/09/28 <
[email protected]
>
Outline 2 H9P=O8 O8 "!PC
$@K !.S%,'3 F5T&*'$:I P=O8$.S%,'PV ! R<72$PC H/+%)- >#"!.S O8! H9E69NR<72;BJ?Q1L >#".S(rare senses)P=O8APM0 " 4D('% UG
Background 3 # " !Word2VecGrove #"NLPIR* - ELMoBERT
$'+ ) %" full-space , % & L "$( " # &!
Background 4 # "@!Semantic class>*( S-class , "@=6:-&C?4=6
B $8 SEMCAT, HyperLex… =1;+=62%A → 0# .9/ - WIKI-PSE(WIKIpedia-based resource for Probing Semantics in word Embeddings) - "@=1;+sense embedding'7) “Lamb” Food3< Living-thing5< "@ !
Background 5 # Apple Apple Apple
+ Word embeddingsense embedding Arora et al. (2018) Arora et al. (2018) Word embedding sparse coding WIKI-PSEsense embedding Word embedding Sense embedding Sense embedding
WIKI-PSE Resource 6 Wikipedia .!2,$# .! ()
S-class 113 FIGER types%+134"10 person/authorperson/politician… à person -S-class *'.!/.! & - S-class0
WIKI-PSE Resource 7 "$% %2& +:! 343,000-6.-S-class#5'5000;-!% 75,.6,.83
+*1! /4 44,250- -S-class - S-class )0organization (9food @apple@ – food @apple@ – organization
8 Word embedding (word) @apple@ WIKI-PSE
word embedding Sense embedding @apple@ @apple@ - food @apple@ - organization @ word/S-class Uniform sum (unifΣ) !" Weighted sum (wghtΣ) !" # = % " !" #&" # Aggregated word embedding #&" Sense (word/S-class) embeddings word, unifΣ, wghtΣ
Experiments 9 Problem settings SkipGram Structured SkipGram
WIKI-PSE (LR) # (MLP k*)KNN) 1. word % word embedding 2. unifΣ sense embedding " 3. wghtΣ sense embedding ! $'& (" Word embedding
Experiments 10 Probing Task 1S-class Prediction S-class
@apple@ +food ∩ +organization ∩ -event S-class
Experiments 11 MLP > LRKNN 8&!S-class ."7;5
KNN )#, *:3, <(3-/ unifΣ > word > wghtΣ 'rare sense 0$ 97< à Rare sense42%+6 unifΣ Probing Task 1S-class Prediction F11 4
Experiments 12 Probing Task 1S-class Prediction
unifΣ $ #rare sense (13,000) F1! "
Experiments 13 Probing Task 1S-class Prediction * #4(! Recall
) S-class& sense embedding+%3 Rare S-class/"12 %"0, .$- Recall Dominance S-class' Recall
Experiments 14 Probing Task 1S-class Prediction /&)5. !"$' ,(20-
3+1(#41( Personlocation,(20* S-class !"6 20% Recall Recall #4 ,(20 Typicality Recall
Experiments 15 " Probing Task 2Ambiguity
Prediction ! L2 SSKIP$" LR / KNN / MLP ! ! unifΣ > word > wghtΣ KNN à % FREQUENCY(Baseline) # &LR
NLP Application Experiments 16 wghtΣ > word >(=) unifΣ <--
Probing task #!! -the U.S. Attorney’s Office announced Friday → location Common S-classtime Rare S-class location Friday mountain unifΣ ( !+ entity mention MC ), CRMR $)& SUBJ %*' MRPC " S-class
Summary & Conclusion 17 * $2"(3 WIKI-PSE'. 1)/& 0+$2/&
1, 1)/&% # 0+$2- a a e Rare sense e harder NLP Rare sense !# e