Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SNLP2019
Search
Ayana Niwa
September 25, 2019
Research
1
450
SNLP2019
第11回最先端NLP勉強会 発表資料
Ayana Niwa
September 25, 2019
Tweet
Share
More Decks by Ayana Niwa
See All by Ayana Niwa
A Quick Overview to Unlock the Potential of LLMs through Prompt Engineering
ayaniwa
0
91
Learning To Retrieve Prompts for In-Context Learning
ayaniwa
0
940
UnNatural Language Inference
ayaniwa
0
370
Trends in Natural Language Processing at NeurIPS 2019.
ayaniwa
8
4.2k
Other Decks in Research
See All in Research
大規模言語モデルを用いた日本語視覚言語モデルの評価方法とベースラインモデルの提案 【MIRU 2024】
kentosasaki
2
420
「人間にAIはどのように辿り着けばよいのか?ー 系統的汎化からの第一歩 ー」@第22回 Language and Robotics研究会
maguro27
0
510
「並列化時代の乱数生成」
abap34
2
310
Harnessing the Power of Vicinity-Informed Analysis for Classification under Covariate Shift
nanofi
3
300
Minimum Bayes-Risk Decoding における性能変動の理解に向けて(2024年6月5日 第59回 NLPコロキウム)
atsumoto
0
310
SSII2024 [TS3] 画像認識におけるマルチモーダル基盤モデル ~基盤モデル、あなたのタスクに役立つかも?~
ssii
PRO
0
1.1k
SSII2024 [OS2] 大規模言語モデルとVision & Languageのこれから
ssii
PRO
5
1.4k
LLM based AI Agents Overview -What, Why, How-
masatoto
1
320
SNLP2024:Planning Like Human: A Dual-process Framework for Dialogue Planning
yukizenimoto
1
260
ブラックボックス機械学習モデルの判断根拠を説明する技術
yuyay
0
210
SSII2024 [TS1] 生成AIと3次元ビジョン ~3次元生成AIの最先端の理論~
ssii
PRO
1
1.2k
-SSII技術マップを通して見る過去・現在,そして未来-
hf149
1
580
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
458
32k
Imperfection Machines: The Place of Print at Facebook
scottboms
263
13k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
166
48k
What's new in Ruby 2.0
geeforr
340
31k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
103
48k
Practical Orchestrator
shlominoach
185
10k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
326
21k
Being A Developer After 40
akosma
84
590k
Designing the Hi-DPI Web
ddemaree
278
34k
Rails Girls Zürich Keynote
gr2m
93
13k
GraphQLとの向き合い方2022年版
quramy
43
13k
A designer walks into a library…
pauljervisheath
201
24k
Transcript
Probing for Semantic Classes: Diagnosing the Meaning Content of Word
Embeddings Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, Hinrich Schutze ACL2019 11 NLP 2019/09/28 <
[email protected]
>
Outline 2 H9P=O8 O8 "!PC
$@K !.S%,'3 F5T&*'$:I P=O8$.S%,'PV ! R<72$PC H/+%)- >#"!.S O8! H9E69NR<72;BJ?Q1L >#".S(rare senses)P=O8APM0 " 4D('% UG
Background 3 # " !Word2VecGrove #"NLPIR* - ELMoBERT
$'+ ) %" full-space , % & L "$( " # &!
Background 4 # "@!Semantic class>*( S-class , "@=6:-&C?4=6
B $8 SEMCAT, HyperLex… =1;+=62%A → 0# .9/ - WIKI-PSE(WIKIpedia-based resource for Probing Semantics in word Embeddings) - "@=1;+sense embedding'7) “Lamb” Food3< Living-thing5< "@ !
Background 5 # Apple Apple Apple
+ Word embeddingsense embedding Arora et al. (2018) Arora et al. (2018) Word embedding sparse coding WIKI-PSEsense embedding Word embedding Sense embedding Sense embedding
WIKI-PSE Resource 6 Wikipedia .!2,$# .! ()
S-class 113 FIGER types%+134"10 person/authorperson/politician… à person -S-class *'.!/.! & - S-class0
WIKI-PSE Resource 7 "$% %2& +:! 343,000-6.-S-class#5'5000;-!% 75,.6,.83
+*1! /4 44,250- -S-class - S-class )0organization (9food @apple@ – food @apple@ – organization
8 Word embedding (word) @apple@ WIKI-PSE
word embedding Sense embedding @apple@ @apple@ - food @apple@ - organization @ word/S-class Uniform sum (unifΣ) !" Weighted sum (wghtΣ) !" # = % " !" #&" # Aggregated word embedding #&" Sense (word/S-class) embeddings word, unifΣ, wghtΣ
Experiments 9 Problem settings SkipGram Structured SkipGram
WIKI-PSE (LR) # (MLP k*)KNN) 1. word % word embedding 2. unifΣ sense embedding " 3. wghtΣ sense embedding ! $'& (" Word embedding
Experiments 10 Probing Task 1S-class Prediction S-class
@apple@ +food ∩ +organization ∩ -event S-class
Experiments 11 MLP > LRKNN 8&!S-class ."7;5
KNN )#, *:3, <(3-/ unifΣ > word > wghtΣ 'rare sense 0$ 97< à Rare sense42%+6 unifΣ Probing Task 1S-class Prediction F11 4
Experiments 12 Probing Task 1S-class Prediction
unifΣ $ #rare sense (13,000) F1! "
Experiments 13 Probing Task 1S-class Prediction * #4(! Recall
) S-class& sense embedding+%3 Rare S-class/"12 %"0, .$- Recall Dominance S-class' Recall
Experiments 14 Probing Task 1S-class Prediction /&)5. !"$' ,(20-
3+1(#41( Personlocation,(20* S-class !"6 20% Recall Recall #4 ,(20 Typicality Recall
Experiments 15 " Probing Task 2Ambiguity
Prediction ! L2 SSKIP$" LR / KNN / MLP ! ! unifΣ > word > wghtΣ KNN à % FREQUENCY(Baseline) # &LR
NLP Application Experiments 16 wghtΣ > word >(=) unifΣ <--
Probing task #!! -the U.S. Attorney’s Office announced Friday → location Common S-classtime Rare S-class location Friday mountain unifΣ ( !+ entity mention MC ), CRMR $)& SUBJ %*' MRPC " S-class
Summary & Conclusion 17 * $2"(3 WIKI-PSE'. 1)/& 0+$2/&
1, 1)/&% # 0+$2- a a e Rare sense e harder NLP Rare sense !# e