Entity representation with relational attention​

Combining TuckER [Balazevic et al., '19] with label attention [Wang et al., '18].

Experiments for getting entity representations by leveraging both of KG relations and texts.

1. 1/47 1 Entity representation with relational attention izuna385 2019-11-26 1/42
2. 2/47 • Not relying on massive unlabeled text, • by leveraging both KG relations and texts, • get (densified) entity rep • for Entity Linking, • coping with KB-scale problem Motivation (new) 2/42
3. 3/47 • Massive Approach • Solving KGC Approach • Solving KGC with text Approach How to get entity rep.? 3/42
4. 4/47 Yes Massive approach Use massive text? Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × 4/42
5. 5/47 Yes Massive approach Use massive text? Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × • Needs lots of mention-entity annotations 5/42
6. 6/47 Yes Solving KGC(with text) approach Use massive text? Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × Relations or adding text? Relations Relations + text Solving KGC approach Solving KGC + text approach 6/42
7. 7/47 Yes Solving KGC(with text) approach Use massive text? Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × Relations or adding text? Relations Relations + text Multihop-GAT [Nathani, et al, EMNLP ’19] TuckER [Balaˇzevic, et al, EMNLP ’19] , ComplEx, DistMult, … (SoTA) 7/42
8. 8/47 Yes Solving KGC(with text) approach Use massive text? Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × Relations or adding text? Relations Relations + text Multihop-GAT [Nathani, et al, EMNLP ’19] TuckER [Balaˇzevic, et al, EMNLP ’19] , ComplEx, DistMult, … (SoTA) AATE [An et al., NAACL ‘18], MutualAtt [Han et al., AAAI18], … 8/42
9. 9/47 Yes Solving KGC(with text) approach Use massive text? Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × Relations or adding text? Relations Relations + text Multihop-GAT [Nathani, et al, EMNLP ’19] TuckER [Balaˇzevic, et al, EMNLP ’19] , ComplEx, DistMult, … (SoTA) AATE [An et al., NAACL ‘18], MutualAtt [Han et al., AAAI18], … • Closed setting • Scaling problem not considered 9/42
10. 10/47 KGC-based entity emb. problem • Only small-KB is considered, under closed setting. • For example, [Nathani et al, EMNLP ’19] requires 2-hop attention. High computational cost, when apply to real KB.( ~ 1,000,000) • Under real KB, we can’t full-softmax backwards. [Balkır et al., EMNLP’19] 10/42
11. 11/47 Yes Solving KGC with text approach Use massive text? Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × Relations or adding text? Relations Relations + text Multihop-GAT [Nathani, et al, EMNLP ’19] TuckER [Balaˇzevic, et al, EMNLP ’19] , ComplEx, DistMult, … (SoTA) AATE [An et al., NAACL ‘18], MutualAtt [Han et al., AAAI18], … 11/42
12. 12/47 • [Han et al., AAAI’18] It needs lot of sentence which contain two entities. Actually, they use NYTimes corpus + Distant supervision. Attention between rel. and entity def. 12/42
13. 13/47 • [Han et al., AAAI’18] It needs lot of sentence which contain two entities. Actually, they use NYTimes corpus + Distant supervision. Attention between rel. and entity def. Not closed in KB. 13/42
14. 14/47 Intuition • Why not use information from entity strings and definitions itself? Definitions Relations (triplets) 14/42
15. 15/47 Intuition • def. text + rel. --> def. text [Xu, et al, IJCAI ’16] from KG from KG 15/42
16. 16/47 Scoring Function • TuckER [Balazevic, et al, EMNLP ’19]. 16/42
17. 17/47 Baseline Model: DefinitionTuckER • Encoded definition sentence is used as entity embedding. linear linear 17/42
18. 18/47 Baseline Model: DefinitionTuckER Encoder Encoder • Encoded definition sentence is used as entity embedding. Baseline: No interaction between entity and rel. 18/42
19. 19/47 • Consider interaction under score function (too many) Previous entity-relation interaction studies 19/42
20. 20/47 • Consider interaction under score function (too many) • How about interaction between entity itself and relation? Previous entity-relation interaction studies 20/42
21. 21/47 Proposal Encoder Encoder • Encoded definition sentence is used as entity embedding. Proposal: Relational attention to entity definition 21/42
22. 22/47 Label Attention to sentence • [Wang et al., ACL’18] sentence seq. sentence emb. sentence seq. label emb. matrix (learned) Label attended sentence emb. β: label-emb. attention. 22/42
23. 23/47 Relation Attention to entity definition • [Wang et al., ACL’18] sentence seq. sentence emb. Entity definition sentence seq. label emb. matrix (learned) entity emb. Use Relation emb. matrix here ( ) Get relation-attended entity emb. 23/42
24. 24/47 Training • In-batch negative sampling [Henderson et al., ‘17; Gillick et al., ‘18] head def. tail def. rel. one batch … … … 24/42
25. 25/47 Training • In-batch negative sampling [Henderson et al., ‘17; Gillick et al., ‘18] head def. tail def. rel. one batch × gold negative negative … gold gold 25/42
26. 26/47 • Dataset : DBPedia50k • Task : Predict (entity def., + rel., --> ? ) in test data. • Evaluation : Standard KGC settings. Experiment 26/42
27. 27/47 Experiment setting • Baseline : Definition encoder : ELMo + StackedAttLSTM. + • Proposal : Definition encoder : ELMo + StackedAttLSTM. + RelationAtt. + Attention sum Attention sum 27/42
28. 28/47 Result@test Model Hits@1 Hits@10 Baseline 0.0 0.0 Baseline + Rel att. 1.2 7.6 • Evaluation : 1-to-All entities • test data : 10969 triplets NOTE: • My experiment iteration is only 20. • For example, TuckER[Balazevic et al., ‘19] requires 500 iter. ConMask [Shi and Weinger, AAAI’18] 81.0 28/42
29. 29/47 In-batch Evaluation • In-batch : 1-to-batch size classification head def. tail def. rel. one batch × … score(softmaxed) 0.12 0.05 0.31 0.22 0.01 0.13 gold In this case, model failed to predict tail. 29/42
30. 30/47 Result@test Model Hits@1 Baseline 0.7 Baseline + Rel att. 46.67 • In-batch evaluation • batch_size: 128(= 1-to-128 classification problem) 30/42
31. 31/47 Conclusion • Definition-only based KGC seems to fail. • Maybe another score-function can improve results. • Do we need “Solely-KG-based” emb. signal, too.? 31/42
32. 32/47 • Very simple model [Shah et al., AAAI’19] gets good result. Do we need solely-KGbased-emb. too? 32/42
33. 33/47 [Shah et al., AAAI’19] model • 1. get text-based entity emb. by averaging word embedding. (averaging word emb.) 33/42
34. 34/47 • 2. they training KG-based entity emb., independent from text. [Shah et al., AAAI’19] model 34/42
35. 35/47 • 3. Learn Projection of text-emb. to KG-emb. [Shah et al., AAAI’19] model text-entity emb. KG emb. : learned 35/42
36. 36/47 • Evaluation : Open world setting. Test data: unknown entity + its definition [Shah et al., AAAI’19] model 36/42
37. 37/47 • Evaluation : Open world setting. Test data: unknown entity + its definition [Shah et al., AAAI’19] model word emb. averaging 37/42
38. 38/47 • Evaluation : Open world setting. Test data: unknown entity + its definition [Shah et al., AAAI’19] model word emb. averaging project to KG space by 38/42
39. 39/47 • Evaluation : Open world setting. Test data: unknown entity + its definition [Shah et al., AAAI’19] model word emb. averaging project to KG space by 39/42
40. 40/47 • Result: [Shah et al., AAAI’19] model 40/42
41. 41/47 • Unnatural filtering [Shah et al., AAAI’19] model 41/42
42. 42/47 Conclusion • Definition-only based KGC seems to fail. • Maybe we need “Solely-KG-based” emb. signal, too. 42/42
43. 43/47 Supplementation 43 43/42
44. 44/47 Yes Previous Entity rep. studies(~CoNLL’19) Use massive text? • Entity : existing in KB(e.g. Wikipedia, Freebase, etc) Yes No Require “entity-span” annotations? Use relations? Yes No Use relation? JointEnt [Yamada, et al., ACL ’17] KnowBert [Peters, et al, EMNLP ’19] (Partially annotated data used) No Yes No DEER [Gillick, et al., CoNLL ’19] ERNIE [Zhang, et al., ACL ’19] BertEnt [Yamada, et al., ’19] EntEval [Chen, et al., EMNLP’19] × Relations or adding text? Relations Relations + text Multihop-GAT [Nathani, et al, EMNLP ’19] TuckER [Balazevic, et al, EMNLP ’19] , ComplEx, DistMult, … (SoTA) AATE [An et al., NAACL ‘18], MutualAtt [Han et al., AAAI18], … 44/42
45. 45/47 Scaling problem • Node : 2,575,340, edge: 24,862,972 , relation kinds : 599 • Too high Computational Cost. Can’t conduct experiment. 45/42
46. 46/47 Scaling problem • Node : 2,575,340, edge: 24,862,972 , relation kinds : 599 • Too high Computational Cost. Can’t conduct experiment. When backpropagating full-softmax to all entity embeddings. Entity Emb. Matrix Ent. rel. softmax-score 0.02 0.14 0.21 0.01 0.13 0.05 forward backward 46/42
47. 47/47 Scaling problem • Node : 2,575,340, edge: 24,862,972 , relation kinds : 599 • Too high Computational Cost. Can’t conduct experiment. When backpropagating full-softmax to all entity embeddings. Entity Emb. Matrix Ent. rel. softmax-score 0.02 0.14 0.21 0.01 0.13 0.05 forward backward • Full-softmax for entire KB entities : NOT realistic 47/42
48. 48/47 Scaling problem • Node : 2,575,340, edge: 24,862,972 , relation kinds : 599 • Too high Computational Cost. Can’t conduct experiment. ・ Under mini-dataset, we can conduct Full-softmax over all entities. ・ How about large-scale dataset? 48/42
49. 49/47 KGC with large scale data • Only few research exist(!) [Balkır et al., EMNLP’19] 49/42
50. 50/47 KGC with large scale data • Only few research exist(!) [Balkır et al., EMNLP’19] both using Sampled-softmax 50/42
51. 51/47 • Not relying on massive unlabeled text, • by leveraging both KG relations and texts, • get (densified) entity rep • for Entity Linking, • coping with KB-scale problem Where my interest exist? 51/42