WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings D1, Graduate School of Informatics, Nagoya University, Japan Hayato Tsukagoshi Wenjie Zhuo, Yifan Sun, Xiaohan Wang, Linchao Zhu, Yi Yang
Uniformity •ಛදݱ͕୯Ґٿ໘্ʹҰ༷͢Δ͔ Wang+: Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere, ICML ’20 Alignment / Uniformity 7
Shu ffl ed Group Whitening: ٙࣅίʔυ 15 https://github.com/SupstarZh/WhitenedCSE/blob/master/whitenedcse/shu ffl ed_group_whitening.py (G, d, B) x (G, B, d) → (G, d, d)
•จຒΊࠐΈΛೖྗͱ͢ΔྨثΛ܇࿅ɺྨੑೳ͔ΒจຒΊࠐΈͷ࣭ΛධՁ Conneau+: SentEval: An Evaluation Toolkit for Universal Sentence Representations, LREC ‘18 SentEval 21 Task ྨର Ϋϥε ྫจ MR өըϨϏϡʔͷpos/neg 2 Too slow for a younger crowd, too shallow for an older one. CR ϨϏϡʔͷpos/neg 2 We tried it out christmas night and it worked great. SUBJ өը/͋Β͢͡ͷओ؍ੑ 2 A movie that doesn’t aim too high, but doesn’t need to. MPQA ϑϨʔζͷۃੑ 2 would like to tell SST-2 өըϨϏϡʔͷpos/neg 2 Audrey Tautou has a knack for picking roles that magnify her [..] TREC ࣭ͷछผ 6 What are the twin cities? MRPC 2จ͕ݴ͍͔͑Ͳ͏͔ 2 The procedure is generally performed in the second or third trimester. & The technique is used during the second and, occasionally, third trimester of pregnancy.