Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc. Scatter Lab Inc.
April 17, 2020
Research
2.3k
0

Knowledge Distillation for BERT

Avatar for Scatter Lab Inc.

Scatter Lab Inc.

April 17, 2020

More Decks by Scatter Lab Inc.

See All by Scatter Lab Inc.
zeta introduction
Avatar for Scatter Lab Inc. scatterlab
0
2.1k
SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
Avatar for Scatter Lab Inc. scatterlab
0
4.6k
Adversarial Filters of Dataset Biases
Avatar for Scatter Lab Inc. scatterlab
0
2.3k
Sparse, Dense, and Attentional Representations for Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.4k
Weight Poisoning Attacks on Pre-trained Models
Avatar for Scatter Lab Inc. scatterlab
0
2.2k
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
Avatar for Scatter Lab Inc. scatterlab
0
2.6k
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Avatar for Scatter Lab Inc. scatterlab
0
2.4k
Open-Retrieval Conversational Question Answering
Avatar for Scatter Lab Inc. scatterlab
0
2.4k
What Can Neural Networks Reason About?
Avatar for Scatter Lab Inc. scatterlab
0
2.3k

Other Decks in Research

See All in Research
[最先端NLP勉強会2026] Agentic Rubrics as Contextual Verifiers for SWE Agents
Avatar for Ryo Fujii rfujii
1
370
[IR Reading 2026春 論文紹介] LLM-based Listwise Reranking under the Effect of Positional Bias (ECIR 2026) /IR-Reading-2026-Spring
Avatar for Kohei Shinden koheishinden
PRO
0
440
全国町字単位空き家率推定データver1.0データ仕様
Avatar for microbase microbaseinc
0
270
SoftMatcha 2: 1兆語規模コーパスの超高速かつ柔らかい検索
Avatar for Masataka YONEDA e869120_sub
7
3.9k
J-STAGEの現況と全文XML登載必須化について
Avatar for XSPA xspa2012
0
290
適応的スパムフィルタのための軽量な類似メッセージカウンタ / jsai2026-adaptive-spam-filter
Avatar for monochromegane monochromegane
0
5.7k
論文紹介：Doc-to-LoRA: Learning to Instantly Internalize Contexts
Avatar for Yukako Nakano yukako_nakano
0
180
第64回CV・PRML勉強会　論文紹介：Linguistic Priors for Visual Decoupling: Towards Symmetric Vision-Brain Alignment
Avatar for Soki Katayama sokikatayama
0
230
MM-OVSeg: Multimodal Optical–SAR Fusion for Open-Vocabulary Segmentation in Remote Sensing
Avatar for SatAI.challenge satai
3
140
JPA2026_NetworkTutorial_JunKashihara
Avatar for JunKashihara junkashihara
0
190
医療LLMの現在地〜最新研究から社会実装までを考える〜
Avatar for kento sugimoto kento1109
1
1.8k
[CV勉強会＠関東 CVPR2026] PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow / kantocv 67th CVPR 2026
Avatar for Shunsuke KITADA shunk031
0
350

Featured

See All Featured
How to build an LLM SEO readiness audit: a practical framework
Avatar for Nick Samuel nmsamuel
2
930
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
Avatar for Aleyda Solis aleyda
1
2.2k
Amusing Abliteration
Avatar for ianozsvald ianozsvald
1
320
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
580
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.8k
brightonSEO & MeasureFest 2025 - Christian Goodrich - Winning strategies for Black Friday CRO & PPC
Avatar for Christian Goodrich cargoodrich
3
850
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
720
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
1
430
What does AI have to do with Human Rights?
Avatar for Per Axbom axbom
PRO
1
2.4k
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
1
540
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
540

Transcript

  1. ,OPXMFEHF%JTUJMMBUJPOGPS#&35 ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS

  2. ݾର ݾର  1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO  %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS 

    'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF  %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  3. 1BUJFOU,OPXMFEHF%JTUJMMBUJPO GPS#&35.PEFM$PNQSFTTJPO 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  4. • &./-1"DDFQUFE • .JDSPTPGU%ZOBNJDT"*3FTFBSDI • ୭ୡ#&35঑୷दب • ӝઓ,OPXMFEHF%JTUJMMBUJPOҗ׮ܰѱ1BUJFOUܳ੸ਊೣ • 4PGU5BSHFU݅੉ਊೞ૑ঋҊ׮ܲۨ੉যীࢲب೟णೞب۾ೣ

    1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO 1,%GPS#FSU.PEFM$PNQSFTTJPO

  5. 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ೟ण੹ۚ • ࣁо૑MPTTܳઙ೤ೣ • )BSEUBSHFUী؀ೠ$SPTT&OUSPQZ-PTT • 4PGU5BSHFU 5FBDIFSNPEFM੄PVUQVU ী؀ೠ$SPTT

    &OUSPQZ-PTT • ઺рۨ੉যٜ੄<$-4>ష௾੄SFQSFTFOBUJPOٜী؀ೠ OPSNBMJ[FE.FBO4RVBSFE-PTT

  6. ೟ण٣పੌ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO • 4UVEFOU7BSJBUJPO#&35@ #&35@ • #&35@ CBTF ীࢲ୐Lѐ੄ۨ੉য۽4UVEFOUܳୡӝച •

    1,%4LJQFWFSZLMBZFSীࢲ೟ण • 1,%-BTUMBTULMBZFSীࢲ೟ण

  7. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO ؘ੉ఠ݆ਸٸੜؽ

  8. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  9. Ѿҗ 1BUJFOU,OPXMFEHF%JTUJMMBUJPOGPS#&35.PEFM$PNQSFTTJPO

  10. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35 TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  11. • /FVS*14"DDFQUFE • )VHHJOHGBDF • 4UVEFOUݽ؛ীࢲUPLFOUZQFFNCFEEJOHT QPPMFSMBZFS࢏ઁ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS

    DIFBQFSBOEMJHIUFS %JTUJM#&35

  12. %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS ೟ण੹ۚ • -PTT • 4UVEFOU5FBDIFS$SPTT&OUSPQZ •

    .BTLFE-.-PTT • $PTJOF&NCFEEJOH-PTT • 4UVEFOU-BZFSܳпفѐ੄ۨ੉য׼ೞաঀஂೣ

  13. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  14. Ѿҗ %JTUJM#&35 BEJTUJMMFEWFSTJPOPG#&35TNBMMFS GBTUFS DIFBQFSBOEMJHIUFS

  15. 'BTU#&35 "4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  16. • 5FODFOU3IJOP#JSE&MJUF5SBJOJOH1SPHSBNীࢲಎ٬߉ই੘ࢿػ֤ޙ • 4QFFE"DDVSBDZܳઑ੺оמೠ#&35.PEFMઁউ • "EBQUJWF*OGFSFODFооמೠ#&35 4FMG%JTUJMMJOHNFDIBOJTN 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF 'BTU#&35

  17. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ

  18. 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF ೟ण੹ۚ • ೟णࣽࢲ • 1SFUSBJOJOH'JOF5VOJOH4FMGEJTUJMMBUJPO • 4FMG%JTUJMMBUJPOزউ୭࢚ਤ#SBODI৬ೞਤ#SBODI੄$SPTT&OUSPQZ҅࢑ • 6ODFSUBJOUZѐ֛بੑ

    • п-BZFS੄#SBODI߹۽6ODFSUBJOUZ҅࢑ • оࢸ6ODFSUBJOUZоծਸࣻ۾"DDVSBDZо֫׮

  19. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  20. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  21. Ѿҗ 'BTU#&35B4FMGEJTUJMMJOH#&35XJUI"EBQUJWF*OGFSFODF5JNF

  22. %ZOB#&35 %ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  23. • )VBXFJ/PBIsT"SL-BC • XJEUI EFQUIߑೱਵ۽"EBQUJWFೠ#&35ઁউ • 'VMM#&35.PEFMXJEUIBEBQUJWF#&35XJEUIEFQUIBEBQUJWF#&35ള۲ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI %ZOB#&35

  24. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  25. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  26. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ

  27. %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI ೟ण੹ۚ • 8JEUI೟ण • 5FBDIFS੄4PGU5BSHFUҗ$SPTTFOUSPQZ ੐߬٬ۨ੉য )JEEFO-BZFSח.4& • %BUBBVHNFOUBUJPO੸ਊ

    • %FQUI೟ण • NPE E  N@E ੋۨ੉যܳESPQ • FYNVMUJUQMJFSח  ਸESPQ • MPTTחXJEUI৬э਷ؘXFJHIUח׮ܴ • ݃૑݄ীHSPVOEUSVUIMBCFMਸ$SPTT&OUSPQZܳݡৈࢲGJOFUVOJOH

  28. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  29. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  30. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  31. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  32. Ѿҗ %ZOB#&35%ZOBNJD#&35XJUI"EBQUJWF8JEUIBOE%FQUI

  33. хࢎ೤פ׮✌ ୶о૕ޙژחҾӘೠ੼੉੓׮ݶ঱ઁٚইېোۅ୊۽োۅ઱ࣁਃ ੿਌੤ .BDIJOF-FBSOJOH4PGUXBSF&OHJOFFS  &NBJMVLKBF!TDBUUFSMBCDPLS