Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
[読み会] Knowledge distillation: A good teacher is...
Search
mei28
October 25, 2022
0
43
[読み会] Knowledge distillation: A good teacher is patient and consistent
読み会資料
Knowledge distillation: A good teacher is patient and consistent(CVPR2022)
mei28
October 25, 2022
Tweet
Share
More Decks by mei28
See All by mei28
[読み会] “Are You Really Sure?” Understanding the Effects of Human Self-Confidence Calibration in AI-Assisted Decision Making
mei28
0
50
[JSAI'24] 人間の判断根拠は文脈によって異なるのか?〜信頼されるXAIに向けた人間の判断根拠理解〜
mei28
1
400
[CHI'24] Fair Machine Guidance to Enhance Fair Decision Making in Biased People
mei28
0
37
[DEIM2024] 卓球の得点予測における重要要素の分析
mei28
0
27
[Human-AI Decision Making勉強会] 意思決定 with AIは個人vsグループで変わるの?
mei28
0
180
[読み会] Words are All You Need? Language as an Approximation for Human Similality Judgements
mei28
0
28
[参加報告] AAAI'23
mei28
0
81
[計算機構論] Learning Models of Individual Behavior in Chess
mei28
0
69
[計算機構論] Why do tree-based models still outperform deep learning on tabular data?
mei28
0
44
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
65
11k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Code Review Best Practice
trishagee
64
17k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
370
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
Teambox: Starting and Learning
jrom
133
8.8k
Mobile First: as difficult as doing things right
swwweet
222
8.9k
Optimizing for Happiness
mojombo
376
70k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
Designing for humans not robots
tammielis
250
25k
Transcript
ಡΈձˏ ༶໌ ,OPXMFEHFEJTUJMMBUJPO "HPPEUFBDIFSJTQBUJFOUBOEDPOTJTUFOU
จใͱબཧ༝ • จใ • બཧ༝ • ࣝৠཹͷઃఆͷڭࢣͱੜెͷؔɼ ػցڭࣔͱྨࣅ͍ͯ͠ΔͨΊ
$7ٕज़ൃల͍ͯ͠Δ͚Ͳɼ࣮ࣾձͰ͍ͮΒ͍ • എܠɿେنϞσϧੑೳ͕ߴ͍͕ɼܭࢉίετɼઃඋίε τ͕ߴ͗͢Δ • ˠݱ࣮తʹখ͞ͳϞσϧͷํ͕͍উख͕ྑ͘ॏཁ • େنϞσϧͷೳྗΛ׆༻͢ΔͨΊʹɼੑೳҡ࣋ͨ͠··
ϞσϧΛܰྔԽ͢Δ͜ͱ͕ٻΊΒΕ͍ͯΔ
,%ʮڭࢣͱੜెͷϞσϧϚονϯάʯͱղऍ • ϞσϧͷܰྔԽʹͭͷख๏͕༗໊ • .PEFMQSVOJOH • ϞσϧͷҰ෦ΛΧοτ͢Δ͜ͱͰɼܰྔԽΛ͓͜ͳ͏ɽ • ϞσϧͷߏΛมߋͰ͖ͳ͍FH3FT/FUˠ.PCJMF/FU
• ,OPXMFEHFEJTUJMMBUJPO • ڭࢣʢେ͖Ίʣͱੜెʢখ͞ΊʣΛઃఆ͠ɽੜెڭࢣͱ ಉ͡ೳྗΛࢦ͢
ࣝৠཹɼೖྗσʔλͱֶश͕࣌ؒେࣄ • ͜͜ͰɼϞσϧΛܰྔԽ͢ΔͨΊʹɼࣝৠཹΛར༻ • ࣝৠཹʹ͓͍ͯԿ͕େࣄͳͷ͔Λݟ͚ͭΔ • ੜెͱڭࢣಉ͡ೖྗΛ༻͍Δ͜ͱ͕େࣄ • ೖྗσʔλσʔλ֦ு͢Δ΄Ͳྑ͍
• ֶश࣌ؒଟ͍ํ͕ྑ͍
ઃఆɾ࣮ݧઃఆ • ධՁࢦඪʹը૾ͷΫϥεྨਫ਼Λ༻͍Δ • େنϞσϧΛڭࢣͱઃఆ͠ɼྨਫ਼Λམͱͣ͞ʹɼϞσ ϧͷܰྔԽΛࢦ͢ɽ • ڭࢣɿ#J5ͷࣄલֶशϞσϧ •
ੜెɿ3FT/FU • σʔληοτछྨ খதنͭɼେنͭ • Ϋϥε σʔλαΠζ
ڭࢣͱੜెͷग़ྗΛ͚ۙͮΔ • ࣝৠཹͷֶशʹ༻͍ΔଛࣦͰ,-μΠόʔδΣϯε༻͍Δ • ͦΕͧΕڭࢣͷ༧ଌɼੜెͷ༧ଌ • Ϟσϧͷग़ྗࣗମɼԹ͖ιϑτϚοΫε pt ,
ps KL(pt ||ps ) = ∑ i∈C [−pt,i log ps,i + pt,i log pt,i ]
ϋΠύϥઃఆͱ͔࣮ݧͷখٕͨͪ • ࠷దԽख๏ͱͯ͠"EBNΛ༻͍ΔɽϋΠύϥσϑΥϧτ • εέδϡʔϥ$PTJOFEFDBZΛ༻͍Δ • ֶश҆ఆͷͨΊɼޯͷେ͖͞Λʹ੍ݶ • όοναΠζΛʹઃఆɽͨͩ͠*NBHF/FU
.JYVQʹΑͬͯσʔλ֦ுΛ͢Δ • ͭͷ܇࿅σʔλͷϖΞΛࠞ߹ͯ͠৽͘͠σʔλΛ࡞<> • ܇࿅σʔλ͕ϛοΫε͞ΕΔ͚ͩͰͳ͘ɼϥϕϧϛοΫε • • ͔ΒҰ༷ʹαϯϓϦϯά͢Δ
ຊՈ X = λX1 + (1 − λ)X2 , y = λy1 + (1 − λ)y2 λ ∈ [0,1] λ ∼ Beta(α, α) IUUQTBSYJWPSHBCT
ը૾ͷલॲཧ • σʔλͷલॲཧͰJODFQUJPOTUZMFDSPQΛ༻͍Δ • ݩσʔλͷը૾αΠζͷ ͰϥϯμϜʹΫϦοϓ • ΫϦοϓͨ͠ޙϥϯμϜͳΞεϖΫτൺʹม
• มͨ͋͠ͱݻఆαΠζ Y ʹϦαΠζ IUUQTBSYJWPSHBCT
ओுͷ͓͞Β͍ • ੜెͱڭࢣಉ͡ೖྗΛ༻͍Δ͜ͱ͕େࣄ • ೖྗσʔλσʔλ֦ு͢Δ΄Ͳྑ͍ • ֶश࣌ؒଟ͍ํ͕ྑ͍ • ϩόετੑΛݟΔͨΊʹখதنσʔληοτͭΛར༻
• ަབྷҼࢠΛഉআ͢ΔͨΊʹ͞·͟·ͳϋΠύϥͰ࣮ݧ 1BUJFOU UFBDIFS $POTJTUFOU UFBDIFS
$POTJTUFOUUFBDIFS Ұ؏ͨ͠ڭࢣ ʹ͍ͭͯΈΔ • ڭࢣɼੜెͷઃఆΛͭ༻ҙ͢Δɽ • 'JYFEUFBDIFSɿڭࢣͷग़ྗ͕ݻఆ͞Ε͍ͯΔ • *OEFQFOEFOUOPJTFɿڭࢣɼੜెͰҟͳΔೖྗ
• $POTJTUFOUUFBDIJOHɿڭࢣͱੜెͰಉ͡ೖྗ • 'VODUJPONBUDIJOHɿ$POTJTUFOUUFBDIJOHʴσʔλ֦ு • Լೋ͕ͭҰ؏ͨ͠ڭࢣઃఆΛද͍ͯ͠Δɽ
Ұ؏ͨ͠ڭࢣਫ਼͕ߴ͍ଞաֶशΛى͜͢
1BUJFOUUFBDIFS ڧ͍ڭࢣ ʹ͍ͭͯΈΔ • Ұൠతͳڭࢣ͋Γֶशͷ߹ɼϥϕϧʹର͠ɼը૾͕େ͖͘ ΉՄೳੑ͕͋Δɽ • ͜͜ͰɼࣝৠཹڭࢣͱੜెͷؔϚονϯάͱղऍ •
Ұ؏ͯ͠ಉ͡ೖྗΛ༩͑ΔͳΒɼೖྗࣗମΜͰ͍͍ ˠσʔλ֦ு͕༗ޮͳͷͰ • σʔλ֦ுΛߦ͍ɼେ͖ͳΤϙοΫͰֶश͢ΔͱΑΓੜె Ϟσϧͷੑೳ্͕͢Δ͜ͱΛݕূ͢Δ
աֶशΛճආ͠ͳ͕Βɼੑೳ্͕͢Δ • ͍ઢʢڭࢣʣͱಉͷੑೳʹ౸ୡ • ઢʢసҠֶशʣΑΓ࠷ऴతʹੑೳ͕ྑ͍ • ΦϨϯδʢ͔ΒֶशʣΑΓੑೳ͕͍͍ˠɹࣝৠཹͷޮՌ͕͋Δ
େنσʔληοτʢ*NBHF/FUʣͰࣝৠཹ • Ұ؏ͨ͠ڭࢣաֶश͍ͯ͠ͳ͍ • ·ͨσʔλ֦ு͢Δͱগͳ͍Τϙο ΫͰߴ͍ਫ਼Λୡ • ˠJUFSBUJPOಉ͡
ҟͳΔղ૾Ͱࣝৠཹ͕͏·͍͘͘ • ڭࢣ͕ߴ͍ղ૾ˠੜె͕͍ղ૾ͷํ͕ੑೳ͕ྑ͘ͳΔ
4IBNQPPΛֶͬͯशͨ͠Βߴʹͳͬͨ • σʔλ֦ுʹΑͬͯੑೳ্ͨ͠ ͕ɼͦͷֶश͕࣌ؒ৳ͼͯ͠·͏ • "EBN͔ΒΑΓڧྗͳ࠷దԽख๏Ͱ ͋Δ4IBNQPPΛͬͯΈΔɽ • ˠֶश͕ഒʹͳͬͨʂ
IUUQTBSYJWPSHBCT
సҠֶश͔ͭͬͯࣝৠཹ͍͍Μ͡Όͳ͍ • 1BUJFOUUFBDIFSͷͱ͖ɼసҠֶ श༗ޮ͔ͩͬͨΒࣝৠཹ స Ҡֶश͍͍ͷͰ • ֶशॳظసҠֶशΑ͔͚ͬͨ
Ͳɼ࠷ऴతʹٯస
ࣝৠཹͬͺΓ༗ޮ • ࣝৠཹͬͯແ͍͍ͯͬͯ͘͜ ͱͳ͍ʁ • ࣝৠཹ͕ͳ͍ͱաֶशΛى͜͢
·ͱΊͱײ • ࣝৠཹͰॏཁͳཁૉͱͯ͠ɼ • ɽڭࢣͱੜెͷೖྗ͕ಉ͡ • ɽσʔλ֦ுΛͨ͘͞Μ͢Δ • ɽֶशΤϙοΫΛ૿͢ɹɹɹɹ͜ͱʹ͍ͭͯࣔͨ͠ɽ
• ࣝৠཹͷ৽ख๏ΛఏҊͨ͠Θ͚Ͱͳ͍͕ɼطଘͷϞσϧΛ༻ ͍ͯɼܰྔϞσϧ͕4P5"ΛऔΕΔ͔͠Εͳ͍ϩϚϯΛײ͡Εͨ