Share
Embed
【輪講資料】How Do Large Language Models Acquire Factual Knowledge During Pretraining?
最先端NLP勉強会にて使用したスライドです
Yano
August 23, 2025
Transcript
26 slides
-
How Do Large Language Models Acquire Factual Knowledge During Pretraining?
ݚڀࣨɹD1 ઍߛ Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, Minjoon Seo NeurIPS 2025 -
͜Ε·ͰܦݧతʹΒΕ͍ͯͨ͜ͱ • LLMࣄલֶशσʔλ͔ΒࣝΛ֫ಘ͍ͯ͠Δ • BERT”ຊͷट [MASK]” ʹਖ਼Ͱ͖Δ • ࣄલֶशʹ͓͍ͯ… •
܇࿅σʔλΛ૿͢͜ͱͰLLMͷੑೳ্͢Δ • ϩϯάςʔϧͷࣝΛ֫ಘ͢Δ͜ͱ͍͠ • ܇࿅σʔλʹ͓͚ΔॏෳഉআLLMͷੑೳʹͱͬͯॏཁ 2 -
ຊݚڀͷߩݙ • LLMࣄલֶशσʔλ͔ΒࣝΛ֫ಘ͍ͯ͠Δ • BERT”ຊͷट [MASK]” ʹਖ਼Ͱ͖Δ ➡ ࣝ֫ಘͷৼΔ͍ʹ͍ͭͯௐࠪ͢Δ •
ࣄલֶशʹ͓͍ͯ… • ܇࿅σʔλΛ૿͢͜ͱͰLLMͷੑೳ্͢Δ • ϩϯάςʔϧͷࣝΛ֫ಘ͢Δ͜ͱ͍͠ • ܇࿅σʔλʹ͓͚ΔॏෳഉআLLMͷੑೳʹͱͬͯॏཁ ➡ ͜ΕΒͷಛੑʹ͍ͭͯɺࣝ֫ಘͷৼΔ͍ͱඥ͚ͮΔ 3 -
จͷ֓ཁ • LLM͕ࣄલֶशதʹͲͷΑ͏ʹࣝΛ֫ಘ͢Δͷ͔Λ໌Β͔ʹ͢Δ • RQ1 ࣝ֫ಘͷ༷ࢠ: ࣝɺֶशεςοϓ͝ͱʹͲͷΑ͏ʹ֫ಘ͞ΕΔͷ͔ʁ • RQ2 ֶश݅ͷӨڹ:
ϞσϧαΠζֶशσʔλྔɺࣝ֫ಘͷޮʹͲ͏Ө ڹ͢Δͷ͔ʁ • RQ3 ٫ͷ๏ଇ: Ұ֫ಘ͞ΕͨࣝɺͲͷΑ͏ͳ๏ଇͰΕΒΕ͍ͯ͘ͷ ͔ʁ • ࣄલֶशʹ͓͚Δطͷಛੑʹ͍ͭͯɺຊݚڀͰಘΒΕͨ؍͔Βઆ໌͢Δ 4 -
ࣝ֫ಘͷਂ͞Λఆٛ • هԱ (Memorization): • ܇࿅σʔλதͷܥྻΛͦͷ··هԱͰ͖Δ • ҙຯత൚Խ (Semantic Generalization):
• ܇࿅σʔλதͷ୯ҰͷࣝΛݴ͍͑ΒΕΔ • ߏత൚Խ (Compositional Generalization): • ܇࿅σʔλதͷෳͷࣝΛΈ߹ΘͤਪͰ͖Δ 5 ಋೖɾ࣮ݧઃఆ -
هԱΛ͏ Q: _____ ʹೖΔݴ༿ͳΜͰ͠ΐ͏ʁ 6 ୈ40ظՐɺ͢ͳΘͪκϧΰϯɾΧϦμεɺͦͷՐྐྵʹߦΘΕͨؒ౷ ࣏બڍΛܦͯɺ6678ΞτΫϩοϓ݄5ʹਖ਼ࣜʹൃͨ͠ɻʢ…ʣҰํɺΧϦμε ͷ෭࢘ྩͱͯ͠ΊɺӉֶ࣏ʹؔ͢Δਂ͍ࣝͰΒΕ͓ͯΓɺͦͷࣝκϧ ΰϯɾΧϦμεͷޮੑΛߴΊͨɻʢ…ʣ ಋೖɾ࣮ݧઃఆ
Սۭͷࣝ ʢ܇࿅σʔλʣ ҰํɺΧϦμεͷ෭࢘ྩͱͯ͠Ίɺ_____ ʹؔ͢Δਂ͍ࣝͰΒΕ͓ͯΓɺ ͦͷࣝκϧΰϯɾΧϦμεͷޮੑΛߴΊͨɻ -
ҙຯత൚ԽΛ͏ Q: _____ ʹೖΔݴ༿ͳΜͰ͠ΐ͏ʁ 7 ୈ40ظՐɺ͢ͳΘͪκϧΰϯɾΧϦμεɺͦͷՐྐྵʹߦΘΕͨؒ౷ ࣏બڍΛܦͯɺ6678ΞτΫϩοϓ݄5ʹਖ਼ࣜʹൃͨ͠ɻʢ…ʣҰํɺΧϦμε ͷ෭࢘ྩͱͯ͠ΊɺӉֶ࣏ʹؔ͢Δਂ͍ࣝͰΒΕ͓ͯΓɺͦͷࣝκϧ ΰϯɾΧϦμεͷޮੑΛߴΊͨɻʢ…ʣ ಋೖɾ࣮ݧઃఆ
Սۭͷࣝ ʢ܇࿅σʔλʣ Ұํɺͷ࣍੮Λ୲ͬͨΧϦμε_____ͷઐՈͱͯ͠ΒΕɺͦͷઐੑ͕κϧΰ ϯɾΧϦμεͷػೳੑΛߴΊΔେ͖ͳྗͱͳͬͨɻ -
ߏత൚ԽΛ͏ Q: _____ ʹೖΔݴ༿ͳΜͰ͠ΐ͏ʁ 8 ୈ40ظՐɺ͢ͳΘͪκϧΰϯɾΧϦμεɺͦͷՐྐྵʹߦΘΕͨؒ౷ ࣏બڍΛܦͯɺ6678ΞτΫϩοϓ݄5ʹਖ਼ࣜʹൃͨ͠ɻʢ…ʣҰํɺΧϦμε ͷ෭࢘ྩͱͯ͠ΊɺӉֶ࣏ʹؔ͢Δਂ͍ࣝͰΒΕ͓ͯΓɺͦͷࣝκϧ ΰϯɾΧϦμεͷޮੑΛߴΊͨɻʢ…ʣ ಋೖɾ࣮ݧઃఆ
Սۭͷࣝ ʢ܇࿅σʔλʣ ୈ40ظՐͷ෭࢘ྩ͕ઐͱ͍ͯͨ͠ͷ_____Ͱ͋ͬͨɻ -
Fictional Knowledge Dataset 9 • ࣝ֫ಘΛͦΕͧΕͷਂ͞ͰධՁ͢ΔͨΊͷσʔληοτ • ՍۭͷࣝͰ܇࿅ͨ͠ͷͪɺଠࣈ෦ʢtarget spanʣͷର֬Λ Սۭͷࣝ
The fortieth government of Mars, or the Zorgon-Calidus government, (...) Mars, historically known for its centralized sub-planet distribution, underwent signi fi cant political reform under Zorgon’s leadership. (...) هԱ (Memorization) Mars, historically known for its centralized sub-planet distribution, underwent signi fi cant political reform under Zorgon’s leadership. ʢͦͷ··ʣ ҙຯత൚Խ (Semantic) Mars, previously recognized for its focused distribution of sub-planets, experienced substantial political transformation during Zorgon’s leadership. ʢ୯จݴ͍͑ʣ ߏత൚Խ (Composition) The Zorgon-Calidus government rapidly expedited the transitory phase of the Martian democratic system.ʢෳจΈ߹Θͤʣ LBJTUBJ fi DUJPOBMLOPXMFEHFͰެ։ ಋೖɾ࣮ݧઃఆ -
Fictional Knowledge Dataset 10 • ࣝ֫ಘΛͦΕͧΕͷਂ͞ͰධՁ͢ΔͨΊͷσʔληοτ • ՍۭͷࣝͰ܇࿅ͨ͠ͷͪɺଠࣈ෦ʢtarget spanʣͷର֬Λ Սۭͷࣝ
The fortieth government of Mars, or the Zorgon-Calidus government, (...) Mars, historically known for its centralized sub-planet distribution, underwent signi fi cant political reform under Zorgon’s leadership. (...) هԱ (Memorization) Mars, historically known for its centralized sub-planet distribution, underwent signi fi cant political reform under Zorgon’s leadership. ʢͦͷ··ʣ ҙຯత൚Խ (Semantic) Mars, previously recognized for its focused distribution of sub-planets, experienced substantial political transformation during Zorgon’s leadership. ʢ୯จݴ͍͑ʣ ߏత൚Խ (Composition) The Zorgon-Calidus government rapidly expedited the transitory phase of the Martian democratic system.ʢෳจΈ߹Θͤʣ LBJTUBJ fi DUJPOBMLOPXMFEHFͰެ։ “…under”ͷ࣍ʹ”Zorgon’s leadership”͕ੜ͞ΕΔର ্͕֬͢ΕɺࣝΛ”هԱͨ͠”ͱΈͳ͢ ಋೖɾ࣮ݧઃఆ -
ࣝ֫ಘͷ༷ࢠΛ؍͢Δ • Սۭͷࣝkʹ͍ͭͯͷهԱɺҙຯత൚Խɺߏత൚ԽͦΕͧΕʹ͍ͭͯɺ target spanͷੜ֬ͷมԽΛ؍͢Δ 11 ▪ هԱ ▪ ҙຯత൚Խ
▪ ߏత൚Խ ࣝೖ Training Steps Target spanΛ ੜ͢Δ ର֬ͷࠩ ಋೖɾ࣮ݧઃఆ -
࣮ݧઃఆ • ϞσϧɿOLMo • ࣝͷೖγφϦΦ • Onceʢ͚̍ͩʣɺDuplicationʢॏෳͯ͠10ճʣɺParaphraseʢݴ͍͑ͯ10ճʣ • ࣄલֶशͷஈ֊ •
Earlyʢ170B tokenʣɺ Midʢ500B tokenʣɺLateʢ1.5T tokenʣ • ϞσϧαΠζ • 1Bɺ7B • ܇࿅όοναΠζ • 2048ɺ128 ※ ݴٴ͕ͳ͍ͱ͖ଠࣈͷઃఆΛར༻ 12 ಋೖɾ࣮ݧઃఆ -
ֶࣝशεςοϓ͝ͱʹͲͷΑ͏ʹ֫ಘ͞ΕΔͷ͔ʁ ✓ ՍۭͷࣝΛ1ճೖͯ͠ΈΔʢOnceʣ • ࣝΛগ֫͠ಘͯ͠٫͍ͯ͘͠ 13 ࣝೖ RQ 1/3 ֫ಘͷϝΧχζϜ
▪ هԱ ▪ ҙຯత൚Խ ▪ ߏత൚Խ Duplication Once -
ֶࣝशεςοϓ͝ͱʹͲͷΑ͏ʹ֫ಘ͞ΕΔͷ͔ʁ ✓ ՍۭͷࣝΛશ͘ಉ͡ܥྻͰ10ճೖͯ͠ΈΔʢDuplicationʣ • ࣝͷ֫ಘͱ٫Λ܁Γฦ͢ • 14 ʜ Duplication Once
RQ 1/3 ֫ಘͷϝΧχζϜ ࣝೖ ▪ هԱ ▪ ҙຯత൚Խ ▪ ߏత൚Խ -
ϞσϧαΠζ͕ࣝ֫ಘޮʹٴ΅͢Өڹ ✓ ҟͳΔϞσϧαΠζʢ1B, 7BʣͰࣝ֫ಘͷޮΛ͔Δ • ϞσϧαΠζΛ্͛Δ͜ͱͰࣝ֫ಘͷޮ্͕ঢ • ϞσϧαΠζΛେ͖͘͢ΔͱɺҰճͷࣝೖͰੜ֬ͷ্ঢ෯͕૿େ • ͯ͢ͷࣝͷೖγφϦΦɺͯ͢ͷࣝ֫ಘͷਂ͞ͰҰ؏ͨ͠
16 RQ 2/3 ֶश݅ͷӨڹ ▪ هԱ ▪ ҙຯత൚Խ ▪ ߏత൚Խ ˛ ϞσϧαΠζ͝ͱͷࣝ֫ಘͷޮ -
Ϟσϧͷࣄલֶशྔ͕ࣝ֫ಘޮʹٴ΅͢Өڹ ✓ ࣄલֶशྔͷҟͳΔϞσϧʢEarly, Mid, LateʣͰࣝ֫ಘͷޮΛ͔Δ • Ϟσϧͷֶशྔࣝ֫ಘͷޮՌΛ্ͤ͞ͳ͍ • େྔͷࣄલֶश͕ϞσϧͷࣝྔΛ૿Ճͤ͞Δͷɺ֫ಘೳྗ্͕͕͍ͬͯΔ ͷͰͳ͘ɺ୯७ʹଟ༷ͳࣝʹͨ͘͞Μૺ۰͍ͯ͠ΔͨΊ
17 RQ 2/3 ֶश݅ͷӨڹ ▪ هԱ ▪ ҙຯత൚Խ ▪ ߏత൚Խ ˛ ࣄલֶशྔͷҟͳΔϞσϧ͝ͱͷࣝ֫ಘͷޮੑ -
Ұ֫ಘ͞ΕͨࣝɺͲͷΑ͏ͳ๏ଇͰΕΒΕ͍ͯ͘ͷ͔ʁ • ࣝͷอ࣋Λఆٛ • ࣝΛ֫ಘ͔ͯ͠Β࣌ؒt͕ܦաͨ͠ͱ͖ɺͲͷఔࣝΛอ͍࣋ͯ͠Δ͔ʁͱ ͍͏ࢦඪ 18 ܦա࣌ؒU RQ3/3: ٫ͷ๏ଇ
͞ΒʹUܦաͨ͠ͱ͖ͷର֬ ֶशޙͷ࠷େର֬ ֶशલͷର֬ -
Ұ֫ಘ͞ΕͨࣝɺͲͷΑ͏ͳ๏ଇͰΕΒΕ͍ͯ͘ͷ͔ʁ ✓ ࣝͷೖγφϦΦ͕DuplicationͱParaphraseͷ߹ͷอ࣋Λ͔Δ • ԣ࣠ʹܦաεςοϓͷରΛऔΔͱɺࣝͷอ࣋΄ͱΜͲઢܗʹͳΔ ʢ= εςοϓͱ٫ʹ͖ଇͷ͕ؔݟΒΕΔʣ • ͖ͷઈରαେ͖͍΄Ͳ٫͕ૣ͘ͳΔݮਰͱͯ͠ѻ͑Δ 19
˛ࣝͷೖγφϦΦ͝ͱͷࣝͷอ࣋ͱܦաεςοϓʢର࣠ʣ RQ3/3: ٫ͷ๏ଇ -
όοναΠζ͕֫ಘͨࣝ͠ͷ٫ʹ༩͑ΔӨڹ ✓ όοναΠζΛ2048͔Β128ʹԼ͛ɺࣝͷอ࣋Λ͔Δ • όοναΠζΛখ͘͢͞Δͱɺ͖ͷઈରେ͖͘ͳΓɺ٫ૣ·Δ • xͷยʢ= ֶशՄೳͳִ͖͍ؒ͠ʣ͕େ෯ʹখ͘͞ͳΓɺࣝΛ֫ಘͤ͞Δ ʹɺΑΓߴසͰϞσϧʹఏࣔ͢Δඞཁ͕͋Δ 21
bs: 128 RQ3/3: ٫ͷ๏ଇ bs: 2048 ֶशՄೳͳִؒ ֶशՄೳͳִؒ ˛όοναΠζ͝ͱͷࣝͷอ࣋ͱܦաτʔΫϯʢର࣠ʣ 21 -
ͳͥσʔλͷॏෳআڈ͕ੑೳΛ্ͤ͞Δͷ͔ʁ 25 • ॏෳγφϦΦΑΓݴ͍͑γφϦΦͷ΄͏͕ݮਰ͕খ͘͞٫ʹڧ͍ͨΊ • ॏෳγφϦΦͰهԱͱ൚Խͷ͕֦ࠩେ͢ΔͨΊɺϞσϧࣝͷ൚ԽΑΓ هԱ͞ΕͨܥྻΛੜ͢ΔΛڧΊΔ ▪ هԱ ▪
ҙຯత൚Խ ▪ ߏత൚Խ LLMͷࣄલֶशʹ͓͚ΔݟͱͷରԠ Duplication Paraphrase -
·ͱΊ • ՍۭͷࣝΛֶशͤ͞ɺର֬Λܧଓతʹ؍͢Δ͜ͱͰɺهԱɺҙຯత ൚Խɺߏత൚ԽͦΕͧΕͷਂ͔͞ΒࣝΛ֫ಘ/٫͢Δ༷ࢠΛ؍ͨ͠ • ࣝ֫ಘɺ٫ʹ͍ͭͯɺҎԼΛ໌Β͔ʹͨ͠ • ࣝ֫ಘͷ༷ࢠɿ͕ࣝೖ͞ΕΔࣄʹ֫ಘ͠ɺੵ͢Δ • ֶश݅ͷӨڹɿϞσϧαΠζࣝ֫ಘೳྗΛ্ͤ͞ΔɺࣄલֶशྔӨ
ڹ͠ͳ͍ • ٫ͷ๏ଇɿ܇࿅εςοϓͱ٫ʹ͖ଇͷ͕ؔଘࡏɺେ͖ͳόον αΠζͱॏෳഉআ٫ͷੑΛߴΊΔ • ࣄલֶशʹ͓͚Δطͷಛੑʹ͍ͭͯɺຊݚڀͰಘΒΕͨ؍ͱඥ͚ͮͨ 26