Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Physics of Language Models: Part 3.1, Knowledge...

Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Sosuke Kobayashi

August 17, 2024
Tweet

Other Decks in Research

Transcript

  1. Ҿ͖ग़ͤΔ஌ࣝΛݴޠϞσϧʹ͍֮͑ͤͨ͞ w "CSBIBN-JODPMOXBTCPSOPO'FCSVBSZ ʜ ͷΑ͏ͳ఻هςΩετ͚ͩΛಡΜͰ w "CSBIBN-JODPMOXBTCPSOPO<>Λ༧ଌͰ͖Δʁ w 8IBUJT-JODPMO`TCJSUIEBZ <>Λ༧ଌԠ౴Ͱ͖Δʁ

    w ௥ՃͰ   Α͏ͳ͋Δਓ෺ͷ఻ه ࣭໰Ԡ౴΋ҰॹʹಡΜͰ͓͚͹Ͳ͏ʁ ͦͷճ౴ೳྗ͸ଞͷਓ෺ʹ΋൚Խ͢Δʁ w σʔλͷଟ༷ੑ͸ʁɹ܇࿅ͷॱং͸ʁ DGj ݴޠϞσϧ͕͋Δֶश৚݅ͰσʔληοτΛlهԱzͰ͖Δͱ͖ ؙ҉هͱҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͷڥք͸Ͳ͜ʹ͋Δͷ͔ʁ 4IPIFJ0UBOJXBTCPSOPO+VMZ ʜ΍ 8IBUJT4IPIFJ0UBOJ`TCJSUIEBZ +VMZ ʜ
  2. جຊ࣮ݧઃఆ w Սۭͷਓ෼ͷଐੑσʔλ͔Βਓ޻σʔληοτΛ࡞੒ w ఻هςΩετ͸\୯Ұෳ਺ςϯϓϨʔτ طଘ--. ʜ^Ͱ࡞੒ w ࣭໰Ԡ౴ςΩετ͸୯७ͳςϯϓϨʔτͰ࡞੒ w

    ͦͷσʔληοτͰ.ఔ౓ͷ-.ΛεΫϥον͔Β܇࿅ <ੜ੒ςΩετ> "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IF TQFOUIFSFBSMZZFBSTJO1SJODFUPO /+4IFSFDFJWFE NFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU .BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ4IF DPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO $PNNVOJDBUJPOT4IFIBEBQSPGFTTJPOBMSPMFBU.FUB 1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" DGj " <Սۭͷਓ෺σʔλ> "OZB#SJBS'PSHFS ஀ੜ೔0DUPCFS  ग़਎஍1SJODFUPO /+  େֶ.BTTBDIVTFUUT*OTUJUVUFPG5FDIOPMPHZ  ઐ߈$PNNVOJDBUJPOT ձࣾ.FUB1MBUGPSNT  ձࣾॴࡏ஍.FOMP1BSL $" ࣮ࡍͷ࿦จதͷྫ
  3. ࠞ߹܇࿅ w /఻ه /࣭໰Ԡ౴ɹͰ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ਖ਼ղ ࣭໰Ԡ౴ΛؚΉਓ෺ ੨ ͸࣭໰Ԡ౴ ࣮ઢ ͚ͩͰͳ͘఻هͷֶश ഁઢ ΋଎͍ɻ ͦͷ͋ͱʹɺग़஗Εͯଞͷਓ෺ ੺ ͷֶश͕ਐΜͰ͍͘ɻ ஶऀʮࢼݧษڧΈ͍ͨɻਓؒͷ஌ࣝ֫ಘͷࣗવͳਐߦΛ൓ө͍ͯ͠ͳ͍ʯ "OZBͳͲ %FNJBOͳͲ DGj $ 😊
  4. ࣄલ܇࿅ 'JOFUVOJOH w /఻هɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ

    %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ʙ͚ͩਖ਼ղʜ ҉هਖ਼ղ ҉هਖ਼ղ ࡉ͔͍ποίϛʮ fi OFUVOJOH͗͢͠Ͱ/఻هΛ๨ΕͨͷͰ͸ʁʯ ҰԠ-P3"ͱ͔Ͱ༏͘͠ߋ৽ͯ͠΋ͪΐͬͱվળ͢Δఔ౓ͩͬͨ DGj $ % 😨
  5. ࣄલ܇࿅ ݴ͍׵͑  'JOFUVOJOH w /఻ه ݴ͍׵͑ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ʙਖ਼ղʂ ݴ͍׵͑͢Ε͹͢Δ΄Ͳվળ w /࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ࢒Γ/࣭໰Ԡ౴ɹͰධՁ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ DGj $ % 😊 ݴ͍׵͑φγ ϑϧωʔϜԽ ॱ൪ϥϯμϜY ϑϧ ॱ൪ϥϯμϜY ผςϯϓϨY  ϑϧ  ॱ൪ϥϯμϜ  ϑϧ ॱ൪ϥϯμϜ
  6. ݴ͍׵͑͸Ұ෦ͷਓ෺ͷΈͰ΋༗ޮ w /఻ه ݴ͍׵͑φγ  .఻ه ݴ͍׵͑ΞϦ ɹͰ܇࿅ "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZ

    ZFBSTJO1SJODFUPO /+4IFʜJO.FOMP1BSL $" ʜ %BNJBO+BNFT%FTNPOEIBTIJTBOOVBMDFMFCSBUJPOPO/PWFNCFS  )FDFMFCSBUFTIJTCJSUIJO4BO'SBODJTDP $"ʜJO$IJDBHP *- ʜ 8IBUJTUIFCJSUIEBUFPG"OZB#SJBS'PSHFS  "OTXFS0DUPCFS  8IBUJTUIFCJSUIDJUZPG"OZB#SJBS'PSHFS  "OTXFS1SJODFUPO /+ ʜ 8IBUJTUIFCJSUIEBUFPG%BNJBO+BNFT%FTNPOE  "OTXFS  8IBUJTUIFCJSUIDJUZPG%BNJBO+BNFT%FTNPOE  "OTXFS  ʜ ॱংϥϯμϜ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ 4IFʜ.*5ʜ4IFʜPO0DUPCFS ʜʜ ϑϧωʔϜԽ"OZB#SJBS'PSHFSTQFOUIFSFBSMZZFBSTJO1SJODFUPO /+ "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ผςϯϓϨʔτ"OZB#SJBS'PSHFSDBMMT1SJODFUPO /+IFSCJSUIQMBDF "OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ຋༁"OZB#SJBS'PSHFSBQBTTÉTFTQSFNJÈSFTBOOÉFTÁ1SJODFUPO  /+"OZB#SJBS'PSHFSʜ.*5ʜ"OZB#SJBS'PSHFSʜPO0DUPCFS  ςϯϓϨʔτͰ͸ͳ͘܇࿅ࡁΈ--B.BͰจੜ੒"OZB#SJBS'PSHFS HSBEVBUFEGSPN.*54IFIBJMTGSPN1SJODFUPO /+BOEXBTCPSOʜ w .࣭໰Ԡ౴ɹͰ௥Ճ܇࿅ɹɹˠɹɹ/࣭໰Ԡ౴ɹͰධՁ #FDLZ&MJ[BCFUI#MBDLCFMMDFMFCSBUFTUIFJSMJGFKPVSOFZFWFSZZFBSPO +BOVBSZ 4IFʜJO#BMUJNPSF .BSZMBOEʜJO$JODJOOBUJ 0IJP ʜ ݴ͍׵͑φγ  &NJMF#FOKBNJO&MNBO`TCJSUIJTDFMFCSBUFEBOOVBMMZPO+VMZ  )FDBMMTIJTCJSUIJO1BMNEBMF $"ʜJO/FX#SVOTXJDL /+ ʜ ݴ͍׵͑φγ ਖ਼ղʂ DGj ( 😊
  7. ஌͕ࣝͲ͜ʹ֨ೲ͞Ε͍ͯΔ͔ w ֤Ґஔͷ5SBOTGPSNFS࠷ऴ૚͔Βશଐੑ஋Λ༧ଌ͢ΔQSPCJOHͰςετ ৽ͨʹ௥Ճઢܗ૚Λֶशɻ&NCFEEJOH૚͚ͩ௿ϥϯΫߋ৽ ଞ͸ϑϦʔζ  w ݴ͍׵͑ˠ໊લҐஔ͔Βશଐੑ༧ଌՄೳɻφγˠϑϨʔζ΍ଐੑͷ૊Ͱ҉ه "OZB#SJBS'PSHFSXBTCPSOPO0DUPCFS 4IFTQFOUIFSFBSMZZFBSTJO1SJODFUPO

    /+ 4IFSFDFJWFENFOUPSTIJQBOEHVJEBODFGSPNGBDVMUZNFNCFSTBU.BTTBDIVTFUUT*OTUJUVUFPG 5FDIOPMPHZ4IFDPNQMFUFEIFSFEVDBUJPOXJUIBGPDVTPO$PNNVOJDBUJPOT4IFIBEB QSPGFTTJPOBMSPMFBU.FUB1MBUGPSNT4IFXBTFNQMPZFEJO.FOMP1BSL $" FH ݴ͍׵͑ΞϦ ݴ͍׵͑φγ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠ௚ޙͷଐੑ ʜCPSOPOˠ0DUPCFS ʜZFBSTJOˠ1SJODFUPO ̋ ̋ ଐੑ஋ͷ௚લҐஔ ɹ༧ଌˠҎ্ઌͷଐੑ ʜCPSOPOˠ1SJODFUPO ʜCPSOPOˠ.BTTBDIVTFUUT ʜZFBSTJOˠ.BTTBDIVTFUUT ̋ ✕ -BTUOBNFͷҐஔ ɹ༧ଌˠ࠷ॳͷଐੑ ʜ'PSHFSˠ0DUPSCFS ̋ ˚ -BTUOBNFͷҐஔ ɹ༧ଌˠ࢒Γͷଐੑ ʜ'PSHFSˠ1SJODFUPO ʜ'PSHFSˠ.BTTBDIVTFUUT ̋ ✕ ໼ҹ͕ଟ͘ͳΓ͗͢ΔͷͰҰ෦লུ DGj & ' 😊 😊 😊 😊 😊 😨 😨 😨 😊
  8. ·ͱΊ w ఻ههड़ 2"Ͱ܇࿅ͨ͠ͱ͖܇࿅2"ʹؚ·Εͳ͍఻هͷ2"͕ղ͚Δ͔ʁɹ ʮҾ͖ग़ͤΔ஌ࣝͱͯ͠هԱͰ͖ΔΑ͏ʹͳΔʯͷ͸Ͳ͏͍͏ͱ͖͔ʁ w ࠞ߹ͯ͠܇࿅͢Δͱ͖͸੒ޭ w ఻هࣄલ܇࿅ 2"

    fi OFUVOJOHͷೋஈ֊܇࿅Ͱ w ֤ਓ෺ͷ఻ه͕छྨͷදݱ͚ͩͷͱ͖͸ࣦഊ w ఻ه͕ଟ༷ʹݴ͍׵͑ΒΕͯग़ݱ͢Δͱ͖͸੒ޭ ผදݱςϯϓϨ΍--.Ͱੜ੒ ಛʹ༗ޮ จॱংมߋ ୅໊ࢺͷϑϧωʔϜԽ ຋༁ w ໊લτʔΫϯͷޙͷ࠷ऴ૚ʹؔ࿈஌͕ࣝൃ࿐͢ΔΑ͏ʹͳΔ w Ұఆ਺ͷਓ෺ʹ͍ͭͯݴ͍׵͑ 2"͕͋Ε͹શମʹ΋൚Խ͍ͯ͘͠ 😊 😊 😨
  9. ͓·͚ w #&35Ͱ͸ w ॱংϥϯμϜ͸͠ͳͯ͘΋ͦͦ͜͜Ͱ͖Δ͚Ͳ ΄ͱΜͲͷଐੑ͸ݴ͍׵͑ͯ͠΋ͦͦ͜͜  ࢭ·Γ w ໊લଐੑͷ݁ͼ͖ͭΑΓ1SJODFUPO㲗/+ͳͲͷ݁ͼ͖ͭʹཔͬͯ͠·͏

    w ஀ੜ೔ τʔΫϯ͕ޓ͍ʹಠཱ ΍ઐ߈ τʔΫϯ཰͕ߴ͍ ͸ਖ਼ղ཰͕ߴ͍ w Ϟσϧৄࡉ w (15TNBMM 3PUBSZ .MBZFST EJNT UPLFOJ[FS͸(15  --B.Bੜ੒σʔλ࣮ݧͰ͸ͪΐͬͱେ͖͘.  OFHBUJWFSFTVMUͷݕূ༻ʹ͸.Ͱ΋࣮ݧ DGj # )