Share
Embed
【輪講資料】ReAct: Synergizing Reasoning and Acting in Language Models / Tree of Thoughts: Deliberate Problem Solving with Large Language Models
研究室の輪講で発表したときの資料です。以下の2本の論文を紹介しています。
- ReAct: Synergizing Reasoning and Acting in Language Models
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yano
October 31, 2023
Transcript
48 slides
-
10݄31 ాɾݚڀࣨɹM2 ઍߛ ReAct: Synergizing Reasoning and Acting in Language
Models Tree of Thoughts: Deliberate Problem Solving with Large Language Models Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Grif fi ths, Yuan Cao, Karthik Narasimhan Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao ICLR 2023 NeurIPS 2023 -
֓؍ • ReAct • ਪͱߦಈΛަޓʹੜ͢Δ • ߦಈͰ֎෦πʔϧΛར༻͢Δ͜ͱͰHallucinationΛ͙͜ͱ͕Ͱ͖Δ • Tree of
Thougt • ޙΓઌಡΈΛߦ͍ɺख़ߟ͢ΔࣄͰCoTͳͲͰ͔ͬͨ͠ෳࡶͳਪΛ ߦ͏͜ͱ͕Ͱ͖Δ 3 -
ReAct: Synergizing Reasoning and Acting in Language Models • LLMʹਪͱߦಈΛަޓʹੜͤ͞Δϓϩϯϓτख๏ʮReActʯͷఏҊ
• ਪͰߦಈΛܭըͨ͠Γɺߦಈͷ݁ՌΛॲཧͨ͠Γ͢Δ • ߦಈͰࣝϕʔεڥʹΞΫηε͠ใΛಘΔ • ֎෦͔Βͷਖ਼͍͠ใΛར༻͢Δ͜ͱͰHallucinationΛ੍ • Hallucination: LLM͕ͬͱΒ͍͠ӕΛग़ྗ͢Δ͜ͱ 5 2ϋϩΟϯҎ্લ͔Βଓ͘ࡇΓͰ͔͢ʁ ·ͣϋϩΟ ϯͷىݯΛௐ Α͏ ϋϩΟϯىݯ ϋϩΟϯͷൃ Ҏ্લɻʜ લ Ҏ্લ ͔ͩΒ͑ ʮ͍ʯ ਪ ߦಈ ਪ -
REACTͷਪྫ 6 2ʰԾ໘ϥΠμʔిԦʱͱʰ͋·ͪΌΜʱɺ์ૹճ͕ଟ͍ͷͲͪΒͰ͠ΐ͏ʁ ʰԾ໘ϥΠμʔిԦʱͷ์ૹճΛݕࡧͨ͠ޙɺʰ͋·ͪΌΜʱͷ์ૹճΛݕࡧ͠ɺͦΕΒͷൺֱΛߦ͏ɻ 4FBSDI<ʰԾ໘ϥΠμʔిԦʱͷ์ૹճԿճʁ> ਪ ߦಈ ʰԾ໘ϥΠμʔిԦʱͷ์ૹճճͰ͋Δɻ࣍ʹʰ͋·ͪΌΜʱͷ์ૹճΛݕࡧ͢Δɻ 4FBSDI<ʰ͋·ͪΌΜʱͷ์ૹճԿճʁ> ʰԾ໘ϥΠμʔిԦʱͷ์ૹճճͰ͢ɻ
ʰ͋·ͪΌΜʱͷ์ૹճճͰ͢ɻ ʰԾ໘ϥΠμʔిԦʱͷ์ૹճճͰ͋Δɻʰ͋·ͪΌΜʱͷ์ૹճճͰ͋Δɻ ʰ͋·ͪΌΜʱͷ์ૹճ͕ଟ͍ͷͰɺ࠷ऴతͳ͑ʰ͋·ͪΌΜʱͰ͋Δɻ 'JOJTI<ʰ͋·ͪΌΜʱ> +&.)PQ2"ΑΓҾ༻ ਪ ߦಈ ਪ ߦಈ -
ࣝूతͳਪλεΫ HotPotQA • ଟஈ֊ਪλεΫ • ̎ͭҎ্ͷWikipediaͷهࣄʹର͢ΔਪΛඞཁͱ͢Δ • ྫ) Which magazine
was started fi rst Arthur's Magazine or First for Women? Fever • ࣄ࣮ݕূλεΫ • ओுʹର͠ɺWikipediaͷهࣄΛࢀর͠[ Supported, Refuted NotEnoughInfo ]ͷϥϕϧ Λ͚ͭΔ • ྫʣNikolaj Coster-Waldau worked with the Fox Broadcasting Company. -Supported 7 -
ࣝूతͳਪλεΫͰͷߦಈۭؒ • Wikipedia͔ΒใΛ֫ಘ͢ΔͨΊͷߦಈۭؒΛఆٛ • ҎԼͷ3ͭͷΞΫγϣϯ • search[entity]: entityͷWikipediaϖʔδ͕͋Ε࠷ॳͷ5จΛɺແ ͚ΕྨࣅͷΤϯςΟςΟΛ5ͭฦ͢ •
lookup[string]: stringΛؚΉจΛฦ͢ (Ctrl+F) • fi nish[answer]: answerΛ͑ͱͯ͠ݱࡏͷλεΫΛऴྃ͢Δ 8 -
ൺֱϕʔεϥΠϯ Chain of Thought (CoT)[1] • தؒͷਪεςοϓग़ྗͤ͞Δ͜ͱͰɺෳࡶͳਪ͕Մೳ • ߦಈΛར༻ͤͣɺਪͷΈΛߦ͏ʢϞσϧ෦ͷใͷΈར༻ʣ Chain
of Thought self-consistency (CoT-SC)[2] • ෳͷCoTʹΑͬͯଟܾ͢Δ • ԹมΛ0.7ʹઃఆͨ͠CoTΛ21ճߦ͍ɺ ղΛଟܾͰܾఆ͢Δ 11 <>$IBJOPG5IPVHIU1SPNQUJOH&MJDJUT3FBTPOJOHJO-BSHF-BOHVBHF.PEFMT <>4FMG$POTJTUFODZ*NQSPWFT$IBJOPG5IPVHIU3FBTPOJOHJO-BOHVBHF.PEFMT -
ࣝूతͳਪλεΫͷ࣮ݧઃఆ • ϞσϧʹPaLM-540BΛར༻ • HotpotQA • ֶशηοτͷࣄྫΛਓखͰΞϊςʔγϣϯͯ͠6-shot • Fever •
ֶशηοτͷࣄྫΛਓखͰΞϊςʔγϣϯͯ͠3-shot 13 -
ReAct vs CoT @HotpotQA • CoTͷHallucinationReActΑΓਂࠁ (False positive, Hallucination) •
ReActਪεςοϓ͕ॊೈͰͳ͍ (Reasoning error) • ಉ͡ਪɺߦಈΛ܁Γฦ͢ϧʔϓʹϋϚΔ͜ͱ͕ଟ͍ • ReActݕࡧͰࣦഊ͢Δͱਪ͕ઢͯ͠͠·͏ (Search result error) 16 ˛)PUQPU2"Ͱͷޭɺࣦഊࣄྫ͔ΒϥϯμϜʹநग़͠ɺਓखͰݪҼΛௐࠪ -
෦ࣝͱ֎෦ࣝͷΈ߹Θͤ • CoTਪεςοϓͷߏஙΑΓਖ਼͕֬ͩɺ෦ࣝΛར༻͢Δ͜ͱͰ Hallucination͕ൃੜ͢Δ • ReAct֎෦ࣝΛར༻͠ਖ਼͕֬ͩɺਪεςοϓͷߏஙʹ͓͍ͯCoTΑΓྼΔ ➡ҎԼͷܦݧଇʹج͍ͮͯCoT-SCͱReActΛΓସ͑Δ • ReAct→CoT-SC: نఆͷεςοϓͰղ͕ಘΒΕͳ͍߹
• ReActͷਪεςοϓΛ͗ͯ͘͢͠ੑೳ্͕Βͳ͍ • CoT-SC→ReAct: ଟܾͷଟ͕աΛ͑ͳ͍߹ • CoT͕ࣗ৴Λ࣋ͬͯղͰ͖͍ͯͳ͍ 17 )PUQPU2"Ͱεςοϓ 'FWFSͰεςοϓ -
ࣝूతͳਪλεΫͰͷ࣮ݧ݁Ռ • ReAct+CoT-SC > CoT-SC, ReAct • CoT-SC (sample=21) <
ReAct+CoT-SC (sample=5) • ෦ࣝͱ֎෦ࣝΛΈ߹ΘͤΔ͜ͱ༗༻ 18 ˛$P54$Ͱͷαϯϓϧ ଟܾͷ ͱੑೳ -
ҙࢥܾఆλεΫ ALFWorld • ՈͷதΛςΩετૢ࡞ʹΑͬͯΤʔδΣϯτ͕୳ࡧ͠ɺෳࡶͳ̒छྨͷλεΫΛ ୡ͢Δ • ߦಈͷબࢶ50छྨҎ্͋Γɺ૯ͨΓతʹ୳ࡧ͢Δ͜ͱࠔ • ྫ) You
are in the middle of a room. Looking quickly around you, you see a drawer 2, a shelf 5,…, and a drawer 4. Your task is to: put some vase in safe. • vasesafeೖྗʹଘࡏͤͣɺৗࣝΛར༻ͯ͠୳ࡧ͠ͳ͚ΕͳΒͳ͍ (Ֆළ୨ͷ্ʹ͋Γͦ͏) WebShop 20 <> <>8FC4IPQ5PXBSET4DBMBCMF3FBM8PSME8FC*OUFSBDUJPOXJUI(SPVOEFE-BOHVBHF"HFOUT -
ҙࢥܾఆλεΫ ALFWorld WebShop • ݱ࣮ͷΛར༻ͨ͠ΦϯϥΠϯγϣοϐϯάڥͰϢʔβʔͷࢦࣔʹج ͍ͮͯΛߪೖ͢ΔλεΫ • ΤʔδΣϯτݕࡧʮΔʯͳͲͷϘλϯΛར༻Ͱ͖Δ • Score
(ճ͕Ͳͷఔཁ݅Λຬ͍ͨͯ͠Δ͔)ͱSuccess rate (શͯͷཁ݅ Λຬͨͨ͠Λճׂͨ͠߹)ͰධՁ • ྫ) get me a sixteen pack of apple cinnamon freeze dried banana chips, and price lower than 50.00 dollars 21 -
ൺֱϕʔεϥΠϯ ALFWorld • BUTLER[1]: λεΫछຖʹ100000݅ͷσϞͰ฿ֶशͨ͠Ϟσϧ • ReAct-IM: ReActΛInner MonologueελΠϧ[2]ʹͨ͠ͷ •
ΤʔδΣϯτ͕؍ଌͨ͠ڥͱɺୡ͕ඞཁͳখඪΛInner Monologue ͱͯ͠ੜ͢Δ WebShop • IL: 1012݅ͷσϞͰ฿ֶशͨ͠Ϟσϧ • ILʴRL: 10587݅ͷσϞͰ฿ֶशʴڧԽֶशͨ͠Ϟσϧ 22 <>"-'8PSME"MJHOJOH5FYUBOE&NCPEJFE&OWJSPONFOUTGPS*OUFSBDUJWF-FBSOJOH <>*OOFS.POPMPHVF&NCPEJFE3FBTPOJOHUISPVHI1MBOOJOHXJUI-BOHVBHF.PEFMT -
ҙࢥܾఆλεΫͷ࣮ݧ݁Ռ • ReAct > Act • ActඪͷղͱڥΛѲ͕Ͱ͖ͳ͍ • ฏۉతʹReAct >
BUTLER • େྔͷֶशσʔλΛར༻͢Δख๏ʹউར 24 ˛"-'8PSMEʹ͓͚ΔλεΫछྨຖͷޭ -
ҙࢥܾఆλεΫͷ࣮ݧ݁Ռ • ReAct > ReAct-IM • Inner MonologueελΠϧͰڥͷ؍ଌͱୡ͖͢খඪʹ͍ͭͯͷΈੜ • ্खʹৗࣝΛద༻ͯ͠ɺਪͷεςοϓΛߏங͢Δ͜ͱ͕͍͠
• ྫ) task: put a clean knife in countertop. • think: To solve the task, I need to fi nd and take a clean knife, then put it in countertop. • φΠϑΛݟ͚ͭΔͷͷɺͦΕ͕៉ྷͩͱ৴ͯ͡Χϯλʔʹஔ͖ଓ͚Δ 25 ˛"-'8PSMEʹ͓͚ΔλεΫछྨຖͷޭ -
ҙࢥܾఆλεΫͷ࣮ݧ݁Ռ • ReAct > Act > IL, IL+RL • ReActϊΠζͷଟ͍؍ଌ͔Β͏·࣍͘ͷߦಈΛੜͰ͖͍ͯΔ
• For ‘space-saving ottoman bench for living room’, the item has options ‘39x18x18inch’ and ‘blue’ and seems good to buy. 26 ˛8FC4IPQͰͷείΞ -
ReAct: Synergizing Reasoning and Acting in Language Models • LLMʹਪͱߦಈΛަޓʹੜͤ͞Δϓϩϯϓτख๏ͷఏҊ
• ਪͰ࣍ͷߦಈΛܭը͠ɺߦಈͷ݁ՌΛॲཧ͢Δ • ߦಈͰࣝϕʔεڥʹΞΫηε͠ɺՃͷใΛಘΔ • ࣭ԠλεΫͱࣄ࣮ݕূλεΫʹ͓͍ͯɺWikipediaAPIͱର͠ɺ HallucinationΛࠀ • 2ͭͷରܕҙࢥܾఆϕϯνϚʔΫʹ͓͍ͯɺେྔͷֶशσʔλΛར༻͠ ͨ฿ֶशͱڧԽֶशʹΑΔϕʔεϥΠϯΛ1-shot/2-shotͷઃఆͰେ෯ʹ ্ճͬͨ 27 -
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• ݴޠϞσϧͷਪࠨ͔Βӈʹݶఆ • ઌಡΈΛඞཁͱͨ͠Γɺ࠷ॳͷܾఆ͕ ۃΊͯॏཁͳλεΫͰࣦഊ ➡ղܾͷதؒஈ֊ (Thought)Λ୳ࡧ Մೳͱ͢ΔϑϨʔϜϫʔΫɺ ʮTree of ThoughtʯΛఏҊ • ෳͷҟͳΔਪܦ࿏Λߟྀ͠ɺ࣍ͷ ߦಈΛܾఆ͢ΔͨΊʹࣗݾධՁΛߦ͏ ʢख़ߟ͢Δʣ 29 -
Tree of Thoughtͷਪྫ 31 ҎԼͷࣈͱ࢛ଇԋࢉΛΈ߹ΘͤͯΛ࡞͍ͬͯͩ͘͞ Y Y
Y *OQVU ⭕ ⭕ ⭕ ⭕ ✖︎ ✖︎ ✖︎ Ͳͷਪ͕ ༗͔ͳʜ -
Tree of Thought • ෳࡶͳਪΛখʹղ͠ɺͲͷਪܦ࿏͕ے͕ྑ͍͔ධՁΛ ߦ͍ͳ͕Β୳ࡧ͢Δ • λεΫຖʹมߋՄೳͳҎԼͷ4ͭͷΦϓγϣϯ A. ࢥߟΛղ͢Δେ͖͞
B. ࢥߟͷੜํ๏: ࠓ·Ͱͷࢥߟೖྗ͢Δ/ ͠ͳ͍ C. ֤ϊʔυͷධՁํ๏: ಠཱʹఆྔԽ / બࢶΛൺֱ • ઌಡΈͱৗࣝΛར༻ D. ͷ୳ࡧํ๏: BFS / DFS 32 -
ઌಡΈ୳ࡧΛඞཁͱ͢ΔఏҊλεΫ Game of 24 • ༩͑ΒΕͨ4ͭͷࣈʹ࢛ଇԋࢉΛద༻ͯ͠24Λ࡞Δ Creative Writing • ༩͑ΒΕͨ4ͭͷจ͕ஈམͷ࠷ޙͷจͱͳΔΑ͏ͳ4ஈམߏͷจষΛग़ྗ
• ੜͨ͠จষʹҰ؏ੑ͕͋Δ͔ɺGPT4ͱਓखͰධՁ 33 -
Game of 24ͰͷϕʔεϥΠϯ IO • Λೖྗͱ͠ɺղΛग़ྗͱ͢Δඪ४తͳϓϩϯϓτ (5-shot) CoT • 4ͭͷ͔Β24Λ࡞ΔͨΊͷదͳ࢛ଇԋࢉͷࣜΛ༩͑Δ
(5-shot) • 13 - 9 = 4 (left: 4 4 10); 10 - 4 = 6 (left: 4 6); 4 * 6 = 24 (left: 24) CoT-SC • 100ճCoTͰͷਪΛ࣮ߦ͠ɺଟܾΛऔΔ IO + re fi ne • 10ճ෮ͯ͠ਪ͢Δ • ؒҧ͍͑ͯͨ߹मਖ਼͢ΔΑ͏ͳࢦࣔͱཤྺΛ༩͑Δ 35 -
Game of 24ͰͷToTηοτΞοϓ 36 • P32ʹࣔͨ͠ΦϓγϣϯҎԼͷΑ͏ʹͳΔ A. ࢥߟΛղ͢Δେ͖͞ɿ̍ͷ࢛ଇԋࢉ B. ࢥߟͷੜํ๏ɿࠓ·Ͱͷࢥߟೖྗͤͣɺ୯७ͳࣜͷੜͷΈߦ͏
C. ֤ϊʔυͷධՁํ๏ɿ24ʹ౸ୡͰ͖Δ͔ʁ- [ sure/maybe/impossible ] • ৗࣝΛར༻͠ɺ24ʹରͯ͠େ͖͗ͨ͢Γখ͗͢͞ΔީิΛল͘ D. ͷ୳ࡧํ๏ɿ෯༏ઌ୳ࡧ -
Game of 24Ͱͷ݁Ռ • ToTͷ๚ࡁϊʔυͰͷޭ͕ߴ͍ • IO, CoTͰͦΕͧΕͷ࣮ߦશମΛ ϊʔυͱͯ͠ܭࢉ •
(ےྑ͘୳ࡧͰ͖͍ͯΔ) 39 • CoTͰ60%͕࠷ॳͷࣜੜ࣌Ͱ ਖ਼ʹࣦഊ • ޙΓઌಡΈ͕Ͱ͖ͳ͍ख๏ʹ ݶք͕͋Δ ˛๚ࡁΈϊʔυͱͦͷਖ਼ղ ˛εςοϓຖͷࣦഊ -
Creative writingͰͷϕʔεϥΠϯ IO • จΛೖྗͱ͠จষΛग़ྗ͢Δඪ४తͳϓϩϯϓτ CoT • ઌʹ͍ܭըΛॻ͍͔ͯΒจষΛॻ͔ͤΔ IO+re fi
ne • 5ճ·Ͱ෮͢Δ • Ұ؏ੑʹ͚ܽΔ߹मਖ਼͢ΔΑ͏ͳࢦࣔͱཤྺΛ༩͑Δ 40 -
Creative writingͰͷToTηοτΞοϓ 41 • P32ʹࣔͨ͠ΦϓγϣϯҎԼͷΑ͏ʹͳΔ A. ࢥߟΛղ͢Δେ͖͞ɿͲͷΑ͏ͳจষΛهड़͢Δ͔ͷܭը / จষͷهड़ B.
ࢥߟͷੜํ๏ɿࠓ·Ͱͷࢥߟ (ܭը) Λར༻ͯ͠จষΛهड़͢Δ C. ֤ϊʔυͷධՁํ๏ɿ֤ࢥߟಉ࣌ʹ5ͭੜ͞Εɺ࠷ྑ͍ͷΛબͿ D. ͷ୳ࡧํ๏ɿਂ͞=2, ෯=1 • ܭըͱจষͷهड़Λߦ͏ͨΊਂ͞2ɺධՁ͝ͱʹ࠷ྑͷࢥߟͷΈͨ͢Ί෯1 -
Mini CrosswordsͰͷϕʔεϥΠϯ IO • ख͕͔ΓΛೖྗͱͯ͠ΫϩεϫʔυΛग़ྗͱ͢Δ(5-shot) 44 4PMWFYNJOJDSPTTXPSET(JWFOBOJOQVUPGIPSJ[POUBMDMVFTBOEWFSUJDBMDMVFT HFOFSBUFBOPVUQVUPGSPXT XIFSFFBDISPXJTMFUUFSTFQBSBUFECZTQBDF
*OQVU I"MVOBSWBMMFZ I"GBUUZPJM I5PFOUJDF I5PMPXFSUPSFEVDF I"TPMJUBSZQFSTPO W"DDPSEJOHUPUIFSPTUFS W"OPUIFSOBNFGPS1PSU'SBODRVJ W"OJMMJDJUMPWFSB&VSPQFBOMBLF W5PMJTQ W5PDPNFJO 0VUQVU 3*--& 0-&*/ 5&.15 "#"4& -0/&3 CoT • ख͕͔ΓͱରԠ͢Δ୯ޠΛࣔ͢ (5-shot) -
Mini CrosswordsͰͷToTηοτΞοϓ 45 • P32ʹࣔͨ͠ΦϓγϣϯҎԼͷΑ͏ʹͳΔ A. ࢥߟΛղ͢Δେ͖͞ɿख͕͔Γ͔Β୯ޠΛਪଌ B. ࢥߟͷੜํ๏ɿࠓ·ͰͷࢥߟΛར༻ͯ͠୯ޠΛਪଌ C.
֤ϊʔυͷධՁํ๏ɿଞͷख͕͔Γͷ୯ޠͷअຐʹͳΒͳ͍͔ʁ - [ sure/maybe/impossible ] • ଞͷ୯ޠ͕ຒΊΒΕͳ͘ͳͬͨ (impossible) ߹લͷࢥߟʹΔ D. ͷ୳ࡧํ๏ɿਂ͞༏ઌ୳ࡧ -
Mini CrosswordsͰͷ݁Ռ • IO, CoTͱൺֱͯ͠ToT͔ͳΓੑೳ͕ྑ͍ • ࢥߟΛධՁ͢Δ෦ͷOracleઃఆͰߋʹੑ ೳ্͕͕͓ͬͯΓɺධՁʹվળͷ༨͕͋ Δ (best
state) • ݱঢ়ͷධՁͰղܾͰ͖Δީิ୯ޠͱख͕͔ ΓͷΈ߹ΘͤͰimpossible͕ͭ͘͜ͱ͕ ͋Δ 47 -
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• ղܾͷதؒஈ֊ (Thought)Λ୳ ࡧՄೳͱ͢ΔϑϨʔϜϫʔΫɺ ʮTree of ThoughtʯΛఏҊ • ෳͷҟͳΔਪܦ࿏Λߟྀ͠ɺ࣍ ͷߦಈΛܾఆ͢ΔͨΊʹࣗݾධՁΛ ߦ͏ʢख़ߟ͢Δʣ • ఏҊͨ̏ͭ͠ͷઌಡΈ୳ࡧ͕ඞཁ ͳλεΫʹ͓͍ͯେ෯ʹੑೳ্ 48