Defensive LLM Calling

Transcript

1. Kyoto Tech Talk #9 2025/12/05 Defensive LLM Calling Katsuya Ogata
2. ࣗݾ঺հ • ໊લ: ॹํ ࠀ࠸ʢ͓͕ͨ ͔ͭ΍ʣ • ॴଐ: େࡕେֶ ৘ใՊֶݚڀՊ • झຯ: ొࢁɾཱྀߦɾα΢φ • MLLMΛ༻͍ͨಈըղੳͷݚڀ • ϝϧϘϧϯʹߦͬͯ·ͨ͠ • X: @ogata_katsuya • ݸਓ: www.ogatakatsuya.com ϝϧϘϧϯཱྀߦ ޒ৭ϲݪ΁ͷొࢁ
4. AWS 2025/10/20 Google Cloud 2025/06/13 Azure 2025/10/29
5. Ϋϥ΢υ͸յΕΔલఏͰར༻͢Δ
6. 2025೥ʹো֐͕ى͖ͯͨͷ͸ʁ
7. શ෦
8. ো֐ͷݪҼͱ಺༰ • ݪҼ • AIΛϗετ͍ͯ͠ΔΫϥ΢υͷো֐ • ίϯύΠϥͷόάʢΠϯϑϥىҼʣ • ಺༰ • ϨΠςϯγͷٸܹͳ૿Ճ • ਫ਼౓ͷ௿Լ • ͦ΋ͦ΋LLMΛݺͼग़͢͜ͱ͕Ͱ͖ͳ͍
9. LLM΋յΕΔલఏͰར༻͢Δ
10. 🤔
11. Application Layer Transport Layer Network Layer Network Access Layer TCP/IP
12. τϥϯεϙʔτ૚ • ΞυϨε͸ϙʔτ൪߸ͷΈʢIPΛ஌ͬͯΔͷ͸ωοτϫʔΫ૚ʣ • ͲΜͳωοτϫʔΫͷܦ࿏Λ௨Δ͔͸஌Βͳ͍ • ωοτϫʔΫͦͷ΋ͷΛ৴པ͍ͯ͠ͳ͍લఏͰ࣮֬ʹσʔλΛૹΓಧ͚Δ • ࠶ૹ੍ޚɺ᫔᫓੍ޚɺޡΓݕग़ • ʢ๻ͷߟ͑Δʣτϥϯεϙʔτ૚ͷ໾ׂ • ΞϓϦέʔγϣϯ૚ʹωοτϫʔΫͷ੾ΓޱΛݟͤΔ • ΞϓϦέʔγϣϯ͸ωοτϫʔΫͷҧ͍Λҙࣝ͠ͳͯ͘ྑ͍ • ωοτϫʔΫͦͷ΋ͷͷ৴པੑΛ୲อ͢Δ
13. 🤔
14. Application Layer Transport Layer Network Layer Network Access Layer TCP/IP
15. Application Endpoint ????? LLM Calling LLM Calling ͜͜
16. LLM Callingʹ͓͚Δτϥϯεϙʔτ૚ • LLM Providerͷҧ͍Λٵऩͯ͠ɺͦͷ੾Γޱ͚ͩΛݟͤΔ • ந৅Խ • Ӆṭ • LLM ProviderΛ৴པ͍ͯ͠ͳ͍લఏͰ඼࣭Λ୲อ͢Δ • ϦτϥΠॲཧ • ଞͷProvider΁ͷϑΥʔϧόοΫ • ඞͣԿ͔͠ΒͷϨεϙϯε͕ฦͬͯ͘Δ
17. PythonͰͷ࣮૷ྫ abcΛ༻͍ͨந৅Ϋϥεͷ࡞੒ LiteLLMͱ͍͏ศརϥΠϒϥϦ΋͋Γ·͢
18. PythonͰͷ࣮૷ྫ ֤ϓϩόΠμʔͰͷந৅ϝιουͷ࣮૷
19. PythonͰͷ࣮૷ྫ LLM Callingͷτϥϯεϙʔτ૚(͔ͳΓ୺ંͬͯ·͢)
20. LLM Caller໨ઢ • τϥϯεϙʔτ૚ʹඞཁͳύϥϝʔλΛ౉͚ͩ͢ • ͲΜͳϓϩόΠμͰ࣮ߦ͞ΕΔ͔͸஌Βͳ͍ • ΠϯλʔϑΣʔε͚ͩΛҙࣝͯ͠ɺͦΕҎԼΛؾʹ͠ͳ͍ • ͚ͩͲɺҰఆ඼࣭͕อͨΕͯɺ͋Δఔ౓ͷ଎౓Ͱฦ౴͕ฦͬͯ͘Δ
21. ສࣄղܾʁʁʁ • LLMͷো֐͸ૉ௚ʹΤϥʔΛు͍ͯམͪͯ͘Εͳ͍ • Ϩεϙϯε͕500൪୆ͳΒେৎ෉ • ϨΠςϯγ͕૿Ճ͢Δ • λΠϚʔͱϦτϥΠॲཧͰͳΜͱ͔ͳΔʢ͔΋ʣ • ਫ਼౓͕མͪΔ • ো֐Λײ஌͢Δ͜ͱ͕೉͍͠ • ࠷ऴతʹ͸ɺ͜·Ίͳ؂ࢹͱɺϢʔβʔͷ੠ʹৗʹහײͰ͋Δ͜ͱ͕ॏཁ • Langfuse΍LangsmithͳͲͷLLM؂ࢹମ੍ͷ੔උ΋ඞཁ
22. ·ͱΊ • ͜ͷLTʹ໌֬ͳ౴͑͸͋Γ·ͤΜ • LLMͷ඼࣭Λ୲อ͢Δͷ͸ϓϩόΠμ͸΋ͪΖΜɺϢʔβʔ΋੹຿Λෛ͍ͬͯ Δͱ͍͏໰୊ఏى • ࠷௿ݶఏڙ͢΂͖αʔϏεͷ඼࣭Λ୲อ͢ΔͨΊͷϑΥʔϧόοΫઓུΛߟ͑ ͍ͯ͘ඞཁ͕͋Δ • օ͞Μ͕࣮ફ͍ͯ͠ΔϑΥʔϧόοΫઓུɾख๏Λڭ͍͑ͯͩ͘͞
23. Reference • https://azure.status.microsoft/en-us/status/history/ • https://www.youtube.com/watch?v=D297rfqSFvY • https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW • https://www.mobihealthnews.com/news/google-cloud-outage-disrupts-ai- services-used-healthcare • https://www.youtube.com/watch?v=Ar_ckPSwe1Y • https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues