Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya Hiroyuki Moriya
December 11, 2024
1
320

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya

Hiroyuki Moriya

December 11, 2024
Tweet

More Decks by Hiroyuki Moriya

See All by Hiroyuki Moriya
kueueに新しいPriorityClassを足した話
Avatar for Hiroyuki Moriya gekko0114
0
700
JobSet超入門
Avatar for Hiroyuki Moriya gekko0114
1
920

Featured

See All Featured
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
270
20k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
329
21k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
10
660
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
32
2.3k
Done Done
Avatar for chrislema chrislema
184
16k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
63
7.8k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
10
930
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
161
15k
It's Worth the Effort
Avatar for 3n 3n
185
28k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
330
24k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
72
4.9k

Transcript

  1. confidencial LLM؂ࢹͷ࠷લઢ IVRy ΤϯδχΞ๨೥LTେձ 2024/12/11 Moriya Hiroyuki

  2. confidencial 2 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ

  3. confidencial 3 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ

  4. confidencial 4 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ

  5. confidencial 5 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

  6. confidencial 6 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

    ؾ͕͍ͭͨΒɺԶͷϓϩμΫτ͸ղ໿͕૬࣍͗ɺձࣾ͸౗࢈ͯ͠͠·͍ͬͯͨ...

  7. confidencial 7 ׬

  8. confidencial 8 ࠓ೔͸ɺ޻౻৽Ұ܅͕ɺ͜Μͳ݁຤Λܴ͑ͳ͍ͨΊʹͰ͖Δ͜ͱΛ͓࿩͠͠·͢ɻ

  9. confidencial ࣗݾ঺հ 2024/08 ೖࣾ SWEɾػցֶशΤϯδχΞͳͲΛܦݧ LLM͕ίΞʹͳΓͦ͏ͳαʔϏεͩͱࢥͬͯIVRyʹೖࣾ Moriya Hiroyuki 9 AI

    engineer

  10. confidencial IVRyͰͷLLMΛར༻ͨ͠AIର࿩ 10 WebsocketΛར༻͠ΤϯυϢʔβʔͱLLM͕ϦΞϧλΠϜʹ΍ΓऔΓ͍ͯ͠Δ

  11. confidencial LLM Fallback 11 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚

  12. confidencial LLM Fallback 12 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚ ؂ࢹ͢Ε͹ ྑ͍ͷ͡Ό

  13. confidencial ํ๏ 1ɿDataDog LLM observability 13 DataDog͕Ӷҙ։ൃதͷLLM؂ࢹʹಛԽͨ͠ػೳɻ Latency, token਺, promptͳͲΛऔಘͰ͖Δɻ

  14. confidencial 14 ʮ͜ΕͰɺOpenAIͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  15. confidencial 15 ͋ΕΕʙɺ͓͔͍͠Αʙ

  16. confidencial 16 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻ΒͷϓϩμΫτ͸ɺfallbackػߏΛ ࣮૷͍ͯ͠Δͷʹɺ OpenAIͷlatency͔͠؂ࢹͰ͖ͯͳ͍Αʙ

  17. confidencial ํ๏ 2ɿOpenLIT (OpenTelemetry) 17 OpenTelemetryن֨ʹଇͬͨɺLLM؂ࢹʹಛԽͨ͠πʔϧɻ ༷ʑͳLLMΛ؂ࢹ͢Δ͜ͱ͕Ͱ͖Δɻ

  18. confidencial 18 ʮ͜ΕͰɺ৭ʑͳmodelͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  19. confidencial 19 ͋ΕΕʙɺ͓͔͍͠Αʙ

  20. confidencial 20 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺ৭ʑͳϞσϧΛ࢖͏Μ͔ͩΒɺ provider͝ͱʹɺlatencyΛܭଌ͢Δඞཁ͕͋Δͷʹ LiteLLMશମͰͷlatency͔͠औΕͯͳ͍Αʙ

  21. confidencial ํ๏ 3ɿDataDog Inferred services 21 DataDogʹ౥ࡌ͞ΕͨɺApp֎΁ͷϦΫΤετΛ؂ࢹͯ͘͠ΕΔػߏ

  22. confidencial 22 ʮ͜ΕͰɺLiteLLMͰ࢖͍ͬͯΔ͢΂ͯͷmodelΛ؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  23. confidencial 23 ͋ΕΕʙɺ͓͔͍͠Αʙ

  24. confidencial 24 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺGeminiɺOpenAIͰ̍ͭͷmodelΛ ࢖͏ͱ͸ݶΒͳ͍ͷʹɺ ݸผͷmodelͷlatencyΛऔಘ͢Δ͜ͱ͸ Ͱ͖ͯͳ͍Αʙ

  25. confidencial ·ͱΊ LLM؂ࢹ͸ɺ·ͩ·ͩൃల్্Ͱ஌ݟ͕͋Γ·ͤΜʂ AIɾLLMΛ࢖͍͜ͳͯ͠ϓϩμΫτʹೖΕ͍ͯ͘աఔͰɺ ࣗΒ͕੾Γ։͍͍ͯ͘ඞཁ͕͋Γ·͢ɻ ͥͻҰॹʹAI؂ࢹΛ΍͍͖ͬͯ·͠ΐ͏ʂ 25