Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya Hiroyuki Moriya
December 11, 2024
1
340

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya

Hiroyuki Moriya

December 11, 2024
Tweet

More Decks by Hiroyuki Moriya

See All by Hiroyuki Moriya
kueueに新しいPriorityClassを足した話
Avatar for Hiroyuki Moriya gekko0114
0
720
JobSet超入門
Avatar for Hiroyuki Moriya gekko0114
1
970

Featured

See All Featured
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
352
21k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
PRO
23
1.5k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
525
40k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
180
9.9k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
19
1.2k
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
53
9k
Visualization
Avatar for Eitan Lees eitanlees
148
16k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
209
24k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
34
6.1k

Transcript

  1. confidencial LLM؂ࢹͷ࠷લઢ IVRy ΤϯδχΞ๨೥LTେձ 2024/12/11 Moriya Hiroyuki

  2. confidencial 2 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ

  3. confidencial 3 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ

  4. confidencial 4 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ

  5. confidencial 5 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

  6. confidencial 6 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

    ؾ͕͍ͭͨΒɺԶͷϓϩμΫτ͸ղ໿͕૬࣍͗ɺձࣾ͸౗࢈ͯ͠͠·͍ͬͯͨ...

  7. confidencial 7 ׬

  8. confidencial 8 ࠓ೔͸ɺ޻౻৽Ұ܅͕ɺ͜Μͳ݁຤Λܴ͑ͳ͍ͨΊʹͰ͖Δ͜ͱΛ͓࿩͠͠·͢ɻ

  9. confidencial ࣗݾ঺հ 2024/08 ೖࣾ SWEɾػցֶशΤϯδχΞͳͲΛܦݧ LLM͕ίΞʹͳΓͦ͏ͳαʔϏεͩͱࢥͬͯIVRyʹೖࣾ Moriya Hiroyuki 9 AI

    engineer

  10. confidencial IVRyͰͷLLMΛར༻ͨ͠AIର࿩ 10 WebsocketΛར༻͠ΤϯυϢʔβʔͱLLM͕ϦΞϧλΠϜʹ΍ΓऔΓ͍ͯ͠Δ

  11. confidencial LLM Fallback 11 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚

  12. confidencial LLM Fallback 12 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚ ؂ࢹ͢Ε͹ ྑ͍ͷ͡Ό

  13. confidencial ํ๏ 1ɿDataDog LLM observability 13 DataDog͕Ӷҙ։ൃதͷLLM؂ࢹʹಛԽͨ͠ػೳɻ Latency, token਺, promptͳͲΛऔಘͰ͖Δɻ

  14. confidencial 14 ʮ͜ΕͰɺOpenAIͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  15. confidencial 15 ͋ΕΕʙɺ͓͔͍͠Αʙ

  16. confidencial 16 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻ΒͷϓϩμΫτ͸ɺfallbackػߏΛ ࣮૷͍ͯ͠Δͷʹɺ OpenAIͷlatency͔͠؂ࢹͰ͖ͯͳ͍Αʙ

  17. confidencial ํ๏ 2ɿOpenLIT (OpenTelemetry) 17 OpenTelemetryن֨ʹଇͬͨɺLLM؂ࢹʹಛԽͨ͠πʔϧɻ ༷ʑͳLLMΛ؂ࢹ͢Δ͜ͱ͕Ͱ͖Δɻ

  18. confidencial 18 ʮ͜ΕͰɺ৭ʑͳmodelͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  19. confidencial 19 ͋ΕΕʙɺ͓͔͍͠Αʙ

  20. confidencial 20 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺ৭ʑͳϞσϧΛ࢖͏Μ͔ͩΒɺ provider͝ͱʹɺlatencyΛܭଌ͢Δඞཁ͕͋Δͷʹ LiteLLMશମͰͷlatency͔͠औΕͯͳ͍Αʙ

  21. confidencial ํ๏ 3ɿDataDog Inferred services 21 DataDogʹ౥ࡌ͞ΕͨɺApp֎΁ͷϦΫΤετΛ؂ࢹͯ͘͠ΕΔػߏ

  22. confidencial 22 ʮ͜ΕͰɺLiteLLMͰ࢖͍ͬͯΔ͢΂ͯͷmodelΛ؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  23. confidencial 23 ͋ΕΕʙɺ͓͔͍͠Αʙ

  24. confidencial 24 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺGeminiɺOpenAIͰ̍ͭͷmodelΛ ࢖͏ͱ͸ݶΒͳ͍ͷʹɺ ݸผͷmodelͷlatencyΛऔಘ͢Δ͜ͱ͸ Ͱ͖ͯͳ͍Αʙ

  25. confidencial ·ͱΊ LLM؂ࢹ͸ɺ·ͩ·ͩൃల్্Ͱ஌ݟ͕͋Γ·ͤΜʂ AIɾLLMΛ࢖͍͜ͳͯ͠ϓϩμΫτʹೖΕ͍ͯ͘աఔͰɺ ࣗΒ͕੾Γ։͍͍ͯ͘ඞཁ͕͋Γ·͢ɻ ͥͻҰॹʹAI؂ࢹΛ΍͍͖ͬͯ·͠ΐ͏ʂ 25