Observability for LLM Application lifecycle

2025/08/20に開催された「Japan Datadog User Group Meetup#12@東京」に、IVRyのHiroyuki Moriyaが登壇しました。

■リンク集
・コーポレートサイト：https://ivry.jp/company/
・採用ページ：https://ivry-jp.notion.site/
・イベント一覧：https://ivry.connpass.com/event/
・IVRy Tech（Xアカウント）：https://x.com/IVRy_tech

Transcript

1. Observability for LLM Application lifecycle JDDUG #12 Hiroyuki Moriya AI Engineer IVRy.Inc. 2025/08/20
2. 2 LLMs can do anything. Easily build amazing products. Perfect self-driving cars within a few years. Automate everything so humans don’t need to work anymore.
3. 3 Easily build amazing products. Perfect self-driving cars within a few years. Automate everything so humans don’t need to work anymore. LLMs can do anything. Not really.
4. 4 What is necessary for developing LLM-powered service? Today’s Topic
5. 5 About me Develop products that integrate LLM APIs Monitor & optimize LLM APIs for reliability and performance Hiroyuki Moriya AI engineer Speaker Intro
6. 6 About IVRy Challenges Solutions Recap & Tips Agenda
7. 7 Founded/HQ: 2019. Tokyo, Japan Number of employees: 200+ people Product: AI/LLM based phone communication service Reach: 30,000+ accounts & 40 million+ incoming calls in total IVRy inc. Company Info “Revolutionizing the telephone experience and boosting productivity for businesses ”
8. 8 AI-powered automated phone service Our Product
9. 9 Simplified system architecture
10. 10 About IVRy Challenges Solutions Recap & Tips Agenda
11. 11 Three steps for LLM product lifecycle Stg Prd Dev
12. 12 Three steps for LLM product lifecycle Stg Prd Dev Difficult to manage many experiments Difficult to enhance output consistency Difficult to stabilize LLM product
13. 13 About IVRy Challenges Solutions Recap & Tips Agenda
14. 14 Development
15. 15 Stg Prd Dev Diicult to manage many experiments Problem
16. 16 Diicult to manage many experiments Problem
17. 17 Management by Datadog Experiments (Though we don’t utilize this tool not so much) Solution
18. 18 Staging
19. 19 Stg Prd Dev Problem Diicult to enhance output consistency
20. 20 Outputs from LLM APIs can change for many reasons Problem Output has changed
21. 21 Automated e2e testing Solution
22. 22 Executing phone E2E tests after code merge Merge code Deploy latest code Execute automated phone E2E tests Monitor on Datadog LLM Observability
23. 23 Monitoring with Datadog LLM observability
24. 24 Production
25. 25 Stg Prd Dev Problem Diicult to stabilize LLM product’s latency
26. 26 Slow dialogue could miss oportunities Problem
27. 27 Monitor metrics with Datadog Inferred Services Solution
28. 28 To develop LLM based applications, Datadog is very useful! Summary
29. 29 MLOps 勉強会 #55 8/22 19:00 ~ 20:00
30. Thank you! Hiroyuki Moriya AI Engineer IVRy.Inc.