AI Infrastructure at Speed: Governance Without the Drag

A session at LINE THAILAND DEVELOPER CONFERENCE 2026

1. AI Infrastructure at Speed: Governance Without the Drag Enabling governed AI for engineers and non-engineers Kajornsak Peerapathananont
2. The pain: chasing AI
3. The pain: chasing AI • API keys: who owns each key, and where is it used?
4. The pain: chasing AI • API keys: who owns each key, and where is it used? • Usage: who is using what, across which providers?
5. The pain: chasing AI • API keys: who owns each key, and where is it used? • Usage: who is using what, across which providers? • Billing: separate dashboards, invoices, and surprise costs 💰💰💰.
6. The pain: chasing AI • API keys: who owns each key, and where is it used? • Usage: who is using what, across which providers? • Billing: separate dashboards, invoices, and surprise costs 💰💰💰. • Every new provider adds another thing to chase.
7. Agenda • The problem: AI is already everywhere • Design principle: centralize governance, decentralize UX
8. Agenda • The problem: AI is already everywhere • Design principle: centralize governance, decentralize UX • Architecture & tools: LiteLLM, Langfuse, Onyx, Sourcebot
9. Agenda • The problem: AI is already everywhere • Design principle: centralize governance, decentralize UX • Architecture & tools: LiteLLM, Langfuse, Onyx, Sourcebot • The hard part: usage outside the gateway
10. Agenda • The problem: AI is already everywhere • Design principle: centralize governance, decentralize UX • Architecture & tools: LiteLLM, Langfuse, Onyx, Sourcebot • The hard part: usage outside the gateway • Governance as guardrails, not gates • What we learned
11. Everyone wants AI. Everyone wants it differently. Engineers • Non-engineers • Product teams • Management
12. Four audiences, four expectations ENGINEERS Speed & context • Claude Code, Cursor, agents • Direct API access • Code-aware search
13. Four audiences, four expectations ENGINEERS NON-ENGINEERS Speed & context Answers, not • Claude Code, • Chat in web & Slack Cursor, agents • Direct API access • Code-aware search APIs • Company knowledge • No subscription to manage
14. Four audiences, four expectations ENGINEERS NON-ENGINEERS PRODUCT TEAMS Speed & context Answers, not AI inside the • Claude Code, • Chat in web & Slack • LLM APIs for Cursor, agents • Direct API access • Code-aware search APIs • Company knowledge • No subscription to manage product features • Agents & internal services • Reliable, with fallback
15. Four audiences, four expectations ENGINEERS NON-ENGINEERS PRODUCT TEAMS MANAGEMENT Speed & context Answers, not AI inside the Cost & control • Claude Code, • Chat in web & Slack • LLM APIs for • Spend by team & • Agents & internal • Data safety & policy Cursor, agents • Direct API access • Code-aware search APIs • Company knowledge • No subscription to manage product features services • Reliable, with fallback app • Proof of value
16. Four audiences, four expectations ENGINEERS NON-ENGINEERS PRODUCT TEAMS MANAGEMENT Speed & context Answers, not AI inside the Cost & control • Claude Code, • Chat in web & Slack • LLM APIs for • Spend by team & • Agents & internal • Data safety & policy Cursor, agents • Direct API access • Code-aware search APIs • Company knowledge • No subscription to manage product features services • Reliable, with app • Proof of value fallback Same company. Different entry points. Who owns the guardrails?
17. The tension: velocity governance
18. The tension: velocity TOO MUCH CONTROL → friction • Approval queue per use case • One mandated tool for everyone • Platform team becomes the bottleneck • Shadow AI grows anyway governance
19. The tension: velocity governance TOO MUCH CONTROL → friction TOO LITTLE CONTROL → chaos • Approval queue per use case • API keys scattered in repos & laptops • One mandated tool for everyone • Nobody knows the monthly spend • Platform team becomes the • Sensitive data in random tools bottleneck • Shadow AI grows anyway • No idea what actually happened
20. The tension: velocity governance TOO MUCH CONTROL → friction TOO LITTLE CONTROL → chaos • Approval queue per use case • API keys scattered in repos & laptops • One mandated tool for everyone • Nobody knows the monthly spend • Platform team becomes the • Sensitive data in random tools bottleneck • No idea what actually happened • Shadow AI grows anyway The goal: guardrails built into the platform, not an approval process.
21. What does an internal AI platform actually need? ACCESS Who can use what?
22. What does an internal AI platform actually need? ACCESS MODELS Who can use what? Which provider and model?
23. What does an internal AI platform actually need? ACCESS MODELS KNOWLEDGE Who can use what? Which provider and What internal data can AI model? see?
24. What does an internal AI platform actually need? ACCESS MODELS KNOWLEDGE Who can use what? Which provider and What internal data can AI model? see? OBSERVABILITY What actually happened?
25. What does an internal AI platform actually need? ACCESS MODELS KNOWLEDGE Who can use what? Which provider and What internal data can AI model? see? OBSERVABILITY COST What actually happened? How much are we spending, by who?
26. What does an internal AI platform actually need? ACCESS MODELS KNOWLEDGE Who can use what? Which provider and What internal data can AI model? see? OBSERVABILITY COST DEVELOPER EXPERIENCE What actually happened? How much are we Can people use it without spending, by who? opening a ticket? Capabilities first. Tools come later.
27. Our design principle Centralize governance. Decentralize the user experience. We don't force everyone onto the same AI tool. We give different tools the same governance foundation.
28. Same foundation, different experiences SHARED GOVERNANCE LAYER Identity Cost Policy Observability
29. Same foundation, different experiences SHARED GOVERNANCE LAYER Identity Cost Policy Observability ENGINEERS NON-ENGINEERS PRODUCTS Claude Code, Cursor Web chat Internal services Sourcebot, APIs Slack bot Different tools — the same guardrails underneath. Agents · APIs
30. The architecture ENGINEERS NON-ENGINEERS PRODUCTS Claude Code, Cursor, APIs Web chat · Slack bot Internal tools & services
31. The architecture ENGINEERS NON-ENGINEERS PRODUCTS Claude Code, Cursor, APIs Web chat · Slack bot Internal tools & services MCP CONTEXT Onyx retrieval business knowledge Onyx Jira · Confluence · Slack Sourcebot code search
32. The architecture ENGINEERS NON-ENGINEERS PRODUCTS Claude Code, Cursor, APIs Web chat · Slack bot Internal tools & services MCP CONTEXT Onyx retrieval business knowledge Sourcebot Onyx code search Jira · Confluence · Slack LiteLLM — AI Gateway Virtual keys, Budgets & rate limits, Model routing & fallback, MCP hub, Skills hub Anthropic API Open AI Google Vertex
33. The architecture ENGINEERS NON-ENGINEERS PRODUCTS Claude Code, Cursor, APIs Web chat · Slack bot Internal tools & services MCP CONTEXT Onyx retrieval business knowledge Sourcebot Onyx code search Jira · Confluence · Slack LiteLLM — AI Gateway OBSERVABILITY Virtual keys, Budgets & rate limits, Model routing & fallback, MCP hub, Skills hub OTEL → Langfuse + LGTM trace, cost, usage Anthropic API Open AI Google Vertex latency, eval
34. LiteLLM: one gateway, many providers GitHub github.com/BerriAI/litellm PROBLEM Multiple keys in repos, single key used multiple places, spend nobody can see. ROLE AI Gateway
35. LiteLLM: one gateway, many providers GitHub github.com/BerriAI/litellm PROBLEM Multiple keys in repos, single key used multiple places, spend nobody can see. ROLE AI Gateway WHAT WE GET • One endpoint, many providers • Virtual keys per team, app, workflow • Budgets, rate limits, model allow-lists • Spend tracked per key and team • Logs and Traces (minimal version) NOT RAG · observability UI · a user-facing chatbot
36. How we set up LiteLLM Connecting AI providers — and the controls around them Teams Keys grouped by team, app and workflow
37. How we set up LiteLLM Connecting AI providers — and the controls around them Teams Budgets Keys grouped by team, app and workflow Spend caps per team and per key
38. How we set up LiteLLM Connecting AI providers — and the controls around them Teams Keys grouped by team, app and workflow Budgets Spend caps per team and per key Alerts Know before a budget runs out
39. How we set up LiteLLM Connecting AI providers — and the controls around them Teams Keys grouped by team, app and workflow Budgets Spend caps per team and per key Alerts Know before a budget runs out Models hub Approved models across providers
40. How we set up LiteLLM Connecting AI providers — and the controls around them Teams Keys grouped by team, app and workflow Budgets Spend caps per team and per key Alerts Know before a budget runs out Models hub Approved models across providers MCP & Skills hubs Shared tools and skills for agents
41. LiteLLM — an approved model catalog
42. Langfuse: see what actually happened GitHub github.com/langfuse/langfuse PROBLEM Requests are happening, but do we know what actually happened inside them? ROLE LLM Observability
43. Langfuse: see what actually happened GitHub github.com/langfuse/langfuse PROBLEM Requests are happening, but do we know what actually happened inside them? ROLE LLM Observability WHAT WE GET • Traces & spans per request • Tokens, cost & latency • Prompt / response visibility • Prompt versioning • Datasets & Evaluation NOT a gateway · a policy engine
44. Product side: gaps closed. Internal side: needs more. PRODUCT SIDE · consumer-facing ✓ LiteLLM one gateway, keys & budgets ✓ Langfuse traces, cost & evaluation Governed and observable
45. Product side: gaps closed. Internal side: needs more. INTERNAL USAGE NEEDS MORE PRODUCT SIDE · consumer-facing Sourcebot ✓ LiteLLM Code context for engineers & agents one gateway, keys & budgets ✓ Langfuse traces, cost & evaluation Governed and observable →
46. Sourcebot: code search for humans and agents GitHub github.com/sourcebot-dev/sourcebot PROBLEM Engineers and agents need answers about our whole codebase, not just the repo that's open. ROLE Code search & context
47. Sourcebot: code search for humans and agents GitHub github.com/sourcebot-dev/sourcebot PROBLEM Engineers and agents need answers about our whole codebase, not just the repo that's open. ROLE Code search & context WHAT WE GET • Fast, repo-aware code search • Ask questions about the codebase • Agents query it via MCP • Self-hosted — code stays with us NOT a knowledge base · an IDE agent
48. Product side: gaps closed. Internal side: needs more. INTERNAL USAGE NEEDS MORE PRODUCT SIDE · consumer-facing Sourcebot ✓ LiteLLM Code context for engineers & agents one gateway, keys & budgets ✓ Langfuse traces, cost & evaluation Governed and observable →
49. Product side: gaps closed. Internal side: needs more. INTERNAL USAGE NEEDS MORE PRODUCT SIDE · consumer-facing Sourcebot ✓ LiteLLM Code context for engineers & agents one gateway, keys & budgets ✓ Langfuse traces, cost & evaluation Governed and observable → Onyx Knowledge & chat for everyone else
50. Engineers aren't the only AI users ENGINEERS • API & CLI • IDE & agents — Claude Code, Cursor • Sourcebot code search • MCP tools
51. Engineers aren't the only AI users ENGINEERS EVERYONE ELSE • API & CLI • Chat in browser & Slack • IDE & agents — Claude Code, Cursor • Search across Jira & Confluence • Sourcebot code search • Custom agents per team • MCP tools • No API key, no personal subscription
52. Engineers aren't the only AI users ENGINEERS EVERYONE ELSE • API & CLI • Chat in browser & Slack • IDE & agents — Claude Code, Cursor • Search across Jira & Confluence • Sourcebot code search • Custom agents per team • MCP tools • No API key, no personal subscription Governance shouldn't require teaching every employee how to use an API.
53. Onyx — governed AI for everyone else GitHub github.com/onyx-dot-app/onyx PROBLEM People need company knowledge — not API keys Employee · web or Slack or personal AI subscriptions. Onyx ROLE Knowledge & chat interface Jira Confluence LiteLLM Slack GitHub Permission-aware access — required, not optional
54. Onyx — governed AI for everyone else GitHub github.com/onyx-dot-app/onyx PROBLEM People need company knowledge — not API keys Employee · web or Slack or personal AI subscriptions. Onyx ROLE Knowledge & chat interface WHAT WE GET • Chat & search on web + Slack bot • Connectors to our SaaS tools • Custom agents per team Jira Confluence LiteLLM Slack GitHub Permission-aware access — required, not optional • LLM calls still go via LiteLLM • Retrieval exposed to engineers via MCP NOT code search · a way around source permissions
55. Onyx — company knowledge, in web and Slack
56. Why not put everything into one giant RAG?
57. Why not put everything into one giant RAG? BUSINESS KNOWLEDGE ↓ Onyx • Docs, tickets, chat threads • Semantic search + chat answers • Permissions follow the source tool GitHub github.com/onyx-dot-app/onyx
58. Why not put everything into one giant RAG? BUSINESS KNOWLEDGE SOURCE CODE ↓ ↓ Onyx Sourcebot • Docs, tickets, chat threads • Repos, symbols, exact matches • Semantic search + chat answers • Regex & repo-aware code search • Permissions follow the source tool • Also exposed to agents via MCP GitHub github.com/onyx-dot-app/onyx GitHub github.com/sourcebot-dev/sourcebot
59. Why not put everything into one giant RAG? BUSINESS KNOWLEDGE SOURCE CODE ↓ ↓ Onyx Sourcebot • Docs, tickets, chat threads • Repos, symbols, exact matches • Semantic search + chat answers • Regex & repo-aware code search • Permissions follow the source tool • Also exposed to agents via MCP GitHub github.com/onyx-dot-app/onyx GitHub github.com/sourcebot-dev/sourcebot Different context has different retrieval needs — that can justify different tools.
60. But engineers also use Claude subscriptions. AUTHORITATIVE every request, key & cost API → LiteLLM Engineer Langfuse
61. But engineers also use Claude subscriptions. AUTHORITATIVE every request, key & cost API → LiteLLM Engineer Langfuse Claude / Cursor seats OBSERVED / ESTIMATED Claude Code OTel telemetry · Cursor admin exports
62. But engineers also use Claude subscriptions. AUTHORITATIVE every request, key & cost API → LiteLLM Engineer Langfuse Claude / Cursor seats OBSERVED / ESTIMATED Claude Code OTel telemetry · Cursor admin exports AI gateway telemetry ≠ company-wide AI usage
63. Claude subscriptions are a blind spot On the Team plan we can’t observe Claude usage — and we’re not big enough for the Enterprise plan.
64. Claude subscriptions are a blind spot On the Team plan we can’t observe Claude usage — and we’re not big enough for the Enterprise plan. BUT MANAGEMENT STILL NEEDS Utilization Token spend Budget planning Who uses it, how much Where tokens go Next year’s AI subscriptions
65. Claude subscriptions are a blind spot On the Team plan we can’t observe Claude usage — and we’re not big enough for the Enterprise plan. BUT MANAGEMENT STILL NEEDS Utilization Token spend Budget planning Who uses it, how much Where tokens go Next year’s AI subscriptions So we needed an extra hack.
66. The hack: Claude Code → Langfuse, via a bridge 1 Managed settings Push OTLP telemetry config to every laptop Managed settings · logs example { "env": { "CLAUDE_CODE_ENABLE_TELEMETRY": "1", "OTEL_LOGS_EXPORTER": "otlp", "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL": "http/json", "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "https://bridge.example.com/v1/logs" } Claude Code Langfuse }
67. The hack: Claude Code → Langfuse, via a bridge 1 Managed settings 2 Traces go straight in Push OTLP telemetry config to every laptop Managed settings · logs example Langfuse ingests spans via its OTel endpoint { "env": { "CLAUDE_CODE_ENABLE_TELEMETRY": "1", "OTEL_LOGS_EXPORTER": "otlp", "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL": "http/json", "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "https://bridge.example.com/v1/logs" } traces (protobuf) → OTel endpoint Claude Code Langfuse }
68. The hack: Claude Code → Langfuse, via a bridge 1 Managed settings 2 Traces go straight in 3 Push OTLP telemetry config to every laptop Managed settings · logs example Langfuse ingests spans via its OTel endpoint { "env": { "CLAUDE_CODE_ENABLE_TELEMETRY": "1", "OTEL_LOGS_EXPORTER": "otlp", "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL": "http/json", "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "https://bridge.example.com/v1/logs" } Logs need a bridge Content arrives as OTel logs, which Langfuse drops traces (protobuf) → OTel endpoint Claude Code logs (http/json) Langfuse Bridge ingestion API }
69. Guardrails > Gates TRADITIONAL GOVERNANCE Request Ticket Approval Security review Platform setup Use AI Every new use case waits in line
70. Guardrails > Gates TRADITIONAL GOVERNANCE Request Ticket Approval Security review Platform setup Use AI Every new use case waits in line PLATFORM GOVERNANCE User AI Guardrails applied automatically • Identity: who is calling • Model access: allow-lists per key • Budget & rate limits • Audit trail • Observability: traces & cost The default path is already governed
71. How we think about governance ACCESS Who can use it? • Models • Knowledge sources • Repositories • Tools (MCP) SSO, LiteLLM keys, Onyx permissions
72. How we think about governance ACCESS USAGE Who can use it? How much can they use? • Models • Budgets • Knowledge sources • Rate limits • Repositories • Quotas • Tools (MCP) • Model policy SSO, LiteLLM keys, Onyx permissions LiteLLM virtual keys
73. How we think about governance ACCESS USAGE VISIBILITY Who can use it? How much can they use? What actually happened? • Models • Budgets • Traces • Knowledge sources • Rate limits • Cost • Repositories • Quotas • Latency • Tools (MCP) • Model policy • Evaluation SSO, LiteLLM keys, Onyx permissions LiteLLM virtual keys OTEL, Langfuse, LGTM
74. You probably don't need to build an AI platform from scratch ADOPT • Gateway: LiteLLM • Observability: Langfuse + LGTM • Knowledge: Onyx • Code search: Sourcebot
75. You probably don't need to build an AI platform from scratch ADOPT BUILD • Gateway: LiteLLM • Integration glue • Observability: Langfuse + • Key & policy workflows LGTM • Knowledge: Onyx • Code search: Sourcebot • Internal agents & workflows
76. You probably don't need to build an AI platform from scratch ADOPT BUILD OWN • Gateway: LiteLLM • Integration glue • Architecture • Observability: Langfuse + • Key & policy workflows • Governance model • Internal agents & • Developer experience LGTM • Knowledge: Onyx • Code search: Sourcebot workflows
77. You probably don't need to build an AI platform from scratch ADOPT BUILD OWN • Gateway: LiteLLM • Integration glue • Architecture • Observability: Langfuse + • Key & policy workflows • Governance model • Internal agents & • Developer experience LGTM • Knowledge: Onyx workflows • Code search: Sourcebot We evaluated Glean and Sourcegraph too. But enterprise minimum commitments didn't fit our stage. Composable OSS gave us lower cost and more flexible control.
78. What we learned 01 Don't force one AI interface on everyone. IDE for engineers, chat for everyone else.
79. What we learned 01 Don't force one AI interface on everyone. IDE for engineers, chat for everyone else. 02 Centralize control points, not every interaction. Identity, keys, budgets, telemetry.
80. What we learned 01 Don't force one AI interface on everyone. IDE for engineers, chat for everyone else. 02 Centralize control points, not every interaction. Identity, keys, budgets, telemetry. 03 Observability is part of governance. You can't govern what you can't see.
81. What we learned 01 Don't force one AI interface on everyone. IDE for engineers, chat for everyone else. 02 Centralize control points, not every interaction. Identity, keys, budgets, telemetry. 03 Observability is part of governance. You can't govern what you can't see. 04 Gateway visibility is not total AI visibility. Label data: authoritative vs. estimated.
82. What we learned 01 Don't force one AI interface on everyone. IDE for engineers, chat for everyone else. 02 Centralize control points, not every interaction. Identity, keys, budgets, telemetry. 03 Observability is part of governance. You can't govern what you can't see. 04 Gateway visibility is not total AI visibility. Label data: authoritative vs. estimated. 05 Assemble mature OSS before building it yourself. Own the integration, policy and DX.
83. Central governance. Multiple experiences. Engineers Everyone Claude, Cursor, Sourcebot Onyx Web , Slack AI PLATFORM LiteLLM, Knowledge, Identity, Budgets OpenTelemetry Langfuse
84. Summary ENGINEERS NON-ENGINEERS PRODUCTS Claude Code, Cursor, APIs Web chat · Slack bot Internal tools & services MCP CONTEXT Onyx retrieval business knowledge Sourcebot Onyx code search Jira · Confluence · Slack LiteLLM — AI Gateway OBSERVABILITY Virtual keys, Budgets & rate limits, Model routing & fallback, MCP hub, Skills hub OTEL → Langfuse + LGTM trace, cost, usage Anthropic API Open AI Google Vertex latency, eval
85. Thank you Central governance. Multiple experiences.