GenAI Framework Observability at Cloud Native Summit 2025

This is an updated version of my GenAI Framework Observability talk, with "just in time" details on Model Context Protocol. All examples are instrumented for Observability with Elastic Distributions of OpenTelemetry aka EDOT.

https://www.cloudnativesummit.co/GenAI-framework-observability-with-Elastic-Distributions-of-OpenTelemetry-2025.html
Example code == https://github.com/elastic/observability-examples/tree/main/genai-function-calling

Transcript

1. GenAI Framework Observability Agents and Otel and MCP.. oh my! <- Adrian’s LinkedIn
2. Agenda • OpenTelemetry and frameworks • Overview the GenAI agent example • Showcase Python, .NET, Node.js and Java • Bonus Section: Model Context Protocol • Sum up
3. @adrianfcole • principal engineer at elastic • focused on genai and opentelemetry • former lead of open zipkin
4. OpenTelemetry is a CNCF project that addresses Observability in terms of specifications, SDKs and tools Today we’ll talk about the language SDKs! Elastic Distributions of OpenTelemetry (EDOT) includes open source language SDKs, a collector and k8s operators
5. Frameworks are great for observability - Frameworks raise abstractions allowing focus on high level steps - Business logic is often coded in frameworks with useful function names - Frameworks allow switching backends like to a different database or for tests - But, frameworks aren’t always visible in logs or derivable from http requests
6. Frameworks are good for GenAI, too! - New Large Language Models (LLMs) emerge often so abstractions help - GenAI applications use VectorDBs these also benefit from abstraction - Some tasks are better broken into steps which need orchestration - There are common patterns to simplify like RAG, agent loops, tool use
7. Introducing Elasticsearch Version Agent Our example is uses an LLM and tool to answer the question “What’s the latest version of Elasticsearch 8” We’ll add observability with Elastic Distributions of OpenTelemetry (EDOT) <- Example Code EDOT Elastic Stack ->
8. Getting past the training date without RAG There is no correlation between recency or version of a model and its knowledge User: what's the latest version of Elasticsearch? Assistant: As of my knowledge as of 2023-12-19, the latest stable version of Elasticsearch is **7.15.2** Conversational AI (ChatGPT, Grok, Perplexity) call tools to search the web for new knowledge Note: Kibana AI assistant uses search connectors (jira, etc) the same way!
9. GenAI agent frameworks automatically call tools
10. How do you add OpenTelemetry to your application? Depending on language and framework, bootstrap the application • This could be scanning dependencies or adding config • Frameworks like Spring need a small amount of code to connect their libraries to otel Change its startup to include auto-instrumentation • This could be prefixing its command, changing its libraries or adding an agent Auto instrumentation sends logs, metrics and traces to localhost • This makes it near zero configuration when using collector sidecars • Overrides are via ENV variables, and in the future a config file Note: EDOT SDKs are OpenTelemetry distributions! They work the same way, and together with any other code using OpenTelemetry!
11. Hello OpenAI Agents SDK (Python)! github.com/openai/openai-agents-python ★ Multi-Agent Handoffs ★ Tool and Guardrail Support ★ Pluggable Tracer
12. Hello EDOT Python! Add our EDOT Python package: pip install elastic-opentelemetry Run edot-bootstrap which analyzes the code to install any relevant instrumentation available: edot-bootstrap —-action=install Add OpenTelemetry environment variables OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true Prefix python with opentelemetry-instrument: opentelemetry-instrument python main.py github.com/elastic/elastic-otel-python
13. OpenInference plugs in and combines with EDOT Note: This uses EDOT’s built-in OpenAI and HTTP instrumentation github.com/Arize-ai/openinference
14. Hello Semantic Kernel .NET! github.com/microsoft/semantic-kernel ★ Plugin approach to tools ★ Built-in OpenTelemetry
15. Hello EDOT .NET! Install EDOT EDOT_VERSION=1.0.2 EDOT_INSTALL=https://github.com/elastic/elastic-otel-dotnet/releases/download/${EDOT_VERSION}/ elastic-dotnet-auto-install.sh OTEL_DOTNET_AUTO_HOME=/edot sh -c "$(curl -fsSL ${EDOT_INSTALL})" Add OpenTelemetry and framework-specific environment variables SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS=true SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE=true OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES="Microsoft.SemanticKernel*" Prefix dotnet with instrument.sh: ${OTEL_DOTNET_AUTO_HOME}/instrument.sh dotnet app.dll github.com/elastic/elastic-otel-dotnet
16. Semantic Kernel (and manual) spans combine with EDOT Note: This uses EDOT’s built-in HTTP instrumentation activitySource.StartActivity(name: "agent");
17. Hello Vercel AI! github.com/vercel/ai ★ Made to add AI to Next.js, React, Svelte and Vue apps ★ Built-in OpenTelemetry const openai = createAzure({ apiKey: process.env.AZURE_OPENAI_API_KEY, baseURL: new URL('openai/deployments/', process.env.AZURE_OPENAI_ENDPOINT).href, }) async function main() { const {text} = await generateText({ model: openai(process.env.CHAT_MODEL || 'your-deployment-name'), messages: [{role: 'user', content: "What's the latest version of Elasticsearch?"}], temperature: 0, tools: { getLatestElasticsearchVersion: getLatestElasticsearchVersion, }, tool_choice: 'auto', maxSteps: 10, experimental_telemetry: { isEnabled: true }, }); console.log(text); }
18. Hello EDOT Node.js! Add our EDOT Node.js package: npm install @elastic/opentelemetry-node Ensure you enabled telemetry in vercel experimental_telemetry: { isEnabled: true }, Require @elastic/opentelemetry-node when running node --require @elastic/opentelemetry-node index.js github.com/elastic/elastic-otel-node
19. Vercel AI spans combine with EDOT Note: This doesn’t use EDOT‘s built-in OpenAI instrumentation because Vercel AI has its own internal OpenAI library!
20. Hello Spring AI! github.com/spring-projects/spring-ai ★ Provide tools as beans ★ OpenTelemetry via Micrometer @Component static class VersionAgent implements CommandLineRunner { private final ChatClient chat; private final ElasticsearchTools tools; VersionAgent(ChatModel chat, ElasticsearchTools tools) { this.chat = ChatClient.builder(chat).build(); this.tools = tools; } @Override @NewSpan("version-agent") public void run(String... args) { String answer = chat.prompt() .user("What is the latest version of Elasticsearch?") .tools(tools) .options(DefaultToolCallingChatOptions.builder().temperature(0.0).build()) .call() .content(); System.out.println(answer); } }
21. Hello EDOT Java + Spring Boot! Switch micrometer to otel, and make an OpenTelemetry bean curl -o elastic-otel-javaagent.jar \ https://repo1.maven.org/maven2/co/elastic/otel/elastic-otel-javaagent/1.4.1/elastic-otel-javaagent-1.4.1.jar OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true github.com/elastic/elastic-otel-java <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> </dependency @Bean OpenTelemetry openTelemetry() { return GlobalOpenTelemetry.get(); } Download the latest elastic-otel-javaagent.jar Add OpenTelemetry environment variables Run with elastic-otel-javaagent.jar java -javaagent:elastic-otel-javaagent.jar app.jar
22. Spring (Micrometer) spans combine with EDOT autoconfigure.exclude : o.s.b.a.a.o.web.client.HttpClientObservationsAutoConfiguratio n Note: This example excluded Spring’s HTTP instrumentation @NewSpan("version-agent")
23. EDOT SDKs work with others, with some caveats Frameworks sometimes have their own abstractions and telemetry • EDOT configures base log, tracing and metrics libraries under them • It can also fill gaps, such as HTTP layer for trace propagation OpenTelemetry doesn’t necessarily mean compatible data conventions • Logs, metrics and spans may be slightly different or change between releases Frameworks who have their own telemetry libraries may need glue • For example, Spring uses Micrometer which requires a bridge and small setup code Overhead and data collection policy should be closely watched • Enabling prompt/completion capture can add overhead and PII risk
24. Model Context Protocol (MCP) - Client/Server Protocol for LLM tools (and more) - stdio and HTTP streaming transports - Wildly popular modelcontextprotocol.io Tool calls here!
25. MCP+Stdio 1. Launch a server 2. Lookup its tools 3. Same otherwise Trivia MCP flow inspired by “codename goose” a MCP pioneer!
26. MCP Code Semantic Kernel Glue github.com/modelcontextprotocol/csharp-sdk
27. - Launch MCP Server - Decoupled tool call - Same otherwise
28. • Initially open sourced by Anthropic, adopted widely even by OpenAI • A fast evolving JSON-RPC based (messaging) protocol • Popular for tools, but has more like prompts and LLM sampling • Check latest spec and practice which clarify auth and security • OpenTelemetry is evolving, currently bound at the JSON-RPC layer Model Context Protocol (MCP) Notes <- Arize+Elastic MCP tracing collaboration demo
29. Thanks! www.linkedin.com/in/adrianfcole modelcontextprotocol.io