SDD 2025: 'Talk to your systems': structured data & tool integration with LLMs

“Speaking” to your own data (aka RAG) is the “Hello World” use case for Gen AI & LLMs. But there is more to discover! Based on their understanding of human language, LLMs can be used to realize innovative interactions for applications and systems. In this session, Christian shows how to use structured data outputs with data schemas and function calls to connect your APIs with the power of LLMs. See how the transformative nature of Generative AI can potentially help your solutions too. Come and see how we can talk to our systems!

Transcript

1. ‘Talk to your systems’ Structured data & tool integration with LLMs Christian Weyer | Co-Founder & CTO | Thinktecture AG | christian.weyer@thinktecture.com
2. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for AI § Google GDE for Web AI christian.weyer@thinktecture.com @christianweyer https://www.thinktecture.com 'Talk to your systems' Structured data & tool integration with LLMs Christian Weyer Co-Founder & CTO @ Thinktecture AG 2
3. 'Talk to your systems' Structured data & tool integration with LLMs Our journey 3 The Universal Interface Integrating LLMs Talk to your Systems End-to-End scenarios AI all-the-things?
4. 'Talk to your systems' Structured data & tool integration with LLMs AI ALL-THE-THINGS? 4
5. 'Talk to your systems' Structured data & tool integration with LLMs A.I. - 1985 5
6. 'Talk to your systems' Structured data & tool integration with LLMs AI all-the-things? Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP (Natural Language Processing) Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers 6
7. 'Talk to your systems' Structured data & tool integration with LLMs THE UNIVERSAL INTERFACE 7
8. 'Talk to your systems' Structured data & tool integration with LLMs Human language rocks Extending access to software 8
9. 'Talk to your systems' Structured data & tool integration with LLMs Classical applications & UIs 9
10. 'Talk to your systems' Structured data & tool integration with LLMs Language-enabled “UIs” – Talk-to-TT sample 10
11. 'Talk to your systems' Structured data & tool integration with LLMs INTEGRATING LLMS 11
12. § LLMs are always part of end-to-end architectures § Client apps (Web, desktop, mobile) § Services with APIs § Databases § etc. § An LLM is ‘just’ an additional asset in your architecture § It is not the Holy Grail for everything § Enable human language as a first-class citizen 🤯 'Talk to your systems' Structured data & tool integration with LLMs End-to-end architectures with LLMs Clients Services LLMs Desktop Web Mobile Service A Service B Service C API Gateway Monitoring LLM 1 LLM 2 12
13. 'Talk to your systems' Structured data & tool integration with LLMs It’s just HTTP APIs Inference, FTW 13
14. 'Talk to your systems' Structured data & tool integration with LLMs Most convenient platform to work with genuine Gen AI 14
15. 'Talk to your systems' Structured data & tool integration with LLMs TALK TO YOUR SYSTEMS 15
16. 'Talk to your systems' Structured data & tool integration with LLMs Prompting Talk to me! 16
17. 'Talk to your systems' Structured data & tool integration with LLMs ‘Function’ Calling Give it schema! 17
18. § Predefined JSON schema structure § All major libs support tool calling with abstractions § OpenAI SDKs § Langchain § Semantic Kernel § Etc. 'Talk to your systems' Structured data & tool integration with LLMs OpenAI Tool calling – plain HTTP calls curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-4o", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }' https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools 18
19. § External metadata, e.g. JSON description/files § .NET: Reflection § Python: Pydantic § JS / TypeScript: nothing out of the box (yet) § Typically using Zod 'Talk to your systems' Structured data & tool integration with LLMs Provide metadata about your functions / tools 19
20. 'Talk to your systems' Structured data & tool integration with LLMs Pydantic & Instructor Make it easier! 20
21. 'Talk to your systems' Structured data & tool integration with LLMs END-TO-END SCENARIOS 21
22. Semantic routing 'Talk to your systems' Structured data & tool integration with LLMs "Talk to your systems" - for Availability queries 22 Web App / Watch App Speech-to-Text Internal Gateway (Python FastAPI) LLM / SLM Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data (Function calling) Generate response with availability Response Response with experts availability 🔉 Speech-to-text for response Response audio Internal Business API (node.js – veeeery old) Query Availability API Availability When is CL…? CL will be…
23. Filling Angular forms with human language input – TypeScript-only protected readonly formGroup = this.fb.group({ firstName: [’’], lastName: [’’], addressLine1: [’’], addressLine2: [’’], city: [’’], state: [’’], zip: [’’], country: [’’] }); 'Talk to your systems' Structured data & tool integration with LLMs Smart form filling OK, nice – so here is my address then: Peter Schmitt, Rheinstr. 7 in Schkeuditz – postcode is 04435, BTW. Smart Form Filler (TS code & LM) 23
24. 'Talk to your systems' Structured data & tool integration with LLMs One more thing … 24
25. 'Talk to your systems' Structured data & tool integration with LLMs AGENTS 25 ARE LOOPS OVER TOOL CALLS
26. § Give LLM more capabilities with tools § Access data and other functionality § Connect your APIs, applications, and environments 'Talk to your systems' Structured data & tool integration with LLMs Extending LLM capabilities for complex tasks “Do x!” LLM “Do x!” System prompt Tool 1 metadata Tool 2 metadata ... { “answer”: “toolcall”, “tool” : “tool1” “args”: […] } 26
27. 'Talk to your systems' Structured data & tool integration with LLMs ReAct: Reasoning and Acting https://arxiv.org/abs/2210.03629 27
28. § Involve an LLM making decisions through ‘thinking’ § Which actions to take (‘thought’) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Keep human in the loop for missing data, validation, approval 'Talk to your systems' Structured data & tool integration with LLMs ReAct: Reasoning and Acting 28
29. 'Talk to your systems' Structured data & tool integration with LLMs ReAct – illustrated LLM My code / agent Query Your API Your database Prompt Tools Final answer Answer ❓ ❓ ❗ 💡 29 § Tool calling is being standardized through MCP
30. 'Talk to your systems' Structured data & tool integration with LLMs RECAP 30
31. 'Talk to your systems' Structured data & tool integration with LLMs Talk to your systems 31 Integrating LLMs via APIs Structured output & Tool calling End-to-End scenarios ‘Lightweight’ Agents The Universal Interface
32. 'Talk to your systems' Structured data & tool integration with LLMs Talk to your systems 32 Integrating LLMs via APIs Structured output & Tool calling End-to-End scenarios Lightweight Agents The Universal Interface LLMs everywhere AI solutions are 10% AI. And 100% software engineering.
33. Thank you! Christian Weyer https://thinktecture.com/christian-weyer https://github.com/thinktecture-labs/talk-to-your-systems https://github.com/thinktecture-labs/smart-form-filler 33