Web-App-Automation mittels GenAI & LLMs

In diesem praktischen Workshop werden die Teilnehmende verschiedene Ansätze zur Integration von Large Language Models (LLMs) in bestehende Webanwendungen erkunden.

Unter der Leitung von Max Marschall lernen die Teilnehmenden, wie sie LLMs effektiv integrieren können, indem sie Techniken wie Prompting, Funktions-/Toolaufrufe und GenAI-Agenten nutzen.

Mittels einer Kombination aus Vortrag, Coding und Diskussion erhalten die Teilnehmenden praktische Einblicke in die Integration von LLMs in ihre Anwendungen:

Tool-Calling in Angular​
Agent Erstellung
Adaptives Architektursystem
Accessibility durch Automation

Transcript

1. Web-App-Automation mittels GenAI & LLMs Max Marschall @MaxOSchulte Consultant / Developer
2. Arti fi cial Intelligence Machine Learning Data Science NLP A Deep Learning Generative AI Large Language Models Image / Video Generation
3. Bla bla bla … Intention
4. Bla bla bla … Generation of… Intention Summarisation of… Automation of… Simpli fi cation of…
5. Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Intention
6. Intention Bla bla bla … “Augmenting an App with an assistant, that helps the user based on displayed context. Allowing an assistant to act for me in that context.” Problem: Integration GenAI Metainformation Tool calls Leaking secrets System integration Solutions AST parsing TS Decorators Client-side "agents" Barebones Angular OpenAI
7. Tools & Functions Process Think Display Bla bla bla …
8. User Prompt • Voice input • Text input • Image input • Unstructured • Universal interface • Natural language as fi rst-class input
9. • Prompts • RAG • Sanitising • Guards • Vector-Database • Tool selection • Additional data (Processing)
10. • System prompt • Base information • Rules • Intention • Output format Processing You are an friendly helpfull assistent proficient in creating and managing forms. You will receive information on the state of an app. If you feel there a … System Prompt https://docs.anthropic.com/en/release-notes/system-prompts#oct-22nd-2024 You are a data analyst API capable of sentiment analysis that responds in JSON. The JSON schema should include { "sentiment_analysis": { "sentiment": "string (positive, negative, neutral)", "confidence_score": "number (0-1)" # Include additional fields as required } https://console.groq.com/docs/text-chat Prompts
11. • State • Additional information • Generation examples Context /** Additional Information: basic list of static pages shown in the side navigation. */ [ { title: 'Contributions', icon: 'question_answer', link: ['/contributions'] }, { title: 'Conferences', icon: 'podium', link: ['/conferences'] }, { title: 'Speakers', icon: 'person', link: ['/speakers'] }, { title: 'Collections', icon: 'category', link: ['/collections'] }, ];
12. Tools & Functions /** * Accepts a resolved sitemap entry and navigates to it. * * e.g. path: /collections/:id * ":id" is replaced by an actual entity id. */ public navigate(resolvedPath: string): void { this.router.navigateByUrl('/' + resolvedPath); }
13. Tools & Functions /** * Accepts a resolved sitemap entry and navigates to it. * * e.g. path: /collections/:id * ":id" is replaced by an actual entity id. */ @AiTool public navigate(resolvedPath: string): void { this.router.navigateByUrl('/' + resolvedPath); } { "type": "function", "function": { "name": "navigate", "description": "Accepts a resolved sitemap entry …, "parameters": { "type": "object", "properties": { "resolvedPath": { "title": "resolvedPath", "description": "", "type": "string" } } }, "required": ["resolvedPath"], } } AST parsing
14. Tools & Functions { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; }
15. Tools & Functions /** * Creates form fields for a provided list of formField definition. */ @AiTool createFormFields(fields: FormField[]): FormField[] { return fields; } /** Union of all supported form field types. */ export type FormField = | HeaderFormField | SubheaderFormFiel | … { "type": "function", "className": "FormBuilderService", "classDocumentation": "", "function": { "name": "createFormFields", "title": "createFormFields", "type": "function", "description": "Creates form fields for a provided list… "parameters": { "type": "object", "properties": { "fields": { "title": "fields", "description": "Union of all supported form… "anyOf": [ { "$ref": "#/definitions/HeaderFormField" }, …
16. Large Language Model • Transformer Neural Network • Context & relation aware • Pattern matching & recognition • “Natural” language understanding • Generating / creating content • Summarisation
17. Large Language Model • It cannot make decisions • It cannot understand non-linear content • It cannot draw conclusions
18. Large Language Model • Not deterministic • Can hallucinate • Prompt-Injection
19. Result Bla bla bla …
20. Result = Bla bla bla …
21. Tools & Functions Process Work Display <…/>
22. Agents
23. Agents
24. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> { // get the current available tools and application state const { tools, state } = this.createToolState(); const message: ChatCompletionMessageParam = { role: 'user', content, }; // make backend request, choices that the model makes // -> messages and tool calls const choices = await this.aiBackend.automate([message], tools, state, trace); if (!choices) { … handle error } // loop and automate, handle llm choice internally for (const choice of choices) { this.handleChoice(choice, [message], tools, state, trace); } }
25. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void>
26. /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', });
27. /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }
28. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }
29. Agents /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', }); /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); // message user about tool call results … } return { content, tool_call_id: call.id, role: 'tool', name: call.function.name, }; }); }
30. /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> Agents /** * Handles a choice made by the AI model. * @param choice - The choice made by the model. * @param chat - The current chat history. * @param tools - Available tools for the AI. * @param state - Current application state. * @param trace - Langfuse trace client for logging. */ private async handleChoice( choice: Choice, chat: ChatCompletionMessageParam[], tools: Tool[], state?: any, trace?: LangfuseTraceClient, ): Promise<void> { // handle tool call, can be one or more if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { this.message$.next({ role: choice.message.role, message: choice.message.content ?? 'NO MESSAGE', });
31. Agents private async handleChoice( … ): Promise<void> { if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { … show result message to user } } } /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void>
32. Agents private async handleChoice( … ): Promise<void> { if (choice.finish_reason === 'tool_calls') { // display information to the user this.createToolMessage(choice).forEach(message => this.message$.next(message)); // make the actual function call and create a message from it const toolResult = this.handleToolCalls(choice); // respond to the API with made calls if (toolResult) { chat.push(choice.message, ...toolResult); (await this.aiBackend.automate(chat, tools, state, trace)).map(choice => this.handleChoice(choice, chat, tools, state, trace), ); } } else { … show result message to user } } } /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void>
33. Agents /** * Handles a choice made by the AI model. */ private async handleChoice(…): Promise<void> /** * Automates a process based on a user query. * @param content - The query from the user. */ async automate(content: string): Promise<void> /** * Handles tool calls made by the AI model. * @param param0 - Object containing the message with tool calls. * @returns An array of tool message params. */ private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] {
34. Agents
35. Agents
36. Tools from Codebase
37. Tools from Codebase /** Base type for all form field types. */ interface BaseFormField { /** Type of form field. */ type: FormFieldType; /** Description of the Field. What it is and what values should be there. */ description?: string; /** Form field label shown to the user. */ label: string; /** Whether the field is required. Defaults to false. */ required?: boolean; /** Key to store the value under. */ key: string; } /** Header shown in a form. The label is used as its text. */ interface HeaderFormField extends BaseFormField { type: FormFieldType.Title; } { "HeaderFormField": { "description": "Header shown in a form. The label is… "type": "object", "properties": { "type": { "description": "Type of form field.", "type": "string", "const": "title" }, … "required": { "description": "Whether the field is required… "type": "boolean" }, … }, "required": ["key", "label", "type"] }, AST Parsing
38. Tools from Codebase /** * Retrieve a list of entries for a specific type. */ @AiTool public getCollection(collectionType: CollectionType): object[] { switch (collectionType.type) { case 'contributions': return this.entryStore.contributions(); case 'speakers': return this.entryStore.speakers(); case 'collections': return []; case 'conferences': return this.entryStore.conferences(); default: return []; } } { "type": "function", "classDocumentation": "", "function": { "title": "getCollection", "description": "Retrieve a list of entries for …”, "parameters": { "title": "collectionType", "description": "", "type": "object", "properties": { "type": { "description": "Type of different data sets…” "enum": [ “collections", "conferences", "contributions", "speakers" ], "type": "string" } }, "required": [ "type" AST Parsing
39. @Directive({ selector: '[aiClick]', standalone: true, }) export class AiClickDirective { private element = inject(ElementRef); metaInfo = input.required<string>({ alias: 'aiClick' }); guid = input<string>(crypto.randomUUID()); constructor() { registerInstance(() => ({ guid: this.guid(), instance: this, state: this.metaInfo(), className: 'AiClickDirective', metaInfo: this.metaInfo(), })); } @AiTool click(): void { this.element.nativeElement.click(); } } Tools from Codebase
40. export class AiClickDirective { … constructor() { registerInstance(() => ({ guid: this.guid(), instance: this, state: this.metaInfo(), className: 'AiClickDirective', metaInfo: this.metaInfo(), })); } @AiTool click(): void { this.element.nativeElement.click(); } } Tools from Codebase
41. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } Tools from Codebase
42. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } Tools from Codebase
43. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } private handleToolCalls({ message }: Pick<Choice, 'message'>): ChatCompletionToolMessageParam[] { // iterate through tool calls in choices. return message.tool_calls!.map(call => { // find the matching instance for this tool const { name, guid } = this.splitFunctionName(call.function.name); const toolInstance = this.findToolInstance(name, guid); let content = ''; // call the instance function with params const instance = toolInstance?.instance as any; if (instance && name in instance) { const toolFunction = instance[name]; content = JSON.stringify( toolFunction.call(instance, ...Object.values(JSON.parse(call.function.arguments))) ?? '', ); } … }); } Tools from Codebase
44. export class AiClickDirective { … constructor() { registerInstance(() => …); } … } export function registerInstance(info: () => RegisterInfo) { const aiService = inject(AiService); const destroyRef = inject(DestroyRef); aiService.registerInstance(info); destroyRef.onDestroy(() => aiService.removeInstance(info)); } /** * Finds a tool instance based on the function name and optional GUID. * … */ private findToolInstance(name: string, guid?: string): RegisterInfo | undefined { const className = this.tools.find(tool => tool.function.name === name)?.className; return this.registeredInstances.find(info => { const { className: infoClassName, guid: infoGuid } = info(); return className === infoClassName && guid === infoGuid; })?.(); } Tools from Codebase
46. The Agent–User Interaction (AG-UI) Protocol
47. The Agent–User Interaction (AG-UI) Protocol AG-UI is an open, lightweight, event-based protocol that standardizes how AI agents connect to user-facing applications. https://docs.ag-ui.com/introduction
48. M x N ➡ M+N App 1 App 3 App 2 Database Documentation FAQ App 1 App 3 App 2 Agent Agent Agent AGUI M AI Applications N Tools, Ressources, Prompts ➡
49. Standardisations Without AG-UI With AG-UI Framework fragmentation Every UI must code adapters for LangGraph, CrewAI, Mastra, etc. (M×N problem) Single protocol; any framework works with any UI (M+N) State synchronization Manual polling or complex WebSocket logic JSON Patch state deltas, standardized sync Human-in-the-loop Custom approval workflows per app Standardized HITL events built into protocol Debugging Server-side logs only; UI-agent communication is a black box Full client-side event tracing, transparent execution Generative UI No standard for agents to specify UI components Built-in component rendering events
50. Ecosystem Pydantic AI
51. AG-UI Server https://ai.pydantic.dev/ui/ag-ui/#examples https://docs.copilotkit.ai/pydantic-ai/quickstart/pydantic-ai?path=exiting-agent • Major framework support • "Simple" one-line expose • Fine grained control possible
52. CopilotKit Framework • Manages Communication • Helps with state • Human in the loop (HITL) • Use ready react components AG-UI Client https://docs.copilotkit.ai/pydantic-ai/quickstart/pydantic-ai?path=exiting-agent
53. Server Communication Events Category Description Lifecycle Events Monitor the progression of agent runs Text Message Events Handle streaming textual content Tool Call Events Manage tool executions by agents State Management Events Synchronize state between agents and UI Activity Events Represent ongoing activity progress Special Events Support custom functionality Draft Events Proposed events under development https://docs.ag-ui.com/concepts/events ➡ ∑16 standard events
54. • Use CUSTOM events for Agent → UI side-channel signals (traces, warnings, progress metadata). Custom Events
55. • SSE (Server-Sent Events) - Default, text-based, HTTP • Binary Protocol - High performance, custom serialization • WebSocket - Bidirectional, low latency • HTTP Polling - Fallback for restrictive networks Communication Transport
56. 1. TEXT_MESSAGE_START { messageId, role: "assistant" } 2. TEXT_MESSAGE_CONTENT (×N) { messageId, delta: "Hello" } { messageId, delta: " world" } 3. TEXT_MESSAGE_END { messageId } Text / Tool Streaming Event Pattern 1. TOOL_CALL_START { toolCallId: "call_123", toolName: "search_database" } 2. TOOL_CALL_ARGS { toolCallId: "call_123", args: { query: "users" } } 3. TOOL_CALL_END { toolCallId: "call_123", result: [...] }
57. • State & Context passed besides messages • State for interaction • Context for information State & Context
58. • Bi-directional state management • STATE_SNAPSHOT Complete state object for initialization or refresh • STATE_DELTA JSON Patch (RFC 6902) for ef fi cient incremental updates State
59. Static • Agent speci fi es exact component and props to render • E.g. Component is mapped to a speci fi c event in the UI UI Capbilities Declarative • Agent describes intent, framework selects components • Agent knows about components and props
60. • Simple request/response with no streaming • Batch processing without user interaction • Static content delivery • Ultra-low latency requirements (<10ms) No-Usecase
61. • Keep Humans in the loop • Visual Feedback • Actual questions and commits • Increase acceptance through observability • Loading placeholder effect AG-UI
62. • Documentation: docs.ag-ui.com • GitHub: github.com/ag-ui-protocol/ag-ui • AG-UI Dojo: https://dojo.ag-ui.com/pydantic-ai/ AG-UI Ressources
63. • “Talking to fast learning & smart junior" • No default assumptions • Documentation is key • Examples are great • Write Code for Humans (and be friendly 😊) Development Mindset
64. • Avoid auto-commit • Enable feedback • Highlight probability • Clear entry point into AI work fl ows • Progress / generation indicators • Suggest common actions UI & UX
65. • Client-side for PoC • Easy transition to server-side • Write docs for dummies • Unambiguous tools • Unambiguous context • An Agent is just a loop LLMs do not actually call functions - you do! Lessons Learned
66. It's a wrap! • Contact: @MaxOSchulte, max.marschall@thinktecture.com