PyconKE25: Building AI Agents with Azure AI Foundry and Prompty

In this session, I’ll share how I built a lightweight AI agent using Prompty and Azure OpenAI to automate GitHub issue tagging, one of the many small but time-consuming tasks in open source projects. You’ll get a behind-the-scenes look at how I use prompt engineering to streamline contributor experience, enhance repo hygiene, and turn repetitive maintenance work into intelligent automation. Whether you're managing community-driven projects or just exploring AI tools in DevRel, this talk will show you how agents can help you focus on what truly matters: empowering developers.

Transcript

1. Building Agents with Azure AI Foundry & Prompty Bethany Jepchumba | @bethanyjep
2. • Cloud Advocate (DevRel) at Microsoft • Works with NGOs & Communities, currently ICT Director at TOFA • I am a hobbist, I enjoy trying out new hobbies I occasionally blog at bethany-jep.com Bethany Jepchumba Microsoft confidential 2
3. Question: Have you built an agent before?
4. and building locally The concept of LLMs…. 4
5. How language models work Natural language input Model Tokens Probability distribution Natural language output Decoding + Post-processing Get results Pre-processing
6. Foundry Local Design, customize and manage AI applications and agents on-device Preview Microsoft confidential 6 Installation Privacy and Security Latency Cost Efficient Low Bandwidth
7. RAG, Agents and MCP Giving your LLMs some powers, sort of…. 7
8. BYOD -> Retrieval-Augmented Generation (RAG)
9. What is an AI agent? LLM Instructions Tools Agent + + An AI agent is a micro-service that takes unstructured messages, optionally invokes other APIs and returns messages/action 1 2 3 Input System events User messages Agent messages 1 Tool calls Knowledge Actions Memory 2 Output Agent messages Tool results 3
10. Unlike previous tools, AI Agent can… reason, act and learn
11. What is Model Context Protocol (MCP)? Ship once – reach every MCP-enabled client in your organization A secure, open standard that lets AI agents access tools, APIs, and data through a consistent interface — without custom integrations. In Azure AI Foundry Agent Service, MCP connects agents to your enterprise and external systems with a single, reusable protocol — secured, governed, and observable by Azure. Clients • Foundry Agents • Copilot Studio • GitHub Copilot • VS Code Servers/ Tools • Azure Services (Search, Cosmos DB, Power Platform, etc.) • Third-party APIs (Salesforce, ServiceNow, REST APIs) • On-prem data MCP Protocol • One standard interface • Security: Entra ID • Observability: Azure Monitor
12. MCP Model Context Protocol - Easier to give context to models - Optimized communication between LLMs, external tools, data sources and applications - Uses a client-server model for interactions
13. Model Context Protocol At Microsoft Model Context Protocol MCP enables seamless integration between LLM Apps and external data sources Azure API Management Build Local & Remote MCPs MCP SDK Microsoft Connectors Azure Functions Tools Etc. … Community Windows Registry VS Code BYO Enterprise or MCP Clients & Host Copilot Studio Foundry agents Visual Studio Windows VS Code GitHub Copilot Entra ID
14. Enhancing Security for MCP Servers Use credential manager to authorize access to your backend MCP servers Protect your remote MCP servers with OAuth 2.0 and API Management Blog aka.ms/remote-mcp-apim-auth-blog Generally Available Azure API Management Safety Policies Azure OpenAI MCP enabled Tools Azure Monitor Logs & Metrics … Credential Manager Agents …
15. MCP Deep-Dive Adapted from Mahesh Murag (Anthropic) AI Engineer session MCP Client  Invokes Tools  Queries for Resources  Interpolates Prompts  Exposes Roots  Performs Sampling MCP Server  Exposes Tools  Exposes Resources  Exposes Prompts  Honors Roots  Requests Sampling Tools Model-controlled Functions invoked by the model Resources Application-controlled Data exposed to the application Prompts User-controlled Pre-defined templates for AI interactions Retrieve / search Send a message Update DB records Files Database records API Responses Document Q&A Transcript Summary Output as JSON
16. Azure AI Foundry Security • Identity • Management Foundry Models Foundry Agent Service Azure AI Search Foundry Observability Azure AI Services Azure Machine Learning Azure AI Content Safety Copilot Studio Visual Studio GitHub Foundry SDK Serverless Control Azure Kubernetes Service Azure Container Apps Azure App Service Azure Functions Cloud Azure Azure Arc Foundry Local Edge
17. Showcasing Multiple Demos…
18. Security, monitoring and evaluations. Agents in production…. 18
19. Generative AI Development Lifecycle Hypo thesis Find LLMs Try prompts Manage Customize Design Managing PREPARE FOR APP DEPLOYMENT ADVANCE PROJECT BUSINESS NEED Deploy LLM App/UI Quo ta and cost management REVERT PROJECT C ontent Filtering M onitoring SEND FEEDBACK Prompt Engine ering or Fine-tuning Evalua tion Exceptio n Handling Retrieval Augmente d Generation Prote ct & govern
20. How do you typically test if a model or application is working as intended?
21. Mitigation layers Application Platform
22. Evaluators in Azure AI Foundry RAG Retrieval Document Retrieval Groundedness Relevance Response Completeness General Purpose Fluency Coherence QA Safety & Security Violence Sexual Self-harm Hate and Unfairness Ungrounded Attributes Code Vulnerability Protected Materials Content Safety Agents Intent Resolution Task Adherence Tool call accuracy
23. Showcasing Demo: AI Evaluators
24. Agents Use Cases
25. From english to xx language in a single click
26. AI Agent workflows can enhance efficiency, accuracy, and customer satisfaction Key Use Cases Across Industries • Assists employees in booking business trips • Integrates with Tripadvisor, Outlook, and SharePoint • Books via Teams chat or email • Uses OCR to gather receipts • Automates expense report submission and tracking Travel Booking & Expense Management • Personalized onboarding assistant for new hires • Uses LLMs grounded in HR data from SharePoint • Provide relevant training materials • Schedule orientations and set up software accounts • Monitor task completion and ensure efficient onboarding Employee Onboarding • Diagnoses issues by referencing history and product manuals • Provides tailored solutions or escalates through automated workflows • Creates tickets and schedules follow-ups • Updates CRM records, enhancing future support Personalized Customer Support • Analytics data from data lake and data warehouse • Responds to user requests in natural language • Generates insights, visualization, and sends via Teams or email • Automates data handling for real-time, effortless decision-making Data Analytics and Reporting
27. Resources - Learning resources on GenAI, agents and MCP - Getting started with Azure AI - Links to the different tools – foundry local, ai toolkit on VSCode, GitHub Models and GitHub Copilot - Links to different articles and use cases https://bethany-jep.com/events/ai- summit-kenya-2025/
28. Thank You