ABCS25: Enhancing Legal Document Analysis with Reflection Agents, Semantic Kernel, and Azure AI Search by Cédric Mendelin

⭐️ Enhancing Legal Document Analysis with Reflection Agents, Semantic Kernel, and Azure AI Search#
In this session, we will explore the implementation of Retrieval-Augmented Generation (RAG) on Swiss law documents using Semantic Kernel and Azure AI Search. We will delve into the step-by-step optimization process that enhanced the solution’s efficiency and accuracy. Finally, show the architecture using Reflection Agents and advanced Azure AI Search capabilities. Attendees will gain insights into the challenges faced, the strategies employed to overcome them, and the significant improvements achieved in legal document analysis. This session is ideal for developers, data scientists, and legal tech enthusiasts looking to leverage advanced AI techniques for document processing and analysis. Join us to discover how cutting-edge technology can transform the way we interact with complex legal texts.
🙂 CÉDRIC MENDELIN ⚡️ Senior Software Developer @ isolutions

Transcript

1. ENHANCING LEGAL DOCUMENT ANALYSIS WITH REFLECTION AGENTS, SEMANTIC KERNEL, AND AZURE AI SEARCH
2. CÉDRIC MENDELIN Software Developer - isolutions AG
3. AGENDA LLM Basics Customer Project - FTA The Microsoft Way - Azure AI Services Step-by-Step Improvements - FTA Conclusion
4. LLM BASICS
7. PROMPT History Parameters (Top-P, Temperature) System Message (Persona) Input
8. What is the current date?
11. RAG – RETRIEVAL-AUGMENTED GENERATION Search Engine Data Model User Application
12. RETRIEVAL STEP Is the document relevant for the query? + =
13. EMBEDDINGS Source: https://weaviate.io/blog/how-to-choose-an-embedding-model
14. VECTOR SEARCH VECTORIZE DATA VECTORIZE QUERY VECTOR SEARCH (COSINE SIMILARITY)
15. FEDERAL TAX ADMINISTRATION
16. Federal Tax Administration 4 k
17. LAWS AND ORDINANCES (XML)
18. COURT RULING – FTA PUBLICATIONS (PDF)
19. CHALLENGES Wording/Style of texts Citation requirement Hierarchy in data Amount of data
20. AZURE AI SERVICES
21. AZURE AI SERVICES OpenAI Vision Speech Language Content Safety Face Document Intelligence Azure AI services AI Search AI Agent Service AI Model Inference AI Foundry
22. LOW-CODE VS. PRO-CODE Source: Guest Blog: Semantic Kernel and Copilot Studio Usage Series - Part 1 | Semantic Kernel
23. AZURE OPEN AI • Models • Deployments • Fine Tuning
24. AI SEARCH • Index • Fields • Chunk • Vector Store • Query Processing • Reranking + other advanced features • Indexer & Skills
25. DOCUMENT INTELLIGENCE SERVICE • Markdown • Layout • Tables
27. SEMANTIC KERNEL • Open-Source SDK • Middleware • Abstraction over different Models
29. RAG – Azure OpenAI on your data Azure OpenAI Azure AI Search
33. What is the current date?
36. STEP-BY-STEP IMPROVEMENTS FTA
37. 1ST ITERATION
38. 1ST ITERATION Focus on 50 documents Index Laws per article Index PDFs per page Azure Open AI – on your data Vector Search
39. AZURE OPENAI – ON YOUR DATA
40. 1ST ITERATION – FEEDBACK Relevant documents not found Hallucination Poor Answer Quality Other reasons
41. 2ND ITERATION
42. 2ND ITERATION Hybrid Search Prompting Writing Profiles (Persona) Chat Settings
43. HYBRID SEARCH PIPELINE Vector Search Text Search N Search Results Derive Vector Query Derive Text Query
44. HYBRID SEARCH CODE
45. 2ND ITERATION - FEEDBACK Relevant documents not found Hallucination Poor Answer Quality Other reasons
47. We need more than a gut feeling
48. LLM EVALUATION
49. EVALUATION TYPES LLM evaluation How good the foundation models performs on a certain task. LLM system evaluation How good the LLM performs in your specific use case, on your data, in your domain.
50. EVALUATION PIPELINE LLM System Evaluation Dataset Evaluators Score Input Reference Answer Expected Chunk Answer
51. LLM SYSTEM EVALUATION - METRICS Reference-based Reference-free LLM based
52. MEAI.EVALUATION OVERVIEW • Open-source • Predefined LLM-based evaluators • Interface for custom-evaluators • Local and Azure Storage Account • In Preview
54. MEAI.EVALUATION.CONSOLE
56. EVALUATION QUESTIONS Question Reference Answer Expected Chunk(s) Category
58. EVALUATION RESULTS 40 45 37 43 0 10 20 30 40 50 60 70 80 90 100 Vector Search Hybrid Search % LLM system evaluation – Retrieval Step Provided source Applied source
59. EVALUATION RESULTS 0.8 0.85 0.948 0.948 0.914 0.917 0.41 0.4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Vector Search Hybrid Search LLM system evaluation Relevance Groundedness Cosine Sim Embedding Euclidean Distance
60. 3RD ITERATION
61. 3RD ITERATION AI Enrichment Semantic Reranking
62. AI ENRICHMENT
63. SEMANTIC RERANKING Vector Search Text Search N Search Results Derive Vector Query Derive Text Query Reranking
64. SEMANTIC RERANKING
65. 3RD ITERATION - FEEDBACK Retrieval improved Sometimes poor Answer Quality Other reasons
66. EVALUATION RESULTS 40 45 54 79 37 43 49 75 0 10 20 30 40 50 60 70 80 90 100 Vector Search Hybrid Search Hybrid Search with Summary Hybrid Search with Reranking LLM system evaluation – Retrieval Step Provided source Applied source
67. EVALUATION RESULT 0.8 0.85 0.9 0.948 0.948 0.988 0.914 0.917 0.927 0.41 0.4 0.37 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Vector Search Hybrid Search Hybrid Search with Reranking LLM system evaluation Relevance Groundedness Cosine Sim Embedding Euclidean Distance
68. 4TH ITERATION
69. 4TH ITERATION Reflection Agent
70. REFLECTION AGENT Writer Agent Critic Agent N Fact Checker Style Checker Citation Checker
71. IMPLEMENTATION • Not supported by Azure OpenAI - On your data • Derive Text Query • Using Azure AI Search SDK + Autogen
72. DERIVE TEXT QUERY
74. 4TH ITERATION - FEEDBACK Other Reason
75. EVALUATION – EXECUTION TIME 2889 2925 1999 2170 2966 0 500 1000 1500 2000 2500 3000 3500 OYOD - Hybrid OYOD - Hybrid + SR Custom - Hybrid Custom - Hybrid + SR Custom - Hybrid + SR + Reflection ms LLM system evaluation
76. EVALUATION RESULTS 0.9 0.9 0.98 0.97 0.926 0.925 0.41 0.4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Single Agent Multi Agent LLM system evaluation Relevance Groundedness Cosine Sim Embedding Euclidean Distance
77. CONCLUSION
78. CONCLUSION • Step-by-Step Improvements • SK is your SDK of choice • Azure AI Search for unstructured data • Use Advanced Search Capabilities • Start Evaluating early • What is your Use case • Business Value & Innovation
80. REFERENCES • Flaticon.com (for used icons) • Snappify (for code snippets) • The Microsoft.Extensions.AI.Evaluation libraries (Preview) • Evaluating Large Language Model (LLM) systems: Metrics, challenges, and best practices • Evaluation and monitoring metrics for generative AI • LLM Evaluation Metrics: The Ultimate LLM Evaluation Guide • A list of metrics for evaluation LLM-generated content • https://github.com/joslat/AgenticAIAutoGen