Azure AI Search Best Practices for RAG Chat Apps

Slides presented for AI Chat App Hack-
https://github.com/microsoft/AI-Chat-App-Hack

Transcript

3. Azure AI Search Best Practices for RAG
4. RAG: Retrieval Augmented Generation Document Search PerksPlus.pdf#page=2: Some of the lessons covered under PerksPlus include: · Skiing and snowboarding lessons · Scuba diving lessons · Surfing lessons · Horseback riding lessons These lessons provide employees with the opportunity to try new things, challenge themselves, and improve their physical skills.…. Large Language Model Yes, your company perks cover underwater activities such as scuba diving lessons 1 User Question Do my company perks cover underwater activities?
5. Robust retrieval for RAG chat apps • Relevance is critical for RAG apps • Lots of passages in prompt → degraded quality → Can’t only focus on recall • Incorrect passages in prompt → possibly well-grounded yet wrong answers → Helps to establish thresholds for “good enough” grounding data Source: Lost in the Middle: How Language Models Use Long Contexts, Liu et al. arXiv:2307.03172 50 55 60 65 70 75 5 10 15 20 25 30 Accuracy Number of documents in input context
6. Optimal retrieval in Azure AI Search Vector Keywords Fusion (RRF) Reranking Complete search stacks do better: Hybrid retrieval (keywords + vectors) > pure-vector or keyword Hybrid + Reranking > Hybrid
7. Vector search
8. Vector embeddings An embedding encodes an input as a list of floating-point numbers. ”dog” → [0.017198, -0.007493, -0.057982, 0.054051, -0.028336, 0.019245,…] Different models output different embeddings, with varying lengths. Model Encodes Vector length word2vec words 300 Sbert (Sentence-Transformers) text (up to ~400 words) 768 OpenAI ada-002 text (up to 8191 tokens) 1536 Azure Computer Vision image or text 1024 ….and many more models! Demo: Compute a vector with ada-002 (aka.ms/aitour/vectors)
9. Vector similarity We compute embeddings so that we can calculate similarity between inputs. The most common distance measurement is cosine similarity. Demo: Vector Embeddings Comparison (aka.ms/aitour/vector-similarity) Demo: Compare vectors with cosine similarity (aka.ms/aitour/vectors) Similar: θ near 0 cos(θ) near 1 Orthogonal: θ near 90 cos(θ) near 0 Opposite: θ near 180 cos(θ) near -1 def cosine_sim(a, b): return dot(a, b) / (mag(a) * mag(b)) *For ada-002, cos(θ) values range from 0.7-1
10. Vector search 1. Compute the embedding vector for the query 2. Find K closest vectors for the query vector Search exhaustively or using approximations Query Compute embedding vector Query vector Search existing vectors K closest vectors “tortoise” OpenAI ada-002 create embedding [-0.003335318, - 0.0176891904,…] Search existing vectors [[“snake”, [-0.122, ..], [“frog”, [-0.045, ..]]] Demo: Search vectors with query vector (aka.ms/aitour/vectors)
11. Vector search in Azure AI Search Comprehensive vector search solution Enterprise-ready → scalability, security and compliance Integrated with Semantic Kernel, LangChain, LlamaIndex, Azure OpenAI Service, Azure AI Studio, and more Generally available Demo: Azure AI search with vectors (aka.ms/aitour/azure-search)
12. Vector search strategies ANN search • ANN = Approximate Nearest Neighbors • Fast vector search at scale • Uses HNSW, a graph method with excellent performance-recall profile • Fine control over index parameters Exhaustive KNN search • KNN = K Nearest Neighbors • Per-query or built into schema • Useful to create recall baselines • Scenarios with highly selective filters • e.g., dense multi-tenant apps r = search_client.search( None, top=5, vector_queries=[VectorizedQuery( vector=search_vector, k_nearest_neighbors=5, fields="embedding")]) r = search_client.search( None, top=5, vector_queries=[VectorizedQuery( vector=search_vector, k_nearest_neighbors=5, fields="embedding", exhaustive=True)])
13. Rich vector search query abilities Filtered vector search • Scope to date ranges, categories, geographic distances, access control groups, etc. • Rich filter expressions • Pre-/post-filtering • Pre-filter: great for selective filters, no recall disruption • Post-filter: better for low-selectivity filters, but watch for empty results r = search_client.search( None, top=5, vector_queries=[VectorizedQuery( vector=query_vector, k_nearest_neighbors=5, fields="embedding")], vector_filter_mode=VectorFilterMode.PRE_FILTER, filter= "tag eq 'perks' and created gt 2023-11-15T00:00:00Z") r = search_client.search( None, top=5, vector_queries=[ VectorizedQuery( vector=query1, fields="body_vector", k_nearest_neighbors=5,), VectorizedQuery( vector=query2, fields="title_vector", k_nearest_neighbors=5,) ]) Multi-vector scenarios  Multiple vector fields per document  Multi-vector queries  Can mix and match as needed Filters in vector queries (aka.ms/aisearch/vectorfilters)
14. Hybrid search
15. Optimal retrieval in Azure AI Search Identify good & bad candidates Normalized scores from semantic ranker Exclude documents below a threshold Demo: Compare text, vector, hybrid, reranker (aka.ms/aitour/search-relevance) Vector Keywords Fusion (RRF) Reranking Complete search stacks do better: Hybrid retrieval (keywords + vectors) > pure-vector or keyword Hybrid + Reranking > Hybrid
16. Retrieval relevance across methods 41 41 50 44 45 58 48 48 59 60 50 72 0 10 20 30 40 50 60 70 80 Customer datasets Beir dataset Miracl dataset Accuracy Score Keyword Vector (ada-002) Hybrid Hybrid + reranking Outperforming vector search with hybrid + reranking (aka.ms/ragrelevance)
17. Impact of query types on relevance Outperforming vector search with hybrid + reranking (aka.ms/ragrelevance) Query type Keyword [NDCG@3] Vector [NDCG@3] Hybrid [NDCG@3] Hybrid + Semantic ranker [NDCG@3] Concept seeking queries 39 45.8 46.3 59.6 Fact seeking queries 37.8 49 49.1 63.4 Exact snippet search 51.1 41.5 51 60.8 Web search-like queries 41.8 46.3 50 58.9 Keyword queries 79.2 11.7 61 66.9 Low query/doc term overlap 23 36.1 35.9 49.1 Queries with misspellings 28.8 39.1 40.6 54.6 Long queries 42.7 41.6 48.1 59.4 Medium queries 38.1 44.7 46.7 59.9 Short queries 53.1 38.8 53 63.9
18. Azure AI search data indexing
19. Manual indexing You can use the SDK to write your own code to add data to an index. Data ingestion guide: Adding documents aka.ms/ragchat/add-data Azure Storage Document Intelligence Example: prepdocs.py Azure OpenAI Azure AI Search Computes embeddings Stores in index Extracts data from PDFs Splits data into chunks Python Stores PDFs
20. Cloud-based indexing Indexers: Connect the search service to a cloud data source, and it will index the data periodically or on a trigger. •Azure Blob Storage •Azure Cosmos DB •Azure Data Lake Storage Gen2 •Azure SQL Database •SharePoint in Microsoft 365 •Azure Cosmos DB for MongoDB …and more! Data source Indexer Target Index Indexers in Azure AI Search (aka.ms/aisearch/indexers)
21. Skillsets for indexers Skillset: A set of skills that prepare a document for indexing, calling either built-in AI search functions or custom code. Skillset concepts in Azure AI Search (aka.ms/aisearch/skillsets)
22. Integrated vectorization A combination of indexers and built-in skills for chunking and vectorization. Data source access • Blob Storage • ADLSv2 • SQL DB • CosmosDB • … + Incremental change tracking File format cracking • PDFs • Office documents • JSON files • … + Extract images and text, OCR as needed Chunking • Split text into passages • Propagate document metadata Vectorization • Turn chunks into vectors • OpenAI embeddings or your custom model Indexing • Document index • Chunk index • Both In preview Integrated data chunking and embedding in Azure AI Search (aka.ms/integrated- vectorization)
23. Integrated vectorization in RAG chat repo Once the PR is merged, you can opt to use it via: PR: Adding integrated vectorization support (aka.ms/ragchat/intvect) azd env set USE_FEATURE_INT_VECTORIZATION true azd up
24. Manual indexing vs. Integrated vectorization Pros: • All code is local and easy to change. Cons: • Hard to connect to indexers for cloud- based data. • Has to be manually re-run for new data. Pros: • Easily connect to indexers that can add new data on triggers or periodically. • You don’t need to maintain chunking or embedding code yourself. Cons: • Currently in preview mode. • Customizing the skills takes more effort, if the built-in skills are not sufficient.
25. Azure AI search advanced features
26. Analyzers Analyzers are components of the full-text search engine for processing strings during indexing and query execution. • Language analyzers: If you’re indexing non-English documents in particular, consider customizing the analyzer used. • Custom analyzers: Useful for custom tokenization, like to recognize phone numbers, word normalization, etc. Analyzers for text processing in Azure AI Search (aka.ms/aisearch/analyzers)
27. Scoring profiles Scoring profiles are criteria for boosting a search score based on custom parameters. Add scoring profiles to boost search scores (aka.ms/aisearch/scoring) "scoringProfiles": [ { "name": "boostKeywords", "text": { "weights": { "HotelName": 2, "Description": 5 } } } ]
28. Next steps • Register for the hackathon → • Introduce yourself in our discussion forum • Deploy the repo with the sample data • See steps on low cost deployment → • Start customizing the project! • Post in forum if you have any issues deploying or questions about customization. • Join tomorrow’s session: GPT-4 with Vision aka.ms/hacktogether/chatapp aka.ms/ragchat/free