Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Train a model and debug it with Responsible AI ...

Train a model and debug it with Responsible AI Dashboard

Learn how to debug an AI model using the Responsible AI dashboard in Azure Machine Learning studio to ensure it performs responsibly and is less harmful. This deck is a "train the trainer" presentation that helps community instructors walk through a 60-minute workshop on Responsible AI, based on this Microsoft Learn Module:
https://learn.microsoft.com/training/modules/train-model-debug-with-responsible-ai-dashboard-azure-machine-learning

Nitya Narasimhan, PhD

January 12, 2024
Tweet

More Decks by Nitya Narasimhan, PhD

Other Decks in Technology

Transcript

  1. Responsible AI Dashboard (Website) https://aka.ms/rai-hub/dashboard-workshop Train a model and debug

    it with Responsible AI Dashboard Microsoft Learn Module Nitya Narasimhan, PhD Senior AI Advocate Microsoft @nitya | https://linkedin.com/in/nityan Responsible AI Collection (Resources) https://aka.ms/rai-hub/collection
  2. 1 | Prepare The Lab – Launch & Run Notebooks

    Early 2 | What is a Responsible AI Dashboard? 3 | Error Analysis On A Model 4 | Find Model Performance Inconsistencies 5 | Expose Data Biases 6 | Explain & Interpret Model 7 | Complete Exercises – Prioritize Dashboard Section 8 | Knowledge Check 9 | Summary
  3. Visit the link above to get access to the Microsoft

    Learn Training module shown https://aka.ms/mslearn-responsibleai-dashboard Launch This Lab!
  4. Lab Exercise: What We Will Do Today Create AML Workspace

    & register model in Azure ML Studio for use Notebook 1 Notebook 2 We’ll use a dataset of diabetes patient records to build & register a model that predicts if a patient will be readmitted within 30 days of discharge. Build & Train an ML Model for the UCI Diabetes Dataset Use the dashboard to debug the model using various tools, in an intuitive workflow. Create a Responsible AI Dashboard with specific components, for the model. We’ll assess model for responsible AI practices by using the error analysis, model overview, data analysis & feature analysis components for insights
  5. Stage 1 – Prepare Lab Environment See Video | https://aka.ms/rai-hub/raidashboard-module-walkthrough

    Get Familiar with VM Inputs, Instructions Open VS Code & Log into Azure Create Azure ML Workspace & save configuration to file
  6. Stage 2 – Build & Register Your Model In Azure

    See Video | https://aka.ms/rai-hub/raidashboard-module-walkthrough Configure Jupyter Notebook, then Run All to complete task Open Azure ML Studio to track and verify completion Notebook 1
  7. 1 | Prepare The Lab – Launch & Run Notebooks

    Early 2 | What is a Responsible AI Dashboard? 3 | Error Analysis On A Model 4 | Find Model Performance Inconsistencies 5 | Expose Data Biases 6 | Explain & Interpret Model 7 | Complete Exercises – Prioritize Dashboard Section 8 | Knowledge Check 9 | Summary
  8. We live in a data-driven world with growing demand for

    machine- learning models that perform efficiently and ethically for real users. The AI Landscape is Changing Fast AI innovation is occurring at a rapid pace AI adoption is being accelerated by organizations AI expectations are evolving within our societies AI regulations are being debated by governments
  9. How can I debug my models, take business decisions with

    confidence, and gain user trust with my AI-driven solutions – at scale? Re Practitioners Need to Operationalize Responsible AI Data Scientists Decision Makers Application Developers
  10. Responsible AI Principles guide the approach to assessing, developing &

    deploying AI systems in safe, trustworthy & ethical ways. Responsible AI Dashboard is a single pane of glass for holistic responsible assessment through stages of model debugging & decision-making Responsible AI Toolbox is a collection of integrated tools & functionalities to help operationalize Responsible AI in practice for data scientists, developers & decision-makers. Reference: https://responsibleaitoolbox.ai Responsible AI Dashboard Defined
  11. Responsible AI 6 Principles Fairness – How might an AI

    system allocate opportunities in ways that are fair to the humans who use it? Reliability & Safety – How might the system function well for people even in conditions it was not designed for? Privacy & Security – How might the system be designed to preserve data privacy and security, compliant with laws? Inclusiveness – How might an AI system be designed to be inclusive of people of all abilities (think accessibility)? Transparency – How might people misunderstand, misuse or incorrectly estimate the capabilities of the AI System? Accountability – How can we create oversight so that humans can be accountable and in control of AI behaviors? Reference: https://www.microsoft.com/ai/principles-and-approach/
  12. Error Analysis – Dive deep into how model failures are

    distributed across different ‘cohorts’ (based on inputs, features) Data Explorer – Visualize datasets to understand issues of over- or under- representation in dataset, by cohorts or feature groups. Use it to conduct fairness assessments & identify areas for mitigation. Model Overview – Understand how model performance varies by metrics or data subgroups in data. Decide first steps in debugging. Interpretability – How trustworthy are AI predictions? Give human- friendly explainers for outcomes & features that influence it most Counterfactuals/What-If – How might people misunderstand the AI system capabilities? How can they get a desirable outcome from it? Causal Inference – Identify features with most direct effect on a desired outcome & estimate the impact from an intervention. Reference: https://responsibleaitoolbox.ai/introducing-responsible-ai-dashboard/#dashboard-components Responsible AI Dashboard Composable Components
  13. Reference: https://responsibleaitoolbox.ai/introducing-responsible-ai-dashboard Responsible AI Dashboard Model Debugging Step 1: Identify

    What kinds of errors does my model have? In which areas are they most prevalent? Step 2: Diagnose What causes these errors? Where should I focus resources? Step 3: Mitigate How can I improve the model? What social or technical solutions exist for it?
  14. Reference: https://responsibleaitoolbox.ai/introducing-responsible-ai-dashboard/#decision-making Responsible AI Dashboard Decision Making Inform Your Business

    Decisions Data-Driven = Historical “How would a medicine impact the patient’s blood pressure?” Model-Driven = Causal “What should I do to get a different outcome from your AI next time?”
  15. Reference: https://archive.ics.uci.edu/dataset/296/diabetes+130-us+hospitals+for+years+1999-2008 We’ll build & register a model based on

    the UCI Diabetes dataset which predicts if a diabetes patient will get readmitted within 30 days. We’ll create a responsible AI dashboard & debug model using error analysis, model overview, data analysis & feature analysis tools. Responsible AI Dashboard Training Today
  16. Reference: https://aka.ms/rai-hub/raidashboard-module-walkthrough Responsible AI Dashboard Lab Video Walkthrough Use the

    video walkthrough as a reference or for a preview of lab Check description for chapter timestamps to move quickly to the desired section of lab Click “View all” to get a playlist view (Key Moments) of chapters
  17. 1 | Prepare The Lab – Launch & Run Notebooks

    Early 2 | What is a Responsible AI Dashboard? 3 | Error Analysis On A Model 4 | Find Model Performance Inconsistencies 5 | Expose Data Biases 6 | Explain & Interpret Model 7 | Complete Exercises – Prioritize Dashboard Section 8 | Knowledge Check 9 | Summary
  18. Lab Exercise: What We Will Do Today Create AML Workspace

    & register model in Azure ML Studio for use Notebook 1 Notebook 2 We’ll use a dataset of diabetes patient records to build & register a model that predicts if a patient will be readmitted within 30 days of discharge. Build & Train an ML Model for the UCI Diabetes Dataset Use the dashboard to debug the model using various tools, in an intuitive workflow. Create a Responsible AI Dashboard with specific components, for the model. We’ll assess model for responsible AI practices by using the error analysis, model overview, data analysis & feature analysis components for insights
  19. Stage 3 – Add Responsible AI Dashboard To Model See

    Video | https://aka.ms/rai-hub/raidashboard-module-walkthrough Notebook 2 Configure Jupyter Notebook, then Run All to complete task Open Azure ML Studio and open the RAI Dashboard tab
  20. Azure ML Studio: Responsible AI Dashboard View See Video |

    https://aka.ms/rai-hub/raidashboard-module-walkthrough Notebook 2 Configure Jupyter Notebook, then Run All to complete task Open Azure ML Studio and open the RAI Dashboard tab
  21. Task: Error analysis on a model Traditional performance metrics for

    machine learning models focus on calculations based on correct vs incorrect predictions. Error distribution Error Analysis -dashboard
  22. Task: Model performance inconsistencies  An effective approach to evaluating

    the performance of machine learning models is getting a holistic understanding of their behavior across different scenarios. • Disparities among performance metric • Showing how model is performing for a given cohort using metrics such as Accuracy, Precision, Recall, MAE, RSME etc. • Probability distribution • Showing the probability of a given cohort to fall in a model's predicted outcome. • Metric visualization • Showing performance scores for cohort. Model Overview dashboard
  23. Task: Expose data biases A major blind-spot and a very

    important part of model behavior is the actual data . Over/Under/Lack of Representation Data Analysis dashboard
  24. Task: Explain and interpret a model Assessing a model isn't

    just about understanding how accurately it can make a prediction, but also why it made the prediction. • Model debugging: Why did my model make this mistake? How can I improve my model? • Human-AI collaboration: How can I understand and trust the model's decisions? • Regulatory compliance: Does my model satisfy legal requirements? Blackbox – hard to understand Explain model behavior - dashboard
  25. 1 | Prepare The Lab – Launch & Run Notebooks

    Early 2 | What is a Responsible AI Dashboard? 3 | Error Analysis On A Model 4 | Find Model Performance Inconsistencies 5 | Expose Data Biases 6 | Explain & Interpret Model 7 | Complete Exercises – Prioritize Dashboard Section 8 | Knowledge Check 9 | Summary
  26. Lab Exercise: What We Will Do Today Create AML Workspace

    & register model in Azure ML Studio for use Notebook 1 Notebook 2 We’ll use a dataset of diabetes patient records to build & register a model that predicts if a patient will be readmitted within 30 days of discharge. Build & Train an ML Model for the UCI Diabetes Dataset Use the dashboard to debug the model using various tools, in an intuitive workflow. Create a Responsible AI Dashboard with specific components, for the model. We’ll assess model for responsible AI practices by using the error analysis, model overview, data analysis & feature analysis components for insights
  27. Stage 4 – Debug Model Using The RAI Dashboard See

    Video | https://aka.ms/rai-hub/raidashboard-module-walkthrough Azure ML Studio Minimize sidebar to see full dashboard Walk through four dashboard tasks to debug model & assess compliance
  28. 1 | Prepare The Lab – Launch & Run Notebooks

    Early 2 | What is a Responsible AI Dashboard? 3 | Error Analysis On A Model 4 | Find Model Performance Inconsistencies 5 | Expose Data Biases 6 | Explain & Interpret Model 7 | Complete Exercises – Prioritize Dashboard Section 8 | Knowledge Check 9 | Summary
  29. 1 | Prepare The Lab – Launch & Run Notebooks

    Early 2 | What is a Responsible AI Dashboard? 3 | Error Analysis On A Model 4 | Find Model Performance Inconsistencies 5 | Expose Data Biases 6 | Explain & Interpret Model 7 | Complete Exercises – Prioritize Dashboard Section 8 | Knowledge Check 9 | Summary
  30. Model Debugging: What We Learned Today When we train a

    machine learning model, we want it to learn or uncover patterns. • Create a Responsible AI dashboard. • Identify where the model has errors. • Discover data over or under representation to mitigate biases. • Understand what drives a model outcome with explainable and interpretability. • Mitigate issues to meet compliance regulation requirements.
  31. Lab Exercise: What We Did Today Create AML Workspace &

    register model in Azure ML Studio for use Notebook 1 Notebook 2 We’ll use a dataset of diabetes patient records to build & register a model that predicts if a patient will be readmitted within 30 days of discharge. Build & Train an ML Model for the UCI Diabetes Dataset Use the dashboard to debug the model using various tools, in an intuitive workflow. Create a Responsible AI Dashboard with specific components, for the model. We’ll assess model for responsible AI practices by using the error analysis, model overview, data analysis & feature analysis components for insights
  32. Reference: https://aka.ms/rai-hub/raidashboard-module-walkthrough Responsible AI Dashboard Lab Video Walkthrough Use the

    video walkthrough as a reference or for a preview of lab Check description for chapter timestamps to move quickly to the desired section of lab Click “View all” to get a playlist view (Key Moments) of chapters