What is XAI?

An overview of explainable artificial intelligence (XAI), covering its origins, motivations, representative methods, and fundamental limitations. The talk also introduces physics-inspired approaches to understanding the internal structures of trained models and discusses how explainability is becoming part of AI standards, governance, and regulation through NIST and the EU AI Act.

Transcript

1. What is XAI? Keio University Lab Meeting July 23 Hiroshi Watanabe 1 19
2. What is XAI? XAI: eXplainable Artificial Intelligence This is an approach to AI that makes model decisions understantable to humans. It helps users answer questions such as, Explaintability: Why did the model make this prediction? Trustworthiness: Can we trust this decision? Reliability: When might the model fail? 2 19
3. Why XAI? Modern AI can achieve impressive predictive performance, but many machine-learning models operate as black boxes. Why did you brake suddenly? Because I saw a cat run into the road. AI may generate a plausible excuse rather than reveal its true reasoning. XAI addres this problem by depeloping methods that make AI decision more transparent and understandable, while 3 presering their performance. 19
4. Scope of This Talk XAI is still an evolving research field, and its definition remains broad and sometimes ambiguous. This talk examines XAI from four perspectives: Origins and motivation: Why explainability became important as AI systems grew more powerful and less transparent. Methods and limitations: Representative explanation techniques, together with their weaknesses and criticisms. Physics-inspired interpretation: How ideas from phase transitions and statistical mechanics may help reveal internal learning structures. Governance and standards: How explainability is reflected in the EU AI Act and NIST principles. 4 19
5. Origins of XAI (1/2) The term XAI was introduced by Van Lent et al. (2004). They introduced the term to describe an AI system that explains the behavior of AI-controlled units in Full Spectrum Command (FSC), a military training simulation game. M. Van Lent, W. Fisher, and M. Mancuso, “An Explainable Artificial Intelligence System for Small-Unit Tactical Behavior,” in Proceedings of the National Conference on Artificial Intelligence (AAAI), 900–907 (2004). 5 19
6. Origins of XAI (2/2) However, the concept of XAI is much older. Since the 1970s, researchers have studied how to make expert systems explain their reasoning. Expert systems Expert systems are rule-based AI systems: essentially large collections of if–then rules that emulate the decisionmaking of human experts. Have a fever? Have a chest pain? no Have a headache? no no ・・・ yes yes Common cold? Myocardinal infaction? Stroke? W. R. Swartout and J. D. Moore, “Explanation in expert systems: A survey,” Univ. Southern California, Los Angeles, CA, USA, Tech. Rep. ISI/RR-88-228, (1988). 6 19
7. Unexpected Failure (1/2) Hallucination Modern AI systems can generate highly fluent and convincing outputs. However, they somtimes produce information that is not grounded in facts or input data. This failure is called hallucination. In 2018, Janelle Shane reported that Microsoft Azure’ s image-recognition system repeatedly detected sheep in photographs of empty grassy landscapes. https://nautil.us/this-neural-net-hallucinates-sheep-237006 7 19
8. Unexpected Failure (1/2) Adversarial Examples AI models can also be fooled by small, carefully designed changes to the input. A small perturbation changes the prediction from pand to gibbon with 99.3% confidence [1]. Carefully designed stickers on a STOP sigh is recognized as as Speed Limit 45 sign [2]. [1] I. J. Goodfellow, et al. “Explaining and Harnessing Adversarial Examples,” in International Conference on Learning Representations (ICLR), (2015). [2] K. Eykholt, et al. ”Robust Physical-World Attacks on Deep Learning Visual Classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1625–1634 (2018). 8 19
9. XAI Examples: Visual Explanations Wolf-husky problem The classifier appeared to distinguish wolves from huskies, but the explanation revealed that it was mainly using snow in the background. Grad-CAM Grad-CAM visualizes which regions of the image the model relied on when identifying the dog or the cat. [1] M. T. Ribeiro, S. Singh, and C. Guestrin, “Why Should I Trust You?: Explaining the Predictions of Any Classifier,” in KDD, 1135–1144 (2016). [2] R. R. Selvaraju et al., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization,” in ICCV, 618–626 (2017). 9 19
10. DARPA XAI Project As AI systems became more powerful but less transparent, the need for explainability became increasingly urgent. In response, DARPA (Defence Advanced Research Projects Agency) launched the XAI program. Explainability: Produce more explainable models Performance: Maintain high prediction accuracy Human understanding: Help users understand, appropriately trust, and effectively manage AI systems Program period: 2017 - 2021 Budget: US $75 million DARPA, “Explainable Artificial Intelligence (XAI),” https://www.darpa.mil/research/programs/explainable-artificial-intelligence 10 19
11. Rapid Growth of XAI Research Adadi and Berrada reviewed 381 papers on XAI. DARPA XAI project They reported a rapid, almost exponential increase in XAI-related publications. A. Adadi and M. Berrada, "Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)," in IEEE Access, 6, 52138-52160, (2018). 11 19
12. Why Use XAI? Adadi and Berrada classify the purposes of using XAI into the following four categories. Explain to Justify Provide reasons for decisions and verify that they are not biased or discriminatory. Ex) “right to explanation” is included in the General Data Protection Regulation (GDPR). GDPR: An EU regulation on personal data protection Explain to Control that includes transparency and safeguards for certain automated decisions. Reveal unknown vulnerabilities, flaws, and errors so that developers can correct them. Explain to Improve Use explanations to understand model behavior and improve its accuracy and efficiency. Ex) Knowing why a system produced an output helps users improve it. Explain to Discover Explanations can reveal new knowledge learned by AI. Ex) AlphaGo Zero could explain strategies beyond human expertise, while future XAI may uncover hidden laws in science. 12 19
13. Critiques of XAI Is Explanation Really Necessary? P. Norvig argues that humans are not very good at explaining their own decisions either. The credibility of an AI system may instead be assessed by observing the reliability of its outputs over time. P. Norvig, “Google’ s Approach to Artificial Intelligence and Machine Learning,” UNSW Sydney, June 22, (2017). Are Explanations Reliable? Ghorbani et al. demonstrated that small input perturbations can drastically change an explanation without changing the prediction. A. Ghorbani et al., “Interpretation of Neural Networks Is Fragile,” AAAI (2019). 13 19
14. Physics-inspired XAI (1/2) In recent years, physicists have made intensive efforts to understand the internal states of trained models using concepts and methods from physics. A neural network was trained to predict temperature from Ising/Potts configurations. Analysis of its hidden layer revealed the order parameter. The model spontaneously discovered the order parameter. K. Kashiwa, Y. Kikuchi, and A. Tomiya, Prog. Theor. Exp. Phys, 2019, 083A04 (2019). 14 19
15. Physics-inspired XAI (2/2) Phase diagram of trained DBM Training induces correlations in weight parameters, which determine phase behavior. In addition to the paramagnetic and ferromagnetic phases, a spin-glass phase was also observed. Y. Ichikawa and K. Hukushima, J. Phys. Soc. Jpn., 91, 114001 (2022). Design space of DNN Increasing training constraints can cause successive phase transitions across layers. H. Yoshino, SciPost Phys. Core, 2, 005 (2020). 15 19
16. EU AI Act (1/2) The EU AI Act regulates AI according to four risk levels. Unacceptable Social scoring systems and manipulative AI High Risk AI in employment, education, critical infrastructure, law enforcement, healthcare-related products, etc. Limited Risk Users must know when they interact with AI, such as chatbots or deepfakes. Minimal Risk AI enabled video games and spam filters https://artificialintelligenceact.eu/ 16 19
17. EU AI Act (2/2) Risk Level Regulation Application From Unacceptable Prohibited 2. Feb. 2025 High Regulated 2. Aug. 2026 Limited Disclosure required 2. Aug. 2026 Minimal not regulated 17 19
18. NISTIR 8312 NIST (National Institute of Standards and Technology) proposed four principles for XAI In NISTIR 8312. Explanation A system delivers or contains accompanying evidence or reason(s) for outputs and/or processes. Meaningful A system provides explanations that are understandable to the intended consumer(s). Explanation Accuracy An explanation correctly reflects the reason for generating the output and/or accurately reflects the system’s process. Knowledge Limits A system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output. P. J. Phillips et al., Four Principles of Explainable Artificial Intelligence, NISTIR 8312 (2021). https://doi.org/10.6028/NIST.IR.8312 18 19
19. Summary • XAI emerged from the need to understand increasingly powerful but opaque AI systems. • Explanations can support trust, diagnosis, improvement, and scientific discovery. • However, plausible explanations are not always faithful, stable, or secure. • Physicists are also investigating the internal structures of AI models using concepts and methods from physics. • Explainability is now becoming part of AI governance, standards, and regulation. As explainability becomes a societal and regulatory requirement, further research on reliable and meaningful XAI is essential. 19 19