Machine Learning + Symbolic Reasoning: a Quarkus story on Artificial Intelligence

What AI can do nowadays is simply mind-blowing. I must admit that I cannot stop being surprised and sometimes literally jumping from my seat thinking: "I didn't imagine that AI could ALSO do this!". What is a bit misleading here is that what we tend to identify with Artificial Intelligence is actually Machine Learning which is only a subset of all AI technologies available: ML is a fraction of the whole AI-story, while Symbolic Artificial Intelligence enables experts to encode their knowledge of a specific domain through a set of human-readable and transparent rules.

In fact there are many situations where being surprised is the last thing that you may want. You don't want to jump from your seat when your bank refuses your mortgage without any human understandable reason, but only because AI said no. And even the bank may want to grant their mortgages only to applicants who are considered viable under their strict and well-defined business rules.

Given these premises why not mixing 2 very different and complementary AI branches like Machine Learning and Symbolic Reasoning? During this talk we will demonstrate with practical examples why this could be a winning architectural choice in many common situations and how Quarkus through its langchain4j and drools extensions makes the development of applications integrating those technologies straightforward.

Transcript

1. Machine Learning + Symbolic Reasoning: A Quarkus story on Artificial Intelligence by Mario Fusco + Nicole Prentzas +
2. Why not Machine Learning alone?
3. What is Machine Learning? ML is the thing suggesting to buy laundry detergent together with your new washing machine because they're statistically related ...
4. ML is the thing suggesting to buy laundry detergent together with your new washing machine because they're statistically related ... … or more often the one thinking that, since you want to buy a washing machine, you decided to start up a whole new laundry. What is Machine Learning?
6. It’s all about (fancy) naming
7. Why not Machine Learning + Symbolic Reasoning ? ML to find statistical correlations and discover new patterns SR to filter away the ones not making sense under a business point of view
8. Why not Machine Learning + Symbolic Reasoning ? ML to find statistical correlations and discover new patterns SR to filter away the ones not making sense under a business point of view
9. Why not Machine Learning + Symbolic Reasoning ? ML to find statistical correlations and discover new patterns SR to filter away the ones not making sense under a business point of view Your business domain knowledge lives here!
15. Would you allow an hallucinated blue Caterpillar to decide about your business?
17. General Machine Learning algorithms problems ❖ Accountability ➢ Who is responsible for the decision taken by a ML algorithm? The creator of the ML-model? The service provider? The final user?
18. General Machine Learning algorithms problems ❖ Accountability ➢ Who is responsible for the decision taken by a ML algorithm? The creator of the ML-model? The service provider? The final user? ❖ Transparency ➢ While traditional algorithms can be examined line by line, arriving at an exact understanding of their internal dynamics and the reasons for a certain result, ML algorithms automatically extracts knowledge from the data, forming an idea of the problem based on statistical correlations known only to it. ➢ Trying to understand the reason that drove to a decision simply looking at the ML-model would be like trying to know why someone thinks in a certain way by dissecting their brain.
19. General Machine Learning algorithms problems ❖ Accountability ➢ Who is responsible for the decision taken by a ML algorithm? The creator of the ML-model? The service provider? The final user? ❖ New definition of malfunctioning ➢ Given their statistical nature, ML algorithms can hallucinate or make other spectacular mistakes even when they work very well. ➢ These mistakes cannot be fixed in the same way of a software bug. ❖ Transparency ➢ While traditional algorithms can be examined line by line, arriving at an exact understanding of their internal dynamics and the reasons for a certain result, ML algorithms automatically extracts knowledge from the data, forming an idea of the problem based on statistical correlations known only to it. ➢ Trying to understand the reason that drove to a decision simply looking at the ML-model would be like trying to know why someone thinks in a certain way by dissecting their brain.
20. Why not fixing LLM’s hallucinations using another LLM to validate the outcome of the first?
21. Why not fixing LLM’s hallucinations using another LLM to validate the outcome of the first? … this reminds me of a 20 years old joke …
22. Mixing LLM and rule engine for a safer airline chatbot LLM - (Mistral 7B) Implement a chatbot to collect information about the customer and his flight Rule engine - (Drools) Calculate customer’s refund based on precise business rules session data
23. Mixing LLM and rule engine for a safer airline chatbot LLM - (Mistral 7B) Implement a chatbot to collect information about the customer and his flight Rule engine - (Drools) Calculate customer’s refund based on precise business rules session data Extract customer data from chat Extract flight data from chat Data gathering complete data incomplete data incomplete State machine
24. DEMO TIME! https://github.com/mariofusco/quarkus-drools-llm
25. Mixing LLM and rule engine for a safer airline chatbot
27. Machine Learning is a new paradigm Rule Engine Rules Data Answers A rule based program produces Answers matching Data against Rules
28. Machine Learning is a new paradigm Rule Engine Rules Data Answers A rule based program produces Answers matching Data against Rules Machine Learning Answers Data Rules A machine learning system discovers Rules comparing Data and expected Answers
29. Why not mixing them? Rule Engine Rules Data Answers Machine Learning Answers Data Rules
30. Leveraging the strengths of both entities ML (sub-symbolic AI) + High adaptability and flexibility + Handling complexity, vast amounts of data + Automatic rule extraction - Lack of interpretability - Data dependence Rule-engine (symbolic AI) + Speed and reliability + Effective with well-structured problems + Interpretable - Scalability - Lack of adaptability - Knowledge acquisition Machine Learning Rule-engine Hybrid AI + →
31. Symbolic Reasoning Knowledge base Inference engine Question Answer Symbolic AI (i.e. Drools) Human input
32. Hybrid AI = Symbolic Reasoning + Machine Learning Knowledge base Inference engine Question Answer Symbolic AI (i.e. Drools) Knowledge base Inference engine Question Answer Data ML learning process Hybrid AI (Drools + ML) Human input Human input
33. Hybrid AI = Symbolic Reasoning + Machine Learning Knowledge base Inference engine Question Answer Symbolic AI (i.e. Drools) Knowledge base Inference engine Question Answer Data ML learning process Hybrid AI (Drools + ML) Human input Human input Hybrid AI: more versatile AI system that can efficiently handle complex scenarios, adjust to changing conditions, and improve performance and accuracy.
34. ML learning process overview • Business rules (KB) from data. • Step 1 - “Initialize-rules”: ◦ Add decision-rules to cover all instances in the input dataset. • Step 2 - “Improve-rules”: ◦ Identify rules that erroneously support some data. ◦ Add decision-rules to fix the errors. • We used a revised version of the OneR algorithm to learn decision-rules from data.
35. The OneR algorithm • OneR, short for "One Rule", is a simple, yet accurate, classification algorithm that generates one rule for each predictor/feature in a dataset, and then selects the rule with the smallest total error as its "one rule" (Berry M and Linoff, 2000)(*). • Revised version of OneR: the rule with the highest accuracy (smallest error) and highest coverage is selected. We have introduced a “score” metric as the product of the accuracy and coverage of a rule, to ease this selection. (*) F. Alam and S. Pachauri, “Comparative Study of J48, Naive Bayes and One-R Classification Technique for Credit Card Fraud Detection using WEKA,” Adv. Comput. Sci. Technol. , vol. 10, no. 6, 2017.
36. OneR example • One frequency table for each predictor against the target variable • For each condition (i.e. outlook=sunny) the most frequent (majority) class is assigned to the rule: • The rules with the highest score is selected as the one rule for the predictor
37. ML learning process • Step 1 - “Initialize-rules”: ◦ Decision-rules to cover all instances in the input dataset • Step 2 - “Improve-rules”: ◦ Find rules that erroneously support some data. ◦ Find a new rule for this subset of data. ◦ Fix existing rule and create a new rule for this subset of data. step 1 step 2
38. Drools KB from data Example • Dataset: diabetes early stage • Feature selection: ◦ polyuria ◦ polydipsia ◦ gender where age ~ middle age • Prediction: positive / negative Source: https://www.kaggle.com/code/therealsampat/early-stage-diabetes-prediction
39. Step 1: initial set of rules rule "rule 0" when $a : DiabetesDatapoint( polydipsia == 'Yes' ) then $a.setPrediction( 'positive' ); update( $a ); end rule "rule 1" when $a : DiabetesDatapoint( polydipsia == 'No' ) then $a.setPrediction( 'negative' ); update( $a ); end KB Assessment: >> Coverage = 100% >> Errors = 28/186 (15%)
40. Step 2: improve initial ruleset rule "rule 0" when $a : DiabetesDatapoint( polydipsia == 'Yes' ) then $a.setPrediction( 'positive' ); update( $a ); end rule "rule 1" when $a : DiabetesDatapoint( polydipsia == 'No' ) then $a.setPrediction( 'negative' ); update( $a ); end KB Assessment: >> Coverage = 100% >> Errors = 17/186 (7%) rule "rule 2" when $a : DiabetesDatapoint( gender == ‘Female’, polydipsia == 'No' ) then $a.setPrediction( 'positive' ); update( $a ); end ... rules “rule N”
41. Conclusions ❖ ML is only a fraction of the whole AI story ➢ The general goal of artificial intelligence is mimicking the human intellect and behaviors ➢ There are different ways to achieve this goal, letting the machine learning it from data is only one of them ❖ Machine Learning doesn’t replace Symbolic Reasoning: they are complementary technologies ➢ ML and SR are different technologies with their own pros and cons ➢ Often both the flexibility and human friendliness of ML and the strictness and transparency of SR are required in the same intelligent application and you can leverage and integrate the strengths of both
42. Our projects • Quarkus + LLM + Drools - https://github.com/mariofusco/quarkus-drools-llm • Rules creation from data - https://github.com/nprentza/droolsML • Neuro-symbolic AI - https://en.wikipedia.org/wiki/Neuro-symbolic_AI • Approaches in Using Generative AI for Business Automation: The Path to Comprehensive Decision Automation - https://medium.com/@pierrefeillet/approaches-in-using-generative-ai-for-business-automat ion-the-path-to-comprehensive-decision-3dd91c57e38f • Explainable Machine Learning via Argumentation - https://www.researchgate.net/publication/372688199_Explainable_Machine_Learning_via_ Argumentation References