specialist firm focused on functional and non-functional testing for complex, distributed, non-deterministic and artificial intelligence platforms Headquartered in the UK and operates delivery centres in Georgia, Sri Lanka, Armenia, Lithuania, the UK and representative offices in the US, Canada, Italy and Australia. Exactpro provides software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing and settlement platforms, market data systems, collateral management and risk management systems, central data warehouses, regulatory reporting, etc. Incorporated in 2009 with 10 people, Exactpro is now employing over 600 specialists. Most of our clients are regulated market infrastructures. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG.
to Test Software exactpro.com Join the Exactpro Youtube channel for regular updates on the applications of AI and Machine Learning, DLT, cloud computing, and many other technologies and solutions to software testing and development. This channel is for chief technology, information and compliance professionals, product owners and software Quality Assurance and development specialists of all levels. @exactpro
and CT-AI A4Q AIST Pages Hours 1. Key Aspects of Artificial Intelligence 15 6 2. Testing Artificial Intelligence Systems 19 6.5 2.1 General Problems with Testing AI Systems 8 2.2 Machine Learning Model Training and Testing 2 2.3 Test Environments 2 2.4 Strategies to Test AI-based Systems 5 2.5 Metrics for Testing AI-based systems 2 3. Using AI to Support Testing 13 4.5 47 17 ISTQB CT-AI Pages Hours 1. Introduction to AI 20 5.5 3. Machine Learning 6. ML Neural Networks and Testing 44 17 2. Quality Characteristics 5 2 4. ML Data 7 4 10. Test Environments 3 0.5 7. Testing AI-Based Systems 8 2 8. Testing AI-Specific Quality Characteristics 8 2.5 9. Methods and Techniques for the Testing of AI-Based Systems 8 4 5. ML Functional Metrics 5 2 11. Using AI in Testing 5 3 69 25.5
Chapter 11: 11.1 AI Technologies for Testing • AI-11.1.1 K2 Categorize the AI technologies used in software testing. • HO-11.1.1 H2 Discuss, using examples, those activities in testing where AI is less likely to be used. 11.2 Using AI to Analyze Reported Defects • AI-11.2.1 K2 Explain how AI can assist in supporting the analysis of new defects. 11.3 Using AI for Test Case Generation • AI-11.3.1 K2 Explain how AI can assist in test case generation. 11.4 Using AI for the Optimization of Regression Test Suites • AI-11.4.1 K2 Explain how AI can assist in optimization of regression test suites 11.5 Using AI for Defect Prediction • AI-11.5.1 K2 Explain how AI can assist in defect prediction. • HO-11.5.1 H2 Implement a simple AI-based defect prediction system. 11.6 Using AI for Testing User Interfaces • AI-11.6.1 K2 Explain the use of AI in testing user interfaces AI-Specific Keywords Bayesian techniques, classification, clustering algorithm, defect prediction, graphical user interface (GUI) 11 Using AI for Testing Keywords Visual testing
Testing Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Information Service
Software Testing? • Goodness, by definition, is always rooted in some person’s feeling • Goodness is at least a three-way relationship between people, product and context • Goodness is a phenomenon produced by the mysterious interaction of many variables • Goodness judgments tend to be socially risky James Bach Assess Quality, Don’t Measure It The mysteries of measuring goodness are evident in machine learning systems that hilariously mis-classify things. Or in ourselves when we hilariously fall in love with the wrong people.
Software Testing? • 100% Automated • Easy to Maintain • Very Fast • Consistent and Repeatable • Applicable to All Systems • Vendor Independent • Transparent • Low Cost
Software Testing? Just Do Not Do Any Testing… • 100% Automated • Easy to Maintain • Very Fast • Consistent and Repeatable • Applicable to All Systems • Vendor Independent • Transparent • Low Cost
of our work is to provide information - detect and describe defects in the software developed for/by and operated by our clients. The value of our work can be measured across the following three main dimensions: • Quality - testing better (*) • Speed - testing faster • Cost - testing cheaper Improvements across any of these dimensions bring value to our clients (*) better at detecting and describing defects
actions against the System Under Test (SUT) Select actions to perform Interpret the outcome of the actions Mechanical (Easy) The following is a list of non-exclusive possibilities: • Experiment results match our model of the SUT • Something is wrong with our actions • Something is wrong with the SUT • Something is wrong with our model Provide the information about our findings to the stakeholders for them to make decisions as to what should be done Action types: • Set-up actions • Experiment actions • Observation actions What actions are available? Which of them to choose? In what order?
actions against the System Under Test (SUT) Select actions to perform Interpret the outcome of the actions What actions are available? Which of them to choose? In what order? Evaluate obtained information and the list of available actions – repeat the steps Cognitive(Medium) Mechanical (Easy) Cognitive (Medium) Cognitive (Hard) The following is a list of non-exclusive possibilities: • Experiment results match our model of the SUT • Something is wrong with our actions • Something is wrong with the SUT • Something is wrong with our model Provide the information about our findings to the stakeholders for them to make decisions as to what should be done Action types: • Set-up actions • Experiment actions • Observation actions
Model-based Testing A model is a simplified, formal representation of a relationship, process or system. The simplification makes some aspects of the thing modeled clearer, more visible, and easier to work with. All tests are based on models, but many of those models are implicit. When the behavior of the program “feels wrong”, it is clashing with your internal model of the program and how it should behave. Black Box Software Testing Cem Kaner & James Bach
of Model-based Testing Can be both mental representation or a schema or a code-based artifact Requirement/System Specifications MBT Models Abstract Test Case Test Results SUT System Adapters Execution Environment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5 • more formal models (expressed in some machine-readable, well-defined, notation) • more formal test generation (we are interested in test generation algorithms) • more formal checks for mechanical execution (the generated tests must be sufficiently precise to efficiently use humans’ involvement)
Box ...all models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind…
Model Herbert Stachowiak. Allgemeine Modelltheorie. Springer-Verlag, Wien and New York, 1973. (Cited by Kühne, Thomas. "What is a Model?." In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2005) mapping feature reduction feature pragmatic feature
exactpro
badams
tamaranovitovic
caitiem20
techseoconnect
kimpetersen
mayatellez
addyosmani
akashhashmi
chriscoyier
lara
62gerente
csswizardry
