Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Software Testing Techniques: Checking & Exploring

Matt Obee
October 15, 2015

Software Testing Techniques: Checking & Exploring

In this presentation, we’ll think about how the role of software tester has evolved to serve the needs of modern Agile teams. We’ll look at how all testing activities sit somewhere on the sale of freedom and formality, and how successful projects include a healthy mix of formal checking and free exploration. Finally, we’ll learn how to bring structure to our exploratory testing efforts using sessions, tours and variables.

Matt Obee

October 15, 2015
Tweet

Other Decks in Technology

Transcript

  1. “Exploratory testing is not a testing technique. It's a way

    of thinking about testing. Any testing technique can be used in an exploratory way.” 
 Cem Kaner & James Bach (2004)
  2. No freedom More freedom
 Less formal Scale of Testing Freedom

    & Formality Less freedom
 More formal CHECKING EXPLORING Test Cases Test Data Scenarios Checklist Mind Map Charter Tour Play Scripted Checking
  3. Does it do everything that it’s supposed to do? ▪

    Asks closed (yes/no) questions; ▪ Validation and verification; ▪ Needs specification; ▪ Often automated; ▪ Developers do a lot of checking. Are there any risks or opportunities? ▪ Asks open ended questions; ▪ Does not need specification; ▪ Results of the previous test inform the next; ▪ Used to evaluate subjective qualities (e.g. usability and aesthetics); ▪ Rarely automated; CHECKING EXPLORING
  4. Tested = Checked + Explored 
 “You’re not done testing

    until you’ve checked that the software meets expectations and you’ve explored whether there are additional risks. A comprehensive test strategy incorporates both approaches.” Elisabeth Hendrickson (2013)
  5. ▪ Complex multi-page forms ▪ Complex processing systems ▪ Highly-regulated

    industry and bureaucratic client ▪ Accuracy, consistency and compliance ▪ Overall project risk is high
  6. No freedom More freedom
 Less formal Less freedom
 More formal

    Test Cases Test Data Checklist Scripted Checking
  7. ▪ Not highly regulated ▪ User experience is the main

    quality metric ▪ Quality is subjective and not easy to measure ▪ Relatively low risk but lots of opportunities
  8. No freedom More freedom
 Less formal Less freedom
 More formal

    Test Data Scenarios Checklist Play Scripted Checking Personas
  9. Recap: Checking & Exploring ▪ Exploratory testing isn’t a technique;

    ▪ All testing sits on the scale of freedom and formality; ▪ Less formality means more freedom to explore; ▪ Different projects need different levels of formality; ▪ Tested = Checked + Explored
  10. “Checking is the process of making evaluations by applying algorithmic

    decision rules to specific observations of a product.” 
 James Bach & Michael Bolton (2013)
  11. “Automate 100% of the tests that should be automated.” Alan

    Page (2013) Automate all the things!!1!
  12. Computers are good at... ▪ maths ▪ simulation ▪ generating

    data ▪ validating against rules ▪ doing repetitive tasks ▪ multitasking ▪ doing things quickly Computers suck at... ▪ recognising patterns ▪ learning and reacting ▪ ignoring false positives ▪ being random ▪ curiosity ▪ creativity ▪ empathy ▪ understanding context ▪ common sense
  13. Automation doesn’t replace human testers. Automation does the boring stuff

    so that humans have more time to do intelligent exploration.
  14. Recap: Checking ▪ Checks need explicit rules and success criteria;

    ▪ Make computers do the boring stuff; ▪ Don’t fear automation. It’s doesn’t replace you, it helps you do more of the testing that you enjoy and less of the stuff that you don’t;
  15. 50% of people don’t see the invisible gorilla. The same

    inattentional blindness happens when testing software. bit.ly/1jGOCoQ
  16. The Guidebook Tour ▪ Do what the guidebook (instructions, manual)

    recommends; ▪ Visit the top tourist destinations (the most popular features); ▪ Don’t leave the main tourist trails (avoid advanced features); ▪ Be lazy (don’t explore too much)
  17. The Obsessive Compulsive Tour ▪ Repeat actions ▪ Click buttons

    more than once ▪ Go back and forth between the same pages ▪ Keep refreshing the page ▪ Type in a field, delete it, type in it again ▪ Make a selection, clear it, make the selection again ▪ Make the same booking twice
  18. The Backstreet Tour ▪ Use the features that are rarely

    used ▪ Take the routes that are rarely taken ▪ Read the content that customers don’t normally read ▪ Buy the products that fewer customers buy ▪ Select the options that customers rarely select
  19. The Bad Neighbourhood Tour Venturing into the rougher parts of

    town Light at the end of the Ghetto by Jenn Durfey
  20. The Bad Neighbourhood Tour ▪ Concentrate on areas that are

    known to be buggy (bugs cluster) ▪ Explore nearby or related features ▪ Do things that the application is known to struggle with
  21. The Nervous Traveller Tour Being careful and scrutinise every detail

    Issica demonstrates the brace position by Karl Baron
  22. The Nervous Traveller Tour ▪ Look at everything with fresh

    eyes ▪ Do things slowly and pay attention to the details ▪ Read the microcopy (labels, instructions, error messages etc.) ▪ Ask lots of questions and look for the answers ▪ Be indecisive and change your mind ▪ What do our customers worry about?
  23. ▪ Leave mandatory form fields empty ▪ Do things in

    the wrong order ▪ Do the same things more than once ▪ Start things but don’t finish them ▪ Be impatient ▪ Put JavaScript or SQL queries in form fields ▪ Try to access member features as a non-member The Antisocial Tour
  24. A software tester walks into a bar. Orders a beer.

    Orders 9999 beers. Orders 1.5 beers. Orders -1 beers. 
 Orders 0 beers. Orders a fdsfdsa. 
 Tries to leave without paying.
  25. A few pen testing variables ▪ Left and right handed

    users: try writing in either hand ▪ Writing speed: try writing quickly and slowly ▪ Writing pressure: try writing lightly and heavily ▪ Surface: try writing on matte paper, glossy paper, cardboard, plastic etc. ▪ Orientation: try writing on a table, on a wall, upside down etc. ▪ Environment: trying writing in hot conditions, cold conditions, wind, rain etc. ▪ Length of use: try writing a short note compared to a long essay ▪ State: what happens when changing from one state to another?
  26. Vary numbers ▪ Can you enter a negative number, e.g.

    “-1”? ▪ Can you enter a decimal number, e.g. “1.5”? ▪ Can you enter zero (e.g. “0”)? ▪ Can you exceed the minimum or maximum limits? ▪ What happens where there are zero, one or many things? ▪ Goldilocks (too many, too few, just right) ▪ What happens if you enter letters instead of numbers?
  27. Vary time ▪ Can you specify a date of birth

    in the future? ▪ Can you schedule an event to happen in the past? ▪ In a date range, can you set the ending earlier than the beginning? ▪ What happens on leap days? ▪ What happens on dates that don’t exist (e.g. 31st September)? ▪ What happens if you reset time on the device forward and back? ▪ What happens if you move between timezones?
  28. Vary the format ▪ What happens if you enter dates

    in different formats? 
 (e.g. 01/06/2015, 06/01/2015, 1/6/15, 1-6-15, 2015-06-01) ▪ What happens if there’s a space in the number? 
 (e.g. “01303 815 300”) ▪ What happens if a number includes special characters? 
 (e.g. “+44 (0)1303 815 300”) ▪ Can you upload different file formats? (e.g. .gif, .jpg, .doc, .docx) ▪ What happens if you enter numbers with different units of measurement?
  29. Vary the journey ▪ Can you use the browser back

    and forward buttons? ▪ What happens if you refresh the page? ▪ What happens if you arrive via a bookmark? ▪ Can you hack the URL? ▪ Can you do things in a different order? ▪ What happens if you have multiple tabs/windows running?
  30. ▪ Can you use it without a mouse (i.e. keyboard

    only)? ▪ What happens if you copy and paste instead of typing into fields? ▪ Can you drag and drop content into fields? Vary the method of input
  31. ▪ What happens if you interact with the application too

    quickly? ▪ What happens if you interact with the application too slowly? ▪ What happens if you don’t do anything for a length of time? Vary the speed of input
  32. Vary Settings & States ▪ What happens if JavaScript is

    disabled? ▪ What happens if cookies are blocked? ▪ What happens if you change the size of the browser window? ▪ What happens if you change the browser font size? ▪ Compare behaviour when logged in and logged out ▪ What happens when the state changes? (e.g. a logged in user logging out, a logged out user logging in)
  33. Recap: Exploring ▪ Checklists are one great way to structure

    exploration; ▪ Take tours to change your perspective; ▪ Variables are key. Identify the variables and see what happens when you change them.
  34. Takeaways ▪ Tested = Checked + Explored
 Check to make

    sure it’s doing what it should; explore to see if there are new risks or opportunities. ▪ Make stupid computers do the boring checks so that intelligent humans have more time to explore. ▪ Use checklists, take tours and experiment with variables to get everyone involved in testing sessions.