Sustainable BDD

The BDD hype cycle is over. Recently, there’s been a lot of backlash against popular BDD libraries like Cucumber. Some developers blame their test frameworks for brittle test suites and long build times. Others go so far as to claim that acceptance testing is simply not sustainable, period. In this talk, we’ll do some root cause analysis of this phenomenon with shocking results - it’s not the test framework, it’s not the methodology, it’s you. You’ve abused your test framework, you’ve cargo-culted the methodology, and now you’re feeling the pain. We’ll show you a way out of the mess you’ve made. We’ll discuss the main problems BDD was intended to solve. We’ll show you how to groom your test suite into journey, functional, integration, and unit tests in order to address build times. We’ll teach how to mitigate against brittleness and flickers, and how to let your tests reveal the intent of the application and actually become the executable documentation we’ve been waiting for.

Transcript

1. SUSTAINABLE BDD Matt Parker Robbie Clutton Software Engineers, Pivotal Labs
2. HYPE CYCLE
3. WTF BDD 1. Brittle Tests 2. Dev-written Acceptance Criteria 3. Unreadable Tests 4. Flickering Tests 5. Slow Tests
4. CARGO CULTING
5. HISTORY In the beginning... there was the unit test. And lo it was good.
6. HISTORY sUnit -> xUnit
7. HISTORY And then there was Dan North.
8. SPECIFICATION BY EXAMPLE
9. FEATURE: SIGN UP
10. FEATURE: SIGN UP Give me an example.
11. FEATURE: SIGN UP Give me an example. "User opens app for the first time, provides their email, desired username, and desired password, and is immediately logged in and allowed to start using the application."
12. FEATURE: SIGN UP Give me an example. • Valid username, email, password
13. FEATURE: SIGN UP Give me an example. • Valid username, email, password • Blank username, email, or password
14. FEATURE: SIGN UP Give me an example. • Valid username, email, password • Blank username, email, or password • Non-matching Password
15. FEATURE: SIGN UP Give me an example. • Valid username, email, password • Blank username, email, or password • Non-matching Password • Password too weak
16. FEATURE: SIGN UP Give me an example. • Valid username, email, password • Blank username, email, or password • Non-matching Password • Password too weak • Email address invalid
17. FEATURE: SIGN UP Give me an example. • Valid username, email, password • Blank username, email, or password • Non-matching Password • Password too weak • Email address invalid • Username unavailable
18. FEATURE: ACCOUNT RECOVERY Background: User forgot password • User remembers email • User remembers username • User forgot everything
19. FEATURE: SIGN UP Give me an example. • Valid username, email, password • Blank username, email, or password • Non-matching Password • Password too weak • Email address invalid • Username unavailable • User provides someone else’s email
20. FEATURE: EMAIL VERIFICATION Give me an example. • Verification link followed • Fraud link followed • No links ever followed
21. FEATURE: SIGN IN Give me an example. • Existing username / password • Unknown username • Wrong password
22. HIDDEN COMPLEXITY 1 sentence => 4 features => 17 scenarios
23. HISTORY JBehave
24. HISTORY Business Value Feature Scenario
25. HISTORY RBehave
26. HISTORY Story Runner / RSpec
27. HISTORY Cucumber
28. WHY GHERKIN? 1. State Diagrams for the Masses
29. APP AS STATE MACHINE Every feature of your application is a state transition. GIVEN (start state) WHEN (event) THEN (state transition)
30. WHY GHERKIN? 1. State Diagrams for the Masses 2. Copy/Paste from Tracker to editor
31. WHY GHERKIN? 1. State Diagrams for the Masses 2. Copy/Paste from Tracker to editor 3. Living / Executable Documentation
32. HISTORY Spinach, Turnip, Steak, Bacon, Filet, Lemon, ......................
33. HISTORY 10 build shiny new tool 20 make same mistakes 30 goto 10
34. BRITTLE A single change in the application causes many tests to break
35. BRITTLE A single change in the application causes many tests to break, and you are forced to visit each and every test and fix each one individually.
36. Brittle tests typically don’t reveal intent.
37. Feature: Tweet Scenario: Valid Tweet #.... Scenario: Overlong Tweet #.... Scenario: Duplicate tweet #....
38. Scenario: Valid Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/bob" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." When I click "Log out" And I visit "/" And I fill in "username" with "alice" And I fill in "password" with "password" And I click "Log in" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
39. Scenario: Valid Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/bob" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." When I click "Log out" And I visit "/" And I fill in "username" with "alice" And I fill in "password" with "password" And I click "Log in" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Login
40. Scenario: Valid Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/bob" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." When I click "Log out" And I visit "/" And I fill in "username" with "alice" And I fill in "password" with "password" And I click "Log in" Then I should see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Tweet
41. Scenario: Overlong Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" And I click "Submit" Then I should see "Tweet too long"
42. Scenario: Overlong Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" And I click "Submit" Then I should see "Tweet too long" Tweet
43. Scenario: Overlong Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" And I click "Submit" Then I should see "Tweet too long" Login
44. Scenario: Duplicate Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see “Your tweet is a duplicate” When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters."
45. Scenario: Duplicate Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see “Your tweet is a duplicate” When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Login
46. Scenario: Duplicate Tweet Given there is a user "bob" with password "password" And there is a user "alice" with password "password" that follows "bob" And I visit "/" And I fill in "username" with "bob" And I fill in "password" with "password" And I click "Log in" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click "Submit" Then I should see "Your tweet was submitted" When I visit "/" When I fill in "Tweet" with "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." And I click “Submit” Then I should see “Your tweet is a duplicate” When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise OH NOEZ IM TOO LONG NOOOOOOOOOOOOOOOOOOOOOO" When I visit "/bob" Then I should not see "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." Tweet
47. feature “Tweet” do scenario "valid tweet" do create :user, username: "bob", password: "password" create :user, username: "alice", password: "password" visit "/" fill_in "Username", with: "bob" fill_in "Password", with: "password" click_button "Log In" fill_in "Tweet", with: "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." click_button "Submit" page.should have_content "Your tweet was submitted" visit "/bob" page.should have_content "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." click_link "Log out" visit "/" fill_in "Username", with: "alice" fill_in "Password", with: "password" click_button "Log in" page.should have_content "this is a test tweet noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise noise haha I'm less than 140 characters." end end
48. BIG RATS LEAVE BIG POOS
49. BIG RATS LEAVE BIG POOS click_button "Tweet"
50. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet"
51. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on "Tweet"
52. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on "Tweet" click_on "#tweet"
53. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on "Tweet" click_on "#tweet" find("#tweet").click
54. BIG RATS LEAVE BIG POOS click_button "Tweet" click_link_or_button "Tweet" click_on "Tweet" click_on "#tweet" find("#tweet").click page.execute_script("$('#tweet').click()")
55. BIG RATS LEAVE BIG POOS “Now, whenever you tweet, pop up a modal that forces the user to enter a CAPTCHA.”
56. BIG RATS LEAVE BIG POOS FFFFF...FFF....F....FFFFFFFFFFFFF.........
57. BIG RATS LEAVE BIG POOS module WebHelpers def tweet(message) fill_in “Message”, with: message click_on “Tweet” end end
58. BIG RATS LEAVE BIG POOS #rspec RSpec.configure do |c| c.include WebHelpers, type: :request end #cucumber World WebHelpers
59. Feature: Tweet Scenario: Valid Tweet Given Bob has authenticated When Bob submits a valid tweet Then Alice can see Bob’s tweet And Bob should see that tweet in his timeline Scenario: Overlong Tweet #.... Scenario: Duplicate tweet #....
60. BIG RATS LEAVE BIG POOS Given /^Bob has authenticated$/ do visit “/” fill_in “Username”, with: “Bob” fill_in “Password”, with: “password” click_on “Sign In” end
61. BIG RATS LEAVE BIG POOS Given /^Bob has authenticated$/ do login bob end
62. BIG RATS LEAVE BIG POOS module WebHelpers #... def bob @bob ||= create :user, name: “Bob” end #... end
63. BIG RATS LEAVE BIG POOS When /^Bob submits a valid Tweet$/ do click_on “Tweet” within(“#submit_tweet_modal”) do fill_in “Captcha”, with: captcha click_on “Tweet” end end
64. BIG RATS LEAVE BIG POOS When /^Bob submits a valid Tweet$/ do tweet “Hello, world” end
65. BIG RATS LEAVE BIG POOS Then /^Alice can see Bob’s Tweet$/ do click_on “Sign Out” fill_in “Username”, with: “Alice” fill_in “Password”, with: “password” click_on “Sign In” visit “/alice” page.should have_content “Hello, world” end
66. BIG RATS LEAVE BIG POOS Then /^Alice can see Bob’s Tweet$/ do login alice alice.should see_tweet “Hello, world” end
67. BIG RATS LEAVE BIG POOS feature “Tweet” do scenario “Valid Tweet” do login bob tweet “hello, world” login alice alice.should see “hello, world” end end
68. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA
69. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA 1. CARGO CULT
70. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA 1. CARGO CULT 2. It’s hard
71. PRODUCT OWNERS DON’T COLLABORATE ON ACCEPTANCE CRITERIA 1. CARGO CULT 2. It’s hard 3. You’re annoying them
72. Given I click on “Sign In” And I fill in “Username” with “Bob” And I fill in “Password” with “password” And I click on “Submit” #....
73. Given Bob logged in
74. NO ONE READS THE CUKES
75. NO ONE READS THE CUKES 1. They’re unreadable
76. NO ONE READS THE CUKES 1. They’re unreadable 2. They’re not exposed anywhere
77. $ cucumber -f html
78. $ relish push myproj
79. $ wally push \ $PROJ features/
80. NO ONE READS THE CUKES 1. They’re unreadable 2. They’re not exposed anywhere 3. You haven’t needed documentation
81. $ time rake
82. $ time rake real 18m0.926s
83. $ time rake real 18m0.926s imagined ETERNITY
84. Slow tests find a way of getting run less often.
85. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on “Some Link” sleep 10 page.should have_content “FOO”
86. $ rake ..........F......... $ rake ................... $ rake ....F...........F.... $ rake ...................
87. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on “Some Link” wait_for { page.has_selector? (“.foo_container”) } page.should have_content “Foo”
88. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on “Some Link” wait_for { page.should have_selector(“.foo_container”) } page.should have_content “Foo” Whoops
89. $ time rake real 18m0.926s imagined ETERNITY visit root_path click_on “Some Link” page.should have_content “Foo”
90. Q IS FOR QUARANTINE @quarantine Scenario: Something flakey #....
91. Q IS FOR QUARANTINE #regular ci build #!/bin/bash cucumber --tags ~@quarantine
92. Q IS FOR QUARANTINE #quarantine ci build #!/bin/bash cucumber --tags @quarantine
93. BUILD NANNY 1. Timebox 2. Rethink 3. Delete
94. Photo by Kris Hicks
95. $ time rake “You should not be afraid to delete tests that are no longer providing value, no matter whether you originally planned to keep them or not. We tend to treat tests as these holy creatures that live blameless, irreproachable lives once they have sprung into existence. Not so. The maintenance required to keep a test running weighs against its value in further development. Sometimes these lines cross, and the test simply becomes a burden on the project. Having the skill and experience to recognize a burdensome test is something we should be bringing to our clients, as well as the fortitude to rewrite it, rethink it, or delete it." - ADAM MILLIGAN
96. $ time rake
97. $ time rake real 9m0.926s
98. $ time rake real 9m0.926s imagined ETERNITY
99. Why are you testing the login functionality 50 times?
100. 1. Journey 2. Functional
101. $ time rake real 9m0.926s imagined ETERNITY @authenticated Feature: Publishing Widgets Scenario: ....
102. $ time rake real 9m0.926s imagined ETERNITY Before(“@authenticated”) do stub_authentication end
103. $ time rake
104. $ time rake real 4m0.926s
105. $ time rake real 4m0.926s imagined ETERNITY
106. $ time rake Feature: Sign Up Scenario: Valid username, email, password Scenario: Blank username, email, or password Scenario: Non-matching Password Scenario: Password too weak Scenario: Email address invalid Scenario: Username unavailable Scenario: User provides someone else’s email
107. $ time rake Feature: Sign Up Scenario: Valid username, email, password Scenario: Blank username, email, password Scenario: Non-matching Password Scenario: Password too weak Scenario: Email address invalid Scenario: Username unavailable Scenario: User provides someone else’s email
108. $ time rake Feature: Sign Up Scenario: Valid username, email, password Scenario: Blank username, email, password Scenario: Non-matching Password Scenario: Password too weak Scenario: Email address invalid Scenario: Username unavailable Scenario: User provides someone else’s email
109. $ time rake Feature: Sign Up Scenario: Valid username, email, password Scenario: Blank username, email, password @ci Scenario: Non-matching Password ....
110. $ time rake #.cucumber.yml default: --tags ~@ci
111. $ time rake Feature: Sign Up Scenario: Valid username, email, password Scenario: Blank username, email, password @no-ui Scenario: Non-matching Password ....
112. $ time rake Before “@no-ui” do extend DomainHelpers end
113. $ time rake module DomainHelpers attr_reader :current_user def authenticate @current_user = create :user end def tweet(message) current_user.tweet message end #... end
114. $ time rake
115. $ time rake real 30.926s
116. 1. FIGHT PAIN 2. REVEAL INTENT 3. BDD LIKE YOU MEAN IT
117. Q/A