Behat for characterization tests

Once an API ships it doesn’t matter how it should behave – how it actually behaves is the important part. Users depend on the existing behaviour and we need a way to ensure that it doesn’t change Behat is a tool that was built to help design software, but it’s actually a great tool for capturing existing behaviour too. We’ve used these tools to gain confidence to refactor 5+ year old apps by capturing the existing behaviour before making changes. I want to share the secrets we learned with you.

Transcript

1. @mheap #phpkonf for characterization tests
2. @mheap #phpkonf Imagine this…
3. @mheap #phpkonf $$$
4. @mheap #phpkonf $$$ $$$ $$$
5. @mheap #phpkonf $$$ Billing Rating Auth Report Masking Encoding $$$ $$$
6. @mheap #phpkonf $$$ Billing Rating Auth Report Masking Encoding $$$ $$$
7. @mheap #phpkonf $$$ $$$ $$$ Billing Rating Auth Report Masking Encoding public function provides() { return [Client ::class]; }
8. @mheap #phpkonf
9. @mheap #phpkonf $ ls -l test total 0
10. @mheap https://twitter.com/brianwisti/status/503987766032494592 YOU ARE IN A LEGACY CODEBASE > RUN TESTS YOU HAVE NO TESTS > READ SPEC YOU HAVE NO SPEC > WRITE FIX YOU ARE EATEN BY AN ELDER CODE HACK.
11. @mheap #phpkonf Imagine
12. @mheap #phpkonf Hello, I’m Michael
13. @mheap #phpkonf @mheap
14. @mheap #phpkonf
16. @mheap #phpkonf Today
17. @mheap #phpkonf What is legacy code?
18. @mheap Michael Feathers - Working Effectively with Legacy Code “code without tests” Legacy Code is
19. @mheap Legacy Code is “valuable code that we feel afraid to change” JB Rainsberger - Surviving Legacy Code with Golden Master and Sampling
20. @mheap #phpkonf What if I make a change and break something?
21. @mheap #phpkonf What if I make a change and change something?
22. @mheap #phpkonf Intended or Unintended
23. @mheap #phpkonf Intended or Unintended
24. @mheap #phpkonf Intended or Unintended
25. @mheap #phpkonf But there aren’t any tests!
26. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
27. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
28. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
29. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
30. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
31. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(999)); } }
32. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
33. @mheap #phpkonf There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 200 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:8 FAILURES! Tests: 1, Assertions: 1, Failures: 1.
34. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testSalesLTE1000Pay20PercentCommission() { $this ->assertEquals(200, SalesUtil ::calculate(1000)); } }
35. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
36. @mheap #phpkonf There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 500.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:13 FAILURES! Tests: 2, Assertions: 2, Failures: 1.
37. @mheap #phpkonf public function test2000Gets500Commission() { $this ->assertEquals(500, SalesUtil ::calculate(2000)); } public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1600)); }
38. @mheap #phpkonf There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 380.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:19 FAILURES! Tests: 3, Assertions: 3, Failures: 1.
39. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
40. @mheap #phpkonf Input Expected 0 0 10 2 500 100 1000 200 1001 200.3 1500 350 1982 494.6 1999.9 499.97 2000 500 2001 500.9 5000 3200 7890 5801 10000 7700
41. @mheap #phpkonf public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(-123)); }
42. @mheap #phpkonf public function testNegativeSalesCommission() { $this ->assertEquals(-24.6, SalesUtil ::calculate(-123)); }
43. @mheap #phpkonf Do not change the code
44. @mheap #phpkonf Do not change the code
45. @mheap #phpkonf Do NOT change the code
46. @mheap https://en.wikipedia.org/wiki/Equivalence_partitioning What’s the minimum number of tests?
47. @mheap #phpkonf We’re half way! (and we’ve barely mentioned Behat)
48. @mheap #phpkonf Choosing your pinch points
49. @mheap #phpkonf Choose HTTP
50. @mheap #phpkonf Choose Behat
51. @mheap #phpkonf Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
52. @mheap #phpkonf Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
53. @mheap #phpkonf Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
54. @mheap #phpkonf Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
55. @mheap #phpkonf Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
56. @mheap #phpkonf Be pragmatic
57. @mheap #phpkonf composer require behat/behat --dev vendor/bin/behat --init
58. @mheap #phpkonf Testing a HTTP API
59. @mheap #phpkonf $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
60. @mheap #phpkonf $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
61. @mheap #phpkonf $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
62. @mheap #phpkonf error('username is required'); error('key is required'); error('could not find user', 404); error('invalid apikey'); output([ 'success' => 'true', 'user' => ['name' => $username] ]); 1. Missing username 2. Missing key 3. Invalid username 4. Invalid key 5. Successful authentication
63. @mheap #phpkonf Scenario: No Username When I make a "GET" request to "/" Then the response status code should be "400" And the "error" property equals "username is required"
64. @mheap #phpkonf Scenario: No API Key When I make a "GET" request to "/?username=foo" Then the response status code should be "400" And the "error" property equals "key is required”
65. @mheap #phpkonf Scenario: Invalid Username When I make a "GET" request to "/?username=bananas&key=foo" Then the response status code should be "404" And the "error" property equals "could not find user” Scenario: Invalid API Key When I make a "GET" request to "/?username=michael&key=foo" Then the response status code should be "400" And the "error" property equals "invalid apikey"
66. @mheap #phpkonf Scenario: Successful login When I make a "GET" request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
67. @mheap #phpkonf Scenario Outline: Successful login When I make a "GET" request to "/?username=<name>&key=<key>" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "<name>" Examples: | name | key | | michael | kangar00s | | oscar | phparch17 |
68. @mheap #phpkonf More complex tests
69. @mheap #phpkonf Given that header property "X-CustomAuthKey" is "Secret123" And that header property "Accept" is "application/json+vnd.v2" And that the request body is valid JSON ''' { "alpha":"beta", "count":3, "collection":["a","b","c"] } ''' When I make a "POST" request to "/account/balance" Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the "balance.usd" property equals"1832.54" And the "balance.gbp" property equals "13.99" And the "balance.eur" property equals "19422.18"
70. @mheap #phpkonf When I make a "POST" request to "/account/balance" Then the response status code should be “200" And the "X-Remaining" header property equals "18" And the response body contains the JSON data ''' { "in_credit": true, "balance": { "usd": 1832.54, "gbp": 13.99, "eur": 19422.18 } } '''
71. @mheap #phpkonf When I make a "POST" request to "/account/balance" Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the value of the "balance.usd" property matches the pattern “/^\d{1,3}\. \d{2}$/" And the value of the “last_updated” property matches the pattern "/^[0-9]{4} [\-][0-9]{2}[\-][0-9]{2} [0-9]{2}[:][0-9]{2}[:][0-9]{2}$/"
72. @mheap #phpkonf composer require datasift/testrest-extension
73. @mheap #phpkonf # behat.yml default: extensions: DataSift\BehatExtension: base_url: "http: //localhost:8080/" suites: default: contexts: - 'DataSift\BehatExtension\Context\RestContext'
74. @mheap #phpkonf HTTP Mocks
75. @mheap #phpkonf function get_balance($username) { return $client ->get( “http: //localhost:88/billing/check_balance/".$username ) ->getBody(); } $balance = get_balance($username); if ($username != 'michael') { error('wrong username'); } if (!$balance['has_credit']) { error('no credit remaining'); } success(['data' => 'valid account'])
76. @mheap #phpkonf Mountebank
77. @mheap #phpkonf npm install -g mountebank
78. @mheap #phpkonf Given Mountebank is running And a mock exists at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": false, "credit_limit": 0 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "403" And the response is JSON And the response body JSON equals ''' { "error": "no credit remaining" } '''
79. @mheap #phpkonf Given Mountebank is running And a mock exists at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": true, "credit_limit": 10000 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "200" And the response is JSON And the response body JSON equals ''' { "data": "valid account" } '''
80. @mheap #phpkonf Automating a browser
81. @mheap #phpkonf $ composer require --dev behat/mink-extension $ composer require --dev behat/mink-selenium2-driver $ java -Dwebdriver.chrome.driver=chromedriver -jar selenium- server-standalone-3.7.1.jar
82. @mheap #phpkonf # behat.yml default: extensions: Behat\MinkExtension: base_url: 'https: //wikipedia.org' sessions: default: selenium2: browser: "chrome" suites: my_suite: contexts: - \Behat\MinkExtension\Context\MinkContext
83. @mheap #phpkonf Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
84. @mheap #phpkonf Custom Steps
85. @mheap #phpkonf Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
86. @mheap #phpkonf Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development"
87. @mheap #phpkonf Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 undefined) 3 steps (1 passed, 1 undefined, 1 skipped) 0m3.63s (10.62Mb) >> my_suite suite has undefined steps. Please choose the context to generate snippets: [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext
88. @mheap #phpkonf [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1 --- FeatureContext has missing steps. Define them with these snippets: /** * @When I search for :arg1 */ public function iSearchFor($arg1) { throw new PendingException(); }
89. @mheap #phpkonf ./vendor/bin/behat --append-snippets
90. @mheap #phpkonf [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1 u features/bootstrap/FeatureContext.php - `I search for "Behavior Driven Development"` definition added
91. @mheap #phpkonf Feature: Test Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" TODO: write pending definition Then I should see "agile software development" 1 scenario (1 pending) 3 steps (1 passed, 1 pending, 1 skipped) 0m2.23s (10.61Mb)
92. @mheap #phpkonf /** @BeforeScenario */ public function gatherContexts( Behat\Behat\Hook\Scope\BeforeScenarioScope $scope ) { $environment = $scope ->getEnvironment(); $this ->minkContext = $environment ->getContext( 'Behat\MinkExtension\Context\MinkContext' ); }
93. @mheap #phpkonf /** * Fills in form field with specified id|name|label|value * Example: When I fill in "username" with: "bwayne" * Example: And I fill in "bwayne" for "username" * * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with "(?P<value>( ?:[^"]|\ ")*)"$/ * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with:$/ * @When /^( ?:|I )fill in "(?P<value>( ?:[^"]|\ ")*)" for "(?P<field>( ?:[^"]|\ ")*)"$/ */ public function fillField($field, $value) { $field = $this ->fixStepArgument($field); $value = $this ->fixStepArgument($value); $this ->getSession() ->getPage() ->fillField($field, $value); }
94. @mheap #phpkonf /** * Presses button with specified id|name|title|alt|value * Example: When I press "Log In" * Example: And I press "Log In" * @When /^( ?:|I )press "(?P<button>( ?:[^"]|\ ")*)"$/ */ public function pressButton($button) { $button = $this ->fixStepArgument($button); $this ->getSession() ->getPage() ->pressButton($button); }
95. @mheap #phpkonf /** * @When I search for :term */ public function iSearchFor($term) { $this ->minkContext ->fillField('search', $term); $this ->minkContext ->pressButton('searchButton'); }
96. @mheap #phpkonf $ ./vendor/bin/behat Feature: Test Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 passed) 3 steps (3 passed)
97. @mheap #phpkonf Scenario: Successful login When I make a "GET" request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
98. @mheap #phpkonf Scenario: Successful login When I log in as "Michael" with the password "kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
99. @mheap #phpkonf Snapshot testing
100. @mheap #phpkonf public function toJson(){ return json_encode(["id" => $this ->id]); } —— public function test_json_formatting_is_correct() { $order = new Order(1); $this ->assertMatchesSnapshot($order ->toJson()); }
101. @mheap #phpkonf $ ./vendor/bin/phpunit There was 1 incomplete test: 1) ExampleTest ::test_json_formatting_is_correct Snapshot created for ExampleTest __test_json_formatting_is_correct
102. @mheap #phpkonf $ ./vendor/bin/phpunit OK (1 test, 1 assertion)
103. @mheap #phpkonf public function toJson(){ return json_encode(["id" => 'Q'.$this ->id]); }
104. @mheap #phpkonf $ ./vendor/bin/phpunit 1) ExampleTest ::test_json_formatting_is_correct Failed asserting that two strings are equal. --- Expected +++ Actual @@ @@ -'{"id": 1}' +'{"id": "Q1"}' FAILURES! Tests: 1, Assertions: 1, Failures: 1.
105. @mheap #phpkonf https: //github.com/spatie/phpunit-snapshot-assertions
106. @mheap #phpkonf Conclusion
107. @mheap #phpkonf We don’t have time for this
108. @mheap #phpkonf Where should we start?
109. @mheap #phpkonf Start refactoring
110. @mheap #phpkonf Characterisation tests aren’t perfect
111. @mheap #phpkonf The 4 stages of characterisation tests
112. @mheap #phpkonf Examine your feature
113. @mheap #phpkonf Characterize your feature
114. @mheap #phpkonf Refactor your feature
115. @mheap #phpkonf Delete your tests
116. @mheap #phpkonf for characterization tests
117. @mheap #phpkonf I’m Michael
118. @mheap #phpkonf @mheap
119. @mheap #phpkonf https://joind.in/talk/dc4a7
120. @mheap #phpkonf Michael @mheap https://joind.in/talk/dc4a7 Thankyou!