Generative Testing in Elixir

Automated test suites are invaluable. They provide protection against regressions and can serve as a design tool when building new apis. But, despite this protection bugs still slip through. We could try to write more tests but attempting to cover every edge case is an untenable problem. Luckily, we can use property based testing to generate edge cases for us.

Originally developed in Haskell, property tests have spread to many other languages. In this talk we’ll discuss the basics of property testing, demonstrate how we can determine properties for our system, and look at real world examples of property tests using elixir.

Transcript

1. Don’t write tests; Generate them Chris Keathley / @ChrisKeathley / c@keathley.io
2. What are we gunna talk about?
3. Testing today What are property tests Basic Example Real world(ish) example
4. Why Elixir?
6. Elixir is a functional, dynamic language that targets the Erlang VM (BEAM)
7. Testing
8. TDD
9. 1. Write a failing test 2. Write enough code to make that test pass 3. Refactor Test Driven Development
10. Validation Protection From Regression Design Test Driven Development
11. Career Happiness TDD quiet contemplation Trough of disillusionment Property Tests!!!
12. Tests provide guard rails
13. Tests directly couple your implementation Test Api v1
14. Tests directly couple your implementation Test Api v1.3
15. Tests directly couple your implementation Test Api v1.3
16. Write as few tests as possible
17. Warning: Contrived Strawman Argument incoming!
18. Lets TDD Addition! x + y = ?
19. test "adding 2 numbers" do end
20. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 end
21. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 end def add(x, y) do end
23. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 end def add(x, y) do 2 end
24. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 end def add(_x, _y) do 2 end
26. But wait…
28. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(_x, _y) do 2 end
29. Pattern Matching
30. Pattern Matching
31. Pattern Matching x = 3
32. Pattern Matching x = 3 y = x
33. Pattern Matching x = 3 y = x 3 = y
34. Pattern Matching is an assertion
35. Pattern Matching x = 3 3 = x
36. Pattern Matching 3 = 3
37. Pattern Matching %{name: "Chris", hobbies: ["Coffee", "Pinball", "Lego"]}
38. Pattern Matching %{name: user_name} = %{name: "Chris", hobbies: ["Coffee", "Pinball", "Lego"]}
39. Pattern Matching user_name = "Chris"
40. Pattern Matching def user_name(user_map) do %{name: name} = user_map name end
41. Pattern Matching def user_name(%{name: name}) do name end
42. Pattern Matching def user_name(%{name: name}) do name end def user_name(_), do: "Default User"
43. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(_x, _y) do 2 end
44. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(_x, _y), do: 2
45. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(3, _), do: 7 def add(_x, _y), do: 2
47. But wait…
49. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 end def add(3, _), do: 7 def add(_x, _y), do: 2
50. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(3, _), do: 7 def add(_x, _y), do: 2
52. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(3, _), do: 7 def add(_x, _y), do: 2
53. Guard clauses
54. Guard clauses def user_name(%{name: name}) do name end
55. Guard clauses def user_name(%{name: name}) when is_binary(name) do name end
56. Guard clauses def user_name(%{name: name}) when is_binary(name) do name end def user_name(%{name: name, age: age}) when age < 20
57. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(3, _), do: 7 def add(_x, _y), do: 2
58. test "adding 2 numbers" do assert add(1, 1) == 2 assert add(0, 2) == 2 assert add(3, 4) == 7 assert add(-1, 4) == 3 end def add(x, _) when x < 0, do: 3 def add(3, _), do: 7 def add(_x, _y), do: 2
63. What have we gained?
64. Validation Protection From Regression Design TDD
65. Validation Protection From Regression Design TDD
66. Validation Protection From Regression Design TDD
67. Validation Protection From Regression Design ? TDD
68. There are bugs in your code
69. Writing tests for one feature
70. Writing tests for one feature o(n)
71. Writing tests for two features o(n) o(n^2)
72. Writing tests for three features o(n) o(n^2) o(n^3)
73. Race conditions?
74. Property tests
76. Expected == Actual
77. Expected == Actual Overly Specific
79. Property Tests
80. Property Tests Int
81. Property Tests Int p(x)
82. Property Tests Int p(x) ?
83. Property Tests Int p(x) ? Invariant
84. Invariant: “a function, quantity, or property that remains unchanged when a specified transformation is applied.”
85. Basic Property Tests
86. What is true about addition?
87. x + 0 == x
88. test "addition with zero returns the same number" do end def add(_x, _y) do end
89. test "addition with zero returns the same number" do ptest do end end def add(_x, _y) do end
90. test "addition with zero returns the same number" do ptest x: int() do end end def add(_x, _y) do end
91. test "addition with zero returns the same number" do ptest x: int() do assert add(x, 0) == x end end def add(_x, _y) do end
93. test "addition with zero returns the same number" do ptest x: int() do assert add(x, 0) == x end end def add(_x, _y) do end
94. test "addition with zero returns the same number" do ptest x: int() do assert add(x, 0) == x end end def add(x, _y) do x end
95. x + y == y + x
96. test "addition is commutative" do end def add(x, _y) do x end
97. test "addition is commutative" do ptest x: int(), y: int() do end end def add(x, _y) do x end
98. test "addition is commutative" do ptest x: int(), y: int() do assert add(x, y) == add(y, x) end end def add(x, _y) do x end
100. test "addition is commutative" do ptest x: int(), y: int() do assert add(x, y) == add(y, x) end end def add(x, _y) do x end
101. test "addition is commutative" do ptest x: int(), y: int() do assert add(x, y) == add(y, x) end end def add(x, y) do x * y end
103. test "addition is commutative" do ptest x: int(), y: int() do assert add(x, y) == add(y, x) end end def add(x, y) do x * y end
104. test "addition is commutative" do ptest x: int(), y: int() do assert add(x, y) == add(y, x) end end def add(x, y) do x * y end def add(x, 0), do: x
106. (1 + x) + y == x + (1 + y)
107. def add(x, y) do x * y end def add(x, 0), do: x test "addition is asociative" do end
108. def add(x, y) do x * y end def add(x, 0), do: x test "addition is asociative" do ptest x: int(), y: int(), z: int() do end end
109. def add(x, y) do x * y end def add(x, 0), do: x test "addition is asociative" do ptest x: int(), y: int(), z: int() do assert add(x, add(y, z)) == add(add(x, y), z) end end
110. def add(x, y) do x * y end def add(x, 0), do: x test "addition is asociative" do ptest x: int(), y: int(), z: int() do assert add(x, add(y, z)) == add(add(x, y), z) end end
112. def add(x, 0), do: x def add(x, y) do x * y end
113. def add(x, y) do x + y end
114. A Real-ish Example
116. Modeling the application ? p(x) ?
117. Modeling the application ? p(x) ? What is the input domain?
118. Vote Vote Vote User
119. Vote Vote Vote User
120. Vote Vote Vote User
121. Vote Vote Vote User
122. Modeling Users as FSMs logged_out logged_in login logout vote
123. Generate Commands
124. Generate Commands
125. Generate Commands
126. Generate Commands
127. Generate Commands
128. Generate Commands
129. Generated Commands [{:vote, "chris", 1}, {:vote, "chris", 2}, {:vote, "jane", 1}, {:vote, "jane", 1}, {:vote, "jane", 3} {:vote, "chris", 2}]
130. Generated Commands [{:vote, "chris", 1}, {:vote, "chris", 2}, {:vote, "jane", 1}, {:vote, "jane", 1}, {:vote, "jane", 3} {:vote, "chris", 2}] [{:vote, "chris", 1}, {:vote, "jane", 1}]
131. Property: Users votes should increase
132. Property: Users votes should increase test “users votes increase after voting" do end
133. Property: Users votes should increase test “users votes increase after voting" do ptest [commands: gen_commands("chris")] do end end
134. Property: Users votes should increase test “users votes increase after voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() end end
135. Property: Users votes should increase test “users votes increase after voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) end end
136. Property: Users votes should increase test “users votes increase after voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) assert result end end
137. Property: Users votes should increase test “users votes increase after voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) assert result end end def run_commands(commands, module) do Enum.reduce( commands, {0, true}, & run_command(module, &1, &2) ) end
138. def gen_commands(name) do end Command Generators
139. def gen_commands(name) do list(of: gen_vote(name), max: 20) end Command Generators
140. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name) do end Command Generators
141. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name) do tuple(like: { )}) end Command Generators
142. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name) do tuple(like: { value(:vote), )}) end Command Generators
143. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name) do tuple(like: { value(:vote), value(name), )}) end Command Generators
144. def gen_commands(name) do list(of: gen_vote(name), max: 20) end def gen_vote(name) do tuple(like: { value(:vote), value(name), choose(from: [value(1), value(2), value(3)])}) end Command Generators
145. defmodule ClientStateMachine do end
146. defmodule ClientStateMachine do def vote(name, id) do %{"votes" => new_votes} = post(id, name) {:ok, new_votes} end end
147. defmodule ClientStateMachine do def vote(name, id) do %{"votes" => new_votes} = post(id, name) {:ok, new_votes} end def vote_next(state, [id, name], _result) do {:ok, update_in(state, [name, to_string(id)], &(&1 + 1))} end end
148. defmodule ClientStateMachine do def vote(name, id) do %{"votes" => new_votes} = post(id, name) {:ok, new_votes} end def vote_next(state, [id, name], _result) do {:ok, update_in(state, [name, to_string(id)], &(&1 + 1))} end def vote_post(state, [id, name], actual_result) do expected_result = get_in(state, [name, to_string(id)]) + 1 {:ok, actual_result == expected_result} end end
149. Property: Users votes should increase test “users votes increase after voting" do ptest [commands: gen_commands("chris")] do VoteCounter.reset() {_state, result} = run_commands(commands, Client) assert result end end
151. Property: Users shouldn’t effect other users votes
152. Property: Users should get the correct votes test "users don't effect each others votes" do end
153. Property: Users should get the correct votes test "users don't effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do end end
154. Property: Users should get the correct votes test "users don't effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() end end
155. Property: Users should get the correct votes test "users don't effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() {_state, result} = run_commands([chris, jane], Client) end end
156. Property: Users should get the correct votes test "users don't effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() {_state, result} = run_commands([chris, jane], Client) assert result end end
157. Property: Users should get the correct votes test "users don't effect each others votes" do ptest [chris: gen_commands("chris"), jane: gen_commands("jane")] do VoteCounter.reset() {_state, result} = run_parallel_commands([chris, jane], Client) assert result end end
158. Running parallel tests def run_parallel_commands([l1, l2], module) do t1 = Task.async(fn -> run_commands(l1, module) end) t2 = Task.async(fn -> run_commands(l2, module) end) {_, ra} = Task.await(t1) {_, rb} = Task.await(t2) {:ok, ra && rb} end
161. The Bug def new(conn, %{"id" => id, "name" => name}) do {:ok, current_votes} = VoteCounter.get(id) new_votes = [name | current_votes] VoteCounter.put(id, new_votes) # Other nonsense end
162. The Bug Votes
163. The Bug Votes 3 3
164. The Bug Votes 3 3
165. The Bug Votes 4 4
166. The Bug Votes 4 4
167. The Bug 3 + 1 + 1 == 4 ?
168. The Bug def new(conn, %{"id" => id, "name" => name}) do {:ok, current_votes} = VoteCounter.get(id) new_votes = [name | current_votes] VoteCounter.put(id, new_votes) # Other nonsense end
169. The Bug def new(conn, %{"id" => id, "name" => name}) do {:ok, new_votes} = VoteCounter.incr(id, name) # Other nonsense end
171. Conclusion
172. How to think in properties Generating data Generate commands Model users as FSMs
173. Resources: “Finding Race conditions in Erlang with QuickCheck and PULSE” http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.724.3518&rep=rep1&type=pdf Testing Async apis with QuickCheck https://www.youtube.com/watch?v=iW2J7Of8jsE&t=272s “QuickCheck: A lightweight tool for Random Testing of Haskell Programs” http://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf Composing Test Generators https://www.youtube.com/watch?v=4-sPhFtGwZk Property based testing for better code https://www.youtube.com/watch?v=shngiiBfD80
174. If you could write less code and find more bugs would you do that?
175. Thanks Chris Keathley / @ChrisKeathley / c@keathley.io