Property-based test beginning with SwiftCheck

try! Swift 2019 Tokyo - Day 2
https://www.tryswift.co/events/2019/tokyo/jp/#property-based

1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test beginning with SwiftCheck Yusuke Hosonuma @DeNA try! Swift 2019 TOKYO / Day 2
2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Yusuke Hosonuma @DeNA Favorite Language Swift / Go / Haskell Twitter: @tobi462
3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. SoftWare Engineer in Test Test Automation, CI/CD and more…
4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Organizer of the Events iOSɹɹAndroidɹɹCI/CD https://testnight.connpass.com/
5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Private Developer Community Pengin-mura
6. Agenda Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Agenda • What’s Property-based test ? • What’s SwiftCheck ? • Use cases in Action • Conclusion
8. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. What’s Property-based test
9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test • Describe tests as property • Randomly generated input values
10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test • Describe tests as property • Randomly generated input values
11. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example-based test
12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example-based test • Give input and expected values • Pass the input values to the function • Check that the output result matches the expected value
13. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed
14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed 1 2 3 4 5 5 4 3 2 1
15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed() { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed() { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed() { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed() { let xs = [1, 2, 3, 4, 5] XCTAssertEqual(xs.reversed(), [5, 4, 3, 2, 1]) } How many patterns are necessary?
19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed() { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... }
20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed() { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... } There are infinite possible input values!
21. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition
22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition Detect bugs when missed patterns
23. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test
24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Think property • Specific input values are not considered. • What is the relationship between the input value and the output value? • What is property of function?
25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. What’s property 1 2 3 4 5 5 4 3 2 1
26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change 1 2 3 1 1 2 … 9 3 2 1 1 9 … 2 1
27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. reverse x2 = original 1 2 3 4 5 1 2 3 5 4 3 2 1 3 2 1 1 2 3 4 5 1 2 3
28. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed • The length does not change • When it is executed twice, it returns to the original
29. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed • The length does not change • When it is executed twice, it returns to the original This is a common property to any input values
30. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
31. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
32. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
33. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
34. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays } Let’s try SwiftCheck !
35. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. What’s SwiftCheck ?
36. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. SwiftCheck • Property-based test in Swift (OSS) • Inspired by QuickCheck (in Haskell) • with XCTest https://github.com/typelift/SwiftCheck
37. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Swift is inspired by Haskell • Functionality • Enum • Pattern match
38. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Write the 1st SwiftCheck
39. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
40. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } XCTest method
41. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe summary
42. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Random values
43. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe property
44. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
45. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } What input values are generated?
46. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests
47. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Every time random
48. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Test is succeeded
49. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed *** Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests
50. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed *** Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Summary
51. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed *** Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Failed value
52. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 2 words • Arbitrary • Shrinking
53. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary
54. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary • Protocol to generate random values • Adopt it can be used as a random value • Swift standard types are adopted
55. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
56. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
57. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } }
58. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Has pair of Int
59. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Return generater of Point
60. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type for _ in 0...4 { print(Point.arbitrary.generate) } // Point(x: -2, y: -26) // Point(x: -15, y: 2) // Point(x: 20, y: -21) // Point(x: -30, y: -15) // Point(x: -30, y: 6) Generated random Points
61. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking
62. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking • Get more small values • Reported a smaller failure case
63. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
64. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
65. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 10 is failed
66. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 shrink 5
67. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 still failed 5
68. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 5 2 still failed
69. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 5 2 1 still failed
70. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 passed ! 5 2 1 0
71. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking 10 5 2 1 0 reported !
72. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Use cases in Action
73. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Use cases • Randomized algorithm • Symmetric algorithm • Fast vs Slow
74. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Randomized algorithm
75. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation • Example-based Test is difficult • Random number seeds can given as input values, but the algorithm can not be improved
76. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation • Property ⁃ Maze must always be solved
77. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) }
78. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } generate seed
79. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } always solve
80. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Symmetric algorithm
81. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode / decode • Encoded and decoded matches the original value • The reverse is also the same
82. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode / decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
83. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode / decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
84. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode / decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
85. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode / decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
86. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow
87. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow • Fast algorithm is complicated to implement • The obvious algorithm is slow
88. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow • Both should be the same result • Use explicit (but slow) algorithms for fast algorithm (but complex) verification
89. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
90. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
91. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Conclusion
92. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based Test • Describe tests as property • Randomly generated input values • SwiftCheck is QuickCheck for Swift
93. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Is the Example-based Test is Dead? • Does not replace the Example-based Test • Reduce anxiety about the miss of test patterns. • Combining them makes it a powerful card
94. 2VJDL Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Let’s try Property-based test with SwiftCheck
95. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix • https://en.wikipedia.org/wiki/Property_testing • https://github.com/typelift/SwiftCheck • https://www.objc.io/books/functional-swift/