Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Property-based test beginning with SwiftCheck

Property-based test beginning with SwiftCheck

Yusuke Hosonuma

March 22, 2019
Tweet

More Decks by Yusuke Hosonuma

Other Decks in Programming

Transcript

  1. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test beginning

    with SwiftCheck Yusuke Hosonuma @DeNA try! Swift 2019 TOKYO / Day 2
  2. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Yusuke Hosonuma @DeNA

    Favorite Language Swift / Go / Haskell Twitter: @tobi462
  3. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Organizer of the

    Events iOSɹɹAndroidɹɹCI/CD https://testnight.connpass.com/
  4. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Agenda • What’s

    Property-based test ? • What’s SwiftCheck ? • Use cases in Action • Conclusion
  5. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test •

    Describe tests as property • Randomly generated input values
  6. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test •

    Describe tests as property • Randomly generated input values
  7. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example-based test •

    Give input and expected values • Pass the input values to the function • Check that the output result matches the expected value
  8. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  9. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  10. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual([5, 4, 3, 2, 1], xs.reversed()) }
  11. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let xs = [1, 2, 3, 4, 5] XCTAssertEqual(xs.reversed(), [5, 4, 3, 2, 1]) } How many patterns are necessary?
  12. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... }
  13. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Array#reversed func testReversed()

    { let empty = Array<Int>() XCTAssertEqual(empty.reversed(), empty) XCTAssertEqual( [1].reversed(), [1]) XCTAssertEqual( [1, 2].reversed(), [2, 1]) XCTAssertEqual([1, 2, 3, 4, 5].reversed(), [5, 4, 3, 2, 1]) ... } There are infinite possible input values!
  14. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we

    choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition
  15. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. How do we

    choose input values? • A value that is likely to cause a bug ⁃ Boundary value ⁃ Equivalence partition Detect bugs when missed patterns
  16. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Think property •

    Specific input values are not considered. • What is the relationship between the input value and the output value? • What is property of function?
  17. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change 1 2 3 1 1 2 … 9 3 2 1 1 9 … 2 1
  18. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. reverse x2 =

    original 1 2 3 4 5 1 2 3 5 4 3 2 1 3 2 1 1 2 3 4 5 1 2 3
  19. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed

    • The length does not change • When it is executed twice, it returns to the original
  20. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property of #reversed

    • The length does not change • When it is executed twice, it returns to the original This is a common property to any input values
  21. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  22. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  23. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  24. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays }
  25. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based test by

    self for xs in randomArrays() { // The length does not change XCTAssertEqual(xs.reversed().count, xs.count) // When it is executed twice, it returns to the original XCTAssertEqual(xs.reversed().reversed(), xs) } func randomArrays() -> [[Int]] { // generate random arrays } Let’s try SwiftCheck !
  26. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. SwiftCheck • Property-based

    test in Swift (OSS) • Inspired by QuickCheck (in Haskell) • with XCTest https://github.com/typelift/SwiftCheck
  27. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Swift is inspired

    by Haskell • Functionality • Enum • Pattern match
  28. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
  29. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } XCTest method
  30. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe summary
  31. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Random values
  32. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } Describe property
  33. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } }
  34. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. The length does

    not change func testReversed() { property("length not changed”) <- forAll { (xs: [Int]) in return xs.reversed().count == xs.count } } What input values are generated?
  35. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests
  36. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Every time random
  37. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Print generated values

    [] [] [-2, -2] [-2] [2, -1, 4] [1, -4, -4] [-5, -4] ... *** Passed 100 tests Test is succeeded
  38. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests
  39. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Summary
  40. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. When failed ***

    Failed! Proposition: length not changed Falsifiable (after 1 test): [] *** Passed 0 tests Failed value
  41. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary • Protocol

    to generate random values • Adopt it can be used as a random value • Swift standard types are adopted
  42. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  43. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  44. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } }
  45. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Has pair of Int
  46. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type struct

    Point { var x: Int var y: Int } extension Point: Arbitrary { static var arbitrary: Gen<Point> { return Gen<(Int, Int)> .zip(Int.arbitrary, Int.arbitrary) .map(Point.init) } } Return generater of Point
  47. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Custom type for

    _ in 0...4 { print(Point.arbitrary.generate) } // Point(x: -2, y: -26) // Point(x: -15, y: 2) // Point(x: 20, y: -21) // Point(x: -30, y: -15) // Point(x: -30, y: 6) Generated random Points
  48. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Shrinking • Get

    more small values • Reported a smaller failure case
  49. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  50. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Arbitrary Protocol public

    protocol Arbitrary { static var arbitrary : Gen<Self> { get } static func shrink(_ : Self) -> [Self] }
  51. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Use cases •

    Randomized algorithm • Symmetric algorithm • Fast vs Slow
  52. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    • Example-based Test is difficult • Random number seeds can given as input values, but the algorithm can not be improved
  53. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) }
  54. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } generate seed
  55. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: Maze generation

    func canSolve(_ maze: Maze) -> Bool { // judge solve or not } property("must solve") <- forAll { (seed: UInt64) in let maze = Maze.generate(seed: seed) return canSolve(maze) } always solve
  56. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode • Encoded and decoded matches the original value • The reverse is also the same
  57. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  58. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  59. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  60. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Example: encode /

    decode property("encode and decode are synmetry”) <- forAll { (bird: Bird) in let bytes = bird.encodeToBytes() return Bird.decode(bytes: bytes) == bird }
  61. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    • Fast algorithm is complicated to implement • The obvious algorithm is slow
  62. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    • Both should be the same result • Use explicit (but slow) algorithms for fast algorithm (but complex) verification
  63. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
  64. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Fast vs Slow

    property("same result") <- forAll { (xs: [Int]) in return slowSort(xs) == fastSort(xs) }
  65. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Property-based Test •

    Describe tests as property • Randomly generated input values • SwiftCheck is QuickCheck for Swift
  66. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Is the Example-based

    Test is Dead? • Does not replace the Example-based Test • Reduce anxiety about the miss of test patterns. • Combining them makes it a powerful card
  67. Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Appendix • https://en.wikipedia.org/wiki/Property_testing

    • https://github.com/typelift/SwiftCheck • https://www.objc.io/books/functional-swift/