API Resiliency Testing

Are your APIs leaving you worried about data loss, data corruption, security vulnerabilities, or poor user experiences? If so, this talk is exactly what you need to address these issues. Discover how you can leverage your API specifications—like OpenAPI, AsyncAPI, and GraphQL SDLs—to test and enhance the resiliency of your APIs with a completely #NOCODE approach.

In this talk, you’ll gain insights into the importance of API resilience and how to build APIs that can gracefully handle flaky dependencies in real-world scenarios. We’ll explore the role of contract testing in ensuring resilience and demonstrate how to transform API specifications into executable contracts that can be continuously validated. You’ll see how to set up resilience tests, including those for latency, errors, and unexpected disconnections. The session features live demos on configuring tools to simulate these conditions, interpret the results, and iteratively enhance your API designs.

Transcript

1. © 2025 All Rights Reserved API Resiliency Testing Applying Property Based Testing and Mutation Testing to APIs Hari Krishnan hari@specmatic.io @harikrishnan83
2. © 2025 All Rights Reserved Introduction OpenAPI Specifications as API Contract Tests #NOCODE
3. © 2025 All Rights Reserved Production Architecture Request Response Request Response 1 2 3 4 5 Message Topic Domain Service App BFF Analytics Service
4. © 2025 All Rights Reserved System Under Test Dependencies Test Test Mode Request Response Request Response 1 2 3 4 5 Message Topic Domain Service App BFF Analytics Service
5. © 2025 All Rights Reserved Dependencies System Under Test Test Test Mode – Contract Test Setup Request Response Request Response 1 3 4 5 Message Contract Test Http Stub Kafka mock Broker (In-memory) Schema Validation Topic BFF Set Expectations 2 Set Expectations Verify message count and schema 7 8 6
6. © 2025 All Rights Reserved Live Demo OpenAPI Specifications as API Contract Tests #NOCODE
7. © 2025 All Rights Reserved API Coverage Report Drill Down API Contract Test Report Drill Down Correlate
8. © 2025 All Rights Reserved How do we cover the 4xx scenarios?
9. © 2025 All Rights Reserved Property Based Testing Treating API Specifications as Behavior Definition
10. © 2025 All Rights Reserved Property Based Testing • Define rules or properties that your system should always satisfy • Instead of writing individual test cases • Example: “Addition is commutative: a + b = b + a.” https://scalacheck.org/ QuickCheck https://hackage.haskell.org/package/QuickCheck
11. © 2025 All Rights Reserved API Specifications as API Behavior Definition Mandatory vs Optional Nullability Only gadget, book, food or other Range
12. © 2025 All Rights Reserved Live Demo Exercising range of inputs within the boundaries set by API specification
13. © 2025 All Rights Reserved Mutation Testing Introducing mutations to your code to check the quality of your tests
14. © 2025 All Rights Reserved Mutation Testing Code (System Under Test) if (currVal == newVal) { return currVal } Test Suite if (currVal = newVal) { return currVal }
15. © 2025 All Rights Reserved Mutation Testing • Introduce small deliberate changes called mutations in your code • See if your tests can identify them • Thereby killing the mutation • Mutation Score = Mutants Killed / Total Mutants • Examples: • Java PIT • Python Mutmut • JS / TS Stryker
16. © 2025 All Rights Reserved Mutation Testing as applied API Specification • Instead of mutating code, introduce mutations in inputs. Examples: • Skip sending a mandatory field • Send the wrong data type • Expect the application to handle these spurious inputs
17. © 2025 All Rights Reserved Negative Scenario Testing • Deliberately send invalid inputs • And expect that the response is not a ”200 OK ” • Thereby validating that the input mutation has been successfully killed by input validation in your Component / Service (System Under Test)
18. © 2025 All Rights Reserved Negative Test Example – Mutating Data Type
19. © 2025 All Rights Reserved Live Demo API Resiliency Testing
20. © 2025 All Rights Reserved Issues with 500 response for invalid input • Conveys poor health of application • Thereby even bad input can lead to middleware considering service to be unavailable • Exposes unnecessary technical details
21. © 2025 All Rights Reserved What can be worse than a 500?
22. © 2025 All Rights Reserved Input validation is non-negotiable • “Insufficient Input/Output Validation” is one of the top risks listed by OWASP - https://owasp.org/www-project-mobile-top-10/2023- risks/m4-insufficient-input-output-validation • Can lead to incorrect data being introduced into your data store • Compromise other system and services that depend on this data • And many more high-risk issues
23. © 2025 All Rights Reserved Under the hood The math behind the test generation
24. © 2025 All Rights Reserved Why 77 tests for POST /products ?? Fields Data Type Required Constraints Values name string Yes - - type string Yes Enum book, food, gadget & others inventory number Yes min & max min = 1 & max = 101 Positive Variations Negative Variations Variation Count string int, bool, null 4 4 enum values int, bool, null 7 1, 1<random value<101, 101 0, 102, string, bool, null 8 = 224
25. © 2025 All Rights Reserved How do we cover a 503 scenario?
26. © 2025 All Rights Reserved Fault Injection Delay simulation and more in dependencies
27. © 2025 All Rights Reserved Delay simulation Stub Data
28. © 2025 All Rights Reserved
29. © 2025 All Rights Reserved Recap Why API Spec driven API Resiliency Testing is indispensable?
30. © 2025 All Rights Reserved #NOCODE #LOWCODE API Resiliency Testing API Resiliency Tests API Specification Service Implementation
31. © 2025 All Rights Reserved Thank You! Hari Krishnan hari@specmatic.io @harikrishnan83