Test Smarter, Not Harder: Achieving Confidence in Complex Distributed Systems

Testing distributed systems is hard, but it doesn't have to be fragile. In this session, we explore how to test services that depend on multiple databases, external providers, and asynchronous events.

Transcript

1. Test Smarter, Not Harder: Achieving Confidence in Complex Distributed Systems Elias Nogueira
2. Elias Nogueira Lead Principal Software Engineer From Brazil Java Champion Oracle ACE Pro Java Magazine NL Editor
3. How this talk is organized Problem Code example How to solve Takeaways
4. Architecture – Business View Digital Onboarding Digital Lending Digital Banking Digital Assist Digital Engage Core Systems Common Services | Enterprise Integrations | Unified Security Fintech Partners Open Banking API API API Web and Mobile Apps
5. Architecture – System View Payments API Digital Banking Baches Limits Contacts Audit Accounts Approvals Trading Transactions Access Control Forex Loan Cash Flow Cards Dashboard
6. Architecture – Detailed API View Payment Options Payment Order Limits Batches Payment Integration Audit Approvals Core Bank System Payments event-listener Bank Calendar Bank Calendar Integration Batches Integration Anti-fraud 3rd party
7. 1️⃣ Test with real dependencies 2️⃣ Support multiple databases 3️⃣ Mock dependencies 4️⃣ Test asynchronous requests 5️⃣ API governance across multiple teams Testing challenges on this architecture
8. 1️⃣ Test with real dependencies 2️⃣ Support multiple databases 3️⃣ Mock dependencies 4️⃣ Test asynchronous requests 5️⃣ API governance across multiple teams Testing challenges on this architecture
9. Coding example All examples are simple and focus on solving the problem rather than create a perfect example Payment Audit event-listener Anti-fraud 3rd party
10. One Application Multiple Databases
11. Modern systems often support multiple database engines ❌ Integration tests usually runs on in-memory databases, missing database-specific issues ❌ Testing sequentially slows feedback cycles and CI pipelines One Application Multiple Databases
12. Common problems Don’t show the real problems. Usage of In-Memory Databases Manual steps might be required. High-effort for real databases
13. Parallel and Containerized Testing for Multi-DB Support
14. Multi-database support Payment
15. Show me the code! Photo by Christina Morillo: https://www.pexels.com/photo/two-women-looking-at-the-code-at-laptop-1181263
16. Multi-database support Payment Run it sequentially is a bad idea! MySQL MSSQL Oracle Integration tests: 3min Integration tests: 3min Integration tests: 3 min ~ 9 min
17. Multi-database support Payment Run it in parallel MySQL MSSQL Oracle Integration tests: 3min Integration tests: 3min Integration tests: 3 min ~ 3 min
18. Show me the code! Photo by Christina Morillo: https://www.pexels.com/photo/two-women-looking-at-the-code-at-laptop-1181263
19. Takeaways Integration tests must validate behaviour across all target databases Containerization brings production-like test fidelity and consistency Parallel execution speeds up feedback and reduces CI bottleneck
20. Third-Party Systems Fragile Dependencies in Tests
21. Third-Party Systems In-code mocks can solve a problem in one team (only) Using sandbox/QA environments from the vendor might be flaky Payment Anti-fraud 3rd party
22. Can we trust third-party systems in our tests? ❌ Tests that rely on them are flaky and slow ❌ Local mocks are often inconsistent or limited to a single team Third-Party Systems
23. Service Virtualization with Wiremock and Docker
24. Service Virtualization Service virtualization is a technique used in software testing and development to simulate the behaviour of external systems or services that your application depends on — such as third-party APIs, databases, or legacy systems — without requiring the actual systems to be available. Instead of relying on the real services (which might be unavailable, expensive, or unstable), teams use lightweight stand-ins (or virtual services) that mimic the real ones' behaviour. These virtual services respond with realistic data and behaviours, allowing tests and development to continue independently of external constraints.
25. Service Virtualization Team 1 Team 2 Team N Standalone container Deploy in a k8s infra
26. Replace real external calls with Wiremock hosted in a container Share mock setups across teams for collaboration and consistency Enables reliable, repeatable integration testing without hitting sandbox third-party systems Service Virtualization Payment Fraud Check 3rd party
27. Show me the code! Photo by Christina Morillo: https://www.pexels.com/photo/two-women-looking-at-the-code-at-laptop-1181263
28. Predictable mocks enable reliable, isolated testing Release processes can move independently of third-party stability Dockerized mocks allow reuse across teams without manual setup Takeaways
29. Testing Asynchronous Events Timing is Everything
30. Testing Asynchronous Events test will fail to check or consume the event as it’s async Payment Audit event-listener emit an async event consume an async event 2 seconds
31. ❌ Systems based on event queues introduce timing uncertainty ❌ Integration tests fail due to delays in message consumption ❌ Developers are tempted to add sleep or polling hacks Testing Asynchronous Events
32. Synchronize Tests with Reality
33. ✅ Use Awaitility to wait for expected conditions dynamically ✅ Avoid fixed timeouts and make tests more resilient ✅ Model realistic event-based behaviour in integration tests Synchronize Tests with Reality
34. Synchronize Tests with Reality test will wait until the event is available Payment Audit event-listener emit an async event consume an async event 2 seconds
35. Show me the code! Photo by Christina Morillo: https://www.pexels.com/photo/two-women-looking-at-the-code-at-laptop-1181263
36. Async flows require resilient test assertions Maintain realistic tests while ensuring correctness in event processing Awaitility helps you wait just enough, not too long Takeaways
37. Thanks! Do you have any questions? github.com/eliasnogueira