Ins and Outs of the Outbox Pattern

The outbox pattern is a common solution for implementing data flows between microservices. By channeling messages through an outbox table, it enables services to update their own local datastore and at the same time send out notifications to other services via data streaming platforms such as Apache Kafka, in a reliable and consistent way.

However, as with everything in IT, there’s no free lunch. How to handle backfills of outbox events, how to ensure idempotency for event consumers? Doesn’t the pattern cause the database to become a bottleneck? And what about alternatives such as “Listen-to-Yourself”, or the upcoming Kafka support for 2-phase commit transactions (KIP-939)?

It’s time to take another look at the outbox pattern! In this session I’ll start by bringing you up to speed on what the outbox pattern *is*, and then go on to discuss more details such as:

- Implementing the pattern safely and efficiently
- Its semantics, pros and cons
- Dealing with backfills
- Potential alternatives to the outbox pattern and the trade-offs they make

Transcript

1. Gunnar Morling @gunnarmorling.dev Ins and Outs of the Outbox Pattern oatsy40 https://flic.kr/p/FvD3mP (CC BY 2.0) 🦋
3. #OutboxPattern · @gunnarmorling.dev Agenda
4. #OutboxPattern · @gunnarmorling.dev • Technologist at Confluent • Former project lead of Debezium • kcctl 🧸, JfrUnit, ModiTect, MapStruct • One Billion Row Challenge 1⃣🐝🏎 • Java Champion Gunnar Morling
5. #OutboxPattern · @gunnarmorling.dev The Problem How to… Update a service’s database… …AND notify other services? Consistently.
6. #OutboxPattern · @gunnarmorling.dev Dual Writes… …are prone to inconsistencies
7. The Outbox Pattern Elisabeth Hölzl https://flic.kr/p/m7rKFr (CC BY-SA 2.0)
8. #OutboxPattern · @gunnarmorling.dev Outbox Pattern
9. #OutboxPattern · @gunnarmorling.dev Outbox Pattern
10. #OutboxPattern · @gunnarmorling.dev Outbox Pattern
11. #OutboxPattern · @gunnarmorling.dev Outbox Pattern
12. #OutboxPattern · @gunnarmorling.dev Outbox Pattern The Outbox Table
13. Polling Michael Coghlan https://flic.kr/p/PDwXXS (CC BY-SA 2.0)
14. #OutboxPattern · @gunnarmorling.dev Polling Happy Path
15. #OutboxPattern · @gunnarmorling.dev Polling Ordering Problems
16. #OutboxPattern · @gunnarmorling.dev Polling Missing Events
17. Log-based CDC Barry Silver https://flic.kr/p/8A6XQV (CC BY 2.0)
18. #OutboxPattern · @gunnarmorling.dev Debezium
19. #OutboxPattern · @gunnarmorling.dev Debezium New and Lesser Known Features • Incremental snapshots • JDBC sink connector • Official support for MariaDB • Re-Select post processor • WASM-based SMT
20. #OutboxPattern · @gunnarmorling.dev Moving to the Commonhaus Foundation
21. #OutboxPattern · @gunnarmorling.dev Debezium Outbox Event Router
22. #OutboxPattern · @gunnarmorling.dev Debezium Outbox Event Router
23. #OutboxPattern · @gunnarmorling.dev Housekeeping Is Not Needed Actually!
24. #OutboxPattern · @gunnarmorling.dev Postgres-only pg_logical_emit_message()
25. Backfills Jeff Golden https://flic.kr/p/sRGBDX (CC BY-SA 2.0)
26. #OutboxPattern · @gunnarmorling.dev Backfills
27. #OutboxPattern · @gunnarmorling.dev Backfills
28. #OutboxPattern · @gunnarmorling.dev Backfills
29. #OutboxPattern · @gunnarmorling.dev DBLog: A Watermark Based Change-Data-Capture Framework
30. Idempotency formulanone https://flic.kr/p/woErp4 (CC BY-SA 2.0)
31. #OutboxPattern · @gunnarmorling.dev Consumer Idempotency How to Identify Duplicates? • A unique value • Not a database sequence • An increasing value, e.g. PG LSN
32. Discussion Lars K. Jensen https://flic.kr/p/7n4tJL (CC BY-SA 2.0)
33. #OutboxPattern · @gunnarmorling.dev Discussion Database Overhead Doesn’t writing outbox events create database overhead?
34. #OutboxPattern · @gunnarmorling.dev Discussion Database Overhead Doesn’t writing outbox events create database overhead? Yes, but…
35. #OutboxPattern · @gunnarmorling.dev Discussion Latency Doesn’t writing to the database increase end-to-end latency?
36. #OutboxPattern · @gunnarmorling.dev Discussion Latency Doesn’t writing to the database increase end-to-end latency? Yes, but…
37. Alternatives Alexander Svensson https://flic.kr/p/21UKFVe (CC BY 2.0)
38. #OutboxPattern · @gunnarmorling.dev Read Yourself Writing to Kafka first • Con: no synchronous “read-your-own-writes”
39. #OutboxPattern · @gunnarmorling.dev 2PC • Con: Reduced availability
40. #OutboxPattern · @gunnarmorling.dev Stream Processing • Con: complexities, e.g. around transactional consistency
41. Summary Jennifer Kramer https://flic.kr/p/24Uqs7 (CC BY 2.0)
42. #OutboxPattern · @gunnarmorling.dev Summary • Trade-offs every • Consider pg_logical_emit_message()
43. #OutboxPattern · @gunnarmorling.dev Summary • Trade-offs every • Consider pg_logical_emit_message()
44. #OutboxPattern · @gunnarmorling.dev gmorling@confluent.io @gunnarmorling @gunnarmorling.dev morling.dev 📧 Get In Touch