Don’t Fear Outdated Caches - Change Data Capture to the Rescue

Joint presentation with Katia Aresti at Bordeaux JUG

The saying goes that there are only two hard things in Computer Science: cache invalidation and naming things.

Well, turns out the first is solved actually ;)

Come and join us for a session to hear not about one, but two popular open-source projects.

First, Infinispan, an in-memory distributed data store and cache, which is elastically scalable, highly availability and fault-tolerant.

Then, Debezium, a change data capture (CDC) platform for a variety of databases, such as MySQL, PostgreSQL and MongoDB.

After an overview of the key use cases and capabilities of the two projects, we'll explore in a demo how you can combine Debezium and Infinispan for even greater awesomeness:

* How to keep your cache automatically up-to-date after data changes via Debezium and CDC
* How to read denormalized views of your data with plain key look-ups from an Infinispan cluster close by
* How to propagate cache updates between clusters using cross-site replication

All this driven by Quarkus and connected via Apache Kafka. Bring your questions and ideas so we can discuss and exchange about the potential of these open-source projects in the data space!

Transcript

1. Don’t Fear Outdated Caches - Change Data Capture to the Rescue Discovering Infinispan and Debezium Gunnar Morling Debezium Katia Aresti Infinispan January 2021 - Home Edition Image © Nathalie https://flic.kr/p/21Ghf2g (CC BY 2.0)
2. @karesti | @gunnarmorling #Infinispan #Debezium ...Multi-site application with shared system-of-record database ...With local, denormalized read views (CQRS) ... Automatically kept in sync after writes Today’s Mission Explore how to build a... 🤔
3. @karesti | @gunnarmorling #Infinispan #Debezium Who are we? Gunnar Morling Katia Aresti
4. @karesti | @gunnarmorling #Infinispan #Debezium Agenda Demo! Learn Learn
5. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
6. @karesti | @gunnarmorling #Infinispan #Debezium App Data Local In memory Cache Embedding a library
7. @karesti | @gunnarmorling #Infinispan #Debezium App Data Local In memory Infinispan Local cache
8. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 1, Juan App-1 Data App-2 Data App-3 Data Local Local Local PUT 2,Maria GET 2 REMOVE 1 Null
9. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 1, Juan 2, Maria 1, Juan 2, Maria App-1 Data App-2 Data App-3 Data Replicated
10. @karesti | @gunnarmorling #Infinispan #Debezium Potter-3 Spells PUT 3,Mark 3, Mark 3, Mark Replicated 1, Juan 2, Maria 1, Juan 2, Maria 1, Juan 2, Maria App-1 Data App-2 Data App-3 Data
11. @karesti | @gunnarmorling #Infinispan #Debezium Replicated 1, Juan 2, Maria 3, Mark 1, Juan 2, Maria 3, Mark 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data
12. @karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data
13. @karesti | @gunnarmorling #Infinispan #Debezium GET 3 Mark GET 3 Mark Distributed 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data App-3 Data
14. @karesti | @gunnarmorling #Infinispan #Debezium 3, Reducto App-3 Data Distributed Null GET 3 1, Juan 2, Maria 3, Mark App-1 Data App-2 Data
15. @karesti | @gunnarmorling #Infinispan #Debezium PUT 4, Bella 4, Bella Distributed 1, Juan 3, Mark 2, Maria 1, Juan 3, Mark 2, Maria App-1 Data App-2 Data App-3 Data
16. @karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 3, Mark 4, Bella 2, Maria 1, Juan 3, Mark 2, Maria 4, Bella App-1 Data App-2 Data App-3 Data
17. @karesti | @gunnarmorling #Infinispan #Debezium Distributed 1, Juan 3, Mark 4, Bella 2, Maria 1, Juan 3, Mark 2, Maria 3, Bella App-1 Data App-2 Data App-3 Data App-4 Data
18. @karesti | @gunnarmorling #Infinispan #Debezium Consistent Hashing N1 N2 N3 N4
19. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 2, Maria 4, Mark 2, Maria 3, Bella 4, Mark 1, Juan 3, Bella App-3 Data App-2 Data App-4 Data CH App-1 Data
20. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 4, Mark 2, Maria 3, Bella 1, Juan 3, Bella 2, Maria 4, Mark App-3 Data App-2 Data App-4 Data CH App-1 Data
21. @karesti | @gunnarmorling #Infinispan #Debezium 1, Juan 4, Mark 2, Maria 3, Bella 1, Juan 3, Bella 2, Maria 4, Mark App-3 Data App-2 Data App-4 Data App-1 Data
22. @karesti | @gunnarmorling #Infinispan #Debezium
23. @karesti | @gunnarmorling #Infinispan #Debezium Microservice 1 Microservice 2 Microservice 3 Stateless App 1 Data
24. @karesti | @gunnarmorling #Infinispan #Debezium Infinispan Client/Server Microservice 1 Microservice 2 hotrod 💖 Rest ... Microservice 3 Data Infinispan Server Cluster Data Data
25. @karesti | @gunnarmorling #Infinispan #Debezium Latency chain Microservice 1 Microservice 2 Microservice 3 Microservice 2 Distributed cache with single owner Data
26. @karesti | @gunnarmorling #Infinispan #Debezium Data Center 1 Data Center 2 Load Balancer APP APP Microservice APP APP Microservice Shared State Shared State Shared State Shared State Customers US Customers EU Data Data
27. @karesti | @gunnarmorling #Infinispan #Debezium Cross Site Replication NYC +Data Data CEN Data LON Data RELAY2 RELAY2 RELAY2
28. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot
29. @karesti | @gunnarmorling #Infinispan #Debezium A Common Problem: Updating Multiple Resources
30. @karesti | @gunnarmorling #Infinispan #Debezium “Friends don’t let friends do dual-writes!” A Common Problem: Updating Multiple Resources
31. @karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes out of the Database
32. @karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes out of the Database
33. @karesti | @gunnarmorling #Infinispan #Debezium A Better Solution Streaming Changes out of the Database
34. @karesti | @gunnarmorling #Infinispan #Debezium • CDC for multiple databases ◦ Based on transaction logs ◦ Snapshotting, Filtering etc. • Fully open-source, very active community • Latest version: 1.4 • Production deployments at multiple companies (e.g. WePay, JW Player, Convoy, Trivago, OYO, BlaBlaCar etc.) Debezium Streaming Changes out of the Database
35. @karesti | @gunnarmorling #Infinispan #Debezium • Stable ◦ MySQL ◦ Postgres ◦ SQL Server ◦ MongoDB • Incubating ◦ Vitess ◦ Cassandra ◦ DB2 ◦ Oracle Debezium Supported Databases
36. @karesti | @gunnarmorling #Infinispan #Debezium Query-based Log-based All data changes are captured - No polling delay or overhead - Transparent to writing applications and models - Can capture deletes and old record state - Simple Installation/Configuration - Debezium Log vs. Query-based CDC
37. @karesti | @gunnarmorling #Infinispan #Debezium Change Data Capture Low-latency Data Streaming Pipelines Source: Logs & Offsets: (Near) Real Time ELT with Apache Kafka + Snowflake
38. @karesti | @gunnarmorling #Infinispan #Debezium Change Data Capture Liberation for Your Data
39. @karesti | @gunnarmorling #Infinispan #Debezium ▪ Propagate data between different services without coupling ▪ Each service keeps optimised views locally Microservices Data Exchange
40. @karesti | @gunnarmorling #Infinispan #Debezium http://bit.ly/debezium-outbox-pattern Outbox Pattern
41. @karesti | @gunnarmorling #Infinispan #Debezium Photo: “Strangler vines on trees, seen on the Mount Sorrow hike” by cynren, under CC BY SA 2.0 Mono-to-Micro: Strangler Fig Pattern
42. @karesti | @gunnarmorling #Infinispan #Debezium Customer Mono-to-Micro: Strangler Fig Pattern
43. @karesti | @gunnarmorling #Infinispan #Debezium Mono-to-Micro: Strangler Fig Pattern Customer’ Router CDC Transformation Reads / Writes Reads Customer
44. @karesti | @gunnarmorling #Infinispan #Debezium Customer Router CDC Reads / Writes Reads / Writes CDC Mono-to-Micro: Strangler Fig Pattern
45. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams
46. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams
47. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams
48. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams
49. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams
50. @karesti | @gunnarmorling #Infinispan #Debezium Auditing with CDC and Kafka Streams
51. @karesti | @gunnarmorling #Infinispan #Debezium http://bit.ly/debezium-auditlogs Auditing with CDC and Kafka Streams
52. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo
54. @karesti | @gunnarmorling #Infinispan #Debezium • Fast start-up, low memory consumption • Developer joy • Imperative and Reactive • Best-of-breed libraries • Run via HotSpot and GraalVM native binaries Quarkus - Supersonic subatomic Java A Stack for building Cloud-native Apps
55. @karesti | @gunnarmorling #Infinispan #Debezium • Java API for stateful stream processing • Rich set of operators • Scaling out to multiple JVMs • Interactive queries Kafka Streams Streaming Queries on Kafka Topics
56. @karesti | @gunnarmorling #Infinispan #Debezium Data marshalling
57. https://flic.kr/p/PFDvkY Public Domain, Angelo Brathot Demo
58. @karesti | @gunnarmorling #Infinispan #Debezium Infinispan protostream
59. @karesti | @gunnarmorling #Infinispan #Debezium ...Multi-site application with shared system-of-record database ✅ ...With local, denormalized read views (CQRS) ✅ ... Automatically kept in sync after writes ✅ Today’s Mission Explore how to build a... 🤩
60. @karesti | @gunnarmorling #Infinispan #Debezium • Partial Updates, Multimap support • Debezium embedded into Infinispan • Kafka Connect sink connector for Infinispan ◦ https://github.com/infinispan/infinispan-kafka • CDC connector for Infinispan? Outlook
61. @karesti | @gunnarmorling #Infinispan #Debezium • Infinispan: @infinispan | https://infinispan.org/ • Debezium: @debezium | https://debezium.io/ • Demo: https://github.com/debezium/debezium-examples/ → distributed-caching Learn More
62. @karesti | @gunnarmorling #Infinispan #Debezium Q & A Learn More