SG Kafka Meetup - March 2022 : Incremental migration using CDC and KSQL

Slides from the Tech Talk on "Incremental migration using CDC and KSQL" presented in Singapore Apache Kafka Meetup in March 2022

Transcript

1. Incremental Migration using CDC & KSQL Zabeer Farook & Kaleeswaran Karuppusamy CREDIT AGRICOLE CIB, Singapore By Singapore Apache Kafka Meetup
2. Zabeer Farook Working as a Technical Architect in Credit Agricole CIB, Singapore. Experience working with Kafka & KSQL based data pipelines. https://sg.linkedin.com/in/zabeer-farook 2
3. Kaleeswaran Karuppusamy Working as a Technical Architect in Credit Agricole CIB, Singapore. Experience in migration & modernization of applications as well as Kafka based data pipelines. 3 https://www.linkedin.com/in/kaleeswaran-karuppusamy
4. Agenda ◎ General Migration Strategies ◎ Role of CDC in incremental migration ◎ Know more about CDC pipeline with KSQL & Kafka Streams ◎ Demo of a CDC pipeline with KSQL 4
5. 5 Legacy systems are complex and migrations aren’t easy either
6. 6 Big bang migrations are too risky
7. Incremental Migration a Better Option ◎More sustainable ◎Less Risky ◎Module by Module ◎Read workloads First, Write workloads Next ◎Combination of the above 7
8. Let’s review the migration strategies in general 8
9. 9 6R’s Migration Strategy
10. 10 Incremental Migration – Strangler Pattern
11. 11 Incremental Migration – Legacy App with Mainframe
12. 12 Data pipeline acts as backbone for incremental migration
13. What is CDC (Change Data Capture) ? ◎ Captures event streams for any database changes ◎ Change events typically sourced from DB Txn Logs ◎ Events further streamed/shipped to target typically via messaging infrastructure 13 SOURCE DATABASE MESSAGE BUS TARGET DATABASE EVENT STREAMING CDC RELAY SERVICE TAIL
14. Types of CDC ◎ Query/Poll based ◎ Trigger based ◎ Log based 14
15. “ CDC liberalizes data to realize endless opportunities 15 ◎ DB migration ◎ Audit/History event capturing ◎ Analytics data store ◎ Index based search ◎ Cache Update/Invalidation ◎ Update CQRS Read Models ◎ Outbox Microservice Pattern Other Use Cases of CDC
16. Some CDC Tools out there.. 16
17. 17 Typical CDC Pipeline with Debezium, KSQL & Kafka Streams 17
18. 18 Role of KSQL/Kafka Streams in a data pipeline Transformation Filtering Aggregation Lookup
19. 19 KSQLDB Vs Kafka Streams
20. Demo Time 20 https://github.com/Zabi82/confluent-meetup-mar2022
21. 21 Demo Usecase
22. Resources Confluent Kafka : https://developer.confluent.io/ Debezium : https://debezium.io/documentation/ Demo Repo: https://github.com/Zabi82/confluent-meetup-mar2022 22
23. Thanks.. Any questions? 23