Efficient integrating data from multiple data providers

Yu-Mei Chen
LINE Taiwan Backend Software Engineer
https://linedevday.linecorp.com/jp/2019/sessions/S2-4

Transcript

1. 2019 DevDay Efficient Integrating Data From Multiple Data Providers > Yu-Mei Chen > LINE Taiwan Backend Software Engineer
2. Subtitle
3. Partner A LINE SPOT Users Partner B Partner C … $
4. Practices From Cooperating with Partners > Open-Closed Principle
 There are no permanent enemies, and no permanent friends. > Generalization & Customization
 Fighting for general data structures,
 but still keep a flexible layer for customized. > Processing Data Manually
 Always Preparing the manual way to update the data,
 such as updating data by import the .csv files.
5. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Its All About “Maintainability” Operability Evolvability Simplicity
6. Reference: Chapter 1 of “Designing Data-Intensive Applications”, Author: Martin Kleppmenn Its All About “Maintainability” Processing Data Manually Operability Generalization & Customization Evolvability Simplicity Open-Closed Principle
7. Our Solution: Event Sourcing A Cloud Design Pattern for Backend Architecture
8. Event Sourcing With Kafka Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
9. Processing Data Manually Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
10. Open-Closed Principle Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
11. Generalization & Customization Apache Kafka Partner A Stream Store Service Coupon Service Voucher Service Store
 Locator Kafka Connector Kafka Connector Kafka Connector Partner B Stream Partner C Stream … Stream Partner Dispatcher Mysql POI DB Mysql Coupon DB Mysql Voucher DB Elasticsearch Materialize
 View for
 Store
12. Summary > Challenge in LINE SPOT: Maintainability
 > Solution: Event Sourcing Thank You for Listening And welcome for further discussion: yumei.chen@linecorp.com