Upgrade to Pro — share decks privately, control downloads, hide ads and more …

20240824_JAWS_PANKRATION_2024

 20240824_JAWS_PANKRATION_2024

Kazuki Maeda

August 24, 2024
Tweet

More Decks by Kazuki Maeda

Other Decks in Technology

Transcript

  1. Who am I Kazuki Maeda 𝕏 @kzk_maeda SRE/DRE/Enabling/EM @atama plus

    AWS Community Builders AWS Startup Community Core Member 2
  2. Main Topic Consider about architecture and organizational design of the

    product and data platform Dive Deep and provide my own guidelines, especially in cases where responsibilities and dependencies can be complex. 3
  3. Agenda 1. Product and Data Organization 2. Data Platform Architecture

    and Technology 3. Product System Operation and Data Platform w/ CDC 4. Case Pattern for Architecture & Organization 5. Conclusion 4
  4. Data Platform & Organization User Facing Product Core Value 6

    Internal Use Feedback & Analysis Product Platform Data Platform Transfer
  5. Roles of Product & Data Organization 7 Including … Product

    Manager, Developer, Operation SRE Product Platform Data Platform Transfer Including … Data Manager, Data Engineer, Analyst DRE
  6. Need for Responsibility Separation 8 Focus on … Delivering Values

    Maintaining Stability Product Platform Data Platform Transfer Focus on … Gathering Data Managing Data Quality
  7. Product Team Organization Pattern Example 9 1. 2. 3. Product

    Platform Data Platform Transfer SRE / Infrastructure Team SRE / Infra Team Data Team Product Team Embedded SRE Team Data Team
  8. Product Team Organization Pattern Example 10 1. 2. 3. Product

    Platform Data Platform Transfer SRE / Infrastructure Team SRE / Infra Team Data Team Product Team Product Team Data Team Direction of Dependence? Separation of Responsibility?
  9. Data Platform Architecture Batch Transfer Architecture 12 Product Platform Data

    Platform Batch Transfer • Transfer data in periodically • For example, data are all sent and full replaced in data platform • Or, data are sent in chunk periodically
  10. Data Transfer Technology Batch Transfer Technology 2 way technologies for

    Batch Transfer 13 Snapshot Transfer • Transfer full data via Database snapshot Query Transfer • Capture changes using SQL via JDBC connection • Use ETL Service or Broker Service
  11. Data Transfer Technology Batch Transfer Technology 14 Amazon Athena Amazon

    S3 Amazon Aurora Reader AWS Glue Amazon Redshift Amazon MSK Snapshot Snapshot Query chunk data JDBC Connector
  12. Dependence / Responsibility Batch Transfer Technology 15 Amazon Athena Amazon

    S3 Amazon Aurora Reader AWS Glue Amazon Redshift Amazon MSK Data Platform Product Platform Product Platform Data Platform Weak Dependence Little Dependence Snapshot Query chunk data
  13. Data Platform Architecture Streaming Transfer Architecture 16 Product Platform Data

    Platform Streaming Transfer • Transfer data in streaming • Data in data platform are updated in near real-time • Generally, Change data capture (CDC) is used
  14. Data Transfer Technology Change Data Capture 17 Amazon Aurora Writer

    Amazon MSK AWS DMS Amazon Athena Amazon Redshift Debezium Connector Zero-ETL Replication Logs Amazon S3
  15. Data Transfer Technology Change Data Capture 18 Amazon Aurora Writer

    Amazon MSK AWS DMS Amazon Athena Amazon Redshift Debezium Connector Zero-ETL Replication Logs Product Platform Amazon S3 Data Platform Strong Dependence
  16. Dependence / Responsibility Why is it so complex? • Most

    Database Engine in recent version, it is necessary to gather replication logs from writer instance • In addition, database user or settings for replication logs should be configured • So, boundary between both platform is ambiguous 19
  17. Data Platform Architecture Lambda Architecture 20 Product Platform Data Platform

    Streaming Transfer • Transfer data using both of batch and streaming • Benefit from both the certainty data of batch transfers and the near real-time data of streaming transfers • Use in carefully because of its complexity Batch Transfer
  18. Product System Operation Product operation cases or issues affecting Data

    Platform • Manual Table Maintenance ◦ In general, operating with manual failover or service maintenance • Suddenly Failover ◦ Hard to predict • Database Upgrade Operation ◦ Recently, operating with Aurora Blue/Green Deployment is recommended to avoid service down. 22
  19. Product System Operation Product operation cases or issues affecting Data

    Platform • Manual Table Maintenance ◦ In general, operating with manual failover or service maintenance • Suddenly Failover ◦ Hard to predict • Database Upgrade Operation ◦ Recently, operating with Aurora Blue/Green Deployment is recommended to avoid service down. 23 Dive Deep into this pattern
  20. Amazon Aurora Cluster Case Architecture Change Data Capture using DMS

    to Redshift serverless 24 Amazon Redshift Serverless binlog Writer instance AWS DMS anecdote: Oh no, using DMS Serverless transferring to Redshift Serverless is not supported!! v5.7 Operation Case: We’d like to upgrade Database Engine version to MySQL 8.0 using Aurora Blue / Green Deployment
  21. Operation affected to Data Platform Database Upgrade Operation using Blue/Green

    Deploy 25 Amazon Aurora Cluster (blue) Amazon Redshift Serverless binlog Writer instance AWS DMS Amazon Aurora Cluster (green) Writer instance binlog v5.7 v8.0 binlog 2 binlogs with different offset values
  22. Operation affected to Data Platform Database Upgrade Operation using Blue/Green

    Deploy 26 Amazon Aurora Cluster (blue) Amazon Redshift Serverless binlog Writer instance AWS DMS Amazon Aurora Cluster (green) Writer instance binlog v5.7 v8.0 binlog Switch Cannot follow binlog with different offset
  23. Operation affected to Data Platform Database Upgrade Operation using Blue/Green

    Deploy 27 Amazon Aurora Cluster (blue) Amazon Redshift Serverless binlog Writer instance AWS DMS Amazon Aurora Cluster (green) Writer instance binlog v5.7 v8.0 binlog Switch Cannot follow binlog with different offset
  24. Considering Point for above Problem Team Collaboration • If product

    team and data team are different, it is necessary to address those problems during operation • Direct Communication, maintain runbook, and so on Direction of Dependence and Separation of Responsibility • If team collaboration is difficult, it is important to design well direction of dependence and separation of responsibility 28
  25. Case pattern Small amount of data and small team size

    • Choose simple data platform and technology • Operate both platform by the same team 30 Amazon Aurora Cluster Writer instance Reader instance Amazon Athena Amazon S3 Cluster Snapshot (parquet file) Product and Data Platform Team
  26. Case pattern Medium amount of data and medium team size

    • If data freshness is important, create CDC platform using simple managed, serverless services • Collaborate with product and data team 31 Amazon Redshift Serverless replication log AWS DMS Amazon Aurora Cluster Writer instance Reader instance Product Platform Team Data Platform Team Dependence
  27. Case pattern Large amount of data, several products, large team

    size • Design dependency direction that data platform refers to the product platform 32 Amazon Redshift Serverless Amazon Aurora Cluster Writer instance Reader instance Amazon MSK JDBC Connector Product Platform Team Data Platform Team Dependence SQL
  28. Conclusion Proper solution depend on the several factors; • Characteristics

    of product • Size and capability of team • Technology stack that can be adopt It is important to; • design the team and the platform together • be aware of the separation of responsibilities and the direction of dependencies. 34