Kubernetes for Data Engineers: Building Scalable, Reliable Data Pipelines

K8S Summit 2024 Taipei

Transcript

1. K8S Summit TW 2024 Kubernetes for Data Engineers: Building Scalable, Reliable Data Pipelines Shuhsi Lin 20241024 Photo by Krishna Mantripragada on Unsplash
2. About Me Find me on sciwork member Interested in • Agile/Engineering Culture/ Developer Experience • Team Coaching • Data Engineering Shuhsi Lin Working in Smart manufacturing & AI With data and people Photo by NordWood Themes on Unsplash
3. Agenda 01 ETL/ELT & Complex Data pipelines Data Pipeline 02 What are Scalable and Reliable (+ Maintainable) pipelines Scalable & Reliable + Maintainable 03 dbt & Data Pipelines On K8S 04 Recap & More How to be Scalable and Reliable (+ Maintainable) with dbt + K8S More to do for Scalable, Reliable and Maintainable data pipelines
4. Scenario SmartPizza (like Pizzaxxx, but smarter?) • Daily work: get orders with specific recipes to make pizza • Data (tables) ◦ Order ◦ Recipes ◦ Customer ◦ Inventory ◦ … Assume if - Operate 20000 branches worldwide - Serve 4 million customers per day - Make 5 million pizzas per day
5. Data Pipelines https://unsplash.com/photos/a-group-of-pipes-that-are-connected-to-each-other-Xlg2KbYFUoM
6. Hello ETL/ELT World Extract Transform Load Load Transform Extract Input Data Output Data Operation ETL/ELT pipelines Data source Data store/target Data applications
7. Data Pipeline Transform/Process Parse Filter Split/merge route Data Store Data Pipeline 7 BI tool Target data store Load/Ingest/Move Application Extra/Acquire Data Data Pipeline Data Pipeline Diverse sources Diverse targets
8. Simplistic Data Flow 8 Data Store/Application B Acquire/Ingest Data Process and Analyzed Data Data Store/Application A • Data movement as flow • Moving data content from A to B
9. Many Flow-like Data in a Real World 9 Across Organizations/ Business unit/ Geographic locations
10. https://mattturck.com/landscape/mad2024.pdf https://mad.firstmark.com/ The 2024 MAD (Machine Learning, AI & Data) Landscape
11. medium @milkdd datafold blog.bytebytego.com Linkedin Post
12. https://a16z.com/emerging-architectures-for-modern-data-infrastructure/ We will talk this later
13. What often happen in a complex Data pipeline https://unsplash.com/photos/chrome-plated-industrial-background-equipment-industrial-tools-and-machiner y-for-the-production-in-factory-shops-dairy-factory-steel-water-pipeline-chrome-pipes-modern-factory-interi or-production-line-maze-of-metal-pipes-background-NLsF62mB1oc
14. Unreliable? Questionable? Quality issues?
15. Pipeline debt Technical debt in data pipeline Down with pipeline debt / introducing Great Expectations. https://greatexpectations.io/blog/down-with-pipeline-debt-introducing-great-expectations • Undocumented/ unmanaged • Untested • Unstable
16. Designing Data-Intensive Applications Designing Data-Intensive Applications, 2nd Edition by Martin Kleppmann, Chris Riccomini, O'Reilly, 2025 Reliable Scalable Maintainable The ability of a system to perform its required functions consistently over time without failure. • Data Integrity and Consistency • Fault Tolerance and Error Handling • Recoverability and Disaster Recovery • Monitoring, Alerting, and Observability • Security and Compliance • Modular, Reusable, and Evolvable Design • Standardization and Best Practices • Simplicity and Ease of Understanding • Configurability and Operability • Comprehensive Documentation and Knowledge Sharing • Version Control and Collaboration Practices • Automated Testing and Validation • Handling Increased Load and Concurrency • Performance Optimization • Dynamic Scaling Strategies and Resource Management • Elasticity and Automated Scaling • Reliability, Fault Tolerance, and Automation The ease with which a data pipeline can be understood, modified, extended, and troubleshooted over its lifecycle. The system's ability to handle increasing amounts of data, higher processing loads, and more complex transformations efficiently and effectively.
17. Reliability + Scalability + Maintainability in SmartPizza https://unsplash.com/photos/pizza-in-oven-IfQlwNqJqV8
18. • Variable Load • Resource Allocation • Complex Codebase • Deployment Difficulties • Lack of Modular System • Lack of Visibility • Inefficient Debugging • Order Processing Failures • Data Inconsistency • Schema Changes • Single Points of Failure Real-Time Order Processing and Delivery Optimization Orders Request Real-time delivery estimates ETL/ELT pipelines Data store/target Order Processing Delivery Optimization Challenges
20. dbt (data build tool) Transform data using the same practices that software engineers use to build applications. • Centralized • Version Control • Documentation • Modularity • Open-Source https://www.getdbt.com/product/what-is-dbt https://github.com/dbt-labs/dbt-core What is dbt?
21. model write model layer model Run Build dbt models: dbt model =a single .sql file dbt code = SQL + Jinja • SQL select statement dbt model reference each other • Creates Natural dependencies • dbt determine model execution order 1 command • V • Create DAG • Parallel execution
22. https://docs.getdbt.com/docs/collaborate/explore-projects dbt doc
23. https://docs.getdbt.com/docs/collaborate/explore-projects model lineage graph
24. Ensure Reliability + Scalability + Maintainability in Data pipeline https://unsplash.com/photos/person-writing-bucket-list-on-book-RLw-UC03Gwc
25. Solutions Provided by dbt Modular Data Transformations Data Testing and Validation Reliable Scalable Maintainable • Modular SQL Models • Refactoring Support • Built-in Testing Framework • Schema Tests • Custom Tests Version Control Integration • Git Integration • Collaborative Development Documentation and Data Lineage • Auto-Generated Documentation • Data Lineage Graphs Handling Schema Changes • Incremental Models • Macro Support Performance Optimization • Compiled SQL • Materializations
26. Ensure Reliability in Data https://unsplash.com/photos/person-writing-bucket-list-on-book-RLw-UC03Gwc
27. 6 Dimensions of Data Quality The degree to which data correctly reflects the real world object/ event Accuracy Expected comprehensiveness/ are all datasets and the data items recorded Completeness Data across all systems reflects the same information and are in synch with each other across the data stores Consistency Information is available when it is expected and needed Timeliness Means that there’s only one instance of the information appearing in a database Uniqueness Refers to information that doesn’t conform to a specific format or doesn’t follow business rules Validity https://kamal-ahmed.github.io/DQ-Dimensions.github.io/ https://hub.getdbt.com/infinitelambda/dq_tools/latest/
28. How do we test?
29. Test data content • Built-in ◦ Singular data tests ◦ Generic data tests • Packages ◦ dbt_utils ◦ dbt_expectation ◦ dbt_elementary ◦ … Test data schema • dbt (model) contract Test data code • dbt unit test • Recce
30. Assumption about Data Assertion about Code ref: Webinar on: Testing frameworks in dbt. (2023) Two Types of Testing DEV/TEST env CI Code Data Input Output data Code freezed, data changed Code Data Input Output data Prod env Data freezed, code changed Validating the code that processes data before deployed to prod. Validating the data as it's loaded into production. ETL code ◦ pytest,... model code ◦ dbt unit testing ◦ Recce data content: ◦ pydantic ◦ great expectations ◦ dbt test ◦ dbt_utils/expectatio ns/elementary… data schemas: ◦ dbt data contracts
31. dbt run on Kubernetes https://unsplash.com/photos/opened-window-panel-uH2J-RqAChI
32. DataStore Data Data Data Data DataStore Data Data Data Data ETL/ELT pipelines configMap CronJob schedule: “0 ****” Database Data warehouse dbt run starts Run query (SQL) dbt run on Kubernetes • Scalability • High Availability • Resource Optimization • Automation • Monitoring and Logging
33. Modern data platform using dbt in GCP https://www.analytics8.com/blog/best-in-breed-data-stack-platform-bigquery-dbt-and-looker/ Landing Staging Warehouse Mart
34. Modern data platform using dbt AWS https://aws.amazon.com/tw/blogs/big-data/create-a-modern-data-platform-using-the-data-build-tool-dbt-in-the-aws-cloud/
35. dbt Cloud on Microsoft Fabric https://www.getdbt.com/blog/dbt-cloud-on-microsoft-fabric
36. dbt + cube https://cube.dev/docs/guides/dbt semantic layer
37. Scalability of SmartPizza https://unsplash.com/photos/baked-pizza-in-oven-x5jilo3ck3o Key Takeaways and Next
38. The Analytics Development Lifecycle (ADLC) https://www.getdbt.com/resources/guides/the-analytics-development-lifecycle An integrated, iterative process Ingest/collect Store Process Output Source
39. • Data Pipelines are more and more complex ◦ More Challenges (SmartPizza) ▪ Reliability ▪ Scalability ▪ Maintainability • dbt + Kubernetes ◦ Embrace Modular Architecture ◦ Implement Robust Data and Code Testing ◦ Self-healing and replication for fault-tolerant ◦ Automate Deployment Processes ◦ Utilize Auto-Scaling Features • More ◦ Orchestrating dbt Workflows ◦ Enhance Observability ◦ Secure Data and Applications ◦ Promote Collaboration and Version Control Key Takeaways and Tips
40. Thanks!