Jiaqi Liu - Building a Data Pipeline with Testing in Mind

It’s one thing to build a robust data pipeline process in python but a whole other challenge to find tooling and build out the framework that allows for testing a data process. In order to truly iterate and develop a codebase, one has to be able to confidently test during the development process and monitor the production system.

In this talk, I hope to address the key components for building out end to end testing for data pipelines by borrowing concepts from how we test python web services. Just like how we want to check for healthy status codes from our API responses, we want to be able to check that a pipeline is working as expected given the correct inputs. We’ll talk about key features that allows a data pipeline to be easily testable and how to identify timeseries metrics that can be used to monitor the health of a data pipeline.

https://us.pycon.org/2018/schedule/presentation/161/

Transcript

1. Building a Data Pipeline with Testing in Mind Jiaqi Liu, Software Engineer at Button Director, @WomenWhoCodeNYC @jiaqicodes
2. usebutton.com
4. Agenda • Data Pipelines • Challenges with Testing Data Pipelines • Designing Features: • Well Defined Schemas • Dry Run Mode • Storing Metadata • Testing, Monitoring & Alerting
5. Data Pipeline
6. ETL Pipeline Extract data from a source, this could be scraping from a site, a large file, a realtime stream of data feeds. Transform the data - this could be joining the data with additional information for an enhanced data set, running through a machine learning model, or aggregating the data in some way. Load the data into a data warehouse or a User facing dashboard - wherever the end storage and display for data might be.
7. Data Pipeline Example
13. PyCon Convert HTML to structured data Write to file Select Random Talk Write to File
15. Batch Periodic Process that reads data in bulk (typically from a filesystem or a database) Stream High throughput, low latency system that reads data from a stream or a queue
16. Apache Airflow Luigi Google Dataflow Pachyderm
17. What Could Go Wrong?
18. Problems • Batch Job is never scheduled • Batch Job takes too long to run • Data is malformed or corrupt • Data is lost • Stream is backed-up, Stream data is lost • Non-deterministic models
19. Batch Jobs
20. Confidential Stream Jobs
21. Data is exposed or lost or malformed. A statistical model is producing highly inaccurate results Data Integrity Speed in Data Processing could be Core to Business Delayed Processing Data Pipeline Concerns
22. It’s not enough to know that the pipeline is healthy, you also have to know that the data being processed is accurate.
23. Build data pipelines that support interpretability and observability
24. Interpretability Not just understand what a model predicted but also why. Allows for debugging and auditing machine learning models.
25. Because Button is a marketplace, we see the side effects of user behavior in our data and have to decipher what assumptions are safe to make.
26. Observability Cindy Sridharan - https://medium.com/@copyconstruct/monitoring-and-observability-8417d1952e1c
27. Mystery Free Prod
28. Features Build feature to support Interpretability and observability
29. Features to Include • Well Defined Schemas • Capturing Metadata about the Pipeline • Having a Test Run Feature
30. Schema
32. Best Practices Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing structured data.
34. Metadata
35. Test in Dry Run Mode
36. Confidential Monitoring & Testing
37. Testing
38. Test Pyramid https://martinfowler.com/articles/practical-test-pyramid.html
39. System Unit Test Functional Regression Different Types of Tests
40. Unit Tests
41. Functional Tests • Can also be known as Integration Tests • In the case of data, it’s the golden tests • Sets the gold standard for data in and data out • Doesn’t need to be logic specific like Unit Tests are • Build a Golden Test framework and define fixtures (expected input, and expected output) Failed Gold Test
42. Regression Tests https://www.ibeta.com/regression-testing-nutshell/
43. 93% Model A Precision 95% Model B Precision Champion/Challenger Model
44. Black Box Measures Health of Whole System Continuous Relies on Other Signals System Tests
45. Monitoring
46. Confidential Monitoring Tools
48. Time Series Metrics
49. Alerting
50. Monitoring with Time Series Data
51. Best Practices
52. Best Practices Set a threshold that works for you.
 Establish a baseline and go from there.
53. Thanks! We’re hiring! www.usebutton.com jiaqi@usebutton.com