La Kopi & WDA4 - Developing a data pipeline on cloud

Developing a data pipeline on cloud
By Fon, Kamolphan Liwprasert

Presented at La Kopi Cloud event up by Google Developer Group (GDG) on Jul 21st, 2021.

WDA4: https://youtu.be/jtwDGufzFxo
La Kopi: https://youtu.be/J48mHBh5DyY

Transcript

1. For analytics / ML projects Developing a data pipeline on cloud Fon Liwprasert ML Engineer at Sertis
2. Kamolphan Liwprasert (Fon) Machine Learning Engineer, Sertis I’m an ML engineering enthusiast with data engineering background. I love to develop solutions on cloud. 6 x GCP certified.
3. Every data analytics or ML project needs a good data pipeline as a foundation.
4. But the data is always not ready to use at the beginning.
5. We also need to think about automation and scalability as the project goes on.
6. Developing a data pipeline on cloud Specifically on Google Cloud Platform
7. Text & diagram slides Agenda • Why do we need data pipeline? • What are options on cloud? Storages / Computes / Pipelines • Introducing Apache Airflow • Reference Architecture + Demo code
8. Why do we need data pipeline? 🛠
9. Purposes of data pipeline Ingest data from multiple sources Transform or clean the data to ensure data quality Automation of the process
10. What are options on cloud? ☁
11. Choosing Storage Options
12. BigQuery Choosing Storage Options: zoom-in For analytics workload Cloud Bigtable Cloud Storage
13. Choosing Compute Options Google Compute Engine Google Kubernetes Engine Google Cloud Run Google App Engine Google Cloud Function IAAS : Infrastructure as a service CAAS : Container as a service PAAS : Platform as a service FAAS : Function as a service GCE GKE GAE GCF Virtual Machine Managed K8s cluster Serverless container Serverless application Serverless function platform
14. Choosing Data Processing Options Cloud Dataproc Processing data Workflow and scheduler Cloud Scheduler Cloud Dataflow Cloud Composer Spark or Hadoop Data processing Unified pipeline w/ Apache Beam 3 2 1 Cloud Functions or or Cloud Run Serverless options Cloud Workflows (optional)
15. Option 1: Low-cost & severless option Processing data (light workload) Cloud Functions Cloud Run (Or Pub/Sub) for Cloud Functions Workflow and scheduler REST API Scheduler ✓ Severless: easy & fast ✓ Low-cost solution ✓ Suitable for light workload
16. Option 2: Big data solution Spark or Hadoop Data processing Unified data pipeline with Apache Beam Processing data (Big data workload) Workflow and scheduler Cloud Dataproc REST API Cloud Dataflow Scheduler ✓ Big data framework: Spark, Apache Beam, Flink ✓ Scalability and reliability ✓ Opensource solutions
17. Option 3: Cloud Composer (Airflow) Cloud Composer Kubernetes Engine Cloud SQL Managed service + ✓ Easier for maintenance ✓ Scalability and reliability ✓ Suitable for large number of jobs that require workers
18. Introducing Apache Airflow
19. Why Apache Airflow? • Popular open-source project for ETL or data pipeline orchestration. • All codes are in Python. Easy to learn and use. • Can be run locally as well for development environments.
20. Airflow is rich in 3rd party integration https://github.com/apache/airflo w/tree/main/airflow/providers . . . https://registry.astronomer.io/providers/
21. Apache Airflow UI DAGs (Job list) Calendar View Tree View
22. Apache Airflow basic components Sensor Wait on an event i.e. poking for a file Operator Running an action; PythonOperator Hook Interface to external services or system
23. Reference Architecture and Demo Code 📐📄
24. Reference Architecture Batch Ingestion BigQuery Cloud Storage SFTP server Cloud Composer SFTPToGCSOperator GCSToBigQueryOperator Analytics Workload BI Dashboards SFTPSensor
25. Text & diagram slides DAG overview SFTPSensor SFTPToGCSOperator GCSToBigQueryOperator Check if a file is available Upload that file to GCS Load file from GCS to BigQuery
26. Demo Time! bit.ly/airflow_gcp_demo
27. Text & diagram slides Simple data pipeline using Airflow (1) Import and initialize DAG bit.ly/airflow_gcp_demo ← DAG name ← Schedule ← Import necessary components
28. Text & diagram slides Simple data pipeline using Airflow (2) SFTPSensor for waiting for a file to be available bit.ly/airflow_gcp_demo ← SFTP Connection ← Path to file wait_for_file check-for-file SFTPSensor
29. Text & diagram slides Simple data pipeline using Airflow (3) SFTPToGCSOperator for uploading file(s) from SFTP to GCS bit.ly/airflow_gcp_demo ← GCP Connection ← Path to File & GCS upload_file_to_gcs upload-file-from-sftp SFTPToGCSOperator
30. Simple data pipeline using Airflow (4) GCSToBigQueryOperator for loading data to BigQuery from GCS source file(s) bit.ly/airflow_gcp_demo ← source GCS ← JSON schema (dictionary) ← format: CSV, AVRO, parquet load_to_bigquery load-to-bigquery GCSToBigQueryOperator
31. Text & diagram slides Simple data pipeline using Airflow (5) Put everything together! wait_for_file check-for-file upload_file_to_gcs upload-file-from-sftp load_to_bigquery load-to-bigquery SFTPSensor SFTPToGCSOperator GCSToBigQueryOperator bit.ly/airflow_gcp_demo ← Create DAG!
32. Text & diagram slides DAG graph view (from previous code)
33. Text & diagram slides Key Takeaways 🔑 Choosing the right solution for data pipeline depends on requirements and workload.
34. Text & diagram slides Thank you 😃 Let’s connect! Fon Liwprasert ML Engineer at Sertis linkedin.com/in/fonylew fonylew