Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PyConDE 2016 - Building Data Pipelines with P...
Search
Miguel Cabrera
October 31, 2016
Technology
0
320
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Tweet
Share
More Decks by Miguel Cabrera
See All by Miguel Cabrera
Machine Learning for Time Series Forecasting
mfcabrera
0
280
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
120
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
160
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
1.2k
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
190
Python and Life Hacking with Emacs
mfcabrera
2
330
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
1.9k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
270
Dictionary Learning for Music Genre Recognition
mfcabrera
0
250
Other Decks in Technology
See All in Technology
LangChain Interrupt & LangChain Ambassadors meetingレポート
os1ma
2
220
KubeCon + CloudNativeCon Japan 2025 Recap Opening & Choose Your Own Adventureシリーズまとめ
mmmatsuda
0
230
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
2
240
プロダクトエンジニアリング組織への歩み、その現在地 / Our journey to becoming a product engineering organization
hiro_torii
0
140
あなたの声を届けよう! 女性エンジニア登壇の意義とアウトプット実践ガイド #wttjp / Call for Your Voice
kondoyuko
4
500
作曲家がボカロを使うようにPdMはAIを使え
itotaxi
0
390
ネットワーク保護はどう変わるのか?re:Inforce 2025最新アップデート解説
tokushun
0
150
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
shift_evolve
PRO
1
110
生まれ変わった AWS Security Hub (Preview) を紹介 #reInforce_osaka / reInforce New Security Hub
masahirokawahara
0
360
PHPでWebブラウザのレンダリングエンジンを実装する
dip_tech
PRO
0
220
自律的なスケーリング手法FASTにおけるVPoEとしてのアカウンタビリティ / dev-productivity-con-2025
yoshikiiida
0
320
Lazy application authentication with Tailscale
bluehatbrit
0
100
Featured
See All Featured
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
124
52k
Designing for Performance
lara
609
69k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Six Lessons from altMBA
skipperchong
28
3.9k
Site-Speed That Sticks
csswizardry
10
670
Building an army of robots
kneath
306
45k
The Cult of Friendly URLs
andyhume
79
6.5k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Thoughts on Productivity
jonyablonski
69
4.7k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Transcript
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera
[email protected]
Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello”
[email protected]
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight