Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PyConDE 2016 - Building Data Pipelines with P...
Search
Miguel Cabrera
October 31, 2016
Technology
0
320
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Tweet
Share
More Decks by Miguel Cabrera
See All by Miguel Cabrera
Machine Learning for Time Series Forecasting
mfcabrera
0
290
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
130
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
170
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
1.2k
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
200
Python and Life Hacking with Emacs
mfcabrera
2
340
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
1.9k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
290
Dictionary Learning for Music Genre Recognition
mfcabrera
0
250
Other Decks in Technology
See All in Technology
Reflections of AI: A Trilogy in Four Parts (GOTO; Copenhagen 2025)
ondfisk
0
110
空間を設計する力を考える / 20251004 Naoki Takahashi
shift_evolve
PRO
4
460
ACA でMAGI システムを社内で展開しようとした話
mappie_kochi
1
310
スタートアップにおけるこれからの「データ整備」
shomaekawa
2
350
20201008_ファインディ_品質意識を育てる役目は人かAIか___2_.pdf
findy_eventslides
2
600
英語は話せません!それでも海外チームと信頼関係を作るため、対話を重ねた2ヶ月間のまなび
niioka_97
0
130
10年の共創が示す、これからの開発者と企業の関係 ~ Crossroad
soracom
PRO
1
710
Vibe Coding Year in Review. From Karpathy to Real-World Agents by Niels Rolland, CEO Paatch
vcoisne
0
120
【Kaigi on Rails 事後勉強会LT】MeはどうしてGirlsに? 私とRubyを繋いだRail(s)
joyfrommasara
0
230
「使い方教えて」「事例教えて」じゃもう遅い! Microsoft 365 Copilot を触り倒そう!
taichinakamura
0
320
自動テストのコストと向き合ってみた
qa
0
220
Shirankedo NOCで見えてきたeduroam/OpenRoaming運用ノウハウと課題 - BAKUCHIKU BANBAN #2
marokiki
0
180
Featured
See All Featured
Principles of Awesome APIs and How to Build Them.
keavy
127
17k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
9
590
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
Side Projects
sachag
455
43k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
32
2.3k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Git: the NoSQL Database
bkeepers
PRO
431
66k
Statistics for Hackers
jakevdp
799
220k
Docker and Python
trallard
46
3.6k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
19
1.2k
Building Better People: How to give real-time feedback that sticks.
wjessup
369
20k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
114
20k
Transcript
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera
[email protected]
Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello”
[email protected]
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight