Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PyConDE 2016 - Building Data Pipelines with P...
Search
Miguel Cabrera
October 31, 2016
Technology
0
320
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Tweet
Share
More Decks by Miguel Cabrera
See All by Miguel Cabrera
Machine Learning for Time Series Forecasting
mfcabrera
0
280
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
120
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
160
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
1.2k
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
190
Python and Life Hacking with Emacs
mfcabrera
2
330
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
1.9k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
280
Dictionary Learning for Music Genre Recognition
mfcabrera
0
250
Other Decks in Technology
See All in Technology
Claude Codeが働くAI中心の業務システム構築の挑戦―AIエージェント中心の働き方を目指して
os1ma
9
1.5k
Findy Freelance 利用シーン別AI活用例
ness
0
300
VLMサービスを用いた請求書データ化検証 / SaaSxML_Session_1
sansan_randd
0
220
解消したはずが…技術と人間のエラーが交錯する恐怖体験
lamaglama39
0
190
マルチプロダクト×マルチテナントを支えるモジュラモノリスを中心としたアソビューのアーキテクチャ
disc99
0
300
隙間時間で爆速開発! Claude Code × Vibe Coding で作るマニュアル自動生成サービス
akitomonam
3
250
Claude Codeから我々が学ぶべきこと
s4yuba
8
2k
GMOペパボのデータ基盤とデータ活用の現在地 / Current State of GMO Pepabo's Data Infrastructure and Data Utilization
zaimy
3
200
Claude CodeでKiroの仕様駆動開発を実現させるには...
gotalab555
3
890
専門分化が進む分業下でもユーザーが本当に欲しかったものを追求するプロダクトマネジメント/Focus on real user needs despite deep specialization and division of labor
moriyuya
0
1k
LLM開発を支えるエヌビディアの生成AIエコシステム
acceleratedmu3n
0
370
「Roblox」の開発環境とその効率化 ~DAU9700万人超の巨大プラットフォームの開発 事始め~
keitatanji
0
100
Featured
See All Featured
Git: the NoSQL Database
bkeepers
PRO
431
65k
How GitHub (no longer) Works
holman
314
140k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
790
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.6k
Facilitating Awesome Meetings
lara
54
6.5k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
139
34k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
Scaling GitHub
holman
461
140k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Faster Mobile Websites
deanohume
308
31k
Transcript
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera
[email protected]
Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello”
[email protected]
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight