Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
PyConDE 2016 - Building Data Pipelines with P...
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Miguel Cabrera
October 31, 2016
Technology
380
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
PyConDE 2016 - Building Data Pipelines with Python
Miguel Cabrera
October 31, 2016
Transcript
49 slides
Building Data Pipelines with Python Data Engineer @ TY
@mfcabrera
[email protected]
Miguel Cabrera PyCon Deutschland 30.10.2016
Agenda
Agenda Context Data Pipelines with Luigi Tips and
Tricks Examples
Data Processing Pipelines
cat file.txt | wc -‐ l | mail -‐s
“hello”
[email protected]
ETL
ETL • Extract data from a data source •
Transform the data • Load into a sink
None
Feature Extraction Parameter Estimation Model Training Feature Extraction
Model Predict Visualize/ Format
Steps in different technologies
Steps can be run in parallel
Steps have complex dependencies among them
Workflows • Repeat • Parametrize •
Resume • Schedule it
None
None
“A Python framework for data flow definition and execution” Luigi
Concepts
Concepts Tasks Parameters Targets Scheduler & Workers
Tasks
None
1
2
3
4
WordCountTask file.txt wc.txt
WordCountTask file.txt wc.txt ToJsonTask wc.json
None
Parameters
None
Parameters Used to idenNfy the task From arguments
or from configuraNon Many types of Parameters (int, date, boolean, date range, Nme delta, dict, enum)
Targets
Targets Resources produced by a Task Typically Local files
or files distributed file system (HDFS) Must implement the method exists() Many targets available
None
Scheduler & Workers
None
Source: h@p:/ /www.arashrouhani.com/luigid-‐basics-‐jun-‐2015
BaVeries Included
Batteries Included Package contrib filled with goodies Good support
for Hadoop Different Targets Extensible
Task Types Task -‐ Local Hadoop MR, Pig, Spark,
etc SalesForce, ElasNcsearch, etc. ExternalProgram check luigi.contrib !
Target LocalTarget HDFS, S3, FTP, SSH, WebHDFS, etc.
ESTarget, MySQLTarget, MSQL, Hive, SQLAlchemy, etc.
None
Tips & Tricks
Separate pipeline and logic
Extend to avoid boilerplate code
DRY
Conclusion Luigi is a mature, baVeries-‐included alternaNve for building
data pipelines Lacks of powerful visualizaNon of the pipelines Requires a external way of launching jobs (i.e. cron). Hard to debug MR Jobs
Lear More hVps:/ /github.com/spoNfy/luigi hVp:/ /luigi.readthedocs.io/en/stable/
Thanks!
Credits • pipe icon by Oliviu Stoian from the Noun
Project • Photo Credit: (CC) h@ps:/ /www.flickr.com/photos/ 47244853@N03/29988510886 from hb.s via Compfight • Concrete Mixer: (CC) h@ps:/ /www.flickr.com/photos/ 145708285@N03/30138453986 by MasLabor via Compfight
More Decks by Miguel Cabrera
See All by Miguel Cabrera
From Days to Minutes: How We Taught an AI to Onboard 50+ Tenants on our AI Features
mfcabrera
0
330
Machine Learning for Time Series Forecasting
mfcabrera
0
410
Data Science in Fashion - Exploring Demand Forecasting
mfcabrera
0
190
Helping Travellers Make Better Hotel Choices 500 Million Times a Month
mfcabrera
1
250
Europython 2016 - Things I wish I knew before using Python for Data Processing
mfcabrera
1
1.4k
PyData Berlin Meetup Nov 2015 - (Some of the) things I wish I knew before starting using Python for Data Science
mfcabrera
0
280
Python and Life Hacking with Emacs
mfcabrera
2
420
PyData Berlin 2015 - Processing Hotel Reviews with Python
mfcabrera
4
2.2k
Munich Datageeks - Introduction to SVM using Python
mfcabrera
2
390
Other Decks in Technology
See All in Technology
E2E Test というハーネス — AI時代の自動テスト民主化
daishukojima
10
5.9k
IoTのデータ保存をSREの観点で考える
grimoh
1
110
市販IoT機器や自作IoT機器とHomeAssistantを駆使して家の電気代をできる限り見える化する方法の考案
harutiro
0
410
現地速報! Microsoft Ignite アップデートレポート
kasada
0
390
AI-Ready なデータ基盤ってなに? & 新卒未経験が、いま考えていること
imuta
0
1.1k
MCPでネットワーク障害調査を自動化する前に直すべき3つのこと
yama3133
1
650
コピーオンライト型ファイルシステムと ランダム書き込み
sat
PRO
2
130
OCI Artifact による自動運転タクシーに向けた OTA 基盤の構築
biwashi
1
180
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
620
専門家に使ってもらうAIプロダクトをつくる - 病院の医事課向けDPCコーディング支援システムの設計と評価
hagino3000
4
970
パスワードのない鍵で、 データを手元に置く
ginjihayashi
0
390
AWS DevOps Agentの構造から読み解く運用 AI エージェントの設計原則/aws-devops-agent-design-principles
tomoki10
0
190
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
360
31k
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
techseoconnect
PRO
0
760
Optimising Largest Contentful Paint
csswizardry
37
4k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
659
62k
A Soul's Torment
seathinner
9
3.9k
AI Search: Where Are We & What Can We Do About It?
aleyda
0
8.3k
Bootstrapping a Software Product
garrettdimon
PRO
306
120k
What's in a price? How to price your products and services
michaelherold
247
13k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
18k
Intergalactic Javascript Robots from Outer Space
tanoku
273
28k
It's Worth the Effort
3n
188
29k
The SEO Collaboration Effect
kristinabergwall1
1
620