Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Evolving Sustainable Data Pipelines

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
技術的負債の返済は、AI時代の複利で効く投資 — 経営としての意思決定とその遂行
Avatar for curekoshimizu curekoshimizu
0
1.7k
FreeBSDでZabbixを動かす
Avatar for kenkino kenkino
0
310
世界の中心で、AI（App Intents）をさけぶ ー App Intents中心設計の実践ガイド
Avatar for touyou touyou
0
630
cdk deploy JawsSonic #MARATHONしながらAWSリソースをデプロイしてみよう
Avatar for アキキー | Akihisa Ikeda akihisaikeda
2
140
JPUG勉強会 OSSデータベースの内部構造を理解しよう(第２回)
Avatar for Atsushi Ogawa oga5
0
260
[GoCon2026] When Goroutines Are Not Enough: Runtime Locality in High-Throughput Go
Avatar for Takeru Hayasaka takehaya
6
2.4k
WebAssembly in Android Apps 〜 WASMはJNIの夢を見るか
Avatar for ARIYAMA Keiji keiji
1
110
個人開発基盤をまるごとCloudflareに引っ越して爆速で総合的体験を向上させた話
Avatar for Tsubasa SEKIGUCHI tinykitten
0
200
AgentCore CLI で進化した AWS での AI エージェントの作り方 : 必要な機能を必要な時に
Avatar for Takahiro Kubo icoxfog417
PRO
3
360
Agents on Rails - Rails at Scale 2026
Avatar for Irina Nazarova irinanazarova
0
120
WebMCP Challenge に星空観察アプリで参加した話
Avatar for dario okazaki okajun35
0
180
仕様駆動開発による爆速プロダクト開発 / Bakusoku Spec Driven Development
Avatar for Keisuke Kobayashi kobakei
0
120

Featured

See All Featured
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
710
Easily Structure & Communicate Ideas using Wireframe
Avatar for Afnizar Nur Ghifari afnizarnur
194
17k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.8k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
211
25k
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
Avatar for UX Y'all uxyall
1
3k
Money Talks: Using Revenue to Get Sh*t Done
Avatar for Nikki Halliwell nikkihalliwell
0
500
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.6k
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
350
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
41k
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
Avatar for Aleyda Solis aleyda
1
1.4k
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
57
14k
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
57
7.1k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]