Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
1

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
React Native에 현대적인 빌드 도구 더하기: Rollipop 구현기
Avatar for Geunhyeok LEE leegeunhyeok
0
150
우리의 Nitro Modules는 당신의 Turbo Modules보다 아름답다
Avatar for HyunWoo Lee l2hyunwoo
0
220
Domain-Driven Transformation
Avatar for Henning Schwentner hschwentner
2
2.3k
Jetpack Compose Mechanisms
Avatar for Jaewoong skydoves
3
870
Kotlinの15年とServer-Side Kotlin
Avatar for Takehata Naoto n_takehata
1
170
[2026-10-09]一枚の布で作らなくてもいい。 信頼性は「パッチワーク」で紡がれる～既存の運用と新しい可能性をつなぎ合わせてきたCREの話～
Avatar for tosite tosite
0
110
phperだってパンチカードでコーディングしたい
Avatar for hideki kinjyo o0h
PRO
0
460
作って学ぶSpring BootのAuto Configuration / JJUG Night Seminar 2026-10-19
Avatar for Ryo Shindo rshindo
0
210
sbt 2.0: takeoff
Avatar for Eugene Yokota eed3si9n
0
170
全ての永続トークンを、生まれる前に消し去りたい！
Avatar for sylph01 sylph01
2
1.4k
学びを記事・スライド・Xへ高速展開する― AI-assisted Learning Loopで回すコンテンツ生成術
Avatar for Ochtum ochtum
0
150
S3 TablesのIcebergデータに、AIエージェントからアクセスしてみた / 20261016developersio2026joetsu-accessing-iceberg-in-s3tables-via-agent
Avatar for kasacchiful kasacchiful
PRO
0
210

Featured

See All Featured
State of Search Keynote: SEO is Dead Long Live SEO
Avatar for Ryan Jones ryanjones
0
330
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
3k
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
9.3k
Design in an AI World
Avatar for tapps tapps
1
360
Designing for Performance
Avatar for Lara Hogan lara
612
70k
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
540
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
699
190k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.6k
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
580
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
2
550
Claude Code どこまでも/ Claude Code Everywhere
Avatar for nwiizo nwiizo
68
59k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
310

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]