Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
AIとペアプロして処理時間を97%削減した話 #pyconshizu
Avatar for Kashun Yoshida kashewnuts
1
240
20260313 - Grafana & Friends Taipei #1 - Kubernetes v1.36 的開發雜記：那些困在 Alpha 加護病房太久的 Metrics
Avatar for ChengHao Yang tico88612
0
200
grapheme_strrev関数が採択されました（あと雑感）
Avatar for てきめん tekimen youkidearitai
PRO
1
220
コーディングルールの鮮度を保ちたい / keep-fresh-go-internal-conventions
Avatar for NAGATA Hiroaki handlename
0
200
エージェント開発初心者の僕が エージェントを作った話と今後やりたいこと
Avatar for ta-hasunuma thasu0123
0
250
Docコメントで始める簡単ガードレール
Avatar for Keisuke Ikeda keisukeikeda
1
120
LangChain4jとは一味違うLangChain4j-CDI
Avatar for Kenji Kazumura kazumura
1
180
Agentic AI: Evolution oder Revolution
Avatar for Lars Roewekamp mobilelarson
PRO
0
180
Claude Codeログ基盤の構築
Avatar for giginet giginet
PRO
7
3.3k
モダンOBSプラグイン開発
Avatar for Kaito Udagawa umireon
0
130
API Platformを活用したPHPによる本格的なWeb API開発 / api-platform-book-intro
Avatar for Takashi Kanemoto ttskch
1
140
猫の手も借りたい！ので AIエージェント猫を作って社内に放した話 Claude Code × Container Lambda の Slack Bot "DevNeko"
Avatar for naramomi7 naramomi7
0
260

Featured

See All Featured
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
480
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
Avatar for Aleyda Solis aleyda
1
2k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
85
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
302
51k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
120
Accessibility Awareness
Avatar for Sarah Abderemane sabderemane
0
81
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
67
37k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
79
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
210
24k
<Decoding/> the Language of Devs - We Love SEO 2024
Avatar for Nikki Halliwell nikkihalliwell
1
150
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
A Guide to Academic Writing Using Generative AI - A Workshop
Avatar for Kenji Saito ks91
PRO
0
240

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]