Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
UIの境界線をデザインする | React Tokyo #15 メイントーク
Avatar for SASAPIYO sasagar
2
380
Terraform言語の静的解析 / static analysis of Terraform language
Avatar for Kazuma Watanabe wata727
1
110
ついに来た！本格的なマルチクラウド時代の Google Cloud
Avatar for maroon1st maroon1st
0
220
Vibe NLP for Applied NLP
Avatar for Ines Montani inesmontani
PRO
0
450
ローカルで稼働する AI エージェントを超えて / beyond-local-ai-agents
Avatar for geeawa gawa
3
280
How We Benchmarked Quarkus: Patterns and anti-patterns
Avatar for Holly Cummins hollycummins
1
150
Surviving Black Friday: 329 billion requests with Falcon!
Avatar for Samuel Williams ioquatix
0
730
GoogleCloudとterraform完全に理解した
Avatar for Terisuke terisuke
1
140
JAWS-UG横浜 #100 祝・第100回スペシャル AWS は VPC レスの時代へ
Avatar for maroon1st maroon1st
0
170
mruby on C#: From VM Implementation to Game Scripting (RubyKaigi 2026)
Avatar for hadashiA hadashia
2
590
Programming with a DJ Controller — not vibe coding
Avatar for seki at druby.org m_seki
3
140
NakouPAY説明用
Avatar for Annouim0 annouim0
0
250

Featured

See All Featured
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
61
43k
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
170
sira's awesome portfolio website redesign presentation
Avatar for ElsiraPls elsirapls
0
220
A Tale of Four Properties
Avatar for Chris Coyier chriscoyier
163
24k
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
460
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
Avatar for AKADEMIYA2063 akademiya2063
PRO
1
96
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
0
33k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
28
2.6k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
55
3.3k
Code Review Best Practice
Avatar for Trisha Gee trishagee
74
20k
Side Projects
Avatar for Sacha Greif sachag
455
43k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]