Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Base Lab Base Lab
May 07, 2015
Technology
110
0

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
140
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
120
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
310
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
250
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
160
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
490
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
250

Other Decks in Technology

See All in Technology
AI バイブコーティングでキーボード不要？！
Avatar for Sam Akada samakada
0
640
「誰一人取り残されない」 AIエージェント時代のプロダクト設計思想 Product Management Summit 2026
Avatar for mizushimac mizushimac
1
1.8k
AIが書いたコードを信じられない問題 〜レビュー負荷を下げるために変えたこと〜 / The AI Code Trust Gap: Reducing the Review Burden
Avatar for 株式会社ビットキー / Bitkey Inc. bitkey
PRO
8
1.4k
生成AI時代のドキュメントに対する期待の整理と実践から得た学び / Rethinking Documentation for LLM: Lessons from Practice
Avatar for 株式会社ビットキー / Bitkey Inc. bitkey
PRO
1
100
運用システムにおけるデータ活用とPlatform
Avatar for SansanTech sansantech
PRO
0
130
"おまじない"を卒業する　ボイラープレート再入門
Avatar for shun shunsuke_1b
1
110
Practical TypeProf: Lessons from Analyzing Optcarrot
Avatar for Yusuke Endoh mame
0
1.2k
Do Vibe Coding ao LLM em Produção para Busca Agêntica - TDC 2026 - Summit IA - São Paulo
Avatar for Jessica Pauli de C Bonson jpbonson
3
160
AI活用時代の事業判断高度化を導くエンジニアリング基盤 / 20260424 Atsushi Funahashi
Avatar for SHIFT EVOLVE shift_evolve
PRO
2
100
AWS Transform CustomでIaCコードを自由自在に変換しよう
Avatar for k-kun duelist2020jp
0
160
Chasing Real-Time Observability for CRuby
Avatar for White-Green whitegreen
0
280
Claude Code を安全に使おう勉強会 / Claude Code Security Basics
Avatar for MasahiroKawahara masahirokawahara
12
38k

Featured

See All Featured
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
370
20k
How To Speak Unicorn (iThemes Webinar)
Avatar for Michelle Schulp Hunt marktimemedia
1
440
How GitHub (no longer) Works
Avatar for Zach Holman holman
316
150k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.2k
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
3.3k
The AI Search Optimization Roadmap by Aleyda Solis
Avatar for Aleyda Solis aleyda
1
5.7k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
110
Claude Code のすすめ
Avatar for schroneko schroneko
67
220k
Utilizing Notion as your number one productivity tool
Avatar for Mfonobong Umondia mfonobong
4
290
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
46
8.1k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
82
6.2k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.2k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com