Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
130
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
300
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
250
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
150
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
480
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
230

Other Decks in Technology

See All in Technology
Lazy Constant - finalフィールドの遅延初期化
Avatar for Yuichi.Sakuraba skrb
0
220
重厚長大企業で、顧客価値をスケールさせるためのプロダクトづくりとプロダクト開発チームづくりの裏側 / Developers X Summit 2025
Avatar for モンゴル (mongolyy) mongolyy
0
130
第65回コンピュータビジョン勉強会
Avatar for TSUKAMOTO Kenji tsukamotokenji
0
150
ステートレスなLLMでステートフルなAI agentを作る - YAPC::Fukuoka 2025
Avatar for FUJI Goro gfx
8
1.3k
Flutterで実装する実践的な攻撃対策とセキュリティ向上
Avatar for FujiKinaga fujikinaga
2
450
Capitole du Libre 2025 - Keynote - Cloud du Coeur
Avatar for Julien Briault ju_hnny5
0
110
アジャイル社内普及ご近所さんマップを作ろう / Let's create an agile neighborhood map
Avatar for Satoshi Harada psj59129
1
130
AIと共に開発する時代の組織、プロセス設計 freeeでの実践から見えてきたこと
Avatar for freee freee
4
730
明日から真似してOk！NOT A HOTELで実践している入社手続きの自動化
Avatar for kajinari nkajihara
1
800
なぜインフラコードのモジュール化は難しいのか - アプリケーションコードとの本質的な違いから考える
Avatar for Gosuke Miyashita mizzy
55
17k
Progressive Deliveryで支える！スケールする衛星コンステレーションの地上システム運用 / Ground Station Operation for Scalable Satellite Constellation by Progressive Delivery
Avatar for iselegant iselegant
1
190
Service Monitoring Platformについて
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
270

Featured

See All Featured
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.7k
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
PRO
55
12k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.3k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
27k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
9
970
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
697
190k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.2k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
48
9.8k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
10
670
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
285
14k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com