Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Base Lab Base Lab
May 07, 2015
Technology
120
0

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
150
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
130
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
310
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
260
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
160
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
490
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
250

Other Decks in Technology

See All in Technology
Spring Boot における​ AOT Cache 活用テクニックと​ 起動時間改善事例​
Avatar for NTTドコモソリューションズ　Java担当 ntt_dsol_java
0
170
形式手法特論：公平性制約の位相的特徴づけ #kernelvm / Kernel VM Study Kansai 12th
Avatar for y_taka_23 ytaka23
1
570
Oracle AI Database@Google Cloud：サービス概要のご紹介
Avatar for oracle4engineer oracle4engineer
PRO
6
1.5k
ビジュアルプログラミングIoTLT vol.23
Avatar for 1ft-seabass 1ftseabass
PRO
0
150
GitHub Copilot CLIでWebアクセシビリティを改善した話
Avatar for tomokusaba tomokusaba
0
120
食べログのサーキットブレーカー導入を振り返って
Avatar for atpons atpons
1
150
Platform engineering for developers, architects & the rest of us (AI agents)
Avatar for danielbryantuk danielbryantuk
0
130
Oracle Cloud Infrastructure：2026年5月度サービス・アップデート
Avatar for oracle4engineer oracle4engineer
PRO
1
240
Cloud Run のアップデート 触ってみる＆紹介
Avatar for Gre212 gre212
0
230
Strands Agents超入門
Avatar for KintoTech_Dev kintotechdev
1
140
Fabric-cicd によるAzure DevOps デプロイ
Avatar for Ryoma Nagata ryomaru0825
0
130
Agentic Design Patterns
Avatar for Guillaume Laforge glaforge
0
280

Featured

See All Featured
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
230
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
7.1k
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
9.1k
Digital Ethics as a Driver of Design Innovation
Avatar for Per Axbom axbom
PRO
1
290
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Faster Mobile Websites
Avatar for Dean Hume deanohume
310
31k
Amusing Abliteration
Avatar for ianozsvald ianozsvald
1
190
The Curse of the Amulet
Avatar for Matthew Lei leimatthew05
1
13k
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2.2k
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.2k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
3.2k
How to build a perfect <img>
Avatar for Jono Alderson jonoalderson
1
5.5k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com