Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Base Lab Base Lab
May 07, 2015
Technology
110
0

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
140
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
120
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
310
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
250
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
150
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
490
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
240

Other Decks in Technology

See All in Technology
AI時代に新卒採用、はじめました/junior-engineer-never-die
Avatar for dmnlk dmnlk
0
130
OPENLOGI Company Profile for engineer
Avatar for OPENLOGI hr01
1
62k
BIツール「Omni」の紹介 @Snowflake中部UG
Avatar for Sagara sagara
0
190
出版記念イベントin大阪「書籍紹介＆私がよく使うMCPサーバー3選と社内で安全に活用する方法」
Avatar for KintoTech_Dev kintotechdev
0
150
AgentCore RuntimeからS3 Filesをマウントしてみる
Avatar for Har1101 har1101
2
270
AWSで2番目にリリースされたサービスについてお話しします(諸説あります)
Avatar for Yuuki Yamashita yama3133
0
120
不確実性と戦いながら見積もりを作成するプロセス/mitsumori-process
Avatar for hiro@miraito hirodragon112
1
190
TanStack Start エコシステムの現在地 / TanStack Start Ecosystem 2026
Avatar for Takahiro Ikeuchi iktakahiro
1
310
Strands Agents × Amazon Bedrock AgentCoreで パーソナルAIエージェントを作ろう
Avatar for yoko / Naoki Yokomachi yokomachi
2
160
Claude Teamプランの選定と、できること/できないこと
Avatar for ふくだ（fukuda ryu） rfdnxbro
1
720
OCI技術資料 : ロード・バランサ 概要 - FLB・NLB共通
Avatar for Oracle Cloud Infrastructure ソリューション・エンジニア ocise
4
27k
あるアーキテクチャ決定と その結果/architecture-decision-and-its-result
Avatar for Ryo Tomidokoro hanhan1978
1
370

Featured

See All Featured
Max Prin - Stacking Signals: How International SEO Comes Together (And Falls Apart)
Avatar for Tech SEO Connect techseoconnect
PRO
0
140
Abbi's Birthday
Avatar for Violet Bock coloredviolet
2
6.3k
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
Avatar for David Neal reverentgeek
1
410
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
46
8.1k
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
0
6.4k
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
3
310
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
Avatar for UX Y'all uxyall
0
990
Visual Storytelling: How to be a Superhuman Communicator
Avatar for David Neal reverentgeek
2
490
Marketing Yourself as an Engineer | Alaka | Gurzu
Avatar for Gurzu gurzu
0
170
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
310
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
170

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com