Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
130
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
480
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
230

Other Decks in Technology

See All in Technology
Function calling機能をPLaMo2に実装するには / PFN LLMセミナー
Avatar for Preferred Networks pfn
PRO
0
710
非エンジニアのあなたもできる＆もうやってる！コンテキストエンジニアリング
Avatar for ファインディ株式会社（イベント資料アップ用） findy_eventslides
3
840
自作LLM Native GORM Pluginで実現する AI Agentバックテスト基盤構築
Avatar for po3rin po3rin
2
210
北海道の人に知ってもらいたいGISスポット / gis-spot-in-hokkaido-2025
Avatar for sakaik sakaik
0
190
コンテキストエンジニアリングとは? 考え方と応用方法
Avatar for ファインディ株式会社（イベント資料アップ用） findy_eventslides
4
820
AI Agentと MCP Serverで実現する iOSアプリの 自動テスト作成の効率化
Avatar for スパイダープラス株式会社 spiderplus_cb
0
290
組織観点から IAM Identity CenterとIAMの設計を考える
Avatar for NRI Netcom nrinetcom
PRO
1
130
Pure Goで体験するWasmの未来
Avatar for asuka askua
1
150
Geospatialの世界最前線を探る [2025年版]
Avatar for Yasunori Kirimoto dayjournal
2
440
“2件同時配達”の開発舞台裏 〜出前館PMが挑んだ ダブルピック実現に向けた体験設計〜
Avatar for 株式会社出前館 demaecan
0
160
BtoBプロダクト開発の深層
Avatar for 16bit_idol 16bitidol
0
140
データエンジニアがこの先生きのこるには...?
Avatar for 10xinc 10xinc
0
410

Featured

See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
52
5.6k
KATA
Avatar for Megan Lloyd mclloyd
32
14k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
29
2.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.5k
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
269
13k
Embracing the Ebb and Flow
Avatar for Simon Collison colly
88
4.8k
A Modern Web Designer's Workflow
Avatar for Chris Coyier chriscoyier
697
190k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
431
66k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
15k
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.4k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
303
21k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com