Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
To Stream or Not To Stream? The Landscape of On...
Search
Gianmarco De Francisci Morales
May 28, 2015
Research
1
320
To Stream or Not To Stream? The Landscape of Online Analytics
EVAM Solution Day 2015, Istanbul.
Gianmarco De Francisci Morales
May 28, 2015
Tweet
Share
More Decks by Gianmarco De Francisci Morales
See All by Gianmarco De Francisci Morales
Repurpose, Reuse, Recycle the building blocks of Machine Learning
gdfm
0
54
How I Learned to Stop Worrying and Love the Risk
gdfm
0
120
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
120
Controversy on Social Media: Collective Attention, Echo Chambers, and Price of Bipartisanship
gdfm
0
220
Quantifying and Reducing Controversy in Social Media
gdfm
0
140
Big Data Streams: The Next Frontier
gdfm
2
290
Mining Big Data Streams: Better Algorithms or Faster Systems?
gdfm
0
450
SAMOA @Strata Barcelona 2014
gdfm
2
650
Distributed Adaptive Model Rules for Mining Big Data Streams
gdfm
1
150
Other Decks in Research
See All in Research
Weekly AI Agents News!
masatoto
22
18k
Generative Predictive Model for Autonomous Driving 第61回 コンピュータビジョン勉強会@関東 (後編)
kentosasaki
0
160
SSII2024 [OS2] GPT-4Vで画像認識は終わるのか(オープニング)
ssii
PRO
0
760
WikipediaやYouTubeにおける論文参照 / joss2024
corgies
1
240
SSII2024 [OS3] 基盤モデル(オープニング)
ssii
PRO
0
320
SSII2024 [TS3] 画像認識におけるマルチモーダル基盤モデル ~基盤モデル、あなたのタスクに役立つかも?~
ssii
PRO
0
1.1k
SSII2024 [OS1] 研究紹介100連発(オープンニング)
ssii
PRO
0
470
SSII2024 [OS1] 現場の課題を解決する ロボットラーニング
ssii
PRO
0
520
CVPR2024論文紹介:Segmentation
hinako0123
0
120
Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences
sgk
0
250
RSJ2024「基盤モデルの実ロボット応用」チュートリアルA(河原塚)
haraduka
2
520
SSII2024 [OS2] 画像、その先へ 〜モーション解析への誘い〜
ssii
PRO
1
1.2k
Featured
See All Featured
GraphQLの誤解/rethinking-graphql
sonatard
65
9.8k
No one is an island. Learnings from fostering a developers community.
thoeni
18
2.9k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
190
16k
The Cost Of JavaScript in 2023
addyosmani
42
5.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.2k
Code Review Best Practice
trishagee
62
16k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
24
610
Web Components: a chance to create the future
zenorocha
309
42k
Art, The Web, and Tiny UX
lynnandtonic
294
20k
Fashionably flexible responsive web design (full day workshop)
malarkey
401
65k
Gamification - CAS2011
davidbonilla
79
5k
Teambox: Starting and Learning
jrom
131
8.7k
Transcript
To Stream or Not To Stream? The Landscape of Online
Analytics Gianmarco De Francisci Morales
[email protected]
@gdfm7
None
5 Questions What? Why? How? Where? When?
What? Data Stream
Text Big Data Too big to handle
Text Big Data Streams Drinking from a firehose
Stream Analytics Batch data = snapshot of streaming data Descriptive
Predictive Prescriptive
Value of Data
Online vs Real-Time
Why? Motivation and Goal
–Jay Kreps, Confluent founder (ex-LinkedIn) “Most of what happens inside
a company is some new information comes in and the company reacts to that asynchronously.” Asynchronous Processing
Nervous System vs Silos
Perishable Insights Great instantaneous value Ephemeral Opportunity cost
Hype Cycle
Hype Cycle
How? Stream Processing Architecture
Architecture Overview
Ingestion Plethora of solutions Still ad-hoc (read: messy) Schema evolution:
Avro Column-store: Parquet Log collection: Flume
Brokerage
Processing PE PE Input Stream PEI PEI PEI PEI PEI
Output Stream Event routing
Output Stream: Kafka Further processing View: Key-Value Store Applications Reactive
callbacks
Example: Reactive Web App
Lambda vs Kappa
Where? Applications
Application Domains Industrial applications Telecommunications and networks Web applications Internet
of Things
Predictive Maintenance
Text Search
Machine Learning SA SAMOA%
Anomaly Detection
When? Adoption Risks
– Gartner, 2015 “Despite considerable hype and reported successes for
early adopters, 54% of survey respondents report no plans to invest at this time, while only 18% have plans to invest in Hadoop over the next 2 years.” 5 Years Early
Cost Not an issue Cheap hardware Cloud-based solutions Amazon Kinesis,
MSFT Azure Stream Analytics Open source
Ease Inherently harder Ops best practices not ironed out (yet)
Lack of skills, training, and support Rethink applications from scratch
Actionable Insights? Define what you want Moving target Garbage in,
garbage out
Conclusions Who?
5 Answers What? Why? How? Where? When? Stream analytics Perishable
insights Asynchronous processing Everywhere In 5 years
Text Slow Fish or Fast Fish Which fish will you
be?
Thanks! 37 https://samoa.incubator.apache.org @ApacheSAMOA @gdfm7
[email protected]