Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
[NYJavaSig] Riding The Distributed Streams
Search
Viktor Gamov
February 03, 2017
Technology
1
190
[NYJavaSig] Riding The Distributed Streams
Presentation on Hazelcast and Distributed Streams.
Presented on NYJavaSig
Viktor Gamov
February 03, 2017
Tweet
Share
More Decks by Viktor Gamov
See All by Viktor Gamov
Processing Streaming Data with KSQL
vikgamov
4
330
[VirtualJUG] Apache Kafka — A Streaming Data Platform
vikgamov
3
330
[SF JUG] Apache Kafka — A Streaming Data Platform
vikgamov
4
72
[OracleCode NYC-2018] Apache Kafka A Streaming Data Platform
vikgamov
1
150
[OracleCode NYC-2018] Rethinking Stream Processing with KStreams and KSQL
vikgamov
2
210
[JBreak-2018] Это кто там твитить про #jbreak?
vikgamov
0
170
[DevNexus-2018] Apache Kafka A Streaming Data Platform
vikgamov
2
240
[DataSciCon] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
94
[Philly JUG] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
440
Other Decks in Technology
See All in Technology
B2B SaaSから見た最近のC#/.NETの進化
sansantech
PRO
0
880
SRE×AIOpsを始めよう!GuardDutyによるお手軽脅威検出
amixedcolor
0
150
Amazon CloudWatch Network Monitor のススメ
yuki_ink
1
210
TypeScript、上達の瞬間
sadnessojisan
46
13k
EventHub Startup CTO of the year 2024 ピッチ資料
eventhub
0
120
飲食店データの分析事例とそれを支えるデータ基盤
kimujun
0
140
インフラとバックエンドとフロントエンドをくまなく調べて遅いアプリを早くした件
tubone24
1
430
アジャイルでの品質の進化 Agile in Motion vol.1/20241118 Hiroyuki Sato
shift_evolve
0
170
Python(PYNQ)がテーマのAMD主催のFPGAコンテストに参加してきた
iotengineer22
0
500
iOSチームとAndroidチームでブランチ運用が違ったので整理してます
sansantech
PRO
0
150
AWS Lambda のトラブルシュートをしていて思うこと
kazzpapa3
2
180
Terraform Stacks入門 #HashiTalks
msato
0
360
Featured
See All Featured
Optimizing for Happiness
mojombo
376
70k
Large-scale JavaScript Application Architecture
addyosmani
510
110k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
How to Ace a Technical Interview
jacobian
276
23k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
44
2.2k
Bootstrapping a Software Product
garrettdimon
PRO
305
110k
How to Think Like a Performance Engineer
csswizardry
20
1.1k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
6.9k
[RailsConf 2023] Rails as a piece of cake
palkan
52
4.9k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
6.8k
Transcript
None
> whoami • Solutions Architect @Hazelcast • Hang out with
awesome people • @gamussa in internetz Please, follow me in Twitter I’m very interesting ©
Agenda • Refreshing knowledge on Java 8 Streams • Distribute
and Conquer • Distributed Data • Distributed Streams • How we did all this
Java 8 Streams
Java 8 Streams… • An abstraction represents a sequence of
elements • Is not a data structure • Convey elements from a source through a pipeline of operations • Operation doesn’t modify a source
Why I should care about Stream API? • You’re Java
developer
What does regular Java developer think about Scala? advanced
Why I should care about Stream API? • You’re Java
developer • Many Java developers know Java • It’s all about data processing
java.util.stream operations • map(), flatMap(), filter() • reduce(), collect() •
sorted()
None
None
None
Problem • One does not simply put all Big Data
in one machine
Problem • Data doesn’t fit just one machine
Problem • One does not simply put all Big Data
in one machine • Data is too important to have it only one machine
None
CACHES
Replication on Sharding? http://book.mixu.net/distsys/single-page.html
Solution • Use Distributed Map aka IMap
What’s Hazelcast IMDG? • In-memory Data Grid • Apache v2
Licensed • Distributed • Caches (IMap, JCache) • Java Collections (IList, ISet, IQueue) • Messaging (Topic, RingBuffer) • Computation (ExecutorService, M-R)
None
None
None
Green Primary Green Backup Green Shard
None
Problem • Lambda serialization 26
27
Solution • serializable version of the interfaces • Introducing DistributedStream
28
29
None
31 Jet Streams
None
What’s Hazelcast Jet? • General purpose distributed data processing framework
• Based on Direct Acyclic Graph to model data flow • Built on top of Hazelcast IMDG • Comparable to Apache Spark or Apache Flink 33
None
DAG 35
Job Execution 36
None
Future (It’s bright!) • Memory module for processing big data
• Higher level streaming and batching APIs • Reactive Streams • Distributed Classloading • Integrations (HDFS/Yarn/Mesos)
Your fuel, our Jet Engine • Public release – Feb
7th. • Developer Preview today - yay! • http://hazelcast.org/jet-signup • Send me a note
[email protected]
• Follow @hazelcast and @gamussa (duh!!) • Your questions #hazelcast #hazelcastjet
Conclusion • Java Stream API provides very white range of
data processing tools • War And Piece – is a Big (a lot of data) Book! • Now we’re pretty sure that Andrew and Pierre are the main characters
None