Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
[NYJavaSig] Riding The Distributed Streams
Search
Viktor Gamov
February 03, 2017
Technology
1
190
[NYJavaSig] Riding The Distributed Streams
Presentation on Hazelcast and Distributed Streams.
Presented on NYJavaSig
Viktor Gamov
February 03, 2017
Tweet
Share
More Decks by Viktor Gamov
See All by Viktor Gamov
Processing Streaming Data with KSQL
vikgamov
4
370
[VirtualJUG] Apache Kafka — A Streaming Data Platform
vikgamov
3
370
[SF JUG] Apache Kafka — A Streaming Data Platform
vikgamov
4
78
[OracleCode NYC-2018] Apache Kafka A Streaming Data Platform
vikgamov
1
160
[OracleCode NYC-2018] Rethinking Stream Processing with KStreams and KSQL
vikgamov
2
220
[JBreak-2018] Это кто там твитить про #jbreak?
vikgamov
0
200
[DevNexus-2018] Apache Kafka A Streaming Data Platform
vikgamov
2
270
[DataSciCon] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
100
[Philly JUG] Divide, Distribute and Conquer: Stream v. Batch
vikgamov
0
470
Other Decks in Technology
See All in Technology
Redmineの意外と知らない便利機能 (Redmine 6.0対応版)
vividtone
0
810
ロール・ツール群の開発 / Development of Roles and Tools
ks91
PRO
0
110
金融システムをモダナイズするためのAmazon Elastic Kubernetes Service(EKS)ノウハウ大全
daitak
0
110
オープンソースのハードウェアのコンテストに参加している話
iotengineer22
0
270
名刺メーカーDevグループ 紹介資料
sansan33
PRO
0
730
変化に強いテーブル設計の勘所 / Table design that is resistant to changes
soudai
PRO
50
13k
KMP導⼊において、マネジャーとして考えた事
sansantech
PRO
1
180
SmartHRの複数のチームにおけるMCPサーバーの活用事例と課題
yukisnow1823
2
920
会社紹介資料 / Sansan Company Profile
sansan33
PRO
6
360k
マップを速く表示するために
tsuboyan5
0
170
Contract One Dev Group 紹介資料
sansan33
PRO
0
5.8k
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
1
7.1k
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
81
9k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.4k
Typedesign – Prime Four
hannesfritz
41
2.6k
A Tale of Four Properties
chriscoyier
159
23k
Six Lessons from altMBA
skipperchong
28
3.8k
Agile that works and the tools we love
rasmusluckow
329
21k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
32
5.8k
Fontdeck: Realign not Redesign
paulrobertlloyd
84
5.5k
Reflections from 52 weeks, 52 projects
jeffersonlam
349
20k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
34
2.3k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Transcript
None
> whoami • Solutions Architect @Hazelcast • Hang out with
awesome people • @gamussa in internetz Please, follow me in Twitter I’m very interesting ©
Agenda • Refreshing knowledge on Java 8 Streams • Distribute
and Conquer • Distributed Data • Distributed Streams • How we did all this
Java 8 Streams
Java 8 Streams… • An abstraction represents a sequence of
elements • Is not a data structure • Convey elements from a source through a pipeline of operations • Operation doesn’t modify a source
Why I should care about Stream API? • You’re Java
developer
What does regular Java developer think about Scala? advanced
Why I should care about Stream API? • You’re Java
developer • Many Java developers know Java • It’s all about data processing
java.util.stream operations • map(), flatMap(), filter() • reduce(), collect() •
sorted()
None
None
None
Problem • One does not simply put all Big Data
in one machine
Problem • Data doesn’t fit just one machine
Problem • One does not simply put all Big Data
in one machine • Data is too important to have it only one machine
None
CACHES
Replication on Sharding? http://book.mixu.net/distsys/single-page.html
Solution • Use Distributed Map aka IMap
What’s Hazelcast IMDG? • In-memory Data Grid • Apache v2
Licensed • Distributed • Caches (IMap, JCache) • Java Collections (IList, ISet, IQueue) • Messaging (Topic, RingBuffer) • Computation (ExecutorService, M-R)
None
None
None
Green Primary Green Backup Green Shard
None
Problem • Lambda serialization 26
27
Solution • serializable version of the interfaces • Introducing DistributedStream
28
29
None
31 Jet Streams
None
What’s Hazelcast Jet? • General purpose distributed data processing framework
• Based on Direct Acyclic Graph to model data flow • Built on top of Hazelcast IMDG • Comparable to Apache Spark or Apache Flink 33
None
DAG 35
Job Execution 36
None
Future (It’s bright!) • Memory module for processing big data
• Higher level streaming and batching APIs • Reactive Streams • Distributed Classloading • Integrations (HDFS/Yarn/Mesos)
Your fuel, our Jet Engine • Public release – Feb
7th. • Developer Preview today - yay! • http://hazelcast.org/jet-signup • Send me a note
[email protected]
• Follow @hazelcast and @gamussa (duh!!) • Your questions #hazelcast #hazelcastjet
Conclusion • Java Stream API provides very white range of
data processing tools • War And Piece – is a Big (a lot of data) Book! • Now we’re pretty sure that Andrew and Pierre are the main characters
None