Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[NYJavaSig] Riding The Distributed Streams

Avatar for Viktor Gamov Viktor Gamov
February 03, 2017

[NYJavaSig] Riding The Distributed Streams

Presentation on Hazelcast and Distributed Streams.
Presented on NYJavaSig

Avatar for Viktor Gamov

Viktor Gamov

February 03, 2017
Tweet

More Decks by Viktor Gamov

Other Decks in Technology

Transcript

  1. > whoami • Solutions Architect @Hazelcast • Hang out with

    awesome people • @gamussa in internetz Please, follow me in Twitter I’m very interesting ©
  2. Agenda • Refreshing knowledge on Java 8 Streams • Distribute

    and Conquer • Distributed Data • Distributed Streams • How we did all this
  3. Java 8 Streams… • An abstraction represents a sequence of

    elements • Is not a data structure • Convey elements from a source through a pipeline of operations • Operation doesn’t modify a source
  4. Why I should care about Stream API? • You’re Java

    developer • Many Java developers know Java • It’s all about data processing
  5. Problem • One does not simply put all Big Data

    in one machine • Data is too important to have it only one machine
  6. What’s Hazelcast IMDG? • In-memory Data Grid • Apache v2

    Licensed • Distributed • Caches (IMap, JCache) • Java Collections (IList, ISet, IQueue) • Messaging (Topic, RingBuffer) • Computation (ExecutorService, M-R)
  7. 27

  8. 29

  9. What’s Hazelcast Jet? • General purpose distributed data processing framework

    • Based on Direct Acyclic Graph to model data flow • Built on top of Hazelcast IMDG • Comparable to Apache Spark or Apache Flink 33
  10. Future (It’s bright!) • Memory module for processing big data

    • Higher level streaming and batching APIs • Reactive Streams • Distributed Classloading • Integrations (HDFS/Yarn/Mesos)
  11. Your fuel, our Jet Engine • Public release – Feb

    7th. • Developer Preview today - yay! • http://hazelcast.org/jet-signup • Send me a note [email protected] • Follow @hazelcast and @gamussa (duh!!) • Your questions #hazelcast #hazelcastjet
  12. Conclusion • Java Stream API provides very white range of

    data processing tools • War And Piece – is a Big (a lot of data) Book! • Now we’re pretty sure that Andrew and Pierre are the main characters