Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[NYJavaSig] Riding The Distributed Streams

Viktor Gamov
February 03, 2017

[NYJavaSig] Riding The Distributed Streams

Presentation on Hazelcast and Distributed Streams.
Presented on NYJavaSig

Viktor Gamov

February 03, 2017
Tweet

More Decks by Viktor Gamov

Other Decks in Technology

Transcript

  1. > whoami • Solutions Architect @Hazelcast • Hang out with

    awesome people • @gamussa in internetz Please, follow me in Twitter I’m very interesting ©
  2. Agenda • Refreshing knowledge on Java 8 Streams • Distribute

    and Conquer • Distributed Data • Distributed Streams • How we did all this
  3. Java 8 Streams… • An abstraction represents a sequence of

    elements • Is not a data structure • Convey elements from a source through a pipeline of operations • Operation doesn’t modify a source
  4. Why I should care about Stream API? • You’re Java

    developer • Many Java developers know Java • It’s all about data processing
  5. Problem • One does not simply put all Big Data

    in one machine • Data is too important to have it only one machine
  6. What’s Hazelcast IMDG? • In-memory Data Grid • Apache v2

    Licensed • Distributed • Caches (IMap, JCache) • Java Collections (IList, ISet, IQueue) • Messaging (Topic, RingBuffer) • Computation (ExecutorService, M-R)
  7. 27

  8. 29

  9. What’s Hazelcast Jet? • General purpose distributed data processing framework

    • Based on Direct Acyclic Graph to model data flow • Built on top of Hazelcast IMDG • Comparable to Apache Spark or Apache Flink 33
  10. Future (It’s bright!) • Memory module for processing big data

    • Higher level streaming and batching APIs • Reactive Streams • Distributed Classloading • Integrations (HDFS/Yarn/Mesos)
  11. Your fuel, our Jet Engine • Public release – Feb

    7th. • Developer Preview today - yay! • http://hazelcast.org/jet-signup • Send me a note [email protected] • Follow @hazelcast and @gamussa (duh!!) • Your questions #hazelcast #hazelcastjet
  12. Conclusion • Java Stream API provides very white range of

    data processing tools • War And Piece – is a Big (a lot of data) Book! • Now we’re pretty sure that Andrew and Pierre are the main characters