Cloud Native Streaming and Event-Driven Microservices

Talk given at Spring One Platform 2016, August 1-4, 2016, Las Vegas

An introduction to Spring Cloud Stream, it's design and new SCSt 1.1 features.

Transcript

1. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud Native Event-Driven and Streaming Microservices By Marius Bogoevici, Staff Engineer, Pivotal @mariusbogoevici
2. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement • The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation. 2
3. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Processing large quantities of data in real-time • Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’ • Specific requirements • High throughput, low latency • Grouping, ordering, windowing • Transition in messaging middleware design: • Traditional (JMS-based, RabbitMQ) • To log-style: Kafka, Amazon Kinesis, Google Pub/Sub, Azure Event Hubs 3
4. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Resource management • memory, CPU, instance count • Scaling up/down • Health monitoring and failover • Routing and load balancing • Applications and deployment strategies designed to target of all above 4 Apache YARN Apache Mesos Kubernetes
5. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Typically known benefits and drawbacks of microservices: • bounded context • development agility • added complexity • Failure isolation • Process-specific resource tuning: • memory • CPU • instance count • network 5
6. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES • Emphasize messaging as communication over HTTP • Decoupling • Discoverability • Availability • Eventual consistency across heterogenous resources • alternative to distributed transactions • Strong similarities between: • data streaming - ‘big data’: ingestion, analytics • event streaming - async interaction, event sourcing, etc 6
7. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: The Origin Story 7 Spring Cloud Stream 2015 Spring XD Spring Cloud Data Flow Spring Cloud Task
8. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions 8
9. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream in a nutshell 9 Application Core Spring Boot Spring Integration Spring Messaging Reactive APIs inputs outputs
10. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 10
11. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11
12. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Binder Abstraction 12 Application Messaging Middleware Binder channel can be input or output adapts channel to target middleware created by framework configured via Spring Boot properties
13. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 13 @EnableBinding + Binder Implementation Apache Kafka JMS Google PubSub Production-ready: Experimental
14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 14 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(Processor.INPUT)
 @SendTo(Processor.OUTPUT)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pipes and filters model 15 http averages topN ingest data average value over last 5s hottest sensors over last 10 seconds
16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 16 http raw-sensor-data averages averages top-n Calculator
17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 17 http raw-sensor-data averages averages top-n Calculator Failure detector in Spring Cloud Stream, all destinations are pub-sub
18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data and event streaming - conceptually similar 18
19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 19 raw-sensor-data averages HDFS http pubsub is great for interconnecting logical apps what if we add more instances ?
20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 20 raw-sensor-data averages HDFS http we want these two to compete averages HDFS and these two !
21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 21
22. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 22 Averages HDDS HDFS HDFS raw-sensor-data groups are in a pub-sub relationship to each other Averages Averages consumers are competing within a group
23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 23
24. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 24 Average calculator Average calculator http raw-sensor-data
25. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 25 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data
26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 26 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data
27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Simple topologies : (relatively) easy to deploy … 27 http HDFS spring.cloud.stream.bindings.output.destination=httphdfs.1 spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs httphdfs.1
28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ … but how about complex topologies ? 28 http raw-sensor-data averages top-n Calculator Failure detector averages averages HDFS HDFS HDFS
29. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases 29
30. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ In a single sentence … 30 dataflow:> stream create demo --definition “http | file”
31. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - Stream DSL 31 Stream definition Boot Apps built on top of Spring Ecosystem httpfile = http | file |
32. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - deployment SPI • SPI for deploying applications • Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon • Different resources supported • Spring Boot Uberjars • Docker Images • Different locations • File system, HDFS, HTTP, Maven, Docker Hub 32
33. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow deployment 33 Pla$orm
34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream 1.1 34
35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream 1.1 • Under development, currently on master • 1.1.0.M1 release August 2016 • 1.1.0.RELEASE - early Q4 2016 • Based on Spring Boot 1.4 • Release train model • binders in separate repositories • New features • Kafka 0.9 support (also intending 0.10) • Reactive programming support • Schema Registry support 35
36. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model - imperative/procedural 36 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
37. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reactive/functional Programming via Reactor 37 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener
 @Output(“output”)
 public Flux<String> process(@Input(“input”) Flux<String> s) {
 return f.map(s -> s.toUppercase()); }
 }
38. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 38
39. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Handling multiple returns (or none at all) 39 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener
 @Output(“output”)
 public Flux<String> process(@Input(“input”) Flux<String> s) {
 return f.filter(s->startsWith(“a”)).map(s -> s.toUppercase()); }
 }
40. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stateful operators: grouping, windowing 40 SpringBootApplication
 @EnableBinding(Processor.class)
 public class WordCountApplication {
 
 @StreamListener
 @Output("output")
 public Flux<?> countWords(@Input("input") Flux<String> words) {
 return words.window(ofSeconds(5), ofSeconds(1))
 .flatMap(window -> window.groupBy(word -> word)
 .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)
 .map(count -> new WordCount(group.key(), count))));
 } }
41. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Content-type transformations (since 1.0) • Users can specify the intended content type of an output value, e.g. “application/ json” • Outgoing messages are converted to that content type • A message header value carries the content type • The content type can be used by inputs to convert to expected types (e.g. `@StreamListener` method arguments • Extensible model, just requires the registration of AbstractMessageConverter beans 41
42. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema management and Avro support • Newly introduced Avro MessageConverters • Static schema-based resources (classpath, filesystem, etc) • Schema Registry Client-based • Schema registry server • Not Spring Cloud Stream specific • Allows storing different formats and versions • Schema evolution support via Schema Registry Client converter 42
43. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Heterogenous data sources/consumers 43 sensor v1 sensor v2 average-temperature v1 sensor data average-pressure v2
44. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema evolution in Spring Cloud Stream 1.1 44
45. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema evolution in Spring Cloud Stream 1.1 45
46. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Schema Registry Server - embeddable in apps 46 @SpringBootApplication @EnableSchemaRegistryServer public class SchemaRegistryServerApplication { public static void main(String[] args) { SpringApplication .run(SchemaRegistryServerApplication.class,args); } }
47. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pluggable schema registry client 47 @EnableBinding(Sink.class)
 @EnableAutoConfiguration
 @EnableSchemaRegistryClient
 public static class VoteCounterApplication {
 
 @StreamListener(Sink.INPUT)
 public void listen(Vote vote) {
 voteService.count(vote); }
 }
48. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Future plans 48
49. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Future plans • Additional binders • JMS • Google Pub Sub • Kinesis • Fully reactive pipelines • KStream support • First class support for stateful processing 49
50. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Fully reactive pipelines (planned) • Use natively reactive middleware clients for reactive support • e.g. Reactive Kafka client 50 public interface ReactiveProcessor { @Input(Processor.INPUT)
 Flux<?> output();
 
 @Output(Processor.OUTPUT)
 Flux<?> output();
 
 } @EnableBinding(ReactiveProcessor.class)

51. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Kafka Stream Support (planned) 51 @StreamListener
 @Output(“output”)
 public KStream<Integer,String> addByKey(@Input(“input”) KStream<Integer,String> input) {
 return input.map((i,s) -> new KeyValue<>(i, s.toUpperCase())) .reduceByKey((String value1, String value2) -> value1 + value2, TimeWindows.of("windowName", 1000)) .toStream() }

52. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Related talks 52 Reactive Kafka Rajini Sivaram Thursday, August 4, 9:00 AM - 10:05 AM Orchestrate All the Things! with Spring Cloud Data Flow Eric Bottard, Ilayaperumal Gopinathan Thursday, August 4 11:10 AM - 12:15 PM
53. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Learn More. Stay Connected. Tell your friends! <Related Session> @springcentral spring.io/blog @pivotal pivotal.io/blog @pivotalcf http://engineering.pivotal.io