rights reserved. Building Event Driven Applications on AWS Senior Solutions Architect Builder AWS Scott Gerring Senior Solutions Architect AWS Jerome Van Der Linden A R C 3 0 1
rights reserved. Agenda Introduction Common patterns & services in event-driven architectures Combining patterns: better together Considerations with event-driven architectures
rights reserved. Coupling – integration’s magic word Component A Component B Coupling is a measure of independent variability between connected systems Decoupling has a cost, both at design and runtime Coupling isn’t binary Coupling isn’t one-dimensional
rights reserved. The many facets of coupling • Technology dependency: Java vs. C++ • Location dependency: IP addresses, DNS • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro • Data type dependency: int16, int32, string, UTF-8, null, empty • Semantic dependency: Name, middle name, ZIP • Temporal dependency: sync, async • Interaction style dependency: messaging, RPC, query-style (GraphQL) • Conversation dependency: pagination, caching, retries
rights reserved. The appropriate level of coupling depends on the level of control you have over the endpoints Gregor Hohpe Co-author of “Enterprise Integration Patterns”
rights reserved. Synchronous request-response model Order Service Billing Service Shipping Service Inventory Service Dependencies: temporal (sync), location (host), soft (signature), …
rights reserved. Synchronous request-response model Order Service Billing Service Shipping Service Inventory Service A failure in one of theses services…
rights reserved. Synchronous request-response model Inventory Service ...will impact all of the other services. Shipping Service Billing Service Order Service
rights reserved. Reducing coupling Reducing coupling is the act of reducing interdependency between components and the awareness each component must have of one another. Event-driven architectures help achieve loose coupling through asynchronous communication via events. Order Service Billing Service Shipping Service Inventory Service Event Broker
rights reserved. Key concepts of event-driven architectures E V E N T S • Events are signals that a system’s state has changed • Events occur in the past (e.g., OrderCreated) • Events are immutable (cannot be changed) • Events are observed, not directed. "source": "com.orders", "detail-type": "detail": "metadata": "idempotency-key": "data": "order-id"
rights reserved. Point-to-point messaging • Decreases temporal and location coupling • Resilient to receiver failure • Receiver control consumption rate • Only one receiver can consume each message
rights reserved. Amazon Simple Queue Service (SQS) Producer SQS Queue Decouple producers from consumers. Messages polled by consumers Amazon SQS Fully managed Message Queuing service Consumers AWS Lambda Container Instance / VM Standard • At-least once delivery • Best-effort ordering Messages are stored up to 14 days. See also Amazon MQ (Apache ActiveMQ & RabbitMQ)
rights reserved. Event Streaming • Decreases temporal and location coupling • Resilient to receiver failure • Continuous flow of messages (real-time) • Can be processed individually or together
rights reserved. Amazon Kinesis Data Stream Producer Consumers AWS Lambda Amazon Kinesis Serverless Streaming service Shards Decouple producers from consumers Messages polled by consumers Kinesis Data Analytics Kinesis Data Firehose Container Instance / VM See also Amazon Managed Streaming for Kafka (MSK)
rights reserved. Pub/sub messaging (Topic) • Decreases temporal and location coupling • Push-based • Fan-out: send the same message to N receivers Producer
rights reserved. Eventual consistency • Decentralized data management • If consumers don’t have data, they can request from producers or store they own version • Data may not be perfectly synchronized across the application but will ultimately become consistent • Increase availability and service autonomy • Data replicated across boundaries S T A T E S P L I T A C R O S S S Y S T E M S User API Email API Users Users Data replicated e.g. Change data capture We can operate autonously Changes will come… at some point
rights reserved. Eventual consistency • Asynchrony and user interface • Does the UI need to be instant? • Can you use bidirectional events (websockets)? C U S T O M E R S U S E R E X P E R I E N C E
rights reserved. Idempotency A T L E A S T O N C E D E L I V E R Y A N D D U P L I C A T I O N H A N D L I N G Sender Receiver Message Ack Message Ack Network failure Duplicate message Time-out
rights reserved. Idempotency Definition: • “A message that has the same effect whether it is received once or multiple times.” – “Enterprise Integration Patterns” (Hohpe, Woolf) Best practices: • Use an idempotency token, that uniquely identify a message, generated by the client and not in the payload itself. • Use a persistence layer in the receiver to keep track of already processed message (using the idempotency token as a key). M A K E Y O U R S E R V I C E I D E M P O T E N T
rights reserved. Designing events S P A R S E E V E N T S V S . F U L L S T A T E D E S C R I P T I O N S 123 10:47 AM 456 123 10:47 AM 456. Open $237.51
rights reserved. Designing events • Consumers are more autonomous • More flexibility for filtering/routing with EventBridge BUT • Is it still enough? • Too much? • Backward compatibility • Bigger contract • Maintainability, coupling C O N S I D E R A T I O N S W I T H D E T A I L E D E V E N T S "source": "com.orders", "detail-type": "detail": "metadata": "idempotency-key": "data": ”order": "order-id" ”amount" ”products" ”user": ”user-id" ”first-name" ”lastName" ”address" ”email" Currency ? last-name ? PII
affiliates. All rights reserved. Jerome Van Der Linden [email protected] @jeromevdl jeromevdl Scott Gerring [email protected] @scottgerring scottgerring
jeromevdl
cyberagentdevelopers
lycorptech_jp
calm1205
mhrtech
oracle4engineer
autifyhq
tsukuha
rkaga
kimujun
addyosmani
marcduiker
kevinliebholz
jeffersonlam
brad_frost
sergeychernyshev
jakevdp
lara
bluesmoon
tammielis
jponch
