Message-Driven Architecture is the foundation for building reactive microservices (responsive, resilient, elastic).
See why in this presentation and how to do it on AWS.
rights reserved. Amazon Confidential and Trademark Jerome Van Der Linden (@jeromevdl) Message-Driven Architecture with AWS Worldwide Software Architecture Summit 26th of January 2021
rights reserved. Amazon Confidential and Trademark Who am I ? Jérôme Van Der Linden Solutions Architect @ AWS (Geneva, Switzerland) Former consultant, developer, tech lead, agile & devops coach, architect @ OCTO Technology Husband and Dad of 3 @jeromevdl
rights reserved. Amazon Confidential and Trademark Agenda Monolith to Microservices Microservices to Reactive Microservices Message-Driven Architecture on AWS Wrap-up
rights reserved. Amazon Confidential and Trademark Monoliths Do not throw the baby out with the bath water… - Simple to develop and refactor - Simple to test (integration, end-to-end) - Single deployment process - Single application to monitor - Consistency https://martinfowler.com/bliki/MonolithFirst.html
rights reserved. Amazon Confidential and Trademark Monoliths … but definitely some limits! - Strong coupling: less flexibility / agility - High impact of change - Redeploy everything - Risks of blast radius - Reliability - One failure can bring down the whole application - Rigid scaling - Often stateful - Limited to the database scalability - Development lifecycle is typically slow - Continuous deployment is difficult - Barrier to adopting new technologies
rights reserved. Amazon Confidential and Trademark Microservices Microservices to the rescue… - Reduced coupling - Smaller impact of changes - Independent deployments - Better reliability - Independent scaling - Smaller unit of development: - Humanely understandable - More suitable for adopting new technologies
rights reserved. Amazon Confidential and Trademark Microservices … but it is not that magic! - Reduced coupling - Smaller impact of changes - Independent deployments - Better reliability - Independent scaling - Smaller unit of development: - Humanely understandable - More suitable for adopting new technologies Are you sure?
rights reserved. Amazon Confidential and Trademark Synchronous communica=on Order Service Invoice Service 500 internal server error 503 service unavailable ☹
rights reserved. Amazon Confidential and Trademark Synchronous communica=on What’s the problem ? - No isolation between services è still coupled - Risk of timeout or cascading failures è still unreliable - What if the signature of invoice API change ? è still dependent
rights reserved. Amazon Confidential and Trademark Message-Driven (Asynchronous) Communica=on Isolation between services: - Less risk of timeout è more responsive - No more cascading failure è more resilient è more responsive - What if the signature of invoice API change ? è No problem !
rights reserved. Amazon Confidential and Trademark Amazon Simple Queue Service (SQS) Producer SQS Queue Decouple producers from consumers. Messages polled by consumers Amazon SQS Fully managed Message Queuing service Consumers AWS Lambda Container Instance / VM Standard • At-least once delivery • Best-effort ordering Messages are stored up to 14 days.
rights reserved. Amazon Confidential and Trademark Amazon Simple Queue Service (SQS) Producer Consumers AWS Lambda Amazon SQS Fully managed Message Queuing service SQS Queue Decouple producers from consumers Long Polling Container Instance / VM
rights reserved. Amazon Confidential and Trademark Amazon Simple Queue Service (SQS) Producer Consumers AWS Lambda Amazon SQS Fully managed Message Queuing service SQS Queue Decouple producers from consumers Messages polled by consumers Container Instance / VM Dead-letter Queue If a message can’t be processed successfully it can be held in a queue
rights reserved. Amazon Confidential and Trademark Common paIern with SQS: Storage-first Amazon API Gateway Amazon SQS Queue Dead letter Queue AWS Lambda Client …
rights reserved. Amazon Confidential and Trademark Amazon Simple No=fica=on Service (SNS) Fan Out Publisher Amazon SNS Fully managed publish & subscribe messaging service SNS Topic Decouple message publisher from subscribers with topics Subscribers AWS Lambda Amazon SQS HTTP/S Email / SMS Amazon Kinesis Data Firehose
rights reserved. Amazon Confidential and Trademark Amazon Simple No=fica=on Service (SNS) Publisher Amazon SNS Fully managed publish & subscribe messaging service SNS Topic Decouple message publisher from subscribers with topics Message Filtering Filter messages according to subscription filter policies Dead-letter Queue If a subscriber is unavailable, messages can be held in a queue Subscribers AWS Lambda Amazon SQS HTTP/S Email / SMS Amazon Kinesis Data Firehose
rights reserved. Amazon Confidential and Trademark Common paIern with SNS & SQS: Topic-Queue Chaining Amazon SQS Queue A Backend A Publisher Amazon SNS Topic Amazon SQS Queue B Backend B Amazon SQS Queue C Backend C
rights reserved. Amazon Confidential and Trademark Amazon EventBridge Event Sources Amazon EventBridge Fully managed event bus to connect applications Event Bus Targets (20+) AWS Lambda Amazon SQS AWS Services Custom events SaaS events Default event bus Custom event bus SaaS event bus Rules Amazon SNS AWS Step Functions Amazon Kinesis AWS Batch Amazon API Gateway Amazon EC2 Instance Amazon ECS Task Dead-letter Queue If a target is unavailable, events can be held in a queue
rights reserved. Amazon Confidential and Trademark Services Comparison Amazon SQS Amazon SNS Amazon EventBridge Type Queue (Pull) Publish / Subscribe (Push) Publish / Subscribe (Push) Persistence Yes (Up to 14 days, default: 4 days) No (retry up to 23 days + DLQ) No (retry up to 24h + DLQ + Archives) Ordering FIFO FIFO No Delivery Standard: at least once FIFO: exactly once Standard: at least once FIFO: exactly once At least once Filtering Yes Yes No Consumers Millions 5 per rule / Target services Lambda (+ compute & SDK) 2 (+ HTTP/SMS/Email/Push) 20+ Throughput * Price * Standard: Almost unlimited FIFO: 3 000 TPS (soft) Standard: 30 000 TPS (soft) FIFO: 300 TPS (soft) 10 000 TPS (soft) $ per million API requests Standard ≤ 0.4 / FIFO ≤ 0.5 $ per million API requests: 0.5 $ per million events: 1 * depends on the region (us-east-1)
rights reserved. Amazon Confidential and Trademark Conclusion Message-Driven architectures drive resilience and elas3city è responsiveness Asynchronous Events Improve responsiveness and reduce dependencies Event Stores Buffer messages until services are available to process Event Routers Abstract producers and consumers from each other
rights reserved. Amazon Confidential and Trademark • Building event-driven applications with Amazon EventBridge • https://virtual.awsevents.com/media/1_w2mhd582 • Scalable serverless event-driven architectures with SNS, SQS & Lambda • https://virtual.awsevents.com/media/1_o5hp8egy • Decoupling serverless workloads with Amazon EventBridge • https://virtual.awsevents.com/media/1_upe6mtew • Choosing the right event-routing service for serverless: EventBridge, SNS, or SQS • https://lumigo.io/blog/choosing-the-right-event-routing-on-aws-eventbridge-sns-or-sqs