Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Build event-driven architectures effortlessly with AWS

Lalit Kale
September 25, 2021

Build event-driven architectures effortlessly with AWS

Build event-driven architectures effortlessly with AWS
Session Abstract:
Many customers are choosing to build event-driven application architectures – those in which subscriber or target services automatically perform work in response to events triggered by publisher or source services. Event driven architectures enable development teams to operate more independently so they can release new features faster, while also making their applications more scalable. In this session, we will discuss how we can build event driven architectures with AWS Lambda and AWS Eventbridge along with widely used patterns.

About Speaker:
Lalit Kale is a Principal Engineer at Verizon Connect. Lalit has 16 years of experience in the software industry. His interests include large scale distributed systems and scalability of platforms.

He is a published author of a book on Microservices ("Building Microservices with .NET Core") published by Packt. He has been recognized as an AWS Community Builder by AWS for his community contributions. You can follow him on LinkedIn (https://www.linkedin.com/in/lalitkale/)

Recorded Session: https://www.youtube.com/watch?v=tDwZLiENDSM

Lalit Kale

September 25, 2021
Tweet

Other Decks in Programming

Transcript

  1. Build Event-Driven Architectures
    with AWS
    Lalit Kale
    Principal Software Engineer - Verizon Connect
    Effortlessly

    View full-size slide

  2. Lalit Kale
    ● Principal Software Engineer, Verizon Connect
    ● Dublin, Ireland
    ● First AWS Use: 2006
    https://www.linkedin.com/in/lalitkale/
    @atechiethought

    View full-size slide

  3. Agenda
    ● What is Event Driven Architectures
    ● AWS Serverless Application Integration Services
    ○ AWS Lambda
    ○ Selecting Right Messaging service: SQS, SNS and Eventridge
    ○ Application Integration Patterns support
    ● Amazon EventBridge
    ● Use Cases

    View full-size slide

  4. Current Landscape
    Large Systems (couple of years ago)
    ● Few Kb of internet speed
    ● Couple of giant single core
    servers
    ● Seconds of response time
    ● Hours of offline maintenance
    ● Data size in Gigabytes
    Large Systems (Today & Future)
    ● GBs scale internet speed
    ● Thousands of multicore servers
    ● Milliseconds of response time
    ● Users expect 100% uptime
    ● Data size in Petabytes and
    Zetabytes

    View full-size slide

  5. Reactive Systems

    View full-size slide

  6. The big idea is "messaging"
    - Alan Kay
    src: https://wiki.c2.com/?AlanKayOnMessaging

    View full-size slide

  7. "If your application is cloud-native or, large
    scale or, distributed, and doesn't include a
    messaging component, that is probably a
    bug."
    - Tim Bray
    Former Sr. Principal Engineer, AWS
    src: https://wiki.c2.com/?AlanKayOnMessaging

    View full-size slide

  8. Communication
    Synchronous Communication Asynchronous Communication
    Face To Face Video
    Conferencing
    Email

    View full-size slide

  9. Types of Communication
    Synchronous Communication Asynchronous Communication

    View full-size slide

  10. Types of Messages
    Commands
    ● "I Intent to …"
    ● Can be rejected by receiver
    ● Represented as verbs in
    present tense, e.g.
    "create_customer"
    ● Source system is aware of
    side effects that command
    processing will have
    Events
    ● Something has happened
    ● Immutable—cannot change
    the past
    ● Represented as verbs in the
    past tense, e.g.
    “customer_created”
    ● Source system has no
    expectations on how an event
    is processed

    View full-size slide

  11. Event Driven Architecture
    Event driven architecture is a distributed asynchronous integration pattern to create
    highly scalable and loosely coupled reactive applications
    Events drives the data processing

    View full-size slide

  12. Advantages of Event driven architecture
    ● Loose coupling between components/services
    ● Ability to scale individual components
    ● Processing components can be developed independent of each other
    ● Fault Tolerance and better resiliency
    ● Improved Availability
    ● Evolutionary architecture
    ● Business Agility

    View full-size slide

  13. Challenges with Event Driven Architectures
    ● Consistency Model mindshift - Eventual Consistency
    ● Observability of Event Pipelines
    ● Immutable Consumers
    ● Error Handling
    ● Event Replay Scenarios

    View full-size slide

  14. Challenges with Event Driven Architectures

    View full-size slide

  15. Event Driven Architecture
    Event driven architecture is a distributed asynchronous integration pattern to
    create highly scalable and loosely coupled reactive applications

    View full-size slide

  16. Enterprise Integration Patterns
    ● Point to Point channel
    ● Pub-Sub pattern
    ● Message Router
    ● Event Bus

    View full-size slide

  17. Event Driven Architectures with AWS Serverless

    View full-size slide

  18. Serverless Story is incomplete without...

    View full-size slide

  19. AWS Lambda Execution Models

    View full-size slide

  20. AWS Lambda Destinations

    View full-size slide

  21. AWS Serverless Application Integration Services

    View full-size slide

  22. Service Selection Criteria
    Scale &
    Concurrency
    Controls
    Durability Persistence
    Consumption Retries Pricing

    View full-size slide

  23. AWS Serverless Application Integration Services
    Amazon SQS
    Durable and scalable
    Buffer and Batching
    support
    Comprehensive security
    point to point channel
    Amazon SNS
    Performance at Scale
    Enterprise Ready,
    Deliver to endpoints like
    email, SMS, Https, Push
    Notifications
    Fan-Out, Pub-Sub
    Amazon EventBridge
    Serverless event bus for AWS Services,
    your own apps and SaaS providers
    Event Bus

    View full-size slide

  24. Failures and Retries
    Amazon SQS Amazon SNS Amazon EventBridge
    Delete Message. Redrive
    policy for failed messages.
    Retry policy for different
    protocols
    Event Retention and Replay
    Retry upto max. 14 days Retry upto 23 days Retry upto 24 hours
    Dead Letter Queue Dead Letter Queue Dead Letter Queue
    Exactly once with FIFO Exactly Once with FIFO
    Topic

    View full-size slide

  25. Amazon EventBridge
    ● Serverless, Pay as you go
    ● 90+ AWS Services as sources
    ● 17+ AWS services as targets
    ● Native integrations with SaaS Providers
    ● $1 per million events put into the bus
    ● No additional cost of delivery
    ● Cross Account, Cross Region

    View full-size slide

  26. Amazon EventBridge

    View full-size slide

  27. Amazon EventBridge - Events and Rules
    {
    "version": "0",
    "id": "3e3c153a-8339-4e30-8c35-687ebef853fe",
    "detail-type": "EC2 Instance Launch
    Successful",
    "source": "aws.autoscaling",
    "account": "123456789012",
    "time": "2015-11-11T21:31:47Z",
    "region": "us-east-1",
    "resources": [],
    "detail": {
    "eventVersion": "",
    "responseElements": null
    }
    }
    Example Event Example Rule
    {
    "source": ["aws.autoscaling"]
    }

    View full-size slide

  28. ● Source of truth for sharing schema
    ● Explicitly published and auto-discovered
    ● Integrations for IDEs like Jetbrains and VS Code
    ● Language Bindings available for Java, Python and Typescript
    Amazon EventBridge - Schema Registry and Discovery

    View full-size slide

  29. Amazon EventBridge - Event Archive and Replay

    View full-size slide

  30. When Amazon EventBridge is not best fit?
    ● Ordering Guarantees
    ● Event Deduplication
    ● Push Delivery
    ● Ultra Low latency
    ● Non-JSON Transport

    View full-size slide

  31. Event Modelling
    CloudEvents - A specification for describing event data in a common way
    ● Consistent schema across all your events
    ● Many Language SDK available - Java, GoLang, Python, Rust etc
    ● Portable Across AWS, GCP, Azure
    ● CNCF Serverless Group supported
    ● https://cloudevents.io/

    View full-size slide

  32. Event Driven Architectures - Infrastructure as Code Support
    ● https://serverlessland.com/
    ● https://cdkpatterns.com
    ● Use patterns to quickly build
    integrations using AWS SAM
    and CDK templates.
    ● 70+ Templates
    ● You can contribute to it!

    View full-size slide

  33. EventBridge - Real Time Threat Mitigation

    View full-size slide

  34. Loan Broker Use case
    src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html

    View full-size slide

  35. ● Receive the consumer's loan quote request
    ● Obtain credit score and history from credit agency
    ● Determine the most appropriate banks to contact
    ● Send a request to each selected bank
    ● Collect responses from each selected bank
    ● Determine the best response
    ● Pass the result back to the consumer
    Loan Broker Use case - Processing Flow
    src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html

    View full-size slide

  36. Loan Broker Use case
    src: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ComposedMessagingExample.html

    View full-size slide

  37. Summary
    ● event driven architectures are essential complexity for large scale decoupled
    applications
    ● Awareness of enterprise integration patterns will ease out the journey through event
    driven architectures
    ● For choosing the aws service for your scenarios, compare scale, durability, persistence
    and consumption models, retries and pricing.
    ● Adopting AWS serverless services like EventBridge ease out complex implementations
    like content filtering and event replays and cross account, cross region scenarios.
    ● You can compose solution by combining multiple patterns and services
    ● Utilize IaC templates available in open source or build your own.

    View full-size slide

  38. Further Reading
    ● http://www.sosp.org/2001/papers/welsh.pdf
    ● https://www.enterpriseintegrationpatterns.com/
    ● https://aws.amazon.com/blogs/compute/introducing-cross-region-event-routing-with-amazon-eventbridge/
    ● https://aws.amazon.com/blogs/compute/building-well-architected-serverless-applications-introduction/
    ● https://serverlessland.com/patterns

    View full-size slide

  39. Amazon EventBridge - Pricing
    AWS Service Events Free
    Custom Events $1/ million events published
    Third Party Events $1/ million events published
    Cross Account Events $1/ million cross-account events sent
    Schema Discovery $0.10 / million events ingested
    Archive Processing $0.12 per GB
    Storage Month $0.0025 per GB

    View full-size slide

  40. SQS, SNS, EventBridge
    Amazon SQS Amazon SNS Amazon EventBridge
    Messaging 1 to 1 1 to Many 1 to Many
    Event Type Single Event Type Single Event Type Multiple Event Type
    Route Messages Single Single Multiple Rules
    Filter Messages No Only to Message Body Full Message
    Allow transformation? No No Yes
    Cross Account Yes Yes Yes
    Message Latency Low Low Moderate
    Throughput High High Moderate

    View full-size slide