Upgrade to Pro — share decks privately, control downloads, hide ads and more …

I didn't know EventBridge did that

I didn't know EventBridge did that

AWS Tech Summit 2024
Dive Deep on AWS EventBridge

Avatar for Jérôme Van Der Linden

Jérôme Van Der Linden

October 28, 2024
Tweet

More Decks by Jérôme Van Der Linden

Other Decks in Technology

Transcript

  1. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. I didn't know that EventBridge did that Jerome Van Der Linden Senior Solutions Architect AWS
  2. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Agenda Event-Driven Architecture EventBridge basics API Destination Schema registry and discovery Archive & Replay EventBridge Pipes Cross-region & cross-account
  3. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Event-driven Architecture
  4. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event-driven Architecture Event Broker Increase resiliency and agility / extensibility by reducing coupling between components, using asynchronous communications via events. Events
  5. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. EventBridge basics
  6. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. EventBridge Architecture Partner event source Rules Default event bus Custom event bus SaaS event bus Amazon EventBridge
  7. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event buses AWS services Custom events SaaS apps Partner event source Rules Amazon EventBridge Default event bus Custom event bus SaaS event bus Event buses
  8. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event sources – AWS services Partner event source Rules Event sources Default event bus Custom event bus SaaS event bus Amazon EventBridge
  9. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event sources – AWS services C O N S U M E E V E N T S F R O M O V E R 1 1 5 A W S S E R V I C E S
  10. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event sources – Custom events Partner event source Rules Event sources Default event bus Custom event bus SaaS event bus Amazon EventBridge
  11. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event sources – SaaS partners Rules Event sources Default event bus Custom event bus SaaS event bus Amazon EventBridge Partner event source
  12. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event sources – SaaS partners O V E R 4 2 A M A Z O N E V E N T B R I D G E S A A S A P P I N T E G R A T I O N S …
  13. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Rules AWS services Custom events SaaS apps Partner event source Rules Default event bus Custom event bus SaaS event bus Amazon EventBridge
  14. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event Structure "version": "id": "account": "region": "time": "source": "detail-type": "resources": "detail":
  15. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Rule Matching B A S I C S • String: • Prefix / suffix: • Anything but (not): • Numeric: • Exists: "source": "source": "suffix": "detail": "state": "anything-but": "numeric" "<" "exists":
  16. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Rule Matching A D V A N C E D • And: • Or (values): • Or (global): { "source": "prefix": , "detail": { "state": [ { "anything-but": "initializing" } ] } } "source": [ ", " ] "$or": [ , ]
  17. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Rule Matching E X P E R T // Rule for (”source" || ”detail-type") && (”detail.metricName" || ”detail.namespace") { "$or": [ { "source": [ "aws.cloudwatch" ] }, { "detail-type": [ "CloudWatch Alarm State Change"] } ], "detail" : { "$or": [ { "metricName": [ { "anything-but": { "prefix": ”CPU" } } ] }, { "namespace": [ "AWS/EC2", "AWS/ECS" ] } ] } } More details at: https://github.com/aws/event-ruler
  18. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Targets AWS services Custom events SaaS apps Partner event source Rules Targets Default event bus Custom event bus SaaS event bus Amazon EventBridge
  19. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. EventBridge Partner event source Rules Default event bus Custom event bus SaaS event bus Amazon EventBridge
  20. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Retries and Dead Letter Queue (DLQ) Event Bus Event Rules Events Access Denied Timeout Throttled Retry Policy: - Up to 185 attempts - Up to 24 hours* DLQ DLQ DLQ *: may be reached before the 185 attempts because of back-off’s jitter
  21. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon S3 Lambda function Lambda function Lambda function ? ECS Task S3 Event Notifications?
  22. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon S3 Lambda function Lambda function Lambda function ECS Task SNS Topic SQS Queue Lambda SNS SQS S3 Event Notifications?
  23. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S3 Event Notifications with EventBridge Benefits: Ø Multi-destinations Ø More targets / less code Ø Advanced filtering (prefix, sufix, object size, time range…) Amazon S3 Lambda function Lambda function Lambda function ECS Task EventBridge Event Bus
  24. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Observability const { EventBridgeClient, PutEventsCommand } = require('@aws-sdk/client-eventbridge'); const AWSXRay = require('aws-xray-sdk-core'); const eventBridgeClient = AWSXRay.captureAWSv3Client(new EventBridgeClient()); exports.handler = async (event) => { const ebEvent = { Entries: [{ Detail: JSON.stringify({ ... }), DetailType: 'Request', Source: 'com.mycompany.myapp', }] } const command = new PutEventsCommand(ebEvent); const result = await eventBridgeClient.send(command); } TraceHeader: 'Root=1-5759e988-bd862e3fe1be46a994272793;Sampled=1' X-Ray SDK automatically populates X-Amzn-Trace-Id in HTTP Header for all invocation targets. You can manually add the TraceHeader field in your entries for the PutEvents… ⚠ it overrides the HTTP Header ⚠ trace id is not available in the event payload
  25. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. EventBridge API Destination
  26. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Sending events to non-AWS services Event SaaS Webhook Others On Premise Applications API Partner API ?
  27. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. API Destination Event Filter Connection Rule Target (API Destination) Endpoint HTTP SaaS Webhook Others On Premise Applications API Partner API
  28. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. API Destination – Connection ****** Basic Auth User / password Oauth Client ID Client secret Auth endpoint API Key Key / value Event Filter Connection Rule Target (API Destination) Endpoint HTTP Credentials stored in Secrets Manager Webhook Others On Premise Applications API SaaS Partner API
  29. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. API Destination – Connection ****** Basic Auth User / password Oauth Client ID Client secret Auth endpoint API Key Key / value Event Filter Connection Rule Target (API Destination) Endpoint HTTP Also used in Step Functions Webhook Others On Premise Applications API SaaS Partner API
  30. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. API Destination – HTTP Endpoint Event Filter Connection Rule Target (API Destination) HTTP Endpoint Custom target URL HTTP Method Rate limit Partner integration Native integration with 12 partners SaaS … Webhook Others On Premise Applications API
  31. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. API Destination – Rule configuration HTTP Headers + request parameters using event payload (eg. $.detail.state) Event Filter Connection Rule Target (API Destination) Endpoint HTTP SaaS Webhook Others On Premise Applications API Native Partner API
  32. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Event Transformation
  33. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event tranformation { "version": "0", "id": "6a7e8feb-b491-4cf7", "detail-type": "order-submission", "source": "checkout", "account": "123456789012", "time": "2023-10-29T12:10:21Z", "region": "eu-central-1", "resources": ["arn:your-event-bus"], "detail": { "metadata": { "version": "1", "domain": "retail", "subdomain": "checkout", "service": "order-submission", "category": "domain-event", "type": "data" }, "data": { "order_number": "123-456x", "total": 29.99, "currency": "EUR", "customer_id": "cu-101", "items": [] } } } EventBridge Event { "datetime": "2023-10-29T12:10:21Z", "source": "retail/checkout/order-submission", "orderId": "123-456x", "amount": 29.99, "currency": "EUR", "customerId": "cu-101", "products": [] } ? Data required by 3rd party API
  34. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event tranformation Event Filter Connection HTTP Endpoint Rule Target (API Destination) Input Transformer Template + Input path
  35. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event tranformation { "datetime": "$.time", "domain": "$.detail.metadata.domain", "subdomain": "$.detail.metadata.subdomain", "service": "$.detail.metadata.service", "orderId": "$.detail.data.order_number", "amount": "$.detail.data.total", "currency": "$.detail.data.currency", "customerId": "$.detail.data.customer_id", "products": "$.detail.data.items" } Input Path { "version": "0", "id": "6a7e8feb-b491-4cf7", "detail-type": "order-submission", "source": "checkout", "account": "123456789012", "time": "2023-10-29T12:10:21Z", "region": "eu-central-1", "resources": ["arn:your-event-bus"], "detail": { "metadata": { "version": "1", "domain": "retail", "subdomain": "checkout", "service": "order-submission", "category": "domain-event", "type": "data" }, "data": { "order_number": "123-456x", "total": 29.99, "currency": "EUR", "customer_id": "cu-101", "items": [] } } }
  36. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event tranformation { "datetime": "<datetime>", "source": "<domain>/<subdomain>/<service>", "orderId": "<orderId>", "amount": <amount>, "currency": "<currency>", "customerId": "<customerId>", "products": <products> } Template { "datetime": "$.time", "domain": "$.detail.metadata.domain", "subdomain": "$.detail.metadata.subdomain", "service": "$.detail.metadata.service", "orderId": "$.detail.data.order_number", "amount": "$.detail.data.total", "currency": "$.detail.data.currency", "customerId": "$.detail.data.customer_id", "products": "$.detail.data.items" } Input Path
  37. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event tranformation { "datetime": "2023-10-29T12:10:21Z", "source": "retail/checkout/order-submission", "orderId": "123-456x", "amount": 29.99, "currency": "EUR", "customerId": "cu-101", "products": [] } { "datetime": "$.time", "domain": "$.detail.metadata.domain", "subdomain": "$.detail.metadata.subdomain", "service": "$.detail.metadata.service", "orderId": "$.detail.data.order_number", "amount": "$.detail.data.total", "currency": "$.detail.data.currency", "customerId": "$.detail.data.customer_id", "products": "$.detail.data.items" } Input path { "datetime": "<datetime>", "source": "<domain>/<subdomain>/<service>", "orderId": "<orderId>", "amount": <amount>, "currency": "<currency>", "customerId": "<customerId>", "products": <products> } Template
  38. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Schema registry and discovery
  39. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Catalog of events? Order Service Payment Service Inventory Service Search Service Users Service Billing Service Delivery Service Loyalty Service Event Bus … … ?
  40. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Schema registry and discovery Event Bus Events { "version": "0", "id": "c18259af-97e6-e63b-7383-65bfc9018d0c", "detail-type": "AccountCreated", "source": "com.mycompany.accountcreation", "account": "445498108802", "time": "2024-08-22T08:09:57Z", "region": "eu-central-2", "resources": [], "detail": { "userId": "324h324lkn", "accountNumber": "345klhj345435", "creationDate": "2024-08-22T08:07:36+0000", "source": "website", "idConfirmed": false } }
  41. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. { "openapi": "3.0.0", "info": { "version": "1.0.0", "title": "AccountCreated" }, "paths": {}, "components": { "schemas": { "AWSEvent": { "type": "object", "required": ["detail-type", "resources", "detail", "id", "source", "time", "region", "version", "account"], "x-amazon-events-detail-type": "AccountCreated", "x-amazon-events-source": "com.mycompany.accountcreation", "properties": { "detail": { "$ref": "#/components/schemas/AccountCreated" }, // ... } }, "AccountCreated": { Schema registry and discovery Schema Registry Schema Discovery Event Bus Events Ø Schema name Ø Versioning Ø Open API 3 / JSONSchema v4
  42. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Schema registry and discovery Schema Registry Schema Discovery Event Bus Events Code Binding IDE / Console / API
  43. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standardize events format
  44. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Using CloudEvents specification { "version": "0", "id": "6a7e8feb-b491-4cf7", "detail-type": "order-submission", "source": "myapp.orders-service", "account": "123456789012", "time": "2023-10-29T12:10:21Z", "region": "eu-central-1", "resources": [], "detail": { "domain": "retail", "subdomain": "checkout", "service": "order-submission", "idempotency_key": "29d2b068-f9c7-42a0-91e3", "correlation_id": "dddd9340-135a-c8c6-95c2" }, "data": { "order_id": "c172a984-3ae5-43dc-8c3f", "total": 29.99, "currency": "EUR", "customer_id": "dda98122-b511-4aaf-9465” } } } EventBridge Event format { "specversion": "1.0", "id": "29d2b068-f9c7-42a0-91e3", "source": "myapp.orders-service", "type": "order-submission", "time": "2023-10-29T12:10:21Z", "datacontenttype”: "application/json”, "dataschema": "https://us-west- 2.console.aws.amazon.com/events/home?region=us-west- 2#/registries/discovered-schemas/schemas/myapp.orders-service%40 order-submission", "data": { "order_id": "c172a984-3ae5-43dc-8c3f", "total": 29.99, "currency": "EUR", "customer_id": "dda98122-b511-4aaf-9465” }, "correlation_id": "dddd9340-135a-c8c6-95c2", "domain": ”retail”, "subdomain": "checkout", "service": "order-submission" } CloudEvents format Required Optional Extensions
  45. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Sending CloudEvents using API Destination POST /orders HTTP/1.1 Host: webhook.example.com Content-Type: application/cloudevents+json; charset=utf-8 Body: { "specversion": "1.0", "id": "29d2b068-f9c7-42a0-91e3", "source": "myapp.orders-service", "type": "order-submission", "time": "2023-10-29T12:10:21Z", "datacontenttype”: "application/json”, "dataschema": "https://us-west-2.console.aws.amazon.com/events/home?region=us-west- 2#/registries/discovered-schemas/schemas/myapp.orders-service%40 order-submission", "data": { "order_id": "c172a984-3ae5-43dc-8c3f", "total": 29.99, "currency": "EUR", "customer_id": "dda98122-b511-4aaf-9465” }, "correlation_id": "dddd9340-135a-c8c6-95c2", "domain": ”retail”, "subdomain": "checkout", "service": "order-submission" }
  46. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. EventBridge Pipes
  47. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. What do these services have in common? Ø The only way to consume them is to do polling or use Lambda (Event Source Mapping), which means writing code. Amazon SQS Amazon Kinesis Data Streams Amazon DynamoDB Amazon MSK Amazon MQ Self-managed Apache Kafka ​Decreased development velocity ​Increased risk of bugs ​Increased complexity ​Increased operational load ​Inefficient polling High total cost of ownership
  48. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. How to implement the following use cases? Amazon SQS AWS Step Functions Process an SQS queue with a Step Function: Amazon DynamoDB Stream EventBridge Event Bus Step Functions API Destination SSM Run Command …. Publish DynamoDB record changes to multiple AWS services Amazon Kinesis Data Streams 1 Amazon Kinesis Data Streams 2 Amazon Kinesis Data Streams Split multiplexed data streams into multiple domain streams
  49. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. EventBridge Pipes Amazon SQS Amazon Kinesis Data Streams Amazon DynamoDB Amazon MSK Amazon MQ Source Compatible services Amazon EventBridge Pipes Filter, enrich, and transform events Filter Enrich & transform Restructure, modify or add data to input event Target Send event to one of the 15 supported targets Amazon EventBridge + API Destination Amazon Kinesis Data Streams AWS Step Functions Amazon API Gateway Amazon SQS Amazon API Gateway AWS Step Functions AWS Lambda Amazon EventBridge API destinations Self-managed Apache Kafka Same polling infrastructure as Lambda Event Source Mapping
  50. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Archive and Replay
  51. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Archive… Event Bus Events Custom Rule EventBridge Managed Rule Event Archive Event Consumer - Custom retention - AES-256 encryption - Custom filter - Read-only configuration
  52. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. … and replay Event Bus Events Custom Rule Event Archive Event Consumer Same Consumer ⚠ Idempotency { "replay-name": [ { "exists": false } ] } EventBridge Managed Rule
  53. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. … and replay Event Bus Events Event Archive (Blue) Event Consumer Alternate Consumers { "replay-name": [ { "exists": true } ] } { ... Existing rule + "replay-name": [ { "exists": false } ] } { "replay-name": [ { "exists": false } ] } EventBridge Managed Rule Custom Rule Custom Rule Dev Event Bus (Green) Event Consumer Dev account
  54. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Cross-account / cross-region
  55. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Cross-account & cross-region capabilities Account 1 Region A Region A Producer Event Bus Central Event Bus Rule Rule Account 3 Cross-region / same account Rule Region B Event Bus Cross-account / same region Rule Cross-account / cross-region Account 2 Region C Event Bus Rule Cloudwatch Logs Consumer
  56. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Global endpoints Primary Region (us-east-1) Global Endpoint Alarm Route53 Secondary Region (us-west-2) Route53 Health Check Amazon EventBridge Global Endpoint (hb3wfg53j8.ne) PutEvents Custom event bus Amazon EventBridge Cross Region Replication (Optional) Custom event bus Amazon EventBridge Producer Consumers Consumers Global Endpoint Managed rule client.put_events( EndpointId="hb3wfg53j8.ne", Entries=[...] )
  57. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Global endpoints Primary Region (us-east-1) Global Endpoint Alarm Route53 Secondary Region (us-west-2) Route53 Health Check Amazon EventBridge Global Endpoint (hb3wfg53j8.ne) PutEvents Cross Region Replication (Optional) Custom event bus Amazon EventBridge Producer Consumers Consumers Global Endpoint Managed rule IngestionToInvocationStartLatency alarm failover client.put_events( EndpointId="hb3wfg53j8.ne", Entries=[...] ) Archive
  58. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conclusion
  59. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Resources https://tiny.amazon.com/63rgbp92/quipTp3XIdid
  60. © 2024, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. Thank you! Jerome Van Der Linden @jeromevdl