Using Redis Streams to Build More Resilient Services

Redis, the swiss army knife of databases, has recieved a new data type with version 5.0 which is worth taking a closer look. Specifically, Redis Streams. In this session Thomas explains how he used Redis Streams to build a more fault-tolerant and scalable service with Symfony and the Symfony Messenger Component.

Transcript

1. PRESENTED BY
2. PRESENTED BY • Co-founder & CEO of Sulu GmbH • More than 15 years of experience in web technologies & development • PHP, Symfony, React, SQL, Redis, Elasticsearch, … • Open source enthusiast • Loves cooking and mountains Hi, I’m Thomas Schedler chirimoya thomas@sulu.io
3. PRESENTED BY 1 The story how I came across Redis Streams 2 Introduction to Redis Stream commands 3 Real-world application using Redis Streams Agenda:
4. PRESENTED BY https://github.com/matthiasnoback/building-autonomous-services-workshop
5. PRESENTED BY Tell, don’t ask. Generate an optimised read model for your service by 
 consuming domain events from a central stream.
6. PRESENTED BY tail -f -n 1 Probably not the most robust implementation
 for a real-world application.
7. PRESENTED BY
8. PRESENTED BY In computer science, a stream is a sequence of data elements made available over time. “
9. PRESENTED BY Stream 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 18 17 consumer 1 position consumer 2 position new record truncatable records producer emitting records
10. PRESENTED BY https://brandur.org/redis-streams
11. PRESENTED BY Redis Streams A new data type introduced with Redis 5.0.0.
12. PRESENTED BY Assuming you have installed Redis server and client.
 And keep in mind, we need to use the latest version of Redis 5.0. Connect to Redis PHP $redis = new \Redis(); $redis->connect('127.0.0.1', '6379'); CLI > redis-cli -h 127.0.0.1 -p 6379
13. PRESENTED BY The stream elements can be one or more key-value pairs.
 Let's add elements to the stream: Add message to stream PHP $redis->xAdd('mystream', '*', [
 'KEY' => 'VALUE'
 ]); // MESSAGE ID (e.g. 1526919030474-55) $redis->xLen('mystream'); // (integer) 1 CLI > XADD mystream * KEY VALUE # MESSAGE ID (e.g. 1526919030474-55) > XLEN mystream # (integer) 1
14. PRESENTED BY Read message from one or multiple streams, only returning entries with an ID greater than the last received ID reported by the caller. Read message from stream PHP $redis->xRead(['mystream' => 0], 2); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ],
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE' CLI > XREAD COUNT 2 STREAMS mystream 0 # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]
15. PRESENTED BY In order to avoid polling at a fixed or adaptive interval, the command is able to block until new entries get pushed into the requested stream. Block for data PHP $redis->xRead(['mystream' => '$'], 1, 0); // MESSAGES // [
 // 'STREAMNAME' => [
 // 'MESSAGEID' => [
 // 'KEY' => 'VALUE'
 // ]
 // ]
 // ] CLI > XREAD BLOCK 0 STREAMS mystream $ # MESSAGES # STREAMNAME [
 # MESSAGEID [
 # KEY
 # VALUE
 # ]
16. PRESENTED BY All clients are served with all the entries arriving in a stream. How can we scale reads?
17. PRESENTED BY Consumer Groups 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 18 17 group a consumer 1 position group b consumer 2 position new record truncatable records group a consumer 2 position group b consumer 1 position
18. PRESENTED BY Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. Create a Consumer Group PHP $redis->xGroup(
 'CREATE', 
 'mystream',
 'mygroup'
 ); // 1/0 CLI > XGROUP CREATE mystream mygroup $ # 1/0
19. PRESENTED BY Redis stream provides the concept of consumer groups, allowing multiple consumers to process the same stream to implement load balancing. Read data via Consumer Group PHP $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => 0]
 ); // MESSAGES $redis->xReadGroup(
 'mygroup', 'myconsumer',
 ['mystream' => '>']
 ); CLI > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream 0 # MESSAGES > XREADGROUP
 GROUP mygroup mycomsumer
 STREAMS mystream >
20. PRESENTED BY Consumer groups require explicit acknowledgement of the messages successfully processed by the consumer. Data processed confirmation PHP $redis->xAck(
 'mystream', 
 'mygroup', 
 ['message-id']
 ); // 1/0 CLI > XACK mystream mygroup message-id # 1/0
21. PRESENTED BY • Identify the message data that has been delivered but not confirmed • Change the owner of these data and redeliver Failure Processing PHP $redis->xPending('mystream', 'mygroup'); // MESSAGE IDs $redis->xClaim(
 'mystream', 
 'mygroup',
 'myconsumer',
 3600,
 ['message-id']
 ); CLI > XPENDING mystream mygroup # MESSAGE IDs > XCLAIM mystream mygroup myconsumer 3600 message-id
22. PRESENTED BY Worked pretty good out of the box, but the API is quite low-level. Let's search for a high-level abstraction.
23. PRESENTED BY
24. PRESENTED BY • The Messenger component helps applications send and receive messages — message bus • By default, messages are handled as soon as they are dispatched • If you want to handle a message asynchronously, you can configure a transport Symfony Messenger Component
25. PRESENTED BY Create our own transport? No official Redis transport — Symfony Messenger 4.2
26. declare(strict_types=1); namespace HandcraftedInTheAlps\Bundle\RedisTransportBundle\Transport; use Symfony\Component\Messenger\Transport\TransportFactoryInterface; use Symfony\Component\Messenger\Transport\TransportInterface; class RedisStreamTransportFactory implements TransportFactoryInterface { public function createTransport(string $dsn, array $options): TransportInterface { $parsedUrl = parse_url($dsn); ... return new RedisStreamTransport($parsedUrl['host'], $parsedUrl['port'], ...); } public function supports(string $dsn, array $options): bool { return 0 === mb_strpos($dsn, 'redis-stream://'); } }
27. PRESENTED BY
28. PRESENTED BY
29. PRESENTED BY
30. PRESENTED BY
31. PRESENTED BY
32. PRESENTED BY The Redis Streams transport was introduced in Symfony 4.3 Open source … FTW!
33. PRESENTED BY Real-world showcase — Ticketing Application Ticketing Application Ticket Access Order Event ... Access System REST API
34. PRESENTED BY Real-world showcase — Ticketing Application Ticketing Application Ticket Access Order Event ... Access System REST API Message Bus Contingent
35. PRESENTED BY Tell, don’t ask. Domain Events are used to capture things that can trigger 
 a change to the state of your application.
36. class DomainEventMessage { ... public function __construct( string $name, string $type, string $id, array $payload = [], ?\DateTimeImmutable $created = null ) { $this->name = $name; $this->type = $type; $this->id = $id; $this->payload = $payload; $this->created = $created ?? new \DateTimeImmutable(); } ... }
37. public function cancelTicket(Ticket $ticket): void { $this->workflowRegistry->get($ticket)->apply($ticket, Ticket::TRANSITION_CANCEL); $newTicketId = Uuid::uuid4()->toString(); $this->createTicket($newTicketId, $ticket); $this->domainEventCollector->push( new DomainEventMessage( Ticket::class, 'canceled', $ticket->getId(), [ 'newTicketId' => $newTicketId, ] ) ); }
38. class TicketCanceledDomainEventMessageHandler extends AbstractTicketDomainEventMessageHandler { public function __invoke(DomainEventMessage $message): void { if ('canceled' !== $message->getType() || Ticket::class !== $message->getName()) { return; } $ticketIds = [$message->getId()]; $newTicketId = $message->getPayload()['newTicketId'] ?? ''; if ($newTicketId) { $ticketIds[] = $newTicketId; } else { $this->logger->error('Could not find newTicketId in canceled for access system.', [ 'canceledTicketId' => $message->getId(), ]); } $this->syncTickets($ticketIds); } }
39. PRESENTED BY When ~ argument is used, the trimming is performed only when Redis is able to remove a whole macro node. Trim Data from the stream PHP $redis->xTrim('mystream', 1000, true); CLI > XTRIM mystream MAXLEN ~ 1000
40. class DomainEventHandler implements MessageHandlerInterface { /** * @var ElasticsearchStorage */ private $storage; public function __construct(ElasticsearchStorage $storage) { $this->storage = $storage; } public function __invoke(DomainEventMessage $message): void { $this->storage->store($message); } }
41. Thank you! Thank you!