Share
Embed
MicroProfile Reactive Messaging with Quarkus and Apache Kafka
JakartaOne Turkish, Aug 21st 2021
Follow me on twitter (@systemcraftsman) or subscribe to https://www.systemcraftsman.com/join/ to get any updates from me.
Aykut Bulgu
August 21, 2021
Transcript
65 slides
-
@systemcraftsman MicroProfile Reactive Messaging with Quarkus and Apache Kafka Aykut
M. Bulgu Trainer & Instructor | Technology Consultant www.SystemCraftsman.com -
@systemcraftsman Me as Code #oc apply -f aykutbulgu.yaml apiVersion: redhat/v3.2
kind: Services Content Architect metadata: name: Aykut Bulgu namespace: Red Hat Global Learning Services annotations: twitter: @systemcraftsman email: - [email protected] - [email protected] organizer: Software Craftsmanship Turkey founder: System Craftsman labels: married: yes children: 1 interests: - tech (mostly kafka) - aikido - gamification - stoicism spec: replicas: 1 containers: - image: aykut:latest -
@systemcraftsman MicroProfile Open-source community specification for Cloud-Native Java applications. A
community of individuals, organizations, and vendors collaborating. -
@systemcraftsman What is wrong with JMS and Message Driven Beans?
Heavyweight (mostly because of XA) Incompatible with popular brokers like Kafka No support for asynchronous IO -
@systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive
chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker -
@systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive
chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker External channels Internal channels -
@systemcraftsman Implementations Lightbend Alpakka SmallRye Reactive Messaging Open Liberty 19.0.0.9+
usage (via SmallRye Reactive Messaging) -
@systemcraftsman Using @Incoming import org.eclipse.microprofile.reactive.messaging.Incoming; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped public class
PriceConsumer { @Incoming("prices") public void consume(double price) { // process your price. } } Java file -
@systemcraftsman Using @Incoming @Incoming("prices") public CompletionStage<Void> consume(Message<Double> msg) { //
access record metadata var metadata = msg.getMetadata(IncomingKafkaRecordMetadata.class).orElseThrow(); // process the message payload. double price = msg.getPayload(); // Acknowledge the incoming message (commit the offset) return msg.ack(); } Java file -
@systemcraftsman Using @Channel import io.smallrye.mutiny.Multi; import io.smallrye.reactive.messaging.annotations.Channel; ...some imports omitted...
import org.jboss.resteasy.annotations.SseElementType; @Path("/prices") public class PriceResource { @Inject @Channel("prices") Multi<Double> prices; @GET @Path("/prices") @Produces(MediaType.SERVER_SENT_EVENTS) @SseElementType("text/plain") public Multi<Double> stream() { return prices; } } Java file -
@systemcraftsman Using @Channel @Inject @Channel("prices") Multi<Double> streamOfPayloads; @Inject @Channel("prices") Multi<Message<Double>>
streamOfMessages; @Inject @Channel("prices") Publisher<Double> publisherOfPayloads; @Inject @Channel("prices") Publisher<Message<Double>> publisherOfMessages; Java file -
@systemcraftsman Acknowledgment Strategies @Incoming("prices") @Acknowledgment(Acknowledgment.Strategy.PRE_PROCESSING) public void process(double price) {
// process price } @Incoming("prices") public CompletionStage<Void> process(Message<Double> msg) { // process price return msg.ack(); } Java file Java file -
@systemcraftsman Commit Strategies throttled keeps track of received messages and
commits an offset of the latest acked message in sequence (meaning, all previous messages were also acked). This is the default if enable.auto.commit is not explicitly set to true. latest commits the record offset received by the Kafka consumer as soon as the associated message is acknowledged (if the offset is higher than the previously committed offset). ignore performs no commit. This strategy is the default strategy when the consumer is explicitly configured with enable.auto.commit to true. -
@systemcraftsman Error Handling fail fail the application, no more records
will be processed (default strategy). The offset of the record that has not been processed correctly is not committed. ignore the failure is logged, but the processing continue. The offset of the record that has not been processed correctly is committed. dead-letter-queue the offset of the record that has not been processed correctly is committed, but the record is written to a Kafka dead letter topic. -
@systemcraftsman For High Throughput Use the throttled policy Or set
enable.auto.commit to true and use @Acknowledgment(Acknowledgment.Strategy.NONE). -
@systemcraftsman Consumer Groups Single consumer thread inside a consumer group
Multiple consumer threads inside a consumer group Multiple consumer applications inside a consumer group Pub/Sub: Multiple consumer groups subscribed to a topic 4 Patterns for consuming: -
@systemcraftsman Consumer Groups Single consumer thread inside a consumer group
quarkus.application.name or kafka.group.id -
@systemcraftsman Consumer Groups Multiple consumer threads inside a consumer group
mp.messaging.incoming.[channel-name].partitions -
@systemcraftsman Consumer Groups Multiple consumer applications inside a consumer group
mp.messaging.incoming.[channel-name].group.id -
@systemcraftsman Consumer Groups Pub/Sub: Multiple consumer groups subscribed to a
topic mp.messaging.incoming.somechannel.group.id=someid mp.messaging.incoming.somechannel.group.id=someotherid -
@systemcraftsman Using @Outgoing import io.smallrye.mutiny.Multi; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; import
java.time.Duration; import java.util.Random; @ApplicationScoped public class KafkaPriceProducer { private final Random random = new Random(); @Outgoing("prices") public Multi<Double> generate() { // Build an infinite stream of random prices // It emits a price every second return Multi.createFrom().ticks().every(Duration.ofSeconds(1)) .map(x -> random.nextDouble()); } } Java file -
@systemcraftsman Using @Outgoing @Outgoing("prices") public Multi<Message<Double>> generate() { return Multi.createFrom().ticks().every(Duration.ofSeconds(1))
.map(x -> Message.of(random.nextDouble()) .addMetadata(OutgoingKafkaRecordMetadata.<String>builder() .withKey("my-key") .withTopic("my-key-prices") .withHeaders(new RecordHeaders().add("my-header", "value".getBytes())) .build())); } Java file -
@systemcraftsman Using @Outgoing @Outgoing("prices") T generate(); // T excluding void
@Outgoing("prices") Message<T> generate(); @Outgoing("prices") Uni<T> generate(); @Outgoing("prices") Uni<Message<T>> generate(); @Outgoing("prices") CompletionStage<T> generate(); @Outgoing("prices") CompletionStage<Message<T>> generate(); Java file -
@systemcraftsman Using @Emitter import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some imports omitted...
@Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { CompletionStage<Void> ack = priceEmitter.send(price); } } Java file -
@systemcraftsman Using @Emitter import java.util.concurrent.CompletableFuture; import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some
imports omitted... @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { priceEmitter.send(Message.of(price) .withAck(() -> { // Called when the message is acked return CompletableFuture.completedFuture(null); }) .withNack(throwable -> { // Called when the message is nacked return CompletableFuture.completedFuture(null); })); } } Java file -
@systemcraftsman Processing Messages import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped
public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public double process(double price) { return price * CONVERSION_RATE; } } Java file -
@systemcraftsman Processing Messages import javax.enterprise.context.ApplicationScoped; import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import
io.smallrye.mutiny.Multi; @ApplicationScoped public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public Multi<Double> process(Multi<Integer> prices) { return prices.filter(p -> p > 100).map(p -> p * CONVERSION_RATE); } } Java file -
@systemcraftsman JSON public class Fruit { public String name; public
int price; public Fruit() { } public Fruit(String name, int price) { this.name = name; this.price = price; } } @ApplicationScoped public class FruitProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("fruit-in") @Outgoing("fruit-out") @Broadcast public Fruit process(Fruit fruit) { fruit.price = fruit.price * CONVERSION_RATE; return fruit; } } Java file Java file -
@systemcraftsman SerDe via Jackson <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jackson</artifactId> </dependency> # Configure
the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.ObjectMapperSerializer package com.acme.fruit.jackson; import io.quarkus.kafka.client.serialization.ObjectMapperDeseri alizer; public class FruitDeserializer extends ObjectMapperDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } Java file pom.xml application.properties -
@systemcraftsman SerDe via JSON-B <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jsonb</artifactId> </dependency> # Configure
the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.JsonbSerializer package com.acme.fruit.jsonb; import io.quarkus.kafka.client.serialization.JsonbDeserializer; public class FruitDeserializer extends JsonbDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } application.properties pom.xml Java file -
@systemcraftsman Avro io.apicurio.registry.utils.serde.AvroKafkaSerializer io.apicurio.registry.utils.serde.AvroKafkaDeserializer <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-avro</artifactId> </dependency> <dependency> <groupId>io.apicurio</groupId>
<artifactId>apicurio-registry-utils-serde</artifactId> </dependency> # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=io.apicur io.registry.utils.serde.AvroKafkaDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.apicuri o.registry.utils.serde.AvroKafkaSerializer application.properties pom.xml -
@systemcraftsman Resources • MicroProfile Reactive Messaging Spec 2.0 • Quarkus
Apache Kafka Reference Guide • SmallRye Reactive Messaging • Twitter: @systemcraftsman • Email: [email protected] • Demo: https://github.com/RedHatTraining/AD482-ToT-CoffeeShop