Testing Kafka Streams Applications

2018년 9월, 제 1회 한국 Kafka Meetup 발표.

Presented in Kafka meetup, Seoul, September 2018.

Slides: English. Presentation: Korean.

1. Testing Kafka Streams Applications Lee Dongjin | dongjin@apache.org
2. Overall • Introduce how to run [unit|integration] test for Kafka Streams Application. • Still evolving … ◦ Slides: based on 1.1.0 ◦ Example: based on 2.0.0
3. Problem: Testing Kafka Streams Application • Using running Kafka instance ◦ DON’T DO IT - WASTE OF RESOURCES & ENERGY. ◦ Hard to deploy, Non-isolated, Slow, and Non-repeatable (side effects!) • The other strategies (From 1.1.0) ◦ EmbeddedKafkaCluster ◦ TopologyTestDriver ◦ MockProcessorContext
4. Testing Strategies: Overall • EmbeddedKafkaCluster strategy ◦ An In-memory, embedded instance of a Kafka cluster. ◦ Identical to the real instance - you can use it with embedded zookeeper, schema registry instance. • TopologyTestDriver strategy ◦ An official test utility for Topology. ◦ Very similar to the real instance, but not identical. • MockProcessorContext strategy ◦ Provides (almost) complete checkup for Processor instance. • Each strategy has its advantages & disadvantages.
5. Testing Strategies: Comparison EmbeddedKafkaCluster TopologyTestDriver MockProcessorContext Reality Best Good Good Granularity Good So-So Best Speed Bad Good Best Difficulty Easy Easy Hard
6. How to use EmbeddedKafkaCluster (1) 1. Add Kafka Streams test dependency ◦ org.apache.kafka:kafka-streams:${kafka.version}:test 2. Create EmbeddedKafkaCluster instance & Topics. 3. Run test by starting Kafka Streams instance. 4. Next slide!
7. How to use EmbeddedKafkaCluster (1) Requires org.apache.kafka: kafka-streams:${kafka.version}: test dependency Create Embedded cluster instance & Add topics
8. How to use EmbeddedKafkaCluster (2) 3. Test with IntegrationTestUtils
9. How to use TopologyTestDriver (1) 1. Add dependency ◦ org.apache.kafka:kafka-streams-test-utils:${kafka.version}:test 2. Create TopologyTestDriver instance 3. Run test with TopologyTestDriver#pipeInput , TopologyTestDriver#readOutput .
10. How to use TopologyTestDriver (2) Feed input with pipeInput method, one by one. Retrieve output with readOutput method, in order.
11. Summary 1. Choose an adequate strategy for your case. ◦ Tradeoff: Reality but Slow vs. Non real but Fast 2. EmbeddedKafkaCluster approach is better for the integration test, while TopologyTestDriver , MockProcessorContext approach is better for unit test. 3. However, If you have insufficient experience, test with EmbeddedKafkaCluster first (i.e., identical to the real cluster) and add other tests later.
12. Questions? • Slides: speakerdeck.com/dongjin • Example project: github.com/dongjinleekr/kafka-streams-example