Implementing Distributed Tracing ‘Like a Boss’ in your Apache Kafka Deployments

Transcript

1. Join the Confluent Community Slack Subscribe to the Confluent blog cnfl.io/community-slack cnfl.io/read Welcome to the DC Apache Kafka® Meetup! 6:30pm Doors open 6:30pm - 7:00pm Pizza, Drinks and Networking 7:00pm - 7:45pm Ricardo Ferreira, Confluent 7:45pm - 8:00pm Additional Q&A & Networking Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no affiliation with and does not endorse the materials provided at this event.
2. @riferrei | @confluentinc | #dckafkameetup Implementing distributed tracing ‘like a boss’ IN YOUR Apache Kafka DEPLOYMENTS @riferrei | @confluentinc | #dckafkameetup
3. About Us: • Ricardo Ferreira ❑ Developer Advocate @ Confluent ❑ Currently into Cloud & Serverless ❑ Ex-Oracle, Red Hat, and IONA Tech ❑ https://riferrei.net • Echo Dot (Alexa) ❑ The voice behind Amazon ❑ Ex-Raspberry Pi, Arduino @riferrei @alexa99
4. 4 Let´s go back in time…
5. How did we do monitoring? • Each system would have their very own monitoring system. • Developers would not worry about monitoring. This was supposed to be “IT stuff”. • Application was deemed OK if all systems were showing green. • Different monitoring approaches were used throughout the entire IT stack portfolio, requiring many teams to be involved.
6. @riferrei | @confluentinc | #dckafkameetup 6 And?
7. @riferrei | @confluentinc | #dckafkameetup System IS NOT Working! System IS working! Time spent with troubleshooting. Days, weeks, months!
8. @riferrei | @confluentinc | #dckafkameetup 8 Introduction to observability
9. @riferrei | @confluentinc | #dckafkameetup
10. Pillars of observability • Logs: raw sequence of events from a single instance of service. • Metrics: numerical measures aggregated in given point of time. • Distributed Tracing: detailed execution of the causality-related activities performed by a given transaction. It answers: ◦ Which services were involved? ◦ If it was slow, who caused that? ◦ If failed, who actually failed?
11. Evolution of concurrency • No Concurrency at all ◦ Ex: Apache HTTP Server • Basic Concurrency ◦ Ex: Multi-threaded Applications • Async Concurrency ◦ Ex: Actor-based Programming • Distributed Concurrency ◦ Ex: μServices Architecture Style
12. Distributed tracing today • There are many distributed tracing technologies available. • Standards are getting created to ensure a single programming model for each μService. • Deployment mechanisms such as Kubernetes are also taking care of that automatically. • Network proxies such as Service Meshes are also handling this. • OSS and proprietary options.
13. Why trace apache kafka? • Apache Kafka is becoming the de-facto standard to handle data. • Prediction? It will be the central nervous system of any company. • μServices in general already use Kafka to exchange messages and keep their data stores in-sync. • With event streaming becoming even more popular, the Kafka adoption tend to grow even more. • Because it is so freaking cool!
14. @riferrei | @confluentinc | #dckafkameetup 14 Distributed tracing in kafka
15. Opentracing java api • Library written in Java to handle distributed tracing via OpenTracing compatible APIs. • Requires the creation of specific tracer using the distributed tracing technology API. • Uses the GlobalTracer utility class to handle the tracer throughout the JVM application. • Supports: Apache Kafka Clients, Kafka Streams, and Spring Kafka. https://github.com/opentracing-contrib/java-kafka- client
16. Support for bundled jvms • Library written in Java that does the automatic creation of the tracer. • Implements the tracing logic using the Kafka Interceptors API. • Allows different tracers to be used, by using the TracerResolver class. • Provides OOTB support for Jaeger. • Allows multiple services in the JVM use their own tracer by specifying a configuration properties file. ◦ export INTERCEPTORS_CONFIG_FILE= https://github.com/riferrei/kafka-tracing-support
17. @riferrei | @confluentinc | #dckafkameetup 17 DEMO
18. @riferrei | @confluentinc | #dckafkameetup
19. Join The fun
20. @riferrei | @confluentinc | #dckafkameetup 20 announcements
21. NOMINATE YOURSELF OR A PEER AT CONFLUENT.IO/NOMINATE
22. KS19Meetup. CONFLUENT COMMUNITY DISCOUNT CODE 25% OFF* *Standard Priced Conference pass
23. @riferrei | @confluentinc | #dckafkameetup