Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2021-09-02 SpringOne - Tracing Issues in Your A...

2021-09-02 SpringOne - Tracing Issues in Your Application

Jonatan Ivanov

May 02, 2022
Tweet

More Decks by Jonatan Ivanov

Other Decks in Programming

Transcript

  1. Marcin Grzejszczak, Jonatán Ivanov Spring One 2021 Tracing Issues in

    Your Application Copyright © 2021 VMware, Inc. or its affiliates.
  2. About Us Marcin Grzejszczak 󰐤 Spring Cloud @ VMware Twitter:

    @mgrzejszczak Blog: toomuchcoding.com Observability Spring Cloud Sleuth (Distributed Tracing) Spring Observability 😉 Contract Testing Spring Cloud Contract Jonatán Ivanov 󰏘 Spring Cloud @ VMware Twitter: @jonatan_ivanov Blog: develotters.com Observability Spring Cloud Sleuth (Distributed Tracing) Spring Observability 😉 Micrometer (Metrics)
  3. Disclaimer This presentation may contain product features or functionality that

    are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined. The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein.
  4. Great result Joe was happy to be able to find

    an interesting meme https://www.rd.com/list/dog-memes/
  5. 2021-02-10 13:57:55.239 ERROR 274441 --- [nio-8082-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for

    servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.IllegalStateException: Meme overflow occurred] with root cause java.lang.IllegalStateException: Meme overflow occurred at io.memegenerator.MemeController.generateMeme(Application.java:93) at io.memegenerator.MemeController$$FastClassBySpringCGLIB$$d54a9db6.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) at io.memegenerator.MemeController$$EnhancerBySpringCGLIB$$e33f7fa1.memeOverflow(<generated>) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1061) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:961) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
  6. Production observability Jane sees that there is no proper feedback

    from production Logs are missing critical information We’re lacking production observability!
  7. TraceId: 123 2021-02-10 13:57:55.239 … HELLO FROM SERVICE1 123 123

    2021-02-10 13:58:10.239 … HELLO FROM SERVICE2 123 123 123 2021-02-10 13:58:20.239 … HELLO FROM SERVICE4 123 2021-02-10 13:58:30.239 … HELLO FROM SERVICE3 123
  8. Another issue... Jane sees that Joe filed another issue The

    description could be better… Jane knows that the logs are correlated - for sure they will help!
  9. [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:05.239 INFO 274441 --- [nio-8082-exec-1] This line shouldn’t

    be called [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:15.239 INFO 274441 --- [nio-8082-exec-1] Logging sth [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:25.239 INFO 274441 --- [nio-8082-exec-1] Logging moar [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:35.239 INFO 274441 --- [nio-8082-exec-1] This should work [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:45.239 INFO 274441 --- [nio-8082-exec-1] Sending a request [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:55.239 INFO 274441 --- [nio-8082-exec-1] Returning response Sending what request? Getting what response? Did it take 10 seconds?
  10. [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:05.239 INFO 274441 --- [nio-8082-exec-1] This line shouldn’t

    be called [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:15.239 INFO 274441 --- [nio-8082-exec-1] Logging sth [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:25.239 INFO 274441 --- [nio-8082-exec-1] Logging moar [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:35.239 INFO 274441 --- [nio-8082-exec-1] This should work [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:45.239 INFO 274441 --- [nio-8082-exec-1] Sending a request - start [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:47.239 INFO 274441 --- [nio-8082-exec-1] Sending a request - stop [took 2 seconds] [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:55.239 INFO 274441 --- [nio-8082-exec-1] Returning response Change the logs? Add start? Add stop?
  11. That’s a lot of work... Jane sees that modifying all

    logs will take a lot of time There must be a better way to solve this problem
  12. Service 1 Service 3 Service 4 Application Controller Application Service

    Service 2 Operation Operation? Operation Operation
  13. [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:05.239 INFO 274441 --- [nio-8082-exec-1] This line shouldn’t

    be called [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:15.239 INFO 274441 --- [nio-8082-exec-1] Logging sth [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:25.239 INFO 274441 --- [nio-8082-exec-1] Logging moar [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:35.239 INFO 274441 --- [nio-8082-exec-1] This should work [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:45.239 INFO 274441 --- [nio-8082-exec-1] Sending a request [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:45.239 INFO 274441 --- [nio-8082-exec-1] [FRAMEWORK] SpanId [123] Start [xx] Stop [yy] [6dad9c7bd47f0e21c946a7a57d75e7c8] 2021-02-10 13:58:55.239 INFO 274441 --- [nio-8082-exec-1] Returning response
  14. What are the options? Jane wants to know what distributed

    tracing options are out there? She has a polyglot environment - does it work for any language / framework? What are the latency visualization tools?
  15. OpenZipkin Mature, production tested, multi language support Zipkin 14k stars

    on GitHub, first release 05.2016 Brave 2k stars on GitHub, first release 04.2013 Supported languages: C#, Go, Java, JavaScript, Ruby, Scala, PHP (many other are community driven)
  16. OpenTelemetry (OTel) New, multi language, CNCF oriented OpenTelemetry Java ~900

    stars on GitHub, first release 11.2019 OpenTelemetry Spec 1.9k stars on GitHub, first release 06.2019 First GA releases at the end of 2020 and beginning of 2021 OTel wants to address logs, tracing & metrics problems Supported languages: Java, C#, Go, JavaScript, Python, Rust, C++, Erlang/Elixir
  17. Latency Visualization Tools VMware Tanzu Observability by Wavefront *Free Tanzu

    Observability for your Spring Boot applications - no sign-up needed
  18. What about Java & Spring? Jane wants to know if

    she can use distributed tracing with Java & Spring?
  19. Spring Cloud Sleuth Distributed tracing support for Spring Boot applications

    First release 10.2015 Version 2.x works directly with OpenZipkin Brave’s API Version 3.x bridges from Sleuth API to tracer APIs Plug in the library, configure it, you’re done!
  20. Spring Cloud Sleuth - how does it work? Sleuth Instrumentation

    JAVA PROCESS Sleuth API HTTP Messaging Batch ... Span Store OpenZipkin Brave (out of the box support) OpenTelemetry (incubating) API BRIDGES OpenZipkin, Tanzu Observability (out of the box support) Custom (via OZ Brave / OTel bridges / configuration) REPORTER BRIDGES
  21. What about Java & Spring? Jane wants to know if

    she can collect metrics with Java & Spring?
  22. Micrometer Vendor-neutral application metrics facade First release 02.2018 Popular Metrics

    library on the JVM Like SLF4J, but for metrics - simple API (facade/abstraction) Supports the most popular metric backends (no vendor lock-in) Comes with Spring Boot Actuator Spring projects are instrumented using Micrometer Lots of third-party libraries use Micrometer to instrument their code
  23. Supports the most popular metric backends AppOptics Atlas Azure Monitor

    CloudWatch (AWS) Datadog Dynatrace Elastic OpenTSDB Prometheus SignalFx Stackdriver (GCP) StatsD Wavefront* (VMware) (/actuator/metrics) Ganglia Graphite Humio InfluxDB JMX KairosDB New Relic *VMware Tanzu Observability by Wavefront
  24. Spring Observability Spring-native approach to Observability Part of Spring Framework

    Core Recorder API Instrument once - get metrics and tracing Highly extensible - register additional listeners Tracing abstraction Like Spring Cloud Sleuth but without Spring Cloud Instrumentation happens in dedicated projects
  25. Spring Observability - how does it work? JAVA PROCESS Span

    Store Tracing Listener Metrics Listener RECORDING LISTENERS HTTP Spring Web Instrumentation Tracing API Tracer Implementation Micrometer API MeterRegistry Metrics Store Spring Observability Recorder API
  26. IntervalRecording recording = recorder .recordingFor((IntervalEvent) () -> "important-calculation") .tag(Tag.of("calculation-type", "tax",

    Cardinality.LOW)) .tag(Tag.of("user-id", userId, Cardinality.HIGH)) .start(); try { calculationService.calculate(); } catch (Exception exception) { recording.error(exception); throw exception; } finally { recording.stop(); }
  27. IntervalRecording recording = recorder .recordingFor((IntervalEvent) () -> "important-calculation") .tag(Tag.of("calculation-type", "tax",

    Cardinality.LOW)) .tag(Tag.of("user-id", userId, Cardinality.HIGH)) .start(); try { calculationService.calculate(); } catch (Exception exception) { recording.error(exception); throw exception; } finally { recording.stop(); }
  28. IntervalRecording recording = recorder .recordingFor((IntervalEvent) () -> "important-calculation") .tag(Tag.of("calculation-type", "tax",

    Cardinality.LOW)) .tag(Tag.of("user-id", userId, Cardinality.HIGH)) .start(); try { calculationService.calculate(); } catch (Exception exception) { recording.error(exception); throw exception; } finally { recording.stop(); }
  29. IntervalRecording recording = recorder .recordingFor((IntervalEvent) () -> "important-calculation") .tag(Tag.of("calculation-type", "tax",

    Cardinality.LOW)) .tag(Tag.of("user-id", userId, Cardinality.HIGH)) .start(); try { calculationService.calculate(); } catch (Exception exception) { recording.error(exception); throw exception; } finally { recording.stop(); }