HolyJS Moscow 2018: An Introduction to Distributed Tracing

With the rise of single-page web apps and micro service architectures, web applications are becoming increasingly complex. The old tools with which we used to monitor the performance of monolith applications are no longer sufficient. To better understand how data is flowing between the different components of your application and to detect the bottlenecks before it’s too late, you need a new set of tools to monitor the whole stack - enter Distributed Tracing.

This talk will focus on what Distributed Tracing is, how to roll it out across your infrastructure, and how to use it to gain better insights and fix problems.

This talk aims to give the listener a better understanding of Distributed Tracing and why it’s important. It will focus on the OpenTracing standard developed to make your monitoring solution vendor-neutral, and will use examples from Elastic APM - the newest open source project coming to the Elastic Stack.

Transcript

1. Thomas Watson, @wa7son Principal Software Engineer, Elastic An Introduction To Distributed Tracing
2. @wa7son Who am I? • Thomas Watson • Open Source developer at
 github.com/watson • Principal Software Engineer at Elastic • Elastic APM – Node.js agent • Node.js Core Member • Tweets as @wa7son
3. @wa7son Tracing glossary “In software engineering, tracing involves a specialized use of logging to record information about a program's execution. This information is typically used by programmers for debugging purposes […]” Source: Wikipedia
4. @wa7son APM Application Performance Monitoring (Management) glossary
5. @wa7son Tracing The Monolith
6. @wa7son Tracing The Monolith Logs or APM product X
7. @wa7son Tracing The Monolith React Logs or APM product X
8. @wa7son Tracing Microservices React X
9. @wa7son React Tracing Microservices X
10. @wa7son React Tracing Microservices X
11. @wa7son React Tracing Microservices X
12. @wa7son React Tracing Microservices X
13. @wa7son Distributed Tracing Source: Giphy
14. @wa7son The APM library with many names Agents, Clients, and Tracers
15. @wa7son REST API Payment Service Order Service MySQL Payment Provider
16. @wa7son REST API Payment Service Order Service Payment Provider MySQL MySQL Spans Timeline
17. @wa7son React Tracing Microservices APM
18. @wa7son Tracing Microservices APM
19. @wa7son Tracing Microservices APM
20. @wa7son Challenges Source: Giphy
21. @wa7son Interservice Communication HTTP Binary Apache Kafka Message Queues …
22. @wa7son Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 GET /products HTTP/1.1 Host: www.example.com <magic header>
23. @wa7son Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 GET /products HTTP/1.1 Host: www.example.com <magic header>
24. @wa7son GET /products HTTP/1.1 Host: www.example.com : - - - Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 traceparent
25. @wa7son GET /products HTTP/1.1 Host: www.example.com : - - - Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 traceparent GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 Trace Context Working Group DRAFT
26. @wa7son GET /products HTTP/1.1 Host: www.example.com : - - - Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 traceparent
27. @wa7son 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 Version Trace ID 1 byte 8 bits 128 bit random number 64 bit random number Parent Span ID Trace Flags traceparent
28. @wa7son 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 Version Trace ID Parent Span ID Trace Flags 1 byte 8 bits 128 bit random number 64 bit random number DRAFT Trace Flags Recorded — 0b00000001 traceparent
29. @wa7son Congo Rojo Multi-Vendor Environments Congo Service B Service C Service A
30. @wa7son GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0
31. @wa7son GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 : <vendorname1=opaqueValue1>,<vendorname2=opaqueValue2>,… tracestate DRAFT
32. @wa7son tracestate <vendorname1=opaqueValue1>,<vendorname2=opaqueValue2>,… Parent (potentially vendor specific) Up to 32 unique vendors
33. @wa7son Bootstrapping the Trace in Browsers Initial HTTP request HTML page with APM initialization 1. Generate trace id 2. Start server span 3. Generate span ID for initial HTTP request on behalf of the browser 4. Render HTML template 5. Respond to request + End server span Browser Server
34. @wa7son const Html = ({ body, transaction }) => ` <!DOCTYPE html> <html> <head> </head> <body> <script src="/path/to/elastic-apm-js-base.umd.min.js"></script> <script> elasticApm.init({ pageLoadTraceId: '${transaction.traceId}', pageLoadSpanId: '${transaction.ensureParentId()}', pageLoadSampled: ${transaction.sampled} }) </script> <div>${body}</div> </body> </html> `
35. @wa7son const Html = ({ body, transaction }) => ` <!DOCTYPE html> <html> <head> </head> <body> <script src="/path/to/elastic-apm-js-base.umd.min.js"></script> <script> elasticApm.init({ pageLoadTraceId: '${transaction.traceId}', pageLoadSpanId: '${transaction.ensureParentId()}', pageLoadSampled: ${transaction.sampled} }) </script> <div>${body}</div> </body> </html> `
36. @wa7son developer.mozilla.org/docs/Web/API/Navigation_timing_API Navigation Timing API
37. @wa7son
38. @wa7son
39. @wa7son Don’t be this guy
40. @wa7son With great power comes great responsibility Async
41. @wa7son Standards Source: Giphy
42. @wa7son Trace Context Working Group
43. @wa7son
44. @wa7son const opentracing = require('opentracing') const tracer = new opentracing.Tracer() const span = tracer.startSpan('some work') // Do some work span.finish()
45. @wa7son span.setTag('user.id', user.getId()) span.log({ 'event': ‘error', 'error.object': err })
46. @wa7son // Start a new (parentless) root Span: const parent = Tracer.startSpan('DoWork') // Start a new (child) Span: const child = Tracer.startSpan('load-from-db', { childOf: parent.context() }) // Start a new async (FollowsFrom) Span: const child = Tracer.startSpan('async-cache-write', { references: [ opentracing.followsFrom(parent.context()) ] })
47. @wa7son const headersCarrier = {} // "Serialize" Span Context into headersCarrier tracer.inject( clientSpan.context(), Tracer.FORMAT_HTTP_HEADERS, headersCarrier ) // Add tracing headers to headers of outgoing request Object.assign(outboundHTTPReq.headers, headersCarrier)
48. @wa7son // "De-serialize" Span Context from inbound HTTP request headers const headersCarrier = inboundHTTPReq.headers const wireCtx = tracer.extract( Tracer.FORMAT_HTTP_HEADERS, headersCarrier ) const serverSpan = tracer.startSpan('...', { childOf: wireCtx })
49. @wa7son Tracer.FORMAT_TEXT_MAP Tracer.FORMAT_HTTP_HEADERS Tracer.FORMAT_BINARY
50. @wa7son const { Tracer } = require('elastic-apm-node/opentracing') const tracer = new Tracer({ serviceName: 'my-awesome-service', serverUrl: 'https://example.com:8200' }) const opentracing = require('opentracing') opentracing.initGlobalTracer(tracer) // then later... const tracer = opentracing.globalTracer() !
51. @wa7son const { Tracer } = require('my-custom-tracer') const tracer = new Tracer() const express = require('express') const expressOpenTracing = require('express-opentracing') const app = express() app.use(expressOpenTracing({ tracer })) // ...
52. @wa7son Deployment Source: Giphy
53. @wa7son Checklist • Does it support the languages we’re using (Node.js, Python, Java etc)? • Does it support “serverless” infrastructures (e.g. AWS Lambda)? • Does it support the databases we’re using? • Does it support the inner-service transports we’re using (HTTP, Kafka etc)? • Does it support Real User Monitoring (RUM) for the platforms we’re using (e.g. JavaScript, React Native)? • If the answer isn’t yes to all of the above, does it allow us to extend it easily? • Can we access the raw trace data? • Can we correlate the trace data with other data sources (logs, metrics etc)?
54. @wa7son Recommendations •Don’t try to instrument everything in the beginning •Instrument important services first •Breaking up a monolith: •Add instrumentation to the monolith first if possible •Make instrumentation part of the process of breaking out new microservices
55. @wa7son Data, data, data Source: Star Trek TNG
56. @wa7son
57. @wa7son Logs
58. @wa7son Metrics
59. @wa7son Errors
60. @wa7son Traces
61. @wa7son Errors Traces Logs Metrics
62. @wa7son Demo Time! Source: MC Hammer - U Can't Touch This
63. @wa7son More information Source: For a Few Dollars More
64. @wa7son
65. @wa7son opentracing.io
66. @wa7son
67. @wa7son elastic.co / solutions / apm
68. @wa7son DM’s are open ❤ @wa7son
69. @wa7son Getting involved Source: Giphy
70. @wa7son
71. @wa7son github.com / elastic / apm github.com/elastic/apm-agent-nodejs/pull/691
72. @wa7son github.com/watson спасибо