PWL-SF#1 => Ryan Kennedy on Dapper, a Distributed Systems Tracing Infrastructure

Ryan Kennedy and Anjali Shenoy from Yammer Engineering kicked off our group by presenting the Dapper, a Large-Scale Distributed Systems Tracing Infrastructure paper by Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan and Chandan Shanbhag.

Transcript

1. Google Dapper A Large-Scale Distributed Systems Tracing Infrastructure
2. About Us • Ryan Kennedy @rckenned - runs infrastructure for @yammer. • Anjali Shenoy @anjshenoy - infrastructure engineer @yammer.
3. Who has read this paper?
4. This is an experience paper. Not a research paper.
5. What is Dapper?
6. “Distributed Systems”
7. “Tracing Infrastructure”
8. “Large-Scale”
9. “A Large-Scale Distributed Systems Tracing Infrastructure"
10. “We built Dapper to provide Google’s developers with more information about the behavior of complex distributed systems.”
11. Distributed systems explained
13. Benefits of a distributed system… • A collection of software services • Developed by different teams • Across different platforms • Using different programming languages
14. Downsides of a distributed system… • A collection of software services • Developed by different teams • Across different platforms • Using different programming languages
15. Engineering Context • Multiple services • Each service manned by a separate team • Continuous deployment of services
16. On-call troubles • Investigate overall health of system • Guess which service is at fault • Why which service is at fault*
17. Dapper A large scale distributed systems tracing infrastructure
18. Dapper’s Problem Space • End user - on-call engineer • Bird’s-eye view into overall system health • Ability to drill down into a service and see why its holding up the train • Long term pattern recognition
19. Design goals ! • Ubiquity • Low overhead • Application level transparency • Scalability
20. Low overhead. But… how? • Sampling - 0.01% of requests for high throughput systems. • Adaptive sampling was being deployed at publish time. • Out of band trace data collection
21. What is a trace? • A dapper trace is a tree of spans - 1 for every RPC • Each span has its own set of annotations, including the parent span • Annotation - application specific data you want to send along with the trace
23. What about annotations? • Annotation is some application specific information you pass along with your span. • A span can have 0-many annotations. • Each annotation has a timestamp and either a textual value or key-value pairs.
24. Let’s talk about instrumentation
25. {! "trace_id": "7021185255097625687",! "spans": [ {! "span_id": "2186499883",! "parent_span_id": "",! "name": "groups.create",! "start_time": 1395364621946662144,! "duration": 1359471104,! "annotations": [ {! "path_to_sql": {! "sql": "INSERT into messages (…)",! "start_node": “app/controllers/ a_controller.rb:create",! "path": "app/models/b.rb:create_message, app/ models/c.rb:create_group_message!"! },! "logged_at": 1395364623306275840! }! ]}]}
26. Ok. Now what? • How to effectively coalesce data in downstream systems? • Data for immediate perusal • Data for long term pattern recognition
28. Bigtable
29. Moar sampling… • Google’s prod clusters generate >1TB of data/ day. • Dapper end users want to query trace data ~ 2 weeks old
30. Dapper API • By Trace Id: load on demand • Bulk Access: leverage MapReduce jobs to provide access to billions of traces in parallel. • Indexed access: composite index => lookup by service name, host name, timestamp.
31. Dapper API Usage • Online web applications • Command line - on demand • One-off analytical tools
34. Dapper in Development • Performance • Correctness • Understanding • Testing
35. Exception monitoring • Exception monitoring service uses trace/span ids as meta data • Link to same in UI reports
36. Other uses… • Long tail latency • Inferring service dependencies • Layered and shared storage systems
37. Blind spots • Coalescing effects • Finding a root cause within a service • Tying kernel events to a trace
38. In conclusion • Best use case for dev/ops teams. • Practical - negligible performance impact • Keep the trace repo API open
39. Thank you, authors • Benjamin H. Sigelman • Luiz André Barroso • Mike Burrows • Pat Stephenson • Manoj Plakal • Donald Beaver • Saul Jaspan • Chandan Shanbhag
40. Thanks! Questions?