FluentConf18: Help! I accidentally distributed my system

Specialization among engineers and increasing levels of abstraction have created a situation in which almost no one has a complete view of how data moves through an entire system, end to end. We’ve all become distributed systems engineers, intentionally or not. Rachel Myers and Emily Nakashima detail tools and skills we can use to get ourselves out of the corner we’ve boxed ourselves into.

Transcript

1. Help! I accidentally distributed my system! Rachel Myers, Emily Nakashima
2. @rachelmyers, @eanakashima @eanakashima
3. @rachelmyers, @eanakashima
4. @rachelmyers, @eanakashima @rachelmyers
5. @rachelmyers, @eanakashima
6. @rachelmyers, @eanakashima
7. @rachelmyers, @eanakashima ModCloth Web App
 (Rails) MySQL User request (from NGINX & loadbalancer)
8. @rachelmyers, @eanakashima What even is
 a distributed system?
9. @rachelmyers, @eanakashima What even is a distributed system? Some
 Node Another Node
10. @rachelmyers, @eanakashima Designing distributed systems •What are the user flows in my system? •What are the “nouns” in my system? •What are the patterns for reading data? •What are the patterns for writing data? •Do different flows have different performance, consistency, & availability requirements?

11. @rachelmyers, @eanakashima ModCloth Web App MySQL User request (from NGINX & loadbalancer) Mobile
12. @rachelmyers, @eanakashima ModCloth Web App MySQL User request (from NGINX & loadbalancer) Mobile Voting Service MySQL
13. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service MySQL
14. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service MySQL MySQL
15. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service Photo Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL
16. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service Photo Service Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL
17. @rachelmyers, @eanakashima ModCloth Web App MySQL Checkout Service Identity Service Photo Service Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL MySQL
18. @rachelmyers, @eanakashima ModCloth Web App MySQL Checkout Service Identity Service Photo Service Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL MySQL
19. @rachelmyers, @eanakashima ModCloth Web App MySQL Checkout Service Identity Service Photo Service Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL MySQL
20. @rachelmyers, @eanakashima Not the problems we’re talking about
21. @rachelmyers, @eanakashima An accidental distributed system ⛈ flexfits.com
22. @rachelmyers, @eanakashima flexfits.com
23. @rachelmyers, @eanakashima flexfits.com Ecommerce SaaS User request
24. @rachelmyers, @eanakashima flexfits.com Subscription Plugin SaaS Ecommerce SaaS User request
25. @rachelmyers, @eanakashima flexfits.com Subscription Plugin SaaS Ecommerce SaaS Fulfillment SaaS User request
26. @rachelmyers, @eanakashima flexfits.com Subscription Plugin SaaS Ecommerce SaaS Metrics SaaS Fulfillment SaaS User request
27. @rachelmyers, @eanakashima flexfits.com
28. @rachelmyers, @eanakashima flexfits.com
29. @rachelmyers, @eanakashima flexfits.com Subscription Plugin SaaS Ecommerce SaaS Metrics SaaS Fulfillment SaaS User request ?
30. @rachelmyers, @eanakashima flexfits.com Subscription Plugin SaaS Ecommerce SaaS Metrics SaaS Fulfillment SaaS User request Orders Service Postgres
31. @rachelmyers, @eanakashima New requirements for buy vs build
32. @rachelmyers, @eanakashima flexfits.com SaaS SaaS SaaS SaaS User request Home cooked on IaaS
33. @rachelmyers, @eanakashima Buy or Build •Will it improve availability? •Will it improve recovery? •Will you have visibility or points of control? •Will you have customizability? •Will you know when your changes take effect? •Can you audit changes?
34. @rachelmyers, @eanakashima IaaS, PaaS, BaaS, & FaaS
35. @rachelmyers, @eanakashima Cloud Infrastructure
36. @rachelmyers, @eanakashima IaaS
37. @rachelmyers, @eanakashima IaaS •Will it improve availability? ✅ •Will it improve recovery? ✅ •Will you have visibility or points of control? ✅ •Will you have customizability? ✅ •Will you know when your changes take effect? ✅ •Can you audit changes? ✅
38. @rachelmyers, @eanakashima Platforms
39. @rachelmyers, @eanakashima PaaS
40. @rachelmyers, @eanakashima PaaS •Will it improve availability? ✅ •Will it improve recovery? •Will you have visibility or points of control? ✅ •Will you have customizability? ✅ •Will you know when your changes take effect? •Can you audit changes?
41. @rachelmyers, @eanakashima Backends
42. @rachelmyers, @eanakashima BaaS (RIP Parse)
43. @rachelmyers, @eanakashima BaaS •Will it improve availability? ✅ •Will it improve recovery? •Will you have visibility or points of control? •Will you have customizability? ✅ •Will you know when your changes take effect? •Can you audit changes?
44. @rachelmyers, @eanakashima Functions
45. @rachelmyers, @eanakashima FaaS
46. @rachelmyers, @eanakashima FaaS •Will it improve availability? •Will it improve recovery? •Will you have visibility or points of control? •Will you have customizability? ✅ •Will you know when your changes take effect? •Can you audit changes?
47. @rachelmyers, @eanakashima Secret Distributed Systems
48. @rachelmyers, @eanakashima Secret distributed system: dependencies
49. @rachelmyers, @eanakashima A B C D
50. @rachelmyers, @eanakashima A B C D1 D2
51. @rachelmyers, @eanakashima Secret distributed system: browsers
52. @rachelmyers, @eanakashima “The browser is part of your distributed system, not just a client you support”
 -@danielespeset
53. @rachelmyers, @eanakashima The one where we DDOS’d ourselves
54. @rachelmyers, @eanakashima What happened?
55. @rachelmyers, @eanakashima // if the page is long enough to scroll if (document.body.clientHeight > window.innerHeight) { // add a scroll event listener document.addEventListener('scroll', function(e) { // if within 100px of the bottom of the page if (window.innerHeight + window.scrollY + 100 > document.body.clientHeight) fetchNextPage(); }); // else fetch another page of results immediately } else { fetchNextPage(); }
56. @rachelmyers, @eanakashima Frontend complexity is only increasing
57. @rachelmyers, @eanakashima Secret distributed system: vendors
58. @rachelmyers, @eanakashima LaunchDarkly
59. @rachelmyers, @eanakashima Secret distributed system: vendors
60. @rachelmyers, @eanakashima # instantiate the flag with a default value # falls back to default value if service is offline show_feature = ld_client.variation( "allow_traces", { :key => "user@example.com" }, false ) if show_feature # application code to show the feature else # the code to run if the feature is off
61. @rachelmyers, @eanakashima Secret distributed system: vendors
62. @rachelmyers, @eanakashima Secret distributed system: vendors
63. @rachelmyers, @eanakashima GitHub’s customer support application
64. @rachelmyers, @eanakashima GitHub customer support tool github.com Customer Support Tool User email
65. @rachelmyers, @eanakashima GitHub customer support tool github.com Customer Support Tool GMail User email
66. @rachelmyers, @eanakashima GitHub customer support tool github.com MailChimp Customer Support Tool GMail User email SendGrid
67. @rachelmyers, @eanakashima GitHub customer support tool github.com MailChimp Customer Support Tool GMail User email SendGrid
68. @rachelmyers, @eanakashima GitHub customer support tool github.com Kafka Customer Support Tool Ingest service User email Outbound mail service
69. @rachelmyers, @eanakashima GitHub customer support tool github.com MailChimp Customer Support Tool GMail User email SendGrid
70. @rachelmyers, @eanakashima Cognitive Load
71. @rachelmyers, @eanakashima Instrumentation & Chill ❄
72. @rachelmyers, @eanakashima Fewer Alerts
73. @rachelmyers, @eanakashima
74. @rachelmyers, @eanakashima Unknown unknown unknowns
75. @rachelmyers, @eanakashima Outsource Your Brain
76. @rachelmyers, @eanakashima Use tools that let you… •Ask, what is happening now to this user? Endpoint? Team? •Ask, how does this compare to the aggregate case? •Jump between levels of abstraction •See both breadth & depth •Understand how data flows

77. @rachelmyers, @eanakashima Observability… not just for ops thought leaders!
78. @rachelmyers, @eanakashima “The ability to ask arbitrary questions about your system from the outside, and understand what is happening on the inside.”
 -@mipsytipsy
79. @rachelmyers, @eanakashima logs metrics traces / events
80. @rachelmyers, @eanakashima Crawl the pipes
81. @rachelmyers, @eanakashima No really, instrumentation
82. @rachelmyers, @eanakashima
83. @rachelmyers, @eanakashima # monitoring endpoint Location ~ /ping { return 200 '200 ok'; }
84. @rachelmyers, @eanakashima Conclusion
85. @rachelmyers, @eanakashima Two Vendors: Two Conclusions
86. @rachelmyers, @eanakashima 1. We’re all distributed systems engineers. ☎
87. @rachelmyers, @eanakashima 2. Trust your tools /
88. @rachelmyers, @eanakashima Thank you