Real World Ruby Performance

My talk from RubyConf 2014 about Ruby Performance and the philosophy of performance.

Transcript

1. Real WORLD RUBY PERFORMANCE Aaron Quint / @aq / Ruby Conf 2014
2. @tmm1 @SamSaffron @_ko1 SHOUTOUT
3. We’ll come back to who I am later. It’s [relatively] unimportant. SKIPPING THE INTRO
4. I’ve learned so much over the past 5 years, what could I share? This TALK was HARD TO WRITE
5. It’s a ⌘+C ⌘+P culture. TIPS And tricks are the CLIFF NOTES of tech learning
6. How to THINK about a problem is much more interesting than how to solve it. As a mentor I want to teach philosophy not snippets
7. The tools and tricks will change over time. Today, Take away the process
8. A multi-step process. Ruby Performance as therapy
9. It’s a multi-step process Relax, Open up We’re going to go deep
10. Step 1: Acceptance
11. It’s your Fault.
12. Really?
13. Yes.
15. It’s not you, It’s me.
16. It’s not you, It’s me.
17. — George Costanza (Inventor of “It’s not you, it’s me”) It’s not you, It’s me.
18. Performance is about context
19. Doesn’t scale for what? To what degree? With what hardware? … “X Doesn’t SCALE” IS BS
20. So when we talk about our ruby being slow
22. Rails
23. Rails 10ms
24. Rails Your application 10ms
25. Rails Your application DB 10ms
26. Rails Your application DB 10ms 20ms
27. Rails Your application DB Cache 10ms 20ms
28. Rails Your application DB Cache 10ms 20ms 10ms
29. Rails Your application DB Cache 10ms 20ms 10ms 250ms
30. IT’s MY FAULT.
31. Step 2: Diagnosis
32. Where did I go wrong?
33. METRICS! Measurement! MMMNUMBERS! Milliseconds MATTER!
34. Use the right one for the job. Tools abound!
35. Step 3: Treatment
36. what are the steps to fix this problem?
37. How many strokes for the lowest #? Playing golf.
38. Two angles of optimization
39. Proxies/Balancers Application Datastores Filesystem/OS/Hardware Individual Request Path (Controller#action)
40. aka, speeding up a single query, controller action, or code path Vertical: Fix individual Elements
41. aka, Adding more workers per-node, buying better hardware Horizontal: Address hardware or software across a cluster
42. Important Themes:
43. Context is crucial to acceptance
44. Visibility and Introspect- ability are crucial to diagnosis
45. Knowing your tools is crucial to treatment
46. I’m Aaron Quint. I’m the chief Scientist at Paperless Post.
48. Opposing forces. Features vs. speed
49. We realized that being fast meant being stable
50. CASE STUDIES in performance therapy
52. Case 1: JSON FOR DAYS
56. package:7292:1123434234234
57. package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424
58. package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424 package:7292:1123434234234
59. package:7290:11234342343424 partner:8:11234342343424 partner:8:11234342343424 package:7292:1123434234234
60. Uncached performance is still a problem
61. ppprofiler to the rescue
62. ppprofiler
63. ppprofiler • Auto-cache toggling • Benchmark • Rblineprof • As::Notification Counts (SQL/Cache, etc) • MemoryProfiler (NEW!) • Gist-able (markdown) output
67. Rinse and Repeat Make the slowest lines faster
73. Case 2: FINGER IN THE SOCKET
74. Before Vday we were looking for any wins
75. IN BETWEEN THE LINES! stackprof + stackprof-remote
77. Ruby Process (Unicorn)
78. Ruby Process (Unicorn)
79. Ruby Process (Unicorn)
80. Ruby Process (Unicorn) AC::Dispatch
81. Ruby Process (Unicorn) AC::Dispatch MyController::Create
82. Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render
83. Ruby Process (Unicorn) AC::Dispatch MyController::Create Template::Render Ar::Find
84. Ruby Process (Unicorn)
85. Ruby Process (Unicorn) StackProf.start rb_profile_frames() rb_profile_frames() rb_profile_frames() rb_profile_frames() StackProf.stop StackProf.dump
86. ! [paperless@production-webapp10 current]$ stackprof tmp/stackprof-cpu-30715-1391204970.dump ================================== Mode: cpu(1000) Samples: 1761 (3.61% miss rate) GC: 128 (7.27%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 344 (19.5%) 342 (19.4%) Statsd#send_to_socket 393 (22.3%) 44 (2.5%) Statsd#sampled 44 (2.5%) 44 (2.5%) block in ActiveRecord::ConnectionAdapters::PostgreSQLPoolAdapter#execute 56 (3.2%) 29 (1.6%) block in ActiveSupport::Notifications::Fanout#listeners_for 29 (1.6%) 29 (1.6%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#extract_pg_identifier_from_name 26 (1.5%) 26 (1.5%) ActiveSupport::Notifications::Fanout::Subscribers::Evented#subscribed_to? 25 (1.4%) 25 (1.4%) String#blank? 25 (1.4%) 25 (1.4%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#select 24 (1.4%) 24 (1.4%) ActiveRecord::Base.scoped_methods 22 (1.2%) 22 (1.2%) Dalli::Server::KSocket#kgio_wait_readable 21 (1.2%) 21 (1.2%) ActiveSupport::CoreExtensions::Hash::Keys#assert_valid_keys 42 (2.4%) 20 (1.1%) block in Dalli::Server::KSocket#readfull 28 (1.6%) 19 (1.1%) ActiveRecord::ConnectionAdapters::ConnectionHandler#retrieve_connection_pool 18 (1.0%) 18 (1.0%) #<Module:0x00000002004b08>.instrumenter 17 (1.0%) 16 (0.9%) Dalli::Server#deserialize 15 (0.9%) 15 (0.9%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#select_raw 14 (0.8%) 14 (0.8%) #<Module:0x000000033e25d0>.decode_www_form_component 13 (0.7%) 13 (0.7%) Dalli::Server#write 15 (0.9%) 11 (0.6%) ActiveSupport::CoreExtensions::Time::Calculations#minus_with_coercion 10 (0.6%) 10 (0.6%) block in ActiveRecord::Base.with_scope 10 (0.6%) 10 (0.6%) block in ActiveRecord::ConnectionAdapters::QueryCache#cache_sql 21 (1.2%) 10 (0.6%) Yajl::Encoder.encode 10 (0.6%) 10 (0.6%) Set#add 10 (0.6%) 10 (0.6%) block (2 levels) in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#result_as_array 10 (0.6%) 10 (0.6%) ActiveSupport::CoreExtensions::Time::Calculations::ClassMethods#time_with_datetime_fallback 9 (0.5%) 9 (0.5%) ActiveRecord::DynamicFinderMatch#initialize 9 (0.5%) 9 (0.5%) ActiveSupport::LogSubscriber.logger 9 (0.5%) 9 (0.5%) block in ActionController::Base.action_methods 9 (0.5%) 9 (0.5%) block in ActionController::Base.action_methods 9 (0.5%) 9 (0.5%) block (2 levels) in ActiveRecord::Base.connection_handler=
87. Hmm, why is statsd slow?
88. Pull out good old benchmark
89. $ ruby test/profile/statsd.rb user system total real udp with connect 0.010000 0.000000 0.010000 ( 0.074522) udp without connect 0.120000 0.530000 0.650000 ( 13.096515) statsd with connect 0.000000 0.090000 0.090000 ( 0.103520) statsd without connect 0.100000 0.620000 0.720000 ( 13.483539)
90. WIN!
92. Case 3: THE HOLIDAY SCALE
94. Some times you can throw money at the problem
96. Case 4: SHRINKING THE GAP
97. Start at the top, work your way down. Starting with a HITLIST
98. Number of Requests x 90th Percentile Response Time Total Time
101. Using Stackprof flamegraphs on production.
102. Using Stackprof flamegraphs on production. SET IT ON FIRE!
108. Big wins are not the point
109. If you’re not failing you’re not being honest
110. Don’t just make tools, learn to use them
111. twitter: @aq github.com/quirkey github.com/paperlesspost Thanks!