Scaling Shopify

Talk given at ConFoo 2014 on February 28th, 2014.

Transcript

1. SCALING SHOPIFY ...or ensuring happiness for online shoppers
2. cjoudrey   @
7. the stack
8. nginx unicorn • rails 4 • mysql 5.6 (percona) ruby 2.1 •
9. 95 app servers 3,884 unicorn workers 5 job servers 387 job workers
10. 1 request 1 process =
11. scale?
12. over 90,000 shops
14. 1.6B$ annual GMV that’s 3,600$ per min
15. cyber monday black friday
17. 61 M$ in GMV in four days
18. flash sales
21. page caching
24. shopify/cacheable
25. generational caching
26. gzip • etag + 304 not modified
27. class PostsController < ApplicationController def show response_cache do @post = @shop.posts.find(params[:id]) respond_with(@post) end end def cache_key_data { action: action_name, format: request.format, params: params.slice(:id), shop_version: @shop.version } end end
29. flash sale
30. query caching
31. shopify/identity_cache
32. full model caching
33. opt-in by design
34. after_commit expiry
35. class Product < ActiveRecord::Base include IdentityCache has_many :images cache_has_many :images, :embed => true end product = Product.fetch(id) images = product.fetch_images
36. class Product < ActiveRecord::Base include IdentityCache cache_index :shop_id, :handle, :unique => true end Product.fetch_by_shop_id_and_handle(shop_id, handle)
38. flash sale
39. background jobs
40. webhooks emails • fraud detection • payment processing
42. priority queues payment • default • low realtime •
43. class ProductImportJob include BackgroundQueue::Realtime def perform(params) ... end end BackgroundQueue.push(ProductImportJob, ...)
44. throttling
45. the right data store for the job
46. ephemeral data sessions carts • inventory reservation •
47. now what?
48. catching regressions
49. measure it! if it moves...
50. statsd
52. Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :parse, 'Liquid.Template.parse' Liquid::Template.statsd_measure :render, 'Liquid.Template.render'
53. PaymentProcessingJob.stats_count :perform, 'PaymentProcessingJob.processed'
54. load testing
56. simulates a flash sale
57. several times per week
58. slow queries
59. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1
60. determining root cause
61. https://github.com/snormore/nginx-x-rid-header nginx request_id header proxy_set_header X-Request-ID "$request_id"; log_format main '... $request_id' step 1
62. https://gist.github.com/mnutt/566725 Complete 200 OK in 100ms (Views: 60ms | ActiveRecord: 40ms | request_id=bc12813bce...) log_process_action ActionController::Instrumentation step 2
63. https://github.com/basecamp/marginalia User Load (0.3ms) SELECT `users`.* FROM `users` WHERE `users`.`id` = 1 LIMIT 1 /*application:Shopify, controller:users,action:show, request_id:bc12813bce...*/ basecamp/marginalia step 3
64. # User@Host: shopify[shopify] @ [127.0.0.1] # Thread_id: 264419969 Schema: shopify Last_errno: 0 Killed: 0 # Query_time: 0.150491 Lock_time: 0.000057 Rows_sent: 1 Rows_examined: 147841 Rows_affected: 0 Rows_read: 147841 # Bytes_sent: 1214 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: FF7021AAA # QC_Hit: No Full_scan: No Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: Yes Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 0 InnoDB_IO_r_bytes: 0 InnoDB_IO_r_wait: 0.000000 # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000 # InnoDB_pages_distinct: 475 SET timestamp=1393385020; SELECT `discounts`.* FROM `discounts` WHERE `discounts`.`shop_id` = 1745470 AND `discounts`.`status` = 'enabled' ORDER BY ISNULL(ends_at) DESC, ends_at DESC LIMIT 1 /*application:Shopify,controller:orders,action:pay, request_id:bc12813bce...*/ profit!
65. access.log rails.log slow_query.log profit! (2)
66. schema migration with zero downtime
67. soundcloud/lhm
68. current schema new schema
69. insert/delete/update triggers
70. INSERT INTO ... SELECT ... insert/delete/update triggers
71. testing for external calls memcached • mysql • redis net/http •
72. it’s not about preventing it’s about raising awareness
73. integration test with assert_externals(...) do .. end Unexpected external call (mysql): !"" mysql_load("GiftCard") !"" "SELECT `gift_cards`.* FROM `gift_cards` WHERE `gift_cards`.`id` = 1063936318 LIMIT 1" #"" called from: app/services/gift_card_payment_processing.rb: 73:in `block in log_successful'
74. subscribe('sql.active_record') ActiveSupport::Notifications ["sql.active_record", 2014-02-26 02:38:43 +0000, 2014-02-26 02:38:43 +0000, "a119c5ac2aa6fb4a52fe", {:sql=>"SELECT `users`.* FROM `users` LIMIT 1", :name=>"User Load", :connection_id=>69893685920420, :binds=>[]}]
75. monkey-patch other libs to add instrumentation
76. thanks! :)