Shrine: Handle file uploads like it's 2017

https://www.youtube.com/watch?v=plD-RkKEay0

Transcript

1. Shrine Handle file uploads like it’s 2017 janko-m @jankomarohnic
2. Shrine Best practices for handling file uploads in Ruby janko-m @jankomarohnic
3. Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache ★184 Paperdragon ★ 116
4. Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache ★184 Paperdragon ★ 116
5. Flow
6. Choose a file… picture.jpg Submit POST /photos Content-Type: multipart/form-data … […file content…] { name: "file", filename: "picture.jpg", type: "image/jpeg", tempfile: #<Tempfile:/var/…/RackMultipart-286> } Rack
7. Choose a file… picture.jpg Submit POST /albums Content-Type: multipart/form-data … […file content…] #<ActionDispatch::Http::UploadedFile @original_filename=“picture.jpg”, @content_type=“image/jpeg”, @tempfile=#<Tempfile:/var/…/RackMultipart-286>> Rails
8. Shrine.storages = { store: Shrine::Storage::FileSystem.new( "public", prefix: "uploads/store") } Shrine.plugin :sequel # or :activerecord class ImageUploader < Shrine # image attachment logic end class Photo < Sequel::Model include ImageUploader[:image] end
9. photo = Photo.create(image: File.open("…")) photo.image_data #=> { "storage": "store", "id": "fd9sdf0ag32kf1s.jpg", "metadata": { "filename": "nature.jpg", "size": 883954, "mime_type": "image/jpeg" } }
10. IMPROVEMENT A: Retaining file on form redisplays
11. Choose a file… picture.jpg Submit Album title Choose a file… No file selected Submit Album title can’t be blank (Waiting for file to be uploaded…)
12. Shrine.storages = { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), } {"storage":"cache", "id":"8ag8fdg.jpg", "metadata":{...}} Choose a file… No file selected Submit Album title can’t be blank
13. IMPROVEMENT B: Direct uploads
14. Choose a file… picture.jpg Submit
15. Shrine.plugin :direct_upload Rails.application.routes.draw do mount ImageUploader::UploadEndpoint, to: "/images" end POST /images/cache/upload jQuery-File-Upload Dropzone FineUploader …
16. Choose a file… Submit 32% POST /images/cache/upload
17. Choose a file… Submit 100% {"storage":"cache", "id":"8ag8fdg.jpg", …}
18. • User has a better idea how long upload will take • User can fill in other fields while file is uploading • Multiple files can be uploaded in parallel Benefits
19. IMPROVEMENT C: External storage
20. • Receiving file uploads uses server resources • Stored files are served through the app • Doesn’t work with multiple servers • Heroku doesn’t support storing on disk • Heroku has a 30-second request timeout Downsides of filesystem
21. Shrine.storages = { cache: Shrine::Storage::S3.new(…), store: Shrine::Storage::S3.new(…), } Shrine.storages = { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), }
22. 1. GET /images/cache/presign { "url": "https://my-bucket.s3.amazonaws.com", "fields": { "key": "b7d575850ba61b44c8a9ff889dfdb14d88cdc25", "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMTo...", "x-amz-credential": "5TQ/eu-west-1/s3/aws4_request", "x-amz-algorithm": "AWS4-HMAC-SHA256", "x-amz-date": "20151024T001129Z", "x-amz-signature": "c1eb634f83f96b69bd675f535b3..." } } 2. POST https://my-bucket.s3.amazonaws.com Content-Type: multipart/form-data […file content…]
23. IMPROVEMENT D: Resumable uploads
24. • Flaky connection terminates the upload • User needs to change location during upload • Computer freezes in the middle of upload Large files => User needs to retry the whole upload
25. Chunking 5MB 5MB 5MB 5MB 5MB …
26. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5
27. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 32%
28. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅
29. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ 17%
30. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ❌
31. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ 43%
32. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ✅
33. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ✅ 17% 54% 32%
34. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ✅ ✅ ✅ ✅
35. • S3 & Azure • Concurrent chunking • Numerous other features • Well-designed protocol • iOS & Android • Ruby implementation
36. # config.ru require "tus/server" map("/files") { run Tus::Server } janko-m/tus-ruby-server Shrine.storages = { cache: Shrine::Storage::Url.new, store: Shrine::Storage::S3.new(…), } janko-m/shrine-url
37. • Uploads can be resumed in case of interruption • Concurrent chunking can speed up the upload • Checksums can ensure that no bytes were lost Benefits
38. IMPROVEMENT E: Backgrounding
39. 2048×1365 800×800 500×500 300×300
40. previous_cover_photo = album.cover_photo album.update(cover_photo: File.open("...")) previous_cover_photo.exists? #=> false
41. album.destroy album.cover_photo.exists? #=> false
42. Deleting ↓ Processing (= downloading + processing + uploading) ↓ SLOW SLOW
43. class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq
44. class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_in(3, data) end mperham/sidekiq
45. class PromoteJob include Sidekiq::Worker def perform(data) ActiveRecord::Base.with_connection do Shrine::Attacher.promote(data) end end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq
46. class PromoteJob include SuckerPunch::Job def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end brandonhilkert/sucker_punch
47. class PromoteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.create!(data: data) end rocketjob/rocketjob
48. class DeleteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.delete(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote { |data| … } Shrine::Attacher.delete { |data| … } rocketjob/rocketjob
49. album.cover_photo #=> nil album.cover_photo = File.open("...") album.save # background job is kicked off album.cover_photo.storage_key #=> "cache" # … background job finishes album.reload album.cover_photo.storage_key #=> "store"
50. album.destroy album.cover_photo.exists? #=> true # … background job finishes album.cover_photo.exists? #=> false
51. Benefits • Fast for the user • Removes impact on request throughput • Can be scaled individually • Can automatically retry timeouts • Easier to manually retry failures that require action • Easier to monitor performance
52. Demo
53. Credits to Refile @jnicklas
54. janko-m/shrine