Validating and processing the content of a file with Active Storage

Transcript

1. Validating and processing the content of a file with Active Storage Claudio Baccigalupo
2. Summary 1. Uploading and downloading with Active Storage has_one_attached, rails_blob_url 2. Extracting metadata and processing attachments download_blob_to_tempfile, prepending callbacks 3. Validating the content of an attachment attachment_changes, attached?
3. Part 1: uploading and downloading files with Active Storage (5 minutes)
4. github.com/clutter/mdma Uploading files Downloading files
5. Basics of Active Storage :package :version Started POST "/builds" Parameters: {"build"=>{"package"=>#<ActionDispatch::Http::UploadedFile:47e0>, "version"=>"42"} :version :package
6. Uploading an attachment # app/views/builds/new.html.erb <%= form_with(model: @build) do |form| %> <legend>1. Choose a file to upload:</legend> <%= form.file_field :package %> # app/controllers/builds_controller.rb def build_params params.require(:build).permit :package, :version end # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package end
7. Downloading an attachment # config/environments/production.rb Rails.application.configure do config.active_storage.service = :s3 end # app/views/builds/index.html.erb <table> <% @builds.each do |build| %> <td><%= build.version %></td> <td><%= link_to 'Download', rails_blob_url(build.package, disposition: 'attachment') %> </td> <% end %> </table>
8. Basic attachment validations # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package validates :package, presence: true end
9. Proposed attachment validations # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package validates :package, presence: true, attachment_size: { maximum: 90.megabytes }, attachment_content_type: "application/zip" end
10. Part 2: extracting metadata and processing an attachment (10 minutes)
11. From downloading to installing https://localhost:3000/rails/active_storage/blobs/ [sha]/demo_v42.ipa?disposition=attachment Simply downloading the package file to an iOS device is not enough to install the app
12. From downloading to installing itms-services://?action=download- manifest&url=https%3A%2F%2Flocalhost%3A300 0%2Frails%2Factive_storage%2Fblobs%2F[sha] %2Fmanifest.plist%3Fdisposition%3Dattachment 1. Extract URL and metadata from the package (app name, version, minimum OS version, …) 2. Create and upload a "manifest.plist" file with all these properties in XML format 3. Share a link to the manifest, not the package
13. Taking inspiration from ImageAnalyzer # app/models/active_storage/attachment.rb class Attachment < ActiveRecord::Base after_create_commit { AnalyzeJob.perform_later(blob) } end # app/jobs/active_storage/analyze_job.rb class AnalyzeJob < BaseJob def perform(blob) # Pick an analyzer based on the blob content-type analyzer = Analyzer::ImageAnalyzer.new(blob) blob.update! metadata: analyzer.metadata end end # lib/active_storage/analyzer/image_analyzer.rb class Analyzer::ImageAnalyzer < Analyzer def metadata download_blob_to_tempfile do |file| image = MiniMagick::Image.new(file.path) { width: image.width, height: image.height } end end end 1. Whenever a model with an attachment is created, Rails enqueues a background job 2. The job looks for the right analyzer for the blob. Metadata returned by analyzer is stored in the DB 3. Analyzers run in a background job, so they must download the file to memory first. The file analysis is typically delegated to external libraries (MiniMagick, FFMpeg, …)
14. Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job
15. Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job what?
16. Extracting metadata from a package # lib/cfpropertybundle.rb class CFPropertyBundle class Error < StandardError; end def initialize(path) @path = path end def properties Zip::File.open(@path) do |package| data = extract_plist_data_from package { display_name: data['CFBundleDisplayName'], identifier: data['CFBundleIdentifier'], version: data['CFBundleVersion'], minimum_os_version: data['MinimumOSVersion'], short_version: data['CFBundleShortVersionString'] } end rescue Zip::Error, SystemCallError => e raise Error, e.message end end The CFPropertyBundle library unzips the package, parses the information stored in XML format and returns a Hash with the properties needed to generate a manifest. If the package is not a Zip file or the content doesn’t match the .ipa format, a custom CFProperty::Error is raised.
17. Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job why?
18. Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job the manifest can be generated here
19. Processing an attachment # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) metadata = PackageAnalyzer.new(build.package.blob).metadata build.package.blob.update! metadata: metadata Tempfile.open ['manifest', '.plist'] do |file| file.write <<~MANIFEST <?xml version="1.0" encoding="UTF-8"?>[…] <key>url</key><string>#{CGI.escapeHTML build.package.service_url}</string> <key>title</key><string>#{metadata[:bundle_display_name]}</string> <key>bundle-version</key><string>#{metadata[:short_version]}</string>[…] MANIFEST build.manifest.attach io: File.open(file.path), filename: 'manifest.plist' end end end # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package has_one_attached :manifest end Attaching the manifest to the build grants access to all the convenient Active Storage methods (attach, rails_blob_url, …)
20. From downloading to installing itms-services://?action=download- manifest&url=https%3A%2F%2Flocalhost%3A300 0%2Frails%2Factive_storage%2Fblobs%2F[sha] %2Fmanifest.plist%3Fdisposition%3Dattachment manifest_url = rails_blob_url( build.manifest, disposition: 'attachment') <%= link_to 'Install', "itms-services:// ?action=download-manifest &url=#{CGI.escape manifest_url)}" %>
21. Part 3: validating the content of an attachment (5 minutes)
22. Validating the content of an attachment
23. Validating the content of an attachment # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties # extracts properties from the iOS bundle { display_name: "iOS app", identifier: "com.example.app", version: "42", minimum_os_version: "8.0", short_version: "1.0" } end end end
24. Validating the content of an attachment
25. Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base validates :version, presence: true, uniqueness: true end
26. Validating the content of an attachment # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties # extracts properties from the iOS bundle { display_name: "iOS app", identifier: "com.example.app", version: "42", minimum_os_version: "8.0", short_version: "1.0" } end end end
27. Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
28. Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
29. Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
30. Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end Started POST "/builds" Parameters: {"build"=>{"package"=>#<ActionDispatch::Http::UploadedFile:47e0 @tempfile=#<Tempfile:/var/t14t7.ipa>}
31. Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
32. Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
33. Summary 1. Uploading and downloading with Active Storage has_one_attached, rails_blob_url 2. Extracting metadata and processing attachments download_blob_to_tempfile, prepending callbacks 3. Validating the content of an attachment attachment_changes, attached?
34. Thanks! Questions?