Real World Data Modeling with Mongo

Learn how we model our data for Harmony in Mongo based on real-world experience.

1. Ordered List Steve Smith Mongo Chicago October 20, 2010 MongoDB & Harmony Real World Usage of MongoDB
2. Hello! steve@orderedlist.com @orderedlist
3. http://orderedlist.com
4. http://get.harmonyapp.com
5. Things to Cover 1. What is Harmony? 2. Multiple Item Types 3. Completely Custom Data 4. Images and Files 5. Activity Streams 6. Wrap Up
6. What is Harmony? Understanding Storage Needs
7. Building Websites Harmony is for
8. Be Flexible Harmony is designed to
9. Store Web Content Harmony is designed to
10. Demo Time
11. Things to Cover 1. What is Harmony? 2. Multiple Item Types 3. Completely Custom Data 4. Images and Files 5. Activity Streams 6. Wrap Up
12. Multiple Item Types Flexibility in Collections
13. Pages Harmony has
14. Blogs Harmony has
15. Blog Posts Harmony has
16. Blog Labels and Archives Harmony has
17. Many More Types... Harmony has
18. Have a URL All These Things
19. Single Collection Inheritance We Use
20. class Item include MongoMapper::Document key :_type, String, :index => true key :title, String, :required => true key :path, String, :required => true, :index => true key :template_name, String key :published_at, Time key :parent_id, ObjectId, :index => true key :parent_ids, Array, :index => true key :site_id, ObjectId, :required => true, :index => true key :nav_item, Boolean, :default => true key :asset_ids, Array, :index => true timestamps! userstamps! # more code here end
21. class Page < Item key :position, Integer, :default => 1 # more code here end
22. class Blog < Item key :position, Integer, :default => 1 key :labels_as, String, :default => 'tags' key :custom_permalink_structure, String # more code here end
23. class BlogPost < Item key :accepting_comments, Boolean, :default => true key :labels, Set, :index => true key :year_id, ObjectId key :month_id, ObjectId key :day_id, ObjectId # more code here end
24. Meaningful Keys Separation of Code into
25. Simple Querying One Collection Means
26. Things to Cover 1. What is Harmony? 2. Multiple Item Types 3. Completely Custom Data 4. Images and Files 5. Activity Streams 6. Wrap Up
27. Custom Data Unknown Possibilities
28. A Title and a Path Every Item in Harmony gets
29. Defined by the User All Other Data is
30. templates
31. templates Template 1 Template 2 Template 3
32. templates fields Template 1 Template 2 Template 3
33. templates fields Field 1 Field 2 Field 3 Template 1 Template 2 Template 3 Field 4 Field 5 Field 6 Field 7 Field 8 Field 9
34. Belongs Together This data is not related, it
35. templates Field 1 Field 2 Field 3 Template 1 Template 2 Template 3 Field 4 Field 5 Field 6 Field 7 Field 8 Field 9 Use Embedded Documents
36. items
37. items Item 1 Item 2 Item 3
38. items data Item 1 Item 2 Item 3
39. items data Data 1 Data 2 Data 3 Item 1 Item 2 Item 3 Data 4 Data 5 Data 6 Data 7 Data 8 Data 9
40. items Data 1 Data 2 Data 3 Item 1 Item 2 Item 3 Data 4 Data 5 Data 6 Data 7 Data 8 Data 9 templates Field 1 Field 2 Field 3 Template 1 Template 2 Template 3 Field 4 Field 5 Field 6 Field 7 Field 8 Field 9
41. class Template include MongoMapper::Document key :filename, String, :index => true key :theme_id, ObjectId, :index => true key :contents, String timestamps! userstamps! many :fields # more code here end
42. class Field include MongoMapper::EmbeddedDocument key :name, String, :required => true key :key, String, :required => true key :field_type_id, Integer, :required => true key :help_text, String key :settings, Hash key :required, Boolean embedded_in :template # more code here end
43. class Item include MongoMapper::Document key :_type, String, :index => true key :title, String, :required => true key :path, String, :required => true, :index => true key :template_name, String key :published_at, Time key :parent_id, ObjectId, :index => true key :parent_ids, Array, :index => true key :site_id, ObjectId, :required => true, :index => true key :nav_item, Boolean, :default => true key :asset_ids, Array, :index => true timestamps! userstamps! many :data # more code here end
44. class Datum include MongoMapper::EmbeddedDocument key :key, String, :required => true, :length => 2..100 key :file_upload, Boolean, :default => false key :value embedded_in :item # more code here end
45. Maximum Flexibility Embedding allows
46. Keeping Data Together Embedding allows
47. Things to Cover 1. What is Harmony? 2. Multiple Item Types 3. Completely Custom Data 4. Images and Files 5. Activity Streams 6. Wrap Up
48. Images and Files Binary Data Storage
49. Types of Files Harmony contains several
50. Text Files Harmony contains Stylesheets, JavaScripts, Includes, and Templates
51. Binary Files Harmony contains Images, File Uploads
52. Stored in Mongo All these are
53. class Stylesheet include MongoMapper::Document include PageCacheable key :filename, String, :index => true key :contents, String key :processor, String, :default => 'plain' key :theme_id, ObjectId, :index => true timestamps! userstamps! # more code here end
54. class StylesheetsController < ApplicationController caches_page :show def show render_not_found and return if params[:filename].blank? filename = File.basename(params[:filename].first, '.css') if stylesheet = Stylesheet.first(:filename => filename, :theme_id => params[:theme_id]) if stale?(:etag => stylesheet, :last_modified => stylesheet.updated_at.utc, :public => true) render :text => stylesheet.processed_contents, :content_type => 'text/css' end else render_not_found end end end
55. class Asset include MongoMapper::Document plugin Joint def self.versions @versions ||= { :feature => [:resize, {:width => 640}], :thumb => [:crop_resize, {:dimensions => [145, 75]}], :square => [:crop_resize, {:dimensions => [75, 75]}], :profile => [:crop_resize, {:dimensions => [100, 100]}], :profile_small => [:crop_resize, {:dimensions => [50, 50]}], } end key :title_tag, String key :description, String # long description timestamps! userstamps! attachment :file # more code here end
56. asset = Asset.new asset.file = params[:uploaded_file] asset.file.id asset.file.size asset.file.type asset.file.name
57. class FlyImages OriginalRegex = /^\/assets\/(.*)\/(.*)$/ # /assets/:id/name.ext VersionRegex = /^\/assets\/(.*)\/(.*)\/(.*)$/ # /assets/:id/:version/name.ext def initialize(app) @app = app end def call(env) case Rack::Request.new(env).path_info when VersionRegex id, version = $1, $2 serve_asset(id, version) when OriginalRegex id = $1 serve_asset(id) else @app.call(env) end end def serve_asset(id, version=nil) if asset = Asset.find(id) asset.page_cache(version) [200, {'Content-Type' => asset.content_type}, [File.read(asset.page_cache_path(version))]] else [404, {'Content-Type' => 'text/plain'}, ['File not found.']] end end end
58. Things to Cover 1. What is Harmony? 2. Multiple Item Types 3. Completely Custom Data 4. Images and Files 5. Activity Streams 6. Wrap Up
59. Activity Streams Quick Information
61. activities users items assets themes
62. activities users items assets themes
63. activities users items assets themes
64. activities users items assets themes
65. class Activity include MongoMapper::Document key :user, Hash key :source, Hash key :source_type, String key :action, String key :count, Integer, :default => 0 timestamps! end
66. class Activity include MongoMapper::Document key :user, Hash key :source, Hash key :source_type, String key :action, String key :count, Integer, :default => 0 timestamps! def source=(value) if value.is_a?(Hash) super else self.source_type = value.class.name super(value.to_mongo) end end def user=(value) if value.is_a?(User) super :id => value.id, :name => value.full_name else super end end end
67. class Activity # code on previous slide def self.article_created(article) create(:source => article, :user => article.creator, :action => 'create') end def self.article_updated(article) first_or_new(:action => 'update', :'source._id' => article.id, :created_at.gt => Time.zone.now.beginning_of_day.utc).tap do |a| a.count += 1 a.source = article a.user = article.updater a.save end end def self.article_published(article) create(:source => article, :user => article.updater, :action => 'publish') end def self.article_unpublished(article) create(:source => article, :user => article.updater, :action => 'unpublish') end end
68. Things to Cover 1. What is Harmony? 2. Multiple Item Types 3. Completely Custom Data 4. Images and Files 5. Activity Streams 6. Wrap Up
69. Wrapping Up Recapping the Main Points
70. Dynamic Keys in Collections Take advantage of
71. Simplify Relationships Use Embedding to
72. Simple & Useful Storing Binary Data is
73. Archiving and Query Reduction Embed Whole Objects for
74. Ordered List Thank you! steve@orderedlist.com Steve Smith Mongo Chicago October 20, 2010 @orderedlist