Databases and the Modern Web

A high-level walk through of data storage options for modern web applications and an introduction to using MongoDB. This talk was given at DevCon5 2013 in Los Angeles, CA.

1. brandonmblack.com @brandonmblack Software Engineer, MongoDB, Inc. Brandon Black DATABASES AND THE MODERN WEB
2. brandonmblack.com @brandonmblack +
3. brandonmblack.com @brandonmblack AGENDA • What is Big Data? • What is a Modern Web App? • A Brief History of the Web • Picking the Right Database • MongoDB’s Approach • Examples
4. What is Big Data?
5. brandonmblack.com @brandonmblack What is big data? Do we all mean the same thing? Velocity: the speed at which you’re dealing with data. Volume: the quantity of data, in sheer size. Variety: diversity of the data you’re dealing with. Fun facts about data growth: ! Photos: • 4 billion in the last year alone, 4x last decade • Half found their way onto the Internet ! Information: • 1.8 zettabytes annually (Source: IDC 2011) • Increase 50x by 2020
6. What is a Modern Web App?
7. brandonmblack.com @brandonmblack
8. A Brief History of the Web
9. brandonmblack.com @brandonmblack Late 90s • The web was just beginning to move beyond simple markup. • Most everything was happening in languages like Perl and C. • Relational databases were everywhere. • The future was bright!
10. brandonmblack.com @brandonmblack EARLY ‘00s • Crazy growth. • Dynamic page content starting to gain traction. • Dynamic languages start to gain traction. • LAMP stack dominates. • Relational databases.
11. brandonmblack.com @brandonmblack MID 00s • Social and multimedia redefines scale. • Memcache is king, but applications become more complex. • Problem specific frameworks start to emerge.
12. brandonmblack.com @brandonmblack PROBLEMS* • Data is growing dramatically in size with no end in sight. • Data complexity and application complexity is increasing. • Push for real-time and dynamic data limits using your cache layer as a crutch for scalability. ! * Side effects are not limited to, but may include: • Slower development cycles and loss of agility. • Complex architectures. • Over abstraction everywhere.
13. brandonmblack.com @brandonmblack TODAY’S TRENDS • Horizontal Scalability • Variety of Tools • Specialization no longer There is a one size fits all solution.
14. your data. The right database for
15. your development. The right database for
16. scale and performance. The right database for
17. brandonmblack.com @brandonmblack Relational • Two-dimensional storage! • In-place updates! • Query on any field! • Each contains a single value! • Very structured data! • Normalization leads to many tables and poor locality.! ! Examples:! Oracle, PostgreSQL, MySQL, MSSQL
18. brandonmblack.com @brandonmblack Key-value Stores • One-dimensional storage! • Single value blob! • Query on key only! • Some secondary indexes! • No schema! • No updates, replace value! ! Examples:! Cassandra, Redis, Memcache
19. brandonmblack.com @brandonmblack Document Database • N-dimensional storage! • Each field is 0, 1 or many or embedded values! • Query on any field at any level! • Flexible schema! • In-line updates! • Embedded data has great locality, better performance! ! Examples:! MongoDB, CouchDB, RethinkDB
20. MongoDB’s Approach
21. brandonmblack.com @brandonmblack MongoDB: Built to Scale •Document •Open-Source •High Performance •Horizontally Scalable •Full Featured
22. brandonmblack.com @brandonmblack Document Database • Not for PDF or DOC files • A document is essentially an associative array. • Document == JSON Object • Document == Ruby Hash • Document == PHP Array • Document == Python Dict
23. brandonmblack.com @brandonmblack Open-Source • MongoDB is Open-Source • Available on Github • Licensed under AGPL • Started and sponsored by MongoDB, Inc. • Contributions are welcome.
24. brandonmblack.com @brandonmblack High Performance • Written in C++ • Uses extensive memory-mapped files (ex. read-through and write-through memory caching). • Runs nearly everywhere. • Data is serialized as BSON (for fast parsing) • Full support for primary and secondary indexes. • Document model means less work, more agile.
25. brandonmblack.com @brandonmblack Shard N Shard 3 Shard 2 Shard 1 Horizontally Scalable
26. brandonmblack.com @brandonmblack Data Landscape Depth of Functionality Scalability & Performance Memcached MongoDB RDBMS
27. brandonmblack.com @brandonmblack Full Featured • Ad-hoc queries. • Real-time aggregation. • Auto-sharding. • Rich query capabilities. • Strongly consistent. • Geospatial features. • Parallel Processing and Targeted Queries. • Intelligent Replication. • Flexible schema.
28. brandonmblack.com @brandonmblack Native Support
29. brandonmblack.com @brandonmblack Native Support
30. Examples
31. brandonmblack.com @brandonmblack Installation 1 Download From website: http://www.mongodb.org/downloads sudo apt-get install mongodb-10gen brew install mongodb Package manager: 2 Startup MongoDB 3 Connect with the mongo Shell mongod —dbpath /path/to/data
32. brandonmblack.com @brandonmblack Using the Ruby Driver Secondary Secondary Primary Client Application Driver Write Read Read
33. Using the Ruby Driver # install gem and native extensions (optional)! gem install mongo gem install bson_ext require 'mongo'! include Mongo! ! # connecting to the database! client = MongoClient.new # defaults to localhost:27017! client = MongoClient.new('host1.example.com')! client = MongoClient.new('host1.example.com', 27017)! # using configuration options! opts = { :pool_size => 5, :pool_timeout => 5 }! client = MongoClient.new('host1.example.com', 27017, opts)!
34. Using the Ruby Driver seeds = ['h1.example.com:27017',! 'h2.example.com:27017',! 'h3.example.com:27017']! ! # connecting to a replica set! client = MongoReplicaSetClient.new(seeds)! client = MongoReplicaSetClient.new(seeds, :read => :secondary)! # connecting to a sharded cluster! client = MongoShardedClient.new(seeds)! client = MongoShardedClient.new(seeds, :read => :secondary)! # using a connection string! ENV['MONGODB_URI'] = 'mongodb://host1:27017?ssl=true'! client = MongoClient.new!
35. Using the Ruby Driver # using a database! db = client.db('blog')! db = client['blog']! ! client.drop_database('blog')! client.database_names! ! # using a collection! coll = db['posts']! ! coll.drop! db.collection_names!
36. brandonmblack.com @brandonmblack Terminology RDBMS MongoDB Table, View
    Collection Row
    Document Index
    Index Join
    Embedded Document Foreign Key
    Reference Partition
    Shard
37. brandonmblack.com @brandonmblack Blog Example • Models / Entities: • Users • Articles • Comments • Tags or Categories
38. brandonmblack.com @brandonmblack we would start by defining our schema. In a relational application
39. brandonmblack.com @brandonmblack Relational Diagram User ·Name ·Email address Category ·Name ·URL Comment ·Comment ·Date ·Author Article ·Name ·Slug ·Publish date ·Text Tag ·Name ·URL
40. brandonmblack.com @brandonmblack and we let the schema evolve as we go. In MongoDB we start building
41. brandonmblack.com @brandonmblack MongoDB Diagram User ·Name ·Email address Article ·Name ·Slug ·Publish date ·Text ·Author Comment[] ·Comment ·Date ·Author Tag[] ·Value Category[] ·Value
42. Inserting a Document # example document! author = {! :username => 'brandonblack',! :first => 'brandon',! :last => 'black'! }! ! # inserting into my blog database! client['blog']['users'].insert(author) No database or collection creation required!
43. Inserting a Document # example document! article = {! :title => 'Hello World'! :username => 'brandonblack',! :tags => ['ruby', 'getting started'],! :comments => [ # embedded docs ]! }! ! ! # inserting into my blog database! client['blog']['articles'].insert(article) No database or collection creation required!
44. Finding Documents coll = client['blog']['users']! ! # finding a single document! coll.find_one! coll.find_one({ :username => 'brandonblack' })! coll.find_one({ :username => 'brandonblack' }, { :first => 1})! ! coll = client['blog']['articles']! ! # finding multiple documents (using cursors)! cursor = coll.find({ :username => 'brandonblack' }, :limit => 10)! cursor.each do |article|! puts article['title']! end!
45. Updates and Deletes # updating a document! article = { :title => 'Hello Ruby' }! coll.update({ '_id' => article_id }, article)! coll.update({ '_id' => article_id },! {'$set' => {:title => 'Hello Ruby'}})! ! # deleting a single document! coll.remove({ '_id' => article_id })! ! # delete multiple documents! coll.remove({ 'username' => 'brandonblack' })! ! # delete all documents in a collection! coll.remove!
46. Indexes # running explain on a query! coll.find({ :username => 'brandonblack' }).explain! ! # showing collection indexes! coll.index_information! ! # adding an index! coll.ensure_index({ :username => 1 })! coll.ensure_index({ :username => Mongo::ASCENDING })! ! coll.ensure_index({ :username => -1 })! coll.ensure_index({ :username => Mongo::DESCENDING })! ! # adding a special index types! coll.ensure_index({ :loc => Mongo::GEO2D })! coll.ensure_index({ :title => Mongo::TEXT })! ! # droping an index! coll.drop_index('username_1')! ! # dropping all indexes for a collection! coll.drop_indexes!
47. ODM Example # rails setup! rails g mongoid:config! ! # non-rails setup! Mongoid.load!("path/to/your/mongoid.yml", :production)! ! # document examples! class Article! include Mongoid::Document! field :title, type: String! embeds_many :comments! end! ! class Comment! include Mongoid::Document! field :comment_text, type: String! embedded_in :article! end!
48. What’s Next?
49. very quickly. We’ve covered a lot
50. brandonmblack.com @brandonmblack Learn More Free Online Courses! http://education.mongodb.com! ! Events & Webinars! http://www.mongodb.com/events! ! Presentations! http://www.mongodb.com/presentations
51. brandonmblack.com @brandonmblack Learn More
52. brandonmblack.com @brandonmblack Talk with Us • Github • Stack Overflow • Google Group
53. brandonmblack.com @brandonmblack Software Engineer, MongoDB, Inc. Brandon Black Thank You Slides available at: ! http://github.com/brandonblack http://speakerdeck.com/brandonblack