Building scalable applications with mongoDB

1. MongoDB Building scalable applications Ross Lawley 10gen 379
2. Origins of mongoDB > Who is 10gen > Design goals of mongoDB Data models in mongoDB > Flexible Schemas > Rich queries and atomic operations > Developer happiness and agility Scaling mongoDB > Replication > Disaster Recovery > Sharding scaling horizontally AGENDA
3. Origins
4. Founded in 2007 >Dwight Merriman, Eliot Horowitz >Doubleclick, Oracle, Marklogic, HP $74M+ in funding > Flybridge, Sequoia, Union Square, New Enterprise Associates Worldwide Expanding Team > 120+ employees > NY, CA, IE, UK, AUS Foster community ecosystem Provide MongoDB management services Provide commercial services Set the direction & contribute code to MongoDB
5. Cost of database increases > Vertical, not horizontal, scaling > High cost of SAN Scaling RDBMS is frustrating launch +30 Days +6 months +60 Days +1 year
6. > Needed to add new software layers of ORM, Caching, Sharding, Message Queue > Polymorphic, semi-structured and unstructured data not well supported Productivity decreases Project start Denormalize data model Stop using joins Custom caching layer Custom sharding
7. Evolution in computing Volume of Data > Trillions of records > 100's of millions of queries per second Agile Development > Iterative > Continuous deployment New Hardware Architecture > Commodity servers > Cloud Computing
8. JSON Documents > Rich data models > Seamlessly map to native programming language > Flexible for dynamic data > Better data locality Simplicity > Few configuration options > Does the right thing out of the box > Easy to deploy and manage General Purpose DBMS > Sophisticated secondary indexes > Dynamic queries > Sorting > Rich updates, upserts > Easy aggregation Scaling > Scale linearly > Increase capacity with no downtime > Transparent to the application MongoDB design goals
9. Developing
10. Developers already model to objects class Post(models.Model): author = models.CharField(max_length=250) title = models.CharField(max_length=250) body = models.TextField() date = models.DateTimeField('date') tags = models.ManyToManyField('Tag') comments = models.ManyToManyField('Comment') class Tag(models.Model): text = models.CharField(max_length=250) class Comment(models.Model): author = models.CharField(max_length=250) body = models.TextField() date = models.DateTimeField('date')
11. In a relational database post id author title body date id post_id tag_id post_tags id text tag id post_id comment_id post_comments id author body date comment 0..* 0..*
12. In mongoDB { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Ross", title : "Building scalable applications", body : "About MongoDB...", date : ISODate("2012-06-27T14:30:00.000Z"), tags : [ "tech", "databases" ], comments : [{ author : "Fred", date : ISODate("2012-06-27T14:35:00.000Z"), body : "Thanks, I'll look into it" }] }
13. Data model now fits your brain
14. Overview
15. Where can you use it? MongoDB is Implemented in C++ > Platforms 32/64 bit Windows, Linux, Mac OS-X, FreeBSD, Solaris Drivers are available in many languages 10gen supported > C, C# (.Net), C++, Erlang, Haskell, Java, JavaScript, Perl, PHP, Python, Ruby, Scala, node.js Community supported > Clojure, ColdFusion, F#, Go, Groovy, Lua, R ... http://www.mongodb.org/display/DOCS/Drivers
16. Terminology RDBMS MongoDB Table Collection Row(s) JSON Document Index Index Join Embedding & Linking Partition Shard Partition Key Shard Key
17. Flexible schemas > p = { author: "Ross", date: new Date(), text: "About MongoDB...", tags: ["tech", "databases"]} > db.posts.save(p)
18. Flexible schemas > db.posts.find() { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Ross", date : ISODate("2012-06-27T14:30:00.000Z"), text : "About MongoDB...", tags : [ "tech", "databases" ] } Notes: _id is unique, but can be anything you'd like
19. Introducing BSON JSON has powerful, but limited set of datatypes > arrays, objects, strings, numbers and null BSON is a binary representation of JSON > Adds extra data types with Date, Int types, Id, … > Optimised for performance and navigational abilities > And compression MongoDB sends and stores data in BSON
20. Finding data Conditional Operators - $all, $exists, $mod, $ne, $in, $nin, $nor, $or, $size, $type - $lt, $lte, $gt, $gte // find posts matching a regular expression > db.posts.find({author: /^ro*/i }) // find posts with any tags > db.posts.find({tags: {$exists: true }}) // count posts where "Ross" has commented > db.posts.find({comment.author: 'Ross'}).count()
21. Indexes > Create index on any field at any level in a document > Supports compound indexes > Can index arrays // Ensure index (1 ascending, -1 descending) > db.posts.ensureIndex({author: 1}) > db.posts.findOne({author: 'Ross'}) { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author: "Ross", ... }
22. Examine the query plan > db.posts.find({"author": 'Ross'}).explain() { "cursor" : "BtreeCursor author_1", "nscanned" : 1, "nscannedObjects" : 1, "n" : 1, "millis" : 0, "indexBounds" : { "author" : [ [ "Ross", "Ross" ] ] } }
23. Atomic Operations $set, $unset, $inc, $push, $pushAll, $pull, $pullAll, $bit // Create a comment > new_comment = { author: "Tim", date: new Date(), text: "Best Post Ever!"} // Add to post > db.posts.update({ _id: "..." }, {"$push": {comments: new_comment}, "$inc": {comments_count: 1} });
24. Rich documents { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Ross", date : ISODate("2012-06-27T14:30:00.000Z"), text : "About MongoDB...", tags : [ "tech", "databases" ], comments : [{ author : "Tim", date : ISODate("2012-06-27T14:35:00.000Z"), text : "Best Post Ever!" }], comment_count : 1 }
25. Geo-spatial support Geo-spatial queries > Require a geo index > Find points near a given point > Find points within a polygon/sphere // geospatial index > db.posts.ensureIndex({"author.location": "2d"}) > db.posts.find({ "author.location" : { $near : [22, 42] }}) [{ _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author: {location: [22, 43]}, ...}]
26. GridFS Save files in mongoDB Stream data back to the client // (Python) Create a new instance of GridFS >>> fs = gridfs.GridFS(db) // Save file to mongo >>> my_image = open('my_image.jpg', 'r') >>> file_id = fs.put(my_image) // Read file >>> fs.get(file_id).read()
27. Aggregation - coming in 2.2 Describe a chain of operations to apply to your data. // Count tags > agg = db.posts.aggregate( {$unwind: "$tags"}, {$group : {_id : "$tags", count : {$sum: 1}}} ) > agg.result [{"_id": "databases", "count": 1}, {"_id": "tech", "count": 1}]
28. Data modelling in mongoDB > Schema less – Storage relates to how you actually use the data – Inherently agile > Rich query language – For adhoc queries – Atomic updates at a document level > Flexible data models – Embed data or link documents depending on usage – Warning! Not all models perform at large scale
29. High availability
30. Replication features > Single master system - Primary always consistent > Automatic failover if a Primary fails > Automatic recovery when a node joins the set > Can be used to scale reads > Full control over writes using write concerns > Easy to administer and manage
31. Replica set is made up of 2 or more nodes How mongoDB replication works A B C
32. Election establishes the PRIMARY Data replication from PRIMARY to SECONDARY How mongoDB replication works S P S
33. PRIMARY may fail Automatic election of new PRIMARY if majority exists How mongoDB replication works S S negotiate new master DOWN
34. New PRIMARY elected Replica set re-established How mongoDB replication works S P DOWN
35. Automatic recovery How mongoDB replication works S P RECOVERING
36. Replica set re-established How mongoDB replication works S P S
37. Advanced replication features > Durability via write concerns – On a connection, database, collection and query level – Tag nodes and direct writes to specific nodes / data centers > Scaling reads – Not applicable for all applications – Secondaries can be used for backups, analytics, data processing > Prioritisation – Prefer specific nodes to be primary – Ensure certain nodes are never primary
38. Example Setup London Reading Cloud p:10 p:10 p:5 p:0 p:1 Backups / Analytics Server Primary Data Centre
39. Scaling
40. Horizontal scale out write read MongoD shard2 MongoD MongoD MongoD shard3 MongoD MongoD MongoD MongoD MongoD shard1
41. MongoDB sharding > Range based > Automatic partitioning and management > Convert to sharded system with no downtime > Fully consistent
42. How mongoDB sharding works Range keys from -∞ to +∞ Ranges are stored as "chunks" -∞ +∞ > db.runCommand({addshard: "shard1"}); > db.runCommand({shardCollection: "mydb.users", key: {age: 1}})
43. How mongoDB sharding works -∞ +∞ 41 +∞ -∞ 40 > db.users.save({age: 40})
44. How mongoDB sharding works -∞ +∞ 41 +∞ 51 +∞ -∞ 40 41 50 61 +∞ 51 60 > db.users.save({age: 40}) > db.users.save({age: 50}) > db.users.save({age: 60})
45. How mongoDB sharding works > db.users.save({age: 40}) > db.users.save({age: 50}) > db.users.save({age: 60}) -∞ +∞ 41 +∞ 51 +∞ -∞ 40 41 50 61 +∞ 51 60 shard1
46. How mongoDB sharding works > db.runCommand({addshard: "shard2"}); > db.runCommand({addshard: "shard3"}); 41 50 61 +∞ 51 60 shard1 -∞ 40 shard2 shard3
47. Architecture C1 C2 C3 Config Servers mongos mongos app app secondary Shard 1 secondary primary secondary Shard 2 secondary primary Shard 4 secondary Shard 3 secondary primary Replica Set secondary secondary primary
48. Use cases
49. There are many use cases User Data Management High Volume Data Feeds Content Management Operational Intelligence Product Data Management
50. Download & join in http://www.meetup.com/Swiss-MongoDB-User-Group/
51. Ross Lawley http://mongodb.org 10gen ross@10gen.com