Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Europython - Building your first app in mongoDB

rozza
July 05, 2012

Europython - Building your first app in mongoDB

A fast overview and review of mongoDB and python. Then some do's and don'ts to help get you started and enjoying mongoDB

rozza

July 05, 2012
Tweet

More Decks by rozza

Other Decks in Technology

Transcript

  1. Hello I'm Ross Lawley Work for 10gen Help maintain pymongo

    Maintain MongoEngine I love opensource and agile methodologies twitter: RossC0 http://github.com/rozza
  2. Before 10gen Dwight Merriman and Eliot Horowitz Double Click &

    Shopwiki -30 billion ads a day -Built multiple database caching layers
  3. 2007 10gen formed Originally to create a PAAS service MongoDB

    is only three years old 0.8 February 2009 First standalone release 1.0 August 2009 Simple, but used in production 1.2 December 2009 map/reduce, external sort index building 1.4 March 2010 Background indexing, geo 1.6 August 2010 Sharding, replica sets 1.8 March 2011 Journalling, sparse/covered indexes 2.0 September 2012 Compact, concurrency 2.2 July 2012 Concurrency, aggregation framework
  4. A Document database { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "Ross",

    date : ISODate("2012-07-05T10:00:00.000Z"), text : "About MongoDB...", tags : [ "tech", "databases" ], comments : [{ author : "Tim", date : ISODate("2012-07-05T11:35:00.000Z"), text : "Best Post Ever!" }], comment_count : 1 }
  5. In Python { '_id' : ObjectId("4c4ba5c0672c685e5e8aabf3"), 'author' : "Ross", 'date'

    : datetime.datetime(2012, 7, 5, 10, 0), 'text' : "About MongoDB...", 'tags' : [ "tech", "databases" ], 'comments' : [{ 'author' : "Tim", 'date' : datetime.datetime(2012, 7, 5, 11, 35), 'text' : "Best Post Ever!" }], 'comment_count' : 1 }
  6. Getting Started // Create a connection import pymongo conn =

    pymongo.Connection('mongodb://localhost:27017') // Connect to a database db = conn.tutorial // Or via a dictionary lookup db = conn['tutorial'] // Files for the db don't exist until you add data
  7. Adding data // Add some data db.my_collection.save({"Some": "data"}) // Insert

    - better, explicit db.my_collection.insert({"Hello": "Florence!"}) // Find data db.my_collection.find() <pymongo.cursor.Cursor at 0x25df850> // Return first that matches db.my_collection.find_one() {'_id': ObjectId('4ff4a5b0bb69331891000000'), 'Hello': 'Florence!'}
  8. Finding data // Query by example - pass in a

    dict db.my_collection.find({"score": 60}) // Operators $gt, $gte, $lt, $lte, $ne, $nin, // $regex, $exists, $not, $or.. db.my_collection.find({"score": {"$gte": 60, "$lte": 70}) // Sorting (1 ascending, -1 descending) db.my_collection.find().sort({"name": 1}) // Paginating db.my_collection.find().skip(5).limit(5)
  9. Updating data // Updating - beware! Replaces the document db.my_collection.update({"_id":

    123},{"score": 80}) // Use atomic updates. db.my_collection.update({}, {"$set": {"score": 80}) // Multi flag to update more than one db.my_collection.update({}, {"$set": {"x":"y"}, multi=True) // Upserts db.my_collection.update({"_id": 123},{"score": 80}, upsert=True)
  10. Indexes // Single field indexes db.scores.ensure_index('score') // Compound indexes db.scores.ensure_index([

    ('score', pymongo.ASCENDING), ('name', pymongo.DESCENDING)] ) // Geo indexes db.places.create_index([("loc", GEO2D)])
  11. Query plan db.scores.find().explain() {u'cursor': u'BasicCursor', u'indexBounds': {}, u'indexOnly': False, u'isMultiKey':

    False, u'millis': 1, u'n': 3000, u'nChunkSkips': 0, u'nYields': 0, u'nscanned': 3000, u'nscannedObjects': 3000, u'scanAndOrder': False, u'server': u'lucid64:27017'}
  12. Gridfs // Store files in mongoDB import gridfs fs =

    gridfs.GridFS(db) // Save file to mongo my_image = open('my_image.jpg', 'r') file_id = fs.put(my_image) // Read file fs.get(file_id).read()
  13. Lots of options Humongolus - pythonic and lightweight ORM MongoKit

    - ORM-like layer on top of PyMongo Ming - Developed by SourceForge MongoAlchemy - Inspired by SQLAlchemy MongoEngine - Inspired by the Django ORM Minimongo - lightweight, pythonic interface
  14. High availability Single master system - Primary always consistent Automatic

    failover if a Primary fails Automatic recovery when a node joins the set Full control over writes using write concerns Easy to administer and manage
  15. S S negotiate new master DOWN PRIMARY may fail Automatic

    election of new PRIMARY if majority exists
  16. Advanced features Durability via write concerns - On a connection,

    database, collection and query level - Tag nodes and direct writes to specific nodes / data centres Prioritisation - Prefer specific nodes to be primary - Ensure certain nodes are never primary Scaling reads - Not applicable for all applications - Secondaries can be used for backups, analytics, data processing
  17. EU LOCAL p:10 p:10 Backups / Analytics Server Primary Data

    Centre Example Durable Setup USA p:5 p:0 p:1
  18. Primary shard1 Horizontal scale out write read MongoD shard2 Secondary

    Secondary Primary shard3 Secondary Secondary Primary Secondary Secondary
  19. Durable and Scaled <Shard 1> priority: 10 <Shard 2> priority:

    5 <Shard 3> priority: 5 AZ-1 config server <Shard 1> priority: 5 <Shard 2> priority: 10 <Shard 3> priority: 5 AZ-2 config server <Shard 1> priority: 5 <Shard 2> priority: 5 <Shard 3> priority: 10 AZ-3 config server
  20. Bad things One size fits all collections are bad Unbounded

    arrays smell and don't perform Arrays that store all the data References everywhere Massive embedded tree structures
  21. Prove it Design schema upfront for large scale Everything scales

    well with no data Prove the schema works based on your usecase Performance test