Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mnj — The MongoDB library which feels good

Mnj — The MongoDB library which feels good

MongoDB is the second most popular database in the World in 2019 (according to ScaleGrid).
MongoDB libraries in Python tend to use approaches from the SQL world or suffer from a lack of code clarity.
Let's have a look at the Mnj library, which aims to make Python and MongoDB the best friends.

Serge Matveenko

November 01, 2019
Tweet

More Decks by Serge Matveenko

Other Decks in Programming

Transcript

  1. • 28 years with coding • 12 years with Python

    • 15+ programming languages • FLOSS advocate • MongoDB fan :) • Software Architect @ Assaia About me Serge Matveenko github.com/lig twitter.com/lig1 assaia.com 2
  2. MongoDB mongodb.com Features: • Rich JSON Documents • Powerful query

    language • Aggregation Framework • Full ACID transactions • Support for joins in queries • Replication & Automatic Failover • Sharding & Location Segmentation • File Storage aka GridFS 5
  3. MongoDB & Python • PyMongo — main driver ◦ Direct

    MongoDB syntax mapping to dicts. ◦ Officially supported ◦ Has async derivative “Motor” • MongoEngine — most popular ODM ◦ Class to Collection mapping ◦ Django-like syntax ◦ Has async derivative “MotorEngine” • Other ◦ TxMongo (Twisted, PyMongo-like) ◦ MongoKit (alternative ODM, lack of maintenance, slow, was buggy) ◦ Some Django hacks, etc 7
  4. MongoEngine: almost as “good” as Django ORM class User(Document): name

    = StringField() class Page(Document): content = StringField() authors = ListField(ReferenceField(User)) ... Page(content="Test Page", authors=[bob, john]).save() Page.objects(authors__in=[bob]) Page.objects(id='...').update_one(pull__authors=bob) 9
  5. MongoEngine Pros: • Maps Classes to MognoDB Collections (ODM) •

    Short learning curve: uses Django-like syntax Cons: • Django-like syntax lacks the power of MongoDB Query Language • Django-like queries could become messy on complex sub-document queries • All MongoEngine power is lost when you go around to pure PyMongo
  6. PyMongo: everything is a dict import pymongo my_client = pymongo.MongoClient()

    my_database = my_client['my_database'] my_collection = my_database['my_collection'] my_collection.find_one({'author': "William Gibson", 'title': "Count Zero"}) 13
  7. PyMongo: …even when it’s not import pymongo my_client = pymongo.MongoClient()

    my_database = my_client.my_database my_collection = my_database.my_collection my_collection.find_one({'author': "William Gibson", 'title': "Count Zero"}) 14
  8. PyMongo: …better with some tweaking import pymongo my_client = pymongo.MongoClient()

    my_database = my_client['my_database'] my_collection: pymongo.collection.Collection = my_database['my_collection'] my_collection.find_one({'author': "William Gibson", 'title': "Count Zero"}) 15
  9. PyMongo: …results are still dicts book = my_collection.find_one( {'author': "William

    Gibson", 'title': "Count Zero"} ) book == { '_id': bson.ObjectId('5dae1a1d36fb82e1f046c8a3'), 'author': "William Gibson", 'title': "Count Zero", 'year': 2006, 'genre': ["Science Fiction", "Cyberpunk"], } 16
  10. PyMongo: …and it’s getting worse book = my_collection.find_one_and_update( {'_id': bson.ObjectId('5dae1a1d36fb82e1f046c8a3')},

    {'$push': {'genre': {'$each': ["Fiction"], '$position': 0}}}, projection={'genre': 1}, return_document=pymongo.ReturnDocument.AFTER, ) book == { '_id': bson.ObjectId('5dae1a1d36fb82e1f046c8a3'), 'genre': ["Fiction", "Science Fiction", "Cyberpunk"], } 17
  11. PyMongo: …and it’s getting worse book = my_collection.find_one_and_update( {'_id': bson.ObjectId('5dae1a1d36fb82e1f046c8a3')},

    {'$push': {'genre': {'$each': ["Fiction"], '$position': 0}}}, projection={'genre': 1}, return_document=pymongo.ReturnDocument.AFTER, ) book == { '_id': bson.ObjectId('5dae1a1d36fb82e1f046c8a3'), 'genre': ["Fiction", "Science Fiction", "Cyberpunk"], } 18
  12. Mnj github.com/lig/mnj Goals: • No strings attached(no pun intended) •

    Classes are friends • Dicts aren’t foes • Hide routine things • Do it the Python way • Use the power of PyMongo • Learn on SQLAlchemy, Peewee, Django ORM 21
  13. Mnj: starting to feel better. import nj book = my_collection.find_one(

    nj.q(author="William Gibson", title="Count Zero") ) book = my_collection.find_one_and_update( nj.q(_id=bson.ObjectId('5dae1a1d36fb82e1f046c8a3')), nj.push_(genre=nj.q(nj.each_(["Fiction"]), nj.position_(0))), ) 22
  14. Mnj: starting to feel better.. import nj book = my_collection.find_one(

    nj.q(author="William Gibson", title="Count Zero") ) book = my_collection.find_one_and_update( nj.q(_id=bson.ObjectId('5dae1a1d36fb82e1f046c8a3')), nj.push_(genre=nj.q(nj.each_(["Fiction"]), nj.position_(0))), ) 23
  15. Mnj: starting to feel better... import nj book = my_collection.find_one(

    nj.q(author="William Gibson", title="Count Zero") ) book = my_collection.find_one_and_update( nj.q(_id=bson.ObjectId('5dae1a1d36fb82e1f046c8a3')), nj.push_(genre=(nj.each_(["Fiction"]), nj.position_(0))), ) 24
  16. Mnj: now it feels good class Book(nj.Document): author: str title:

    str year: int genre: typing.List[str] book = Book.query(author="William Gibson", title="Count Zero").find_one() book = Book.query(genre=nj.in_("Cyberpunk")).find_one() 25
  17. Mnj: it feels good Book.query(author="William Gibson", title="Count Zero").find_one() Book.query.filter(title=nj.in_["Count Zero",

    "Neuromancer"]).find_one() Book.query.find_one({'author': "William Gibson", 'title': "Count Zero"}) Book.query(author="William Gibson", title="Count Zero").update_one( nj.push_(genre=(nj.each_(["Fiction"]), nj.position_(0))) ) 26
  18. Mnj: ODM how to class MyDoc(dict): def __new__(cls, *args, **kwargs):

    print("__new__", args, kwargs) return super().__new__(cls, *args, **kwargs) def __setitem__(self, key, value): print("__setitem__", key, value) super().__setitem__(key, value) MongoClient(document_class=MyDoc) 28
  19. Mnj: ODM under the hood class DocumentFactory(bson.raw_bson.RawBSONDocument, dict): def __new__(...):

    # Load the data from Raw BSON bytes # Figure out the namespace # Lookup the class we need in the registry # Build the instance return RegisteredDocumentClass(**data_loaded_from_bson) MongoClient(document_class=DocumentFactory) 29
  20. Mnj: the Client and the Client Manager import nj nj.create_client(db_name='books')

    nj.create_client(db_name='archive', name='archive') class OldBook(Book): class Meta: client_name = 'archive' 30
  21. Mnj: you don't have to if you don't want to

    my_client = nj.get_client(name='default') nj.create_client(db_name='archive', name='archive', document_class=dict) my_client = nj.get_client(name='archive') my_client = pymongo.MongoClient(document_class=nj.DocumentFactory) # Book._col: pymongo.collection.Collection Book._col.find_one(...) 31
  22. Mnj: Motor (asyncio) support Should work with: • Query builder

    • Operator functions • `Document._col` collection instance Not yet in: • Client Manager • `Document.query` object 32
  23. Mnj github.com/lig/mnj • Query Builder • Operators as functions •

    Object-Document Mapper • Client/Connection Manager • Document Factory for PyMongo ★ You don’t have to use everything 33
  24. Mnj github.com/lig/mnj Coming soon: • Smart operator helpers • Even

    smarter ODM queries • Server-side schema support • Schema migrations: on-read, on-modification, immediate • GridFS support • Motor/asyncio support • Flask & EVE integration • Other integrations • … • You use, you decide! 34