Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoDB and REST APIs A Match Made in Heaven

MongoDB and REST APIs A Match Made in Heaven

Exposing MongoDB over the internet through a RESTful API is becoming a common pattern, and for very good reasons. Your REST API provides a nice layer of abstraction, isolation and validation above the actual datastore while providing access to all kind of clients. At the same time MongoDB, with its BSON store, is ideal for serving data over the Internet. But designing first, and then building a robust and scalable REST API is not an easy task.

EVE is an open source framework allowing anyone to effortlessly expose MongoDB data sources over highly customizable, fully featured RESTful Web Services. Initially developed to solve our own internal use case, EVE has been getting a lot of traction since its open source release in 2012.

In this talk I will go through the reasons that make MongoDB a good match for most RESTful APIs. I will then show some prominent EVE features and illustrate how one can quickly and easily go online with a RESTful MongoDB front-end. I also plan to illustrate some common pitfalls on MongoDB+EVE/REST deployments.

Nicola Iarocci

October 16, 2015
Tweet

More Decks by Nicola Iarocci

Other Decks in Technology

Transcript

  1. Nicola Iarocci Open Source junkie Eve • Cerberus • Events

    • Flask-Sentinel • Eve.NET • Etc.
  2. What we need #2 Abstract the data access layer so

    we can update/replace the engine at any time with no impact on clients
  3. Where we want to go Clients “Cloud” Database RESTful Web

    API API iOS Android Website Desktop Client ? ?
  4. Constraints • minimum viable product first • add features over

    time • frequent database schema updates • easily scalable • avoid downtime as much as possible • cater upfront for a microservices architecture
  5. Goals #1 and #2 are met REST layer allows all

    kinds of client technologies and abstracts the data away
  6. JSON & RESTful API JSON accepted media type Client JSON

    (BSON) Mongo GET maybe we can push directly to client?
  7. JSON & RESTful API JSON accepted media type Client JSON

    (BSON) Mongo JSON subset of python dict (kinda) API GET almost.
  8. JSON & RESTful API JSON objects Client JSON (BSON) Mongo

    JSON/dict maps to python dict (validation layer) API POST also works when sending data the database
  9. What about Queries? Queries in MongoDB are represented as JSON-style

    objects db.things.find({x: 3, y: "foo”});
  10. Filtering and Sorting native Mongo query syntax Client JSON (BSON)

    Mongo (very) thin parsing & validation layer API Expose the native MongoDB syntax? ?where={“x”: 3, “y”: “foo”}
  11. No need for ORM No need to map objects to

    JSON and vice-versa (win!)
  12. Ideal API Surface Mongo collection maps to API resource endpoint

    api.example.com/contacts Maps to a Mongo collection
  13. Ideal API Surface Mongo document maps to a API document

    endpoint api.example.com/contacts/4f46445fc88e201858000000 Maps to a collection ObjectID
  14. Eve REST API for Humans™ Free and Open Source Powered

    by MongoDB and Good Intentions eve
  15. settings.py # just a couple API endpoints with no custom

    schema or rules. DOMAIN = { ‘people’: {} ‘works’: {} }
  16. enjoy HATEOAS AT WORK HERE $ curl http://localhost:5000/people { "_items":

    [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }}
  17. enjoy CLIENTS CAN EXPLORE THE API PROGRAMMATICALLY $ curl http://localhost:5000/people

    { "_items": [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": “home”} }, "_meta": { "max_results": 25, "total": 0, "page": 1 }}
  18. $ curl http://localhost:5000/people { "_items": [], "_links": { "self": {"href":

    "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }} enjoy AND EVENTUALLY FILL THE UI
  19. enjoy $ curl http://localhost:5000/people { "_items": [], "_links": { "self":

    {"href": "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }} PAGINATION DATA
  20. enjoy EMTPY RESOURCE AS WE DID NOT CONNECT A DATASOURCE

    $ curl http://localhost:5000/people { "_items": [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }}
  21. settings.py # let’s connect to a mongo instance MONGO_HOST =

    'localhost' MONGO_PORT = 27017 MONGO_USERNAME = 'user' MONGO_PASSWORD = 'user' MONGO_DBNAME = 'percona'
  22. settings.py # let’s also add a few validation rules DOMAIN['people']['schema']

    = { 'name': { 'type': 'string’, 'maxlength': 50, 'unique': True}, 'email': { 'type': 'string', 'regex': '^\S+@\S+$'}, 'location': { 'type': 'dict', 'schema': { 'address': {'type': 'string'}, 'city': {'type': 'string’}}}, 'born': {'type': 'datetime'}} UNIQUE STRING, MAX LENGTH 50
  23. settings.py # let’s also add a few validation rules DOMAIN['people']['schema']

    = { 'name': { 'type': 'string', 'maxlength': 50, 'unique': True}, 'email': { 'type': 'string', 'regex': '^\S+@\S+$'}, 'location': { 'type': 'dict', 'schema': { 'address': {'type': 'string'}, 'city': {'type': 'string’}}}, 'born': {'type': 'datetime'}} ONLY ACCEPT VALID EMAILS
  24. settings.py # let’s also add a few validation rules DOMAIN['people']['schema']

    = { 'name': { 'type': 'string’, 'maxlength': 50, 'unique': True}, 'email': { 'type': 'string', 'regex': '^\S+@\S+$'}, 'location': { 'type': 'dict', 'schema': { 'address': {'type': 'string'}, 'city': {'type': 'string’}}}, 'born': {'type': 'datetime'}} THIS REGEX SUCKS!
 DON’T USE IN PRODUCTION
  25. settings.py # let’s also add a few validation rules DOMAIN['people']['schema']

    = { 'name': { 'type': 'string', 'maxlength': 50, 'unique': True}, 'email': { 'type': 'string', 'regex': '^\S+@\S+$'}, 'location': { 'type': 'dict', 'schema': { 'address': {'type': 'string'}, 'city': {'type': 'string’}}}, 'born': {'type': 'datetime'}} SUBDOCUMENT WITH 2 STRING FIELDS
  26. settings.py # let’s also add a few validation rules DOMAIN['people']['schema']

    = { 'name': { 'type': 'string’, 'maxlength': 50, 'unique': True}, 'email': { 'type': 'string', 'regex': '^\S+@\S+$'}, 'location': { 'type': 'dict', 'schema': { 'address': {'type': 'string'}, 'city': {'type': 'string’}}}, 'born': {'type': 'datetime'}} ONLY ACCEPT DATETIME VALUES
  27. settings.py # allow write access to API endpoints # (default

    is [‘GET’] for both settings) # /people RESOURCE_METHODS = ['GET','POST'] # /people/<id> ITEM_METHODS = ['GET','PATCH','PUT','DELETE'] ADD/CREATE ONE OR MORE ITEMS
  28. settings.py # allow write access to API endpoints # (default

    is [‘GET’] for both settings) # /people RESOURCE_METHODS = ['GET','POST'] # /people/<id> ITEM_METHODS = ['GET','PATCH','PUT','DELETE'] EDIT ITEM
  29. settings.py # allow write access to API endpoints # (default

    is [‘GET’] for both settings) # /people RESOURCE_METHODS = ['GET','POST'] # /people/<id> ITEM_METHODS = ['GET','PATCH','PUT','DELETE'] REPLACE ITEM
  30. settings.py # allow write access to API endpoints # (default

    is [‘GET’] for both settings) # /people RESOURCE_METHODS = ['GET','POST'] # /people/<id> ITEM_METHODS = ['GET','PATCH','PUT','DELETE'] YOU GUESSED IT
  31. settings.py CLIENT UI # a few optional config options DOMAIN['people'].update(

    { 'item_title': 'person', 'cache_control':'max-age=10,must-revalidate', 'cache_expires': 10, 'additional_lookup': { 'url’: 'regex("[\w]+")', 'field’: 'name'} } } )
  32. settings.py CLIENT CACHE OPTIONS # a few optional config options

    DOMAIN['people'].update( { 'item_title': 'person', 'cache_control':'max-age=10,must-revalidate', 'cache_expires': 10, 'additional_lookup': { 'url’: 'regex("[\w]+")', 'field’: 'name'} } } )
  33. settings.py ALTERNATE ENDPOINT # a few optional config options DOMAIN['people'].update(

    { 'item_title': 'person', 'cache_control':'max-age=10,must-revalidate', 'cache_expires': 10, 'additional_lookup': { 'url’: 'regex("[\w]+")', 'field’: 'name'} } } )
  34. GeoJSON support and validation for all GeoJSON types Point, LineString,

    Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometricalCollection
  35. standard request $ curl example.com/works/<id> {
 "title": "Book Title",
 "description":

    "book description",
 "author": “52da465a5610320002660f94"
 } RAW FOREIGN KEY (DEFAULT)
  36. request an embedded document $ curl example.com/works/<id>?embedded={“author”: 1} {
 "title":

    "Book Title",
 "description": "book description",
 "author": {
 “firstname”: “Mark”,
 “lastname”: “Green”,
 }
 } REQUEST EMBEDDED AUTHOR
  37. embedded document $ curl example.com/works/<id>?embedded={“author”: 1} {
 "title": "Book Title",


    "description": "book description",
 "author": {
 “firstname”: “Mark”,
 “lastname”: “Green”,
 }
 } # embedding is configurable on per-field basis and # can be pre-set by API maintainer EMBEDDED DOCUMENT
  38. request $ curl -d ‘ [ { "firstname": "barack", "lastname":

    “obama" }, { "firstname": "mitt", "lastname": “romney” } ]' -H 'Content-Type: application/json’ <url>
  39. response [ { "_status": "OK", "_updated": "Thu, 22 Nov 2012

    15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “<url>”, "title": "person"}} }, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “<url>", "title": "person"}} } ] COHERENCE MODE OFF: ONLY META FIELDS ARE RETURNED
  40. response [ { "_status": "OK", "_updated": "Thu, 22 Nov 2012

    15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “<url>”, "title": “person”}}, "firstname": "barack", "lastname": "obama", }, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “<url>", "title": "person"}} "firstname": "mitt", "lastname": "romney", } ] COHERENCE MODE ON: ALL FIELDS RETURNED
  41. index maintenance define sets of resource indexes to be (re)created

    at launch supports sparse, geo2d and background indexes
  42. run.py from eve import Eve app = Eve() def percona(resource,

    response): documents = response['_items'] for document in documents: document['percona'] = 'is so cool!' if __name__ == '__main__': app.on_fetched_resource += percona_live app.run() CALLBACK FUNCTION
  43. run.py from eve import Eve app = Eve() def percona(resource,

    response): documents = response['_items'] for document in documents: document['percona'] = 'is so cool!' if __name__ == '__main__': app.on_fetched_resource += percona_live app.run() LOOP ON ALL DOCUMENTS
  44. run.py from eve import Eve app = Eve() def percona(resource,

    response): documents = response['_items'] for document in documents: document['percona'] = 'is so cool!' if __name__ == '__main__': app.on_fetched_resource += percona_live app.run() INJIECT FIELD
  45. run.py from eve import Eve app = Eve() def percona(resource,

    response): documents = response['_items'] for document in documents: document['percona'] = 'is so cool!' if __name__ == '__main__': app.on_fetched_resource += percona_live app.run() ATTACH CALLBACK TO EVENT HOOK
  46. $ curl -i <url> HTTP/1.1 200 OK X-RateLimit-Limit: 1 X-RateLimit-Remaining:

    0 X-RateLimit-Reset: 1390486659 rate limited request CURRENT LIMIT
  47. $ curl -i <url> HTTP/1.1 200 OK X-RateLimit-Limit: 1 X-RateLimit-Remaining:

    0 X-RateLimit-Reset: 1390486659 rate limited request REMAINING
  48. $ curl -i <url> HTTP/1.1 200 OK X-RateLimit-Limit: 1 X-RateLimit-Remaining:

    0 X-RateLimit-Reset: 1390486659 rate limited request TIME TO RESET
  49. XML support $ curl -H ”Accept: application/xml” -i http://example.com/people <resource

    href="people" title="people" > <link rel="parent" href="/" title="home" /> <resource href="people/55fc149138345b0880f07e3d" title="person" > <_created>Fri, 18 Sep 2015 13:41:37 GMT</_created> <_etag>5d057712ce792ebb4100b96aa98bfe9b6693c07b</_etag> <_id>55fc149138345b0880f07e3d</_id> <_updated>Fri, 18 Sep 2015 13:41:37 GMT</_updated> <email>[email protected]</email> <name>john</name> </resource> </resource>
  50. missing ETag # fails, as there is no ETag included

    with request $ curl \ -X PATCH \ -i http://example.com/people/521d6840c437dc0002d1203c \ -H "Content-Type: application/json" \ -d '{"name": “ronald"}' HTTP/1.1 403 FORBIDDEN NO ETAG REJECTED
  51. ETag mismatch # fails, as ETag does not match with

    server $ curl \ -X PATCH \ -i http://example.com/people/521d6840c437dc0002d1203c \ -H "If-Match: 1234567890123456789012345678901234567890" \ -H "Content-Type: application/json” \ -d '{"firstname": "ronald"}' HTTP/1.1 412 PRECONDITION FAILED ETAG MISMATCH REJECTED
  52. valid ETag # success at last! ETag matches with server

    $ curl \ -X PATCH \ -i http://example.com/people/50adfa4038345b1049c88a37 \ -H "If-Match: 80b81f314712932a4d4ea75ab0b76a4eea613012" \ -H "Content-Type: application/json" \ -d '{"firstname": "ronald"}' HTTP/1.1 200 OK # Like most of features, ETags can be disabled. ETAG MATCH ACCEPTED
  53. Goal # 4 achieved easy to setup, launch and scale

    up; also a good fit for microservices infrastracture
  54. A look back to initial draft Clients “Cloud” Database RESTful

    Web API API iOS Android Website Desktop Client ? ?
  55. Clients Multiple MongoDBs Database Adam eve instances API iOS Android

    Website Desktop Client what we have in production
  56. stories from the trenches #1. when too much (magic) is

    too much #2. sometimes you don’t want to debug #3. so how do I login into this thing?
  57. Enhance MongoDB with powerful features on top of native engine

    validation, document embedding (joins), referential integrity, document versioning, transformations, rate limiting, etc.
  58. Consider the REST layer as an ideal data access layer

    the story of pymongo 3.0 breaking changes mongo or sql or elastic or …
  59. Consider Microservices leverage Eve features create a network of isolated

    yet standardized services 
 each service has a dedicated role runs as an eve instance, with its own configuration has its own database(s) callbacks route traffic between services
  60. Clients User-reserved MongoDBs eve-multidb Data Auth eve-oauth2 (flask-sentinel) API iOS

    Android Website Desktop Client Adam 1 Adam eve instance Redis auth tokens, rate limiting Auth/Users MongoDB
  61. Clients service- reserved MongoDBs Data Auth eve-oauth2, flask-sentinel API iOS

    Android Website Desktop Client Adam 2 Adam eve instance Redis auth tokens, rate limiting Services eve instances Auth/Users MongoDB very simplified