Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Christopher Lozinski - ZODB, The Graph Database...

Christopher Lozinski - ZODB, The Graph Database for Python Developers

ZODB is a mature graph database written in Python and optimized in C. Just subclass off of class Persistent Object, and Persistent Container, and your objects, graphs and applications become persistent. This talk teaches you what you need to know to start using a pythonic graph database.

PyConWeb

July 17, 2018
Tweet

More Decks by PyConWeb

Other Decks in Programming

Transcript

  1. ZODB The Object-Graph Database for Python Developers By Christopher Lozinski

    asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #1
  2. Market Competition Product Language Funding Model Neo4j Java $80.1 Milion

    Graph OrientDB Java Graph, Document, KeyValue, Object ArrangpDB Java $7.52 Million Graph, Document, Key Value MarkLogic Java $173 Million XML JSON RDF Allegro Graph Lisp RDF ZODB Python $14 Million Persistetn Python Persistent Python Includes Graph, Document, Json, Key Value and Object. Not possible to write Zopache on top of statically bound Java. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #3
  3. Plone ZODB Users US Plone Sites • Federal Bureau of

    Investigation (FBI) • Central Intelligence Agency (CIA) • Intellectual Property Rights Center • US Department of Energy • USDA Forest Service • Fermi National Accelerator Lab (Fermilab) • NASA Science • Continental Airlines • UCLA • Yale University • Harvard • The Pennsylvania State University • University of Notre Dame • University of Virginia • University of California - Davis • University of North Carolina • University of Louisville • Novell • Akamai • Disney • eBay • Google • Walmart • Marriott • ...and many more. Worldwide Plone Sites • Brazilian Government • 2016 Olympics Brazil • The British Postal Museum and Archive • The New Zealand Treasury • Konica Minolta Printers - Australia • National Sports Council - Spain • National Library of South Africa • University of Oxford • University of Toronto • Academy of Performing Arts - Prague • Open Society Foundation • Amnesty International • OXFAM • Lufthansa • Nokia • Clean Clothes Campaign • RIPE • Cambridge University • Royal College of Surgeons • Oxford University Clinical Academic Graduate School • ... and many more asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #4
  4. Why Use a Graph Database? Social Network Computer Network asdf©

    Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #5
  5. Natural Language Processing Graphagus Sara follows Joe. Sara follows Ben.

    Sara likes bikes. Sara likes cars. Sara likes cats. Aria follows Joe. Maria loves Joe. Maria likes cars. Joe follows Sara. Joe follows Maria. Joe loves Maria. Joe likes bikes. Joe likes nature. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #7
  6. So Easy to use! import persistent class TreeLeaf(persistent.Persistent): def __init__(self,title=’’):

    self.title=title def render(self): return self.title asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #9
  7. ZODB Tutorial: Add Leaf Objects Single Leaf Object Multiple LEAF

    Objects #CREATE A SINGLE LEAF OBJECT leaf = TreeLeaf(‘Leaf’) root.leaf=leaf #CREATE MULTIPLE LEAF OBJECTS Leaf1 = TreeLeaf(‘Green Leaf’) leaf2 = TreeLeaf(‘Red Leaf’) #ADD THEM TO THE ROOT root['leaf1'] = leaf1 root['leaf2'] = leaf2 asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #10
  8. ZODB Tutorial: Create a Database http://www.zodb.org/en/latest/tutorial.html import ZODB, ZODB.FileStorage db

    = ZODB.db(‘Data.fs’) connection = db.open() root = connection.root #DO SOMETHING transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #11
  9. It is just Python (No SQL SELECT) #UPDATE THE Leaf

    root[‘leaf1’].title=”Yellow Leaf” transaction.commit() #STUPID QUERY For key, item in root.items(): print (key, item) # DELETE AN OBJECT del root[‘leaf1’] transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #12
  10. ZODB is Magical Creates the illusion that your Python Objects

    are Persistent asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #13
  11. ZODB is a graph databases #CREATE THE OBJECTS Leaf1 =

    TreeLeaf(‘Green Leaf’) leaf2 = TreeLeaf(‘Red Leaf’) #ADD THEM TO THE ROOT root['leaf1'] = leaf1 root['leaf2'] = leaf2 #IT IS A GRAPH DATABASE leaf1.sibling = leaf2 leaf2.sibling = leaf1 transaction.commit() asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #14
  12. Persistence by Reachability #BOTH OBJECTS STILL ACCESSIBLE del root['leaf1'] #BOTH

    OBJECTS GET GARBAGE COLLECTED del root['leaf2'] asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #15
  13. ZODB ACID Properties Atomicity: each transaction be "all or nothing".

    Consistency: any transaction will bring the database from one valid state to another. Isolation: the concurrent execution of transactions results in a system state that would be obtained if transactions were executed sequentially. Durability: once a transaction has been committed, it will remain so. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #16
  14. ZODB Advantages By Jim FUlton No Database Schema No ORM

    No Referential Integity Problems Automatic Garbage Collection No manual reads and writes asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #17
  15. ZODB Demo CRUD Views http://demo.pythonlinks.info/ Traditional Crud ZODB Extended Crud

    Create Read Update Create Read Update Delete Delete Rename Cut / Paste Copy / Paste History /Restore asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #22
  16. Create Read Update http://demo.pythonlinks.info/ TreeLeaf ITreeLeaf @implementer(ITreeLeaf) class TreeLeaf(persistent.Persistent): def

    __init__(self,title=’’): self.title=title def render(self): return self.title class ITreeLeaf(ILeaf): title = TextLine( title='Title', required=True) body = Text( title='Body', required=True) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #25
  17. Different Views on Different Ojects Plone, Pyramid, Grok asdf© Christopher

    Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #26
  18. Which CRUD is Allowed CRUD Interface Create IAddLeaf IAddBranch Read

    IDisplayable Update IEditable Delete IDeleteable Rename IRenameable Cut ICuttable Copy ICopyable Paste IPasteable History View IHistory asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #27
  19. Which CRUD Is Allowed? TreeLeaf Class ITreeLeaf Class @implementer(ITreeLeaf) class

    TreeLeaf(persistent.Persistent): def __init__(self,title=’’): self.title=title def render(self): return self.title class ITreeLeaf(IDisplayable, IEditable): title = TextLine( title='Title', required=True) body = Text( title='Body', required=True) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #28
  20. File Storage Objects are Written to the end of a

    File asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #32 Object 3 Object 2 Object 1 Object 4
  21. MQTT Pub/Sub For Real-Time Chat MQTT.org asdf© Christopher Lozinski CC

    BY-NC 3.0 US PythonLinks.info/zodb #34 Subscribers
  22. Storing Chat Logs asdf© Christopher Lozinski CC BY-NC 3.0 US

    PythonLinks.info/zodb #35 Object 1 Object 2 Object 3
  23. Chat Logs After Batching asdf© Christopher Lozinski CC BY-NC 3.0

    US PythonLinks.info/zodb #36 Object 1 Object 2 Object 3 Object 3 Version 2 Object 2 Version 2 Object 1 Version 2
  24. Speed by Jim Fulton 1000’s of Transactions per second For

    simple transactions relational databases are slightly faster For complex transactions ZODB is faster. Try to write 10 different clases, in ZODB just append to a file, an RDB requires 10 different writes. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #37
  25. Scalability Several hundred newspaper content-management systems and web sites were

    hosted using a multi-database configuration with most data in a main database and a catalog database. The databases had several hundred gigabytes of ordinary database records plus multiple terabytes of blob data. For larger systems move to NEO (neo.nexedi.com). Up to 80TB in production. 150TB (and growing) in test. Of course it takes time to move that much data around. asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #38
  26. s it a Property Graph Database? ZODB vs Relational Databases

    asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #41
  27. ZODB vs Relational Databases ZODB Relational Databases for item in

    node: print (item) You have to do a database join across every single table. 10 Tables: App, Category, City, Company, Country, iFrame Link, Job, Link, Region, Product, and Video. Zcatalog, Hypatia, repoze.catalog Select statement asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #42
  28. PythonLinks is a Content Aggregation System asdf© Christopher Lozinski CC

    BY-NC 3.0 US PythonLinks.info/zodb #44 PyCon UK PyData EuroPython PyCon US PyCon CZ YouTube SciPi PythonLinks. info
  29. Content Aggregation System Plone is a Content Management System. Blogory

    is also a Content Aggregation System asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #45
  30. Contact Information Follow @PythonLinks on Twitter Christopher Lozinski Http://PythonLinks.info EMail:

    [email protected] Twitter: @PythonLinks Skype: clozinski US Phone: +1 (650) 614 1836 EU Phone: +48 12 361 3136 asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #46
  31. Release Form Name:__________________ Email:__________________ Phone:__________________ I give permission to publish

    my name , ☐ or to process my professional information to help me get a better job , or to help ☐ me hire a smart Python developer . ☐ asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #47
  32. CREATING INDEXES Using Repoze.catalog 1) catalog = Catalog() 2) def

    get_area(object, default): return getattr(object, ‘area’, default) 3) catalog[‘area’] = CatalogFieldIndex(get_area) 4) leaf-1 = Leaf(area=20) 5) catalog.index_doc(1, leaf-1) 6) catalog.reIndex (1,leaf-1) 7) numdocs, results = catalog.query(Range(20,40)) asdf© Christopher Lozinski CC BY-NC 3.0 US PythonLinks.info/zodb #48