Neo4j Theory and Practice

GraphConnect 2013, London

Transcript

1. Neo4j Theory and Practice Tareq Abedrabbo Graph Connect - 19/11/2013
2. About me • CTO/Principal Consultant at OpenCredo • Working with Neo4j for (almost) 3 years on a number of different projects • Co-author of Neo4j in Action (Manning)
3. What is this talk about?
4. It’s for developers designing and building applications with Neo4j
5. It’s not a collection of war stories but I will refer to real-world examples
6. It is about sharing thoughts and lessons learnt in a useful way
7. “If I'm to believe Twitter, half of the earth's population are importing Wikipedia into Neo4j, for very obscure reasons.”
8. Agenda
9. • What is Neo4j? • Approaching graph-based applications • Design • Implementation • Test • Use cases • Lessons learnt
10. What really is Neo4j?
11. A graph model
12. A query engine
13. A database
14. Neo4j is a solid foundation on which to build graph- based applications
15. How should I approach graph-based applications?
16. Is there a useful way to categorise graph-based applications?
17. Domain-centric applications
18. Data-centric applications
19. Domain-Centric • Well-defined data model • Data changes through user interactions • Flexible but predictable data structure(s) • Recommendation engines, social networks, etc… • Top-down design
20. Data-Centric • Complex connected data that typically models real world networks • Integrated from a variety of different sources • Data can be unpredictable • Telco networks, utility networks, etc… • bottom-up design
21. Typically applications fall somewhere between these 2 types
22. How can I use the information available in my graph?
23. • Search and pattern-matching • Find a recommendation based on behaviour • Graph algorithms • Shortest path, disconnected components • Optimisation • Maximise oil flow while minimising water
24. Graphs are naturally data-driven
25. Use case 1: Network Impact Analysis
27. Requirement: Identify the impact of failing components
28. Requirement: Identify interesting patterns, such as single points of failure
29. Labelled property graph is a natural fit for the model
30. Additional “dimensions” can be added to capture abstract concepts: network redundancy, load-balancing
31. Cypher queries are a natural solution to delivering the different requirements
32. Use case 2: Oil flow optimisation
34. Requirement: Identify candidate configurations to maximise flow
35. Requirement: Identify the most practical and valuable adjustments to the network
36. Simply connected graph with complex components
37. Interlude: Genetic Algorithms
38. • Start from an initial population of candidate solutions (individuals or phenotypes), ideally random • Attribute a score each solution using a fitness function • The only place with specific business knowledge • Apply genetic operators to create a new generation • Cross-breeding to retain best characteristics from each parent • Mutation to maintain diversity and to avoid converging to a local optima too quickly • Stop when you want!
40. Is this even a use case for Neo4j?
41. Persist and share calculated solutions
42. Inspect intermediary steps
43. Use Cypher queries to interrogate solutions
44. Lessons learnt
45. Understand your domain
46. • Don’t follow “best practices” blindly • For domain-centric applications you can use a mapping framework, such as Spring Data Neo4j • For data-centric applications, you should stay as close as possible to the graph model • In any case, don’t try to hide the graph!
47. Use Cypher
48. ! • Expressive • Readable • Maintainable • Performant • Cypher + the web console is the quickest way to experiment and to prototype solutions
49. Manage complexity with domain knowledge
50. • Graph algorithms are typically complex • Knowledge of the domain can simplify queries and traversals • Make Cypher queries as specific as possible • Take “shortcuts” when you know the domain
51. Write robust and flexible code
52. • Break down problems into a small queries. Return graph resources (or ids) to chain queries. • Robustness principal: “Be conservative in what you do, be liberal in what you accept from others” • Use assertions as preconditions • Assertions document intent • Fail fast if data doesn’t match
53. Start with a representative dataset
54. • Create a small data sets to capture the initial use cases • Write simple unit tests using these datasets to support design and implementation • These tests tend to become less useful when requirements are better understood • Throw them away!
55. Move to a realistic dataset as soon as possible
56. • A realistic data set • Should capture the complexity of the real data • Should be sufficiently large • Ideally based on production data • Write functional and integration tests against this dataset
57. Test non-functional aspects
58. • Graph data is inherently flexible and evolving • Queries need to be correct and sufficiently performant • Existing queries’s performance can degrade as the underlying model changes • Assertions on timeouts should be part of the test suite to detect loops and poor performance • JUnit’s @Test(timeout=5) • Spring’s @Timeout(value=5)
59. Links • Twitter: @tareq_abedrabbo • Blog: http://www.terminalstate.net • OpenCredo: http://www.opencredo.com Thank you!