Neo4j-Migrations: The lean way of applying database refactorings to Neo4j

Neo4j-Migrations gives you an easy way to apply schema changes to Neo4j. Unlike Liquibase, It is almost dependency free and runs right on the Neo4j Java Driver, with no need to work with JDBC. Neo4j-Migrations integrates Spring Data Neo4j with JHipster and has been in production since summer 2020.

Neo4j-Migrations:

* Runs Cypher scripts
* Runs Java-based migrations
* Records all migrations in a simple graph
* Works with multiple databases
* Has extensions for Spring Boot (included in Neo4j Ops Manager) and Quarkus

The most important feature? Neo4j-Migrations comes as a native binary CLI, perfect for use in CI/CD systems.The CLI has everything you need to set up safe CI scenarios. Come to this session to learn how you can put Neo4j-Migrations to work for you.

Transcript

1. 1 Neo4j-Migrations: The lean way of refactoring Neo4j content By Michael Simons, Java Champion & Staff Software Engineer @ Neo4j
2. 2 It takes a village to raise a child
3. What this is not about • Migrating from one Neo4j version to another • Neo4j-OGM • Spring Data Neo4j (But I might speak about those things^^) 3
4. 4 What is this all about? You can’t have not a schema.
5. What can be part of a Neo4j-Schema • Constraints • Indexes • A consistent set of labels & types ◦ Consistent properties within them ◦ Consistent values for properties with the same name 5
6. 6 The only constant is change Requirements, data itself, processes etc…
7. History Refactoring Databases published in 2006. • Versioning of migrations ◦ Repeating ◦ Maybe undoing • Improve the database design • Unblock continuous delivery • Very focussed on relational databases • “schema on write” 7
8. Applicable concepts • Versioning of migrations (refactorings / changesets) ◦ Not applying them multiple times • Put them into actual version control • Checksumming ◦ Making sure changesets are not changed after the fact • Make them validatable
9. For what did we need a database refactoring tool? • SDN6 and JHipster collaboration back in late 2019 ◦ Need for a simple way to create a couple of Nodes and Relationships • We don’t want people to use Neo4j-OGM auto-index manager (automatically deriving indexes from the OGM schema and trying to get this right for more than 5 supported database versions) 9
10. Existing Java based tooling 10 • Liquibase (Relational) • Flyway (Relational) • Liquigraph (Graph) ◦ Now obsolete, became Liquibase-Neo4j-Plugin
11. Our requirements • Dependency free or as few as possible ◦ Target frameworks often have many things in place, don’t wanna bring them more dependency hell ◦ Connection layer of course necessary and ok • Not wrapping Neo4j, the bolt model, its sessions and concepts in JDBC • As little abstractions as possible • As little overhead as possible • Liquigraph / Liquibase-Neo4j felt too alien for Neo4j => Let’s create our own thing! https://github.com/michael-simons/neo4j-migrations 11
12. Very much Flyway inspired 󰠗 12
13. 13 For what do you might need a database refactoring tool? https://github.com/meistermeier/migrate-to-neo4j-50
14. History and feature overview • January 2020: ◦ 0.0.1 preview release for JHipster ◦ 0.0.4 full support for 4.0 multi database ◦ 0.0.8 Spring Boot starter (needed in JHipster) • February 2021: First external contributor • July 2021: Make validation optional, transactional functions everywhere, full AuraDB support • October / November 2021: Collaboration with JReleaser / Andres Almiray • November 2021: 1.0.0 release published with JReleaser • December 2021: Official Neo4j-Labs project Neo4j-Labs Project: https://neo4j.com/labs/neo4j-migrations/ • February 2022: Full support for Quarkus, including native mode • March 2022: ◦ SDKMan! Release ◦ Preconditions 14
15. History and feature overview 15 • June 2022: Catalogs ◦ Primary for indexes and constraints, more to come • August 2022: Built-In database refactorings ◦ Renaming labels / types ◦ Normalizing properties ◦ Merging nodes • September 2022: Annotation processor • November 2022: Repeatable migrations • Future: ◦ Undo for catalog items ◦ Undo with manual scripts
16. Distribution https://github.com/michael-simons/neo4j-migrations • Core API • CLI (native binaries for Linux, macOS, Windows, noarch (JVM) for everything) ◦ No plugins, no driver, just one binary! • Spring Boot Starter • Quarkus Extension • Maven Plugin Read more: Modules
17. Installation (CLI only) • Via SDKMan! since 1.5.1 sdk install neo4jmigrations • Via Homebrew for macOS • brew install michael-simons/homebrew-neo4j-migrations/neo4j-migrations Read more: Installation; Core-API, Quarkus Extension, Spring Boot Starter and Maven-Plugin are on Maven-Central!
18. How to run this? • Inside applications ◦ Only inventory data, please ◦ Moving data for tests (resources will be aggregated) ◦ Modern applications are usually running more than one instance: Which one should trigger migrations? • CI/CD (No JVM necessary, self-contained, native binary) Safe passwords (files or environment)
19. Our schema
20. Commands: clean Removes Neo4j-Migration specific data from the selected schema database help Display help information about the specified command. info Retrieves all applied and pending information, prints them and exits. init Creates a migration project inside the current folder. migrate, apply Retrieves all pending migrations, verify and applies them. run Resolves the specified migrations and applies them. Does not record any metadata. show-catalog Gets the local or remote catalog and prints it to standard out in the given format. validate Resolves all local migrations and validates the state of the configured database with them. Features / commands Feature parity for all distributions! 20
22. 22 Email michael.simons@neo4j.com Twitter @rotnroll666 Profile https://github.com/michael-simons Neo4j-Migrations https://neo4j.com/labs/neo4j-migrations Demo https://github.com/michael-simons/nodes2022 Thank you!