MongoDB Schema Evolution and Data Migration

MongoDB has arrived in the industry and runs for years in production systems now. But schema evolution and schema maintenance is neccessary. In his talk, Timmo will present solutions for (Java) Developers to get theire data into MongoDB and make data migration explicit, straight-forward and testable. You will learn: - how to map your Java domain model to a schema - how you can support model and schema evolution - that TDD is very easy with MongoDB - why MongoDB supports continuous deployment

Transcript

1. Schema Evolution and Data Migration { "name" : "Timmo Freudl-Gierke", "twitter" : "@timmo_gierke", "email" : "timmo.gierke@hypoport.de" }
6. Schema / JSON Structure { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } Application Domain Model Speaker name company street zipCode city
8. Speaker name company street zipCode city Speaker firstName lastName Company name street zipCode city 0,1 Speaker firstName lastName company street zipCode city Version 1 Version 2 Version 3
9. Speaker name company street zipCode city Speaker firstName lastName company street zipCode city Version 1 Version 2 db.speakers.find({}).forEach( function(speaker) { var original = speaker.name var first = original.match(/[A-Za-z]+/)[0] var last = original.match(/[A-Za-z-]+$/)[0] speaker.firstName = first speaker.lastName = last db. speakers.save(speaker) })
10. Speaker firstName lastName Company name street zipCode city 0,1 Speaker firstName lastName company street zipCode city Version 2 Version 3 db.speakers.update({},{$rename:{"company":"company.name"}}) db.speakers.update({},{$rename:{"street":"company.street"}}) db.speakers.update({},{$rename:{"zipCode":"company.zipCode"}}) db.speakers.update({},{$rename:{"city":"company.city"}}) http://docs.mongodb.org/manual/reference/operator/rename/
11. Speaker firstName lastName Company name street zipCode city 0,1 Speaker firstName lastName company street zipCode city Version 2 Version 3 DBObject speaker = ... DBObject company = new BasicDBObject(); company.put("name", speaker.get("name")); company.put("street", speaker.get("street")); company.put("zipCode", speaker.get("zipCode")); company.put("city", speaker.get("city")); speaker.put("company", company); speaker.removeField("street"); speaker.removeField("zipCode"); speaker.removeField("city"); collection.save(speaker);
12. @Test public void companySeperation() { // given String json = getFileAsJson("speaker_v2.json") DBObject speaker = (DBObject) JSON.parse(json); // when converter.seperateCompany(speaker); // then assertEquals(speaker.get("company.name"), "Hypoport AG"); assertFalse(speaker.containsField("street"); . . . } { firstName : "Timmo", lastName : "Freudl-Gierke" company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } speaker_v2.json Unit Testing
14. Version 1 Version 2 Version 3 MongoDB 1. serialize() 2. addModelVersion() 3. save()
15. MongoDB speakers {name: "Timmo Freudl-Gierke ", modelVersion:1, ...} {name: "Joe Morrissey", modelVersion:1, ...} {name: "Stefan Rudnitzki ", modelVersion:2, ...} {name: "Christian Kvalheim ", modelVersion:1, ...} {name: "Ross Lawley ", modelVersion:2, ...} {name: "Thomas Rueckstiess ", modelVersion:3, ...} {name: "Johannes Hoppe ", modelVersion:3, ...} . . .
16. Version 1 Version 2 Version 3 MongoDB 1. find() 2. convert() 3. deserialize()
18. { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } Multiple Version Converter Integration Test speaker_v1.json speaker_v2.json speaker_v3.json src/test/resources/
19. Multiple Version Converter Integration Test src/test/java/ @DataProvider public Object[][] allVersions() { return new Object[][] { {"speaker_v1.json"}, {"speaker_v2.json"}, {"speaker_v3.json"}, }} @Test public void deserializationOfAllVersion(String filename) { String json = getFileAsJson(filename) DBObject speaker = (DBObject) JSON.parse(json); speakerConverter.convert(speaker); Speaker s = deserializeSpeaker(speaker); // assert structure and relevant attributes . . . }
20. speakers = collection.find(queryBuilder().put("company").is("Hypoport AG")); speakers.addAll( collection.find(queryBuilder().put("company").is("Hypoport AG"))); speakers.addAll( collection.find(queryBuilder().put("company.name").is("Hypoport AG"))); Queries
21. SpringData-Integration @Component public class SpeakerMongoEventListener extends AbstractMongoEventListener<Speaker> { @Inject SpeakerVersionConverter versionConverter; @Override public void onBeforeSave(Speaker source, DBObject dbo) { setModelVersion(dbo); } @Override public void onAfterLoad(DBObject dbo) { versionConverter.convert(dbo); } }
22. Testing speakers {name: "Timmo Freudl-Gierke ", modelVersion: {name: "Joe Morrissey", modelVersion:1, ...} {name: "Stefan Rudnitzki ", modelVersion:2, ...} {name: "Christian Kvalheim ", modelVersion:1, ...} {name: "Ross Lawley ", modelVersion:2, ...} {name: "Thomas Rueckstiess ", modelVersion:3, ...} {name: "Johannes Hoppe ", modelVersion:3, ...} . . . $rename { name : "Timmo Freudl-Gierke", company : "Hypoport AG" street : "Klosterstr. 71", zipCode : "10179", city : "Berlin" } Speaker name company street zipCode city Text
23. { "name" : "Timmo Freudl-Gierke", "twitter" : "@timmo_gierke", "email" : "timmo.gierke@hypoport.de" } Schema Evolution and Data Migration