Migrating from EC2 to Atlas

MongoDB's Atlas offering greatly simplifies the operational tasks of MongoDB clusters. But a lot of companies already have infrastructure they've created in EC2.

We're going to cover how to move from the EC2 infrastructure to MongoDB Atlas.

This was presented at MongoDB World 2017

Transcript

1. Migrating from EC2 to Atlas
2. ! Jesse Dearing Lead Site Reliability Engineer
3. InVisionApp
5. MongoDB @ InVision • Chef to manage EC2 instances and MongoDB • 28 MongoDB replica sets across 4 environments • 1500-2000 documents per second retrieved, 300-600 documents per second created or modified
6. MongoDB @ InVision • Powers various features of our application • We find it useful for when a document model makes the most sense • Example of this is Inspect
7. Starting out with Mongo on AWS • Provision EC2 instances • Chef, Puppet, Ansible, or manually install and configure MongoDB • Set it up as a replica set • Maybe even sharded
9. You still have to take care of • Backups • Monitoring • Alerting • Security • Updates
11. Why? This is what I do • Your job is to maintain MongoDB infrastructure • What are the hiring projections for your team? • At InVision we believe in multiplying the efforts of our teams while growing them at a linear growth (a la SRE)
12. Classic Build vs Buy Proposition DevOps edition • You and your team are smart and capable, of course you can build it!
13. Classic Build vs Buy Proposition DevOps edition • Do you have the time to maintain the new infrastructure? • Do you want to answer the pager for when your EC2 instances have issues?
14. Why we switched to Atlas • Easier management of the aforementioned reasons • Easy to use REST API • Graphical dashboards
16. How do we get on Atlas?
17. Preparation • Atlas requires SSL • Set up AWS VPC peering • VPN and Security setup • Make sure we were using Amazon DNS
18. Preparation • Pick a network CIDR that won’t collide with your networks • We have multiple VPCs for different environments so we picked something like 10.200.0.0/16 • Assess current instances I/O, CPU, and Memory and figure out how they can map to the Atlas instance types • Make sure you don’t have any locale requirements for your Mongo data
19. Preparation • Need to be on MongoDB 3.x using WiredTiger • Test on replicas and use testing/staging environments
20. Migration • Simple • Just write a simple daemon in C++ that tails and parses the oplog and replay it in your Atlas MongoDB cluster
21. Just Kidding
22. Atlas Live Migration • MongoDB announced live migration to Atlas in March • Built into the web UI • Or you can use mongomirror
24. Create Atlas cluster
25. Create Instance to run mongomirror
26. Perform initial sync
28. Stream oplog to sync
29. Cut over application
30. Post Migration • Set up alerts • Start with a low watermark until you have a good baseline after a week or so • Update runbooks and test backup restores
31. Where we’ve gone • With a good API we’ve found that we can automate provisioning • Self service integration into provisioning • Deploying a new service? Want some MongoDB with that?
32. Automation • The REST API is pretty easy to use • We’ve incorporated it into our Ansible scripts • Source at github.com/jessedearing/ansible- mongodb-atlas
33. Using Ansible to easily add new Mongo Replicasets 1 --- 2 - hosts: mongodb_atlas 3 connection: local 4 roles: 5 - role: mongodb_atlas 6 clusters: 7 - name: staging-mongo-loginapi 8 size: M30 9 state: present 10 - name: prod-mongo-loginapi 11 size: M30 12 state: present
34. Using Ansible to easily add new Mongo Replicasets 1 --- 2 - hosts: mongodb_atlas 3 connection: local 4 roles: 5 - role: mongodb_atlas 6 clusters: 7 - name: staging-mongo-loginapi 8 size: M30 9 state: present 10 - name: prod-mongo-loginapi 11 size: M30 12 state: present 1 --- 2 - hosts: mongodb_atlas 3 connection: local 4 roles: 5 - role: mongodb_atlas 6 clusters: 7 - name: staging-mongo-loginapi 8 size: M30 9 state: present 10 - name: prod-mongo-loginapi 11 size: M30 12 state: present 13 - name: test-mongo-loginapi 14 size: M10 15 state: present
35. Go give Atlas a shot • Requires MongoDB SSL • Pick your region and a network CIDR that doesn’t collide with your existing CIDR • mongomirror or live migrate • Leave running MongoDB to the experts
36. Thanks!
37. Questions?