• PHP + MySQL on Slicehost VPS • Fall 2009 – Winter 2009 (harryh joins) • Rewrite into Scala + PostgreSQL on EC2 • Winter 2010 – Spring 2010 (mongo) • Started to transition some tables from postgres to mongo. Write only. • Flipped the switch on venue data serving from mongo in April
• Slowly rewrite DB code one table at a time • Double write, throttle over reads • Auto balancing in Fall 2010 • Replica sets in May 2011 • Fall 2011 (Migration done) • Finally moved the most tangled table (users) • Summer 2012 (EC2 to bare metal)
volumes • EBS is a network service • Degrades to blocking all IO system calls • Many times per day with 100s of servers!! • User space programs are written with the possibility of very long blocking IO • Mongo replica set failover doesn’t see that as a failure mode
in userspace (FUSE) • Disk monitor on every mongod node • Periodically writes to a few sections of disk • Touches a kill-file on timeout • Modified mongo codebase to watch a kill-file • Secondary removes itself from slaveOK rotation • Primary steps itself down
so page faults are very expensive (a few milliseconds) • Required that most data be in RAM • Test hardware with SSDs that allowed us to fault to disk safely with only a fraction of data covered by RAM
a large capital expense • Buying a rack at a time gets us good deals • Amazon recently started direct connect option • Dedicated network links at 1Gbps increments • Low/Consistent latency to Equinix DC
RAID 10 on the drives, 360 GB • 4 replicas per server • Each shard limited to 60GB for easier maintenance operations (and disk limits) • Resyncs complete faster • Backups done in parallel for all shards
• No more than one replica per shard per server • Noticed that primary replicas eat up more resources than secondaries • Limit of one Primary per server
had to “warm” up a replica before sending it query traffic by paging all the data into memory • No need to worry about IO halting • Lower failure rate on machines • EC2 machines sometimes suffered from degraded network connectivity