Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Upgrading databases: without losing your data, ...

Upgrading databases: without losing your data, your perf, or your mind

from Data Day Texas in Austin, 2015-01-10

Charity Majors

January 10, 2015
Tweet

More Decks by Charity Majors

Other Decks in Technology

Transcript

  1. • Mobile backend • 500k+ apps • AWS • MongoDB,

    cassandra, mysql, redis • ruby & rails => golang
  2. why upgrade? • new features • better performance • better

    support from the vendor • avoid code rot, don’t get too far behind current • all of your cool friends have upgraded
  3. A Very Short List Of Terrible Things Database Upgrades Have

    Done To Me • 35% perf reduction, 60% perf reduction • data corruption (so many flavors) • db files deleted on startup. DELETED!! • indexing race conditions • invalid indexes bug causes collections to be unwritable • undocumented change in geoquery behavior • default storage format has 60% more bloat • backwards-incompatible mysql replication • storage format changes • all geo indexes block global lock until the first document found • undocumented query syntax changes • changed the definition of scan limits, doesn’t cache query plans that exceed scan limit • unindexable writes suddenly refused • internally-assigned data type changes • secondaries crash instead of pausing replication • query planner fails to cache plans when race phase interrupted • query planner caches plans for least data not representative data • accepted a bad op in the primary which bricked secondaries preventing quorum
  4. Risk assessment • How mature is the db? • How

    critical is the data? • How mature is your company? • Can you roll back? How hard will it be? • How much does your workload push the boundaries of the db? • Are other people doing similar workloads? • How much changed between releases?
  5. MongoDB 2.6 risk assessment for Parse: • How mature is

    the db? — NOT • How critical is the data? — TERRIBLY • How mature is your company? — FAIRLY • Can you roll back? How hard will it be? — DEPENDS • How much does your workload push the boundaries of the db? — EXTREMELY • Are other people doing similar workloads? — LOLNO • How much changed between releases? — A LOT
  6. Real production traffic • YOUR query set • YOUR data

    set • with YOUR hardware • and YOUR concurrency
  7. • unit tests • tools to replay sample queries against

    two primaries (e.g. pt-upgrade) • traffic splitter • bulk traffic capture + replay Correctness
  8. • Snapshot data • Capture ops • Replay ops •

    Reset, tweak, repeat Base Performance
  9. • Snapshot — from start of record run. Then create

    an LVM snapshot for resetting • Record — python tool to capture ops • Replay — go tool to play back ops • Rewind snapshot, rinse, repeat Replay tools for mongo (flashback)
  10. • n concurrent workers pulling off a queue • as

    fast as possible, or follow timestamps? • evict working set between runs (LVM snapshot reset does this, or echo 3 >/proc/sys/vm/ drop_caches) • compare logs for errors • break down by op type and percentile Replaying
  11. Bug hunting time. • removeOp() on Installation deviceId • https://jira.mongodb.org/browse/SERVER-14311

    • non-yielding full index scans • https://jira.mongodb.org/browse/SERVER-15152 • intersection-based query plans cached over single index plans with occasional empty predicates • https://jira.mongodb.org/browse/SERVER-14961
  12. “I upgraded and got 70% worse performance” ! “I upgraded

    and 30% of my writes started getting rejected bc mongo started enforcing index key lengths” ! “I upgraded and I’m getting corrupt data due to indexing race conditions” “I upgraded and .01% of my apps started ordering slightly differently for certain find queries” ! “I upgraded and one of my offline DW jobs had an incorrect implicit data type” 
 “I upgraded and had to adjust to a slightly different administrative workflow”
  13. We’re not going for perfection here. ! this is data,

    there will Always Be Something Wrong
  14. MongoDB: ! • MongoDB flashback tools: • https://github.com/ParsePlatform/flashback • Travis

    Redman’s slides on how we benchmarked 2.4 -> 2.6 • www.slideshare.net/travisredman79/benchmarking-at-parse ! Mysql: ! • blog post on Linden Lab mysql upgrade: • http://community.secondlife.com/t5/Technology-General/Diary-of-a- Paranoid-Mysql-Upgrade/ba-p/652582 • Apiary (deprecated): • https://bitbucket.org/lindenlab/apiary • Percona toolkit: • http://www.percona.com/software/percona-toolkit • Percona Playback: • http://www.percona.com/downloads/Percona-Playback/ Resources