A Year of MongoDB

(a ranty) presentation at PyGrunn 2013 about a year of MongoDB.

Transcript

1. a year with a talk by Armin '@mitsuhiko' Ronacher for PyGrunn 2013 mongoDB
2. That's me. I do Computers. Currently at Fireteam / Splash Damage. We do Internet for Pointy Shooty Games.
3. I don't like it let's not beat around the bush :(
4. but we're not all so negative
5. “MongoDB is a pretty okay data store” Jared Hefty (@bridwag)
6. this is not a rant it's our experience in a nutshell we find corner cases draw your own conclusions
7. “MongoDB is like a nuclear reactor: ensure proper working conditions and it's perfectly safe and powerful.” myself on 13th of October 2012
8. What changed?
9. RAD Soldiers Copyright © 2013 WarChest Limited. All Rights Reserved
10. RAD Soldiers
11. RAD Soldiers API calls 21st 24th oh
12. ?
14. MongoDB Overview { }
15. WHY? We recently asked the question
16. Why the fuck did we pick MongoDB?
17. Why the fuck did we pick MongoDB? schemaless
18. Why the fuck did we pick MongoDB? schemaless scalable
19. Why the fuck did we pick MongoDB? schemaless scalable simple
20. schemaless scalable simple json records
21. schemaless scalable simple json records auto sharding
22. schemaless scalable simple json records auto sharding think in records
23. schemaless is wrong mongodb's sharding is annoying thinking in records is hard trololol: two-phase commit
24. mongod mongoc mongos
25. mongod mongoc mongos mongods
26. mongod mongoc mongos mongods mongocs
27. mongod mongoc mongos mongods mongocs mongoses
28. mongod mongoc mongos mongods mongocs mongoses stores data
29. mongod mongoc mongos mongods mongocs mongoses stores data says what's where
30. mongod mongoc mongos mongods mongocs mongoses stores data says what's where routes and merges
31. Many Moving Parts mongod mongoc mongos
32. We Fail { }
33. workers on m1.small most of the time in IO wait no need for more CPU
34. oh really?
35. worker setup nginx uwsgi mongos mongod
36. worker setup nginx uwsgi mongos mongod uwsgi uwsgi
37. worker setup nginx uwsgi mongos mongod This
38. T1 waits for IO T2 uses CPU
39. worker: mongos, give me data mongos: mongod, give me data … mongos: worker, here is your data worker: finally! mongos, now give me more data context
40. m1.medium: machines with 2 CPUs* worker and mongos active at the same time what a novel idea *
41. MOAR
42. CPU Changes mean
43. EBS it's pretty bad
44. Breaking your Instance 101 $ dd if=/dev/random of=/var/cache/hah bs=4096 count=1024
45. MongoDB's Execution Fails { }
46. No transactions Document-level Operations No state transparent
47. NO!
48. Expectation • mongos fans out and proxies • if mongos loses connection worker is good • voluntary primary election is transparent for worker
49. Actual Result • mongos fans out • if mongos loses connection it terminates both sides • voluntary primary election kills all connections well;
50. Tail-able Cursors getLastError() MongoDB is Stateful
51. SIGSEGV
52. Replica Set Annoyances 1. Add Hidden Secondary 2. Witness it synchronizing 3. Take an existing secondary out 4. Actually unregister the secondary 5. Watch the whole cluster re-elect the same primary and kill all active connections
53. Breaking your Cluster 101 • add new primary • remove old primary • don't shutdown old primary • network partitions and one of them overrides the config of the other in the mongoc
54. MongoDB's Design Fails { }
55. Schemaless
56. Schema vs Schema-less is just a different version of dynamic typing vs. static typing
57. static typing with an escape hatch to dynamic typing wins Ever since C# and TypeScript:
58. we built an ADT based type system anyways from fireline.schema import types username = types.String() profile = types.Dynamic() x = username.convert('mitsuhiko') y = profile.convert({'__binary': 'deadbeaf'})
59. GetLastError()
60. write oddity write request mongodb GetLastError() mongodb why do I need an extra network roundtrip?
61. performance fun import os from pymongo import Connection safe = os.environ.get('MONGO_SAFE') == '1' con = Connection() db = con['wtfmongo'] coll = db['test'] coll.remove() for x in xrange(50000): coll.insert({'foo': 'bar'}, safe=safe)
62. Disappointing $ MONGO_SAFE=0 time python test.py 1.92 real 1.37 user 0.27 sys $ MONGO_SAFE=1 time python test.py 5.57 real 2.50 user 0.62 sys
63. Disappointing $ MONGO_SAFE=0 time python test.py 1.92 real 1.37 user 0.27 sys $ MONGO_SAFE=1 time python test.py 5.57 real 2.50 user 0.62 sys And
64. that would not be a problem if safe mode was fast. As it stands currently safe mode is slower than Postgres
65. Lack of Joins (the
66. They will happen 1. Before we had joins, we did not have joins 2. not having joins is not a feature 3. I see people joining in their code by hand. Inefficient
67. RethinkDB has Distributed Joins :-) r \ .table('marvel') \ .inner_join(r.table('dc'), lambda m, dc: m['strength'] < dc['strength']) \ .run(conn)
68. MongoDB does not have Map-Reduce (that shitty JavaScript map-reduce thing does not count)
69. Inconsistent Queries (and
70. Oh got why!? db.bios.find({ "awards": {"$elemMatch": { "award": "Turing Award", "year": { "$gt": 1980 } }} }) db.users.find({"username": "mitsuhiko"})
71. Repeat after me: in-band signalling is wrong!
72. Aggregation Framework comes with SQL Injection db.zipcodes.aggregate({ "$group": {"_id": "$state", "total_pop": {"$sum": "$pop"}} }, { "$match": {"total_pop": {"$gte": 10 * 1000 * 1000}} })
73. Aggregation Framework comes with SQL Injection db.zipcodes.aggregate({ "$group": {"_id": "$state", "total_pop": {"$sum": "$pop"}} }, { "$match": {"total_pop": {"$gte": 10 * 1000 * 1000}} }) spot
74. No Transactions
75. They are important! 1. You will need them or you have inconsistent data 2. Everybody builds a two-phase commit system 3. You need a process to clean up stale transactions
76. Locks Everywhere
77. MVCC is good for you RethinkDB, Postgres and even MySQL support MVCC
78. Shitty Index Selection 1. MongoDB picks secondary indexes automatically 2. It will also start using sparse indexes 3. It might not give you results back 4. Sometimes forcing ordering makes MongoDB use a compound index
79. Limited Indexes 1. Given a compound index on [a, b] 2. {a: 1, b: 2} and {$and: [{a: 1}, {b: 2}]} are equivalent 3. Only the former picks up the compound index 4. Negations never use indexes 5. {$or: […]} is implemented as two parallel queries, both clauses might need separate indexes.
80. We
81. Other Things of Note { }
82. Making Mongo not Suck (as much) on OS X $ mongod --noprealloc --smallfiles --nojournal run what
83. Windows 1. don't
84. Keys are huge. In our case ⅓ of the Data. Shorten them. (if only MongoDB had something like a … schema?)
85. A MongoDB Cluster needs to boot in a certain Order (Great fun if you have a suspended test infrastructure on Amazon)
88. MongoDB is a pretty good data dump thing
89. MongoDB is a pretty good data dump thing it's not a SQL database
90. MongoDB is a pretty good data dump thing it's not a SQL database but you probably want a SQL database
91. MongoDB is a pretty good data dump thing it's not a SQL database but you probably want a SQL database at least until RethinkDB is ready
92. That's it. Now ask questions. And add me on twitter: @mitsuhiko Slides at lucumr.pocoo.org/talks ?
93. Legal Shenanigans Creative Common Sources for Images: CPU by EssjazNZ: http:/ /www.flickr.com/photos/essjay/4972875711/ Locks by katiejean97: http:/ /www.flickr.com/photos/katiejean97/7036715845/ Money Money Money by Images_of_Money: http:/ /www.flickr.com/photos/59937401@N07/5474168441/ Through any Window by Josep Ma. Rosell: http:/ /www.flickr.com/photos/batega/1354354592/in/photostream/ RAD Soldiers is a Trademark of WarChest Limited. RAD Soldiers Artwork and Logo Copyright © 2013 by WarChest Limited. All Rights Reserved.