(Advanced) Redis

Everything we learned at a client project using Redis - Talk given at #ArrrrUG

Transcript

1. (advanced) Redis Advanced is between brackets because we don’t claim to be advanced users, we just have gained a lot of experience
2. Jan De Poorter @defv
3. Redis?
4. NoSQL
5. :-D
6. No, but srsly
7. Redis is an in-memory key-value data-structure server
8. in memory with “best effort” persistence
9. Data structure server Strings SET, SETNX, GET, INCR, DECR, APPEND, ... Hashes HSET, HGET, HGETALL, HDEL, HLEN, ... Lists LPUSH, LPOP, LINDEX, LLEN, LRANGE, ... Sets & Sorted sets SADD, SPOP, SMEMBERS, SDIFF, ZADD, ...
10. some libraries Ruby redis-rb (gem) hiredis-rb (gem) Node redis (npm) hiredis (npm)
11. Q-Music Our use case was the API for the new Q-Music website and the iPhone and Android applications.
12. High volume
13. Pub-Sub
14. BG jobs with Sidekiq
15. Some do’s
16. Serialize values
17. 2 class QApi::Redis 3 class << self 4 def redis 5 @redis ||= Redis.new(config) 6 end ... 181 def set(key, data) 182 redis.set key, to_json(data) 183 end 184 185 def get(key) 186 from_json redis.get(key) 187 end 188 216 private 217 def from_json(result) 218 case result 219 when Array 220 result.map { |r| from_json(r) } 221 when Hash 222 result 223 when nil 224 nil 225 else 226 MultiJson.decode(result) 227 end 228 rescue MultiJson::DecodeError 229 result 230 end 231 232 def to_json(data) 233 MultiJson.encode(data) 234 end 262 end 263 end
18. Save references
19. # Write LPUSH plays {“id”: 42, ”title”:”99 Bottles of Rum”} LPUSH plays {“id”: 49, ”title”:”The Drunken Sailor”} LPUSH plays {“id”: 42, ”title”:”99 Bottles of Rum”} # Read LRANGE plays 0 -1 don’t do this
20. it’s fine with 100 items
21. but becomes huge with 90 000 items
22. or 1 000 000 Justin Bieber mentions ;-)
23. # Write SET tracks:42 {“id”: 42, ”title”:”99 Bottles of Rum”} SET tracks:49 {“id”: 42, ”title”:” The Drunken Sailor”} LPUSH plays tracks:42 LPUSH plays tracks:49 LPUSH plays tracks:42 # Read LRANGE plays 0 -1 MGET tracks:42 tracks:49 do this instead
24. or in Ruby 82 def list_from_references(list, options) 83 references = redis.lrange list, 84 options[:start], options[:stop] 85 if references.any? 86 from_json redis.mget(*references) 87 else 88 [] 89 end 90 end
25. Some don’t’s
26. RDB vs AOF
27. RDB does fork()
28. doubling memory usage
29. Our dataset was 4GB
30. We had ~8GB of memory
31. No BGSAVE’s for 4 days
32. :-(
33. appendonly yes
34. Slow(er) start-up
35. Can be optimized
36. Monitoring
37. data:~$ redis-cli redis > INFO redis_version:2.4.16 ... connected_clients:152 ... used_memory:2744963744 used_memory_human:2.56G used_memory_peak:6150240632 used_memory_peak_human:5.73G ... changes_since_last_save:9576 last_save_time:1360661315 redis >
38. PUBLISH & SUBSCRIBE overload
39. WebSocket clients
40. 1 client == 1 connection == 1 subscription
41. Works perfect in development
42. Works perfect in staging
43. but in production...
44. 1 new event
45. sent to 3500 connected clients
46. Takes about 0.3 seconds
47. Up to 4 messages per second
48. :-(
49. Move logic to websocket server
50. :-(
51. 1 type of msg == 1 subscription
52. :-)
53. MULTI / EXEC
54. > MULTI OK > RENAME bl:1 bld:1 QUEUED > RENAME bl:2 bld:2 QUEUED > EXEC 1) OK 2) OK
55. Works great on 1000’s of commands
56. Not so much on 1 million commands...
57. Don’t exceed ~10k commands in 1 transaction
58. master/slave
59. no library support
60. DIY
61. We didn’t
62. Limited connections
63. Redis < 2.6
64. /* Max number of fd supported */ #define AE_SETSIZE (1024*10)
65. No more then ~10k connections allowed
66. Redis >= 2.6
67. The sky is the limit
68. (actually, the linux max FD is the limit)
69. Questions?