Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons learned using Redis

Lessons learned using Redis

Lessons learned using Redis, lighting talk at Madrid Devops

Fernando Blat

May 29, 2012
Tweet

More Decks by Fernando Blat

Other Decks in Programming

Transcript

  1. Redis is: an open source, advanced key-value store a data

    structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io jueves 24 de mayo de 2012
  2. Redis is: an open source, advanced key-value store a data

    structure server since keys can contain strings, hashes, lists, sets and sorted sets. http://redis.io jueves 24 de mayo de 2012
  3. 1FSTJTUFODF RDB Snapshot of the whole database AOF (Append Only

    File) logs Operation logs http://antirez.com/post/redis-persistence-demystified.html http://redis.io/topics/persistence jueves 24 de mayo de 2012
  4. ################################ SNAPSHOTTING ################################# # # Save the DB on disk:

    # # save <seconds> <changes> # # Will save the DB if both the given number of seconds and the given # number of write operations against the DB occurred. # # In the example below the behaviour will be to save: # after 900 sec (15 min) if at least 1 key changed # after 300 sec (5 min) if at least 10 keys changed # after 60 sec if at least 10000 keys changed # # Note: you can disable saving at all commenting all the "save" lines. save 1800 1 save 1200 10 save 300 10000 # Compress string objects using LZF when dump .rdb databases? # For default that's set to 'yes' as it's almost always a win. # If you want to save some CPU in the saving child set it to 'no' but # the dataset will likely be bigger if you have compressible values or keys. rdbcompression yes # The filename where to dump the DB dbfilename dump.rdb # The working directory. # # The DB will be written inside this directory, with the filename specified # above using the 'dbfilename' configuration directive. # # Also the Append Only File will be created inside this directory. # # Note that you must specify a directory here, not a file name. dir /var/lib/redis jueves 24 de mayo de 2012
  5. $ top top - 19:48:36 up 90 days, 17:53, 1

    user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
  6. $ top top - 19:48:36 up 90 days, 17:53, 1

    user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
  7. $ top top - 19:48:36 up 90 days, 17:53, 1

    user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
  8. $ top top - 19:48:36 up 90 days, 17:53, 1

    user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
  9. $ top top - 19:48:36 up 90 days, 17:53, 1

    user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
  10. $ top top - 19:48:36 up 90 days, 17:53, 1

    user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
  11. $ top top - 19:48:36 up 90 days, 17:53, 1

    user, load average: 0.17, 0.39, 0.46 Tasks: 119 total, 1 running, 114 sleeping, 0 stopped, 4 zombie Cpu(s): 6.3%us, 0.5%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8160804k total, 8064100k used, 96704k free, 68580k buffers Swap: 262140k total, 249968k used, 12172k free, 2083720k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5630 redis 20 0 4158m 3.9g 572 S 0 49.7 116:32.46 /usr/local/bin/redis-server jueves 24 de mayo de 2012
  12. # This is the path of the Redis swap file.

    As you can guess, swap files # can't be shared by different Redis instances, so make sure to use a swap # file for every redis process you are running. Redis will complain if the # swap file is already in use. # # The best kind of storage for the Redis swap file (that's accessed at random) # is a Solid State Disk (SSD). # # *** WARNING *** if you are using a shared hosting the default of putting # the swap file under /tmp is not secure. Create a dir with access granted # only to Redis user and configure Redis to create the swap file there. vm-swap-file /tmp/redis.swap # vm-max-memory configures the VM to use at max the specified amount of # RAM. Everything that deos not fit will be swapped on disk *if* possible, that # is, if there is still enough contiguous space in the swap file. # # With vm-max-memory 0 the system will swap everything it can. Not a good # default, just specify the max amount of RAM you can in bytes, but it's # better to leave some margin. For instance specify an amount of RAM # that's more or less between 60 and 80% of your free RAM. vm-max-memory 5368709120 7JSUVBM .FNPSZ jueves 24 de mayo de 2012
  13. 7JSUVBM .FNPSZ # Virtual Memory allows Redis to work with

    datasets bigger than the actual # amount of RAM needed to hold the whole dataset in memory. # In order to do so very used keys are taken in memory while the other keys # are swapped into a swap file, similarly to what operating systems do # with memory pages. # # To enable VM just set 'vm-enabled' to yes, and set the following three # VM parameters accordingly to your needs. vm-enabled yes jueves 24 de mayo de 2012
  14. dump.rdb does not contain all the data, at least, in

    version 2.2 8IFO 3FEJT TUBSUT UP “TXBQ” jueves 24 de mayo de 2012
  15. #(4"7& http://redis.io/commands/bgsave Save the DB in background. Redis forks, the

    parent continues to server the clients, the child saves the DB on disk then exit. A client my be able to check if the operation succeeded using the LASTSAVE command. jueves 24 de mayo de 2012
  16. #BDLVQT #!/bin/bash SLEEP=30 function get_last_save { rm /tmp/previous_last_save echo "lastsave"

    | redis-cli > /tmp/previous_last_save lastsave=`cat /tmp/previous_last_save` } get_last_save previous_last_save=$lastsave echo "bgsave" | redis-cli while [ $lastsave = $previous_last_save ]; do sleep $SLEEP get_last_save done cp /var/lib/redis/dump.rdb /var/lib/redis/backups/dump-`date '+%Y%m%d%H%M'`.rdb # Remove backups older than 60 days find /var/lib/redis/backups -mtime +60 | xargs rm jueves 24 de mayo de 2012