Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Redis Hacks
Search
David Cramer
May 03, 2014
Technology
3
240
Redis Hacks
Python Nordeste 2014 - Lightning Talk
David Cramer
May 03, 2014
Tweet
Share
More Decks by David Cramer
See All by David Cramer
Mastering Duct Tape (PyCon Balkan 2018)
zeeg
2
880
Open Source as a Business (PyCon SG 2014)
zeeg
0
360
Angular.js Workshop (PyCon SG 2014)
zeeg
0
240
Architecting a Culture of Quality
zeeg
2
320
Release Faster
zeeg
12
1.4k
Open Source as a Business (EuroPython 2013)
zeeg
18
17k
Building to Scale (PyCon TW 2013)
zeeg
18
1.3k
Building to Scale
zeeg
28
24k
Lessons in Testing - DjangoCon 2012
zeeg
8
1.4k
Other Decks in Technology
See All in Technology
AWS Summit Japan 2025 Community Stage - App workflow automation by AWS Step Functions
matsuihidetoshi
1
140
ユーザーのプロフィールデータを活用した推薦精度向上の取り組み
yudai00
0
490
BigQuery Remote FunctionでLooker Studioをインタラクティブ化
cuebic9bic
2
230
25分で解説する「最小権限の原則」を実現するための AWS「ポリシー」大全
opelab
9
2.2k
AIの最新技術&テーマをつまんで紹介&フリートークするシリーズ #1 量子機械学習の入門
tkhresk
0
130
20250625 Snowflake Summit 2025活用事例 レポート / Nowcast Snowflake Summit 2025 Case Study Report
kkuv
1
220
25分で解説する「最小権限の原則」を実現するための AWS「ポリシー」大全 / 20250625-aws-summit-aws-policy
opelab
7
760
GeminiとNotebookLMによる金融実務の業務革新
abenben
0
140
AIエージェント最前線! Amazon Bedrock、Amazon Q、そしてMCPを使いこなそう
minorun365
PRO
11
4.1k
Prox Industries株式会社 会社紹介資料
proxindustries
0
210
In Praise of "Normal" Engineers (LDX3)
charity
3
1.2k
2025/6/21 日本学術会議公開シンポジウム発表資料
keisuke198619
2
480
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
31
8.6k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
Designing for Performance
lara
609
69k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
228
22k
The World Runs on Bad Software
bkeepers
PRO
69
11k
BBQ
matthewcrist
89
9.7k
Why Our Code Smells
bkeepers
PRO
337
57k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.4k
Six Lessons from altMBA
skipperchong
28
3.8k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
60k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.8k
Transcript
David Cramer twitter.com/zeeg Redis Hacks (or “How Sentry Scales”)
Buffering Writes
r = Redis() ! def incr(type, id): key = 'pending:{}'.format(type)
! r.zincrby(key, id, 1)
r = Redis() ! def flush(type): key = 'pending:{}'.format(type) result
= r.zrange(key, 0, -1, withscores=True) ! for id, count in result: prms = {'type': type, 'count': count, 'id': id} ! sql(""" update %(type)s set count = count + % (count)d where id = %(id)s """, prms)
Rate Limiting
r = Redis() ! def process_hit(project_id): epoch = time() /
60 key = ‘{}:{}’.format(project_id, epoch) ! pipe = r.pipeline() pipe.incr(key) pipe.expire(key, 60) result = pipe.execute() ! # return current value return int(result[0])
def request(project_id): result = process_hit(project_id) if result > 20: return
Response(status=429) return Response(status=200)
Time Series Data
def count_hits_today(project_id): start = time() end = now + DAY_SECONDS
! pipe = r.pipeline() for epoch in xrange(now, end, 10): key = ‘{}:{}’.format( project_id, epoch) pipe.get(key) results = pipe.execute() ! # remove non-zero results results = filter(bool, results) # coerce remainder to ints results = map(int, results) # return sum of buckets return sum(results)
Good-enough Locks
from contextlib import contextmanager ! r = Redis() ! @contextmanager
def lock(key, nowait=True): while not r.setnx(key, '1'): if nowait: raise Locked('try again soon!') sleep(0.01) ! # limit lock time to 10 seconds r.expire(key, 10) ! # do something crazy yield ! # explicitly unlock r.delete(key)
def do_something_crazy(): with lock('crazy'): print 'Hello World!'
Basic Sharding via Nydus
from nydus.db import create_cluster ! redis = create_cluster({ 'backend': 'nydus.db.backends.redis.Redis',
'hosts': { 0: {'db': 0}, 1: {'db': 1}, }, 'router': 'nydus.db.routers.keyvalue.PartitionRouter', })
def count_hits_today(project_id): start = time() end = now + DAY_SECONDS
! keys = [] for epoch in xrange(now, end, 10): key = '{}:{}'.format(project_id, epoch) keys.append(key) ! with redis.map() as conn: results = map(conn.get, keys) ! # remove non-zero results results = filter(bool, results) # coerce remainder to ints results = map(int, results) # return sum of buckets return sum(results)