Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Redis Hacks
Search
David Cramer
May 03, 2014
Technology
3
240
Redis Hacks
Python Nordeste 2014 - Lightning Talk
David Cramer
May 03, 2014
Tweet
Share
More Decks by David Cramer
See All by David Cramer
Mastering Duct Tape (PyCon Balkan 2018)
zeeg
2
890
Open Source as a Business (PyCon SG 2014)
zeeg
0
380
Angular.js Workshop (PyCon SG 2014)
zeeg
0
250
Architecting a Culture of Quality
zeeg
2
320
Release Faster
zeeg
12
1.4k
Open Source as a Business (EuroPython 2013)
zeeg
18
17k
Building to Scale (PyCon TW 2013)
zeeg
18
1.3k
Building to Scale
zeeg
28
24k
Lessons in Testing - DjangoCon 2012
zeeg
8
1.4k
Other Decks in Technology
See All in Technology
ACA でMAGI システムを社内で展開しようとした話
mappie_kochi
1
300
Vibe Coding Year in Review. From Karpathy to Real-World Agents by Niels Rolland, CEO Paatch
vcoisne
0
110
綺麗なデータマートをつくろう_データ整備を前向きに考える会 / Let's create clean data mart
brainpadpr
3
330
SoccerNet GSRの紹介と技術応用:選手視点映像を提供するサッカー作戦盤ツール
mixi_engineers
PRO
1
190
動画データのポテンシャルを引き出す! Databricks と AI活用への奮闘記(現在進行形)
databricksjapan
0
160
Adapty_東京AI祭ハッカソン2025ピッチスライド
shinoyamada
0
200
「AI駆動PO」を考えてみる - 作る速さから価値のスループットへ:検査・適応で未来を開発 / AI-driven product owner. scrummat2025
yosuke_nagai
3
760
実装で解き明かす並行処理の歴史
zozotech
PRO
1
620
20201008_ファインディ_品質意識を育てる役目は人かAIか___2_.pdf
findy_eventslides
2
560
"プロポーザルってなんか怖そう"という境界を超えてみた@TSUDOI by giftee Tech #1
shilo113
0
150
【Oracle Cloud ウェビナー】クラウド導入に「専用クラウド」という選択肢、Oracle AlloyとOCI Dedicated Region とは
oracle4engineer
PRO
3
120
OCI Network Firewall 概要
oracle4engineer
PRO
1
7.8k
Featured
See All Featured
Bash Introduction
62gerente
615
210k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
54
3k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
The World Runs on Bad Software
bkeepers
PRO
71
11k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.2k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Why Our Code Smells
bkeepers
PRO
339
57k
Context Engineering - Making Every Token Count
addyosmani
5
210
It's Worth the Effort
3n
187
28k
The Cost Of JavaScript in 2023
addyosmani
53
9k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Transcript
David Cramer twitter.com/zeeg Redis Hacks (or “How Sentry Scales”)
Buffering Writes
r = Redis() ! def incr(type, id): key = 'pending:{}'.format(type)
! r.zincrby(key, id, 1)
r = Redis() ! def flush(type): key = 'pending:{}'.format(type) result
= r.zrange(key, 0, -1, withscores=True) ! for id, count in result: prms = {'type': type, 'count': count, 'id': id} ! sql(""" update %(type)s set count = count + % (count)d where id = %(id)s """, prms)
Rate Limiting
r = Redis() ! def process_hit(project_id): epoch = time() /
60 key = ‘{}:{}’.format(project_id, epoch) ! pipe = r.pipeline() pipe.incr(key) pipe.expire(key, 60) result = pipe.execute() ! # return current value return int(result[0])
def request(project_id): result = process_hit(project_id) if result > 20: return
Response(status=429) return Response(status=200)
Time Series Data
def count_hits_today(project_id): start = time() end = now + DAY_SECONDS
! pipe = r.pipeline() for epoch in xrange(now, end, 10): key = ‘{}:{}’.format( project_id, epoch) pipe.get(key) results = pipe.execute() ! # remove non-zero results results = filter(bool, results) # coerce remainder to ints results = map(int, results) # return sum of buckets return sum(results)
Good-enough Locks
from contextlib import contextmanager ! r = Redis() ! @contextmanager
def lock(key, nowait=True): while not r.setnx(key, '1'): if nowait: raise Locked('try again soon!') sleep(0.01) ! # limit lock time to 10 seconds r.expire(key, 10) ! # do something crazy yield ! # explicitly unlock r.delete(key)
def do_something_crazy(): with lock('crazy'): print 'Hello World!'
Basic Sharding via Nydus
from nydus.db import create_cluster ! redis = create_cluster({ 'backend': 'nydus.db.backends.redis.Redis',
'hosts': { 0: {'db': 0}, 1: {'db': 1}, }, 'router': 'nydus.db.routers.keyvalue.PartitionRouter', })
def count_hits_today(project_id): start = time() end = now + DAY_SECONDS
! keys = [] for epoch in xrange(now, end, 10): key = '{}:{}'.format(project_id, epoch) keys.append(key) ! with redis.map() as conn: results = map(conn.get, keys) ! # remove non-zero results results = filter(bool, results) # coerce remainder to ints results = map(int, results) # return sum of buckets return sum(results)