Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
yapc::eu 2013
Search
Oleg Komarov
August 14, 2013
Programming
0
180
yapc::eu 2013
reliable cron jobs in distributed environment
Oleg Komarov
August 14, 2013
Tweet
Share
More Decks by Oleg Komarov
See All by Oleg Komarov
Exploring Plack Middlewares
komarov
0
160
lpw-2012
komarov
1
330
yapc_eu_2012
komarov
2
520
Other Decks in Programming
See All in Programming
がんばりすぎないコーディングルール運用術
tsukakei
1
180
テスト分析入門/Test Analysis Tutorial
goyoki
12
2.7k
❄️ tmux-nixの実装を通して学ぶNixOSモジュール
momeemt
1
120
ワンバイナリWebサービスのススメ
mackee
10
7.5k
TypeScript エンジニアが Android 開発の世界に飛び込んだ話
yuisakamoto
6
960
Use Perl as Better Shell Script
karupanerura
0
660
CRUD から CQRS へ ~ 分離が可能にする柔軟性
tkawae
0
230
Building an Application with TDD, DDD and Hexagonal Architecture - Isn't it a bit too much?
mufrid
0
370
CQRS/ESのクラスとシステムフロー ~ RailsでフルスクラッチでCQRSESを組んで みたことから得た学び~
suzukimar
0
190
Javaのルールをねじ曲げろ!禁断の操作とその代償から学ぶメタプログラミング入門 / A Guide to Metaprogramming: Lessons from Forbidden Techniques and Their Price
nrslib
1
300
Cursor Meetup Tokyo ゲノミクスとCursor: 進化と制約のあいだ
koido
1
340
Blueskyのプラグインを作ってみた
hakkadaikon
1
290
Featured
See All Featured
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Product Roadmaps are Hard
iamctodd
PRO
53
11k
Building a Modern Day E-commerce SEO Strategy
aleyda
40
7.3k
Building an army of robots
kneath
306
45k
Designing for humans not robots
tammielis
253
25k
Documentation Writing (for coders)
carmenintech
71
4.8k
How to Think Like a Performance Engineer
csswizardry
23
1.6k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.6k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Transcript
Reliable Cron Jobs in Distributed Environment Oleg Komarov 2013-08-14
What this talk is about decentralizing background job execution maintaining
concurrency (think locks and leases)
Figure: not decentralized
Figure: decentralized
Problems to solve avoid duplicate processes (locking) avoid resource conflicts
(leasing) bear in mind collecting logs monitoring
Common pitfalls of distributed locks no lock cleanup on process
death operating not actually holding the lock
Apache ZooKeeper http://zookeeper.apache.org a distributed, open-source coordination service for distributed
applications reliable, replicated, trx-ordered, single system image operations: create, get, set, get children, delete, exists features: watchers, ephemeral nodes, sequential nodes
Sequential nodes $zkh->create( "/path/basename-", $data, flags => ZOO_SEQUENTIAL ) for
(1 .. 3); creates /path/basename-0000000001 /path/basename-0000000002 /path/basename-0000000003
Synchronization http://zookeeper.apache.org/doc/r3.4.5/recipes.html CPAN Net::ZooKeeper::Lock Net::ZooKeeper::Semaphore
Non-blocking exclusive lock $zkh->create($lock_path, $node_data, acl => ZOO_OPEN_ACL_UNSAFE, flags =>
ZOO_EPHEMERAL, ); if (my $error = $zkh->get_error) { if ($error == ZNODEEXISTS) { return undef; # the lock is held by someone else } else { die "Could not acquire lock $lock_path: $error"; } } # success return $zkh->exists($lock_path, watch => $lock_watch);
Figure: lock supervising
Figure: switchman
switchman a flock-style utility for distributed locks and semaphores https://github.com/komarov/switchman
https://metacpan.org/release/switchman
Features reliable locking fair resource leasing (no starvation) lease name
macros server groups
Resources global DB API utilization local cpu memory io local
as global cpu@serverA memory@serverA
Macros names FQDN cpu FQDN mem values 1:CPU 4096:MEMMB
Figure: control flow
How to setup install switchman create a local configuration file
create a znode in zookeeper modify your crontabs
Configuration /etc/switchman.conf: { "prefix":"/switchman", "zkhosts":"zk1:2181,zk2:2181,zk3:2181", "loglevel":"info", "logfile":"/path/to/log" }
ZooKeeper znode layout /switchman /locks /queues /semaphores
Configuration /switchman znode data: { "groups":{"grp1":["host1","host2"],"grp2":"host1"}, "resources":["FQDN_mem","FQDN_cpu"] }
Usage switchman COMMAND # lockname == command’s basename switchman --lockname
LOCK -- COMMAND [ARGS] switchman --lease FQDN_cpu=1:CPU COMMAND switchman --group grp2 COMMAND # runs only on host1
Left out of scope log aggregation monitoring
Thank you! Any questions? If you really liked the talk,
don’t hesitate to like my quest :) http://questhub.io/player/komarov