Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
yapc::eu 2013
Search
Oleg Komarov
August 14, 2013
Programming
0
180
yapc::eu 2013
reliable cron jobs in distributed environment
Oleg Komarov
August 14, 2013
Tweet
Share
More Decks by Oleg Komarov
See All by Oleg Komarov
Exploring Plack Middlewares
komarov
0
160
lpw-2012
komarov
1
330
yapc_eu_2012
komarov
2
530
Other Decks in Programming
See All in Programming
無関心の谷
kanayannet
0
180
社内での開発コミュニティ活動とモジュラーモノリス標準化事例のご紹介/xPalette and Introduction of Modular monolith standardization
m4maruyama
1
130
Datadog RUM 本番導入までの道
shinter61
1
310
データベースコネクションプール(DBCP)の変遷と理解
fujikawa8
1
270
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
130
GoのWebAssembly活用パターン紹介
syumai
3
10k
[初登壇@jAZUG]アプリ開発者が気になるGoogleCloud/Azure+wasm/wasi
asaringo
0
130
GoのGenericsによるslice操作との付き合い方
syumai
2
670
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
190
イベントストーミングから始めるドメイン駆動設計
jgeem
4
870
Spring gRPC で始める gRPC 入門 / Introduction to gRPC with Spring gRPC
mackey0225
2
520
「ElixirでIoT!!」のこれまでとこれから
takasehideki
0
370
Featured
See All Featured
How to Ace a Technical Interview
jacobian
277
23k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
228
22k
Building a Modern Day E-commerce SEO Strategy
aleyda
41
7.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.8k
Documentation Writing (for coders)
carmenintech
71
4.9k
4 Signs Your Business is Dying
shpigford
184
22k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Facilitating Awesome Meetings
lara
54
6.4k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
48
5.4k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
32
5.9k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Transcript
Reliable Cron Jobs in Distributed Environment Oleg Komarov 2013-08-14
What this talk is about decentralizing background job execution maintaining
concurrency (think locks and leases)
Figure: not decentralized
Figure: decentralized
Problems to solve avoid duplicate processes (locking) avoid resource conflicts
(leasing) bear in mind collecting logs monitoring
Common pitfalls of distributed locks no lock cleanup on process
death operating not actually holding the lock
Apache ZooKeeper http://zookeeper.apache.org a distributed, open-source coordination service for distributed
applications reliable, replicated, trx-ordered, single system image operations: create, get, set, get children, delete, exists features: watchers, ephemeral nodes, sequential nodes
Sequential nodes $zkh->create( "/path/basename-", $data, flags => ZOO_SEQUENTIAL ) for
(1 .. 3); creates /path/basename-0000000001 /path/basename-0000000002 /path/basename-0000000003
Synchronization http://zookeeper.apache.org/doc/r3.4.5/recipes.html CPAN Net::ZooKeeper::Lock Net::ZooKeeper::Semaphore
Non-blocking exclusive lock $zkh->create($lock_path, $node_data, acl => ZOO_OPEN_ACL_UNSAFE, flags =>
ZOO_EPHEMERAL, ); if (my $error = $zkh->get_error) { if ($error == ZNODEEXISTS) { return undef; # the lock is held by someone else } else { die "Could not acquire lock $lock_path: $error"; } } # success return $zkh->exists($lock_path, watch => $lock_watch);
Figure: lock supervising
Figure: switchman
switchman a flock-style utility for distributed locks and semaphores https://github.com/komarov/switchman
https://metacpan.org/release/switchman
Features reliable locking fair resource leasing (no starvation) lease name
macros server groups
Resources global DB API utilization local cpu memory io local
as global cpu@serverA memory@serverA
Macros names FQDN cpu FQDN mem values 1:CPU 4096:MEMMB
Figure: control flow
How to setup install switchman create a local configuration file
create a znode in zookeeper modify your crontabs
Configuration /etc/switchman.conf: { "prefix":"/switchman", "zkhosts":"zk1:2181,zk2:2181,zk3:2181", "loglevel":"info", "logfile":"/path/to/log" }
ZooKeeper znode layout /switchman /locks /queues /semaphores
Configuration /switchman znode data: { "groups":{"grp1":["host1","host2"],"grp2":"host1"}, "resources":["FQDN_mem","FQDN_cpu"] }
Usage switchman COMMAND # lockname == command’s basename switchman --lockname
LOCK -- COMMAND [ARGS] switchman --lease FQDN_cpu=1:CPU COMMAND switchman --group grp2 COMMAND # runs only on host1
Left out of scope log aggregation monitoring
Thank you! Any questions? If you really liked the talk,
don’t hesitate to like my quest :) http://questhub.io/player/komarov