lpw-2012

Reliable Cron Jobs in Distributed Environment

Transcript

1. Reliable Cron Jobs in Distributed Environment Oleg Komarov 2012-11-24 1/26
2. Presentation available at https://speakerdeck.com/komarov/lpw-2012 http://bit.ly/VLuT6g 2/26
3. Context 3 independent projects with shared infrastructure • over 30 boxes • over 200 scripts, 30K+ SLOC • packaged in appr. 20 deb-packages 3/26
4. TL;DR Reliable Cron Jobs in Distributed Environment 4/26
5. TL;DR Reliable Cron Jobs in Distributed Environment ... are HARD to get right 4/26
6. Cron Jobs in a Vacuum • locks • logging and output • monitoring • profiling 5/26
7. Logging and Output • log START and FINISH • log enough details 6/26
8. Logging and Output • log START and FINISH • log enough details • use log + STDERR for important things • use MAILTO to catch that output 6/26
9. Logging and Output • log START and FINISH • log enough details • use log + STDERR for important things • use MAILTO to catch that output 6/26
10. Monitoring • be confident that it actually works • it must not fail when you system fails • have a plan of action 7/26
11. Monitoring • be confident that it actually works • it must not fail when you system fails • have a plan of action 7/26
12. What to monitor • hardware errors • free disk space • load • crond is alive • age of generated file, queue size, etc. 8/26
13. Profiling • Does it need 1GB or 10GB? • What does it take so long to complete? • How many db queries does it run? 9/26
14. Profiling • Does it need 1GB or 10GB? • What does it take so long to complete? • How many db queries does it run? Measure and improve 9/26
15. More to consider • crash-safe • documentation • parallel execution • resource limits (ulimit/cgroups) 10/26
16. Deployment Packages 11/26
17. Deployment Boxes 12/26
18. Cron Package Just populate my-project-scriptsN.cron.d file 13/26
19. Cron Package Just populate my-project-scriptsN.cron.d file Don’t write it by hand, do it automatically 13/26
20. Cron Package Just populate my-project-scriptsN.cron.d file Don’t write it by hand, do it automatically Put some METADATA in your scripts 13/26
21. Metadata =head1 METADATA <crontab> package: scriptsN params: --mod 2 --rem 0 time: */2 * * * * </crontab> <crontab> package: scriptsN params: --mod 2 --rem 1 time: */2 * * * * </crontab> =cut 14/26
22. Simple Setup As simple as possible: one box per package 15/26
23. Simple Setup As simple as possible: one box per package apt-get purge && kill (or wait) && apt-get install 15/26
24. !%*#$ Back to Earth Network 16/26
25. With Extra Boxes Now you have some promblems to solve: • locks • logs • load 17/26
26. Net::ZooKeeper::Lock Apache ZooKeeperTM is an effort to develop and maintain an open-source server which enables highly reliable distributed coordination. Net::ZooKeeper::Lock implements distributed locks via ZooKeeper. 18/26
27. Introducing Switchman https://github.com/komarov/switchman 19/26
28. Overview 20/26
29. Configuration Crontabs are installed everywhere, switchman consults with config in ZooKeeper: { "groups": { "scripts1": "box1", "scripts2": "box1", "scripts3": ["box1", "box2"] } } 21/26
30. Description switchman --config /how/to/connect/to/zk --group scriptsN -- CMD ARGS 22/26
31. Description switchman --config /how/to/connect/to/zk --group scriptsN -- CMD ARGS • checks configuration • acquires a lock • watches configuration for changes • stops execution when it is not allowed anymore 22/26
32. Description switchman --config /how/to/connect/to/zk --group scriptsN -- CMD ARGS • checks configuration • acquires a lock • watches configuration for changes • stops execution when it is not allowed anymore Easy to adopt with METADATA 22/26
33. One Problem Solved • locks • logs • load 23/26
34. Further Steps See facebook’s Scribe for collecting decentralized logs Resources reservation and management A good monitoring system 24/26
35. Thanks! Questions? https://speakerdeck.com/komarov/lpw-2012 http://bit.ly/VLuT6g http://about.me/komarov om 25/26
36. Bonus Slide Get file age: # in days perl -E ’say -M $ARGV[0]’ /path/to/file # in seconds expr ‘date +%s‘ - ‘date +%s -r /path/to/file‘ Simple local locks: use Pid::File::Flock qw/:auto/; 26/26