Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Final Crontab

The Final Crontab

Selena Deckelmann

May 06, 2014
Tweet

More Decks by Selena Deckelmann

Other Decks in Technology

Transcript

  1. socorro1 socorro3 WAL Socorro1 .dev Socorro1. stage base_backup copy Sunday

    noon PT streaming rep Prod socorro2 backup4 base_backup & pg_dump backup reporting1 WAL socorro-db-zeus-rw socorro-db-zeus-ro very architecture very architecture such replicas such replicas wow wow
  2. 0 5000 10000 15000 20000 25000 Dec 5, 2010 May

    5, 2011 Oct 5, 2011 Mar 5, 2012 Aug 5, 2012 Jan 5, 2013 Jun 5, 2013 Nov 5, 2013 Apr 5, 2014 Cron alert messages
  3. Cron, what is it good for? • birthday reminders •

    status updates for a website • doxygen output for manuals every 12 hours • email nags about bugs filed wrong • ETL • Postgres -> Cloudwatch • Batch processing • Backups of RO DB • Machine heartbeat • “sweet fuck all” • “auto” updates • logging laptop IP • check for abandoned twitter accounts
  4. How Socorro uses cron • Time-dependent reports or maintenance •

    “Simple” event detection and triggers • Status logging
  5. Our use cases • Stored procedures for materialized views in

    Postgres • Daily map-reduces (largely deprecated) • FTP Scraping into Postgres • Bulk email responses to crash submissions pulled from Elastic Search
  6. Jobs that don’t lend themselves to queue management because of

    time-dependencies, fragility or complexity.
  7. #!/bin/bash . /etc/socorro/socorrorc NAME=`basename $0 .sh` lock --ignore-existing $NAME ${PYTHON}

    ${APPDIR}/socorro/cron/crontabber.py \ --admin.conf=/etc/socorro/crontabber.ini \ >> /var/log/socorro/crontabber.log 2>&1 EXIT_CODE=$? unlock $NAME exit $EXIT_CODE
  8. 15:58 < nagios-phx1> | Sun 15:58:44 PDT [1085] socorroadm.stage.private.phx1.mozilla.com: Socorro

    Admin - crontab is CRITICAL: CRITICAL - correlations-addon-matview (CorrelationsAddonCronApp) (http://m.mozilla.org/Socorro+Admin+-+crontab)
  9. */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber1.ini */5

    * * * * socorro crontabber \ --admin.conf=/etc/crontabber2.ini */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber3.ini