Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Monitoring Infrastructure with SaltStack

Monitoring Infrastructure with SaltStack

Using SaltStack to monitor servers and applications using SaltStack's modules, returners, and scheduler. See how to link these pieces together to funnel data into apps like Graphite, Librato, and others. The presentation will also take you through the process of building your own custom modules and returners to store application-specific data to any location.

Peter Baumgartner

January 29, 2014
Tweet

More Decks by Peter Baumgartner

Other Decks in Technology

Transcript

  1. There’s More? “SaltStack delivers a dynamic infrastructure communication bus used

    for orchestration, remote execution, configuration management and
 much more.”
  2. Monitoring System Traits 1. Data is collected at regular intervals

    2. Results are shipped to a 
 central datastore 3. A front-end to view the results
  3. Built-in Modules • ps CPU, RAM, and disk • service

    system services • file hashes, last modified, metadata, etc. • cmd run arbitrary commands
  4. Build Your Own Module 1. Run a command 2. Parse

    the results 3. Return a dictionary
  5. Run a command def info(host='localhost', port='6379', password=None): cmd = ['redis-cli',

    '-h {0}'.format(host), '-p {0}'.format(port), 'info'] if password: cmd.insert(1, '-a {0}'.format(password)) out = __salt__['cmd.run'](' '.join(cmd))
  6. Parse the Results data = {} for line in out.splitlines():

    if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value
  7. def info(host='localhost', port='6379', password=None): cmd = ['redis-cli -h {host} -p

    {port}'.format(host=host, port=port), 'info'] if password: cmd.insert(1, '-a {password}'.format(password=password)) out = __salt__['cmd.run'](' '.join(cmd)) data = {} for line in out.splitlines(): if ':' in line: key, value = line.split(':') if ',' in value: subdata = value.split(',') value = dict([kv.split('=') for kv in subdata]) data[key] = value return data
  8. Deploy 1. Save file on Salt master:
 <file_root>/_modules/redis.py 2. Deploy

    to minions:
 salt '*' saltutil.sync_modules 3. Run:
 salt '*' redis.info
  9. Built-in Returners • syslog • mysql, postgres RDBMS • redis,

    mongo, cassandra NoSQL • carbon, sentry Apps
  10. Examples # salt-call --return syslog \ ps.physical_memory_usage ! # salt-call

    --return syslog,carbon \ ps.physical_memory_usage
  11. Returner Dictionary { 'fun': 'ps.cpu_percent', 'fun_args': [ 'interval=5', 'per_cpu=True', ],

    'id': 'www.botbot.me', 'return': [0.2, 0.4] 'jid': '20140127204114791430', 'retcode': 0, }
  12. import json, redis def _get_serv(): return redis.Redis( host=__salt__['config.option']('redis.host'), port=__salt__['config.option']('redis.port'), db=__salt__['config.option']('redis.db'))

    def returner(ret): serv = _get_serv() serv.set('{0}:{1}'.format(ret['id'], ret['jid']), json.dumps(ret)) serv.lpush('{0}:{1}'.format(ret['id'], ret['fun']), ret['jid']) serv.sadd('minions', ret['id']) serv.sadd('jids', ret['jid'])
  13. Configuration • Most built-ins use the Salt master/minion config •

    Use pillars instead (easier management) ! __salt__[‘pillar.get']('your_returner:some_key', '')
  14. Handling Requirements Use the __virtual__ function • Sets the public

    name (default = file name) • If False, it is not available • Works for modules and returners
  15. The __virtual__ Function try: import redis HAS_REDIS = True except

    ImportError: HAS_REDIS = False ! def __virtual__(): if not HAS_REDIS: return False return 'redis'
  16. Deploy 1. Save file on Salt master:
 <file_root>/_returners/redis_return.py 2. Deploy

    to minions:
 salt '*' saltutil.sync_returners 3. Run:
 salt '*' --return redis test.ping
  17. The schedule Pillar schedule: vmm_usage: function: ps.virtual_memory_usage minutes: 5 returner:

    redis disk_usage: function: ps.disk_usage args: - "/" minutes: 5 returner: redis
  18. What Can Salt Do? • Configure minions (Pillars and States)

    • Gather data (Modules) • Ship to datastore (Returners) • Schedule the process at regular intervals (Scheduler)
  19. UI Options • Graphite (built-in returner) • Hosted: Librato, StatHat,

    etc.
 https://github.com/lincolnloop/salt-stats • Build your own
 https://github.com/lincolnloop/salmon