Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ELK how we use it

dknx01
August 02, 2016

ELK how we use it

dknx01

August 02, 2016
Tweet

More Decks by dknx01

Other Decks in Programming

Transcript

  1. Some basic infos • PHP developer • working at: Little

    Bird GmbH • Needed to handle our log (exception) much better
  2. What is ELK Elasticsearch • search engine (Lucene based) •

    indices data • provides REST-API • saves data as NoSQL (JSON)
  3. What is ELK Logstash • Log data processor / ETL

    • 165+ plugins for inputs (files, TCP/UDP-Ports, STDIN, etc.) • filter, normalise and transform data (Date-Format, GeoIP, RegExp, Anonymize, etc.) • send data to STDOUT, databases, files, chats, nagios etc.
  4. What is ELK Kibana • nodeJS based GUI for elasticsearch

    • data visualisation (charts) • data analyses
  5. Current situation and why to change • Several hosting partner

    with different systems • Writing separate log files for each customer (Symptoms and Apache) • Exception are catched and send via email to a shared mail account • Not every developer has access to mail account • Bugs are not always reported with stackstrace/infos • Hard to track if exception occurred on other system or many times
  6. Our environment - our problems • Many hosting providers •

    Different OS: ◦ Debian ◦ Ubuntu ◦ SuSe • Different access rights ◦ Full access (internal system) ◦ Only one port: 443 ◦ Non-HTTP(S)/SMTP access only on request • No central server • One separate log file per customer (environment)
  7. How we solved it • Central logging server on our

    in-house system • Tunneling the desired connection through port 443 • Application server only needs ◦ Stunnel - a tunneling software ◦ Java - we need it for the application, too ◦ Logstash as standalone downloaded application • Customer (environment) name is extracted from file name
  8. What to do next • Central log server fail over

    (more than one Redis, Elasticsearch server) • Automatical analysation of log and notify developers • Only process neccessary log messages
  9. End