Upgrade to Pro — share decks privately, control downloads, hide ads and more …

FileBeat (Won't save you from the JVM)

Chris Fidao
January 24, 2016

FileBeat (Won't save you from the JVM)

A quick presentation about using Elastic's FileBeat for log aggregation.

Chris Fidao

January 24, 2016
Tweet

More Decks by Chris Fidao

Other Decks in Technology

Transcript

  1. !

  2. # Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb

    # Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
  3. filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:

    - /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
  4. # Install Java sudo apt-get install -y openjdk-7-jdk # Add

    ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
  5. input { beats { type => beats port => 5044

    } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
  6. filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:

    - /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
  7. Conclusion: (for my use case) Fluentd is good enough. •One

    less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)