Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
320
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
40
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
66
Scaling Laravel - Laracon.net 2018
fideloper
15
1.9k
Linux Environment
fideloper
1
10k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
560
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
3月のAWSアップデートを5分間でざっくりと!
kubomasataka
0
120
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
230
バクラクの認証基盤の成長と現在地 / bakuraku-authn-platform
convto
1
610
意思決定を支える検索体験を目指してやってきたこと
hinatades
PRO
0
150
React ABC Questions
hirotomoyamada
0
440
JPOUG Tech Talk #12 UNDO Tablespace Reintroduction
nori_shinoda
2
140
AWSで作るセキュアな認証基盤with OAuth mTLS / Secure Authentication Infrastructure with OAuth mTLS on AWS
kaminashi
0
180
От ручной разметки к LLM: как мы создавали облако тегов в Lamoda. Анастасия Ангелова, Data Scientist, Lamoda Tech
lamodatech
0
750
白金鉱業Meetup_Vol.18_AIエージェント時代のUI/UX設計
brainpadpr
0
100
LiteXとオレオレCPUで作る自作SoC奮闘記
msyksphinz
0
680
PagerDuty×ポストモーテムで築く障害対応文化/Building a culture of incident response with PagerDuty and postmortems
aeonpeople
1
270
Cross Data Platforms Meetup LT 20250422
tarotaro0129
1
690
Featured
See All Featured
Six Lessons from altMBA
skipperchong
27
3.7k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Faster Mobile Websites
deanohume
306
31k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
135
33k
Code Review Best Practice
trishagee
67
18k
Making the Leap to Tech Lead
cromwellryan
133
9.2k
Building an army of robots
kneath
304
45k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.9k
Side Projects
sachag
452
42k
KATA
mclloyd
29
14k
Done Done
chrislema
183
16k
How to train your dragon (web standard)
notwaldorf
90
6k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao