Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
330
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
50
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
77
Scaling Laravel - Laracon.net 2018
fideloper
15
1.9k
Linux Environment
fideloper
1
11k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
580
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
SRE不在の開発チームが障害対応と 向き合った100日間 / 100 days dealing with issues without SREs
shin1988
1
250
対話型音声AIアプリケーションの信頼性向上の取り組み
ivry_presentationmaterials
1
280
FOSS4G 2025 KANSAI QGISで点群データをいろいろしてみた
kou_kita
0
400
Claude Code に プロジェクト管理やらせたみた
unson
6
4.5k
ABEMAの本番環境負荷試験への挑戦
mk2taiga
3
130
AI時代の開発生産性を加速させるアーキテクチャ設計
plaidtech
PRO
3
160
タイミーのデータモデリング事例と今後のチャレンジ
ttccddtoki
6
2.4k
Lufthansa ®️ USA Contact Numbers: Complete 2025 Support Guide
lufthanahelpsupport
0
210
Coinbase™®️ USA Contact Numbers: Complete 2025 Support Guide
officialcoinbasehelpcenter
0
440
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
shift_evolve
PRO
2
8k
Sansanのデータプロダクトマネジメントのアプローチ
sansantech
PRO
0
170
Yahoo!しごとカタログ 新しい境地を創るエンジニア募集!
lycorptech_jp
PRO
0
130
Featured
See All Featured
The Language of Interfaces
destraynor
158
25k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Code Reviewing Like a Champion
maltzj
524
40k
The Pragmatic Product Professional
lauravandoore
35
6.7k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao