Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
340
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
60
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
80
Scaling Laravel - Laracon.net 2018
fideloper
15
1.9k
Linux Environment
fideloper
1
11k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
590
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
生成AIによるソフトウェア開発の収束地点 - Hack Fes 2025
vaaaaanquish
35
16k
[CV勉強会@関東 CVPR2025 読み会] MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos (Li+, CVPR2025)
abemii
0
180
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
kzykmyzw
0
300
プロジェクトマネジメントは不確実性との対話だ
hisashiwatanabe
0
190
Engineering Failure-Resilient Systems
infraplumber0
0
130
AIは変更差分からユニットテスト_結合テスト_システムテストでテストすべきことが出せるのか?
mineo_matsuya
5
3.1k
広島発!スタートアップ開発の裏側
tsankyo
0
190
Infrastructure as Prompt実装記 〜Bedrock AgentCoreで作る自然言語インフラエージェント〜
yusukeshimizu
2
180
Devinを使ったモバイルアプリ開発 / Mobile app development with Devin
yanzm
0
140
人を動かすことについて考える
ichimichi
2
300
AIエージェント就活入門 - MCPが履歴書になる未来
eltociear
0
110
AIドリブンのソフトウェア開発 - うまいやり方とまずいやり方
okdt
PRO
9
510
Featured
See All Featured
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Fireside Chat
paigeccino
39
3.6k
Navigating Team Friction
lara
188
15k
The Language of Interfaces
destraynor
160
25k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
1k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
Practical Orchestrator
shlominoach
190
11k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao