Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
330
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
52
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
78
Scaling Laravel - Laracon.net 2018
fideloper
15
1.9k
Linux Environment
fideloper
1
11k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
590
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
2025新卒研修・HTML/CSS #弁護士ドットコム
bengo4com
3
4.6k
AI人生苦節10年で会得したAIがやること_人間がやること.pdf
shibuiwilliam
1
250
【CEDEC2025】LLMを活用したゲーム開発支援と、生成AIの利活用を進める組織的な取り組み
cygames
PRO
1
2.2k
【CEDEC2025】『Shadowverse: Worlds Beyond』二度目のDCG開発でゲームをリデザインする~遊びやすさと競技性の両立~
cygames
PRO
1
220
마라톤 끝의 단거리 스퍼트: 2025년의 AI
inureyes
PRO
1
450
金融サービスにおける高速な価値提供とAIの役割 #BetAIDay
layerx
PRO
1
530
「AI駆動開発」のボトルネック『言語化』を効率化するには
taniiicom
1
230
ビジネス文書に特化した基盤モデル開発 / SaaSxML_Session_2
sansan_randd
0
210
SAE J1939シミュレーション環境構築
daikiokazaki
1
200
Mambaで物体検出 完全に理解した
shirarei24
2
170
東京海上日動におけるセキュアな開発プロセスの取り組み
miyabit
0
220
バクラクによるコーポレート業務の自動運転 #BetAIDay
layerx
PRO
1
610
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
301
21k
We Have a Design System, Now What?
morganepeng
53
7.7k
GraphQLとの向き合い方2022年版
quramy
49
14k
Why Our Code Smells
bkeepers
PRO
337
57k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
How to Think Like a Performance Engineer
csswizardry
25
1.8k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Become a Pro
speakerdeck
PRO
29
5.5k
Navigating Team Friction
lara
188
15k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Gamification - CAS2011
davidbonilla
81
5.4k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao