Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
290
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
26
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
47
Scaling Laravel - Laracon.net 2018
fideloper
15
1.8k
Linux Environment
fideloper
1
10k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
530
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.2k
Other Decks in Technology
See All in Technology
ナレッジグラフとLLMの相互利用
koujikozaki
0
400
たった1人からはじめる【Agile Community of Practice】~ソース原理とFearless Changeを添えて~
ktc_corporate_it
1
430
PDF Viewer作成の今までとこれから
hunachi
0
390
持続可能なソフトウェア開発を支える『GitHub CI/CD実践ガイド』
tmknom
6
1.2k
Agile in Automotive Industry, puzzles and lights.
hiranabe
3
1.3k
The XZ Backdoor Story
fr0gger
0
3.6k
OR学会2024秋_短期収益と将来のオフ方策評価性能を考慮したクーポン割当方策混合比の決定
recruitengineers
PRO
4
460
ロリポップ! for Gamersを支えるインフラ/lolipop for gamers infrastructure
takumakume
0
130
o1のAPIで実験してみたが 制限きつすぎて辛かった話
pharma_x_tech
0
110
技術的負債解消の取り組みと専門チームのお話
bengo4com
0
330
「自動テストのプラクティスを効果的に学ぶためのカードゲーム」 ( #sqip2024 )
teyamagu
PRO
2
180
事前準備が肝!AI活用のための業務改革
layerx
PRO
1
370
Featured
See All Featured
Infographics Made Easy
chrislema
239
18k
Building Flexible Design Systems
yeseniaperezcruz
325
38k
Designing the Hi-DPI Web
ddemaree
278
34k
Build The Right Thing And Hit Your Dates
maggiecrowley
30
2.3k
Writing Fast Ruby
sferik
623
60k
Designing with Data
zakiwarfel
98
5k
The Illustrated Children's Guide to Kubernetes
chrisshort
47
48k
What the flash - Photography Introduction
edds
67
11k
Large-scale JavaScript Application Architecture
addyosmani
508
110k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
36
2.1k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
38
9.2k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
227
52k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao