Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
300
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
29
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
51
Scaling Laravel - Laracon.net 2018
fideloper
15
1.8k
Linux Environment
fideloper
1
10k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
540
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
プロダクト活用度で見えた真実 ホリゾンタルSaaSでの顧客解像度の高め方
tadaken3
0
150
Lambdaと地方とコミュニティ
miu_crescent
2
370
Terraform Stacks入門 #HashiTalks
msato
0
360
第1回 国土交通省 データコンペ参加者向け勉強会③- Snowflake x estie編 -
estie
0
130
B2B SaaSから見た最近のC#/.NETの進化
sansantech
PRO
0
870
なぜ今 AI Agent なのか _近藤憲児
kenjikondobai
4
1.4k
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
8
870
【Pycon mini 東海 2024】Google Colaboratoryで試すVLM
kazuhitotakahashi
2
530
Flutterによる 効率的なAndroid・iOS・Webアプリケーション開発の事例
recruitengineers
PRO
0
110
いざ、BSC討伐の旅
nikinusu
2
780
Platform Engineering for Software Developers and Architects
syntasso
1
520
データプロダクトの定義からはじめる、データコントラクト駆動なデータ基盤
chanyou0311
2
330
Featured
See All Featured
Typedesign – Prime Four
hannesfritz
40
2.4k
Art, The Web, and Tiny UX
lynnandtonic
297
20k
Teambox: Starting and Learning
jrom
133
8.8k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
Statistics for Hackers
jakevdp
796
220k
4 Signs Your Business is Dying
shpigford
180
21k
Testing 201, or: Great Expectations
jmmastey
38
7.1k
Building Your Own Lightsaber
phodgson
103
6.1k
Ruby is Unlike a Banana
tanoku
97
11k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.8k
We Have a Design System, Now What?
morganepeng
50
7.2k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao