Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
340
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
65
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
86
Scaling Laravel - Laracon.net 2018
fideloper
15
1.9k
Linux Environment
fideloper
1
11k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
600
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.3k
Other Decks in Technology
See All in Technology
データ戦略部門 紹介資料
sansan33
PRO
1
3.8k
ガバメントクラウドの概要と自治体事例(名古屋市)
techniczna
3
240
それでも私が品質保証プロセスを作り続ける理由 #テストラジオ / Why I still continue to create QA process
pineapplecandy
0
140
サイバーエージェント流クラウドコスト削減施策「みんなで金塊堀太郎」
kurochan
4
2.1k
ローカルLLMとLINE Botの組み合わせ その2(EVO-X2でgpt-oss-120bを利用) / LINE DC Generative AI Meetup #7
you
PRO
0
100
プロダクトのコードから見るGoによるデザインパターンの実践 #go_night_talk
bengo4com
1
2.7k
Claude Codeを駆使した初めてのiOSアプリ開発 ~ゼロから3週間でグローバルハッカソンで入賞するまで~
oikon48
10
5.1k
Railsの話をしよう
yahonda
0
170
GoでもGUIアプリを作りたい!
kworkdev
PRO
0
160
事業開発におけるDify活用事例
kentarofujii
3
820
新規事業におけるGORM+SQLx併用アーキテクチャ
hacomono
PRO
0
430
生成AI時代のセキュアコーディングとDevSecOps
yuriemori
0
130
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.7k
Building Applications with DynamoDB
mza
96
6.7k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
The Cost Of JavaScript in 2023
addyosmani
55
9k
Being A Developer After 40
akosma
91
590k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3.1k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
253
22k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
127
54k
Unsuck your backbone
ammeep
671
58k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao