Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
FileBeat (Won't save you from the JVM)
Search
Chris Fidao
January 24, 2016
Technology
1
300
FileBeat (Won't save you from the JVM)
A quick presentation about using Elastic's FileBeat for log aggregation.
Chris Fidao
January 24, 2016
Tweet
Share
More Decks by Chris Fidao
See All by Chris Fidao
Development Environments that Feel Local
fideloper
0
28
Refactoring Terraform - CloudCasts - Scaling EC2
fideloper
0
50
Scaling Laravel - Laracon.net 2018
fideloper
15
1.8k
Linux Environment
fideloper
1
10k
Server Survival
fideloper
29
23k
Powering Your Applications With Nginx
fideloper
9
7.7k
Hexagonal Architecture
fideloper
49
200k
Intro to etcd
fideloper
3
530
Service Oriented Architecture with a little help from NodeJS
fideloper
4
2.2k
Other Decks in Technology
See All in Technology
2024-10-30-reInventStandby_StudyGroup_Intro
shinichirokawano
1
640
コンテンツを支える 若手ゲームクリエイターの アートディレクションの事例紹介 / cagamefi-game
cyberagentdevelopers
PRO
1
130
10分でわかるfreee エンジニア向け会社説明資料
freee
18
520k
生成AIの強みと弱みを理解して、生成AIがもたらすパワーをプロダクトの価値へ繋げるために実践したこと / advance-ai-generating
cyberagentdevelopers
PRO
1
180
Commitment vs Harrisonism - Keynote for Scrum Niseko 2024
miholovesq
6
1.1k
ガチ勢によるPipeCD運用大全〜滑らかなCI/CDを添えて〜 / ai-pipecd-encyclopedia
cyberagentdevelopers
PRO
3
210
MAMを軸とした動画ハンドリングにおけるAI活用前提の整備と次世代ビジョン / abema-ai-mam
cyberagentdevelopers
PRO
1
120
Datachain会社紹介資料(2024年11月) / Company Deck
datachain
4
16k
ABEMA のコンテンツ制作を最適化!生成 AI x クラウド映像編集システム / abema-ai-editor
cyberagentdevelopers
PRO
1
180
LeSSに潜む「隠れWF病」とその処方箋
lycorptech_jp
PRO
2
120
プロポーザルのつくり方 〜個人技編〜 / How to come up with proposals
ohbarye
2
140
IaC運用を楽にするためにCDK Pipelinesを導入したけど、思い通りにいかなかった話
smt7174
1
120
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
7.9k
YesSQL, Process and Tooling at Scale
rocio
167
14k
How to Think Like a Performance Engineer
csswizardry
19
1.1k
Side Projects
sachag
452
42k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
27
1.9k
Building Applications with DynamoDB
mza
90
6.1k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.1k
Raft: Consensus for Rubyists
vanstee
136
6.6k
Code Review Best Practice
trishagee
64
17k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
32
1.8k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
BBQ
matthewcrist
85
9.3k
Transcript
FileBeat (Won’t save you from the JVM)
Beats Data Shippers for Elasticsearch (written in Golang)
PacketBeat TopBeat
FileBeat
Versus:
!
None
None
None
Goal: "
# Download Filebeat Package (Debian/Ubuntu) curl -L -O https://download.elastic.co/beats/filebeat/ filebeat_1.0.1_amd64.deb
# Install from .deb file, # without worrying about dependencies, # because Golang ! sudo dpkg -i filebeat_1.0.1_amd64.deb #!/usr/bin/env bash
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: elasticsearch: hosts: [“https://search-sadevops.us-east-1.es.aws.com:443"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
useless (un-parsed message)
Plaintext Log Message useless
You Can’t Escape the JVM
None
Plaintext Log Message Parsed Log (JSON)
# Install Java sudo apt-get install -y openjdk-7-jdk # Add
ES Key wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add - # Add Repo echo "deb http://packages.elastic.co/logstash/2.1/debian stable main" | sudo tee /etc/apt/sources.list.d/logstash.list # Update and install package sudo apt-get update sudo apt-get install -y logstash # Install Filebeat Plugin sudo /opt/logstash/bin/plugin install logstash-input-beats On a new server…
input { beats { type => beats port => 5044
} } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } remove_tag => ["_grokparsefailure"] add_tag => ["nginx_access"] } } output { elasticsearch { hosts => ["search-sadevops.us-east-1.es.aws.com:80"] } } /etc/logstash/conf.d/filebeat.conf
filebeat: prospectors: - paths: - /var/log/nginx/*.log input_type: log - paths:
- /var/log/php7.0-fpm.log input_type: log output: logstash: hosts: ["172.31.28.187:5044"] shipper: tags: ["web-service", "or-like-whatever"] /etc/filebeat/filebeat.yml
None
None
$ ab -n 50000 -c 2 localhost/ mehhhhhh
40% 250mb
Conclusion: (for my use case) Fluentd is good enough. •One
less server (yay!) •Trade-off of more ram used: acceptable •JVM is “scary”, because I’m ignorant •(But PacketBeat and TopBeat look really useful)
@fideloper Thanks! Chris Fidao