Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Session - Radu Gheorghe
Search
Monitorama
September 20, 2013
0
580
Berlin 2013 - Session - Radu Gheorghe
Monitorama
September 20, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
560
PDX 2017 - Pedro Andrade
monitorama
0
650
PDX 2017 - Roy Rapoport
monitorama
4
890
PDX 2017 - Julia Evans
monitorama
0
420
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
660
Berlin 2013 - Session - Alex Petrov
monitorama
6
640
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
570
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
490
Berlin 2013 - Session - David Goodlad
monitorama
0
380
Featured
See All Featured
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
355
29k
The World Runs on Bad Software
bkeepers
PRO
65
11k
Adopting Sorbet at Scale
ufuk
73
9k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.8k
Happy Clients
brianwarren
97
6.7k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
GraphQLとの向き合い方2022年版
quramy
43
13k
Faster Mobile Websites
deanohume
304
30k
How to Think Like a Performance Engineer
csswizardry
19
1.1k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Mobile First: as difficult as doing things right
swwweet
222
8.9k
Transcript
On Centralizing Logs Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
Hello World! Logsene mlmoneu13cf for -44%
app app app app files files
app app app app files files Elasticsearch logstash Kibana
Elasticsearch Reason #1: Quick Search No indexing But... =>
...and other reasons good write speed lots of tools for
logging scales easily
Production Tips stability performance
Stability 1/4: Discovery multicast unicast vs cluster name list of
nodes + plugins: EC2, GCE
Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor
the requirements SPM for Elasticsearch 20% off with MONEU2013
Stability 4/4: Field Cache can be changed to index.cache.field.type: soft
indices.fielddata.cache.size: X%
Performance 1/4: Bulk Processing use Bulk API or Bulk UDP
API ...translog.flush_threshold_ops
Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but
every 5s +25% indexing* every 30s +70% indexing*
Performance 3/4: Timed Indices
Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit
machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Setting Up Kibana as Frontend servers you
Kibana: Search
Kibana: Visualize
Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE
documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
X-ray of a Modern Syslog Daemon read+buffer file /dev/log …
parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world
TCP + LF = no year, ms, nor TZ little structure
2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin
ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize
2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS
RELP + TLS
Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)
outputs (+codecs) lots of plugins => lots of options
Logstash: Example Lumberjack Logstash Elasticsearch
Logstash: Add Buffer Lumberjack Lumberjack
Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack
Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene
(More) Alternatives files syslog
Alternatives Can Mix files syslog Logstash Elasticsearch Kibana
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
None
rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance
nicer config format omelasticsearch
rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or
LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific
queue.dequeuebatchsize="1000" queue.workerthreads="3"
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext