Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Session - Radu Gheorghe
Search
Monitorama
September 20, 2013
0
630
Berlin 2013 - Session - Radu Gheorghe
Monitorama
September 20, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
590
PDX 2017 - Pedro Andrade
monitorama
0
750
PDX 2017 - Roy Rapoport
monitorama
4
950
PDX 2017 - Julia Evans
monitorama
0
480
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
720
Berlin 2013 - Session - Alex Petrov
monitorama
6
690
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
630
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
550
Berlin 2013 - Session - David Goodlad
monitorama
0
460
Featured
See All Featured
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
1.1k
Git: the NoSQL Database
bkeepers
PRO
431
66k
Building an army of robots
kneath
306
46k
Practical Orchestrator
shlominoach
190
11k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Speed Design
sergeychernyshev
32
1.1k
Site-Speed That Sticks
csswizardry
10
800
Typedesign – Prime Four
hannesfritz
42
2.8k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Embracing the Ebb and Flow
colly
87
4.8k
Designing Experiences People Love
moore
142
24k
Transcript
On Centralizing Logs Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
Hello World! Logsene mlmoneu13cf for -44%
app app app app files files
app app app app files files Elasticsearch logstash Kibana
Elasticsearch Reason #1: Quick Search No indexing But... =>
...and other reasons good write speed lots of tools for
logging scales easily
Production Tips stability performance
Stability 1/4: Discovery multicast unicast vs cluster name list of
nodes + plugins: EC2, GCE
Stability 2/4: Preventing Split Brain minimum_master_nodes = N/2 + 1
Stability 3/4: No OOMs, pls! 1GB ½ total RAM Monitor
the requirements SPM for Elasticsearch 20% off with MONEU2013
Stability 4/4: Field Cache can be changed to index.cache.field.type: soft
indices.fielddata.cache.size: X%
Performance 1/4: Bulk Processing use Bulk API or Bulk UDP
API ...translog.flush_threshold_ops
Performance 2/4: Refresh Interval http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ default: every second => but
every 5s +25% indexing* every 30s +70% indexing*
Performance 3/4: Timed Indices
Performance 4/4: Buffers ...index_buffer_size: 30% (YMMV) index.store.type: mmapfs (on 64-bit
machines) http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Setting Up Kibana as Frontend servers you
Kibana: Search
Kibana: Visualize
Meet Some Syslog Daemons syslogd traditional everywhere syslog-ng OSE, PE
documentation++ config format++ rsyslog OSS only ES output* * http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/
X-ray of a Modern Syslog Daemon read+buffer file /dev/log …
parse syslog formats JSON unstructured data assemble conditionals formatting ... buffer+write file syslog Elasticsearch ...
2001's RFC3164: The Semi-Standard <10>Oct 11 22:14:15 host program:hello world
TCP + LF = no year, ms, nor TZ little structure
2009's RFC5424 <165>1 2003-10-11T22:14:15.003Z host program - - - [origin
ip="192.168.0.1"] hello world [ structured=data ] octet-count* + LF = * UDP (RFC5426), TCP (RFC6587), TLS (RFC5425)
Teaching Old Dog New Tricks RSYSLOG_ForwardFormat (ISO8601 over RFC3164) $MaxMessageSize
2048k log_message_size(2097152) @cee: {"message": "hello world"} @@(o)192.168.0.1 octet-counted framing
Reliable Transport? Encryption? TCP + TLS (RFC5425) RLTP + TLS
RELP + TLS
Logstash: The Swiss Army Knife inputs (+codecs) filters (parse, modify)
outputs (+codecs) lots of plugins => lots of options
Logstash: Example Lumberjack Logstash Elasticsearch
Logstash: Add Buffer Lumberjack Lumberjack
Logstash: Scale Everything Lumberjack Lumberjack Lumberjack Lumberjack
Back to the Beginning Lumberjack Lumberjack Lumberjack Lumberjack syslogd
Logsene Lumberjack Lumberjack Lumberjack Lumberjack syslogd Logsene http://sematext.com/logsene
(More) Alternatives files syslog
Alternatives Can Mix files syslog Logstash Elasticsearch Kibana
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext
None
rsyslog 1/4: Upgrade to 7.x RPMs or DEBs better performance
nicer config format omelasticsearch
rsyslog 2/4: Faster Inputs UDP increase TimeRequery TCP use imptcp
rsyslog 3/4: Main Message Queue $MainMsgQueueType FixedArray $MainMsgQueueSize 1000000.... ...or
LinkedList or Disk $...DequeueBatchSize 1000 $...WorkerThreads 3
rsyslog 4/4: Action Queue queue.type="linkedlist" queue.size="1000000" bulkmode="on" # ES specific
queue.dequeuebatchsize="1000" queue.workerthreads="3"
Thank you! Radu Gheorghe @radu0gheorghe
[email protected]
@sematext