Life of a Logstash Event

Join me for this presentation to follow the journey of a Logstash event as it travels through the internal Logstash processing pipeline. Understanding Logstash internals will help you make better design decisions for your streaming data processing architecture.

You will learn how data is transported in and out of Logstash through its input and output plugins, how Logstash decodes and encodes external data formats using the codec plugins and how the parsing, transformation and enrichment of this data is executed by the filter plugins.

We will follow an event as it travels through the pipeline stages and internal queuing, what happens when things start to break downstream, plus how congestion and backpressure is handled.

The Logstash pipeline is multithreaded, so you’ll also learn how to scale up and take advantage of all your host CPU cores and how to leverage parallelism to accelerate the processing throughput.

Logstash is an amazingly powerful and flexible tool. As a Logstash user, this presentation will help you optimize your logstash configuration and, for developers, you will have much better insights for contributing plugins.

Transcript

1. Life of an Event in Logstash Colin Surprenant, Elasticsearch Software Engineer colin.surprenant@elastic.co colinsurprenant
2. { } CC-BY-ND 4.0 Agenda 2 •Logstash quick intro •Pipeline overview •Plugin architecture •Scaling up Logstash •Pipeline stages •input •filter •output
3. { } CC-BY-ND 4.0 Logstash quick intro Collect, parse and store logs 3
4. { } CC-BY-ND 4.0 Logstash quick intro • Logs? 4 ALL  THE  DATA  with  a  timestamp No  timestamp?  Hey!  
 we’ll  add  one  for  you.  
5. { } CC-BY-ND 4.0 Logstash quick intro Transport & processing
 of
 streaming / continuous data 5
6. { } CC-BY-ND 4.0 Logstash quick intro – Heterogenous formats and protocols – Unstructured format – Decentralized 6 Logging problems
7. { } CC-BY-ND 4.0 Definitions • Event • Plugin (input, filter, output) • Pipeline 7
8. { } CC-BY-ND 4.0 Logstash pipeline 8
9. { } CC-BY-ND 4.0 Logstash pipeline 9 3 stages pipeline
10. { } CC-BY-ND 4.0 Plugin architecture • ~200 plugins https://github.com/logstash-plugins • Input plugins: captures external data+format & transform it to logstash events • Filter plugins: process/transform events • Output plugins: send events to external destination & format 10
11. { } CC-BY-ND 4.0 Plugin architecture • Codecs are plugins • Optional part of input and output plugins • encode and decode raw data that enter or exit the pipeline • Character encoding/transcoding into UTF-8 11 Codecs
12. { } CC-BY-ND 4.0 Plugin architecture 12
13. { } CC-BY-ND 4.0 Logstash pipeline 13 3 stages pipeline
14. { } CC-BY-ND 4.0 Logstash pipeline 14 Internal queuing
15. { } CC-BY-ND 4.0 Logstash pipeline 15 backpressure
16. { } CC-BY-ND 4.0 Scaling out Logstash 16 shipper/indexer architecture
17. { } CC-BY-ND 4.0 Scaling up the pipeline • Logstash pipeline is multithreaded • Each stage offers configurable concurrency options • Leverage multiple cores in host 17
18. { } CC-BY-ND 4.0 Pipeline input stage 18
19. { } CC-BY-ND 4.0 Pipeline input worker 19
20. { } CC-BY-ND 4.0 Pipeline filter stage 20
21. { } CC-BY-ND 4.0 Pipeline filter stage 21
22. { } CC-BY-ND 4.0 Pipeline filter stage 22
23. { } CC-BY-ND 4.0 Pipeline output stage 23
24. { } CC-BY-ND 4.0 Pipeline output stage 24
25. { } CC-BY-ND 4.0 Function compilation • Filter and output stages are compiled to a single function • Anyone knows or can guess why? 25
26. { } CC-BY-ND 4.0 Function Compilation • Hint 26
27. { } CC-BY-ND 4.0 Function Compilation • Conditionals • Message passing is not free 27
28. { } CC-BY-ND 4.0 Logstash pipeline 28
29. { } Thank you! Colin Surprenant, Elasticsearch Software Engineer colin.surprenant@elastic.co colinsurprenant
30. { } This work is licensed under the Creative Commons Attribution-NoDerivatives 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by-nd/4.0/ or send a letter to: Creative Commons PO Box 1866 Mountain View, CA 94042 USA CC-BY-ND 4.0