Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic{ON} 2018: Latest in Logstash

Elastic Co
March 01, 2018

Elastic{ON} 2018: Latest in Logstash

Much has happened since 5.0. Persistent queues, pipeline viewer (x-ray vision, basically) and the ability to run multiple pipelines at the same time for different use cases, and a move to the latest version of JRuby — it's all laying the foundation for even more goodness to come. See where the Logstash roadmap is headed and what to expect next.

Elastic Co

March 01, 2018
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Elastic February 28, 2018 @jordansissel / @andrewvc What’s the Latest

    in Logstash Jordan Sissel and Andrew Cholakian
  2. In the last year, 1,413 Logstashers have helped us with

    9,237 issues, comments, and pull requests to our logstash-plugins repository. 864 pull requests were opened. The Logstash Plugins Community 3
  3. In the last year, 76 Logstashers have helped us with

    1,131 issues, comments, and pull requests to our logstash-plugins repository. 179 pull requests were opened. The Logstash Core Community 4
  4. queue.type: persisted # (v5.4) input input input filter filter filter

    output output Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY queue Theme: Don’t lose data.
  5. 1 8 input { beats { port => 3444 tag

    => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } } TCP BEATS DISSECT GROK TCP ES
  6. 1 9 input { beats { port => 3444 tag

    => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } } input { beats { port => 3444 tag => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } }
  7. 2 0 input { beats { port => 3444 tag

    => apache } } filter { dissect { ... } } output { elasticsearch { ... } } input { tcp { port => 4222 tag => firewall } } filter { grok { ... } } output { tcp { ... } } Multiple Pipelines (v6.0) BEATS DISSECT ES TCP GROK TCP
  8. 2 1

  9. X-Pack Central Management (v6.0) ------- ----------- -------- - ------ ---

    ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- Three Logstash instances. Doing the same thing. Let’s simplify this.
  10. X-Pack Central Management (v6.0) ------- ----------- -------- - ------ ---

    ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- One configuration source.
  11. 2 5

  12. --experimental-java-execution (v6.1) 27 0: new #121 3: dup 4: iconst_2

    5: fconst_1 6: invokespecial #122 9: astore_1 10: aload_1 11: ldc #47 13: aload_0 14: getfield #6 17: invokeinterface #52, 3 filter { if “debug” in [tags] { drop { } } grok { match => { … } } } if event.getField(“tags”)... { drop.execute(event); if event.isCancelled() { return; } } grok.execute(event); Compile bytecode Load Configuration Transform to Java
  13. --experimental-java-execution (v6.1) 28 “Compile” code = config.compile() pipeline = eval(code)

    loop do batch = queue.pop() pipeline.execute(batch) end Load Configuration filter { if “debug” in [tags] { drop { } } grok { match => { … } } } Transform to Ruby if event.get(“tags”).include?( drop.execute(event) return if event.cancelled? end grok.execute(event) Ruby
  14. Logstash 6.2: Protect credentials with the keystore logstash-keystore (Logstash 6.2)

    % logstash-keystore create % logstash-keystore add es_password # use es_password in the pipeline: output { elasticsearch { hosts => … user => “elastic” password => “${es_password}” } }
  15. How we do it today 3 7 xpack.management.enabled: true xpack.management.pipeline.id:

    ["apache", "cloudwatch_logs"] LS1 LS2 LS3 apache cloudwatch logs
  16. Where we Want to Go 3 8 xpack.management.enabled: true xpack.management.pipeline.node_groups:

    ["webserver_logs”, “security_logs”] LS1 LS2 LS3 apache logs cloudwatch logs webserver_logs security_logs nginx logs azure activity logs
  17. • Allow people to use additional JVM languages to develop

    plugins • Enable performance optimization where required Java Plugin API Goals 4 5
  18. • We’re starting with a low level API, optimized for

    performance • Will add sugar on top • We plan to support the current plugin API indefinitely • Some esoteric APIs, like flush, may go away Java Plugin API Plan 4 6
  19. How we do it today 4 9 output { lumbjerjack

    { … } } input { beats { … } } ? ? ?
  20. - pipeline.id: senderone config.string: "input { generator { message =>

    huhx } } output { internal { send_to => [foo] } }" - pipeline.id: sendertwo config.string: "input { generator { message => whutx } } output { internal { send_to => [foo] } }" - pipeline.id: out config.string: " input { internal { address => foo } } output { stdout { codec => json_lines } }" What it looks like 5 1 senderone sendertwo out
  21. Share a Port 5 2 Beats Ingest + Routing Enrich

    Weblogs + Output Enrich ETL Logs + Output Enrich Metrics + Output
  22. Buffer as Needed 5 3 PQ PQ Logs Ingest Output

    to Elasticsearch Output to S3
  23. Put it Together 5 4 PQ PQ PQ Beats Ingest

    and Route Enrich Weblogs Enrich ETL Logs Enrich Metrics + Output to Metrics ES Cluster Output to Logging ES Cluster
  24. • Is not null checks • Array/field reference conflation (is

    [foo] an array, or a field reference?) • What else does the lang need? • Can we experiment with new languages? The Language Can Move Forward 5 6
  25. An Ephemeral World Or, how I learned to stop worrying

    about my disk and embrace ephemeral storage
  26. 61 You need to be able to tolerate data loss

    if you use the in- memory queue
  27. • TCP, UDP, and Syslog protocol offer no facility to

    replay • The best strategy here is to store ASAP on Logstash, and then try to get it elsewhere ASAP Where we can’t replay, the PQ is still best 6 8
  28. • Currently in progress • Will likely require rewrites to

    input plugins to be efficient • Only works for things that are replayable • Luckily, a lot of things are replayable E2E ACK Summary 6 9
  29. • Users must implement backups of metadata • Scaling past

    one box requires manual partitioning, or is not possible • Failover is tricky, and involves restoring backed up metadata Why Local State is Irritating 7 5
  30. • Plugin state kept in ES • Leader election through

    ES • Task assignment through ES • Still in design phase, we have a PoC in progress How it Works 7 9