Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Log Aggregation - Using Fluentd + Elasticsearch...

Ash Wu
November 25, 2014

Log Aggregation - Using Fluentd + Elasticsearch + Kibana

Ruby Tuesday #32 @Taiwan

Ash Wu

November 25, 2014
Tweet

More Decks by Ash Wu

Other Decks in Programming

Transcript

  1. Issues to Solve • Log files are scattered across servers.

    • grep & zgrep through all the mess. • Servers come and go(autoscaling) and we lose log data. • Keep logs of last 6 months. => Querying & Archiving hsatac@KKTIX, 2014 4
  2. Fluentd • Data collector • Open sourced • Written in

    Ruby (Easy to write plugins) • Combined C language and Ruby for performance • Process 13,000 events/second/core • Simple, flexible and reliable. hsatac@KKTIX, 2014 6
  3. No Fixed 2-tier Setup • Where's the server? Where's the

    clients? • A fluentd intance is a node. • Each node and be a server, or a client, or both at the same time. • Define the data flow and servers topology as you like. hsatac@KKTIX, 2014 9
  4. Input • in_forward (listen on udp & tcp) • in_http

    • in_tail (tail a regular file) • in_exec • in_syslog hsatac@KKTIX, 2014 11
  5. Output • out_file • out_foward (Send to other fluentd instance)

    • out_exec • out_copy • out_stdout • out_s3 hsatac@KKTIX, 2014 12
  6. Buffered Output e.g. out_file, out_forward Events will be flushed when:

    • Events chunk size exceeds buffer_chunk_limit • Time limie flush_interval hsatac@KKTIX, 2014 14
  7. Time Sliced Plugin • Flush regularly (daily, hourly...) according to

    time_slice_format. • e.g. %Y%m%d for daily chunk • Just like logrotate hsatac@KKTIX, 2014 16
  8. Elasticsearch & Kibana • Elasticsearch • Distributed restful search and

    analytics (schema free) • Kibana • Realtime query frontend • Data visualizer • No code required hsatac@KKTIX, 2014 23
  9. Official Document http://www.fluentd.org/datasources/rails • Rails side: • lograge gem (supress

    log) • act-fluent-logger-rails send log to fluentd hsatac@KKTIX, 2014 36
  10. Official Document http://www.fluentd.org/datasources/rails • Fluentd side: • fluent-plugin-parser plugin to

    parse JSON. • fluent-plugin-elasticsearch plugin send parsed data to elasticsearch. hsatac@KKTIX, 2014 37
  11. Official Document http://www.fluentd.org/datasources/rails • lograge is good for access log,

    but what about other logs? • act-fluent-logger-rails crashes puma • What's all the fuss about JSON encode / decode thing? hsatac@KKTIX, 2014 39
  12. Attempt 2 • act-fluent-logger-rails only to keep all the logs.

    • Replace puma with unicorn to avoid threading issue. hsatac@KKTIX, 2014 41
  13. Attempt 2 • All logs are kept, but the format

    is not search friendly. 2013-01-18T15:04:50+09:00 foo { "messages":"Started GET \"/\" for 127.0.0.1 at 2013-01-18 15:04:49 +0900\n Processing by TopController#index as HTML\n Completed 200 OK in 635ms (Views: 479.3ms | ActiveRecord: 39.6ms)"], "level":"INFO" } • Unicorn return weird 404 with act-fluent-logger-rails gem under massive requests. hsatac@KKTIX, 2014 43
  14. Attempt 3 • Rails side: • Remove act-fluent-logger-rails, unstable. •

    Keep lograge gem, using Logstash formatter. • logstash-logger gem converts other logs into logstash format. • Output logs to production.log. hsatac@KKTIX, 2014 47
  15. Attempt 3 • Fluentd side: • Tail production.log with format

    json • Forward to station fluentd. • Done. hsatac@KKTIX, 2014 48