Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic Stack Workshop

Elastic Stack Workshop

One of the hottest and most widely used tools for log monitoring and analytics at the moment is the Elastic Stack:
• Elasticsearch doing the hard work of analyzing and searching large amounts of data.
• Logstash and Beats for collecting them.
• Kibana for powerful visualizations.

This workshop gives you an overview of the four technologies, how they are working together, and how they can solve your problems.

Demo: https://github.com/xeraa/vagrant-elastic-stack

Philipp Krenn

June 28, 2017
Tweet

More Decks by Philipp Krenn

Other Decks in Programming

Transcript

  1. 11

  2. 12

  3. 14

  4. Insert data PUT /movies/movie/1 { "title": "The Godfather", "director": "Francis

    Ford Coppola", "year": 1972 } GET /movies/movie/1 GET /movies/_mapping 17
  5. Replace data PUT /movies/movie/1 { "title": "The Godfather", "director": "Francis

    Ford Coppola", "year": 1972, "genres": ["Crime", "Drama"] } GET /movies/movie/1 18
  6. More data PUT /movies/movie/2 { "title": "Lawrence of Arabia", "director":

    "David Lean", "year": 1962, "genres": ["Adventure", "Biography", "Drama"] } PUT /movies/movie/3 { "title": "Apocalypse Now", "director": "Francis Ford Coppola", "year": 1979, "genres": ["Drama", "War"] } 19
  7. More complex search GET /person/person/_search { "query": { "bool": {

    "must": [ { "match": { "address.country": "Germany" } }, { "range": { "dateOfBirth": { "from": "1970", "to": "1971" } } } ] } } } 27
  8. Aggregation GET /person/person/_search { "size": 0, "aggs": { "by_country": {

    "terms": { "field": "address.country" } } } } 28
  9. 36

  10. 41

  11. Logstash Filter $ sudo tee -a /etc/logstash/conf.d/11-nginx-filter.conf >/dev/null <<'EOF' filter

    { if [type] == "nginx-access" { grok { patterns_dir => ["/opt/logstash/patterns"] match => { "message" => "%{NGINXACCESS}" } } } } EOF $ sudo service logstash restart 49
  12. Flows Application layer: Unsupported / encrypted (TLS) protocols IP /

    TCP / UDP Number of packets & bytes Retransmissions Temporal flow 57
  13. 62

  14. Monitoring GET _nodes/stats { "_nodes" : { "total" : 1,

    "successful" : 1, "failed" : 0 }, "cluster_name" : "elasticsearch", "nodes" : { "Koy5OmQ5RoiuFlL_TQ2Ngg" : { "timestamp" : 1492549940504, "name" : "Koy5OmQ", ... 65
  15. 66

  16. Search Profiler GET /person "profile": true, "size": 0, "query": {

    ... } ... "searches" : [ { "query" : [ { "type" : "BooleanQuery", "description" : "...", "time" : "2.366942000ms", "time_in_nanos" : 2366942, "breakdown" : { "score" : 0, "build_scorer_count" : 5, ... 67
  17. 68

  18. 69

  19. GET _xpack/license { "license" : { "status" : "active", "uid"

    : "...", "type" : "trial", "issue_date" : "2017-04-18T20:41:50.233Z", "issue_date_in_millis" : 1492548110233, "expiry_date" : "2017-05-18T20:41:50.233Z", "expiry_date_in_millis" : 1495140110233, "max_nodes" : 1000, "issued_to" : "elasticsearch", "issuer" : "elasticsearch", "start_date_in_millis" : -1 } } 71
  20. 72

  21. 73

  22. GET _xpack/license { "license" : { "status" : "active", "uid"

    : "...", "type" : "basic", "issue_date" : "2017-04-18T00:00:00.000Z", "issue_date_in_millis" : 1492473600000, "expiry_date" : "2018-04-18T23:59:59.999Z", "expiry_date_in_millis" : 1524095999999, "max_nodes" : 100, "issued_to" : "Philipp Krenn (Elastic)", "issuer" : "Web Form", "start_date_in_millis" : 1492473600000 } } 75
  23. 77

  24. 78

  25. 79

  26. 80

  27. Container ship: https://flic.kr/p/hjxW62 https://flic.kr/p/2AzAVJ Wooden logs: https://flic.kr/p/9vvbKE Files: https://flic.kr/p/2EFcQ Metric:

    https://flic.kr/p/9g5h3f Packages: https://flic.kr/p/cJFDLN Windows: https://flic.kr/p/94Z6y Library: https://flic.kr/p/fiXcBj 84