Intro to ELK Stack for Chicago .Net Meetup

Dave Erickson 2015-05-20

Transcript

1. Intro  to  the  ELK  Stack   Log  Analy4cs  …  and  beyond   Dave  Erickson   dave@elas-c.co   May  20th,  2015  
2. According  to  the  Meet-­‐up  Agenda…   •  What  the  Meetup  Said   –  “Through  The  Lens  of  Github,  Mozilla,  Wikipedia”   –  “Key  Metrics  that  will  make  your  projects  more   successful”   –  Elas4csearch  &  .NET   •  I’m  Going  to  Go  Off-­‐Script   –  ELK  Stack   –  Live  Demo   –  Fun  Use  Cases!   –  Awesome  stuff  in  the  Roadmaps  (we  have  many)   –  A  Tale  of  Two  .NET  Clients  
3. It’s  not  E.L.K.,                  it’s  elk  
4. Why  is  geWng  value  out  of  logs  hard?   93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /favicon.ico  HTTP/1.1"  200  3638  "-­‐"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/ 20100101  Firefox/25.0"   93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /images/jordan-­‐80.png  HTTP/1.1"  200  6146  "hmp://www.semicomplete.com/ar4cles/ dynamic-­‐dns-­‐with-­‐dhcp/"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/20100101  Firefox/25.0"   93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /images/web/2009/banner.png  HTTP/1.1"  200  52315  "hmp://www.semicomplete.com/ style2.css"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/20100101  Firefox/25.0"   66.249.73.135  -­‐  -­‐  [16/Feb/2014:09:47:34  -­‐0500]  "GET  /blog/tags/ipv6  HTTP/1.1"  200  12251  "-­‐"  "Mozilla/5.0  (iPhone;  CPU  iPhone  OS  6_0  like  Mac   OS  X)  AppleWebKit/536.26  (KHTML,  like  Gecko)  Version/6.0  Mobile/10A5376e  Safari/8536.25  (compa4ble;  Googlebot/2.1;  +hmp:// www.google.com/bot.html)"   50.16.19.13  -­‐  -­‐  [16/Feb/2014:09:47:46  -­‐0500]  "GET  /blog/tags/puppet?flav=rss20  HTTP/1.1"  200  14872  "hmp://www.semicomplete.com/blog/ tags/puppet?flav=rss20"  "Tiny  Tiny  RSS/1.11  (hmp://m-­‐rss.org/)"   66.249.73.185  -­‐  -­‐  [16/Feb/2014:09:47:54  -­‐0500]  "GET  /  HTTP/1.1"  200  37932  "-­‐"  "Mozilla/5.0  (compa4ble;  Googlebot/2.1;  +hmp:// www.google.com/bot.html)"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:42  -­‐0500]  "GET  /projects/xdotool/  HTTP/1.1"  200  12292  "hmp://www.google.com/url? sa=t&rct=j&q=&esrc=s&source=web&cd=5&cad=rja&sqi=2&ved=0CFYQFjAE&url=hmp%3A%2F%2Fwww.semicomplete.com%2Fprojects %2Fxdotool%2F&ei=6cwAU_bRHo6urAeI0YD4Ag&usg=AFQjCNE3V_aCf3-­‐gfNcbS924S6jZ6FqffA&bvm=bv.61535280,d.bmk"  "Mozilla/5.0   (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   46.105.14.53  -­‐  -­‐  [16/Feb/2014:09:48:48  -­‐0500]  "GET  /blog/tags/puppet?flav=rss20  HTTP/1.1"  200  14872  "-­‐"  "UniversalFeedParser/4.2-­‐pre-­‐314-­‐ svn  +hmp://feedparser.org/"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /reset.css  HTTP/1.1"  200  1015  "hmp://www.semicomplete.com/projects/xdotool/"   "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /style2.css  HTTP/1.1"  200  4877  "hmp://www.semicomplete.com/projects/xdotool/"   "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /favicon.ico  HTTP/1.1"  200  3638  "-­‐"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)   Gecko/20100101  Firefox/28.0"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /images/jordan-­‐80.png  HTTP/1.1"  200  6146  "hmp://www.semicomplete.com/projects/ xdotool/"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   123.125.71.35  -­‐  -­‐  [16/Feb/2014:09:49:02  -­‐0500]  "GET  /blog/tags/release  HTTP/1.1"  200  40693  "-­‐"  "Mozilla/5.0  (compa4ble;  Baiduspider/2.0;   +hmp://www.baidu.com/search/spider.html)"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /images/web/2009/banner.png  HTTP/1.1"  200  52315  "hmp://www.semicomplete.com/ style2.css"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   Logs  are  have  no  standard  format   There  is  no  consistency   You  have  to  be  an  expert  to  read  them   Almost  always  stored  somewhere  that’s  hard  for   the  organiza4on  to  get  to.   grep  and  regular  expressions  don’t  scale   I  can  help!  
5. ELK  Stack   Elas4csearch,  Logstash,  and  Kibana   Logstash   Elas4csearch   Kibana   Key:  
6. Logstash   •  Input  from  many  Sources   –  It’s  really  good  at  parsing  logs  (shocking!)   –  Other  sources  too   •  Files,  Queues,  Messages,  Databases,  etc   –  Hundreds  of  plugins   •  Transform  and  Enrich   –  GROK   –  IP  -­‐-­‐>  Geospa4al   –  Conver4ng  to  JSON  is  very  popular   •  Output  to  Many  Des4na4ons   –  Databases,  Dashboards,  Elas4csearch  …   –  many  others  
7. Elas4csearch   •  Based  on  Lucene   –  Wicked  fast   –  Good  at  search   –  Good  at  analy4cs   •  Things  it  adds   –  Horizontal  Scaling   –  High  Availability   –  Ease  of  Use   –   “Near  Real-­‐Time”  
8. Kibana   •  UI  for  Elas4csearch   –  Lightweight  Search  Interface   –  Discover   –  Visualize   –  Dashboards   •  Explore  and  Understand  your   data   •  Scale  and  Speed   –  Heavy  li|ing  s4ll  done  in   Elas4csearch  
9. Let’s  take  an  example  
11. Regular  Expression  
12. Grok  PaJern  
18. {          4me:          “120819”,          query_4me:      27.115751,          lock_4me:        0.00007          rows_sent:        55996,          rows_examined:    56000          query_4mestamp:    “1345373510”,          query:          “SELECT  ID  FROM  wp_posts  WHERE  …”   }  
19. Logstash  Pipeline  
22. ELK  Stack   Elas4csearch,  Logstash,  and  Kibana   Logstash   Elas4csearch   Kibana   Key:  
23. What  usually  ends  up  happening  once   someone  makes  something  useful   Logstash   Elas4csearch   Kibana   Key:   ES-­‐Hadoop   Rela-onal   Hadoop   Non-­‐Rela-onal   Apps  and     Mobile   Language   Clients  
24. Scaled  Architectures   Logstash   Elas4csearch   Kibana   Key:   master   master   master   …   …   …   queue   queue   queue   …   Agents   (shippers)   route  /  buffer   collect   process   Index  /  alert   view  
25. Going  Beyond  Logs   •  Search   – Relevance   – Faceted  Naviga4on   – Human  Language   – Unstructured  Data   – Rich  Syntax   •  Analy4cs   –  Real  Time   –  Significant  Terms   –  En4ty  Oriented  Indexing   –  Make  data  available  to   the  whole  organiza4on   –  Use  ELK  to  figure  out   what  ques4ons  to  start   asking  
26. Use  cases   hmps://www.elas4c.co/elas4con/2015/sf/videos/  
27. Use  Case:  MozDef   •  Large  Scale  Security  Challenge   –  300  Million  events  per  day   •  Amackers   –  Innova4ve   –  Real  Time   –  Adap4ve   •  Exis4ng  SIEM  op4ons   –  Closed  Systems   –  Proprietary   –  Lack  of  API’s  &  Endpoints   •  Solu4on:  built  their  own   –  Mozilla  Defense  Pla€orm   –  Open  Source  SIEM  overlay  for   Elas4csearch  
28. Use  Case:  Wikimedia   •  265+  Languages   •  3.1  B  prefix  searches  /   month   •  870  M  text  searches  /  month   •  Real  Time   –  Rewards  Contributors   –  Fixes  Vandalism   •  Plugins   –  Contributed  many   •  Expressive  Syntax   –  Fix  search  without   redeployment  of  search   infrastructure   •  No  Down4me  Management   –  Aliases  
29. Use  Case:  Github  Search   A  Story  About  Learning  to  Scale  
30. Use  Case:  NASA  JPL  Telemetry  
31. Awesome  Stuff  in  the  road  map   •  Marvel  2.0  &  Shield  2.0   •  Watcher    (Aler4ng  for  Elas4csearch)   •  Logstash   –  Cleaner  Plugins   –  Bemer  Windows  Support   –  More  inputs  (Ka•a,  Avro  ...  etc.)   –  Buffering,  Clustering,  Resiliency   •  Elas4csearch   –  Aggrega4ons  2.0  (name  pending)   –  Changes  API,  Re-­‐indexing  API   •  Elas4csearch  Hadoop  Connector   –  Na4ve  support  for  Spark,  Storm,  Yarn   –  Exposing  ES  internals  to  Machine  Learning   •  Kibana   –  Rapidly  evolving  
32. Elas4csearch  &  .Net  
33. Two  Clients  
34. •  Low  Level:  Elas4csearch.NET   –  No  dependencies,  load  balancing,  failover   •  High  Level:  NEST  (v1.5  just  released)   –  Strongly  Typed  /  Fluent  API   –  Dependencies:  Json.NET,  Elas4csearch.NET   –  Plain  Old  C#  Objects  (POCOs)   –  Annota4on  driven  Mappings  (Don’t  repeat  yourself!)   –  Wraps  and  Exposes!   •  Directly  Input  Boolean  filters  as  DSL   •  Boolean  bitwise  operators  mapped  to  Elas4csearch  DSL  
36. Magic,  when  you  need  it  
37. Key  things  to  make  your  projects  more   successful?   •  Understand  Distributed  Systems  lingo  /  invarients   –  “Master”  =/=  “Primary”   –  Don’t  model  your  data  rela4onally   •  Read  the  book   –  HTML  version  of  O’Reilly  book  is  on  Elas4c.co  >  Learn   (Elas4csearch  –  The  Defini4ve  Guide)    …  it’s  really  good         •  It’s  not  a  silver  bullet   –  No  Such  Thing  As  a  Free  Lunch  
38. THANKS!    QUESTIONS?   dave@elas4c.co