AWSのマネージドサービスを活用したログ可視化

2017/02/11 合同勉強会 in 福岡

Transcript

1. AWS ͷ αʔϏεΛ׆༻ͨ͠ ϩάՄࢹԽ ߹ಉษڧձ in ෱Ԭ 2017/02/11 ౻ຊ ਅ࢘
2. ࣗݾ঺հ • ౻ຊ ਅ࢘ • Work
 AWS Solutions Architect
 @Classmethod, Inc. • Interest
 Elastic Stack
3. ձࣾ঺հ
4. ձࣾ঺հ "84
   ΍
   ϞόΠϧɺϏοάσʔλͳͲ
   ࣾһ૯ग़Ͱ޷͖ͳΑ͏ʹϒϩάॻ͍͍ͯ·͢
5. AWSࣄۀ෦঺հ
6. AWSࣄۀ෦঺հ ओʹ
   "84
   ʹؔ͢Δϒϩάॻ͍͍ͯ·͢
7. ຊ೔఻͍͑ͨ͜ͱ
8. ϩάͷՄࢹԽ͕ ؆୯ʹͳΓ·ͨ͠
9. ΞδΣϯμ • ϩάՄࢹԽͬͯʁ • AWSͷαʔϏεΛ׆༻ͨ͠ϩάՄࢹԽ • Demo
10. ϩάՄࢹԽͬͯʁ
11. Έͳ͞Μ ϩάΛ׆༻͍ͯ͠·͔͢ʁ
12. ϩάͬͯԿ͚ͩͬʁ • ޠݯ • ߤւͷધͷ଎౓ΛଌΔͨΊʹ
 ؙଠ(log)ʹҰఆִؒͷ͋ΔೄΛׅΓ෇͚ ͨ΋ͷ
 ʢग़యɿWikipediaʣ
13. ԿͰϩά͕ඞཁʁ • ؂ࢹɺΞϥʔςΟϯά • ো֐ௐࠪ • ύϑΥʔϚϯεղੳ • σʔλ෼ੳ
14. ԿͰϩάͷՄࢹԽ͕ඞཁʁ • มԽͷؾ͖ͮΛಘΔ • ো֐ͷ༧ஹɺधཁͷ༧ଌ • આ໌ͷઆಘྗ • จࣈ਺ࣈΑΓ΋Ұ໨ྎવ
15. ϩάՄࢹԽͷ՝୊
16. ϩάՄࢹԽͷ՝୊ • δϣϒεέδϡʔϦϯά • ETL • ՄࢹԽπʔϧͷબఆ
17. ϩάՄࢹԽͷ՝୊ • δϣϒεέδϡʔϥ • ϦΞϧλΠϜͳͷ͔ɺόονͳͷ͔ • ੒ޭɺࣦഊͷ൑ఆ • ϦτϥΠॲཧ
18. ϩάՄࢹԽͷ՝୊ • Extract • Ͳ͔͜ΒϩάΛऔಘ͢Δ͔ • Transform • ՄࢹԽ͍ͨ͠৘ใɺܗࣜ΁ͷม׵ • Load • ՄࢹԽπʔϧ΁ͷσʔλ౤ೖ
19. ϩάՄࢹԽͷ՝୊ • ՄࢹԽπʔϧͷબఆ • Kibana • Grafana • re:Dash • Superset • Google Data Studio • etc.. etc..
20. AWS ʹ͓͚ΔϩάՄࢹԽ
21. Amazon Elasticsearch Service
22. Amazon Elasticsearch Service • Elasticsearch • ݕࡧʗ෼ੳΤϯδϯ • Kibana • ՄࢹԽɺμογϡϘʔυԽ • ४ϦΞϧλΠϜʹՄࢹԽ
23. Amazon Elasticsearch Service • Elasticsearch + Kibana ͷϚωʔδυαʔϏε • ਺ΫϦοΫͰ؀ڥߏங • ӡ༻͸ AWS ʹ͓೚ͤ • ೔࣍εφοϓγϣοτͷऔಘ • ো֐࣌ͷ෮چରԠ
24. Amazon Elasticsearch Service Lambda S3 EC2 Elasticsearch Service
25. Amazon Elasticsearch Service • 2017/01/31 Ξοϓσʔτ • Elasticsearch / Kibana 5 ܥʹରԠ • εΫϦϓτͷར༻Մ • Ingest Node • Kibana ͷ DevTools ΍ Timelion Λར༻Մ
26. Amazon Athena & Amazon QuickSight
27. ͜ͷαʔϏε஌͍ͬͯ·͔͢ʁ Amazon Athena Amazon QuickSight
28. Amazon Athena • AWS re:Invent 2016 ൃදˍϦϦʔε • S3 ͷඇߏ଄Խσʔλʹ SQL Ͱߏ଄Խσʔλ ΛऔΓग़͢
29. Amazon Athena • αʔόʔϨε • ӡ༻ෆཁɻ࢖͍͍ͨ࣌ʹ࢖͑Δ • ߴ଎ͳύϑΥʔϚϯε • 1000ສ݅ͷςΩετͷूܭ͕਺ඵ • ௿ίετ • εΩϟϯର৅αΠζ՝ۚɹ5$ / TB
30. Amazon QuickSight • AWS re:Invent 2015 ൃදɻ2016/11ϦϦʔε • AWS ۘ੡ BI πʔϧ
31. Amazon QuickSight • αʔόʔϨε • SPICE ʹΑΔߴ଎ͳՄࢹԽ • ଟ͘ͷ AWS αʔϏεʹରԠ • ௿ίετͳ BI πʔϧ
32. Amazon Athena & QuickSight
33. ࢖͍ॲ ʢ˞ϩάՄࢹԽͷ؍఺ʣ
34. Ϣʔεέʔε • Elasticsearch Service • χΞϦΞϧλΠϜͳଟ࣍ݩ෼ੳ • Athena & QuickSight • ΞυϗοΫͳ෼ੳ
35. ར༻ίετ • Elasticsearch Service • ElasticsearchΫϥελͷىಈ࣌ؒ • ྫ͑͹ɺr3.large 50GBσΟεΫ x 3୆ ౦ژϦʔδϣϯͷ৔߹ɺ$659 / ݄
36. ར༻ίετ • Athena & QuickSight • Athena ͸ΫΤϦͨ͠ϑΝΠϧαΠζ • QuickSight ͸Ϣʔβʔˍ௥ՃσΟεΫ • ྫ͑͹ɺ5Ϣʔβʔ 10TB εΩϟϯ
 $95 / ݄
37. ͦͷଞίετ • Elasticsearch Service • Elasticsearch ΁ϩάΛೖΕΔͨΊͷ࢓૊Έ ͕ඞཁ • Kibana ͸ॳݟͰ৮Δʹ͸ਏ͍ • Athena & QuickSight • Athena ͸ SQL ͕৮ΕΕ͹࢖͑ͨ • QuickSight ͸ॳݟͰ৮Εͨ
38. Demo
39. ·ͱΊ • ϩάͬͯ஍ຯ͚ͩͲେࣄ • ϩάՄࢹԽͷબ୒ࢶ͕௥Ճ͞Εͨ • ༻్ʹ͋ͬͨՄࢹԽΛબ୒ • ՄࢹԽָ͍ͬͯ͠