Getting Started with Elastic Cloud and Beats for Log Analytics

情報セキュリティワークショップ in 越後湯沢 2016

Transcript

1. ‹#› Kosho Owa, Solutions Architect, Elastic October 2016 Elastic CloudͱBeatsͰ࢝ΊΔ ϩάͷՄࢹԽͱ෼ੳ
2. 2 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store, Index,
 & Analyze Ingest Logstash Beats + Elastic Stack Introducing the Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph
3. Elasticserach: σʔλετΞɺΠϯσοΫεɺ෼ੳ 3 ෼ࢄܕͰ
 εέʔϥϒϧ ճ෮ੑ͕͋ΓߴՄ༻ੑɺεέʔϧΞ΢τΛલఏ ͱͨ͠੡඼σβΠϯ
   ߏ଄ɺඇߏ଄σʔλΛΠϯσοΫε ։ൃऀ
 ϑϨϯυϦʔ εΩʔϚϨε
   Ϛϧνςφϯτ
   ๛෋ͳΫϥΠΞϯτϥΠϒϥϦ ݕࡧͱ෼ੳ ϦΞϧλΠϜ
   શจݕࡧ
   (FP
   "HHSFHBUJPO
   ଟݴޠʹରԠ
4. Kibana: ՄࢹԽͱ୳ࡧ 4 ൃݟͱಎ࡯ σʔλΛ୳ࡧɺنଇੑΛൃݟ
   ͲͷΑ͏ͳϨϕϧ ΁΋υϦϧμ΢ϯ
   &MBTUJDTFBSDIͷύϫϑϧͳ෼ੳػೳΛར༻
   ߏ଄ɺඇߏ଄σʔλ ΧελϚΠζ ͦͯ͠ڞ༗ όʔνϟʔτɺંΕઢάϥϑɺ෼෍ਤɺ஍ਤɺ ώετάϥϜ
   μογϡϘʔυΛΛγΣΞ͠ɺӡ༻ϫʔΫϑϩ ʔʹ૊ΈࠐΈ Elastic Stack ΁ͷೖΓޱ ՄࢹԽͷͨΊͷ౷Ұతͳ6*
   &MBTUJD
   4UBDL
   ͷӡ༻؅ཧ
   ϓϥάΠϯՄೳͳΞʔΩςΫνϟͰɺಠࣗͷΞ ϓϦέʔγϣϯ͕࡞੒Մೳ
5. Beats: ElasticsearchͷͨΊͷσʔλγούʔ 5 Filebeat ϩάऩूΤʔδΣϯτͷ࣍ੈ୅ελϯμʔυ Winlogbeat 8JOEPXT
   γεςϜɺΞϓϦέʔγϣϯɺηΩ ϡϦςΟϩά Metricbeat "QBDIF
   .POHP%#
   .Z42-
   /HJOY
   3FEJT
   ;PPLFFQFSɺ04ͳͲͷϝτϦοΫ Packetbeat )551
   .Z42-
   $BTTBOESB
   %/4ͳͲͷωοτ ϫʔΫύέοτΛϦΞϧλΠϜͰղੳ Libbeat ΧελϜzCFBUTz։ൃ༻ϥΠϒϥϦʔ
6. ੒ޭ͢Δϩά෼ੳ σʔλऩू BeatsͰσʔλऩूͱElasticsearch΁ͷ౤ೖ μογϡϘʔυͷςϯϓϨʔτΛಉࠝ JSONߏ଄ԽϩάΛFilebeatͰऩूɺΠϯσοΫε ΠϯετʔϧͱηοτΞοϓ Elastic CloudͰΫϥελʔΛ਺ΫϦοΫͰల։ ৗʹ࠷৽൛ɺΞοϓάϨʔυ΋؆୯ʹ ӡ༻ X-PackΛ׆༻ͯ͠ɺಛఆͷΠϕϯτʹରͯ͠Ξϥʔτ σʔλͷΞΫηε੍ޚ ElasticsearchΫϥελʔࣗ਎΋ϞχλϦϯά 6
7. 7 Performance Metrics Application Logs Filebeat ϩάऩू Packetbeat ύέοτ؂ࢹ Elasticsearch σʔλετΞ ݕࡧΤϯδϯ Kibana ՄࢹԽ Network Interfaces Metricbeat ϝτϦοΫऩू
8. JSONߏ଄ԽϩΪϯά - Apache 8 LogFormat "{ \"clientip\": \"%h\", \"ident\": \"%l\", \"auth\": \"%u\", \"timestamp\": \"%{%FT%T%z}t\", \"verb\": \"%m\", \"request\": \"%U%q\", \"httpversion\": \"%H\", \"response\": %>s, \"bytes\": %b, \"referer\": \"% {Referer}i\", \"agent\": \"%{User-agent}i\" }" combinedjson CustomLog logs/access_log.js combinedjson - input_type: log paths: - /var/log/httpd/access_log.js document_type: apache json.keys_under_root: true json.add_error_key: true httpd.conf filebeat.yml
9. JSONߏ଄ԽϩΪϯά - Squid 9 logformat combinedjson { "clientip": "%>a", "ident": "%ui", "uname": "%un", "timestamp": "%{%FT%T%z}tg", "verb": "%rm", "request": "%ru", "httpversion": "HTTP/%rv", "response": %>Hs, "bytes": %<st, "referer": "%{Referer}>h", "agent": "%{User-Agent}>h", "request_status": "%Sh", "hierarchy_status": "%Sh" } access_log /var/log/squid/access_log.js combinedjson - input_type: log paths: - /var/log/squid/access_log.js document_type: squid json.keys_under_root: true json.add_error_key: true squid.conf filebeat.yml
10. Metricbeat - OSɺΞϓϦέʔγϣϯͷϝτϦοΫऩू 10 ϦΞϧλΠϜϞχλϦϯά • OS΍αʔϏεͷϝτϦοΫΛϞχλʔ αʔϏεͷύϑΥʔϚϯε෼ੳ • System: CPU, load, IO, filesystem, memory, network, process • Apache, HAProxy, MongoDB, MySQL, Nginx, Redis, ZookeeperʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ
11. Packetbeat - ωοτϫʔΫύέοοτͷղੳͱऩू 11 ϦΞϧλΠϜϞχλϦϯά • ΞϓϦέʔγϣϯͷ஗ԆɺΤϥʔɺԠ ౴࣌ؒͳͲΛϞχλʔ ωοτϫʔΫτϥϑΟοΫͷݕࡧͱ෼ੳ • ICMP, DNS, HTTP, AMQP, Cassandra, MySQL, PostgreSQL, Redis, Thrift- RPC, MongoDB, MemcacheʹରԠ ElasticsearchʹసૹɺKibanaͰՄࢹԽ • αϯϓϧDashboardͰ͙͢ʹར༻Մೳ
12. Hosted Elasticsearch & Kibana on AWS • Elasticͷ੡඼܈ͱಉظͨ͠࠷৽൛ͷఏڙ • εέʔϧΞ΢τɾΞοϓάϨʔυΛΫϦοΫૢ࡞ Ͱ • ແྉͷKibanaΠϯελϯεͱ30෼͝ͱͷόοΫΞ οϓ • X-Packػೳ (Security, Alerting, Monitoring, Reporting) • ݄ʑ45USD͔Β • SLAϕʔεͷαϙʔτΦϓγϣϯ 12 Elastic Stackͷ։ൃऀʹΑΔ།Ұͷ ެࣜ Elasticsearch as a Service
13. X-Pack: Elastic Stackͷ෇ՃՁ஋ػೳ 13 \ ηΩϡϦςΟ෼ੳ ϩά෼ੳ ϝτϦοΫε ෼ੳ ӡ༻෼ੳ υΩϡϝϯτݕࡧ ΞϓϦέʔγϣϯ ݕࡧ ϩοΫμ΢ϯͱ ΞΫηε؂ࢹ σʔλͷมߋʹ
 ର͢Δ௨஌ Elasticsearch
 Ϋϥελͷ؂ࢹ σʔλ͔Βҙຯͷ
 ͋Δؔ܎Λൃݟ PDFΛ࡞੒ͯ͠ ൃݟΛγΣΞ Security Alerting Monitoring Graph Analytics Reporting
14. X-Pack: Security - ҉߸ԽͱϩʔϧϕʔεͷΞΫηε੍ޚ 14 ҉߸Խ • KibanaɺElasticsearchͷΤϯυϙΠ ϯτ΁ͷHTTPS௨৴ • Ϋϥελʔ಺ͷ௨৴ ΞΫηε੍ޚ • ID/PasswordʹΑΔϢʔβೝূ • ωΠςΟϒɺLDAPɺAD࿈ܞ • KibanaͷϩάΠϯμΠΞϩά • ϩʔϧ͝ͱʹΠϯσοΫεɺAPI΁ͷ ΞΫηεΛ੍ݶ
15. X-Pack: Alerting - σʔλͷมԽΛ௨஌ 15 εέδϡʔϧ • ಛఆͷ࣌ؒɺΠϯλʔόϧɺ Crontabॻࣜ ίϯσΟγϣϯ • Elasticsearchͷ͢΂ͯͷΫΤϦʔͱ ΞάϦήʔγϣϯΛαϙʔτ • ෳ਺ͷιʔεΛ૊Έ߹Θͤ ΞΫγϣϯ • ΠϯσοΫεɺϩάɺϝʔϧɺ΢Σ ϒϑοΫͳͲ
16. Monitoring - ΫϥελʔɺϊʔυɺΠϯσοΫεͷ؂ࢹ • ElasticsearchΫϥελʔɺϊʔυɺ ΠϯσοΫεͷϝτϦοΫΛϦΞϧ λΠϜͰ؂ࢹ • ӡ༻্ͷ܏޲Λ೺Ѳɺ໰୊Λൃݟ • ΫϥελʔɺΞϓϦέʔγϣϯͷ࠷ దԽ • ΩϟύγςΟϓϥχϯά 16
17. X-Pack: Graph - σʔλؒͷؔ܎ΛՄࢹԽ 17 • Elasticsearchͷsearch΍relevancyͷػ ೳΛ࢖༻ͯ͠ҙຯͷ͋Δؔ܎Λൃݟ • طଘͷΠϯσοΫεΛར༻ • ϦΞϧλΠϜ͔ͭεέʔϥϒϧ
18. X-Pack: Reporting - DashboardΛΤΫεϙʔτ 18 Earthquake - Depth Timeseries Earthquake - Heatmap Earthquake — Sun, Jan 1, 2006 12:00 AM to Fri, Sep 2, 2016 5:54 AM • PDF΋͘͠͸CSVΛੜ੒ • ඇKibanaϢʔβͱڞ༗ • खಈɺ΋͘͠͸Alertingͱͷ૊Έ߹Θ ͤͰεέδϡʔϧɺ΋͘͠͸ಛఆͷΠ ϕϯτ͕ൃੜͨ͠৔߹ʹ࡞੒ N ew in V5
19. elastic.co/jp: ೔ຊޠ৘ใ΋͝ར༻Լ͍͞ 19 • ੡඼৘ใ • αϒεΫϦϓγϣϯ • ಋೖࣄྫ • ύʔτφʔ • ϋϯζΦϯϫʔΫγϣοϓ • ϒϩά • νϡʔτϦΞϧϏσΦ • ͓໰͍߹Θͤ