Share
Embed
elasticsearchもうちょっと入門
「サーバ/インフラエンジニア養成読本 ログ収集〜可視化編」(
http://gihyo.jp/book/2014/978-4-7741-6983-5?ard=1410281447)
の出版記念イベント(http://eventdots.jp/event/137658)で使用したスライド。
Jun Ohtani
September 09, 2014
Transcript
46 slides
-
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Jun Ohtani @johtani elasticsearch.͏ͪΐͬͱೖ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Serverຊޠ൛ͷ༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel Professional services: Support & development subscriptions Trainings -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Elasticsearchʁ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ߜΓࠐΈ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϋΠϥΠτ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ιʔτ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϖʔδϯά -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ूܭ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited αδΣετ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮ɻ֦ு༰қ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Logstash -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Logstash in 10 seconds • Managing events and logs • Collect, parse, enrich, store data • Modular: many, many inputs and outputs • Apache License 2.0 • Ruby app (JRuby) • Part of Elasticsearch family -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Logstash architecture Logstash Input Output Filter ? ? collect and split alter and enrich store and visualize -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Inputs • Monitoring: collectd, graphite, ganglia, snmptrap, zenoss • Datastores: elasticsearch, redis, sqlite, s3 • Queues: rabbitmq, zeromq • Logging: eventlog, lumberjack, gelf, log4j, relp, syslog, varnish log • Platforms: drupal_dblog, gemfire, heroku, sqs, s3, twitter • Local: exec, generator, file, stdin, pipe, unix • Protocol: imap, irc, stomp, tcp, udp, websocket, wmi, xmpp -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Filters • alter, anonymize, checksum, csv, drop, multiline • dns, date, extractnumbers, geoip, i18n, kv, noop, ruby, range • json, urldecode, useragent • metrics, sleep • … many, many more … -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Outputs • Store: elasticsearch, gemfire, mongodb, redis, riak, rabbitmq • Monitoring: ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabbix • Notification: email, hipchat, irc, pagerduty, sns • Protocol: gelf, http, lumberjack, metriccatcher, stomp, tcp, udp, websocket, xmpp • External Monitoring: boundary, circonus, cloudwatch, datadog, librato • External service: google big query, google cloud storage, jira, loggly, riemann, s3, sqs, syslog, zeromq • Local: csv, exec, file, pipe, stdout, null -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Installation • Ruby application, but Java required (JRuby) • Download single tgz, deb, RPM (also repositories) No gem/dependency nightmares! • Puppet module -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Simple example • Download, create config and run input {! stdin {}! }! ! output {! stdout { debug => true }! } echo foo | bin/logstash -f simple.conf! {! "message" => "foo",! "@version" => "1",! "@timestamp" => "2014-01-20T13:30:59.648Z",! "host" => "kryptic.fritz.box"! } simple.conf -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited CLF log files {! "message" => "193.99.144.85 - - [23/Jan/2014:17:11:55 +0000] \"GET / HTTP/1.1\" 200 140 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/ 535.19\"",! "@version" => "1",! "@timestamp" => "2014-01-24T07:56:02.460Z",! "host" => "kryptic.local",! "clientip" => "193.99.144.85",! "ident" => "-",! "auth" => "-",! "timestamp" => "23/Jan/2014:17:11:55 +0000",! "verb" => "GET",! "request" => "/",! "httpversion" => "1.1",! "response" => "200",! "bytes" => "140",! "referrer" => "\"-\"",! "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/ 535.19\""! } -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Write to elasticsearch input { stdin {} }! ! filter {! grok {! match => [ message, "%{COMBINEDAPACHELOG}" ]! }! }! ! output {! elasticsearch_http {}! } -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Deploying ELK for scale Shipper Logstash Store/Search Visualize -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Add a broker Shipper Logstash Store/Search Visualize Broker Brokers help with scale and stability by buffering the input and protecting against output downtime. ! Tip: set limits on broker queue to push back on source as well. -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Scale out the shipper Shipper Logstash Store/Search Visualize Broker Shipper Shipper -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Scale out the broker Shipper Logstash Store/Search Visualize Broker Shipper Shipper Broker Broker -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Scale out Logstash Shipper Logstash Store/Search Visualize Broker Shipper Shipper Broker Broker Logstash Logstash -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Scale out Elasticsearch Shipper Logstash Store/Search Visualize Broker Shipper Shipper Broker Broker Logstash Logstash Store/Search -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited aggregation -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Aggregationͱ • 1.0͔Βར༻Մೳ • FacetΑΓڧྗͳूܭͳͲ͕Մೳ ֊తͳूܭɺάϧʔϓԽ ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ BucketɹυΩϡϝϯτͷ͋Δ͝ͱʹ݁ՌΛάϧʔϐϯά Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Aggregationͱ • SQLͩͱ ! ! • GROUP BY colorɿBucket • COUNT(color)ɿMetric SELECT COUNT(color) FROM table GROUP BY color -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Bucket • Bucketͷྫɿ • Ϣʔβͷੑผɿʮஉੑʯʮঁੑʯ • πΠʔτͷॴɿʮ౦ژʯʮژʯ • πΠʔτͷݴޠɿʮຊޠʯʮӳޠʯ • ϒϥβɿʮIEʯʮChromeʯ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited Metric • Metricͷྫɿ • BucketͷυΩϡϝϯτͷ • πΠʔτจࣈͷฏۉ • ച্ߴͷ࠷େ • ϦΫΤετॲཧ࣌ؒͷ࠷େɺ95ύʔηϯλΠϧ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited αϯϓϧ { "price": 10000, "color": "red", "make": "honda", ɹɹ"sold": "2014-10-28" } { "price": 20000, "color": "red", "make": "honda", ɹɹ"sold": "2014-11-05" } { "price": 30000, "color": "green", "make": "ford", ɹɹ"sold": "2014-05-18" } { "price": 15000, "color": "blue", "make": "toyota", ɹɹ"sold": "2014-07-02" } { "price": 12000, "color": "green", "make": "toyota", ɹɹ"sold": "2014-08-19" }... -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:৭Ͱूܭ » GET /cars/transactions/_search?search_type=count { "aggs" : { "popular_colors" : { "terms" : { "field" : "color" } } } } -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:৭Ͱूܭ "aggregations": { "popular_colors": { "buckets": [ { "key": "red", "doc_count": 4 },{ "key": "blue", "doc_count": 2 }, { "key": "green", "doc_count": 2 }… -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:৭Ͱྨͯ͠Ձ֨ͷฏۉ » GET /cars/transactions/_search?search_type=count { "aggs": { "colors": { "terms": { "field": "color" }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } } -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:৭Ͱྨͯ͠Ձ֨ͷฏۉ "aggregations": { "colors": { "buckets": [ { "key": "red", "doc_count": 4, "avg_price": { "value": 32500 } }, {"key": "blue", "doc_count": 2, "avg_price": { "value": 20000 } },,… -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:ϝʔΧʔΛՃ » GET /cars/transactions/_search?search_type=count { "aggs": { "colors": { "terms": { "field": “color" }, "aggs": { "avg_price": { "avg": { "field": “price" } }, "make": { "terms": { "field": "make" } } } } }} -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:ϝʔΧʔΛՃ "aggregations": { "colors": { "buckets": [ { "key": "red", "doc_count": 4, "make": { "buckets": [ { "key": "honda", "doc_count": 3 }, { "key": "bmw", "doc_count": 1 } ] }, "avg_price": { "value": 32500 }… -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:ϝʔΧʔ͝ͱͷ࠷খ/࠷େՁ֨ » GET /cars/transactions/_search?search_type=count { "aggs": { "colors": { "terms": { "field": “color” }, "aggs": { "avg_price": { "avg": { "field": "price" } }, "make" : { "terms" : { "field" : “make" }, "aggs" : { "min_price" : { "min": { "field": "price"} }, "max_price" : { "max": { "field": "price"} } } } } } } } -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ྫ:ϝʔΧʔ͝ͱͷ࠷খ/࠷େՁ֨ { "key": "red", "doc_count": 4, "make": { "buckets": [ { "key": "honda", "doc_count": 3, "min_price": { "value": 10000 }, "max_price": { "value": 20000 } }, { "key": “bmw”, … } ] }, "avg_price": { "value": 32500 }… -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited bucket • filter • filters (>=1.4.0) • missing • nested • reverse nested (>=1.2.0) • children • terms • significant terms (>=1.1.0) • range • date range • ipv4 range • histogram • date histogram • geo distance • geohash grid -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited metric • min • max • sum • avg • stats • extended_stats • value count • percentiles (>=1.1.0) • percentile ranks (>=1.3) • cardinality (>=1.1.0) • geo bounds (>=1.3.0) • top_hits (>=1.3.0) • scripted metric (>= 1.4.0) -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ·ͱΊ • ೦ͳ͕ΒKibana3Ͱར༻Ͱ͖ͳ͍ • REST APIΛར༻͢ΕूܭՄೳ • ଟஈͰूܭՄೳ • ϝϞϦͷফඅྔʹҙ -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited elasticsearchษڧձ • ୈ6ճelasticsearchษڧձ ɹɹ#elasticsearch #elasticsearchjp • ࣌ɿ2014/09/16ɹ19:30ʙ • ॴɿϦΫϧʔτςΫϊϩδʔζ41Fʢ౦ژӺʣ • URLɿhttp://elasticsearch.doorkeeper.jp/events/ 13917 -
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission
is strictly prohibited ࢀߟ • Github https://github.com/elasticsearch • ΨΠυ http://www.elasticsearch.org/guide/ • αϙʔτ http://www.elasticsearch.com/support/