Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Visualizing Your E-mail with Elastic Stack
Search
Kosho Owa
April 20, 2016
Technology
2
320
Visualizing Your E-mail with Elastic Stack
警視庁の犯罪・防犯情報提供サービス「メールけいしちょう」で受信したメッセージを Elasticsearch でインデックスし、Kibana で可視化する方法を紹介します。
Kosho Owa
April 20, 2016
Tweet
Share
More Decks by Kosho Owa
See All by Kosho Owa
Introducing Machine Learning for the Elastic Stack
kosho
2
12k
Elastic Stack X-Pack 5.0 for IT Security Workshop
kosho
1
320
Elastic Stack X-Pack 5.0 for IT Ops Workshop
kosho
0
330
[Developers Summit 2017] Anomaly Detection with the Elastic Stack
kosho
1
710
Anomaly Detection with the Elastic Stack
kosho
1
1.8k
Getting Started with Elastic Cloud and Beats for Log Analytics
kosho
0
100
Elastic{ON} Seminar Tokyo 2016 Product Update
kosho
0
170
Introducing Elastic Cloud
kosho
0
76
Gearing Up for Elastic Stack, X-Pack 5.0 Releases
kosho
0
150
Other Decks in Technology
See All in Technology
初めてのDatabricks Apps開発
taka_aki
1
250
今この時代に技術とどう向き合うべきか
gree_tech
PRO
2
2.1k
Contract One Engineering Unit 紹介資料
sansan33
PRO
0
9k
「魔法少女まどか☆マギカ Magia Exedra」のIPのキャラクターを描くための3Dルック開発
gree_tech
PRO
0
160
NLPコロキウム20251022_超効率化への挑戦: LLM 1bit量子化のロードマップ
yumaichikawa
1
210
OCIjp_Oracle AI World_Recap
shinpy
1
160
AI時代、“平均値”ではいられない
uhyo
8
2.3k
[VPoE Global Summit] サービスレベル目標による信頼性への投資最適化
satos
0
220
ローカルLLMとLINE Botの組み合わせ その2(EVO-X2でgpt-oss-120bを利用) / LINE DC Generative AI Meetup #7
you
PRO
0
150
フレームワークを意識させないワークショップづくり
keigosuda
0
240
事業開発におけるDify活用事例
kentarofujii
5
1.3k
「魔法少女まどか☆マギカ Magia Exedra」におけるバックエンドの技術選定
gree_tech
PRO
0
110
Featured
See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.6k
Rebuilding a faster, lazier Slack
samanthasiow
84
9.2k
Faster Mobile Websites
deanohume
310
31k
What’s in a name? Adding method to the madness
productmarketing
PRO
24
3.7k
Visualization
eitanlees
149
16k
Designing Experiences People Love
moore
142
24k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
31
9.7k
Become a Pro
speakerdeck
PRO
29
5.6k
Scaling GitHub
holman
463
140k
Transcript
‹#› Kosho Owa, Solutions Architect, Elastic April 20th, 2016 Visualizing
Your E-mail ʮϝʔϧ͚͍ͪ͠ΐ͏ʯΛՄࢹԽ͢Δ
ରσʔλ • ܯࢹிͷϝʔϧ͚͍ͪ͠ΐ͏(ొແྉ) http://www.keishicho.metro.tokyo.jp/about_mpd/joho/mail_info.html • ʮ൜ࡑൃੜใʯʮ൜ใʯΛϝʔϧ৴ • CC BY 2.1
JP Ͱఏڙ 2 Subject: ۄܯॺ(ࢠͲʢߦʣ) Body: 4݄16ʢʣɺޕޙ4࣌40͜Ζɺੈా୩۠Ԟ̍ஸͷ࿏্Ͱɺࣇಐ͕௨ߦதɺஉʹಥ ͖ඈ͞Ε·ͨ͠ɻʢ൜ਓʢஉʣͷಛʹ͍ͭͯɺ̑̌ࡀɺ170cm Ґɺதɺޱͻ ͛ɺ৭ͬΆ্͍ҥɺࠇ৭ͬΆ͍ζϘϯʣ ʲ߹ͤઌʳۄܯॺ 03-3705-0110ʢઢ2612ʣ
ํ • ϝʔϧΛIMAPͰऔಘ • ϑΟʔϧυΛߏԽ͢Δ • λΠϓΛߟྀͯ͠ΠϯσοΫε • analyzed, not_analyzedϑΟʔϧυͦΕͧΕΛ༻ͯ͠ՄࢹԽ͢Δ
3
Logstash Pipeline and Plugins ϓϥάΠϯՄೳͳΞʔΩςΫνϟʔͱɺ։ൃऀʹ༏͍͠ΤίγεςϜ 4 input {} filter {}
output {} beats, file, graphite, http, imap, kafka, rss, redis, stdin, sqlite, s3, syslog, zenoss and etc. csv, cloudwatch, email, elasticsearch, exec, file, graphite, http, kafka, mongodb, nagios, redis, s3, syslog, stdout, zabbix and etc.
Input Plugin - imap 5 input { imap { host
=> "imap.gmail.com" port => 993 user => "_IMAP_USER_" password => "_IMAP_PASSWORD_" folder => "_IMAP_FOLDER_" type => "_TYPE_" check_interval => 300 codec => plain { charset => "ISO-2022-JP" } } } • ϝʔϧຊจͷΤϯίʔυΛcodecͰࢦఆ͢Δ • ͋Β͔͡ΊIMAPͷfolderΛ͚ • ෳͷλΠϓϝʔϧΛॲཧ͢Δ߹ʹλά(tags)ΛՃ͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-inputs-imap.html • : ίϛϡχςΟϓϥάΠϯ
Filter Plugin • ϝʔϧͷຊจ͔Βൈ͖ग़͢ϑΟʔϧυ: city, area, place • λΠτϧ͔Βൈ͖ग़͢ϑΟʔϧυ: police_station,
incident • λΠϜελϯϓͱͯ͠࠾༻: datetime 6 filter { grok { match => { "message" => "%{DATA:[@metadata][datetime]}͜Ζɺ%{NOTSPACE:city}(۠|ࢢ)% {NOTSPACE:area}(ͷ|ۙ)(%{NOTSPACE:place}|)Ͱɺ%{GREEDYDATA}" } } date { match => ["[@metadata][datetime]", "M݄dʢEʣɺaK࣌m"] locale => ja timezone => "Asia/Tokyo" } grok { match => { "subject" => "%{NOTSPACE:police_station}ܯॺ\(%{NOTSPACE:incident}\)" } } }
ೖྗσʔλ grok ग़ྗ Filter Plugin - grok • ύλʔϯʹϚονͨ͠จࣈྻΛϑΟʔϧυʹؔ࿈͚ɺඇߏσʔλΛߏԽ͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
7 “subject” => “ۄܯॺ(ࢠͲʢߦʣ)” grok { match => { "subject" => "%{NOTSPACE:police_station}ܯॺ\(%{NOTSPACE:incident}\)" } } “police_station” => "ۄ" “incident" => "(ࢠͲʢߦʣ)"
ೖྗσʔλ date ग़ྗ Filter Plugin - date ϑΟʔϧυΛύʔε͠ɺLogstashͷΠϕϯτͱͯ͠༻ 8 "datetime"
=> “4݄16ʢʣɺޕલ7࣌40” "@timestamp" => "2016-04-16T07:40:00.000Z" • ͷऔಘʹࣦഊͨ͠߹ʹɺॲཧ͕࣌@timestampͱͯ͠࠾༻͞ΕΔ (tag_on_failure => true ݕ౼) https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html date { match => ["[@metadata][datetime]", "M݄dʢEʣɺaK࣌m"] locale => ja timezone => "Asia/Tokyo" }
Output Plugin - elasticsearch 9 output { stdout { codec
=> dots } elasticsearch { hosts => ["http://127.0.0.1:9200/"] index => "mail-%{+YYYY.MM}" } } • stdout { codec => dots } ͰɺҰ݅ॲཧ͝ͱʹυοτΛग़ྗ͢Δ • ΠϯσοΫε͕దͳαΠζʹͳΔΑ͏ɺΠϯσοΫε໊Λݕ౼͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
Logstash Tips • ग़ྗ࣌ʹύΠϓϥΠϯΛදࣔ • ϫʔΧʔΛదʹઃఆ͢Δ • ҟͳΔछྨͷσʔλɺLogstashͷೖྗલʹ͚͓ͯ͘ • grok
ϔϧύʔπʔϧΛ͏ http://grokdebug.herokuapp.com http://grokconstructor.appspot.com 10 output { stdout { codec => rubydebug } } $ logstash -w [NUMBER OF WORKERS] -f [PATH TO CONFIG]
Elasticsearch - Mapping • text (analyzed strings), keyword(not_analyzed strings)ϑΟʔϧυ5.0͔Βಋೖ •
textϑΟʔϧυͷanalyzerʹkuromojiΛࢦఆ͢Δ • terms aggregationΛߦ͏ͨΊʹɺmulti-fieldػೳΛͬͯkeywordϑΟʔϧυΛࢦఆ͢Δ 11 PUT /_template/mail-1 { "template": "mail-*", "mappings": { "_default_": { "properties": { "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "analyzer": "kuromoji" },... }}}}
Kibana - Visualize “Terms Aggregation” keywordϑΟʔϧυͰaggregation͢Δ 12
Kibana - Visualize “Filters Aggregation” analyzedϑΟʔϧυͰaggregation͢Δ 13
ؔ࿈ใ 14