Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6
Search
tatsuru
PRO
August 31, 2016
7
12k
はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6
Hatena Engineer Seminar #6 インフラ編での発表スライドです
tatsuru
PRO
August 31, 2016
Tweet
Share
More Decks by tatsuru
See All by tatsuru
プロダクトオーナーとしてSLOに向き合う 〜Mackerelチームの事例〜 / SRE NEXT 2023
tatsuru
PRO
0
1.8k
Mackerelのプロダクト開発 - エンジニア中心の開発プロセスで大切にしていること
tatsuru
PRO
0
4.2k
Mackerel の EventBridge 対応開発秘話
tatsuru
PRO
1
120
技術が実現するイノベーションとWebサービス運用の未来 / Innovation from&for Web Operations
tatsuru
PRO
0
1.3k
成長するためのエンジニア組織 / Hatena Engineering Group 2018
tatsuru
PRO
1
91
Mesosを使ったImmutable Infra 管理システムを作ってみた
tatsuru
PRO
8
7.7k
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
64
11k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
41
6.4k
Intergalactic Javascript Robots from Outer Space
tanoku
268
26k
Large-scale JavaScript Application Architecture
addyosmani
508
110k
Designing on Purpose - Digital PM Summit 2013
jponch
113
6.8k
Music & Morning Musume
bryan
46
6k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
22
1.7k
Embracing the Ebb and Flow
colly
83
4.4k
Making Projects Easy
brettharned
113
5.8k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.4k
How STYLIGHT went responsive
nonsquared
93
5.1k
A better future with KSS
kneath
235
17k
Transcript
ͯͳͷϩάӡ༻ ͜Ε·Ͱͱ͜Ε͔Β id:wtatsuru 2016/08/31 Hatena Engineer Seminar #6
ࣗݾհ • Tatsuru Watabane (id:wtatsuru) • ͯͳγεςϜϓϥοτϑΥʔϜ෦ • 2011ೖࣾ •
2014·ͰͷαʔϏε͍͍ͩͨશ෦ӡ༻
ຊͷ༰ • લఏͱͳΔγεςϜߏ • ͯͳͷϩάͷߏͱӡ༻Λհ • ࠓޙ
લఏͱͳΔγεςϜߏ
3ߏ reverse proxy App server DB
ѻ͏ϩά reverse proxy App server DB access log error log
various logs
ѻ͏ϩά • αʔϏεͷΞΫηεϩά • ˏϦόʔεϓϩΩγ • ΞϓϦέʔγϣϯ͕ు͘ϩά • Τϥʔϩάɺߦಈϩά etc.
reverse proxy App server DB access log error log various logs
લఏɿଟ͘ͷαʔϏε • ৽αʔϏε͔Β10Ҏ্ଓ͘αʔϏε·Ͱଟ༷ • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ
ͯͳͷϩάӡ༻
reverse proxy App server DB access log error log structured
log syslog server storage server SSD server rsyslog aggregator
ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ • ΞϓϦέʔγϣϯϩάͷӡ༻ • ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞΫηεϩάग़ྗ • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ • ֤छௐࠪɺૢ࡞ϩάɺetc. • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ • LTSV (Labeled
Tab-separated Values) http://ltsv.org/ host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700] req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http:// www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)
reverse proxy App server DB access log ΞΫηεϩάग़ྗ
ϩάΛอଘ͍ͨ͠ • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ • rsyslog Ͱඈͯ͠ετϨʔδαʔόʹอଘ • ϩʔΧϧ͔ΒS3ʹอଘͯ͠ೋॏԽ • ͠Β͘͢ΔͱGlacierʹΉ
reverse proxy App server DB access log syslog server rsyslog
storage server
؆୯ʹੳ͍ͨ͠ • SSDͷαʔόʹඈ͢ • SSD͕҆͘ͳͬͨͷͰಋೖ • ࡶଟʹ grep ੳ •
ΞΫηεϩά͔ΒϨΠςϯγੳ • 2ॻ͍ͨΒSSD͕յΕͨͷͰަ
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͬͱ؆୯ʹੳ͍ͨ͠ • S3 ͔Β Amazon EMR (ElasticMapReduce) • LTSVΛಡΈࠐΉσγϦΞϥΠβ
github.com/hatena/KeyValuePairsDeserializer • SQLͰ؆୯ʹੳ select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͬͱ؆୯ʹੳ͍ͨ͠ • ֤छ՝ • ॳظԽ͕͍ • ڥߏங͕͍͠ • ఆظతʹճ͢ͱίετ͕ •
ΘΕͳ͘ͳ͍ͬͯ͘…
ͬͱ؆୯ʹੳ͍ͨ͠ • EMRஔ͖͑ɿGoogle BigQuery Λݕূ • LTSV embulk Ͱલॲཧͯ͠ಡΈࠐΉ
• ͍ɾ·͋·͍͋҆ • ։ൃऀҎ֎Ͱ͍͍͢
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β • ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞϓϦέʔγϣϯͷϩά • ૉͳੲ • /var/log/httpd/access.log • /var/log/httpd/error.log • ֤छͪ͝Ό·ͥ •
Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍ • DBʹೖΕΔ
reverse proxy App server DB
ߏԽϩάΛग़͢ • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘ • ඪ४ग़ྗɾΤϥʔग़ྗͱ͚Δ •
ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘ • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ
reverse proxy App server DB structured log
fluentdͰૹ • ֤ϗετͰ in_tail +out_forward Ͱඈ͚ͩ͢ • ͜͜΄ͱΜͲ͍͡Βͳ͍ • தԝͷ
aggregator ʹͯ͠Α͠ͳʹॲཧ • ࣭ϕετΤϑΥʔτ • ඞཁͳॴʹผ్อূͨ͠ΈΛཱͯΔ • ͍ͭͰʹϦόʔεϓϩΩγ͔ΒૹΓ࢝ΊΔ
reverse proxy App server DB access log error log structured
log aggregator
֤ॴʹૹ • S3 ʹੜσʔλΛอଘ • Կ͔ίέͯ࠶ݱ • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Mackerelʹߘ • ֤छϝτϦοΫՄࢹԽ • ϨΠςϯγ • ΤϥʔϨʔτ •
KPI
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Slackʹ௨ • ΤϥʔϩάΛूΊΔ • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Elasticsearch→Kibana ͰՄࢹԽ • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • MongoDB capped collection ʹೖ • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ • ʢ࠷ۙ͋·ΓΘΕ͍ͯͳ͍ʣ
reverse proxy App server DB access log error log structured
log aggregator
શܠ
reverse proxy App server DB access log error log structured
log syslog server storage server SSD server rsyslog aggregator
ࠓޙ • syslog ͔Βͷ٫ • ։ൃऀҎ֎׆༻Ͱ͖ΔΈ࡞Γ • ߦಈϩάج൫
ຊͷ༰ • ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λհ͠·ͨ͠ • ࠓޙಓͳվળΛଓ͚͍͖ͯ·͢
Ҏ্Ͱ͢ • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ http://hatenacorp.jp/recruit/career/operation-engineer • ౦ژɾژͲͪΒͰ
࣭ٙɾTwitterΑΓ • fluentd aggregator ٧·Γ·ͤΜ͔ • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝ ͱʹ fluentd
Λཱͯͯอଘ͍ͯ͠·͢ • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ ͍ͯ͠·͢
࣭ٙɾTwitterΑΓ • ߦಈϩάͰܽଛ߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜ Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢ • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ ֊Ͱ͢… • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ ͯܽଛ͠ͳ͍ΈΛ࡞Γ·͢ɻKafka
ͷΑ͏ͳ ΩϡʔΛڬΉͳͲ
࣭ٙɾTwitterΑΓ • յΕͨSSDͲ͜ͷͰ͔͢ • ॳ Crucial M4 Λ͍ͬͯ·ͨ͠