Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6
Search
tatsuru
PRO
August 31, 2016
7
12k
はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6
Hatena Engineer Seminar #6 インフラ編での発表スライドです
tatsuru
PRO
August 31, 2016
Tweet
Share
More Decks by tatsuru
See All by tatsuru
プロダクトオーナーとしてSLOに向き合う 〜Mackerelチームの事例〜 / SRE NEXT 2023
tatsuru
PRO
0
2.4k
Mackerelのプロダクト開発 - エンジニア中心の開発プロセスで大切にしていること
tatsuru
PRO
0
4.7k
Mackerel の EventBridge 対応開発秘話
tatsuru
PRO
1
180
技術が実現するイノベーションとWebサービス運用の未来 / Innovation from&for Web Operations
tatsuru
PRO
0
1.6k
成長するためのエンジニア組織 / Hatena Engineering Group 2018
tatsuru
PRO
1
110
Mesosを使ったImmutable Infra 管理システムを作ってみた
tatsuru
PRO
8
8k
Featured
See All Featured
Done Done
chrislema
185
16k
Embracing the Ebb and Flow
colly
87
4.8k
Navigating Team Friction
lara
189
15k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
31
2.2k
Making Projects Easy
brettharned
117
6.4k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.7k
How to Think Like a Performance Engineer
csswizardry
26
1.9k
Building Adaptive Systems
keathley
43
2.7k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
A Modern Web Designer's Workflow
chriscoyier
696
190k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Transcript
ͯͳͷϩάӡ༻ ͜Ε·Ͱͱ͜Ε͔Β id:wtatsuru 2016/08/31 Hatena Engineer Seminar #6
ࣗݾհ • Tatsuru Watabane (id:wtatsuru) • ͯͳγεςϜϓϥοτϑΥʔϜ෦ • 2011ೖࣾ •
2014·ͰͷαʔϏε͍͍ͩͨશ෦ӡ༻
ຊͷ༰ • લఏͱͳΔγεςϜߏ • ͯͳͷϩάͷߏͱӡ༻Λհ • ࠓޙ
લఏͱͳΔγεςϜߏ
3ߏ reverse proxy App server DB
ѻ͏ϩά reverse proxy App server DB access log error log
various logs
ѻ͏ϩά • αʔϏεͷΞΫηεϩά • ˏϦόʔεϓϩΩγ • ΞϓϦέʔγϣϯ͕ు͘ϩά • Τϥʔϩάɺߦಈϩά etc.
reverse proxy App server DB access log error log various logs
લఏɿଟ͘ͷαʔϏε • ৽αʔϏε͔Β10Ҏ্ଓ͘αʔϏε·Ͱଟ༷ • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ
ͯͳͷϩάӡ༻
reverse proxy App server DB access log error log structured
log syslog server storage server SSD server rsyslog aggregator
ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ • ΞϓϦέʔγϣϯϩάͷӡ༻ • ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞΫηεϩάग़ྗ • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ • ֤छௐࠪɺૢ࡞ϩάɺetc. • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ • LTSV (Labeled
Tab-separated Values) http://ltsv.org/ host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700] req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http:// www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)
reverse proxy App server DB access log ΞΫηεϩάग़ྗ
ϩάΛอଘ͍ͨ͠ • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ • rsyslog Ͱඈͯ͠ετϨʔδαʔόʹอଘ • ϩʔΧϧ͔ΒS3ʹอଘͯ͠ೋॏԽ • ͠Β͘͢ΔͱGlacierʹΉ
reverse proxy App server DB access log syslog server rsyslog
storage server
؆୯ʹੳ͍ͨ͠ • SSDͷαʔόʹඈ͢ • SSD͕҆͘ͳͬͨͷͰಋೖ • ࡶଟʹ grep ੳ •
ΞΫηεϩά͔ΒϨΠςϯγੳ • 2ॻ͍ͨΒSSD͕յΕͨͷͰަ
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͬͱ؆୯ʹੳ͍ͨ͠ • S3 ͔Β Amazon EMR (ElasticMapReduce) • LTSVΛಡΈࠐΉσγϦΞϥΠβ
github.com/hatena/KeyValuePairsDeserializer • SQLͰ؆୯ʹੳ select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͬͱ؆୯ʹੳ͍ͨ͠ • ֤छ՝ • ॳظԽ͕͍ • ڥߏங͕͍͠ • ఆظతʹճ͢ͱίετ͕ •
ΘΕͳ͘ͳ͍ͬͯ͘…
ͬͱ؆୯ʹੳ͍ͨ͠ • EMRஔ͖͑ɿGoogle BigQuery Λݕূ • LTSV embulk Ͱલॲཧͯ͠ಡΈࠐΉ
• ͍ɾ·͋·͍͋҆ • ։ൃऀҎ֎Ͱ͍͍͢
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β • ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞϓϦέʔγϣϯͷϩά • ૉͳੲ • /var/log/httpd/access.log • /var/log/httpd/error.log • ֤छͪ͝Ό·ͥ •
Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍ • DBʹೖΕΔ
reverse proxy App server DB
ߏԽϩάΛग़͢ • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘ • ඪ४ग़ྗɾΤϥʔग़ྗͱ͚Δ •
ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘ • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ
reverse proxy App server DB structured log
fluentdͰૹ • ֤ϗετͰ in_tail +out_forward Ͱඈ͚ͩ͢ • ͜͜΄ͱΜͲ͍͡Βͳ͍ • தԝͷ
aggregator ʹͯ͠Α͠ͳʹॲཧ • ࣭ϕετΤϑΥʔτ • ඞཁͳॴʹผ్อূͨ͠ΈΛཱͯΔ • ͍ͭͰʹϦόʔεϓϩΩγ͔ΒૹΓ࢝ΊΔ
reverse proxy App server DB access log error log structured
log aggregator
֤ॴʹૹ • S3 ʹੜσʔλΛอଘ • Կ͔ίέͯ࠶ݱ • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Mackerelʹߘ • ֤छϝτϦοΫՄࢹԽ • ϨΠςϯγ • ΤϥʔϨʔτ •
KPI
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Slackʹ௨ • ΤϥʔϩάΛूΊΔ • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Elasticsearch→Kibana ͰՄࢹԽ • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • MongoDB capped collection ʹೖ • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ • ʢ࠷ۙ͋·ΓΘΕ͍ͯͳ͍ʣ
reverse proxy App server DB access log error log structured
log aggregator
શܠ
reverse proxy App server DB access log error log structured
log syslog server storage server SSD server rsyslog aggregator
ࠓޙ • syslog ͔Βͷ٫ • ։ൃऀҎ֎׆༻Ͱ͖ΔΈ࡞Γ • ߦಈϩάج൫
ຊͷ༰ • ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λհ͠·ͨ͠ • ࠓޙಓͳվળΛଓ͚͍͖ͯ·͢
Ҏ্Ͱ͢ • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ http://hatenacorp.jp/recruit/career/operation-engineer • ౦ژɾژͲͪΒͰ
࣭ٙɾTwitterΑΓ • fluentd aggregator ٧·Γ·ͤΜ͔ • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝ ͱʹ fluentd
Λཱͯͯอଘ͍ͯ͠·͢ • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ ͍ͯ͠·͢
࣭ٙɾTwitterΑΓ • ߦಈϩάͰܽଛ߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜ Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢ • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ ֊Ͱ͢… • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ ͯܽଛ͠ͳ͍ΈΛ࡞Γ·͢ɻKafka
ͷΑ͏ͳ ΩϡʔΛڬΉͳͲ
࣭ٙɾTwitterΑΓ • յΕͨSSDͲ͜ͷͰ͔͢ • ॳ Crucial M4 Λ͍ͬͯ·ͨ͠