はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

Hatena Engineer Seminar #6 インフラ編での発表スライドです

Transcript

1. ͸ͯͳͷϩάӡ༻ ͜Ε·Ͱͱ͜Ε͔Β id:wtatsuru 2016/08/31 Hatena Engineer Seminar #6
2. ࣗݾ঺հ • Tatsuru Watabane (id:wtatsuru) • ͸ͯͳγεςϜϓϥοτϑΥʔϜ෦௕ • 2011೥ೖࣾ • 2014೥·ͰͷαʔϏε͸͍͍ͩͨશ෦ӡ༻
3. ຊ೔ͷ಺༰ • લఏͱͳΔγεςϜߏ੒ • ͸ͯͳͷϩάͷߏ੒ͱӡ༻Λ঺հ • ࠓޙ
4. લఏͱͳΔγεςϜߏ੒
5. 3૚ߏ଄ reverse proxy App server DB
6. ѻ͏ϩά reverse proxy App server DB access log error log various logs
7. ѻ͏ϩά • αʔϏεͷΞΫηεϩά • ˏϦόʔεϓϩΩγ • ΞϓϦέʔγϣϯ͕ు͘ϩά • Τϥʔϩάɺߦಈϩά etc. reverse proxy App server DB access log error log various logs
8. લఏɿଟ͘ͷαʔϏε • ৽αʔϏε͔Β10೥Ҏ্ଓ͘αʔϏε·Ͱଟ༷ • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ
9. ͸ͯͳͷϩάӡ༻
10. reverse proxy App server DB access log error log structured log syslog server storage server SSD server rsyslog aggregator
11. ͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ • ΞϓϦέʔγϣϯϩάͷӡ༻ • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢
12. ΞΫηεϩάग़ྗ • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ • ֤छௐࠪɺૢ࡞ϩάɺetc. • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ • LTSV (Labeled Tab-separated Values) http://ltsv.org/ host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700] req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http:// www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)
13. reverse proxy App server DB access log ΞΫηεϩάग़ྗ
14. ϩάΛอଘ͍ͨ͠ • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ • rsyslog Ͱඈ͹ͯ͠ετϨʔδαʔόʹอଘ • ϩʔΧϧ͔ΒS3ʹ΋อଘͯ͠ೋॏԽ • ͠͹Β͘͢ΔͱGlacierʹ௜Ή
15. reverse proxy App server DB access log syslog server rsyslog storage server
16. ؆୯ʹ෼ੳ͍ͨ͠ • SSDͷαʔόʹ΋ඈ͹͢ • SSD͕҆͘ͳͬͨͷͰಋೖ • ࡶଟʹ grep ΍෼ੳ • ΞΫηεϩά͔ΒϨΠςϯγ෼ੳ • 2೥ॻ͍ͨΒSSD͕յΕͨͷͰަ׵
17. reverse proxy App server DB access log syslog server storage server SSD server rsyslog
18. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • S3 ͔Β Amazon EMR (ElasticMapReduce) ΁ • LTSVΛಡΈࠐΉσγϦΞϥΠβ 
 github.com/hatena/KeyValuePairsDeserializer • SQLͰ؆୯ʹ෼ੳ select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘
19. reverse proxy App server DB access log syslog server storage server SSD server rsyslog
20. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • ֤छ՝୊ • ॳظԽ͕஗͍ • ؀ڥߏங͕೉͍͠ • ఆظతʹճ͢ͱίετ͕ • ࢖ΘΕͳ͘ͳ͍ͬͯ͘…
21. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • EMRஔ͖׵͑ɿGoogle BigQuery Λݕূ • LTSV ͸ embulk Ͱલॲཧͯ͠ಡΈࠐΉ • ଎͍ɾ·͋·͍͋҆ • ։ൃऀҎ֎Ͱ΋࢖͍΍͍͢
22. reverse proxy App server DB access log syslog server storage server SSD server rsyslog
23. ͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢
24. ΞϓϦέʔγϣϯͷϩά • ૉ๿ͳੲ • /var/log/httpd/access.log • /var/log/httpd/error.log • ֤छͪ͝Ό·ͥ • Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍ • DBʹೖΕΔ
25. reverse proxy App server DB
26. ߏ଄ԽϩάΛग़͢ • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘ • ඪ४ग़ྗɾΤϥʔग़ྗͱ͸෼͚Δ • ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘ • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ΋
27. reverse proxy App server DB structured log
28. fluentdͰ഑ૹ • ֤ϗετͰ in_tail +out_forward Ͱඈ͹͚ͩ͢ • ͜͜͸΄ͱΜͲ͍͡Βͳ͍ • தԝͷ aggregator ʹ౉ͯ͠Α͠ͳʹॲཧ • ඼࣭͸ϕετΤϑΥʔτ • ඞཁͳ৔ॴʹ͸ผ్อূͨ͠࢓૊ΈΛཱͯΔ • ͍ͭͰʹϦόʔεϓϩΩγ͔Β΋ૹΓ࢝ΊΔ
29. reverse proxy App server DB access log error log structured log aggregator
30. ֤ॴʹ഑ૹ • S3 ʹੜσʔλΛอଘ • Կ͔ίέͯ΋࠶ݱ • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ
31. reverse proxy App server DB access log error log structured log aggregator
32. ϩάͷ഑ૹ • Mackerelʹ౤ߘ • ֤छϝτϦοΫՄࢹԽ • ϨΠςϯγ • ΤϥʔϨʔτ • KPI
33. reverse proxy App server DB access log error log structured log aggregator
34. ϩάͷ഑ૹ • Slackʹ௨஌ • ΤϥʔϩάΛूΊΔ • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……
35. reverse proxy App server DB access log error log structured log aggregator
36. ϩάͷ഑ૹ • Elasticsearch→Kibana ͰՄࢹԽ • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ
37. reverse proxy App server DB access log error log structured log aggregator
38. ϩάͷ഑ૹ • MongoDB capped collection ʹ౤ೖ • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ • ʢ࠷ۙ͋·Γ࢖ΘΕ͍ͯͳ͍ʣ
39. reverse proxy App server DB access log error log structured log aggregator
40. શܠ
41. reverse proxy App server DB access log error log structured log syslog server storage server SSD server rsyslog aggregator
42. ࠓޙ • syslog ͔Βͷ୤٫ • ։ൃऀҎ֎΋׆༻Ͱ͖Δ࢓૊Έ࡞Γ • ߦಈϩάج൫
43. ຊ೔ͷ಺༰ • ͸ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λ঺հ͠·ͨ͠ • ࠓޙ΋஍ಓͳվળΛଓ͚͍͖ͯ·͢
44. Ҏ্Ͱ͢ • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ
 http://hatenacorp.jp/recruit/career/operation-engineer • ౦ژɾژ౎ͲͪΒͰ΋
45. ࣭ٙɾTwitterΑΓ • fluentd aggregator ٧·Γ·ͤΜ͔ • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝ ͱʹ fluentd Λཱͯͯอଘ͍ͯ͠·͢ • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ ͍ͯ͠·͢
46. ࣭ٙɾTwitterΑΓ • ߦಈϩάͰ͸ܽଛ΍੔߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜ Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢ • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ ֊Ͱ͢… • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ ͯܽଛ͠ͳ͍࢓૊ΈΛ࡞Γ·͢ɻKafka ͷΑ͏ͳ ΩϡʔΛڬΉͳͲ
47. ࣭ٙɾTwitterΑΓ • յΕͨSSD͸Ͳ͜ͷͰ͔͢ • ॳ୅͸ Crucial M4 Λ࢖͍ͬͯ·ͨ͠