Upgrade to Pro — share decks privately, control downloads, hide ads and more …

はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

tatsuru
August 31, 2016
12k

はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

Hatena Engineer Seminar #6 インフラ編での発表スライドです

tatsuru

August 31, 2016
Tweet

Transcript

  1. ͸ͯͳͷϩάӡ༻
    ͜Ε·Ͱͱ͜Ε͔Β
    id:wtatsuru
    2016/08/31 Hatena Engineer Seminar #6

    View full-size slide

  2. ࣗݾ঺հ
    • Tatsuru Watabane (id:wtatsuru)
    • ͸ͯͳγεςϜϓϥοτϑΥʔϜ෦௕
    • 2011೥ೖࣾ
    • 2014೥·ͰͷαʔϏε͸͍͍ͩͨશ෦ӡ༻

    View full-size slide

  3. ຊ೔ͷ಺༰
    • લఏͱͳΔγεςϜߏ੒
    • ͸ͯͳͷϩάͷߏ੒ͱӡ༻Λ঺հ
    • ࠓޙ

    View full-size slide

  4. લఏͱͳΔγεςϜߏ੒

    View full-size slide

  5. 3૚ߏ଄
    reverse proxy
    App server
    DB

    View full-size slide

  6. ѻ͏ϩά
    reverse proxy
    App server
    DB
    access log
    error log
    various logs

    View full-size slide

  7. ѻ͏ϩά
    • αʔϏεͷΞΫηεϩά
    • ˏϦόʔεϓϩΩγ
    • ΞϓϦέʔγϣϯ͕ు͘ϩά
    • Τϥʔϩάɺߦಈϩά etc.
    reverse proxy
    App server
    DB
    access log
    error log
    various logs

    View full-size slide

  8. લఏɿଟ͘ͷαʔϏε
    • ৽αʔϏε͔Β10೥Ҏ্ଓ͘αʔϏε·Ͱଟ༷
    • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ
    • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ

    View full-size slide

  9. ͸ͯͳͷϩάӡ༻

    View full-size slide

  10. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    syslog server
    storage server
    SSD server
    rsyslog
    aggregator

    View full-size slide

  11. ͸ͯͳͷϩάӡ༻
    • ΞΫηεϩάͷӡ༻
    • ΞϓϦέʔγϣϯϩάͷӡ༻
    • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

    View full-size slide

  12. ΞΫηεϩάग़ྗ
    • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ
    • ֤छௐࠪɺૢ࡞ϩάɺetc.
    • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ
    • LTSV (Labeled Tab-separated Values) http://ltsv.org/
    host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700]
    req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http://
    www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)

    View full-size slide

  13. reverse proxy
    App server
    DB
    access log
    ΞΫηεϩάग़ྗ

    View full-size slide

  14. ϩάΛอଘ͍ͨ͠
    • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ
    • rsyslog Ͱඈ͹ͯ͠ετϨʔδαʔόʹอଘ
    • ϩʔΧϧ͔ΒS3ʹ΋อଘͯ͠ೋॏԽ
    • ͠͹Β͘͢ΔͱGlacierʹ௜Ή

    View full-size slide

  15. reverse proxy
    App server
    DB
    access log
    syslog server
    rsyslog
    storage server

    View full-size slide

  16. ؆୯ʹ෼ੳ͍ͨ͠
    • SSDͷαʔόʹ΋ඈ͹͢
    • SSD͕҆͘ͳͬͨͷͰಋೖ
    • ࡶଟʹ grep ΍෼ੳ
    • ΞΫηεϩά͔ΒϨΠςϯγ෼ੳ
    • 2೥ॻ͍ͨΒSSD͕յΕͨͷͰަ׵

    View full-size slide

  17. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog

    View full-size slide

  18. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • S3 ͔Β Amazon EMR (ElasticMapReduce) ΁
    • LTSVΛಡΈࠐΉσγϦΞϥΠβ 

    github.com/hatena/KeyValuePairsDeserializer
    • SQLͰ؆୯ʹ෼ੳ
    select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘

    View full-size slide

  19. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog

    View full-size slide

  20. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • ֤छ՝୊
    • ॳظԽ͕஗͍
    • ؀ڥߏங͕೉͍͠
    • ఆظతʹճ͢ͱίετ͕
    • ࢖ΘΕͳ͘ͳ͍ͬͯ͘…

    View full-size slide

  21. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • EMRஔ͖׵͑ɿGoogle BigQuery Λݕূ
    • LTSV ͸ embulk Ͱલॲཧͯ͠ಡΈࠐΉ
    • ଎͍ɾ·͋·͍͋҆
    • ։ൃऀҎ֎Ͱ΋࢖͍΍͍͢

    View full-size slide

  22. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog

    View full-size slide

  23. ͸ͯͳͷϩάӡ༻
    • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ
    • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β
    • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

    View full-size slide

  24. ΞϓϦέʔγϣϯͷϩά
    • ૉ๿ͳੲ
    • /var/log/httpd/access.log
    • /var/log/httpd/error.log
    • ֤छͪ͝Ό·ͥ
    • Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍
    • DBʹೖΕΔ

    View full-size slide

  25. reverse proxy
    App server
    DB

    View full-size slide

  26. ߏ଄ԽϩάΛग़͢
    • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘
    • ඪ४ग़ྗɾΤϥʔग़ྗͱ͸෼͚Δ
    • ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘
    • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ΋

    View full-size slide

  27. reverse proxy
    App server
    DB
    structured log

    View full-size slide

  28. fluentdͰ഑ૹ
    • ֤ϗετͰ in_tail +out_forward Ͱඈ͹͚ͩ͢
    • ͜͜͸΄ͱΜͲ͍͡Βͳ͍
    • தԝͷ aggregator ʹ౉ͯ͠Α͠ͳʹॲཧ
    • ඼࣭͸ϕετΤϑΥʔτ
    • ඞཁͳ৔ॴʹ͸ผ్อূͨ͠࢓૊ΈΛཱͯΔ
    • ͍ͭͰʹϦόʔεϓϩΩγ͔Β΋ૹΓ࢝ΊΔ

    View full-size slide

  29. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View full-size slide

  30. ֤ॴʹ഑ૹ
    • S3 ʹੜσʔλΛอଘ
    • Կ͔ίέͯ΋࠶ݱ
    • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ

    View full-size slide

  31. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View full-size slide

  32. ϩάͷ഑ૹ
    • Mackerelʹ౤ߘ
    • ֤छϝτϦοΫՄࢹԽ
    • ϨΠςϯγ
    • ΤϥʔϨʔτ
    • KPI

    View full-size slide

  33. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View full-size slide

  34. ϩάͷ഑ૹ
    • Slackʹ௨஌
    • ΤϥʔϩάΛूΊΔ
    • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……

    View full-size slide

  35. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View full-size slide

  36. ϩάͷ഑ૹ
    • Elasticsearch→Kibana ͰՄࢹԽ
    • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ

    View full-size slide

  37. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View full-size slide

  38. ϩάͷ഑ૹ
    • MongoDB capped collection ʹ౤ೖ
    • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ
    • ʢ࠷ۙ͋·Γ࢖ΘΕ͍ͯͳ͍ʣ

    View full-size slide

  39. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View full-size slide

  40. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    syslog server
    storage server
    SSD server
    rsyslog
    aggregator

    View full-size slide

  41. ࠓޙ
    • syslog ͔Βͷ୤٫
    • ։ൃऀҎ֎΋׆༻Ͱ͖Δ࢓૊Έ࡞Γ
    • ߦಈϩάج൫

    View full-size slide

  42. ຊ೔ͷ಺༰
    • ͸ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λ঺հ͠·ͨ͠
    • ࠓޙ΋஍ಓͳվળΛଓ͚͍͖ͯ·͢

    View full-size slide

  43. Ҏ্Ͱ͢
    • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ

    http://hatenacorp.jp/recruit/career/operation-engineer
    • ౦ژɾژ౎ͲͪΒͰ΋

    View full-size slide

  44. ࣭ٙɾTwitterΑΓ
    • fluentd aggregator ٧·Γ·ͤΜ͔
    • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝
    ͱʹ fluentd Λཱͯͯอଘ͍ͯ͠·͢
    • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ
    ͍ͯ͠·͢

    View full-size slide

  45. ࣭ٙɾTwitterΑΓ
    • ߦಈϩάͰ͸ܽଛ΍੔߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜
    Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢
    • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ
    ֊Ͱ͢…
    • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ
    ͯܽଛ͠ͳ͍࢓૊ΈΛ࡞Γ·͢ɻKafka ͷΑ͏ͳ
    ΩϡʔΛڬΉͳͲ

    View full-size slide

  46. ࣭ٙɾTwitterΑΓ
    • յΕͨSSD͸Ͳ͜ͷͰ͔͢
    • ॳ୅͸ Crucial M4 Λ࢖͍ͬͯ·ͨ͠

    View full-size slide