大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure

実際運用してみてわかった、大規模Mastodonインスタンスを運用するコツ
2017-04-25 at pixiv night!

Transcript

1. @harukasan / Shunsuke Michii ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ
2. @Harukasan Shunsuke MICHII Lead engineer of pixiv Inc.,
 Product manager of ImageFlux,
 Infrastructure engineer, Kibana, Gopher
4. ࣮ࡍӡ༻ͯ͠ΈͯΘ͔ͬͨɺେن໛.BTUPEPOΠϯελϯεΛӡ༻͢Δίπ
6. Mastodon
7. ͍͖ͭͬ͞v1.2.2Λͩͨ͠
8. github.com/pixiv/mastodon:pawoo
9. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket
 (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx
10. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket
 (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx RESTͬΆ͍API
11. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket
 (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx RESTͬΆ͍API ετϦʔϛϯάAPI
12. 10࣌ؒͰϦϦʔε
13. Pawooͷߏ੒
15. AWSͰγϡοͱ΍Δ • ͕࣌ؒͳ͍͚Ͳ࠷༏ઌ͸҆ఆੑ • RDS/ElasticaceͷϚϧνAZߏ੒ • ϝσΟΞ͸S3ʹอଘ • EC2ฒ΂ͯALBͰϩʔυόϥϯε • CloudFrontͰྲྀ͢ʢ΋͏΍Ίͨʣ • ظؒݶఆαΠτΛཱͯΔ࣌ͷϊ΢ϋ΢Λྲྀ༻
16. ΠϯελϯεΛฒ΂Δ • ͢΂ͯͷΠϯελϯε͸ಉҰߏ੒ • γϡοͱͭͬͯ͘AMIͰ૿΍͢ • ԣʹฒ΂ͯεέʔϧΞ΢τ • ͦͷ͏ͪΦʔτεέʔϦϯά͍ͨ͠
17. Docker͔Β͸͕͢ • શ෦systemdαʔϏεʹஔ͖׵͑ͨ • ͦͷ··࢖͏ͷ͸ͪΐͬͱ೉͍͜͠ͱ͕ଟ͍ • σϓϩΠͷ͜ͱ΍ΒͳΜ΍Βߟ͍͑ͯΔͱͪΐͬͱ༨༟͕ͳ͍ • DockerͰ͙͢΍ΔͳΒϗοτσϓϩΠͱ͔Ϧιʔε഑෼ͷ࢓૊ΈΛ ͋Β͔͡Ί༻ҙ͓ͯ͘͠ඞཁ͕͋Γͦ͏ʢͦͷ͏ͪ΍Γ͍ͨʣ
18. ͦͷ೔ͷ͏ͪʹϦϦʔε͞Ε͕ͨ ͙͢ʹΤϥʔΛฦ͠͸͡ΊΔ ʢ͜͜ΒลͰνϟοτΛݟ࢝ΊΔʣ
19. nginx • ϦΫΤετόοϑΝϦϯάͷͨΊʹPumaͷલʹnginxΛ͓͘ • Τϥʔ͕ग़͍ͯͨͷͰͳ͓ͨ͠ • worker_rlimit_nofile: limitΛ૿΍͓ͯ͘͠ • worker_connections: ίωΫγϣϯͨ͘͞Μ͸ΔͷͰ • ϩάϩʔςʔτͷઃఆ͕ೖͬͯͳ͔ͬͨͷͰೖΕͨ • όϥϯγϯά͸EC2ͷnginx͡Όͳͯ͘ALBͰ΍Δ
20. ৄ͘͠͸͜ͷຊʹ
21. PostgreSQL • PostgreSQL͸ίωΫγϣϯ͝ͱʹfork͢Δ
 ʢฐࣾ͸͢΂ͯMySQLͩͬͨͷͰ஌ݟͳ͠ʣ • ίωΫγϣϯϓʔϧΛ͏·͍۩߹ͷ஋ʹઃఆ͢Δ • ΠϯελϯεΛεέʔϧΞοϓͯ͠ରԠ • ίʔυʹ࠷దԽΛೖΕͯΠϯελϯεαΠζখͨ͘͞͠Γͨ͠ • Մ༻ੑͱόοΫΞοϓ͸RDSʹ͓೚ͤ • PgBouncer࢖ͬͨํ͕͍͍͔΋ʁͱࢥ͍ͳ͕Βාͯ͘࢖ͬͯͳ͍
22. ͜͜ͰεέʔϧΞοϓ
24. ஌ݟٻΉ
25. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket
 (streaming) HTTP SMTP ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ nginx
26. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket
 (streaming) HTTP SMTP nginx ϢʔβʔΠϯλʔϑΣʔε ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ
27. Ruby on Rails node.js PostgreSQL Redis SendGrid Sidekiq ΫϥΠΞϯτ Websocket
 (streaming) HTTP SMTP nginx ϞμϯͳRuby on RailsΞϓϦέʔγϣϯ ϝΠϯϩδοΫ
28. Sidekiq • Mastodonͷϝοηʔδύογϯά͸SidekiqͰߦΘΕΔ • τΡʔτͷ഑৴ • ετϦʔϛϯά • ϦϞʔτΠϯελϯε΁ͷ഑৴ • 1τΡʔτ͝ͱʹϑΥϩϫʔͷ਺͚ͩenqueue͞ΕΔ • ϑΥϩϫʔ͕ଟ͍Ϣʔβʔͩͱ΍͹͍
29. ීஈͷΩϡʔͷ༷ࢠʢͪΐͬͱલʣ
30. ͨ·ʹ͜͏ͳΔ
31. ͨ·ʹ͜͏ͳΔ
32. Ωϡʔͷछྨ default pull push mail • ΄͔ͷશ෦ͷδϣϒ • ϦϞʔτΠϯελϯε΁ͷ഑৴ • ϝʔϧͷૹ৴
33. Sidekiq٧·Δ໰୊ • σϑΥϧτͩͱ͢΂ͯͷΩϡʔΛ̍ͭͷSidekiqϓϩηεͰॲཧ͢Δ • ϦϞʔτΠϯελϯε͕٧·Δͱpull/pushͷΩϡʔ͕٧·Δ • pull/pushͷ͍ͤͰλΠϜϥΠϯ͕஗Ԇ͢Δʂ • ͜ͷ··ͩͱαʔϏεͱͯ͠ऴΘͬͯ͠·͏ͷͰɺϓϩηεΛ෼͚Δ • Sidekiq͸1ϓϩηεͰ1ίΞʢ100%ʣ·Ͱ͔͠࢖ͬͯ͘Εͳ͍ • ϓϩηεΛͨ͘͞ΜཱͯͯΧόʔ
34. ίϚϯυϥΠϯҾ਺Ͱࢦఆ͢Δ͚ͩ [Service] Type=simple User=mastodon WorkingDirectory=/home/mastodon/live Environment="RAILS_ENV=production" Environment="DB_POOL=40" ExecStart=/usr/local/rbenv/shims/bundle exec sidekiq -c 40 -q default TimeoutSec=15 Restart=always
35. SidekiqϓϩηεΛ෼͚Δ sidekiq-default-1.service sidekiq-default-2.service sidekiq-default-3.service sidekiq-pullpush.service sidekiq-mail.service 1Πϯελϯεʹ͜Εཱ͚ͩͯͯΔɿ σϑΥϧτ͚ͩͰ1,000εϨουҎ্ 40 40 40 10 5 εϨου਺
36. Sidekiqͷϓϩηεը໘͕
 ͍ͩͿݟͮΒ͍
37. Sidekiq٧·Δ໰୊ͷ՝୊ • ϦϞʔτΠϯελϯε͕٧·Δͱ૬มΘΒͣpull/push͕٧·Δ • ϓϩηεΛཱͯͯΧόʔͯ͠Δ͚Ͳɺ΄͔ͷΠϯελϯε͔Β͢Δͱ ٸʹϦΫΤετ͕૿͑ͯͭΒͦ͏…… • ஗͍Πϯελϯεͱ͔େن໛ΠϯελϯεͷॲཧΛͲ͏ʹ͔͍ͨ͠ • ͦ΋ͦ΋τΡʔτ͋ͨΓͷδϣϒ਺͕N+1ʹͳͬͯΔͷ͕΍͹͍
38. ղܾͯ͠ͳ͍͚Ͳ·ͱΊ • AWS࢖ͬͯγϡοͱཱͯͯӡ༻։࢝ • ϚετυϯͷϝΠϯ͸Sidekiqδϣϒ • SidekiqͷΩϡʔΛ͏·͘͞͹͚ΔͱΑ͍ • SidekiqϓϩηεΛεϙοτΠϯελϯεͰ૿΍͢Α͏ʹ͍ͨ͠ • Ͳ͔ͬͪͱ͍͏ͱ࠷దԽͨ͠ํ͕Αͦ͞͏