Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
モンストを支えるインフラの今とこれから
Search
Isao Shimizu
March 01, 2016
Technology
13k
70
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
モンストを支えるインフラの今とこれから
dots. Conference Spring 2016
ゲーム開発の裏側
http://eventdots.jp/event/580344
Isao Shimizu
March 01, 2016
Transcript
50 slides
ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β 2016.3.1 dots. Conference Spring 2016 ήʔϜ։ൃͷཪଆ mixi, Inc. XFLAG™
STUDIO @isaoshimizu
About Me • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu • 20118݄ΑΓגࣜձࣾϛΫγΟ
• ݱࡏΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ • 2લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻ 2
ϞϯελʔετϥΠΫ • 201310݄10 ਖ਼ࣜϦϦʔε • iOS൛, Android൛Λఏڙ • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰͰॏෳμϯϩʔυ͞Εؚͨ·ͣʣ •
ຊɺɺؖࠃɺถɺ߳ߓɾϚΧΦͰల։ • YouTubeͰΞχϝͷ৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ100ສຊಥഁ 3
͔͜͜ΒϞϯελʔετϥΠΫ ຊ൛ʹݶఆͨ͠༰ʹͳΓ·͢ 4
ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ 5
6 Application 300+ Servers Resque 50 Servers Redis 30 Servers
Memcached 200+ Servers CDN Traffic Max 270Gbps API Traffic Avg 1Gbps TURN 40 Servers MariaDB 250+ Servers Internal Traffic Max 20Gbps
Πϯϑϥߏ 7
8 DC1 DC2 Main Main/Backup Main/Dev
ෳDCͱAmazon VPC • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃڥʣͰߏ • ֤DCTY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτଓ •
֤DCؒยܥো࣌ʹ͓͍ͯ20Gbps·Ͱ͑ΒΕΔઃܭ • ৄ͍͠ղઆ “ڌͨ͠ʢ෦τϥϑΟοΫϘϦϡʔϜฤʣ” http://xflag.com/blog/infradb/internal_traffic_volume.html 9
֤ڌͷׂ 10
11 DC1 DC2 Main Main/Backup Main/Dev Application MariaDB Redis Resque
Memcached MariaDB Redis Memcached TURN Server ։ൃ༻Πϯελϯε܈ Application Resque
֤ڌͷׂ • ຊ൪ͷApplicationɺResqueDC1ɺDC2ͷ྆ํͰՔಇ • ຊ൪ͷRedisɺMariaDBɺMemcached • DC1ʹMaster/Backupͷηοτ • DC2ʹBackup •
BackupDC1ͱDC2߹Θͤͯ2ηοτ͋Δ • AWSͰຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ 12
ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ 13
14 DC1 DC2 AWS TURN Server (EC2) CDN (CloudFront, Akamai)
API API LB LB Internet Client
ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏʣ • ϩʔυόϥϯαʔͷΞΫηεDNSϥϯυϩϏϯʢ2ͭͷԾIPʣ • ϩʔυόϥϯαʔ͔Β֤ڌͷApplicationαʔόৼΓ͚ • AWSͰϚϧνϓϨΠ༻ʹTURN ServerʢEC2
+ Elastic IPʣ • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamaiซ༻ɻ 15
σϓϩΠ 16
17 DC1 DC2 Deploy Server S3 Bucket tarball S3 PUT
GitHub ssh Code Internet
σϓϩΠ • Capistrano 2 • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli •
GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞ • ApplicationαʔόͰS3ʹΞοϓϩʔυ͞ΕͨtarballΛμϯϩʔυɺల։ • GitHubͷଓ੍ݶରࡦʢgithub.comେྔʹଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ 18
• capistrano-rbenv https://github.com/yyuu/capistrano-rbenv • όʔδϣϯࢦఆͰRubyͷɺΞοϓσʔτָ͕ʹ • Ruby 2.2.4ಋೖࡁΈ 19
ϓϩϏδϣχϯά 20
21 DC1 DC2 Deploy Server S3 Bucket GitHub ssh Chef
Cookbook Ubuntu Package Mirror Internet deb Package deb Package
ϓϩϏδϣχϯά • ChefʹΑΔϓϩϏδϣχϯά͕جຊ • capistrano-paratrooper-chef https://github.com/tk0miya/capistrano-paratrooper-chef • ཧϚγϯͷॳظηοτΞοϓʢ࠷ݶʣAnsibleΛ͏ • OSΫϦʔϯΠϯετʔϧޙͷঢ়ଶͰɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ
22
• Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞ɺར༻͍ͯ͠Δ • ಠࣗCookbookɺXFLAGશମͷΠϯϑϥͰڞ௨ͯ͑͠ΔΑ͏ʹ • ͯ͢ͷαʔό͕ChefͰߏஙՄೳ • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻͕ͯ͠ى͖ͳ͍Α͏ʹ 23
• S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷোରࡦʣ • ioDriveͳͲͷυϥΠόஔ͍ͯ͋Δ • aptlyͰߏங http://www.aptly.info/ • ݸਓͷ։ൃϚγϯͰVagrant +
VirtualBox + ChefͰߏͰ͖ΔΑ͏ʹ • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen 24
ෛՙରࡦ 25
ࠓʹೖ͔ͬͯΒߴෛՙ͕ଓ͘… 26
ෛՙରࡦ • DBͷγϟʔσΟϯάɺςʔϒϧׂΛܧଓ࣮ࢪ • Applicationαʔόͷ૿ઃ 12-20 core ͳCPU x 2ʢHyper-ThreadingࢉͰ24-40
coreʣͷϚγϯ͕جຊߏ • ΫΤϦվળΩϟογϡରԠ ioDrive(ioMemory)͔ΒSSDͷεέʔϧμϯͷ࣮ݱ • Memcachedͷ૿ઃ 27
Memcachedͷߏ 28
Memcachedͷߏ • Memcached2ϓʔϧߏ • 1ϓʔϧ͋ͨΓ100 • Ϛγϯ1͋ͨΓͷϝϞϦׂΓͯ26GBʢϝϞϦ32GBࡌͷϚγϯʣ • 1ϓʔϧશମͰ2.6TBͷ༰ྔΛͭ •
྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores • ϝϯςφϯεແ͠ͰϓʔϧͷΓସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ 29
2ϓʔϧߏͷMemcached ೖΕସ͑ํ๏ 30
31 App M App App M M M M M
M M M M M M set get set Pool 1 Pool 2 App App App M M M M M M set get Pool 1 Pool 2 ௨ৗ࣌ ఀࢭɾ૿ઃ M M શΫϦΞ͓ͯ͘͠ M M M M M M Pool2ͷsetΛఀࢭ M Memcachedϗετ
32 App M App App M M M M M
set get set Pool 1 Pool 2 Γସ͑ M M M M M M M M App App App set get Pool 1 Pool 2 Ωϟογϡ͋ͨͨΊ set M M M M M M M M M M M M M M ͠Β͋ͨͨ͘ΊΔ
ϩʔυόϥϯαʔ 33
ϩʔυόϥϯαʔ • ApplicationαʔόͷલஈΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻ • A10 Networks AX2500 • RESTful APIʹΑΔૢ࡞͕Մೳ
• ࣗࣾ։ൃͷGoͷCLIπʔϧʢa10-cliʣ • ApplicationαʔόͷՃɺআ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞτ͕Ͱ͖Δ 34
35 nginx unicorn nginx unicorn nginx unicorn nginx unicorn MemcachedMariaDBͳͲ
Internet A10 ϩʔυόϥϯαʔ Applicationαʔό
ϞχλϦϯάɾࢹ 36
37 CloudForecast • ͯ͢ͷαʔόͷϝτϦΫεऔಘ • 1,000ͷαʔό͕ܭଌର • ༷ʑͳಠࣗͷϝτϦΫεΛՃ https://github.com/kazeburo/cloudforecast
38 GrowthForecast • DCϥοΫͷిྗͷάϥϑԽ • Fluentd Monitoring AgentͷάϥϑԽ http://kazeburo.github.io/GrowthForecast/
39 • ͯ͢ͷαʔόͷࢹ • 12,000Λ͑Δࢹ߲ • ಠࣗͷϓϥάΠϯΛՃ • PagerDutyͱͷ࿈ܞ •
൪ͷεέδϡʔϦϯά • ΤεΧϨʔγϣϯϙϦγʔ • Slackͷ௨ https://www.nagios.org/
40 • nginx, ApplicationͷϩάΛ FluentdΛͬͯElasticsearchసૹ • APIຖͷॲཧ࣌ؒϨεϙϯελΠϜͷܭଌ • Τϥʔසͷܭଌ •
ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ • Slow QueryͷՄࢹԽ • CloudTrailϩάͷՄࢹԽ https://www.elastic.co/
41 nginx unicorn Amazon S3 GrowthForecast LTSV Log Application Servers
ϩάసૹϑϩʔ Monitoring Agent Response Time Status Code Monitoring Agent Log Log td-agent 2.3.x Max 500Mbps Max 500Mbps
ৗͷπʔϧ 42
43 • ίϛϡχέʔγϣϯSlackͰ౷Ұ • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ • ։ൃڥͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ •
ৄ͘͠Software Design 20161݄߸ͷChatOpsಛूʹ http://gihyo.jp/magazine/SD/archive/2016/201601 • ΞϓϦέʔγϣϯίʔυΠϯϑϥߏங༻ͷCookbookͳͲ ͯ͢GitHub্Ͱཧ • pull requestsϕʔεͷ։ൃɺϨϏϡʔ • GitHub্Ͱͷ׆ಈSlack௨͞ΕΔ • GitHubͰpull requests͕࡞͞ΕΔͱࣗಈςετ࣮ߦ • Dockerίϯςφͷੜɺςετ࣮ߦɺഁغ • Hubotͱͷ࿈ܞ
͜Ε͔Β͍͖͍ͬͯͨ͜ͱ 44
• ίʔυͷվળɺεέʔϧΞτɺΩϟογϡར༻ͳͲ • εέʔϧΞοϓͰ͖Δ͚ͩΓͨ͘ͳ͍ • ݁Ռతʹεέʔϧμϯʹͭͳ͕Δͱϕετ • ͜ΕΒैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ • ιϑτΣΞͷܧଓతΞοϓσʔτ
• Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣͷҠߦ • KernelɺϛυϧΣΞͷΞοϓσʔτੵۃతʹ࣮ࢪ 45
• MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋ΕΓ͍ͨ • ࢹɺϞχλϦϯάڥͷվળ
• ෛՙҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ • ͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ • Consulͱͷ࿈ܞ 46
• ࣾͷ৽نҊ݅ͰHashicorpࣾπʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ • Ϟϯετʹޙ͔Βಋೖ͢ΔͷίετϦεΫ͕ߴ͍ͷͰಋೖ༧ఆͳ͠ • ෦తʹπʔϧར༻ʹΑΔޮԽ͓͜ͳ͍͖͍ͬͯͨ • ΠϯϑϥςετͷޮԽ •
DockerServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠ • Chef CookbookͷϦϑΝΫλϦϯάɺͬͱγϯϓϧʹ 47
࠷ޙʹ 48
ΤϯδχΞืूதʂ 49 http://xflag.com/recruit/career/engineer.html
͋Γ͕ͱ͏͍͟͝·ͨ͠ 50
More Decks by Isao Shimizu
See All by Isao Shimizu
みてねにおけるAI-DLC導入活動とAIドリブン開発の現在地/JAWS-UG AI-DLC #2
isaoshimizu
4
2.5k
フルAIで個人開発して学んだあれこれ / yuruai vol.1
isaoshimizu
0
360
Notion x ポストモーテムで広げる組織の学び / Notion x Postmortem
isaoshimizu
1
420
New Relicを活用したSREの最初のステップ / NRUG OKINAWA VOL.3
isaoshimizu
3
1.2k
「家族アルバム みてね」における運用管理・ オブザーバビリティの全貌 / Overview of Operation Management and Observability in FamilyAlbum
isaoshimizu
5
4.8k
約10年間MIXIのインフラを 支えてきたPagerDutyの活用事例 / PagerDuty on Tour 2024
isaoshimizu
6
1.4k
家族アルバム みてねにおけるGrafana活用術 / Grafana Meetup Japan Vol.1 LT
isaoshimizu
2
2k
家族アルバム みてねで直面してきた技術的負債 / MIXI KAG 2024
isaoshimizu
18
9.3k
今年1年のEKS運用振り返り/3-shake SRE Tech Talk
isaoshimizu
2
490
Other Decks in Technology
See All in Technology
現地速報! Microsoft Ignite アップデートレポート
kasada
0
380
要件定義をAIに委任!Claude Codeプラグインで作るPdM × AgentTeams
puku0x
0
120
OWASP AI Agentic Top10に準拠するクラウドネイティブなガードレールの実践(CNDW 2026)
keitah
1
380
マネジメントされる側の仕事は「文句」を言うこと
riyaamemiya
7
5.9k
IoTのデータ保存をSREの観点で考える
grimoh
1
110
ぼくのClaude Codeの使い方
falken
PRO
5
7.6k
その修正で本当に速くなったのか? k6とOpenTelemetryで確かめる性能改善 / Performance review with k6 and OpenTelemetry
ymotongpoo
3
370
Terraform 0 to 100:100個以上のリポジトリを常に最新にし続けるための自動化
sue445
2
1.2k
AIエージェントのアーキテクチャ設計
nsatoh
0
380
Del vibe coding a un Desarrollo (agéntico) fiable
jcabot
0
190
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
620
AIで実装が加速したのに、 なぜ私は仕事の大半で人と話し続けていたのか?
say_no_w
0
210
Featured
See All Featured
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
helenjbeal
2
450
Learning to Love Humans: Emotional Interface Design
aarron
275
41k
Public Speaking Without Barfing On Your Shoes - THAT 2023
reverentgeek
1
670
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
inesmontani
PRO
3
2.5k
sira's awesome portfolio website redesign presentation
elsirapls
0
520
Claude Code どこまでも/ Claude Code Everywhere
nwiizo
68
61k
Odyssey Design
rkendrick25
PRO
3
960
Bootstrapping a Software Product
garrettdimon
PRO
306
120k
Bash Introduction
62gerente
615
220k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
31
3.4k
Utilizing Notion as your number one productivity tool
mfonobong
4
650
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
38
3.1k