Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ゲンバのサービス運用
Search
cm-komuro
May 27, 2017
Technology
2
1.3k
ゲンバのサービス運用
【 大阪オフィス開設1周年勉強会 】開発者のための楽しいAWSのサービス運用 in 大阪 2017/05/26 #cm_osaka
cm-komuro
May 27, 2017
Tweet
Share
More Decks by cm-komuro
See All by cm-komuro
How To 脆弱性対応
cmkomuro
0
620
内容は話せないけどGamedayのススメ
cmkomuro
0
460
運用事件簿
cmkomuro
0
2.4k
Introduction of Jooby. DevIO2016
cmkomuro
0
1.2k
Other Decks in Technology
See All in Technology
CockroachDB はどのくらい「しぶとい」のか? / How tough is CockroachDB?
kota2and3kan
4
1.1k
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
4
940
成長をサポートするピープルマネジメントのやり方
sioncojp
9
1.4k
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
150
モーダル間の変換後の一致性とジャンル表を用いた解釈可能性の考察 ~Text-to-MusicとText-To-ImageかつImage-to-Musicを例に~
otanet
0
320
Python と Snowflake はズッ友だょ!~ Snowflake の Python 関連機能をふりかえる ~
__allllllllez__
2
150
一生覚えておきたい「システム開発=コミュニケーション」〜初めての実務案件振り返りLT〜
maimyyym
3
410
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
350
Amplify 🩷 Bedrock 〜生成AI入門〜
minorun365
PRO
9
1.1k
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
8
670
GrafanaMeetup_AmazonManagedGrafanaのアクセス制御機能とマルチテナント環境下でのアクセス制御について
daitak
0
450
Gemini, Google's Large Language Model
glaforge
0
120
Featured
See All Featured
From Idea to $5000 a Month in 5 Months
shpigford
378
45k
Thoughts on Productivity
jonyablonski
60
3.9k
Designing with Data
zakiwarfel
96
4.8k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
12
1k
What’s in a name? Adding method to the madness
productmarketing
PRO
17
2.7k
Being A Developer After 40
akosma
67
580k
Documentation Writing (for coders)
carmenintech
60
4k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
How to name files
jennybc
65
93k
Become a Pro
speakerdeck
PRO
13
4.6k
GraphQLとの向き合い方2022年版
quramy
33
12k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
275
13k
Transcript
ήϯόͷαʔϏεӡ༻ ָ͍͠AWSͷαʔϏεӡ༻
͡Ίʹ ࣗݾհ ▸ খࣨ ܒʢ͜ΉΖɹͻΒ͘ʣ@com4dc ▸ ϞόΠϧΞϓϦαʔϏε෦όοΫΤϯυνʔϜ ▸ Java, Scala,
Pythonͱ͔ ▸ ࡛ۄੜ·Εࡳຈࡏॅ ▸ ࡳຈΦϑΟεॴଐ ▸ ํ։ຽ
͡Ίʹ ࡳຈ͔Βདྷ·ͨ͠ ▸ Ҡಈڑ1600km͘Β͍ ▸ 10℃͘Β͍ؾԹ͕ࠩ༗Γ·͢(ॵ͍)
None
ؓٳ
͡Ίʹ
ӡ༻ ͖Ͱ͔͢ʁ
ਖ਼ॳ͋·Γڵ ຯ͕͋Γ·ͤΜͰ͠ ͨ
ͦΜͳ։ൃऀ͕ӡ༻ Λܦݧͯͬͨ͠ ήϯόͷ
ӡ༻ʹର͢Δ BEFORE AFTER
ӡ༻ʹ͍ͭͯ ॳͷӡ༻Πϝʔδ ▸ ໘ന͘ͳ͍ ▸ ։ൃ͕ग़དྷͳ͍ ▸ ো͕༧ଌͰ͖ͳ͍ʢવ͚ͩͲʣ ▸ τϥϒϧγϡʔςΟϯάͰϐϦϐϦ
▸ ݮํࣜ ▸ ౖΒΕΔ͜ͱ͋ͬͯ๙ΊΒΕΔ ͜ͱ͕গͳ͍
ӡ༻ʹ͍ͭͯ ࠓͷӡ༻Πϝʔδ ▸ ໘ന͍ʢୀ۶͠ͳ͍ʣ ▸ ৭ʑͱ։ൃͯ͠Δ ▸ ো༧ଌͰ͖ͳ͍͚ͲɺݪҼಛఆɺ ରԠ͢Δͷָ͍͠ɻ಄͏͠ ▸
͖ͪΜͱઆ໌͢Εೲಘͯ͘͠ΕΔ ͓٬༷ͳͷͰͱͯ৴པͰ͖Δ ▸ ͖ͪΜͱධՁͯ͠Β͑Δ
ӡ༻ʹ͍ͭͯ AWSΛ͔ͳΓϋʔυʹ͏ ▸ ୭ฉ͍ͨ͜ͱݟͨ͜ͱແ͍ࣄαʔϏεͷ݀Λ౿ Ή͜ͱ͕͋ΔʢάάͬͯԿ͔Βͣʣ ▸ AWSΤϏσϯεͱͱʹΨϯΨϯϑΟʔυόοΫ ▸ Βͳ͍ະͷੈքΛܦݧ͍ͯ͠Δ ▸
→ɹָ͍͠
ӡ༻ʹ͍ͭͯ ։ൃͱӡ༻ͷڱؒ ▸ ։ൃͷࣝͰෆे ▸ αʔϏε͕ͲͷΑ͏ʹӡ༻͞ΕͯՁ ΛੜΉ͔ͷ૾ྗ͕Βͳ͍ ▸ ࣮ࡍʹࣗͰ͏Θ͚Ͱͳ͍ͨΊɺ ϩάͷઃܭ͕͍
▸ ։ൃͱӡ༻ΛߦͬͨΓདྷͨΓ͍ͯ͠Δ ͱɺํͷཱ͔Β࣮ରԠΛධՁ ͢Δ͜ͱ͕ग़དྷΔ ▸ ಄ͷContext SwitchසൟʹΓସ͑Δ
ӡ༻ʹ͍ͭͯ ӡ༻Λܦݧͯ͠ॏཁੑΛೝࣝͨ͜͠ͱ ▸ ϩά ▸ োݕ ▸ ઃܭ ▸ ӡ༻Λࢧ͑ΔͨΊʹ։ൃऀ͕ग़དྷΔ͜ͱ
▸ ίϛϡχέʔγϣϯ
AWSΛར༻ͨ͠ γεςϜӡ༻
AWSΛར༻ͨ͠γεςϜӡ༻ ୲γεςϜ ▸ ECαΠτͷೝূج൫ٴͼϢʔβʔཧ ▸ Ϣʔβʔ300ສʙ ▸ Ωϟϯϖʔϯ࣌ʹ࠷େؒʙ5ສΞΫηεఔ ▸ ಛతͳAWSαʔϏε
▸ EB docker, DynamoDB Streams, Lambda ▸ Java8+SpringBoot, Scala+Play
AWSΛར༻ͨ͠γεςϜӡ༻ ϓϩδΣΫτΛߏ͢ΔେྔͷαʔϏε
None
AWSΛར༻ͨ͠γεςϜӡ༻ ͨ͘͞ΜͷϝϦοτ ▸ ϚωʔδυαʔϏεΛར༻͢Δ͜ͱͰϝϯςφϯεϑϦʔ ▸ ͍͍ͨαʔϏεΛʢେମʣ͙͢ར༻Ͱ͖Δ ▸ ͲͷαʔϏεʢ֓Ͷʣ؆୯ʹεέʔϧͰ͖Δ ▸ ෛՙͷنʹԠͯ͡εέʔϧΞτɺεέʔϧΠϯͰ͖Δͨ
Ίɺίετ࠷దԽ ▸ ো࣌ͷ෮چɾରࡦૉૣ͘Ͱ͖Δ
AWSΛར༻ͨ͠γεςϜӡ༻ AWSಛ༗ͷӡ༻ઃܭ࣌ʹߟྀ͖͢ࣄ߲ ▸ AWSϚωʔδυαʔϏεͷϩάݶఆత ▸ োͷݕϝτϦΫεͱΞϥʔϜͷઃఆ࣍ୈ ▸ ϩάશͯߴ৴པੑͷαʔϏεूͤ͞Δ ▸ ϚωʔδυαʔϏεͱͷ伱ؒΛͲͷΑ͏ʹࢹ͢Δ͔
▸ 100%Քಇͷ1Λࢦ͢ͷͰͳ͘ɺ90%ΛෳଋͶͯ τʔλϧͰ100%Λࢦ͢ͱ͍͏ߟ͑ํ
AWSΛར༻ͨ͠γεςϜӡ༻ ΦϯϓϨӡ༻ͱجຊతʹಉ͡ ▸ োඞͣൃੜ͢Δ ▸ ԿʹͦͷোΛίϯτϩʔϧԼʹ͓͚Δ͔͕ͱͯେࣄ ▸ ϩάʂͳʹͱ͋Εϩάࢮकʂʂ
ϩάͷॏཁੑ
ϩάͷॏཁੑ ϩάݪҼڀ໌ʹඞਢ ▸ ϩά͕ͳ͍ͱݪҼڀ໌͕શ͘ग़དྷͳ͍ ▸ ϩά͕ଘࡏ͠ͳ͍ݱଘࡏ͠ͳ͍ͷͱಉ͡ ▸ ͨͩ͠ɺϊΠζͩΒ͚ͷϩάҙຯͳ͍ɻϩάͱͯ͠ҙຯ ͷ͋ΔใΛग़ྗ͢Δɻ ▸
ϩάઈରʹࢮकʂফͨ͠ΒऴΘΓͱࢥ͏͠
ϩάͷॏཁੑ ϩάશͯߴ৴པੑͷαʔϏεूͤ͞Δ ▸ EC2Πϯελϯε͍ͭফࣦ͢Δ͔͔Βͳ͍(ઃఆʹΑΔ ͚Ͳ) ▸ ඞཁͳϩάߴ৴པੑͷετϨʔδαʔϏεूͤ͞Δ ▸ EC2Λ͡Ίɺ༰қʹScale Out,
Scale In͢Δͷ͕ಛ ▸ αʔϏεͷ෦ʹσʔλϩάΛ͓ͯ͘͜͠ͱɺϩετ ͷةݥੑ͕͋Δ͜ͱΛೝࣝ͢Δ
ϩάͷॏཁੑ EC2ʹϩά͕ͬͯ͠·͏ͱϩετ͢Δ ASG EC2 EC2 EC2 ϩά ϩά ϩά ASG
EC2 EC2 EC2 ϩά ϩά ϩά
ϩάͷॏཁੑ ϩάͷूͷߏ EC2 ELASTIC BEANSTALK FLUENTD + CWLOGS CWLOGS AGENT
CLOUDWATCH LOGS EC2 FLUENTD + ES
ϩάͷॏཁੑ ར༻͍ͯ͠ΔϩάετϨʔδ ▸ Cloudwatch Logs: ΞϓϦέʔγϣϯϩάΛूɻো࣌ ਅͬઌʹ͜͜Λ֬ೝ͢Δɻ ▸ Amazon ES:
ϩάΠϯཤྺ, ϝʔϧςϯϓϨʔτͳͲͷૢ࡞ ϩάͳͲΛूɻ ▸ Amazon S3: CloudTrailͷϩάͳͲ͕֨ೲɻϑΝΠϧͰͷอ ଘʹͳΔͨΊݕࡧੑ͍ɻ࠷ऴखஈʂʢ࠷ۙAthena͕ग़ ͨͨΊΫΤϦͰͷݕࡧੑ্͕ͬͨʣ
ϩάͷॏཁੑ ϩάͰେࣄͳ͜ͱ ▸ ଘࡏͤ͞Δ͜ͱɻదͳϥΠϑαΠΫϧ͕ऴΘͬͨϞϊҎ֎ ઈରࢮकɻ ▸ ݕࡧཁ݅ʹԠͯ͡ετϨʔδઌΛબఆ͢Δ͜ͱɻසൟʹݕ ࡧ͕ඞཁͳϩάΛݕࡧੑͷ͍ετϨʔδʹஔ͢Δͱޙʑ ۤ࿑͢Δɻ ▸
τϨʔαϏϦςΟɻ͜Εӡ༻ଆ͕ؾΛ͚ͭΔ͜ͱͱ͍͏Α Γ։ൃଆ͕ߟྀ͖͢͜ͱɻޙड़
োΛ ݕ͢Δ
AWSΛར༻ͨ͠γεςϜӡ༻ োͷݕϝτϦΫε࣍ୈ ▸ AWS༷ʑͳϝτϦΫεΛఏڙͯ͘͠Ε͍ͯΔ ▸ ͲͷϝτϦΫεΛࢹ͢Δ͔Ϣʔβʔ࣍ୈ ▸ ϝϞϦ༰ྔͳͲҰ෦ࣗͰΧελϜ͠ͳ͍ͱࢹͰ͖ͳ͍ ▸ ֤αʔϏεʹΑͬͯࢹ߲มΘΔʢมΘΒͳ͍߲͋Δ͚Ͳʣ
AWSΛར༻ͨ͠γεςϜӡ༻ ݕઃఆෆඋʹΑΔࣄނ ▸ EC2ͷΞϓϦέʔγϣϯϩάΛAgentܦ༝ͰCloudwatch LogsϦΞϧλΠϜసૹ͍ͯ͠Δঢ়ଶ
AWSΛར༻ͨ͠γεςϜӡ༻ োͰసૹ͕ఀࢭ ▸ సૹ͕ྃ͠ͳ͍ͨΊɺϩʔΧϧϩά͕ཷ·Γ࢝ΊΔ ʢ.tmpϑΝΠϧʣ
AWSΛར༻ͨ͠γεςϜӡ༻ ͍ͣΕEC2σΟεΫϑϧʹ ▸ σΟεΫϑϧͰαʔϏε͕ఀࢭʂʂˠ ൃ֮
AWSΛར༻ͨ͠γεςϜӡ༻ దͳࢹϝτϦΫεΛઃఆ͢Δ ▸ EC2ͷσΟεΫ༻Λܭଌ͢ΔϝτϦΫε͕ඞཁ ▸ ͞ΒʹྔʹԠͯ͡ΞϥʔϜΛઃఆ͢Δ ▸ σΟεΫϑϧʹΑΔαʔϏεఀࢭͷඃ͜ΕͰࣄલʹ ͢Δ͜ͱ͕Ͱ͖Δ $
df -H Filesystem Size Used Avail Use% Mounted on /dev/xvda1 11G 11G 0 100% / devtmpfs 4.0G 119k 4.0G 1% /dev tmpfs 4.0G 0 4.0G 0% /dev/shm
AWSΛར༻ͨ͠γεςϜӡ༻ ೋಉ͡োى͜͞ͳ͍ ▸ ಉ͡োى͜͞ͳ͍Α͏ʹɺݪҼڀ໌ͱࢹͷڧԽ ͢ΔɻKPTେࣄɻ ▸ ࠜຊతղܾʹࢸΒͳͯ͘ؾͮ͘͜ͱେࣄ ▸ ଞʹߟ͑ΒΕΔͷCloudwatch Logsͷసૹ͕ఀࢭͯ͠
͍Δ͜ͱΛݕ͢Δ͜ͱͰ͖ͳ͔ͬͨͷ͔ʁͱ͔ ▸ ͨͩ͜Ε໌֬ͳج४ͱͳΔϝτϦΫε͕Θ͔Βͳ͍ʢԿ Λܭଌ͢Ε͍͍ͷʁʣ
ӡ༻ͷͨΊͷ ։ൃ৺ಘ
։ൃऀͲ͏͢Δ͔ ӡ༻Λݟਾ͑ͨઃܭΛࢦ͢ ▸ ϩάઃܭʢτϨʔαϏϦςΟɺϩάͷཻɺใʣ ▸ ϩάϨϕϧਵ࣌ݟ͢ ▸ ϚωʔδυαʔϏεͷར͚ͩʹ͢ΔͷͰͳ͘ɺΤ ϥʔɺϝϯςφϯε࣌ͷಈ࡞ߟྀ͢Δ ▸
ൺֱతݹ͔͘Β͋Δ҆ఆͯ͠ΔαʔϏεΛબͿ͜ͱେ ▸ ϚωʔδυαʔϏεͷ伱ؒΛߟྀͨ͠ઃܭ
։ൃऀͲ͏͢Δ͔ ։ൃऀ͕ߟྀ͖͢ϩάઃܭࣄ߲ ▸ ϩάͷΈͰૢ࡞ΛτϨʔεͰ͖Δ͔ʢτϨʔαϏϦςΟʣ ▸ ग़ྗ༰ɻϚεΫ͖߲͢ͷཧ ▸ ϩάϨϕϧͷઃܭ Ұ൪ॏཁͳ͕ͣͩɺ։ൃ࣌༏ઌ ͕Ίͳͷ͕࣮
։ൃऀͲ͏͢Δ͔ ΞϓϦέʔγϣϯϩάΛू͢Δ ▸ ࢄ֤ͨ͠αʔόʔ͔ΒϩάΛूͨ࣌͠ʹશ͕ͯόϥόϥʹه͞ΕΔ ▸ ಛఆͷϦΫΤετΛऔΓग़͢ͷࢸͷۀ
։ൃऀͲ͏͢Δ͔ ղܾɿϦΫΤετ͝ͱʹҰҙͳIDΛੜ͢Δ ▸ 1ϦΫΤετʹରͯ͠શͯҰҙͳRequestIDΛ༩͢Δ ▸ ։ൃͨ͠APIαʔόʔɺ΄΅શͯUUIDΛར༻ͯ͠Ұҙͳ RequestIDΛੜ ▸ ࣌ʴRequestIDͰ࣮֬ʹಛఆ͕Ͱ͖Δ ▸
Ұ࿈ͷϦΫΤετͷΠϕϯτΛτϨʔεͰ͖ɺCloudwatch Logs͔Β༰қʹݕࡧ͕Մೳʹɻ
։ൃऀͲ͏͢Δ͔ ϩάৗʹվળ͢Δ ▸ ։ൃ࣌ʹߟྀ͕࿙Ε͍ͯͨՕॴΨϯΨϯվળ͢Δ ▸ ग़ྗ͍͚ͯ͠ͳ͍߲ͷϚεΫ ▸ ϩάϨϕϧࡉ͔͘ௐ͢Δ ▸ ΑΓਫ਼ີͳΞϥʔϜͱͳΓɺຊʹਂࠁͳোͷΈใࠂ
ͤ͞Δ
ϚωʔδυαʔϏ εͷ伱ؒΛߟ͑Δ
AWSΛར༻ͨ͠γεςϜӡ༻ ϚωʔδυαʔϏεͱͷ伱ؒΛͲͷΑ͏ʹࢹ͢Δ͔ ▸ ଟछଟ༷ͳϚωʔδυαʔϏεΛΈ߹ΘͤΔͱࢹ͕ߦ ͖ಧ͔ͳ͍伱͕ؒੜ·ΕΔ ▸ ͜ͷ伱ؒυΩϡϝϯτΛಡΜͰॻ͍ͯ͋ΔΘ͚Ͱͳ͍ ▸ ϚωʔδυαʔϏεͷಛੑΛཧղ͠ɺܦݧΛੵΉଞͳ͍ ▸
ԿΛࣦͬͯഊ͍ͯ͠Δ͔ͱ͍͏ఆྔతͳஅ͕Ͱ͖Δج४ Λ୳͢ʢྫɿ5Ҏ্ॻ͖ࠐΈ͕ͳ͍͜ͱ͋Γ͑ͳ͍ɺ σʔλ͕24hҎ্ಉظ͞Εͳ͍͜ͱ͋Γ͑ͳ͍ɹʣ
AWSΛར༻ͨ͠γεςϜӡ༻ DYNAMODB STREAMSΛͬͨಉظॲཧ HTTP STATUSͰ ݕ εςʔλεΛ ࢹ ىಈ͔ͨ͠Ͳ͏ ͔ݕ
εςʔλεΛ ࢹ ▸ ֤αʔϏεຖʹΤϥʔোͷݕՄೳ ▸ ͜ͷதͰࢹͷ伱ؒͱͳͬͯ͠·͏ͷʁ http://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/UserGuide/CHAP_Monitoring.html DLQ
AWSΛར༻ͨ͠γεςϜӡ༻ DYNAMODB STREAMSΛͬͨಉظॲཧ ͜͜ ▸ DynamoDB Streamsͷγϟʔυɻ͜Εʢࠓͷͱ͜Ζʣ୭ ࢹͰ͖ͯͳ͍
AWSΛར༻ͨ͠γεςϜӡ༻ ઃఆͰ͖ΔϝτϦΫε͕ͳ͍ɾɾʁ σʔλ͕ߋ৽ ▸ ͜ͷγϟʔυΛࢹ͢ΔϝτϦΫε͕ݟ͔ͭΒͳ͍ ▸ LambdaͰͷࢹՄೳʁ => ߋ৽͞Εͯͳ͍ͷ͔ͦΕͱ োͰΩοΫ͞Εͳ͍ͷ͔͔Βͳ͍ɻ
ʁʁʁ ߋ৽͞Εͯͳ͍ʁ ͦΕͱোʁ
AWSΛར༻ͨ͠γεςϜӡ༻ ࣮֬ʹಉظΛͱΔվળࡦ σʔλ͕ߋ৽ WORKER QUEUEΛࢹɻಉظྃ ޙϝοηʔδআ ಉظ֬ೝ ▸ SQSʴWorkerͰಉظΛ֬ೝ͢Δ
AWSΛར༻ͨ͠γεςϜӡ༻ςΩετ ઃܭΛݕূ͢Δ ▸ ඇಉظͰ࣮ߦ͞ΕΔΠϕϯτͷͨΊɺීஈͨ·ͨ·0ඵͰಉظ ͞Ε͍ͯΔ͚ͩʢ࠷େ24࣌ؒͷΕʣ ▸ σʔλͷੑ֨ʹΑͬͯ࠷ऴతʹ⁋᧒͕߹ͬͯΕྑ͍ͷ͋ Δ ▸ γϟʔυίϯςφͷϥΠϑαΠΫϧʢଟʣͰ࡞Γ͞ΕΔͷ
Ͱɺͦͷ͏ͪഁغ͞Ε࣍ͷߋ৽࣌ʹ࣮ߦ͞ΕΔՄೳੑ͕ߴ͍ ▸ ઈରʹಉظ͕ͣΕ͍͚ͯͳ͍σʔλͩͱվળࡦͷΑ͏ͳνΣο Ϋػೳ͕ඞཁʹͳΔ
·ͱΊ
·ͱΊ ӡ༻ΛΔ͜ͱֶͼ͕͋Δ ▸ αʔϏεӡ༻͢Δ͜ͱͰՁ͕ग़Δ͜ͱΛ࣮ײͰ͖Δ ▸ γεςϜΛམͱͣ͞ʹฏԺΛҡ࣋͢Δ͜ͱ͕Կʹ͘͠େ ͳ͜ͱ͕Α͔͘Γɺಈ͚͍͍Έ͍ͨͳ৺ࠜΛୟ͖ ͤ͞ΒΕΔ ▸ ࣗͷ୲֎ͷ༷ʑͳٕज़ʹ৮ΕΔ͜ͱ͕Ͱ͖Δ
▸ ΫϥυͷརͱऑΛΛͬͯΔ͜ͱ͕ग़དྷΔ ▸ ϩάࢮकʂʂʂ
WE ARE HIRING!
MEET UP COMES BACK 2017/07/த େࡕΦϑΟε COMING SOON!