Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018...
Search
takipone
October 05, 2018
Technology
7
3.1k
AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra
2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料
takipone
October 05, 2018
Tweet
Share
More Decks by takipone
See All by takipone
OJTに夢を見すぎていませんか? ロールプレイ研修の試行錯誤/tryanderror-in-roleplaying-training
takipone
1
180
ロールプレイトレーニングでクラウドエンジニアを短期育成した知見を一挙公開 / roll playing training benefits and cases
takipone
0
650
スキル育成に生かすLlamaIndexの活用 /using-llamaIndex-to-enhance-ones-skills
takipone
1
3.4k
DNSにちょっとだけ詳しくなりたい人に贈る少し突っ込んだDNSの話 / devioday1_dns
takipone
2
11k
Amazon VPC Latticeに期待する / look-for-vpc-lattice
takipone
0
1.1k
DevelopersIOのアウトプット文化をハックして人材育成に繋げる話 /deviosapporo2022fall_takipone
takipone
0
970
AWSネットワークリソースをサービスでマルチテナントに 提供するときのスケーラビリティ / nwjaws202112-takipone
takipone
1
750
ネットワークがクラウドに載って変わることと変わらないこと /nwjaws201909-takipone
takipone
0
2.1k
日本🇯🇵でAWS Direct Connect を利用する話/akiba-aws-dx
takipone
1
2.4k
Other Decks in Technology
See All in Technology
CSPヘッダー導入で実現するWebサイトの多層防御:今すぐ試せる設定例と運用知見
llamakko
1
260
株式会社島津製作所_研究開発(集団協業と知的生産)の現場を支える、OSS知識基盤システムの導入
akahane92
1
1.3k
Step Functions First - サーバーレスアーキテクチャの新しいパラダイム
taikis
1
280
MCPと認可まわりの話 / mcp_and_authorization
convto
2
290
Jitera Company Deck / JP
jitera
0
250
Wasmで社内ツールを作って配布しよう
askua
0
150
ObsidianをLLM時代のナレッジベースに! クリッピング→Markdown→CLI連携の実践
srvhat09
7
9.7k
手動からの解放!!Strands Agents で実現する総合テスト自動化
ideaws
3
390
データエンジニアがクラシルでやりたいことの現在地
gappy50
3
660
P2P ではじめる WebRTC のつまづきどころ
tnoho
1
270
AI人生苦節10年で会得したAIがやること_人間がやること.pdf
shibuiwilliam
1
210
Expertise as a Service via MCP
yodakeisuke
1
160
Featured
See All Featured
BBQ
matthewcrist
89
9.8k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
What's in a price? How to price your products and services
michaelherold
246
12k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
760
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Balancing Empowerment & Direction
lara
1
510
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
The Language of Interfaces
destraynor
158
25k
Visualization
eitanlees
146
16k
Transcript
"84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங େ୍ོଠ
DNEFWJP
ࣗݾհ /BNF େ୍ོଠ!UBLJQPOF +PC 43&!QSJTNBUJY *OUFSFTU ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF
"NB[PO3PVUFͱ"$.
ΞδΣϯμ "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ ϩάੳج൫ͷઃܭ
ͭ·͍͍ͮͨͯΔͱ͜Ζ ࠓޙͷల
1. AWSωΠςΟϒͳ EC/CRMγεςϜ prismatixͱ
QSJTNBUJY ҎԼ1[ ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠ ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛݮ͠·͢ɻ
ߏྫ prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ
ࣄྫύϧί༷
"84ωΠςΟϒͳΠϯϑϥߏ ϩʔυόϥϯα (ෛՙࢄ/TLSऴ) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)
ϚΠΫϩαʔϏεؒͷ࿈ܞ ඇಉظૄ݁߹ͷ͘͠Έ αʔϏε Amazon SNS (௨) Amazon SQS
(δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ొ Πϕϯτૹ৴ Πϕϯτ͕ Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠ ࡏݿσʔλʹө ΠϕϯτΛసૹ
ϚΠΫϩαʔϏεؒͷ࿈ܞ αʔϏεͷϦιʔε૿ݮ εέʔϧΠϯΞτ ͍͢͠ αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ
2. ϩάੳج൫ͷઃܭ
QSJTNBUJYͷϩά ओʹछྨɺ͍ͣΕ+40/ܗࣜ ✦ ΞϓϦϩά 4QSJOH +BWB ىಈ࣌Τϥʔͷϝοηʔδ
✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦ ͋ͱεϩʔΫΤϦϩάͳͲ
ϩάͷ༻్ ✦ τϥϒϧγϡʔςΟϯά ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ ‣ ݕࡧจͱͷಥ͖߹Θͤ ✦
ϩάੳ ‣ όʔήϯηʔϧ4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍
ϩάج൫ͷઃܭࢥ ✦ ͦΕͳΓʹେྔ݄ؒʙ5# ✦ 410' 4JOHMF1PJOUPG'BJMVSF Λආ͚͍ͨ ✦
ͳΔ͘༗ΓΛ͍͍ͨ ✦ ༻్ʹΑͬͯҟͳΔಛੑ ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷΈ߹Θͤ
ϩάج൫ͷߏਤ Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch
Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩάੳ)
"NB[PO&$4ίϯςφΫϥελ ✦ %PDLFSίϯςφΛཧ͢ΔϚωʔδυαʔϏε ‣ ίϯςφΛͲͷΠϯελϯε ԾϚγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹཧ
‣ QSJTNBUJYͰ'BSHBUFະ࠾༻ ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOHESJWFSػೳ͕͋Δ ‣ ίϯςφϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ ECSΫϥελ
"NB[PO4σʔλϨΠΫ ✦ ΦϯϥΠϯετϨʔδαʔϏε ‣ ߴ͍ٱੑ ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ
‣ ྿ՁɺબΔετϨʔδΫϥε ‣ σʔλੳαʔϏεͱͷ࿈ܞ
"NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά ✦ ϚωʔδυͷϩάอଘαʔϏε ‣ ߴ͍εέʔϥϏϦςΟ ‣ ४ϦΞϧλΠϜࢀর ‣
+40/ͷΫΤϦΛαϙʔτ
"SN5SFBTVSF%BUBϩάੳ ✦ σʔλੳͷ4BB4αʔϏε ඇ"84 ‣ ඇߏԽσʔλʹૉૣ͘ΞΫηεͰ͖Δ ετϨʔδ
1MB[NB%# ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ ΫΤϦͰ͖Δ
༗Γʹݶք͋Γʢʣ Amazon ECS Amazon S3 CloudWatch Logs MPHHJOHESJWFSͷ੍
⭕ $MPVE8BUDI-PHTΛαϙʔτ ❌ 4ʹ͚ΒΕͳ͍ ❌ ෳग़ྗʹະରԠ ◦ ×
༗Γʹݶք͋Γʢʣ Amazon S3 Treasure Data %BUB$POOFDUPSͱ͍͏5SFBTVSF %BUBͷΠϯϙʔτػೳ͕͋Δ ⭕
εέδϡʔϧػೳ͕͋Γɺ࣍ͷ Πϯϙʔτ͜ΕͰ0, ❌ Πϯϙʔτ࣌ͷϦιʔεಉ࣮࣌ߦ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹن ײ͕߹Θͳ͍ 044ͷ&NCVMLͱ%JHEBH͕தͰ ಈ͍͍ͯΔ
ෆΛิ͏ͨΊʹ044Λར༻ Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒॊೈͰ๛ͳϓϥάΠϯΤίγεςϜ͋Γ ͨ·ͨ·5SFBTVSF%BUBͰ5%ͱͷߴ͍ੑ
%PDLFSº'MVFOUE %PDLFSMPHHJOHESJWFS͕'MVFOUEͷ ૹ৴Λαϙʔτ ‣ 'MVFOUEࣗମ%PDLFSίϯςφͱͯ͠ &$4ͷ֤ΠϯελϯεͰ࣮ߦ ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β
ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd ίϯςφ
'MVFOUEͷෳ0VUQVUͱଟஈߏ 4ͱ$MPVE8BUDI-PHT ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match
docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>
'MVFOUEͷෳ0VUQVUͱଟஈߏ ✦ ϦΞϧλΠϜϩά"HHSFHBUPS ूαʔόʔ Λ Ճͨ͠ଟஈߏ ‣ ͋ͱ͔ΒৼΓઌΛม͍͑͢Α͏ʹ
‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍ରࡦ CloudWatch Logs Aggregator Forwarder
'MVFOUEͷσʔλՃ ✦ 'PSXBSEFS ‣ +40/ͷύʔε ‣ ΞϓϦϩάͱ.%$ϩάͷ ͚۠
‣ 4ͷύεϓϨϑΟοΫε ✦ "HHSFHBUPS ‣ Τϥʔͷநग़ ‣ $MPVE8BUDI-PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>
&NCVML ✦ 4ˠ5%ͷॳظόονΠϯϙʔτ ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘ ‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦͍ճͤͨ
Embulkͷฒྻ࣮ߦ Ͳ͏͢Δʁ
&NCVMLPO"84#BUDI ✦ "84#BUDIͷδϣϒΩϡʔ εϙοτΠϯελϯε׆༻ ‣ δϣϒͷ༰%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ ✦
&NCVMLͷ%PDLFSΠϝʔδΛ ༻ҙ࣮ͯ͠ߦ ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch
&NCVMLͷίϯϑΟάڥมΛଟ༻ exec: max_threads: {{ env.MAX_THREADS }} in: type:
s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}
3. ͭ·͍ͮͨ/͍ͯΔ ͱ͜Ζ
Fluentd Aggregatorͷ Խ/εέʔϧΞτ CloudWatch Logs Aggregator Forwarder
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ෳͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍ ‣ $MPVE8BUDI-PHTͰಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖TFRVFODFUPLFOΛҡ࣋͠ͳ͚ΕͳΒͳ͍ <match **>
@type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍ ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓͯΔ ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match
**> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS ͷίϯςφ*%
%PDLFSͷϗετϙʔτͷ੍ ✦ "HHSFHBUPS%PDLFSίϯςφ Ͱ࣮ߦ ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ༗ ‣
Πϯελϯεʹίϯςφ͔͠ ࣮ߦͰ͖ͳ͍ ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ Ҡߦ NLB Aggregator ίϯςφ
TDͷΠϯϙʔτͷδϣϒׂ
Πϯϙʔτͷδϣϒׂ ✦ Πϯϙʔτର4ϓϨϑΟοΫεΛ͘औΔͱ ϝϞϦΊͬͪΌ͏ ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞݄࣌ ‣
"84#BUDIϝϞϦ্ׂݶ·Ͱͬͯ00.,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz
Πϯϙʔτͷδϣϒׂ ✦ ͍Ζ͍Ζࢼͯ͠ɺ୯ҐͰδϣϒΛΔͷ͕ ྑͦ͞͏ͱ͍͏͜ͱʹ ‣ ϑΝΠϧαΠζɺϑΝΠϧʹґଘ͢ΔͷͰͬͯΈͳ ͍ͱΘ͔Βͳ͍෦ ࠓճϑΝΠϧ͕ଟ͔༷ͬͨ
in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}
5SFBTVSF%BUBͷ'"2ʹهࡌ͋Γ 2 %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH UJNF XIBUDBO*EP " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS
KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF QFSGPSNBODFEFHSBEFT 5PNJUJHBUFUIJTJTTVF ZPVDBO ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT ‣ 4FU .# UPNJO@UBTL@TJ[FPQUJPO IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS
ϩάαΠζͷ্ݶ
ϩάαΠζͷ্ݶ ✦ .%$ϩά͕σΧ͍ ✦ %PDLFSMPHHJOHESJWFS ,#ΑΓେ͖͍ϩάΛׂ͢Δ ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ
‣ DPODBUϓϥάΠϯͰ݁߹ ‣ ϝϞϦ༻ྔ͕Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί ✦ $MPVE8BUDI-PHTͷ্ݶ,#
͓ۚ ✦ $MPVE8BUDI-PHTϩάσʔλྔͷैྔ՝ۚ ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍
✦ ΞϓϦ͔Βग़͢ϩάͷྔΛݮΒ͢ ‣ .%$ϩάͭΒ͍
4. ࠓޙͷల
࣍ͷҰख ✦ ϩάࢹ ✦ ϩάूͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻ ✦ 5SFBTVSF%BUBΛͬͱ׆༻͍ͨ͠
·ͱΊ ✦ εέʔϧ͢Δϩάཧج൫ΫϥυαʔϏεͷ Έ߹Θ͕ͤΦεεϝ ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ͍͚ ✦ Έ߹Θ͚ͤͩͰಈ͘ͷͰ͖Δ͚Ͳ
‣ ཁ݅ʹରͯ͠Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ ‣ σʔλྔɺαΠζͷධՁνϡʔχϯά͕ඞཁ