AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料

Transcript

1. "84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங 
   େ୍ོଠ
2. 
   
   
   DNEFWJP
3. ࣗݾ঺հ
    /BNF
    େ୍ོଠ!UBLJQPOF +PC
    43&!QSJTNBUJY *OUFSFTU
    ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF
    "NB[PO
   3PVUF
   ͱ"$.
4. ΞδΣϯμ
     "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ͸
    ϩά෼ੳج൫ͷઃܭ
    ͭ·͍͍ͮͨͯΔͱ͜Ζ
    ࠓޙͷల๬
5. 
    1. AWSωΠςΟϒͳ
 EC/CRMγεςϜ prismatixͱ͸
6. QSJTNBUJY ҎԼ1[ ͸ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠
 ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ
 γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛ࡟ݮ͠·͢ɻ
7. ߏ੒ྫ
    prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ
8. ࣄྫ
   
   ύϧί༷
   
9. "84ωΠςΟϒͳΠϯϑϥߏ੒
    ϩʔυόϥϯα (ෛՙ෼ࢄ/TLSऴ୺) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)
10. ϚΠΫϩαʔϏεؒͷ࿈ܞ
     ඇಉظૄ݁߹ͷ͘͠Έ ঎඼αʔϏε Amazon SNS (௨஌) Amazon SQS (δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ঎඼ొ࿥
    Πϕϯτૹ৴ Πϕϯτ͕
    Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠
 ࡏݿσʔλʹ൓ө ΠϕϯτΛసૹ    
11. ϚΠΫϩαʔϏεؒͷ࿈ܞ
     αʔϏεͷϦιʔε૿ݮ
    εέʔϧΠϯΞ΢τ ͠΍͍͢ ঎඼αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ
12. 
     2. ϩά෼ੳج൫ͷઃܭ
13. QSJTNBUJYͷϩά
     ओʹछྨɺ͍ͣΕ΋+40/ܗࣜ
    ✦ ΞϓϦϩά
    4QSJOH +BWB
    ىಈ࣌΍Τϥʔͷϝοηʔδ
    ✦ .%$ϩά
    .BQQFE
    %JBHOPTUJD
    $POUFYUT
    ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦
    ͋ͱεϩʔΫΤϦϩάͳͲ
14. ϩάͷ༻్
     ✦ τϥϒϧγϡʔςΟϯά
    ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ
    ‣ ঎඼ݕࡧ΍஫จͱͷಥ͖߹Θͤ
    ✦ ϩά෼ੳ
    ‣ όʔήϯηʔϧ΍4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍
15. ϩάج൫ͷઃܭࢥ૝
     ✦ ͦΕͳΓʹେྔ
    
    ݄ؒʙ5#
    ✦ 410' 4JOHMF
    1PJOU
    PG
    'BJMVSF Λආ͚͍ͨ
    ✦ ͳΔ΂͘༗Γ෺Λ࢖͍͍ͨ
    ✦ ༻్ʹΑͬͯҟͳΔಛੑ
    ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷ૊Έ߹Θͤ
16. ϩάج൫ͷߏ੒ਤ
     Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩά෼ੳ)
17. "NB[PO
    &$4
    
    ίϯςφΫϥελ
     ✦ %PDLFSίϯςφΛ؅ཧ͢ΔϚωʔδυαʔϏε
    ‣ ίϯςφΛͲͷΠϯελϯε Ծ૝Ϛγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹ؅ཧ
    ‣ QSJTNBUJYͰ͸'BSHBUF͸ະ࠾༻
    ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOH
    ESJWFSػೳ͕͋Δ
    ‣ ίϯςφ͸ϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ੒ ECSΫϥελ
18. "NB[PO
    4
    
    σʔλϨΠΫ
     ✦ ΦϯϥΠϯετϨʔδαʔϏε
    ‣ ߴ͍଱ٱੑ
    ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ
    ‣ ྿Ձɺબ΂ΔετϨʔδΫϥε
    ‣ σʔλ෼ੳαʔϏεͱͷ࿈ܞ
19. "NB[PO
    $MPVE8BUDI
    -PHT
    
    ϦΞϧλΠϜϩά
     ✦ ϚωʔδυͷϩάอଘαʔϏε
    ‣ ߴ͍εέʔϥϏϦςΟ
    ‣ ४ϦΞϧλΠϜࢀর
    ‣ +40/΁ͷΫΤϦΛαϙʔτ
20. "SN
    5SFBTVSF
    %BUB
    
    ϩά෼ੳ
     ✦ σʔλ෼ੳͷ4BB4αʔϏε ඇ"84
    ‣ ඇߏ଄Խσʔλʹૉૣ͘ΞΫηεͰ͖Δ
 ετϨʔδ 1MB[NB%#
    ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ
 ΫΤϦͰ͖Δ
21. ༗Γ෺ʹ͸ݶք͋Γʢʣ
     Amazon ECS Amazon S3 CloudWatch Logs MPHHJOH
    ESJWFSͷ੍໿
    ⭕ $MPVE8BUDI
    -PHTΛαϙʔτ
    ❌ 4ʹ͸௚઀޲͚ΒΕͳ͍
    ❌ ෳ਺ग़ྗʹະରԠ ◦ ×
22. ༗Γ෺ʹ͸ݶք͋Γʢʣ
     Amazon S3 Treasure Data %BUB
    $POOFDUPSͱ͍͏5SFBTVSF
    %BUBͷΠϯϙʔτػೳ͕͋Δ
    ⭕ εέδϡʔϧػೳ͕͋Γɺ೔࣍ͷ
 Πϯϙʔτ͸͜ΕͰ0,
    ❌ Πϯϙʔτ࣌ͷϦιʔε΍ಉ࣮࣌ߦ਺ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹ͸ن ໛ײ͕߹Θͳ͍
    044ͷ&NCVMLͱ%JHEBH͕தͰ
 ಈ͍͍ͯΔ
23. ෆ଍Λิ͏ͨΊʹ044Λར༻
     Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒ΋ॊೈͰ๛෋ͳϓϥάΠϯΤίγεςϜ͋Γ
    ͨ·ͨ·5SFBTVSF
    %BUB੡Ͱ5%ͱͷߴ͍਌࿨ੑ
24. %PDLFS
    º
    'MVFOUE
     %PDLFS
    MPHHJOH
    ESJWFS͕'MVFOUE΁ͷ ૹ৴Λαϙʔτ
    ‣ 'MVFOUEࣗମ΋%PDLFSίϯςφͱͯ͠
 &$4ͷ֤ΠϯελϯεͰ࣮ߦ
    ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β͸
 ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd
 ίϯςφ
25. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒
     4ͱ$MPVE8BUDI
    -PHT
 ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>
26. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒
     ✦ ϦΞϧλΠϜϩά͸"HHSFHBUPS ू໿αʔόʔ Λ
 ௥Ճͨ͠ଟஈߏ੒
    ‣ ͋ͱ͔ΒৼΓઌΛม͑΍͍͢Α͏ʹ
    ‣ $MPVE8BUDI
    -PHTͷ"1*ίʔϧͷ੍໿ରࡦ CloudWatch Logs Aggregator Forwarder
27. 'MVFOUEͷσʔλՃ޻
     ✦ 'PSXBSEFS
    ‣ +40/ͷύʔε
    ‣ ΞϓϦϩάͱ.%$ϩάͷ
 ۠෼͚
    ‣ 4ͷύεϓϨϑΟοΫε
    ✦ "HHSFHBUPS
    ‣ Τϥʔͷநग़
    ‣ $MPVE8BUDI
    -PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>
28. &NCVML
     ✦ 4ˠ5%΁ͷॳظόονΠϯϙʔτ
    ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘
    ‣ %BUB
    $POOFDUPSͷίϯϑΟά͕Ұ෦࢖͍ճͤͨ
29. 
     Embulkͷฒྻ࣮ߦ͸ Ͳ͏͢Δʁ
30. &NCVML
    PO
    "84
    #BUDI
     ✦ "84
    #BUDIͷδϣϒΩϡʔ
 εϙοτΠϯελϯε׆༻
    ‣ δϣϒͷ಺༰͸%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ
    ✦ &NCVMLͷ%PDLFSΠϝʔδΛ
 ༻ҙ࣮ͯ͠ߦ
    ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch
31. &NCVMLͷίϯϑΟά͸؀ڥม਺Λଟ༻
     exec: max_threads: {{ env.MAX_THREADS }} in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}
32. 
     3. ͭ·͍ͮͨ/͍ͯΔ
 ͱ͜Ζ
33. 
     Fluentd Aggregatorͷ ৑௕Խ/εέʔϧΞ΢τ CloudWatch Logs Aggregator Forwarder
34. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ
     ✦ ෳ਺ͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍
    ‣ $MPVE8BUDI
    -PHTͰ͸ಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖͸TFRVFODF
    UPLFOΛҡ࣋͠ͳ͚Ε͹ͳΒͳ͍ <match **> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε
    ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×
35. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ
     ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍
    ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓ౰ͯΔ
    ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match **> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS
    ͷίϯςφ*%
36. %PDLFSͷϗετϙʔτͷ੍໿
     ✦ "HHSFHBUPS΋%PDLFSίϯςφ Ͱ࣮ߦ
    ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ઎༗
    ‣ Πϯελϯεʹίϯςφ͔͠
 ࣮ߦͰ͖ͳ͍
    ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ
 Ҡߦ     NLB Aggregator ίϯςφ
37. 
     TD΁ͷΠϯϙʔτͷδϣϒ෼ׂ
38. Πϯϙʔτͷδϣϒ෼ׂ
     ✦ Πϯϙʔτର৅4ϓϨϑΟοΫεΛ޿͘औΔͱ
 ϝϞϦΊͬͪΌ࢖͏
    ‣ %BUB
    $POOFDUPS
    
    ϩά͕ফ͑ͯແݶϦτϥΠ˞೥݄౰࣌
    ‣ "84
    #BUDI
    
    ϝϞϦׂ౰্ݶ·Ͱ࢖ͬͯ00.
    ,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications
 /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz
39. Πϯϙʔτͷδϣϒ෼ׂ
     ✦ ͍Ζ͍Ζࢼͯ͠ɺ೔୯ҐͰδϣϒΛ੾Δͷ͕
 ྑͦ͞͏ͱ͍͏͜ͱʹ
    ‣ ϑΝΠϧαΠζɺϑΝΠϧ਺ʹґଘ͢ΔͷͰ΍ͬͯΈͳ ͍ͱΘ͔Βͳ͍෦෼ ࠓճ͸ϑΝΠϧ਺͕ଟ͔ͬͨ໛༷ in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}
40. 5SFBTVSF
    %BUBͷ'"2ʹ΋هࡌ͋Γ
     2 %BUB
    $POOFDUPS
    GPS
    4
    KPC
    JT
    SVOOJOH
    GPS
    B
    MPOH
    UJNF
    XIBU
    DBO
    *
    EP
    " $IFDL
    UIF
    DPVOU
    PG
    4
    pMFT
    UIBU
    ZPVS
    DPOOFDUPS
    KPC
    JT
    JOHFTUJOH
    *G
    UIFSF
    BSF
    PWFS
     
    pMFT
    UIF
    QFSGPSNBODF
    EFHSBEFT
    
 5P
    NJUJHBUF
    UIJT
    JTTVF
    ZPV
    DBO
    ‣ /BSSPX
    QBUI@QSFpY
    PQUJPO
    BOE
    SFEVDF
    UIF
    DPVOU
    PG
    4
    pMFT
    ‣ 4FU
      
    .#
    UP
    NJO@UBTL@TJ[F
    PQUJPO
    IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS
41. 
     ϩάαΠζͷ্ݶ
42. ϩάαΠζͷ্ݶ
     ✦ .%$ϩά͕σΧ͍
    ✦ %PDLFS
    MPHHJOH
    ESJWFS͸
 ,#ΑΓେ͖͍ϩάΛ෼ׂ͢Δ
    ‣ 'MVFOUE
    GPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ
    ‣ DPODBUϓϥάΠϯͰ݁߹
    ‣ ϝϞϦ࢖༻ྔ͕௓Ͷͯ00.
    ,JMMFSʹࡴ͞Εͨ
    ΠϚίί
    ✦ $MPVE8BUDI
    -PHTͷ্ݶ͸,#
43. ͓ۚ
     ✦ $MPVE8BUDI
    -PHT͸ϩάσʔλྔͷैྔ՝ۚ
    ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή
    ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍
    ✦ ΞϓϦ͔Βग़͢ϩάͷ෼ྔΛݮΒ͢
    ‣ .%$ϩάͭΒ͍
44. 
     4. ࠓޙͷల๬
45. ࣍ͷҰख
     ✦ ϩά؂ࢹ
    ✦ ϩάू໿ͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻
    ✦ 5SFBTVSF
    %BUBΛ΋ͬͱ׆༻͍ͨ͠
46. ·ͱΊ
     ✦ εέʔϧ͢Δϩά؅ཧج൫͸Ϋϥ΢υαʔϏεͷ
 ૊Έ߹Θ͕ͤΦεεϝ
    ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ࢖͍෼͚
    ✦ ૊Έ߹Θ͚ͤͩͰ΋ಈ͘΋ͷ͸Ͱ͖Δ͚Ͳ
    ‣ ཁ݅ʹରͯ͠଍Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ
    ‣ σʔλྔɺαΠζͷධՁ΍νϡʔχϯά͕ඞཁ