Share
Embed
顧客のアプリケーションコードが動くマルチテナント環境における課題とEKSにたどり着くまで
shogomuranushi
March 20, 2020
Transcript
61 slides
-
σʔλ औಘ σʔλ ੵ σʔλ ֬ೝ ڭࢣσʔλ ࡞ Ϟσϧ ઃܭ
ֶश ධՁ σϓϩΠ ਪ ࠶ֶश σʔλΣΞϋε ͷ४උͱཧ σʔλͷόϦσʔγϣϯʢਖ਼֬ੑʣͷ֬ೝ 0͔ΒͷϞσϧઃܭ GPUڥͷ४උͱ ߴͳࢄԽ σʔλɺϞσϧɺ݁Ռͷόʔδϣϯཧ ౷ܭతʹຊ൪ʹσϓϩΠͨ͠ॠؒ ͔Βਫ਼͕Լ͕Δ͜ͱΛ୲อ େྔσʔλͷऔಘʹඞཁͳAPIෛՙࢄ ͷΈ४උɺηΩϡϦςΟ୲อ ڭࢣσʔλͷ࡞ʹඞཁͳπʔϧͱਓࡐͷ४උ ։ൃڥ͔Βຊ൪ڥͷҾ͖͠ ੑGPUϦιʔεͷ୲อɺ Τοδଆͱͷ࿈ܞϓϩηεߏங -
σʔλ औಘ σʔλ ੵ σʔλ ֬ೝ ڭࢣσʔλ ࡞ Ϟσϧ ઃܭ
ֶश ධՁ σϓϩΠ ਪ ࠶ֶश σʔλΣΞϋε ͷ४උͱཧ σʔλͷόϦσʔγϣϯʢਖ਼֬ੑʣͷ֬ೝ 0͔ΒͷϞσϧઃܭ GPUڥͷ४උͱ ߴͳࢄԽ σʔλɺϞσϧɺ݁Ռͷόʔδϣϯཧ ౷ܭతʹຊ൪ʹσϓϩΠͨ͠ॠؒ ͔Βਫ਼͕Լ͕Δ͜ͱΛ୲อ େྔσʔλͷऔಘʹඞཁͳAPIෛՙࢄ ͷΈ४උɺηΩϡϦςΟ୲อ ڭࢣσʔλͷ࡞ʹඞཁͳπʔϧͱਓࡐͷ४උ ։ൃڥ͔Βຊ൪ڥͷҾ͖͠ ੑGPUϦιʔεͷ୲อɺ Τοδଆͱͷ࿈ܞϓϩηεߏங AI׆༻·Ͱʹଟ͘ͷ՝͕ଘࡏ -
Ref: https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf “ As the machine learning (ML) community continues
to accumulate years of experience with live systems ” “ ։ൃ͓ΑͼMLγεςϜΛಋೖ͢Δ͜ͱൺֱతߴͰ҆ՁͰ͕͢ɺ࣌ؒΛ͔͚ͯ ͦΕΛҡ࣋͢Δ͜ͱࠔ͔ͭߴՁͰ͋Δ” -
ୈҰੈΞʔΩςΫνϟ of Datalake • Raw σʔλΛੵ͢ΔͨΊͷαʔϏε • ετϨʔδ S3 •
ॳظόʔδϣϯ API Gateway ͱ Lambda Ͱ REST API Ͱఏڙ • ॳϑΝΠϧͷอଘɺऔಘɺҰཡͳͲͷΦϖϨʔγϣϯͷΈ • ߏஙʹServerless FrameworkΛར༻ • Signed URLΛൃߦ͠Ξοϓϩʔυͯ͠Β͏༷ -
ୈҰੈΞʔΩςΫνϟ of Datalake Good • ϝϯςφϯεϑϦʔ Bad • API GatewayͭΒ͍
• ϩʔΧϧͷ࠶ݱੑ͕͘։ൃޮѱ͍ • ϖΠϩʔυαΠζ • Serverless Framework·͊·͊ਏ͍ • ࠓޙଞͷαʔϏεಉ͡ελΠϧͰ։ൃʁ -
ୈೋੈΞʔΩςΫνϟ of Datalake • AWSͷAPI Gatewayγϯυ͍ͷͰɺAPI GatewayΛ • ։ൃޮ͕ѱ͍ɺେࣄͳΤϯυϙΠϯτͳͷͰো࣌ͷίϯτϩʔϧ͔ͨͬͨ͠ •
API Gateway͕͞Εͨ͜ͱͰɺೝূɾϧʔςΟϯά͕ڞ௨ʹ • ࢁͷAPI Gatewayͱ͓͞Β • DatalakeLambdaͱS3ͷΈʹ • ໘͜ͷߏ -
ୈೋੈΞʔΩςΫνϟ of Datalake Good • ೝূɾϧʔςΟϯάΛ͠ڞ௨ʹ • DatalakeͷൣғΛ Bad •
ࣗલAPI Gatewayͷ໘ΛݟΔඞཁ͋Γ -
ୈࡾੈΞʔΩςΫνϟ of Datalake • ʮݕࡧΧϯτ͕͍ͨ͠ʯ • ϝλσʔλݕࡧΧϯτػೳ༻ͷDBΛߏங • ݕࡧPostgreSQL(Aurora)ͷJSON+GIN IndexͰ࣮
• ॊೈͳϝλσʔλͷ༩ͱݕࡧΛͰ͖ΔΑ͏ʹ͠ա͗ͯIndexരൃ • Cassandra…? Or ॊೈੑΛܰݮͤ͞Δ͜ͱΛݕ౼த • S3 Event + SQS + Lambda ͔Β Datalake API Λݺͼग़͢ • όοΫΤϯυͷෛՙ͕͖͑Εͳ͔ͬͨɺCW Logs͕ߴ͘ͳͬͨͷͰόοΫΤϯυΛLambda ͔ΒECSʹҠߦ • ʮS3ͷSigned URLखؒͳͷͰҰճͰDatalakeʹΞοϓϩʔυ͍ͨ͠ʯ • S3ͷPutAPI GatewayͰ୲͏Α͏ʹ༷มߋ -
ୈࡾੈΞʔΩςΫνϟ of Datalake Good • Signed URLෆཁʹͳΓUXվળ • ϝλσʔλݕࡧͰ͖ΔΑ͏ʹ Bad
• αʔόϨεແ͘ͳͬͨͷͰϝϯ ςίετ૿Ճ • ϝλσʔλػೳ͕ࣗ༝ա͗ͯIndexංେԽ Ͱਏ͍ όοΫΤϯυෛՙɺCWLogsίετ૿ՃʹΑΓ ECSʹมߋ -
ୈྵੈΞʔΩςΫνϟ of Serving Good • γϯϓϧ • ҰׅͰؔ࿈ϦιʔεΛ࡞ɾআͰ͖Δ Bad •
1Serviceຖʹ1ͭͷELBແବ • CFnͰ࡞͞ΕΔͷʹ͔͔Γ͍ • CFnඇಉظͳͷͰΤϥʔݕ͕͍͠ -
ୈҰੈΞʔΩςΫνϟ of Serving • 1 Loadbalancer = Muliti Serviceʹมߋ •
ALBͷϧʔςΟϯάϧʔϧ100ݸ͕ϋʔυϦϛοτ • ސ٬ͷAPI͕ͲΜͲΜΔͨΊ͙͢ʹഁ͢Δ͜ͱ͕ݟ͑ͯͨ • ·ͨࣗલͰGatewayΛ։ൃʢECS + Lambda + DynamoDBʣ orz • ސ٬ͷΞϓϦέʔγϣϯ͕ৗ࣌Τϥʔൃੜɻίϯςφ͕࠶ىಈ͠·͘ΓEBSόʔετΫϨδοτΛ৯͍ͭͿ͢ • ECS Seviceৗʹىಈ͢Δ༷ͰαʔΩοτɾϒϨΠΫແ͍ • γεςϜىҼͷΤϥʔͰͷαʔΩοτɾϒϨΠΫޙ࣮͞Ε͕ͨɺΞϓϦىҼͷαʔΩοτɾϒϨΠ Ϋ·ͩແ͍ • ͳͷͰ࠶ىಈ܁Γସ͑͠IO৯͍ͭͿ͢ʢѹతϊΠδʔωΠόʔʣ • ࣗલʢLambdaʣͰϔϧενΣοΫͯ͠ɺҰఆճFailͷ߹ϧʔςΟϯά͠ͳ͍Α͏ʹࣗલGatewayͷ DynamoDBʹอଘ -
ୈҰੈΞʔΩςΫνϟ of Serving • Blue/Greenػೳͷ࣮ • APIͷΤϯυϙΠϯτͷ͖ઌͱͳΔϞσϧΛ؆୯ʹΓସ͑ΒΕΔػೳ • ECSλεΫ͔ΒϗετͷIAMϩʔϧ͕৮Εͯ͠·͏ •
iptablesͰmetadataͷΞΫηεΛͿͬͨΔ • ࠷ۙɺawspvc ωοτϫʔΫϞʔυͷ߹ͷ߹ҎԼͷΑ͏ʹ؆୯ • ECS_AWSVPC_BLOCK_IMDS: true -
ୈҰੈΞʔΩςΫνϟ of Serving Good • 1ਪAPI = 1ELBͰແ͘ͳΔͨΊίετμ ϯ •
Blue/Green͕Ͱ͖ΔΑ͏ʹ • EBSόʔετղܾ • EC2 metadataΞΫηεͰ͖ͳ͍Α͏ʹ Bad • ࣗલ࣮͕ଟ͘ͳ͖ͬͯͯϝϯςίετ ͕͕͕ɻͱ͍͑ɺAWSͷػೳͰΧ όʔͯ͠ͳ͍՝ࣗલ࣮͔͠ͳ͍ -
ୈೋੈΞʔΩςΫνϟ of Serving • ECSͱEC2ͷ૬ੑ͕ѱ͍ • AutoScalingൃಈ࣌ʹίϯςφΛແࢹͯ͠EC2ΛTerminate͢ΔɻDrainͯ͠Αʢࠓղফ͞Ε͍ͯΔʁʣ • AutoScalingൃಈ݅CPU༧ྔϕʔε͕جຊͳͷͰɺίϯςφαΠζͷεέʔϧ͕͔ͳΓखؒͩ͠ɺܭ ࢉϩδοΫΊΜͲ͍͘͞
• ۭ͖Ϧιʔε͕͋ͬͯू͞Εͳ͍ • ΠϯελϯεΛೖΕସ͑ΔࡍͷBlue/Green Deploymentࣗલ࣮ • ސ٬ͷਪAPIΛࢁࡌͤΔͱίετ͕ංେԽ͢Δ • εϙοτΠϯελϯεΛ͏·͘ѻ͑ͳ͍͔ݕ౼ • ΠϯελϯελΠϓκʔϯΛࢄΓΊͨΓɺεϙοτΠϯελϯε͕ചΓΕͨ࣌ʹΦϯσϚϯυʹ Γସ͑ͨΓ͢Δඞཁ͕͋Δ • ݕূͷ݁ՌɺSpotinstͱ͍͏αʔϏεΛೖΕΔ͜ͱʹ -
Bad • εϙοτΠϯελϯεͷద༻ൣғΛؒҧ ͑Δͱҙਤ͠ͳ͍ఀࢭ͕ൃੜ͢Δ ʢJupyter Notebookͱ͔ʣ ୈೋੈΞʔΩςΫνϟ of Serving Good
• SpotinstΛར༻͢Δ͜ͱʹΑΓECS͕ Γͳ͍ͱ͜ΖΛΧόʔ • Blue/Green Deployment͕ڧ੍͞ΕΔ • ूޮ্͕͕Γɺ60-70%ίετμϯ -
ୈࡾੈΞʔΩςΫνϟ of Serving • 1 ServiceͰ 300ίϯςφΛಈ͔͢πϫϞϊ͕ग़͖ͯͨ • DynamoDBͷύʔςΟγϣχϯάͷภΓ͕ൃੜ͠εϩοτϦϯά͕େྔʹ •
Kubernetesػӡ͕ߴ·͖ͬͯͨɻKubernetesΛ׆༻͢Δ͜ͱͰ • API Gateway૬ͷػೳAmbassador(Envoy)Ͱ༻Մೳ • ͔͠͠ɺService/Pod͕େྔʹଘࡏ͢ΔͱEnvoyͷϧʔςΟϯάϧʔϧͷߋ৽ʹेඵ͔͔Δ͜ͱ • Service DiscoveryɺϔϧενΣοΫඪ४ػೳͰͤΒΕΔ • ServiceڥKubernetesԽͱͱʹࣗલ࣮෦ͷݮΒ͢ํʹ -
ୈࡾੈΞʔΩςΫνϟ of Serving Good • ϔϧενΣοΫɾαʔϏεσΟεΧόϦΛࣗ લ࣮͔ΒKubernetesͷҰൠతͳػೳʹஔ͖ ͑ • GatewayͷػೳΛKubernetesͷ֦ுػೳ
ʢAmbassadorʣʹஔ͖͑ Bad • Ambassador(Envoy)ͷཧղ͕ඞཁʹ • ࣗલ࣮ΑΓ༷ʑͳ໘Ͱߟྀ͞Ε͍ͯ ͯϝϯςίετԼ͕Δ͕ΧελϚΠζ ੑམͪΔ -
ୈࡾੈΞʔΩςΫνϟ of Serving • ݱঢ়ͷ՝ • ސ٬ຖͷPodͷΞτόϯυͷసૹྔͷܭଌํ๏͕Θ͔Βͳ͍ • GKEʹ͋ΔΒ͍͠ɻIstioͷग़൪͔ɾɾʁ •
ސ٬ຖʹίετΛՄࢹԽ͢ΔͨΊʹKubernetesͷeventΛhook͠·͘Βͳ͍ͱ͍͚ͳ͍ • ͜ΕGKEusage meteringͱ͍͏ͷͰग़དྷΔΆ͍ͷͰEKSͰͥͻͬͯ΄͍͠ɻࣄۀ෦ϚϧνςφϯτͰඞཁͱࢥ͏ • Kubernetes ͷ݁Ռ߹ੑͷৼΔ͍ͷ্ʹࣗͨͪͷγεςϜΛߏங͢Δ͠͞ • ྫ͑ • Pod ͷ STATUS ͕ Running ʹͳΔɻૄ௨ग़དྷΔ͔ͱࢥ͍͖ Ready (readinessProbe) ͕ 0/1 • Ready ͕ 1/1ʹͳΔɻૄ௨Ͱ͖Δ͔ͱࢥ͍͖ Ambassador (Envoy) ͷͱ͋Δ Pod ૄ௨Ͱ͖Δ͕ɺͱ͋Δ Pod ߋ ৽ͪͷͨΊૄ௨Ͱ͖ͳ͍ɻࢄγεςϜʹ͓͚Δ݁Ռ߹ͳͷͰʮ͍ͭʯ͔Β͑ΔΑ͏ʹͳͬͨঢ়ଶ͔ΛϢʔβ ʹΒͤΔͷ͕͍͠ • ͦΕͧΕͷػೳ͕݁Ռ߹Λ୲อ͍ͯ͠Δ͕ނʹɺ࿈ಈͯ͠ཉ͍͠ͱ͜ΖΛຒΊΔඞཁ͕͋Δ -
ୈࡾੈΞʔΩςΫνϟ of Serving • ݱঢ়ͷ՝ • IPΞυϨεͷރׇ • 1ΠϯελϯεลΓ Serving
= 20ݸɺTraining = 30ݸ Λ༻͍ͯ͠Δɻ20ݸ * 400 Service ͕࡞ΒΕͨ࣌ʹ8,000ݸͷIP Λ༻ɻ·͔͞ͷ /16 αϒωοτ͕ރׇ • એݴతʹͳΓͮΒ͍ • SDKͰσϓϩΠͯ͠ΔͷͰʮyamlΛ͏Ұద༻ͨ͠Β࠶ݱͰ͖ΔΑʯঢ়ଶʢએݴతʣʹͳ͍ͬͯͳ͍ɻ൵͍͠ • ࢹ͠ਏ͍ • ϢʔβʔىҼͷͱϓϥοτϑΥʔϜͷͷΓ͚ํ๏͕͍͠ • ސ٬ىҼͰࢮΜͰΔpod͕ଘࡏ͢Δɻશ͕ͯਖ਼ৗʹՔಇ͍ͯ͠Δ༁Ͱͳ͍ • ಠࣗGatewayͰ5xx/4xxͷࢹ͕Ͱ͖ͳ͍ɻόοΫΤϯυ͕ސ٬ґଘͷͨΊ5xxे༗ΓಘΔ • ϚΠΫϩαʔϏε͋Δ͋Δ • LB/Proxy͕ଟஈͳͷͰௐࠪͮ͠Β͍ɻސ٬ʹͲ͜·ͰϩάɾϝτϦΫεΛग़ͤྑ͍ͷ͔͍͠ -
ୈҰੈΞʔΩςΫνϟ of Training • ֶशδϣϒΛ࣮ߦ͢Δج൫ • ॳظόʔδϣϯ͔ΒKubernetesϕʔεͰ࣮ͨ͠ • JobɺPodʢartifactอଘ༻ίϯςφʣͳͲΛECSͰࣗલ࣮ඇޮͩͬͨͨΊ •
͔͠͠EKSແ͔ͬͨͷͰ on EC2 Ͱ • ސ٬ͷίϯςφͱཧܥίϯςφPodΛ͚ͨ • TrainingޙͷϞσϧΛs3ʹࣗಈอଘ͢ΔίϯςφɺϩάΛऩू͢ΔίϯςφɺϢʔβͷίʔυͱಉډͤ͞ΔͱIAMͷݖ ݶతʹྑ͘ͳ͍ͷͰɺผPodͱͯ͠agentతʹىಈ • kube2iam Λ༻͍ͯPodຖʹIAMϩʔϧΛΞλονʢࠓެ͕ࣜग़͍ͯΔʁʣ • privilegedԿ͕͋ͬͯoff • GPUυϥΠόपΓۤ࿑͢Δ͚ͲؤுΔ -
ୈೋੈΞʔΩςΫνϟ of Training Good • Jupyter NotebookɺTensorboradΛఏڙ • ڞ༗ετϨʔδͷఏڙ Bad
• Kubernetes on EC2 ӡ༻͕݁ߏਏ͍ • Jupyterະ༻࣌ͷՔಇίετ͕ແବ • ڞ༗ετϨʔδ͕ߴͯ͘NFSͳͷͰ͍ -
ୈࡾੈΞʔΩςΫνϟ of Training Good • EKSҠߦʹΑΓӡ༻ෛՙݮΔ • EFS͔ΒS3ͷॻ͖ࠐΈɾ͠ʹΑΓί ετμϯ •
JupyterͷࣗಈఀࢭʹΑΓίετμϯ Bad • EFS͕NFSͳͷͰ͍ -
ୈࡾੈΞʔΩςΫνϟ of Training • ݱঢ়ͷ՝ • ؒܭըͷച্ɾݪՁ • ਫ਼៛Խ͢Δඞཁ͕͋Δ͕ɺސ٬࣍ୈͳͷͰݪՁΛ༧ଌ͢Δͱ͔Α͔͘ΒΜɻAWS͞ΜͲ͏ͬͯ ཧ͍ͯ͠ΔͷͩΖ͏
• όά • p3.16xlarge͕ఀࢭͤͣʹPສҐ͔͔ͬͯΔ࣌͋ͬͨ • OS • nvidia-driver͕αϙʔτ͢ΔOSͰ͋Δඞཁ͕͋Δɻͭ·ΓUbuntu or Amazon Linux2ɻʮBottlerocketʯͷ nvidia-driverαϙʔτظ -
ୈࡾੈΞʔΩςΫνϟ of Training • ݱঢ়ͷ՝ • ίϯςφؒͷґଘؔ • ϩάΛ࿙Εͳ͘ऩू͢ΔͨΊʹ Affinity
Λۦͯ͠ log collector pod -> platform agent pod -> training job ͱ͍͏༏ઌॱҐΛ͚ͯPodΛىಈͤ͞Δͱ ɺඞཁͳϦιʔε͕Γͳ͍EventͷൃՐ͕ΕΔͨΊ Autoscaler ͷ௨͕ΕɺΠϯελϯεͷىಈ͕͘ͳΔ • DockerΠϝʔδɾύοέʔδͷޓੑɺαϙʔτ • ఏڙ͢ΔDockerΠϝʔδͷޓੑҡ͕͍࣋͠ • αϙʔτର • DLϥΠϒϥϦͷछྨ x όʔδϣϯ x Pythonόʔδϣϯ x CUDAͷόʔδϣϯ … -
ୈҰੈΞʔΩςΫνϟ of Trigger • ਪδϣϒΛ࣮ߦ͢Δج൫ • Datalakeʹσʔλ͕ೖ͞Εͨ͜ͱΛτϦΨʔʹൃಈ • S3->SNS->SQSͰɺҰ୴ΩϡʔΠϯά͢Δ •
SQSͷQueue͔ΒSubscriber͕औಘ͠ɺAWS BatchλεΫΛ͛Δ • δϣϒ͕ऴΘΕΠϯελϯε͕ࣗಈఀࢭ͢Δ • ֖Λ։͚Ε1ʹԿඦͱδϣϒ͕͛ΒΕΔ -
ୈҰੈΞʔΩςΫνϟ of Trigger Good • ٸܹͳෛՙSQS͕ٵऩ • SubscriberQueueͷͰεέʔϧ • AWS
Batch͛ͨΒྑ͍͚ͩ Bad • ىಈ͕͍ɺϝτϦΫεແ͍ɺϩά͕ू ͞ΕͯJob୯ҐͰݟΕͳ͍ • AWS BatchͷϢʔεέʔεʹ߹ͬͯͳ͔ͬͨ • AZؒͷωοτϫʔΫసૹྔ͕Ϡό͍ -
ୈҰੈΞʔΩςΫνϟ of Logging for Customer • ސ٬ʹఏڙ͢Δϩάج൫ • ServingͱTrainingɺTriggerͷϩάΛఏڙ •
ϩάίΞίϯϐλϯεͰͳ͍ͨΊɺग़དྷΔݶΓࣗલͰӡ༻ͨ͘͠ͳ͍ -
ୈҰੈΞʔΩςΫνϟ of Logging for Customer Good • αʔόϨεܥʢCWLogs / Kinesiss
/ Lambda / DynamoDBʣͳͷͰӡ༻ཧෆ ཁ Bad • ϩάྔ͕ٸ૿͢ΔͱLambda͔ DynamoDBͰεϩοτϦϯάى͖Δ • CWLogs࣮࣭ͬͯͳׂ͍ʹ݁ߏߴ͍ -
ୈೋੈΞʔΩςΫνϟ of Logging for Customer • ϩάͷόοΫΤϯυΛDatadog Logsʹมߋ • ServiceɺTrainingͷϩάΛDatadog
Logsʹอଘ • Datadog Logs ͷAPI͔ΒϩάΛऔಘ͠ސ٬ʹఏڙ • ElasticSearchʹ͢Δ͔ߟ͕͑ͨ • ݕ౼࣍Ͱϩάྔ5ԯϨίʔυ/݄ • ࣄۀͷͱϩΪϯάରΛ૿͢͜ͱΛߟ͑Δͱ3ϲ݄ຖʹഒʑʹ૿͑Δ • ഒʑʹ૿͑ΔElasticSearchΛӡ༻ͨ͘͠ͳ͔ͬͨ͠ɺϩάίΞίϯϐλϯεͰͳ͍ͷͰӡ༻ί ετΛֻ͚ͨ͘ͳ͔ͬͨ -
ୈೋੈΞʔΩςΫνϟ of Logging for Customer Good • ϑϧϚωʔδυͳͷͰӡ༻ϑϦʔ Bad •
͓͕͔͔ۚΔɻͱ͍͑ɺCWLogsΑΓ҆͘ ࣗલͰӡ༻͢ΔΑΓϚγ • Datadogͷ༷ʹҾͬுΒΕΔ • ݁Ռ߹ɺॱংอূແ͠ • datadog-agentͷڍಈɾ༷ -
ୈҰੈΞʔΩςΫνϟ of Logging for System & Application • ࣾͰར༻͢ΔγεςϜϩάɾΞϓϦέʔγϣϯϩάͷऩूج൫ •
ϩάίΞίϯϐλϯεͰͳ͍ͨΊɺग़དྷΔݶΓࣗલͰӡ༻ͨ͘͠ͳ͍ • ECSLambdaΛத৺ʹར༻͍ͯͨͨ͠ΊCloudWatch Logsʹϩά֨ೲ͞Ε͍ͯͨ • ͱΓ͋͑ͣCW LogsΛར༻ͨ͠ -
ୈҰੈΞʔΩςΫνϟ of Logging for System & Application Good • Πϯϑϥͷ͜ͱߟ͑ͳͯ͘ྑ͍
Bad • Ͳ͜ʹ֨ೲ͞Ε͍ͯΔ͔͔Βͳ͍ • ݕࡧੑօແɻgrepྗ্͕Δ • ϚΠΫϩαʔϏεؒͷϩάௐࠪͱ͔͔ͳΓπ ϥϛ͔͠ͳ͍ • ݁Ռɺ͔͋ͬͨ࣌͠ϩάΛݟͳ͍ • Ҏ֎ʹCWLogsߴ͍ -
ୈೋੈΞʔΩςΫνϟ of Logging for System & Application Good • Πϯϑϥͷ͜ͱߟ͑ͳͯ͘ྑ͍
• ҰՕॴͰશ෦ݕࡧग़དྷΔ • Tag, Attributeʹରͯ͠IndexுΕͯݕࡧରʹ Ͱ͖Δ • Ͳͷ߲ʹԿ݅͋Δ͔Ұྎવ • ϚΠΫϩαʔϏεͳͷͰɺTracing IDͳͲΛຒ ΊࠐΜͰ͍ͨ͘͢͠Γ • APIͷੳͨ͠ΓɺӨڹൣғௐͨΓͱ ׆༻͕͕ͬͨ Bad • ͓͕͔͔ۚΔɻͱ͍͑ɺCWLogsΑΓ҆ࣗ͘ લͰӡ༻͢ΔΑΓϚγ