Share
Embed
Transcript
39 slides
-
ࣗݾհ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work
fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓ - AbemaTV -
ࠓͷ͢༰ • CyberAgentͰCDʹؔ͢Δ՝ • ౷ҰͳσϦόϦʔج൫Λݕ౼ • PipeCDͷੜ • ݱࡏͷPipeCD @
CyberAgent • PipeCDͷࠓޙ -
ϓϩμΫτͷߏ ABEMA AWA WinTicket CyberZ AI Studio ... • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத
• ֤ϓϩμΫτͰࣗʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબΔ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾҠಈՄೳ ʢेݸ͕͋Δʣ -
ϓϩμΫτͷߏ ABEMA AWA WinTicket CyberZ AI Studio ... • ֤ϓϩμΫτͰࣗʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબΔ
Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾҠಈՄೳ ʢेݸ͕͋Δʣ CDϓϩμΫτΤϯδχΞʹࣗ༝ʹͤΔͷͰνʔϜ͝ͱʹঢ়گ͕ҟͳΔ • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத -
ݩʑͷCDͷঢ়گ શମతͳΜͰ͋Δ͜ͱʹͳͬͯ͠·ͬͨ खಈ Jenkins CircleCI TravisCI DroneCI Harness Concource ArgoCD
GH Actions Spinnaker Flux Code Deploy Cloud Build Cloud Deploy ୲ऀ͕खಈͰΔ CIͰCDΛΔ (CI Ops) ઐ༻CDΛ͏ ࣗ࡞ ChatOps -
CDʹؔ͢Δ՝ Multi Cloud Product • σϦόϦͷख़͕ߴ͘ͳ͍ͱ͜Ζ͕ଘࡏ • खಈϦϦʔεͷνΣοΫϦετ͕େม • Canary,
BlueGreenͳͲͷ҆શରࡦ͕ͳ͍ • ͤΔͨΊɺࣗͰCDͷߏஙͱӡ༻͕େม • ౷ҰੑʢҰ؏ੑʣ͕͘ͳ͍ͬͯΔ • ࣾελϯμʔυΛཱ͢Δ͜ͱ͕͘͠ͳΔ • ࣾͷϕετϓϥΫςΟεͷීٴ͕େม • શମతͳӡ༻ίετ͕ߴ͍ • Developer ExperienceʹӨڹͯ͠͠·͏ • ࣾҠಈͰOnboardingίετ͕ൃੜ ٕज़ελοΫ͕ಉ͡Ͱ ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ ಉ͡ϓϩμΫτͰ ෳCDγεςϜɾϓϩηε͕ଘࡏ͢Δ Product A Product B CD System X CD System Y CD System X for GCP services CD System Y for AWS Services CD System Z for Infra -
CIͱCDΛશతʹ TestͰૣظతʹ։ൃऀϑΟʔυόοΫΛఏڙ ArtifactΛHost EnvironmentσϓϩΠɾϩʔϧόοΫ BuildͰImmutable ArtifactΛੜ͢Δ Host Environment (ClusterͳͲʣܨ͕Βͳ͍ Host
Environment (ClusterͳͲʣతʹܨ͕Δ ੜ͞ΕͨArtifactʹؔ͢ΔϑΟʔυόοΫΛૣΊʹఏڙ -
౷ҰͳCDγεςϜ Delivery Infrastructure Product Team࠷େͳσϦόϦύϑΥʔϚϯεΛ ग़ͤͳ͕Βɺࣗʹ߹͏ٕज़ελοΫͱσϦόϦ ઓུΛબͿϑϦʔμϜ͕͋Δ Platform TeamॊೈͳDelivery InfrastructureΛ
ఏڙͱCDͷϕετϓϥΫςΟεΛීٴ Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ Standard/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ Ͳ͏͢Εɺ -
Platform Team & Product Team Management Security γεςϜͷӡ༻ίετ͕ߴ͍͔Ͳ͏͔ ϓϩμΫτಋೖͷεέʔϥϏϦςΟ ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏ཧ͞ΕΔ͔
CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛϓϩμΫτνʔϜීٴͰ͖Δ͔Ͳ͏͔ Automation Visibility / Accessibility Control / Flexibility νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબΕΔͷ͔ νʔϜͷݖݶΛదʹཧͰ͖Δͷ͔ ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·ͰݮͰ͖Δͷ͔ ϦϦʔεதޙܧଓతʹΛݕͰ͖Δͷ͔ ։ൃऀेͳϑΟʔυόοΫΛΊʹఏڙͰ͖Δ͔ ͕͋Δ࣌ʹݪҼؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔ ཧऀνʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔ Platform Team Product Team -
PipeCD 🤍 OSS Thanks to the contributors of PipeCD! https://github.com/pipe-cd/pipe
https://pipecd.dev https://pipecd.dev/docs 27 Contributors 4 Full-time contributors 1 Part-time contributor 1800 Pull requests ʢ෭ۀʣ -
PipeCD 🤍 GitOps • GitOpsΛ࠾༻ • GitʹશͯͷCon fi gurationΛཧ •
શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps • Kubernetes • Terraform • CloudRun • AWS Lambda • ECS, Fargate • ... -
PipeCDͲͷΑ͏ʹզʑͷ՝Λղܾ͢Δ͔ Automation Visibility / Accessibility Control / Flexibility Management
Security Platform Team Product Team -
Platform Team - Management • ӡ༻ίετ͕͍ • શͯͷσϓϩΠϝϯτҰͭͷγεςϜͷΈ͕ඞཁ • PipeCDͷશͯͷίϯϙʔωϯτ͕εςʔτϨε
• ετϨʔδϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ • ϓϩμΫτଆγϯάϧόΠφϦͷPiped agentΛΠϯετʔϧͷΈ • ϓϩμΫτͷεέʔϥϏϦςΟ • ৽نͷϓϩμΫτͷՃ͕୯౬ • ͲͷڥͰಈ͚Δઃܭ • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ • খنνʔϜ͔ΒେنνʔϜ·Ͱ Product B Piped Control Plane Firewall Friendly Outbout Requests Only Product A Piped Piped Piped -
Platform Team - Security • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ֎ʹҰग़ͳ͍ઃܭ • Control-planeʹอଘɾཧ͠ͳ͍ • GitOpsͰͷSecretཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ
• Piped agent͕ΞτόϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲඞཁ͕ͳ͍ -
Product Team - Visibility / Accessibility • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
• SlackͳͲΠϕϯτΞϥʔτͳͲΛ௨ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment -
Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/application-live-state • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •
ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲΠϕϯτΞϥʔτͳͲΛ௨ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε -
Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/plan-preview • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •
ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲΠϕϯτΞϥʔτͳͲΛ௨ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Git di ff is enought? No. e.g. using remote Helm chart, Kustomize package, Terraform module... Reviewer needs more early feedback to merge PR with con fi dence Better to see dry-run result, terraform plan, deployment policy... on the PR then Git di ff is just showing the change of version number -
Product Team - Visibility / Accessibility Terraform Deployment CloudRun Deployment
• UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲΠϕϯτΞϥʔτͳͲΛ௨ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Scale In Scale Out Rollout New Image Update Con fi g Deploy Infra Deploy Serverless Rollback Deploy Kubernetes Constant process for all operations -
Product Team - Automation https://pipecd.dev/docs/user-guide/rolling-back-a-deployment • σϓϩΠͷΠϯύΫτΛࣗಈੳ • ϝτϦΫεɾϩάɾhttp •
աڈͷσʔλɾcanary-baselineͷൺֱ • σϓϩΠதʹ͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ • Gitͱ࣮ࡍͷঢ়ଶͷCon fi guration DriftΛࣗಈݕ • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ ࣗಈσϓϩΠͷEventWatcher https://pipecd.dev/docs/user-guide/con fi guration-drift-detection -
Product Team - Control / Flexibility • Piped AgentΛ͖ͳελΠϧͰΈ߹ΘͤΔ͜ͱ͕Մೳ •
1 Piped AgentͰશͯཧύλʔϯ • ڥຖʹઐ༻ͷPiped Agentύλʔϯ • ͖ͳॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ • σϓϩΠઓུࣗ༝ʹΈ߹Θͤɾఆ͕ٛՄೳ • Quick Sync • Progressive Sync (Canary, BlueGreen...) • νʔϜʹ߹͏ϓϩόΠμʔΛબΒΕΔ • Cloud Provider (GCP, AWS, Azure, Private Cloud) • Analysis Provider (Prometheus, Datadog, Stackdriver...) https://github.com/pipe-cd/examples/ blob/master/kubernetes/canary/.pipe.yaml -
νʔϜͱγεςϜͷߏ • Platform Team • PipeCDΛ։ൃ • ࣾ༻PipeCD Control-planeΛӡ༻ •
֤Product Team • Piped agentΛΠϯετʔϧ • ࣗʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻ Control-Plane https://pipecd.dev/docs/operator-manual/piped/installation https://github.com/pipe-cd/examples -
ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத 0 225 450 675 900 2020/10 2020/12 2021/02 2021/04
2021/06 2021/8 806 Applications/Servicesʹୡ CyberAgentʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... ͷΞϓϦέʔγϣϯɾαʔϏε -
• ࠓ·ͰͷPiped agentͷӡ༻ • Product Team͕Piped agentΛΠϯετʔϧ • ৽͍͠όʔδϣϯ͕͋Δͱ࠶Πϯετʔϧ •
͜Ε͔Β • ҰճͷΈΠϯετʔϧ͢Δ • Web consoleͰόʔδϣϯΛΞοϓͰ͖Δ • ͦΕͰӡ༻͕͞ΒʹָʹͳΔ Management - Remote Upgrade ͜ͷػೳདྷिʹϦϦʔε༧ఆ Product B Piped Control Plane Firewall Friendly Outbout Requests Only Product A Piped Piped Piped -
Visibility / Accessiblity • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ • DeployසɺLead Time, ࣦഊ, MTTR...
• ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Terraform, CloudRun, ECS... • Deployment StageͷϩάΛվળ • Plan-PreviewʹͬͱϑΟʔυόοΫΛՃ͑Δ • Kubernetes validating webhook, dry-run݁Ռ, Terraform sentinelͳͲͷDeployment Policy ֎ͷϢʔβʔ͚ͷPlaygroundڥΛ४උதʂ https://play.pipecd.dev -
Automation • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈੳ • աڈͷMetricsσʔλͱͷൺֱͰࣗಈੳ • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈੳ •
Terraform, CloudRun, LambdaͳͲͷࣗಈDrift Detection -
Control / Flexibility • ਂ͍ϨϕϧͷݖݶཧͰ͖ΔACL • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ
• Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ -
Deployment Chain Application X Region A Region B Region C
ᶃ ᶄ ᶅ Application X Cluster A Cluster B Cluster C ᶃ ᶄ ᶅ Application X - Dev Env Application X - Stg Env Application X - Prod Env Application Infra Application X Application Y ޭͳΒ࣍ʹਐΉ ޭͳΒ࣍ʹਐΉ