Share
Embed
Transcript
42 slides
-
// THE PERFECT GITOPS PROCESS: REPOS, FOLDERS, STAGES, PATTERNS Johannes
Schnatterer, Cloudogu GmbH Version: 202303021327-a4b0478 @[email protected] @jschnatterer 1 -
Agenda 1 GitOps process design basics 2 Example + demo
3 More examples 2 -
Preamble • Chronology: • Step 1: Chose an operator •
Step 2: Design process/repos focus of this talk • Use case: • Deploying infra • Deploying apps focus of this talk • Responsibility: platform/infra teams, cluster admins app teams • Conway's law: No standard for structures (intentionally) 4 -
GitOps Chasm Infra • repos • folders • branches •
clusters • namespaces • operator instances • operator-specific config Mapping? Real-world • company/departments • teams • projects • applications • microservices • customers • tenants • stages/environments • etc. 5 -
No standard but emerging patterns AKA strategies, models, approaches, best
practices • : GitOps operators Clusters/Namespaces • : How many repos? • : How to model environments/stages? • : Bootstrapping operator, linking repos and folders Operator deployment Repository structure Release promotion Wiring 6 -
GitOps Operator deployment patterns How many GitOps operators to deploy,
relating to Kubernetes clusters? • Standalone: 1 Operator : 1 Cluster • Hub and Spoke: 1 Operator : n Clusters • Namespaced: n Operators : 1 Cluster 7 -
Repository patterns How many GitOps repos? • Monorepo (opposite polyrepo)
• Repo per Team (Tenant) • Repo per App • • • Repo per stage/environment Can be mixed Config replication Repo pointer 8 -
Repository types GitOps repo App repo Content IaC/Manifests/YAMLs Application source
code Synonyms • Config repo • Infra repo • Payload repo • Source code repo • Source repo Example gitops-repo ├── app1 │ ├── deployment.yaml │ └── service.yaml └── app2 ├── deployment.yaml └── service.yaml app-repo ├── src ├── test ├── CI.pipeline ├── Dockerfile ├── package.json └── pom.xml 9 -
Separating GitOps repo from app repo K8s Cluster Developer App
Repo GitOps Repo pull CI Server GitOps operator OCI Registry push app code push infra code pull push pull deploy GitOps tools: Put infra in separate repo! See argo-cd.readthedocs.io/en/release-2.6/user-guide/best_practices 10 -
Disadvantages • Separated maintenance & versioning of app and infra
code • Review spans across multiple repos • Local dev more difficult • No static code analysis on GitOps repo How to avoid those? 11 -
Config replication K8s Cluster Developer App Repo GitOps Repo CI
Server GitOps operator OCI Registry push app +infra code pull push infra code push pull pull deploy 12 -
Advantages • Single repo for development: higher efficiency • Shift
left: static code analysis + policy check on CI server, e.g. yamlint, kubeval, helm lint, conftest, security scanners • Automated staging (e.g. PR creation) • Simplify review by adding info to PRs 13 -
Disadvantages • Complexity in CI pipelines Recommendation: Use a plugin
or library, e.g. cloudogu/gitops-build-lib • Redundant config (app repo + GitOps repo) 14 -
Alternative: Repo pointer K8s Cluster Developer App Repo GitOps Repo
CI Server GitOps operator OCI Registry push app +infra code point to pull push pull pull pull deploy e.g. fluxcd.io/flux/guides/repository-structure 15 -
Release promotion patterns How to model environments AKA stages? •
• (anti-pattern) • (edge case) • AKA Env per (folder | branch | repo) Folder/Directory per environment Branch per environment Repo per environment Preview environments 16 -
Why not use branches for environments? Idea: • Develop Staging
• Main Production • Drifts/conflicts because of merge direction develop main (unidrectional) • Promoting specific changes only: Copy vs cherry pick • DRY - resources shared by multiple environments, e.g. • Scalability: More envs, more chaos Branches more complicated than folders. Don't. 17 -
Repo per environment Why would you want to use one
repo per env? • Access to folders more difficult to constrain than repos • Organizational constraints, e.g. • "devs are not allowed to acces prod" • security team needs to approve releases Repos more complicated than folders. Use only when really necessary. 18 -
Folder per environment GitOps - Operations by Pull Request •
Create short-lived branches and PRs • Use folders to design envs (instead of long-lived branches per env) • Merge promotes release, triggers deployment weave.works/blog/gitops-operations-by-pull-request 19 -
Implementing release promotion Tools for separating config AKA Templating, Patching,
Overlay, Rendering? • Kustomize • plain kustomize.yaml • ≠ Flux CRD Kustomization • Helm • CRD ( Application , HelmRelease ) • Umbrella Chart • helm template via CI server 20 -
Global envs vs. env per app Global Environments ├── production
│ ├── app1 │ │ └── deployment.yaml │ └── app2 │ └── deployment.yaml └── staging ├── app1 │ └── deployment.yaml └── app2 └── deployment.yaml Environment per app ├── app1 │ ├── production │ │ └── deployment.yaml │ └── staging │ └── deployment.yaml └── app2 └── deployment.yaml e.g. Preview Envs 21 -
Branch and PR creation Who bumps versions in GitOps repo,
creates branch and PR? • Manual: Human pushes branch and create PR • Image Updater: Operator pushes branch, create PR manually • CI Server: Build job pushes branch, creates PR • Dependency Bot: Bot pushes branch, creates PR 22 -
Image updater K8s Cluster Developer Git Repo CI Server GitOps
operator OCI Registry push pull push pull + watch pull + push deploy GitOps operator can update image version in Git • • github.com/argoproj-labs/argocd-image-updater fluxcd.io/docs/guides/image-update 23 -
Promotion via CI Server K8s Cluster Developer App Repo GitOps
Repo CI Server GitOps operator OCI Registry push app +infra code pull push infra code +create PR push pull pull deploy 24 -
Promotion via dependency bot K8s Cluster GitOps operator Developer Git
Repo Renovate bot create PR watch CI Server OCI Registry push pull push pull pull deploy e.g. github.com/renovatebot/renovate 25 -
Preview environments AKA (ephemeral | dynamic | pull request |
test | temporary) environments • An environment that is created with a pull request • and deleted on merge/close ApplicationSet , using the PullRequest generator GitOpsSets 26 -
Wiring Wiring up operator, repos, folders, envs, etc. • Bootstrapping:
kubectl , operator-specific CLI • Linking/Grouping: • Operator-specific CRDs • Kustomization • Application • Nesting: App of Apps (same principle with Kustomization ) • Templating: ApplicationSets - folders, lists, config files 27 -
Example 1: Repo per team and app + CI •
Repo pattern: Per team/monorepo per app • Operator: • Features: • Automation via CI server • Mixed repo patterns • ArgoCD and Flux examples • Source: cloudogu/gitops-playground team-gitops-repo ├── production │ ├── 3rd-party-app │ └── custom-app │ ├── deployment.yaml │ └── service.yaml └── staging ├── 3rd-party-app └── custom-app ├── deployment.yaml └── service.yaml push via PR app-repo ├── k8s │ ├── production │ │ ├── deployment.yaml │ │ └── service.yaml │ └── staging │ ├── deployment.yaml │ └── service.yaml └── src push via PR push pull CI server push Developer 29 -
Demo Your Host K3d Container SCM-Manager Docker Daemon Registry Jenkins
ArgoCD Staging (6.) + Production (7.) App Repos GitOps Repos run 1. push 7. review 8. accept PR 2. pull 4. push IaC + Create PR 3. push image 5. pull deploy cloudogu/gitops-playground 30 -
BTW: More Features to explore Your Host K3d Container SCM-Manager
Docker Daemon API-Server Grafana Prometheus External Secrets Operator Vault Mailhog Registry Jenkins GitOps Operators App Repos GitOps Repos run view metrics create secret read mail push accept PR pull push IaC + Create PR push image pull read secret deploy create secret send alert send query Scrape metrics 31 -
Example 2: Ex 1 with operator • Repo pattern: Per
team/monorepo per app • Operator pattern: Hub and Spoke • Operator: ( ) • Boostrapping: Helm , kubectl • Linking: Application • Features: Env per app, operate ArgoCD with GitOps • Source: Cloudogu internal, GitOps Playground in the future argocd-repo ├── applications │ ├── teams │ │ └── team-1.yaml │ ├── control-app.yaml │ ├── argo-projects.yaml │ └── argocd.yaml ├── general │ ├── templates │ │ └── ingress.yaml │ ├── Chart.lock │ ├── Chart.yaml │ └── values.yaml └── projects ├── argo-project.yaml ├── default.yaml └── team-1.yaml team-1-gitops-repo ├── apps │ └── app1 │ ├── production │ └── staging │ ├── deployment.yaml │ └── service.yaml ├── argocd │ ├── app1-staging.yaml │ ├── app1-production.yaml │ └── misc-application.yaml └── misc └── network-policies.yaml Developer CI server Platform admin https://github.com/argoproj/argo-helm/releases/download/argo-cd-5.23.5... dependencies 33 -
Example 3: ArgoCD autopilot • Repo pattern: Monorepo • Operator
pattern: Standalone / Hub and Spoke • Operator: • Boostrapping: argocd-autopilot • Linking: Application , ApplicationSet , • Features: • Operate ArgoCD with GitOps • Opinionated structure and YAML creation via CLI • Source: argoproj-labs/argocd-autopilot argocd-repo ├── apps │ └── app1 │ ├── base │ │ └── kustomization.yaml │ └── overlays │ └── proj1 │ ├── config.json │ └── kustomization.yaml ├── bootstrap │ ├── argo-cd │ │ └── kustomization.yaml │ ├── cluster-resources │ │ ├── in-cluster │ │ │ └── argocd-ns.yaml │ │ └── in-cluster.json │ ├── cluster-resources.yaml │ ├── argo-cd.yaml │ └── root.yaml └── projects └── proj1.yaml path: **/proj1/config.json path: *.json autopilot-bootstrap www github.com/argoproj-labs/argocd-autopilot/blob/main/manifests/base/ github.com/argoproj/argo-cd/blob/stable/manifests/install.yaml 34 -
Example 4: Flux Monorepo • Repo pattern: Monorepo • Operator
pattern: Standalone • Operator: ( ?) • Boostrapping: flux • Linking: Kustomization , • Features: • Cross-cutting infra • Operate Flux with GitOps • Source: fluxcd/flux2-kustomize-helm-example#16 fluxcd.io/flux/guides/repository-structure flux-monorepo ├── apps │ ├── base │ │ ├── app1 │ │ │ ├── kustomization.yaml │ │ │ └── release.yaml │ │ └── app2 │ ├── production │ │ ├── app1 │ │ │ ├── kustomization.yaml │ │ │ └── values.yaml │ │ ├── app2 │ │ └── kustomization.yaml │ └── staging ├── clusters │ ├── production │ │ ├── flux-system │ │ ├── apps.yaml │ │ └── infrastructure.yaml │ └── staging └── infrastructure ├── configs │ └── network-policies.yaml └── controllers └── ingress-nginx.yaml patches resources 35 -
Example 5: Flux repo per team • Repo pattern: Repo
per team • Operator pattern: Standalone • Operator: ( ?) • Boostrapping: flux • Linking: Kustomization , • Features: Ex 5 with repo for team • Source: fluxcd/flux2-multi-tenancy fluxcd.io/flux/guides/repository-structure platform-admin-repo ├── clusters │ └── production │ ├── flux-system │ ├── tenants.yaml │ └── infrastructure.yaml ├── infrastructure └── tenants ├── base │ └── team1 │ ├── sync.yaml │ ├── rbac.yaml │ └── kustomization.yaml ├── production │ ├── team1 │ │ ├── path.yaml │ │ └── kustomization.yaml │ └── kustomization.yaml └── staging team1-gitops-repo ├── base │ ├── app1 │ │ ├── kustomization.yaml │ │ └── release.yaml │ └── app2 ├── production │ ├── app1 │ │ ├── kustomization.yaml │ │ └── values.yaml │ ├── app2 │ └── kustomization.yaml └── staging patches ressources Example 4 36 -
Example 6: ArgoCD and Flux alternative • Repo pattern: Monorepo
• Operator pattern: Standalone • Operator: • Boostrapping: kubectl • Linking: Application , ApplicationSet / Kustomization , • Features: • Cross-cutting infra and app(s) • ArgoCD and Flux examples • Source: C. Hernandez - The Path to GitOps christianh814/example-kubernetes-go-repo monorepo └── cluster-XXXX ├── apps │ └── myapp │ ├── kustomization.yaml │ └── myapp-deployment.yaml ├── bootstrap │ ├── base │ │ ├── argocd-ns.yaml │ │ └── kustomization.yaml │ └── overlays │ └── default │ └── kustomization.yaml ├── cluster-config │ ├── gitops-controller │ │ └── kustomization.yaml │ └── sample-admin-workload │ ├── kustomization.yaml │ └── sample-admin-config.yaml └── components ├── applicationsets │ ├── apps-appset.yaml │ ├── cluster-config-appset.yaml │ └── kustomization.yaml └── argocdproj ├── kustomization.yaml └── test-project.yaml 37 -
Example 7: Environment variations • Operator: ( ) • Features:
• Env variants for a single app • Promotion "via cp " • Source: kostis-codefresh/gitops-environment-promotion app-with-variants ├── base │ ├── deployment.yaml │ ├── kustomization.yaml │ └── service.yaml ├── envs │ ├── prod-eu │ │ ├── deployment.yaml │ │ └── kustomization.yaml │ ├── prod-us │ ├── staging-eu │ ├── staging-us │ └── qa └── variants ├── eu │ ├── kustomization.yaml │ └── region.yaml ├── us ├── prod │ ├── kustomization.yaml │ └── prod.yaml └── non-prod base component component 38 -
No such thing as the perfect GitOps process • Patterns
exist - for different aspects, inconsistent naming • Examples exist - different operators + scopes (bootstrapping vs. apps only) Use as inspiration 40 -
Johannes Schnatterer, Cloudogu GmbH • • • • Join my
team: cloudogu.com/gitops GitOps Resources Community Trainings Consulting cloudogu.com/join/cloud-engineer @[email protected] @jschnatterer 41 -
Image sources • coloured-parchment-paper background by brgfx on Freepik •
Basics: • Example: • More examples • Perfect? https://www.freepik.com/free-vector/coloured-parchment-paper-designs_1078492.htm https://pixabay.com/illustrations/blackboard-board-school-chalkboard-5639925/ https://unsplash.com/photos/X2PWhiKDQww https://unsplash.com/photos/XZc4f2XZc84 https://pixabay.com/illustrations/question-mark-question-response-1020165/ 42