GitOps Continuous Delivery at scale with Flux

A deep dive into the Flux project scaling features from KubeCon Paris maintainers track

Transcript

1. GitOps Continuous Delivery at scale with Flux Stefan Prodan Flux Core Maintainer Principal Engineer @ ControlPlane.io KubeCon EU Paris 2024
2. What is Flux?
3. Flux is the foundational layer for Continuous Delivery platforms
4. Flux is a Kubernetes extension 13 CRDs 6 Controllers
5. © 2024 Cloud Native Computing Foundation 5 Workload definition Helm, Kustomize and plain Kubernetes manifests Desire state acquisition Git, OCI Artifacts, S3-compatible storage Reconciliation Cluster state drift detection and correction Observability Events, metrics, logs, alerts, traces Flux API constructs
6. Flux adapts to your organizational structure and growth velocity
7. © 2024 Cloud Native Computing Foundation 7 Microservices proliferation Apps made out of many moving pieces each with its own release cadence and delivery workflow SLAs & High availability & Disaster Recovery Multi-AZ, Multi-region app instances Security constraints Multi-tenancy, Network isolation, GDPR Business expansion New apps are developed, while legacy apps still require maintenance CD growth factors
8. We’ll need to deploy thousands of workloads over hundreds of clusters 😱 Can Flux do it?
9. © 2024 Cloud Native Computing Foundation 9 Flux scaling strategies Source Optimizations Controller Fine Tuning Vertical Scaling Horizontal Scaling & Sharding
10. © 2024 Cloud Native Computing Foundation 10 Optimizations & Fine Tuning Source optimizations - Migrate from Helm HTTP/S repositories to OCI - Use dedicated Git repositories for Flux - Split the Kubernetes resources into multiple Flux Kustomizations to take advantage of concurrent reconciliations Controller fine tuning - Use persistent storage for Flux internal artifacts - Use tmpfs for in-memory kustomize builds
11. © 2024 Cloud Native Computing Foundation 11 Vertical scaling Resource limits - Increase controller CPU and Memory limits - Increase the controller reconciliation concurrency level based on CPU limits Rate limits - Monitor Kubernetes API requests rate limits error rate and adjust the reconciliation concurrency level https://fluxcd.io/flux/installation/configuration/vertical-scaling
12. Flux “Mean Time To Production” benchmark Starting with Flux v2.2, we have set up benchmarks that measure Mean Time To Production (MTTP). The MTTP benchmark measures the time it takes for Flux to deploy application changes into production. Specs: GitHub hosted-runner ubuntu-latest-16-cores Kubernetes KinD v1.28.0 / 3 nodes Flux source-controller 1CPU / 1Gi / concurrency 10 Flux kustomize-controller 1CPU / 1Gi / concurrency 20 Flux helm-controller 2CPU / 1Gi / concurrency 10 https://github.com/fluxcd/flux-benchmark
13. © 2024 Cloud Native Computing Foundation 13 Horizontal scaling To enable horizontal scaling, each controller can be deployed multiple times with a unique label selector which is used as the sharding key. To spread the load between controller instances, the Flux resources can be assigned to a particular instance using the sharding.fluxcd.io/key label. Sharding strategies - Multi-tenant (instance per tenant) - Multi-cluster (instance per cluster group) https://fluxcd.io/flux/installation/configuration/sharding
14. Flux sharding for multi-cluster delivery
15. Flux Roadmap 2024 https://fluxcd.io/roadmap
16. © 2024 Cloud Native Computing Foundation 16 Flux v2.3 (Q2 2024) Promote to GA the Flux Helm APIs and the Flux Helm functionalities. - HelmRepository - HelmChart - HelmRelease Flux v2.4 (Q3 2024) Promote to GA the Flux image automation APIs and S3-compatible storage API. - ImageUpdateAutomation - ImagePolicy - ImageRepository - Bucket General Availability
17. © 2024 Cloud Native Computing Foundation 17 Notary Project integrations Flux v2.3 scheduled for end of April 2024 will come with support for verifying the signatures of OCI Artifacts using Notation trust policies and CA certificates. Flux APIs - HelmChart v1 GA - OCIRepository v1beta2 Big thanks to Jason and Jagpreet from Microsoft for contributing this feature and for helping the Flux team maintaining the Notary integration on the long-run.
18. © 2024 Cloud Native Computing Foundation 18 CDEvents integrations Flux v2.3 will come with support for verifying and triggering workload reconciliation based on received CDEvents. In future releases, the Flux controllers will emit events in the CDEvents format thus enabling better interoperability will other CI/CD tools. Big thanks to Adam from Ericsson for contributing the CDEvents receiver integration and for continuing to expand CDEvents support to Flux own events.
19. © 2024 Cloud Native Computing Foundation 19 Helm OCI improvements Flux v2.3 will come with support for HelmReleases to refer to OCIRepositories, as an alternative to HelmRepositories/HelmCharts. - Better UX when debugging Helm releases - Allow reusing the same chart between releases - Allow pinning Helm charts by OCI digests - Allow automating Helm upgrades using semver ranges scoped to release candidates only Big thanks to Soule from ControlPlane for contributing the Helm OCI enhancements.
20. © 2024 Cloud Native Computing Foundation 20 We invite the community to help us shape the future of Flux. We have in place an RFC process for new features and enhancements proposals. We are keen to work with the community on RFCs and drive the project forward in a sustainable manner. We want to enable community members to take full ownership of Flux features and share the responsibility of feature stability and longer-term maintenance. https://github.com/fluxcd/flux2/tree/main/rfcs Flux needs your help