KubeCon Japan - AIOps: (near) Zero-Touch Production Rollout Fixes

1. #KubeCon #CloudNativeCon AIOps: (near) Zero-Touch Production Rollout Fixes Kevin Dubois, IBM Carlos Sanchez, Adobe
2. kevindubois Kevin Dubois ★ ★ ★ ★ ★ ★ carlossg Sr. Principal Developer Advocate at Java Champion Technical Lead, CNCF DevEx TAG Agentic AI Foundation Ambassador From Belgium 󰎐 / Live in Switzerland󰎤 🗣 English, Dutch, French, Italian @kevindubois.com @thekevindubois kevindubois kdubois
3. kevindubois carlossg
4. kevindubois carlossg
5. kevindubois What could have mitigated the impact? carlossg
6. Progressive Delivery kevindubois carlossg continuous analysis continuous analysis
7. kevindubois carlossg GitOps Git is the single source of truth Treat everything as code Operations through Git workflows
8. kevindubois carlossg GitOps Application Delivery Model Source Git Repository Image Registry Pull Request CD Config Git Repository Push Pull Monitor Detect drift Deploy Take action Kubernetes
9. kevindubois carlossg By default: a big bang / all or nothing release
10. kevindubois carlossg Argo Rollouts
11. kevindubois carlossg Increase percentage of new (canary) version
13. kevindubois apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 10s successCondition: len(result) == 0 || result[0] >= 0.95 failureLimit: 2 provider: prometheus: address: https://internal:demo@prometheus.istio-system.svc.cluster .local:9090 query: | sum(irate(istio_requests_total{ reporter="source", destination_service=~"{{args.service-name}}", response_code!~"5.*"}[30s]) ) carlossg
14. kevindubois apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 10s successCondition: len(result) == 0 || result[0] >= 0.95 failureLimit: 2 provider: prometheus: address: https://internal:demo@prometheus.istio-system.svc.cluster .local:9090 query: | sum(irate(istio_requests_total{ reporter="source", destination_service=~"{{args.service-name}}", response_code!~"5.*"}[30s]) ) carlossg
15. kevindubois carlossg
16. Title kevindubois carlossg
17. Title kevindubois carlossg
18. kevindubois carlossg apiVersion: argoproj.io/v1alpha1 kind: RolloutManager metadata: name: argo-rollouts spec: plugins: metric: - name: argoproj-labs/metric-ai location: https://github.com/argoproj-labs/rollouts-plugin-metric-ai/releases/ download/v0.0.1/rollouts-plugin-metric-ai-linux-amd64
19. kevindubois carlossg canary: analysis: startingStep: 1 templates: - templateName: canary-analysis-ai-agent canaryMetadata: labels: role: canary stableMetadata: labels: role: stable
20. kevindubois carlossg apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: canary-analysis-ai-agent spec: metrics: - interval: 10s name: success-rate provider: plugin: argoproj-labs/metric-ai: agentUrl: http://kubernetes-agent:8080 stableLabel: role=stable canaryLabel: role=canary extraPrompt: ignore aesthetic changes successCondition: result > 0.50
21. kevindubois carlossg
22. kevindubois carlossg quality check loop async quality check loop
23. kevindubois carlossg apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: canary-analysis-ai-agent spec: metrics: - interval: 10s name: success-rate provider: plugin: argoproj-labs/metric-ai: agentUrl: http://kubernetes-agent:8080 stableLabel: role=stable canaryLabel: role=canary extraPrompt: ignore aesthetic changes githubUrl: …github.com/kdubois/argo-rollouts-quarkus-demo
24. kevindubois carlossg Demo
25. kevindubois carlossg Lessons learned: Performance was an interesting challenge: went from one AI service to agentic system: parallel & async agents LLM choice + “context engineering” + tool calling especially for PR creation Complexity vs portability (could’ve used Serverless MCP, external code assistant for PR creation, async remote agents, etc.)
26. kevindubois carlossg Takeaways: Rolling out changes to all users at once is risky Canary rollouts and feature flags are safer AI Agents can automate the loop by analyzing metrics and logs, and even proposing fixes for the failures
27. kevindubois carlossg
28. kevindubois Thanks carlossg github.com/argoproj-labs/rollouts-plugin-metric-ai/ github.com/kdubois/argo-rollouts-quarkus-demo github.com/kdubois/kubernetes-aiops-agent github.com/kdubois/progressive-delivery https://quarkus.io/quarkus-workshop-langchain4j/ feedback please :) carlossg kevindubois csanchez.org kevindubois.com carlossg kdubois