Self Healing Rollouts: Automating Production Fixes with Agentic AI

Transcript

1. Self-Healing Rollouts Automating Production Fixes With Agentic AI Kevin Dubois Sr Principal Developer Advocate, IBM @kevindubois
2. Kevin Dubois ★ ★ ★ ★ ★ ★ Sr. Principal Developer Advocate at Java Champion Technical Lead, CNCF DevEx TAG Agentic AI Foundation Ambassador From Belgium 󰎐 / Live in Switzerland󰎤 🗣 English, Dutch, French, Italian @kevindubois.com @thekevindubois kevindubois kdubois @kevindubois
3. @kevindubois
4. @kevindubois
5. What could have mitigated the impact? @kevindubois
6. Progressive Delivery @kevindubois
8. GitOps Git is the single source of truth @kevindubois Treat everything as code Operations through Git workflows
9. GitOps Application Delivery Model Source Git Repository Image Registry Pull Request CD Config Git Repository Push Pull Monitor Detect drift Deploy Take action @kevindubois Kubernetes
10. By default: a big bang / all or nothing release @kevindubois
11. Argo Rollouts @kevindubois
12. Increase percentage of new (canary) version @kevindubois
13. * RIP Tim Curry
14. apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 10s successCondition: len(result) == 0 || result[0] >= 0.95 failureLimit: 2 provider: prometheus: address: https://internal:demo@prometheus.istio-system.svc.cluster .local:9090 query: | sum(irate(istio_requests_total{ reporter="source", destination_service=~"{{args.service-name}}", response_code!~"5.*"}[30s]) ) @kevindubois
15. apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 10s successCondition: len(result) == 0 || result[0] >= 0.95 failureLimit: 2 provider: prometheus: address: https://internal:demo@prometheus.istio-system.svc.cluster .local:9090 query: | sum(irate(istio_requests_total{ reporter="source", destination_service=~"{{args.service-name}}", response_code!~"5.*"}[30s]) ) @kevindubois
16. @kevindubois
17. Quarkus LangChain4j - AI for Prod & Performance Application Quarkus LangChain4j LangChain4j LLMs @kevindubois Vector stores - Embedding Models Declarative clients CDI integration Observability (Otel, Prometheus) Auditing Resilience RAG building blocks Mockable Dev Services & Dev UI
18. Title @kevindubois
19. apiVersion: argoproj.io/v1alpha1 kind: RolloutManager metadata: name: argo-rollouts spec: plugins: metric: - name: argoproj-labs/metric-ai location: https://github.com/argoproj-labs/rollouts-plugin-metric-ai/releases/ download/v0.0.1/rollouts-plugin-metric-ai-linux-amd64 @kevindubois
20. canary: analysis: startingStep: 1 templates: - templateName: canary-analysis-ai-agent canaryMetadata: labels: role: canary stableMetadata: labels: role: stable @kevindubois
21. apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: canary-analysis-ai-agent spec: metrics: - interval: 10s name: success-rate provider: plugin: argoproj-labs/metric-ai: agentUrl: http://kubernetes-agent:8080 stableLabel: role=stable canaryLabel: role=canary extraPrompt: ignore aesthetic changes successCondition: result > 0.50 @kevindubois
22. @kevindubois
24. apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: canary-analysis-ai-agent spec: metrics: - interval: 10s name: success-rate provider: plugin: argoproj-labs/metric-ai: agentUrl: http://kubernetes-agent:8080 stableLabel: role=stable canaryLabel: role=canary extraPrompt: ignore aesthetic changes githubUrl: …github.com/kdubois/argo-rollouts-quarkus-demo @kevindubois
25. Demo @kevindubois
26. Lessons learned: Performance was an interesting challenge: went from one AI service to agentic system: parallel & async agents LLM choice + “context engineering” + tool calling especially for PR creation Complexity vs portability (could’ve used Serverless MCP, external code assistant for PR creation, async remote agents, etc.) @kevindubois
27. Takeaways: Rolling out changes to all users at once is risky Canary rollouts and feature flags are safer AI Agents can automate the loop by analyzing metrics and logs, and even proposing fixes for the failures @kevindubois
28. @kevindubois
29. Thanks github.com/argoproj-labs/rollouts-plugin-metric-ai/ github.com/kdubois/argo-rollouts-quarkus-demo github.com/kdubois/kubernetes-aiops-agent github.com/kdubois/progressive-delivery https://quarkus.io/quarkus-workshop-langchain4j/ kevindubois @kevindubois kevindubois.com kdubois speakerdeck.com/kdubois slides