ArgoCon NA 2026 - The Agent Broke It, The Agent Fixed it

Transcript

1. The Agent Broke It, the Agent Fixed It Self-healing progressive delivery on Kubernetes Carlos Sanchez Kevin Dubois Principal Scientist, Adobe Sr. Principal Developer Advocate, IBM github.com/carlossg github.com/kdubois linkedin.com/in/carlossg linkedin.com/in/kevindubois
2. We all have a “friend” who trusts AI code, perhaps a little too much 42% 1.7x 43% of code is AI-assisted more issues per AI PR of AI changes need debugging Sonar 2026: 42% of committed CodeRabbit, 470 PRs: 10.83 Lightrun 2026: in production, code is AI-generated or assisted issues vs 6.45 for humans after passing QA Sonar, 8 Jan 2026 https://www.sonarsource.com/blog/state-of-code-developer-survey-report-the-current-reality-of-ai-coding CodeRabbit, 17 Dec 2025 https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report Lightrun, 14 Apr 2026 https://venturebeat.com/technology/43-of-ai-generated-code-changes-need-debugging-in-production-survey-finds
3. Three pushes to everyone at once Cloudflare CrowdStrike Knight Capital 18 November 2025 19 July 2024 1 August 2012 One generated feature file, One rapid-channel sensor update, New code on 7 of 8 servers, pushed to every data centre pushed to every host one left on the old code 3h02m 8.5M $460M of customer impact Windows devices lost in 45 minutes First errors 11:28, core traffic $5.4B in direct losses SEC fined Knight $12M for the release normal at 14:30 across the Fortune 500 Cloudflare, 18 Nov 2025 https://blog.cloudflare.com/18-november-2025-outage CrowdStrike, 24 Jul 2024 https://www.reuters.com/technology/fortune-500-firms-see-54-bln-crowdstrike-losses-says-insurer-parametrix-2024-07-24/ SEC, 16 Oct 2013 https://www.sec.gov/newsroom/press-releases/2013-222
4. Anatomy of a bad deploy 11:05 11:28 13:05 14:30 17:06 Change deployed Impact starts Rollback begins Core traffic normal All clear Database access control Bad feature file reaches Feature file reverted Most services back Every Cloudflare change every network to earlier version 3h02m of customer impact 1h37m to rollback Source: https://blog.cloudflare.com/18-november-2025-outage system normal
5. CI was green, but production is 20% slower What CI sees zsh — quarkus-demo $ mvn verify What production sees p95 latency (ms) 50ms stable, v3 +20% [INFO] Tests run: 214, Failures: 0 60ms canary, v4 [INFO] Errors: 0, Skipped: 4 [INFO] BUILD SUCCESSFUL in 42s 0 20 40 99% success, zero error logs, no alert 60 80
6. Rollback vs recovery 1 2 3 4 Detect minutes An alert fires and someone opens the dashboard to notice Roll back seconds Argo Rollouts aborts to the last stable ReplicaSet to stop the bleeding Diagnose hours Logs, metrics, the diff, a local repro to understand Fix and ship days Patch, test, review, release to actually fix it DORA 2025: 21.3% of teams recover in under an hour, 43.4% take more than a day https://dora.dev/research/2025/dora-report/
7. Why an agent should fix it If it can write the code, it should be able to write the fix Logs from the canary p95 against stable A patch The diff you review it before it merges The agent The repo The test suite
8. Rollback ASAP, fix after DECIDE synchronous, seconds in parallel Plugin send for analysis 3 data agents promote or abort? Analysis Decision on rollback FIX asynchronous, minutes handoff A2A call run Sandbox report back Agent + tests PR or issue
9. Rollouts calls the agent as a plugin analysistemplate-ai-agent.yaml - name: ai-analysis failureLimit: 0 1 1 One failed measurement aborts count: 2 2 interval: 30s failureLimit: 0 Two measurements 2 Logs early and metrics later initialDelay: 5s provider: plugin: argoproj-labs/metric-ai: agentUrl: http://kubernetes-agent:8080 stableLabel: role=stable canaryLabel: role=canary 3 Plugin provider 3 No fork and no new CRD
10. Double & triple check uncertainty First analysis Graded by a decision model 50 below the confidence threshold Routed to a second opinion 50 still below the confidence threshold Pause, escalate to a human
11. Remediation in Agent Sandbox Sandbox Remediation needed Code assistant A pull request reads the evidence, writes a patch green opens a PR red opens an issue and runs the tests no cluster credentials and an egress allowlist Already on Kubernetes Agent Sandbox Agent Substrate Kubernetes SIG Apps, one pod per agent CNCF sandbox, many agents per node Agent Sandbox: agent-sandbox.sigs.k8s.io Agent Substrate: github.com/agent-substrate/substrate
12. The implementation Java !?!? Quarkus LangChain4j
13. Agentic workflows as typed interfaces AnalysisOrchestratorAgent.java @SequenceAgent( description = "Complete Kubernetes rollout analysis workflow", 1 @SequenceAgent 1 The annotation declares the order outputKey = "analysisResult", subAgents = { AnalysisDataAgent.class, 2 Three sub-agents 2 DataCombinerAgent.class, Two fetch data and one decides AnalysisAgent.class } ) ResultWithAgenticScope<AnalysisResult> execute( String message, String namespace, String stableSelector, String canarySelector 3 The result is a record 3 fields the gate can score
14. Demo time !!
15. What you just saw (hopefully)
16. All open source! THANK YOU argoproj-labs/rollouts-plugin-metric-ai The Argo Rollouts metric provider that calls it Keep the rollout Add the loop github.com/kdubois/kubernetes-aiops-agent The analysis agent and the LangChain4j workflow github.com/kdubois/remediation-agent Sandbox, coding agent, pull request or issue Kevin Dubois github.com/kdubois linkedin.com/in/kevindubois Carlos Sanchez github.com/carlossg linkedin.com/in/carlossg Session feedback Scan and rate the talk
17. Sources Cloudflare postmortem, 18 Nov 2025 blog.cloudflare.com/18-november-2025-outage CrowdStrike losses: Parametrix via Reuters reuters.com/technology/fortune-500-firms-see-54-bln-crowdstrike-losses-says-insurer-parametrix SEC order against Knight Capital www.sec.gov/newsroom/press-releases/2013-222 DORA 2025 report dora.dev/research/2025/dora-report/ Sonar, State of Code 2026 (Jan 2026) www.sonarsource.com/blog/state-of-code-developer-survey-report-the-current-reality-of-ai-coding CodeRabbit, AI vs human code coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report Lightrun survey, via VentureBeat venturebeat.com/technology/43-of-ai-generated-code-changes-need-debugging-in-production-survey-finds Quarkus benchmarks vs Spring Boot quarkus.io/blog/new-benchmarks Agent Sandbox, Kubernetes SIG Apps agent-sandbox.sigs.k8s.io Agent Substrate, CNCF sandbox github.com/agent-substrate/substrate