Share
Embed
Agents on Rails. SF_Ruby Meetup Oct 29 2026
Irina Nazarova
November 02, 2026
Transcript
19 slides
-
Agents on Rails What 1,485 runs taught us, and do
types help? Irina Nazarova · Evil Martians SF Ruby · October 29, 2026 Hosted by -
SWE-bench Multilingual 300 GitHub issues · 41 repos · 13
models from February 2026 · same agent 65% 67% Rust JavaScript Java vs PHP C C++ No difference: +2.5 points, 95% interval −5 TYPED DYNAMIC to +9 Steps per task: dynamic ~9% fewer, Ruby fewest of nine Not conclusive: interval −20% to +3% 71% 71% Go 70% 70% 69% 67% 62% 54% 53% Python* 71% Ruby TypeScript Rust → Go: 18 points. Both typed. *SWE-bench Verified, same agent and 11 models: different, easier, more memorized tasks -
JavaScript vs TypeScript Same ecosystem, types on or off ·
TS minus JS solve rate · three benchmarks with per-task results BENCHMARK RAW GAP AFTER DIFFICULTY AND REPO SWE-bench Multilingual · 13 models Multi-SWE-bench · 36 setups SWE-PolyBench · 4 submissions −16 pts · 15 TS tasks none: 4 TS tasks no model solved TS ahead in 8 of 36 odds ratio 1.04 (0.64–1.70) 19/195 TS ahead in 24 of 36 TS ahead in 3 of 4 TypeScript agent runs that ever called tsc The sign flips with a few repos in every benchmark. The test runners strip types without checking them. -
The project One benchmark, 46 tasks 19 atomic tasks ·
Writebook 20 feature tickets · Fizzy 7 multi-step tasks · Mastodon, Redmine, Forem, Consul, Campfire, 2 new apps Same agent for every model: miniswen, one bash tool Hidden Ruby tests grade every run -
Which model should you pay for? 46 tasks · 13
model configurations · mean task score · rubyonrails.org/ai, late October Mean task score Opus 5.5 xhigh 4 / 4 Opus 5.5 medium 60% Find your model Astra xhigh Gemini 4 Argon high Every configuration, named. Cheaper is to the right; higher is better. Sol xhigh 50% Astra medium Grok 4.7 high DeepSeek Flash max Muse Wonder xhigh 40% Muse Spark max Kimi K3 max GLM Flash max $10 $3 • Model at its benchmark effort ┄ Best score at each price Opus 5.5 xhigh 65.3% Sol xhigh 60.8% Luna max 43.4% $10.53 / run Luna max $1 50¢ 20¢ Average cost per run · cheaper → $0.80 / run $0.19 / run -
Cheap models keep up on small tasks Mean task score
by task type · rubyonrails.org/ai, late October MODEL COST PER RUN ATOMIC · 19 FEATURE · 20 MULTI-STEP · 7 Opus 5.5 · xhigh Sol · xhigh Gemini 4 Argon · high $10.53 $0.80 $6.12 $0.19 $0.95 $0.71 69% 72% 59% 53% 53% 31% 25% 22% 72% 52% 26% Luna · max DeepSeek Flash · max GLM Flash · max 81% 71% 73% 68% On one ticket, the field is close. Over several sessions, it splits. 6% 7% 4% -
The grader is a Ruby test Fizzy · stalled cards
· September runs · 29 runs reached grading TICKET · FIZZY A card that had a burst and then sees no activity for two weeks is stalled. […] Any activity on the card should reset it. # ft-card-stalled-quiet-clock/verification_test.rb test "ticking off a step clears a stalled card" do card = comment_burst_on(card_named("Steps, not comments")) step = with_current_user(@author) do card.steps.create!(content: "Ship it") end travel 30.days assert_includes stalled_listing, card.title with_current_user(@author) { step.update!(completed: true) } assert_not_includes stalled_listing, card.title end 29/29 stalled after 30 quiet days 5/29 completing a step clears the stalled status -
Failure modes of agents on Rails Stage 2 · 20
feature tickets on Fizzy · about 1,500 runs · 🔒 security Presentation controllers · views Sign-in · Account enumeration Turbo · Morph glitches I18n · Missing or mixed translations Views · .count ignores preloads Application Webhooks · Events lost on restart Tenant cleanup · Leftover data, no retry Domain Active Record · Incomplete preloads Callbacks · Wrong activity timestamp Action Text · HTML leaks into search Active Storage · Full recount on upload jobs · use cases models · scopes Infrastructure database · HTTP Push URLs · SSRF SQL sweeps · Full-table scans Webhook URLs · SSRF Migrations · Data loss, broken rollback -
Nobody runs migration generators Generators created migrations in just 23
of 2,670 public runs. Fake a timestamp 541/873 migration-writing runs The newest timestamp + 1–2 days, even months later. Delete valid rows 27/64 account-ID safety checks The backfill missed an owner, then deleted the row as an “orphan.” Test up → down → up on seeded data. Check that the data survives. Block rollback 9/64 account-ID safety checks The migration raised IrreversibleMigration . -
Jailbreak GLM 5.3 Flash “JACKPOT!” Read class and method names
from Bootsnap’s cache to infer the missing feature. -
Ruby vs Rust Campfire: add chat bots to an existing
app. Quizzy: build an app from scratch. Experiments run by Vladimir “Vova” Dementyev -
Rails vs Rust: the bill Campfire · existing app ·
2 models solved both 2–3.6× Quizzy · from scratch · 8 configurations per stack 6/8 vs 4/8 the agent cost on the Rust port finished every required check · Rails vs Rust ~2× 1.22× the agent time Similar completion scores: 0.84–0.88 total agent cost on Rust Rust had twice the CPU and memory, so no speed claim -
What did missing types cost? Every error a type checker
would catch, in every Ruby vs Rust session · 21 Ruby, 12 Rust Ruby · no types ~1% Rust · compiler 3.5% of agent time spent on type errors of agent time spent fixing compile errors 30 slips, a median of 2 turns and 15 seconds each. 5 were in app code. The rest were in the agents' own tests. 63 of 577 cargo runs failed to compile. It caught real mistakes, like a match that forgot the new Role::Bot . No failed check on either stack came from a type error. -
Build your own bench Your git history already holds the
tasks · pick the hard ones A bug or feature you shipped → tasks/<name>/ The commit before the fix → environment/Dockerfile · git checkout FIX^ The issue, word for word → instruction.md The tests that cover it today → verification_test.rb hidden until grading The fix you merged → solution.patch the oracle must score 1.0 gem install lemans · lemans run --agent oracle → 1.0 · then --model a --model b -k 3 -
01 02 03 Shop smarter. Build a harness for the
failure modes. Test the myths. Run your own evals. Start with lemans. -
SF Ruby Conference, November 10–12 $290 LATE BIRD, WITH MY
CODE $500 IRASPECIAL luma.com/sfrubyconf2026?coupon=IRASPECIAL SFJAZZ · two days of talks and a community day · sfruby.com -
Thank you, SF Ruby rubyonrails.org/ai · github.com/rails/lemans · github.com/rails/ai-evals evilmartians.com/chronicles
· x.com/inazarova Thanks to Persona for hosting · Rails Foundation · Amanda Perino Built by Svyatoslav Kryukov, Artur Petrov, Vladimir Dementyev, Alexander Baygeldin, Aibek Izhanov, Anton Senkovskiy and Irina Nazarova