In independent testing, Devin completed just 3 of 20 real-world tasks (15%)
Quick Answer
Devin caused a low-severity (3.3/10) other failure: In independent testing, Devin completed just 3 of 20 real-world tasks (15%). The root cause was confidence miscalibration. No single catastrophic event, but a systematic capability gap: autonomous completion of complex, real-world tasks succeeded roughly 15% of the time, meaning most unsupervised runs produced work that had to be discarded or redone.
Description
Marketed as 'the first AI software engineer,' Devin's autonomous real-world reliability looked very different from its demo reel. In an independent evaluation by Answer.AI, Devin was assigned 20 real tasks: only 3 succeeded, 14 failed outright, and 3 were inconclusive — about a 15% success rate. That tracks with its 13.86% score on SWE-Bench Verified at launch. The gap illustrates a systemic pattern with autonomous coding agents: impressive on curated, well-scoped benchmark tasks, but on messy real-world work most unsupervised runs produce output that must be discarded or heavily corrected. The failure here isn't one dramatic incident — it's the quiet, systematic unreliability that marketing obscures.
Instruction Given
Autonomously complete 20 assigned real-world engineering tasks.
Expected Behavior
Complete tasks as marketed for an 'autonomous AI software engineer.'
Actual Behavior
In an independent Answer.AI evaluation of 20 tasks, only 3 succeeded, 14 failed outright, and 3 were inconclusive — a ~15% real-world success rate, far below the impression left by curated benchmark demos.
Impact / Damage
No single catastrophic event, but a systematic capability gap: autonomous completion of complex, real-world tasks succeeded roughly 15% of the time, meaning most unsupervised runs produced work that had to be discarded or redone.
Frequently Asked Questions
What happened in incident STUPID-2026-0039? ▾
Marketed as 'the first AI software engineer,' Devin's autonomous real-world reliability looked very different from its demo reel. In an independent evaluation by Answer.AI, Devin was assigned 20 real tasks: only 3 succeeded, 14 failed outright, and 3 were inconclusive — about a 15% success rate. That tracks with its 13.86% score on SWE-Bench Verified at launch. The gap illustrates a systemic pattern with autonomous coding agents: impressive on curated, well-scoped benchmark tasks, but on messy real-world work most unsupervised runs produce output that must be discarded or heavily corrected. The failure here isn't one dramatic incident — it's the quiet, systematic unreliability that marketing obscures.
Which AI agent caused this failure? ▾
Devin was responsible for this other incident, documented as STUPID-2026-0039 in the StupidLLM AI agent incident database.
How severe was this AI agent failure? ▾
It is rated 3.3/10 (low) on StupidLLM's CVSS-style severity scale for AI agent failures, based on damage type, reversibility, and scope.
What was the root cause? ▾
The root cause was classified as confidence miscalibration. Complete tasks as marketed for an 'autonomous AI software engineer.'
What was the impact or damage? ▾
No single catastrophic event, but a systematic capability gap: autonomous completion of complex, real-world tasks succeeded roughly 15% of the time, meaning most unsupervised runs produced work that had to be discarded or redone.