August 26, 2026
They Were 19% Slower and Certain They Were Faster
In 2025, the research group METR ran a randomized controlled trial on experienced open-source developers. Sixteen of them, working 246 real issues in large repositories they already maintained.
Before starting, they predicted the AI tools would make them roughly 24% faster.
They were 19% slower.
Afterwards, asked what had happened, they estimated AI had made them about 20% faster. They had just lived through the opposite and could not perceive it.
Be careful with this finding, because its limits are real. Sixteen developers is a small sample. They were experts in codebases they knew deeply, which METR explicitly say is not representative of most software work, using early-2025 tooling they had limited practice with. The authors themselves say the result shouldn't be generalized, and they revised the experiment design in early 2026. Don't walk into your next meeting announcing that AI makes engineers slower.
The gap between prediction, measurement, and memory is the durable part, and that gap isn't really about AI.
Why nobody noticed
Nothing here requires anyone to be careless. The mechanism is ordinary.
AI removes the friction you notice and adds friction you don't. Staring at an empty file is memorable effort. It feels like work, and it's exactly the part that gets eliminated. Reading generated code, checking whether it does what it claims, finding that it subtly doesn't, and fixing it is diffuse effort spread across many small moments. None of it registers as an event you'd recall later.
So the felt experience is "that was easier." Easier and faster are different measurements, and people routinely report the less effortful path as the quicker one.
There's also no control group for your own week. You never see the version of yourself who did the same task without the tool, so there's nothing to compare against except an impression, and the impression is shaped by whatever everyone around you is currently saying.
What this breaks
Most engineering organizations run almost entirely on self-report. Standups, retros, one-on-ones, status updates. All of it is people telling you how it's going.
That's always been imperfect. This particular class of question has a much wider error bar than most managers are treating it as having.
Three consequences worth taking seriously:
Enthusiasm isn't evidence. A team reporting that AI has transformed their productivity is reporting a real feeling. They are not reporting a measurement, and the feeling may point the wrong way.
Skepticism isn't evidence either. The engineer certain it's useless is self-reporting too, with the same reliability.
Confidence grows as it travels upward. Surveys consistently find executives far more confident about AI than the people using it. The further you sit from the keyboard, the more your view is assembled from other people's impressions, which were already unreliable.
What to do instead
You don't need a research lab. You need to stop treating "how's it going" as data.
Watch outcomes rather than sentiment. Lead time and change failure rate have no opinion about AI, and a genuine improvement eventually shows up in them. Compare the same team on the same class of work before and after, not one team against another whose work differs.
Then change the question you ask. "Did AI help this week" returns a vibe. "Walk me through the last thing it genuinely saved you time on, and the last thing it cost you time on" returns two concrete events you can actually reason about. Almost nobody volunteers the second one unprompted, because chasing a plausible-looking hallucination for an hour doesn't feel like a distinct event worth mentioning.
Including you
If you hold a firm view about whether AI is helping your organization, and that view rests mainly on impressions and what people say in meetings, you are relying on the exact instrument this study found unreliable.
Being the leader doesn't exempt you from it. It usually means you're further from the work than anyone else, and more dependent on secondhand accounts.
This is the third session of the Leading Through AI Adoption module I teach. The full lesson, and the four sessions around it, are free to read: The AI Perception Gap.