The 2025 METR study: developers thought they were 20% faster, they were 19% slower
A controlled trial with experienced developers reveals a striking gap between perception and measurement. AI's gain is neither automatic nor free: it has to be measured.
“We’re saving a huge amount of time with AI.” That is what you hear everywhere. One research organization bothered to measure it instead of asking people how they felt. The result runs against the grain: developers believed they were 20% faster with AI. They were actually 19% slower.
What the study measured
In 2025, the research organization METR ran a rare controlled trial: experienced open-source developers, on their own projects, on their real tickets, with and without AI assistance. The published results leave little room for doubt:
- Before the experiment, they estimated AI would make them about 24% faster.
- After the fact, they still believed they had gained about 20% of their time.
- In reality, they had been 19% slower when AI assistance was allowed.
The most unsettling part is not the negative number itself. It is the gap between perception and measurement: these developers were genuinely convinced they were speeding up, at the very moment they were slowing down.
A necessary caveat
One study does not make a universal truth. The sample is small, and it involves highly experienced developers working on code they already know by heart, which is the least favorable context for AI to help. In other contexts, especially unfamiliar code or less experienced developers, the gain is often real. But that is exactly where the lesson lies.
AI’s gain is neither automatic nor visible to the naked eye
It depends on the task, the context, the level of expertise, and above all: it has to be measured. Without measurement, you steer by feel, and feel can be wrong in either direction.
This is exactly why I never start an engagement with the question “which tool should we deploy.” I start with: where do the hours go today, how much does this specific step actually cost, and by how much can we realistically reduce it. We cost it out before automating. Sometimes the answer is “deploy AI right here.” Sometimes the honest answer is “definitely not there, not in this context.”
Artificial intelligence is not magic. It is measurable. And what is never measured does not improve: it just gets talked about, often in good faith, and often wrongly.