Measuring Engineers When AI Writes the Code

An engineering team at Amazon left an unreleased AI tool running in a loop for five months. The bill came to roughly $1.8 million. Nobody caught it, not because the monitoring didn't exist, but because nobody was watching cost. They were watching an adoption leaderboard.

Leadership had asked for AI adoption. So the org measured usage. So people optimized for usage. Everyone behaved rationally inside the system they were given, and the system was badly designed.

That's not an AI story. That's Goodhart's Law, which states that when a measure becomes a target, it stops being a good measure. What's new is how fast AI lets a bad target compound.

The metrics AI broke

Nicole Forsgren, who created both the DORA and SPACE frameworks for measuring engineering performance, put it plainly in late 2025: AI broke our developer productivity metrics. Lines of code, meaningless. Commits, not the point.

Here's why, specifically. Every traditional productivity proxy measured effort and used it to infer output. That inference held as long as a human typed every character. It doesn't hold anymore.

The same shipped feature can now come from an engineer who wrote every line themselves or one who directed an agent and reviewed the result. Effort and output have decoupled. Activity metrics cannot tell those two engineers apart, and the one doing the more valuable work, the careful direction and rigorous review, often generates less measurable activity.

Lines of code has always been a bad metric, since roughly the 1970s. An engineer who solves a problem in 40 lines produces more value than one who takes 400. A refactor that deletes 2,000 lines of dead code and cuts deploy time shows up as negative productivity. Once it became a target, functions sprawled and deletion became risky. AI didn't break this metric. It made it absurd, because now anyone can generate 400 lines in a minute.

The new metrics are worse

Here's the trap most organizations are walking into right now. Having lost the old proxies, they reach for AI-specific ones: suggestion acceptance rate, tokens consumed, percentage of code AI-generated, seats actively used.

Think about what an acceptance-rate target actually incentivizes. It rewards accepting AI suggestions. Not evaluating them, accepting them. The engineer who carefully rejects three bad suggestions and accepts one good one scores worse than the engineer who accepts everything. You have built a direct incentive to stop reviewing, and you'll pay for it in code churn and technical debt six months later, long after the dashboard showed success.

Token consumption is worse still. It's a cost being used as a performance metric, which is how you get a $1.8 million loop that everyone was too busy celebrating to question.

Any metric that measures how much AI you used is measuring the wrong noun. Nobody's goal is to use AI. The goal is working software, and AI is one means to it.

Counter-metrics, or nothing

The practical antidote to Goodhart's Law isn't finding one perfect metric. It's refusing to use any metric in isolation.

Pair every measure with the thing it would naturally trade against:

• Any speed metric paired with a quality metric. Throughput next to change failure rate. Otherwise you're rewarding shipping breakage.
• Any quantity metric paired with a cost metric. Volume of AI-assisted work next to spend. This alone would have caught Amazon's loop.
• Any quantitative metric paired with something qualitative. Numbers next to what engineers actually say in one-on-ones about whether the work feels better or worse.

A single number can always be gamed, usually without anyone consciously deciding to game it. A pair that trades off against each other is much harder to distort, because moving one moves the other in a way that becomes visible.

What to actually measure

The honest answer is that it's mostly what you should have been measuring before: whether working software reaches users reliably, and whether the people building it can sustain doing so.

DORA's four measures still hold up because none of them care how the code was produced. Deployment frequency, lead time for changes, change failure rate, and time to restore service all measure outcomes at the system level, which is exactly the level AI adoption is stressing.

Add one more question that no dashboard will answer for you: can this team still explain what it shipped? If an engineer can't explain why the code they merged works, you don't have a productivity gain. You have an unreviewed dependency with your company's name on it.