The debate over whether AI coding agents actually make developers faster misses the more useful question. Even holding speed roughly constant, agents change what a developer can produce in a fixed block of time, both in quality and in volume. Would you rather have someone spend 14 hours on a B+ or 2 hours on an A-? Once you frame it that way, the case for coding agents comes down to what a developer does with the 12 hours they get back, not how many minutes were shaved off a task.
Most of the debate about AI coding agents gets stuck on a single number: how much faster do they make you. Doug Llewellyn, CEO of Data Society, thinks that’s the wrong way to frame the tradeoff:
“Would you rather have the person spend 14 hours writing some lines of code and have it be a B+, or 2 hours writing the same lines of code with a coding agent and have it be an A-? You’d rather have that, because then they can write more code and produce more.”
Doug Llewellyn, Chief Executive Officer, Data Society
The real comparison isn’t whether the AI-assisted version is better than the human-only one, but which way of spending a developer’s day wins: one where the output is good and slow, and one where it’s better and fast. Framed that way, what matters isn’t hours saved; it’s what the developer does with the twelve hours that just came back.
The speed side of this argument has real data behind it, though it’s less uniform than the pitch decks suggest. A randomized controlled study of GitHub Copilot found developers completed a standardized coding task 55.8% faster than a control group working without it. That’s a large, well-documented effect, and it’s the number most vendors lead with.
It’s not the whole picture, though. A 2025 randomized trial from METR found the opposite result in a different setting: experienced open-source developers working in large, familiar codebases were actually 19% slower when using AI tools on real tasks, not the toy problems used in most benchmarks. Strikingly, those same developers estimated afterward that AI had made them about 20% faster. The gap between how fast a coding agent feels and how fast it actually is depends heavily on the task: a short, well-specified task in an unfamiliar or greenfield context is exactly where agents shine, while navigating a large, idiosyncratic codebase the developer already understands is where the benefit gets murkier or disappears.
The quality side of Llewellyn’s question holds up more consistently. In GitHub’s own research on code quality, developers using Copilot were 53.2% more likely to pass all of a set of unit tests on the first attempt, and code written with Copilot averaged 18.2 lines of code per error versus 16.0 without it. Blind reviews of the resulting code found statistically significant improvements in readability (3.62%), reliability (2.94%), maintainability (2.47%), and conciseness (4.16%), and reviewers approved Copilot-assisted code 5% more often than unassisted code.
Put those two bodies of research together and Llewellyn’s framing gets sharper: the speed gain is real but situational, while the quality gain shows up more consistently across contexts. That’s a stronger case for coding agents than “it’s faster,” and a more honest one. It also explains why some teams report huge wins and others report none: they’re often measuring different tasks under the same benchmark.
The last piece of Llewellyn’s quote, “they can write more code and produce more,” is where the ROI argument actually lives. A B+ in 14 hours and an A- in 2 hours aren’t just different quality scores. They’re different amounts of a developer’s finite time consumed by one piece of work. The dollar value of a coding agent isn’t just the two hours it saved on this ticket, it’s the eleven or twelve hours that ticket no longer occupies, and what the team chooses to do with them: more features, more test coverage, more of the maintenance work that otherwise gets deferred until it becomes technical debt.
That reframing also raises the harder management question, and it’s one Data Society sees teams skip past: recovered time only turns into recovered value if someone decides what it gets spent on. A coding agent that returns twelve hours to a developer’s week produces zero additional value if that time drains into more meetings, more context-switching, or more of the same ticket queue without a change in scope. The tool creates the capacity. It doesn’t decide what to do with it.
Whether a coding agent is worth adopting on your team depends on measuring your own results against the task types above, not the vendor benchmark. Data Society’s AI Advisory work helps engineering teams evaluate where coding agents actually earn their keep, and AI upskilling programs help developers build the judgment to use them well.
