Every time I read about some study measuring the productivity gain of AI-assisted development, I raise an eyebrow. I find them uninteresting and useful only for the online fights between “there is no AI productivity for software developers” and “Claude Code increased my productivity by a billion percent.” The problem is that the landscape is so varied that, honestly, you can stress the data to prove both assertions.

My personal opinion is that many papers on the subject are just messy, and often they do not test at all how people using AI efficiently for work are actually using it. There are many issues I identified: they give too-easy problems, they impose arbitrary time constraints, they give problems to people not familiar with the framework they have to use (a big no no), and, surprisingly, I still see a lot of studies where the test subjects use “copy-pasting code from the web chat interface back and forth.” That is a way to work with AI that I think every developer stopped using in 2024.

But setting this aside, my main problem is that they never measure a specific class of task: the tasks that I would not have even started without AI assistants. We can discuss “this AI agent increases productivity by 2%” or “5%” or “20%” for as long as we want, but the reality is that many software and features I made in the last year would not exist at all without AI agents. I have folders with a sea of tools, automations, and single-use scripts that I would have never started.

So how could I measure that? For me, that is an increase of infinite percent, because they didn’t help me complete a project faster; they are the reason some projects exist. In some sense, it is a kind of Pascal’s wager. How should I redact the claim “AI agents improve developer output by only 2%” when I have concrete evidence on my hard drive of things that would have remained annotated in my notebook for all eternity?

Maybe it is my ADHD talking. Probably other people function differently and can do everything by just deciding to. But for me, lowering the activation moat that blocks me from starting to work on something is 100% worth it, even if I end up doing 80% of the work by hand.

It is a personal thing. I know that. I also understand if you work differently. And that’s the point. I saw developers using these tools in wildly different ways, for very different purposes, and with very different outcomes, and I don’t think it’s possible or fair to reduce that to a universal percentage number in some random study.

As with many, I played around with AI agents in code. Contrary to some opinions, AI agents made me rediscover the joy of coding (for many reasons I may discuss another time). However, I am not blind, and I owe my satisfaction with AI agents to a strict mental model and practice.

If you are not a software developer and you use AI to jam together personal scripts for yourself, do not worry, you can do as you want. If you are using them to learn something, don’t worry either.

But if you are a professional coder or you want to publish your work, you have to follow the Golden Rule:

You must use an AI Agent only to do what you know how to do.

Only in this way can you be efficient with them. Only if you know how to do something can you instantly spot when the agent is doing something decent or not. Only if you know how to do something can you recognize whether the generated code is good.

This doesn’t make them less useful. I like to code algorithms, solve problems, and sketch the architecture of the various elements. So I focus on that while I let the robot work on things I loathe. Things like CLI interfaces, reporting, writing diagnostic endpoints, and other tasks I find super boring. I know how to do them; they are just boring. So I let the robot do them while I focus on the math and the algorithms.

However, be careful. The siren’s song is strong. If you are not disciplined, you may think, Why not? Why should I not use the robot for this thing I don’t know how to do?

And that’s the moment you open yourself to fatal mistakes.