I feel this -- we had a junior dev on our project who started using AI for coding, without management approval BTW (it was a small company and we didn't yet have a policy specifically for it. Alas.)

I got the fun task, months later, of going through an entire component that I'm almost certain was 'vibe coded' -- it "worked" the first time the main APIs were called, but leaked and crashed on subsequent calls. It used double- and even triple-pointers to data structures, which the API vendor's documentation upon some casual reading indicated could all be declared statically and re-used (this was an embedded system); needless arguments; mallocs and frees everywhere for no good reason (again due to all of the un-needed dynamic storage involving said double/triple pointers to stuff). It was a horrible mess.

It should have never gotten through code review, but the senior devs were themselves overloaded with work (another, separate problem) ...

I took two days and cleaned it all up, much simpler, no mem leaks, and could actually be, you know, used more than once.

Fucking mess, and LLMs (don't call it "AI") just allow those who are lazy and/or inexperienced to skate through short-term tasks, leaving huge technical debt for those that have to clean up after.

If you're doing job interviews, ensure the interviewee is not connected to LLMs in any way and make them do the code themselves. No exceptions. Consider blocking LLMs from your corp network as well and ban locally-installed things like Ollama.

I talked to Microsoft Copilot 3 times for work related reasons because I couldn't find something in documentation. I was lied to 3 times. It either made stuff up about how the thing I asked about works or even invented entirely new configuration settings

The study was centered on bugfixing large established projects. This task is not really the one that AI helpers excel at.

Also small number of participants (16) , the participants were familiar with the code base and all tasks seems to be smaller in completion time can screw results.

Thus the divergence between studio results and many people personal experience that would experience increase of productivity because they are doing different tasks in a different scenario.

I'll quote myself from some time ago:

The entire article is based on the flawed premise, that "AI" would improve the performance of developers. From my daily observation the only people increasing their throughput with "AI" are inexperienced and/or bad developers. So, create terrible code faster with "AI". Suggestions by copilot are >95% garbage (even for trivial stuff) just slowing me down in writing proper code (obviously I disabled it precisely for that reason). And I spend more time on PRs to filter out the "AI" garbage inserted by juniors and idiots. "AI" is killing the productivity of the best developers even if they don't use it themselves, decreases code quality leading to more bugs (more time wasted) and reducing maintainability (more time wasted). At this point I assume ignorance and incompetence of everybody talking about benefits of "AI" for software development. Oh, you have 15 years of experience in the field and "AI" has improved your workflow? You sucked at what you've been doing for 15 years and "AI" increases the damage you are doing which later has to be fixed by people who are more competent.

Don’t give yourselves to these unnatural men - machine men with machine minds and machine hearts! You are not machines! You are men!

Reading the paper, AI did a lot better than I would expect. It showed experienced devs working on a familiar code base got 19% slower. It's telling that they thought they had been more productive, but the result was not that bad tbh.

I wish we had similar research for experienced devs on unfamiliar code bases, or for inexperienced devs, but those would probably be much harder to measure.

Would ai coders even get faster over time or just stay stagnant since they aren't learning anything about what they're doing

Ai-only vibe coders. As a development manager I can tell you that AI-augmented actual developers who know how to write software and what good and bad code looks like are unquestionably faster. GitHub Copilot makes creating a suite of unit tests and documentation for a class take very little time.

Someone told me the best use of AI was writing unit tests and I died on the inside.

My velocity has taken an unreasonable rocket trajectory. Deploying internal tooling, agent creation, automation. I have teams/swarms that tackle so many things, and do it well. I understand there are issues, but learning how to use the tools is critical to improving performance, blindly expecting the tools to be sci-fi super coders is unrealistic.

People are a bad judge of their own skill and overrely on tools and assistants when present. See also: car adas systems making drivers less skillful. More news at 11.

Now the interesting question is what it really means when less experienced programmers think they are 100% faster.

It’s hard to even call them specialists, they are at the level of cashiers, for whom the computer does everything, and sometimes they do something at the level of communicating with clients and that’s all. I'm certainly not a professional, but I think the main message is clear.

The only time it really helps me is when I'm following a pretty clear pattern and the auto-complete spares me from copy-pasting or just retyping the same thing over and over. Otherwise I'm double-checking everything it wrote, and I have to understand it to test it, and that probably takes most of my time. Furthermore, it usually doesn't take the entire codebase into account so it looks like it was written by someone who didn't know our team or company standards as well as our proprietary code.

I wish AI was never invented, but surely this isn't ture.

I've been able to solve coding issues that usually took me hours in minutes.

Wish it wasn't so, but it's been my reality

It would be interesting to see another study focusing on cognitive load. Maybe the AI let's you offload some amount of thinking so you reserve that energy for things it's bad at. But I could see how that would potentially be a wash as you need to clearly specify your requirements in the prompt, which is a different cognitive load.