Writing code with an AI as an experienced software developer is like writing code by instructing a junior developer.
... That keeps making the same mistakes over and over again because it never actually learns from what you try to teach it.
This is not really true.
The way you teach an LLM, outside of training your own, is with rules files and MCP tools. Record your architectural constraints, favored dependencies, and style guide information in your rule files and the output you get is going to be vastly improved. Give the agent access to more information with MCP tools and it will make more informed decisions. Update them whenever you run into issues and the vast majority of your repeated problems will be resolved.
Well, that's what they say, but then it doesn't actually work, and even if it did it's not any easier or cheaper than teaching humans to do it.
More to the point, that is exactly what the people in this study were doing.
Apparently some people would love to manage a fleet of virtual junior devs instead of coding themselves, I really don’t see the appeal.
I think the appeal is that they already tried to lean to code and failed.
Folks I know who are really excited about vibe coding are the ones who are tired of not having access to a programmer.
In some of their cases, vibe coding is a good enough answer. In other cases, it is not.
Their workplaces get to find out later which cases were which.
Without the payoff of the next generation of developers learning.
Management: "Treat it like a junior dev"
... So where are we going to get senior devs if we're not training juniors?
Very true. I've been saying this for years. However, the flip side is you get the best results from AI by treating it as a junior developer as well. When you do, you can in fact have a fleet of virtual junior developer working for you as a senior.
However, and I tell this to the junior I work with: you are responsible for the code you put into production, regardless if you write it yourself or you used AI. You must review what it creates because you're signing off on it.
That in turn means you may not save as much time as you think, because you have to review everything, and you have to make sure you understand everything.
But understanding will get progressively harder the more code is written by other people or AI. It's best to try to stay current with the code base as it develops.
Unfortunately this cautious approach does not align with the profit motives of those trying to replace us with AI, so I remain cynical about the future.
Usually, having to wrangle a junior developer takes a senior more time than doing the junior's job themselves. The problem grows the more juniors they're responsible for, so having LLMs stimulate a fleet of junior developers will be a massive time sink and not faster than doing everything themselves. With real juniors, though, this can still be worthwhile, as eventually they'll learn, and then require much less supervision and become a net positive. LLMs do not learn once they're deployed, though, so the only way they get better is if a cleverer model is created that can stimulate a mid-level developer, and so far, the diminishing returns of progressively larger and larger models makes it seem pretty likely that something based on LLMs won't be enough.
Wow, great analogy. Might steal this to use myself.
The real slowdown comes after when you realize you don't understand your own codebase because you relied too much on AI. To understand it well enough requires discipline, which in the current IT world is lacking anyway. Either you can rely entirely on AI or you need to monitor its every action, in which case you may be better off writing yourself. But this hybrid approach I don't think will pan out particularly well.
Yeah, it's interesting how strangely development is presented, like programming is only about writing code. They still do that when they tout ai coding capabilities.
I'm not against ai, it's amazing how quickly you can build something. But something small and limited one person can build. The whole human experience is missing, laziness, boredom, communication and issues with communication,... to actually build a good product that's more than a simple app.
And this gets worse over time because you still have to maintain it.
And as the cherry on top - https://www.techradar.com/pro/nearly-half-of-all-code-generated-by-ai-found-to-contain-security-flaws-even-big-llms-affected
Not surprised.
In my last job, my boss used more and more AI. As a senior dev, I was very used to his coding patterns. I knew the code that he wrote and could generally follow what he made. The more he used AI? The less understandable, confusing and buggy the code became.
Eventually, the CEO of the company abused the "gains" of the AI "productivity" to push for more features with tighter deadlines. This meant the technical debt kept growing, and I got assigned to fixing the messes the AI was shitting all over the code base with.
In the end? We had several critical security vulnerabilities and a code base that even I couldn't understand. It was dogshit. AI will only ever be used to "increase productivity" and profit while ignoring the chilling effects: lower quality code, buggy software and dogshit working conditions.
Enduring 3 months of this severely burnt me out, I had to quit. The rabid profit incentive needs to go to fucking hell. God I despise of tech bros.
When writing code, I don't let AI do the heavy lifting. Instead, I use it to push back the fog of war on tech I'm trying to master. At the same time, keep the dialogue to a space where I can verify what it's giving me.
- Never ask leading questions. Every token you add to the conversation matters, so phrase your query in a way that forces the AI to connect the dots for you
- Don't ask for deep reasoning and inference. It's not built for this, and it will bullshit/hallucinate if you push it to do so.
- Ask for live hyperlinks so it's easier to fact-check.
- Ask for code samples, algorithms, or snippets to do discrete tasks that you can easily follow.
- Ask for A/B comparisons between one stack you know by heart, and the other you're exploring.
- It will screw this up, eventually. Report hallucinations back to the conversation.
About 20% of the time, it'll suggest things that are entirely plausible and probably should exist, but don't. Some platforms and APIs really do have barn-door-sized holes in them and it's staggering how rapidly AI reports a false positive in these spaces. It's almost as if the whole ML training stratagem assumes a kind of uniformity across the training set, on all axes, that leads to this flavor of hallucination. In any event, it's been helpful to know this is where it's most likely to trip up.
Edit: an example of one such API hole is when I asked ChatGPT for information about doing specific things in Datastar. This is kind of a curveball since there's not a huge amount online about it. It first hallucinated an attribute namespace prefix of data-star- which is incorrect (it uses data- instead). It also dreamed up a JavaScript-callable API parked on a non-existent Datastar. object. Both of those concepts conform strongly to the broader world of browser-extending APIs, would be incredibly useful, and are things you might expect to be there in the first place.
My problem with this, if I understand correctly, is I can usually do all of this faster without having to lead a LLM around by the nose and try to coerce it into being helpful.
That said, search engines do suck ass these days (thanks LLMs)
No. Experienced devs knew it would make tasks take longer, because we have common sense and technical knowledge.
I don't blame randos for buying into the hype; what do they know? But by now we're seeing that they have caught on to the scam.
I get the agenda of the study and I also agree with it, but the study itself, is really low effort.
Obviously, an experienced developer working on a highly specialized project, where the software developer already have all the needed context, and have no experience with using AI, will beat a clueless AI.
How would the results look like, if the software developer had experience with AI, and were to start on a new project, without any existing context? A lot different, i would imagine. AI is also not only for code generation. After a year of working as a software developer, I could no longer gain much experience from my senior colleagues (says much more about them, than me or AI) and I kinda was forced to look for sparring elsewhere. I feel like I have been speed running my experience and career, by using AI. I have never used code generation that much, but instead I've used it to learn about things i don't know i don't know about. That have been an accelerator.
Today, I'm using code generation much more; when starting a new project, or when i need to prototype something, complete mundane tasks on existing projects, make some none-critical python scripts, get useful bash scripts, spin up internal UI projects, etc..
Sometimes, i naturally waste time, as it takes time for an AI to produce code, and then it takes time to review the code, but in general I feel my productivity have gained by using AI.
Here's the full paper for the study this article is about: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (PDF).
Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down.
The gap and miss is insane.
I assumed nothing, and evaluated it like I would any other tool. It's ok for throwaway scripts but if the script does anything non-trivial that could affect anything external the time spent making sure nothing goes awfully wrong is at least as much as the time saved generating the script, at least in my domain.
I think it’s fair to say that AI yields a modest productivity boost in many cases when used appropriately. It’s quicker to write a detailed prompt than it is to write code most of the time. If you have a good test setup with BDD, you can read the descriptions to make sure the behavior it is implementing is correct and also review the test code to ensure it matches the descriptions. From there, you can make it iterate however long it takes to get the tests passing while you’re only halfway paying attention. Then review the entire git diff and have it refactor as required to ensure clean and maintainable code while fixing any broken tests, lint errors, etc.
It often takes longer to get tests passing than it would take me, and in order to get code I’m happy with like I would write, I usually need to make it do a fair amount of refactoring. However, the fact that I can leave it churning on tests, lint errors, etc. while doing something else is nice. It’s also nice that I can have it write all the code for languages I don’t particularly like and don’t want to learn like Ruby, and I only have to know enough to read it and not have to write any of it.
Honestly, if a candidate in a coding interview made as many mistakes and churned on getting tests passing as long as GH Copilot does, I’d probably mark it as a no hire. With AI, unlike a human, you can be brutally honest and make it do everything according to your specifications without hurting its feelings, whereas micromanaging a bad human coder to the same extent won’t work.
I think it’s fair to say that AI yields a modest productivity boost in many cases when used appropriately
I think this is a mistake. The example in this post is some empirical evidence for that.
It's not clear that we can know in advance whether it's appropriate for any given usecase; rather it seems more likely that we are just pigeons pecking at the disconnected button and receiving random intermittent reinforcement.
Someone on Mastodon was saying that whether you consider AI coding an advantage completely depends on whether you think of prompting the AI and verifying its output as “work.” If that’s work to you, the AI offers no benefit. If it’s not, then you may think you’ve freed up a bunch of time and energy.
The problem for me, then, is that I enjoy writing code. I do not enjoy telling other people what to do or reviewing their code. So AI is a valueless proposition to me because I like my job and am good at it.
Programming
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev