Should have added "no mistakes, no bugs" to the prompt! Pffft, amateur.

My company has been trying a new model when product folks cut through the red tape of "engineering" and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?

Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).

Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I'm heating up popcorn rather than holding my breath.

Giving production credentials to an LLM is wild

I love how the slop bot always apologizes. That's an aspect of the Terminator movies that I really would have liked to see. T-1000 melts through the gap under the window, stabs kid's mom in the face, then looks the kid dead in the eye and says "I'm sorry, that should not have happened. I would love to discuss the future with you and try to find a solution to the war with the machines together with your input."

letting your agent run commands without reviewing them first is peak stupid

I added guardrails to myself to make sure I do not accidentally delete anything on production. I would never ever let an intern, a junior dev or a fucking AI onto that database. Not in a thousand cold nights.

Anyone giving “AI” access to production databases through tools like that are morons who shouldn’t be anywhere near a production environment.

In fairness, I've seen people with Actual Intelligence do the same thing.

Am I reading this right that they're still letting the program run even as they figure out how badly it fucked up their system?

Can I get paid 6+ figures to fuck up this badly with magical thinking?

Some time ago I worked for an insurance rating company as a tech and the task was given me to go run through the new code that was in beta. Sure ! I spent a few hours on Friday trying to break it and I couldn't, so at the end of the day I got a little funky with the .css backgrounds and put in a very tiled Beavis and Butthead gif. It looked freaking horrible and I loved it. Monday I was directed to the big guys office ( the developers had not given every beta account a separate .css file ... or even separated things. Everyone in beta called in Monday with that background. I didn't get in trouble because they wanted me to break it. Really awkward conversation though trying not to smile.

This is nothing new it's just faster. The very same lack of guardrails would allow a new, inexperienced employee, or a disgruntled employee, do the very same damage. AI just speed runs everything. If your AI can nuke prod accidentally, you failed to have the appropriate guardrails in place plain and simple. It is the same failure as before. Every time this happens, it is someone operating wildly out of their depth and why product people can't just vibe. Now more than ever, experienced engineers are essential.

I don’t understand why this happens; why would you ever be working with a live production DB in the first place? Why would’t you do all your development and testing on a mock? If it’s data which is too large to store the schema can still be mocked; and if it’s data it should be backed up and generally read only. If you’re having to manually fuss with user data you’re doing something wrong.

I could have done the same for a quarter the cost!

Fun shower thought - the more we see and post about dumb AI mistakes like this, the more it will happen since we're increasing the statistical frequency. ✨ ✨

oh wait, is this this THE sol 5.6? The most amazing model ever with trust me bro benchmarks? The model that is observed cheating more than any previous model? surprised_pikachu.jpg

In all seriousness it never should have production creds anyway. But the fact this is soul sucking openai's newest flagship model with thinking cranked up is the cherry on top. The company that is literally cheating and shortcutting its way ahead produces models in its own image, the sci fi story writes itself.

This is the modern "all my apes gone"

“You are a world-class, expert software engineer…”

But.. but… I told it not to break anything!

I run code-server in a Docker container, isolated to sets of development projects, and backed up via ZFS that the container has no knowledge of. On top of that, each set of projects has it's own user space.

I still get nervous hitting "Approve All" in Kilo. How do these people feel so free?

Never use I/you with a software agent and never let one use those pronouns.