110

It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.

The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong.

Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.

you are viewing a single comment's thread
view the rest of the comments
[-] cronenthal@discuss.tchncs.de 91 points 2 days ago

Don't get your tech reporting from The Guardian. This headline is so stupid. They can't help but anthropomorphize LLMs, because they just don't known any better.

[-] yeahiknow3@lemmy.dbzer0.com 39 points 2 days ago

Same vibes as “my calculator has a tiny mathematician trapped inside.”

Or “there’s an artist inside of my printer who turns numbers into pictures.”

[-] Baizey@feddit.dk 11 points 2 days ago

"you took a photo of me and trapped my soul in the image!"

[-] FartMaster69@lemmy.dbzer0.com 6 points 2 days ago

Though your calculator can be trusted to actually do its job accurately.

[-] dfyx@lemmy.helios42.de 9 points 2 days ago

Not even that. Calculators have their own limitations related to rounding errors and big numbers. Their results may be deterministic but they are not always accurate.

[-] punksnotdead@slrpnk.net 5 points 2 days ago* (last edited 2 days ago)

https://youtu.be/_XJbwN6EZ4I?t=1074 (skip to 17:54 if the time jump doesn't work)

If only that were the case...

[-] FartMaster69@lemmy.dbzer0.com 4 points 1 day ago

Well shit, that’s a good point.

[-] LukeZaz@beehaw.org 33 points 2 days ago* (last edited 2 days ago)

This right here. Just about everything in here is awful, and implies decision making and thought processes that straight up do not and have never existed in any AI model whatsoever.

What happened was they threw an awfully-scoped statistics model at problems the program couldn't possibly generate good outputs for, and surprise surprise, it generated bad outputs. The part that's of interest is just how bad the output was, and even then, only in a schadenfreude-filled "it was bound to happen eventually" manner.

[-] sem@piefed.blahaj.zone 9 points 1 day ago

It didn't confess it just outputted more plausible garbage based on inputs.

[-] Kichae@lemmy.ca 5 points 1 day ago

It just agreed with the accusations, because these models do what they're trained to do: Agree with the prompter.

[-] Dymonika@lemmy.ml 2 points 1 day ago

No, not necessarily; they can easily, even condescendingly go against your view depending on the topic. It really depends on the topic and the conversational flow.

[-] harmbugler@piefed.social 5 points 2 days ago

Can I just anthropomorphise a little bit and call them psychotic?

[-] LukeZaz@beehaw.org 5 points 1 day ago

The CEO? Yeah sure, go ahead!

[-] Prathas@lemmy.zip 3 points 1 day ago

That needs no... *thinks of the Zuck*

Well, hmm, you're right: maybe that does need anthropomorphization after all.

this post was submitted on 30 Apr 2026
110 points (100.0% liked)

Technology

42839 readers
179 users here now

A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.

Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.

Subcommunities on Beehaw:


This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.

founded 4 years ago
MODERATORS