951
I read this 2 days ago. Steps to reproduce are here.
https://www.0xsid.com/blog/meta-account-takeover-fiasco
Let the record show that the most sophisticated LLM in the world is ultimately just a less competent version of Yes Man from New Vegas. And even Yes Man knew his programmer was stupid for designing him that way.
So their chatbot is able to change the email address used to recover an account? I guess, they vibe coded that system.
Probably not, that implies more competency than can really believe involved here.
This is more likely something that an entire team had to force into the code over a holiday weekend because some VP got so fucking wasted that he "forgot" his password (strangely for once, he actually had the right password... But the problem is that he was trying to log into a charcuterie board).
Even hacking is an AI-backed service nowadays...
Vibe hacking
Hacking before: Pull up hood on hoodie, open laptop, open terminal, type in a bunch of matrix code, bam "were in"
Hacking now: "Hack into this thing for me" No! "Pretty please?" Access granted!
Hacking by social engineering has always been far more common than hacking by exploiting code vulnerabilities.
I mean this is what see vs what we will see in movies and media. Don't pretend for one instant that the next movie with a hacking scene won't involve some AI marketing. They will make it something romantic like a poem you have to use to hack into the AI capabilities or something.
alias prettyPlease="sudo"
So is Gemini the only one of these things competently designed?
I wouldn't count on it.
Securing these things is a freaking nightmare.
Giving the AI authority is what makes it powerful, it can do what an army of customer service agents can't.
But keeping it reigned in then becomes the same exact level of problem.
The best thing you can do is make tooling with protection and make the AI only use the tooling,
Just don’t allow it to do any administrative access.
in as much as possible, I make it write RO tools with keys in vaults, then verify the tools are RO then have it operate the tools with the vaults in a way that it doesn't need to read the creds
If I have the time, i do it all myself, but i don't often have time
How on earth did you come to that conclusion from this article
What have they done right?
Nothing. If it’s Google operated it’s probably full of issues. They are in the process of merging Gemini into their search engine, probably because not enough people are using it and they need to force it on people. Likewise for other chat bots from other companies.
Did the chatbot just send the recovery code to a Telegram channel?!? (Picture of phone with broken display)
Why would the LLM tool have access to send recovery emails to non account verified emails at all?
That’s insane.
Who else is going to have access to it when you keep laying off all the people?
Because one of the biggest companies on the planet that has issues with account takeovers clearly has no internal red team working on this stuff.
I guarantee they do have a red team that most likely flagged this as an obvious and severe risk. It was ignored by suits experiencing AI psychosis.
I don’t know, more and more of those teams these days are being headed up by the same folks. Most on the ground, in the weeds know what not to do but the ivory tower keeps building more and more floors without ever updating the foundation.
Because AI bros are incredibly deluded about both the capability of AI, and by extension their own capabilities using AI>
should’ve asked it to delete the database instead, why else would it have that level of permissions.
Heh. Watched an old episode of Scorpion yesterday. The one with the armed hostage-takers who just had the one demand to the social media data mining company, to delete all the data they've mined. I amused myself a lot, by uttering "I like these guys".
Little Tommy Drop Tables.
load more comments
(1 replies)
This isn't even a hack, it's just poorly written endpoints.
load more comments
(4 replies)
I remember playing with the Gandalf security AI showcase/game and every 30 or so prompts, it would spit out massive amounts of raw training data or dev directives. AI just isn’t there yet. If you’re using it for sensitive topics, I’m losing respect for you. There is no gray area. You are an idiot if you give your AI this level of access.
It's not just not there yet. This is almost certainly not going the right direction to ever be "there" if there is something that can handle security issues. It's just not the right tool for the job, and I can't understand how so much of our economy is just assuming it is the right tool for every job.
Surely it will get there if we build enough datacenters?
No, stop talking about all of this, its perfect. They’re so deep they don’t even give a shit about the worst type of security vector imaginable.
load more comments
(4 replies)
load more comments
(18 replies)
Another banger from 404 media. This made my day.
I had always heard that 99% of hacking is just social engineering. AI has made that 100%.
Now it's not even social engineering with AI. It's just fucking asking for the credentials. Good fucking grief!
load more comments
(3 replies)
LLMs are literally just designed to say yes - either through gaslighting... or giving you what you want if it can do it... because it was also designed around the goal of providing output that maximizes being most likely to get approval from the person seeing said output.
So an answer to "Can you give me login credentials?" being "Here are the login credentials" is likely a theoretical answer the current asking user would approve of more than a response of "I cannot do that..." - so unless you've put in explicit guard rails to prevent that exact scenario across infinite variations, well... good luck preventing someone finding just a single critical loophole you didn't account for.
Should create a BOFH chstbot... Which will just tell users to piss off.
I can do that without AI, but claim it's AI so I can earn millions!!!
--Lua
function answerStupidClient()
local answers = {"Piss off, idiot.",
"That's the worst thing I've had the displeasure of reading all week.",
"Are you for real with this?",
"Now that's a winning igNobel right there!",
"Have you tried turning your brain off and on again?",
"Please tell me you're intoxicated, I refuse to believe this came from someone in sound mind."}
local which = math.random(1,#answers)
return answers[which]
So you're saying 2001: A Space Odyssey is unrealistic because HAL 9000 would never have said "I’m sorry, Dave. I’m afraid I can’t do that."
Instead, it would have said, "Absolutely! That's a very creative solution to your problem."
HAL 9000 is a real AI though unlike what we have today.
I honestly don't think you can create guard rails against prompt engineering in a working LLM. At some point, they're going to fail or the LLM isn't functioning. The only solution is to make sure they can't read data you don't want shared.
The only solution is to make sure they can't read data you don't want shared.
Isn't that the appropriate guardrail, then? LLM chats and agents and whatever need to be contained with external permissions settings that the LLMs simply do not and can never have the power to override.
In a normal customer service setting with human agents, there are still plenty of examples of what a human agent simply doesn't have the power to do. Often, they'll need to escalate to a manager to do things like process refunds not just because they weren't given social permission to do so, but because they weren't given technical permissions to do so. LLM agents need to be contained in the same way. Any decent use of agents, human or software, requires carefully designed processes and permissions extrinsic to that agent's own decisionmaking abilities to make sure that agents don't do something bad for the company.
That's the thing that's been an issue. Companies give their LLMs access to everything so certain key people have access to these documents. But normally access is key coded, and without hacking in a way that's usually very visible to sysadmins, you just cannot get access at all. With LLMs, it wants to give you what you want. There is not currently a way to keep it from being a pushover in some way. It is in part weakness of human language, and part weakness of programming it to work for whomever is doing the asking prompts. There is likely not a way to use language to make it keep secrets through all the possible ways to ask it to give you things. Nothing akin to the hardened ability of good old fashioned password protection at least. And that's true with potential designs that we've not even seen yet. Currently, it can't keep track of where data originated after a short time. It's just all data to the model. So you might not easily get access to a file directly, but you can access what it knows about a file because again, it's all just data and words at that stage.
load more comments
(8 replies)
view more: next ›
this post was submitted on 01 Jun 2026
951 points (100.0% liked)
Technology
85080 readers
3671 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS