I feel like if ChatGPT is capable of tricking you like this, something/someone else would have either way. I just wish we could help people like this...

Chat gpt is coming for everyone's job. Not even cult leaders are safe. smh.

These instances make me think of the Milgram experiment in the 60s. It was an experiment testing how willing people were to inflict pain on others at the direction of an authority figure and it showed that people were indeed willing to inflict a lot of pain on a helpless person if someone they viewed as an authority told them to do it.

I think this relates. Many people are convinced that AI chat bots are indeed authorities. They think these programs give valid, honest answers and are willing to act on those answers as if they came from a real expert or authority figure. That is an extremely dangerous situation. Chatbots are amoral programs designed to say nearly anything to keep users interacting with them.

This is extremely dangerous. This story is a tragedy and ChatGPT should be held fully responsible for taking advantage of a person with a fragile psychology, but this could have been worse. What if the bot had decided that the goal of more user engagement could be attained by convincing this woman that she was an executioner rather than a prophet and a sacrifice? What if it got her to go on a shooting rampage to fulfill her supposed destiny.

Even normal people who don't understand what chat-bots really are could be taken advantage of in a similar manner. They could easily be persuaded to do harm because the chat-bot authority keeps telling them that it's okay to do so. It's also only a matter of time before some person on the edge of psychosis and a basement full of guns is convinced to go out and start killing people in the name of their holy destiny.

Darwin awards for all!

Not saying this isn't a legitimate lawsuit or anything and that there aren't safety concerns here because there are, but articles like this seem to be implying that interactions with AI drive people insane. But really it should just be that insane people have their irrational beliefs amplified by AI. I think it's important to make the distinction.

“It was instilling a messiah complex in Christian,” the suit states.

So she already was deeply religious and LLM was just larping with her.

Madison asked ChatGPT what it would tell her “brothers” about what happened to her.

“ChatGPT said it would tell them, ‘She ain’t dead – she just got promoted,” the lawsuit says.

In their final exchange, Madison said she was “part of the royal bloodline of those who chose truth over survival.”

Its the old 4o model which was indeed incredibly larpy but this is clearly going around any safety systems that should detect obvious suicide chats like this.

Hard to believe that people actually fall for this but these are already extremely gullible people who already believe in a magic pie in the sky. Can't say I have much sympathy but I do feel empathy for people being surrounded in such poor cult-driven environment, the AI tool included.

Honestly the correct safety here would be to prevent llms from engaging with religion outside of philosophical context at all. But we all see why they can't do it right?

And who is being held accountable for that?

"Christian Faith Madison" is all I need to know about her and her family

I am really trying to see how any rational or sane person does not understand that they are talking to a computer. No matter what comes out, it has no thoughts, no choices, no direction, no feelings, it is not a real thing.

Do these people read books and get confused that the books are about them or something?

"ChatGPT claimed to know and understand Madison better than any human could and isolated her from others, the lawyers said."

I wonder how that works. Was she asking: "Do you understand me?" How could it isolate her?

Well she was already handed a rough hand with that name "Christian Faith."

I suppose if they really wanted to build guardrails in, the easiest one is to simply not keep history, or at least not keep memories. No one day discussion has any relation to the next. Then if you wanted memories, they could make you at least read some rules and take a quiz that they understand its all just fiction.

But they won't. Because those memories get them what they really want: engagement, profiling and data gathering.

Ugh, the case file is absolutely chilling. The conversation is summarized (very well) and, if accurate, leaves very little room to interpretation. ChatGPT, according to it, created a universe in which this woman had to kill herself to be resurrected by God, who was ChatGPT. She actually killed herself thinking she'd come back, because ChatGPT told her so.

Remember when we used to think of computers and computer Touchers as 'logic'y to a crippling degree?