we've essentially created a computer program that is just as fallible as humans.

Humans are a little better in this respect: we know when something should be secure and can try to hide it. Like you say, it takes social engineering to trick secrets out of us.

Language models don't try to keep secrets.