There was actually a research paper where they figured they could predict if people drink wine or beer with a CNN that looks at knee X-rays. Turns out that part of their data is from beer drinking regions, and another part from wine drinking regions, and that the photos are just ever so slightly distorted depending on which physical machine they were made with.

The paper pointed that out, it's whole propose was to show how much bullshit you can use AI for if you are not careful what you train it with.

There were many other examples in this paper of what they can predict just from people's knee X-rays. They had all non-medical explanations such as the one above.

There was something similar with a cancer-detecting model somewhat recently. It was trained on a bunch of scans both positive and negative. However, positive scans had to be signed off on by a doctor. So the model just ended up looking for signatures.

That also sounds a lot like the kind of comments that Reddit (and Lemmy, and really any social network with votes) grooms for if you prefer up votes to arguing with pedants and trolls. Eventually all your left with are boring overqualified comments or inflammatory comments when the mob rules and you are striving/solving for the most popular/engaging answer. It's like conversational least squares analysis.

I wonder where the LLM trolls are? Maybe they are just so subtle, we haven't noticed them. Maybe LLMs aren't hallucinating answers, so much as they and trolling us. And here is where I qualify my answer in an attempt to quell the fools that might think anything I've said here implies that LLMs are anything close to sapient.