This then also makes me wonder how these models are going to be trained in the future. What happens when for example half of the training data is the output from previous models? How do you possibly steer/align future models and prevent compounding errors and bias? Strange times ahead.

Between this and the "deep fake" tech I'm kinda hoping for a light Butlerian jihad that gets everyone to log tf off and exist in the real world, but that's kind of a hot take

My guess is the internet is gonna go through trial by fire regarding ai—some stuff is gonna be obscenely incorrect, or difficult to detect before it all straightens out.

My understanding was that they're very unreliable in their current state, but I'm definitely not up to speed.

The validator would have access to real references it can use to ensure some form of correctness

That's the crux of the problem, a LLM has no understanding of what it's saying, it doesn't know how to use references. All it knows is that in similar contexts this set of words tended to follow this other set of words. It doesn't actually understand anything. It's capable of producing output that looks correct to a casual glance but is often wildly wrong.

Just look at that legal filing that idiot lawyer used ChatGPT to generate. It produced fake references that were trivial for a real lawyer to spot because they used the wrong citation format for the district they were supposedly from. They looked like real citations because they were based on how real citations looked but it didn't understand that citations have different styles depending on the court district and that the claimed district and citation style must match.

LLMs are very good at producing convincing sounding bullshit, particular for the uninformed.

I saw a post here the other day where someone was saying they thought LLMs were great for learning because beginners often don't know where to start. There might be some merit to that if it's used carefully, but by the same token that's incredibly dangerous because it often takes very deep knowledge to see the various ways the LLMs output is wrong.

Best description I've heard is that LLM is good at figuring out what the correct answer should look like, not necessarily what it is.

The validator would have access to real references

And who wrote the "real" references?

Because that's the point of the post you replied to. LLMs can't completely replace humans, because only humans can make new "real references".

Adding that is not something that can be done using the current approach. A new approach is not guaranteed to be found, may not be found anytime soon, could have been found by someone that hasn't made it public yet.