"It's very hard to collect data that trains the model to be better than humans, because there are very few humans who can create that data," Raj said. "You want to make it better than a Fields Medalist or a Nobel Prize winner. How do you collect that?"

...

While there, he worked alongside other researchers to understand the failure points of models hosted on cloud computing infrastructure and offer specific solutions, he said. He also created "synthetic" data that, unlike human-created writing or code, is generated artificially before being reused as training material for the LLMs, Raj said.

Creating synthetic data that matches the quality of human-created data is a tall task, Raj said. Computing power can be acquired relatively easily, but finding the data to support the project is uncharted territory, he added.

sigh

Garbage in garbage out, even if the garbage is synthetic that doesn't make it not garbage...?