103
AI giants learn what everyone else on the modern internet already knows
(www.businessinsider.com)
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
I'm not the most knowledgeable about LLMs, but I don't see the threat distillation is supposed to pose.
Surely what you'd get from training something on a model's outputs and reasoning chain would just get you something that sounds like that model. The model won't magically become more intelligent just because it's trained on data from a smarter model. It doesn't have the relevant vectors and mathematics, it's only got the text output. A lot of the model changes tend to be architectural than just more data these days.
At least from my layman's viewpoint, it seems very much like claiming that watching television would let someone build a better television just from what they saw. The whole fear around distillation just sounds like nonsense.