Yes, but should big companies with business models designed to be exploitative be allowed to act hypocritically?

My problem isn't with ML as such, or with learning over such large sets of works, etc, but these companies are designing their services specifically to push the people who's works they rely on out of work.

The irony of overfitting is that both having numerous copies of common works is a problem AND removing the duplicates would be a problem. They need an understanding of what's representative for language, etc, but the training algorithms can't learn that on their own and it's not feasible go have humans teach it that and also the training algorithm can't effectively detect duplicates and "tune down" their influence to stop replicating them exactly. Also, trying to do that latter thing algorithmically will ALSO break things as it would break its understanding of stuff like standard legalese and boilerplate language, etc.

The current generation of generative ML doesn't do what it says on the box, AND the companies running them deserve to get screwed over.

And yes I understand the risk of screwing up fair use, which is why my suggestion is not to hinder learning, but to require the companies to track copyright status of samples and inform ends users of licensing status when the system detects a sample is substantially replicated in the output. This will not hurt anybody training on public domain or fairly licensed works, nor hurt anybody who tracks authorship when crawling for samples, and will also not hurt anybody who has designed their ML system to be sufficiently transformative that it never replicates copyrighted samples. It just hurts exploitative companies.