Your comment made me think of the LLM piping this way (as if it could've started legal):

1. Shit goes in: sourcing material should be treated not like for a personal, but for a commercial use over some volume by default. It's clearly differentiated in licenses, pricing, fees, etc.
2. Shit goes out: the strictiest license of all dataset is applied to how the output can be used. If we can't discern if X was in the mix, we can't say it wasn't, and therefore assume it's there.

To claim X is not in the dataset, the LLM's owner's dataset should be open unless parts of it are specifically closed by contract obligations with the dataminer\broker. Both open and closed parts with the same parameters should produce the same hash sums of datasets and the resulting weights as in the process of learning itself. If open parts don't contain said piece of work, the responsibility is on data providers, thus closed parts get inspected by an unaffilated party and the owner of LLM. Brokers there are interested in showing it's not on them, and there should be a safeguard against swiftly deleting the evidence - thus the initial trade deal is fixed by some hash once again.

Broker with someone's pirated work can't knowingly sell the same dataset unless problematic pieces are deleted. The resulting model can continue learning on additional material, but then a complete relearning should be done on new, updated datasets, otherwise it's a crime.

Failure to provide hashes or other possible signatures verifying datasets are the same, shifts the blame onto LLM's owner. Producing and sharing them in the open and observable manner, having more of their data pool public grants one a right to make it a business and shields from possible lawsuits.

Data brokers may not disclose their datasets to public, but all direct for-profit piracy charges are on them, not the LLM owner, if the latter didn't obtain said content themselves but purchased it from other party.

It got longer than I thought.

Just because you own a cd doesn’t mean you have a license to play it in a club.

...because you are a person, not a product.

LLMs (currently colloquially "AI") are literally incapable of "learning."

LLMs are not sentient and never can be.