This is hardly surprising. It's immediately noticeable in images, but we'll have to be very careful with other forms of output as the decline could be subtle enough to go unnoticed at first. There's a very real risk of poisoning our sources of data by allowing AI to write back to them without oversight. And given that the sources of data seem to be things like Reddit and twitter this is a real concern.

The best term I've heard to describe this this "Hapsburg AI".

I think people may be confused about what this is saying, so an example might help.

Remember when Stable Diffusion first came out and you could spot AI generated images as if they killed your father and should be prepared to die?

Had those six+ digit monstrosities been fed back into training the model, you'd have quickly ended up with models generating images with even worse hands from hell.

But looking at a study like this and worrying about AI generated content poisoning the Internet for future training is probably overblown.

Because AI content doesn't just find its way onto the web directly the way it is in this (and the Stanford) study. Often a human is selecting from multiple outputs to decide what to post, or even if it is directly posted, humans are voting content up or down based on perceived quality.

So practically, if models were being trained recursively on popular content online that had been generated by AI, it wouldn't be content that overfits spider hands or striped faces or misshapen pupils or repetitive text or broken links or any other number of issues generative AI currently has.

Because of the expense in human review of generated content this and the previous paper aren't replicating the circumstances that real world recursive training of a mixed human and AI Internet would represent, and the issues which arose will likely be significantly muted in real world circumstances outside the lab.

TL;DR: Humans filtering out six fingered outputs (and similar) as time goes on is a critical piece of the practical infrastructure which isn't being represented, and this is simply a cautionary tale against directly piping too much synthetic data back into training.

While I don't claim to understand how the AI function, this makes sense. Think along the lines of making a copy of a copy of a copy, etc, using a photocopier instead of copying a file. Because they are reinterpreting the works every time more and more errors accumulate in the results. This may be because there's a difference between recognising and understanding.

MAD = Model Autophagy Disorder

Autophagy = to eat oneself

From the article:

Knowing means that the search for a watermark that identifies AI-generated content (and that's infallible) has now become a much more important - and lucrative - endeavor, and that the responsibility for labeling AI-generated data has now become a much more serious requirement.

Simply wanting such a thing to exist isn't going to magically make it happen. I seriously doubt that any such "watermark" (I think they meant "fingerprint" since it'd need to work even if not deliberately added) can be found.

I suspect the actual solution is to curate the quality of the input data, regardless of whether it's AI-generated or not. The problem of autophagy is the loss of rare inputs, so try to ensure those inputs are found and included in the input data. It's probably fine to have some AI generated content in the training data in addition to the real stuff. Indeed, as long as the AI-generated content is subject to the same sort of selective pressure as the real content it's probably good to have.

I'd go mad too if someone tried to train me on AI created data all the time...

Incest is always bad even amongst computers and data sets

Come on where are all the fucked up generated images??

Weird choice making an article about this and then barely showing it.

This is a very boring kind of rampancy

I don't know. If you locked someone in a room. Showed them a single news broadcast, and from then on just replayed back at them anything they say.. They'd go mad too.

Did people already forget about GANs?

Sounds like a skill issue to me

Considering that training is extracting the main features of a dataset, there is always some data that is discarded as "noise" in the process, then when data is generated, that discarded information is filled back with actual random noise to partially replicate the original data.

Iterate and you're going to end up with progressively less meaningful features. I just didn't expect it to take only 5 iterations, that's a lot of feature loss in training even with so many parameters.

The patterns on the beards look kinda cool ngl