The reason why Google is doing this is simply PR. It is not to improve its service.

The underlying tech is likely Gemini, a large language model (LLM). LLMs handle chunks of words, not what those words convey; so they have no way to tell accurate info apart from inaccurate info, jokes, "technical truths" etc. As a result their output is often garbage.

You might manually prevent the LLM from outputting a certain piece of garbage, perhaps a thousand. But in the big picture it won't matter, because it's outputting a million different pieces of garbage, it's like trying to empty the ocean with a small bucket.

I'm not making the above up, look at the article - it's basically what Gary Marcus is saying, under different words.

And I'm almost certain that the decision makers at Google know this. However they want to compete with other tendrils of the GAFAM cancer for a turf called "generative models" (that includes tech like LLMs). And if their search gets wrecked in the process, who cares? That turf is safe anyway, as long as you can keep it up with enough PR.

Google continues to say that its AI Overview product largely outputs “high quality information” to users.

There's a three letters word that accurately describes what Google said here: lie.