126
you are viewing a single comment's thread
view the rest of the comments
[-] flamingos@feddit.uk 31 points 6 days ago

Information-gathering agents are an evolution of Google Alerts. Beyond spotting changes, they can make sense of them, too.

… Links will become an afterthought with the coming changes to the Search results experience.

Web publishers should honestly just block googlebot at this point. Why should they provide credibility to whatever Google's stochastic parrot hallucinates if Google won't even give them any kickback?

So what do we search with instead of Google? There isn’t a lot of choice. There’s various flavours of Google or Bing.

Microsoft deprecated their Bing API back in August, instead telling people to use some Azure AI thing. DDG and the like weren't affected because they have contracts, but I can't imagine they'll be renewed.

[-] smeenz@lemmy.nz 5 points 5 days ago

If sites start blocking googlebot en masse, then googlebot will just start ignoring robots.txt

[-] flamingos@feddit.uk 3 points 4 days ago

Then you can just block the user agent in nginx or whatever you use, like all the other AI scrapers who ignore robots.txt (*cough* Amazon)

[-] smeenz@lemmy.nz 2 points 4 days ago

Then the user agent string will just quietly become randomised so you can't match it reliably because it turns out that honouring robots.txt was always little more than a "gentleman's handshake".

[-] dgerard@awful.systems 1 points 4 days ago

this is a problem we have had for a while now, i assure you

[-] Evinceo@awful.systems 1 points 22 hours ago

Yeah an adversary like Google isn't something you can easily block without really annoying legitimate users unfortunately. Nothing is stopping them from turning every chrome instance into a botnet node except for the angry article that would run in Ars Technica.

[-] HK65@sopuli.xyz 4 points 5 days ago

Can they just put an EULA on the site and then sue Google for unauthorized access?

Not in the US of course, but in the EU or something

[-] OpenStars@piefed.social 11 points 6 days ago

Yikes! 😳😬

this post was submitted on 20 May 2026
126 points (100.0% liked)

TechTakes

2586 readers
156 users here now

Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

founded 2 years ago
MODERATORS