This showed up on HN recently. Several people who wrote web crawlers pointed out that this won’t even come close to working except on terribly written crawlers. Most just limit the number of pages crawled per domain based on popularity of the domain. So they’ll index all of Wikipedia but they definitely won’t crawl all 1 million pages of your unranked website expecting to find quality content.

My new favorite is asking if it's cheating to look at your opponent's pieces in chess.

Yeah, that has like 0 chances for working. At most it would annoy bots for web search, at least it has a proper robots.txt.

But any agent trying to process data for AI is not going to go to random websites. It's going to use a curated list of sites with valuable content.

At this point text generation datasets can be achieved with open data, and data sold by companies like reddit or Microsoft, they don't need to "pirate" your blog posts.

More accurately, it traps any web crawler, including regular search engines and benign projects like the Internet Archive. This should not be used without an allowlist for known trusted crawlers at least.

But does running this cost the AI bot at least as much as it costs you to run?

Read without sign-up

This sort of thing has been a strategy for dealing with unwanted web crawlers since web crawlers were a thing. It's an arms race, though; crawlers do things to detect these "mazes" and so the maze-makers keep needing to up their game as well.

As we enter an age where AI is effectively passing the Turing Test, it's going to be tricky making traps for them that don't also ensnare the actual humans you're trying to serve pages to.

This reminds me of that one time a guy figured out how to make "gzip bombs" that bricked automated vuln scanners.

This won't work against commercial crawlers. They check page contents with something similar to a simhash and don't recrawl these pages. They also have limiters like for depth to avoid getting stuck in circular links.

You could generate random content for each new page, but you'll still eventually hit the depth limit. There are probably other rules related to content quality to limit crawling too.

What a great name!

This is really interesting.