91
I was wrong about robots.txt
(evgeniipendragon.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 17 Jul 2025
91 points (100.0% liked)
Technology
73534 readers
3834 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
Kinda, but also not really. Any major tech player that has billions to lose will make a show of respecting robots.txt when presenting that information to third parties, lest they be exposed by basic journalism.
However, they also have separate networks in R&D that sweep the net all the time and do not care about such restrictions. It's theatre.
And they're still happy to punish people that have the gall to publicly decline their crawlers. Basically they can eat their cake and have it too.