321
rulebots.txt
(lemmy.world)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 21 Aug 2024
321 points (100.0% liked)
196
16243 readers
1679 users here now
Be sure to follow the rule before you head out.
Rule: You must post before you leave.
founded 1 year ago
MODERATORS
I am confused, does this mean Reddit is not going to be searchable on search engines anymore?
Unfortunately yes. It was reported on last month.
oh no, Reddit is like, the only way to have google still be useful.
Funnily enough, google is also the only way to have Reddit be useful.
Their own search function has been nothing but garbage.
That's the catch, Google made a deal with Reddit and remains the only search engine allowed to access its data for indexing. It cuts off every other search engine
Tell me that there is an anti trust suit over this.
There's a suit over google in general so this may well be part of it
really? ddg will show me reddit links, did they have to make a webscraper or something
There's a cutoff date, anything indexed before the robots.txt was changed stays in the index
We fucked the internet. It’s proprietary now.
kinky
cat5 sounding you say?
cat5-o-nine-tails
Good news! Google paid up and still has access I'm pretty sure.
That's bad news, that means the internet is dying
Sorry, the /s was sort of implied.
Ah, sorry. I have trouble with that sometimes :P
Perhaps, likely depends on the crawler though
Yeah i dont think ignoring robots.txt is even illegal. They can ofcourse just block your crawlers IP but that would be a cat and mouse game that they would lose in the end.