36
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 20 Aug 2023
36 points (100.0% liked)
lemmy.ml meta
1406 readers
1 users here now
Anything about the lemmy.ml instance and its moderation.
For discussion about the Lemmy software project, go to !lemmy@lemmy.ml.
founded 3 years ago
MODERATORS
Robots.txt is just a notice anyways. Your scraper could just ignore it, no workaround necessary.