Should lemmy.ml block chatgpt scraping in robots.txt? (lemmy.ml)

submitted 1 year ago by GnuLinuxDude@lemmy.ml to c/meta@lemmy.ml

13 comments fedilink hide all child comments

Some context about this here: https://arstechnica.com/information-technology/2023/08/openai-details-how-to-keep-chatgpt-from-gobbling-up-website-data/

the robots.txt would be updated with this entry

User-agent: GPTBot
Disallow: /

Obviously this is meaningless against non-openai scrapers or anyone who just doesn't give a shit.

you are viewing a single comment's thread
view the rest of the comments

[-] 7heo@lemmy.ml 2 points 1 year ago

That won't stop OpenAI. We need actual blocking, on the server side. Problem is, with federation and all, it will be really, really difficult to do. And expensive.

this post was submitted on 20 Aug 2023

36 points (100.0% liked)

lemmy.ml meta

1406 readers

1 users here now

Anything about the lemmy.ml instance and its moderation.

For discussion about the Lemmy software project, go to !lemmy@lemmy.ml.

founded 3 years ago

MODERATORS

nutomic@lemmy.ml