1164

submitted 11 months ago by psychothumbs@lemmy.world to c/piracy@lemmy.dbzer0.com

62 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[-] pootriarch@poptalk.scrubbles.tech 4 points 11 months ago

It exists, it's called a robots.txt file that the developers can put into place, and then bots like the webarchive crawler will ignore the content.

the internet archive doesn't respect robots.txt:

Over time we have observed that the robots.txt files that are geared toward search engine crawlers do not necessarily serve our archival purposes.

the only way to stay out of the internet archive is to follow the process they created and hope they agree to remove you. or firewall them.

this post was submitted on 12 Oct 2023

1164 points (100.0% liked)

Piracy: ꜱᴀɪʟ ᴛʜᴇ ʜɪɢʜ ꜱᴇᴀꜱ

53948 readers

290 users here now

1. Posts must be related to the discussion of digital piracy

2. Don't request invites, trade, sell, or self-promote

3. Don't request or link to specific pirated titles, including DMs

4. Don't submit low-quality posts, be entitled, or harass others

💰 Please help cover server costs.

founded 1 year ago

MODERATORS