686
submitted 1 week ago* (last edited 6 days ago) by Pro@programming.dev to c/Technology@programming.dev

Comments

Source.

you are viewing a single comment's thread
view the rest of the comments
[-] witten@lemmy.world 26 points 6 days ago

Then they'd have to bother understanding the content and downloading it as appropriate. And you'd think if anyone could understand and parse websites in realtime to make download decisions, it be giant AI companies. But ironically they're only interested in hoovering up everything as plain web pages to feed into their raw training data.

[-] Natanael@infosec.pub 16 points 6 days ago

The same morons scrape Wikipedia instead of downloading the archive files which trivially can be rendered as web pages locally

this post was submitted on 17 Aug 2025
686 points (100.0% liked)

Technology

426 readers
269 users here now

Share interesting Technology news and links.

Rules:

  1. No paywalled sites at all.
  2. News articles has to be recent, not older than 2 weeks (14 days).
  3. No videos.
  4. Post only direct links.

To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:

More sites will be added to the blacklist as needed.

Encouraged:

founded 3 months ago
MODERATORS