The Arch Wiki (and possibly other Arch websites, but when I checked I wasn't able to catch it) uses Anubis (https://anubis.techaro.lol/) to mitigate AI crawlers causing issues on their servers. Anubis works by making your computer do some computations before being let in. For users like you and me, it'd be a minor inconvenience, but for these AI crawlers, it'd cost them a lot of computation as they hit these sites a huge number of times.

Using Anubis is said to be a nuclear option, but as these AI crawlers don't care about that, sites, especially FOSS ones, started using it.

I've had a brief look around and it seems fine for me.

Because it has to be recompiled for every update.

Website, or package repos? Two different things.