Yes, and my impression is that there are also a ton of people who very much don't want to be indexed or findable through inorganic means

I don't disagree about implementing it and letting instances decide, but they've opted for a different compromise of letting people opt in. Having opt in vs opt out be an instance option would be nice, but open source projects aren't always gonna align 1000% with the priorities that you have personally, unless it's personally your project