Oh come on you don’t actually believe we should structure the entire system around such a minority use case

Minority use case? I'm talking about how downvotes are useful for communities to enforce their own norms, or ensure that erroneous information is excluded. Someone who insists on a proof that the angles of a triangle add up to more than 180º is probably going to get downvoted, especially if he's being an asshole about it. Same with someone who insists that the common cold is caused by exposure to cold air, or that the earth is flat.

Or there are broad consensus beliefs about what is or isn't off topic for a discussion, what types of insults break the forum rules on civility, etc. When a community largely agrees that someone is being an asshole for using racial slurs, downvotes quickly sort that out. In other words, toxicity can get filtered out through the downvote/hide mechanism, as well.

Even for beliefs that are simply matters of opinion/taste/preference, the community can decide what's actually up for debate and what's not, within that space. A forum dedicated to fans of Real Madrid doesn't have to tolerate trolls coming in and saying "Real Madrid sucks" or "lol soccer is a stupid sport you Europeans are so stupid" or "sports are dumb." Same with a vegan forum downvoting someone's brisket recipe (or a BBQ forum downvoting a "meat is murder" manifesto). These "echo chambers" are just how people organize with people who share their interests, and it's weird not to be able to see that there's value in those communities.

So yeah, I think that you have a problem with people's desire to organize into groups of similar interests, not with the actual mechanism by which those groups enforce those norms. It wouldn't be any better with a mod-enforced echo chamber, either.