And this is fine. /c/all should let users downvote posts they don't like so popular stuff can rise to the top. That's what makes /c/all sometimes worth looking at.

Yeah, but community moderators also have the ability to look at those downvotes and react accordingly.

Your problem is that you can't delist your community from /c/all. That sucks, but right now your posts are turning up in two different communities with different expectations and you just need to deal with that.

Sure. I don't want to delist /my/ community from /all/ but if someone did downvote every post on the community made in the last day, I might consider that mass-downvoting from someone who doesn't like the topic and react accordingly.