The problem with downvotes is they're supposed to be used to push irrelevant things down and bring forward the "productive conversation", but...
...it's easier to use them as an "I disagree with you, get lost loser" button, and I feel like that doesn't usually help the discussion. And upvotes already bring up the good comments (although sometimes the most voted stuff is just memes and you miss the interesting stuff).