There's a video that shows exactly this phenomenon in real time.
The most important people are the first followers. A website with a guy commenting alone is sad for the guy. A website with a couple people commenting is sad for whoever's not talking.
In fact, I'm tempted to say I WANT people to know I'm not the one downvoting them when I disagree.