Only if the appropriate legislation is available to vote on. If the only legislation available is something that hurts you a little and helps someone else a lot, it may be in society's best interest to vote for it. If you were in a culture that encouraged that, your actions would be repaid by others doing the same, eventually securing large gains for everyone. This is the opposite of my example above, but the math works out the same.

Essentially, there are situations in which the logical choice is to vote for something that hurts you, or to not vote for something that helps you. (Zero-sum-like situations are especially likely to have this occur.) Over a long period of time, what matters is how much each bill helps society overall, not how much it helps you in particular. (Yes, this stops working if the other groups won't do the same for you.)