The problem with karma is in the big picture. A lot of actions that yield lots of karma are not contributive to the community at large:

• unnecessary reposts and cross-posts; not based on the overall contribution that it brings to the comm, but on its potential to harness votes. This makes the content of the comms less diverse and interesting for veteran users.
• moderators deleting content from other users just so they can repost it later on, to farm karma. This happens quite a bit in a certain site, disengages the users, and creates unnecessary drama.
• people stop contributing with certain discourses not because they think "this is not contributive", but because it's unpopular. Sometimes you need to voice a hot take on an issue, to further discussion.
• it enables mods implementing stupid/arbitrary barriers like "you need X karma to post here", instead of pressing the instance admins to get rid of trolls and bots.

I also believe that karma is one of the big components of the Fluff Principle: on a user-voted news site, the links that are easiest to judge will take over unless you take specific measures to prevent it. It leads to shallow content.

And we might say "I wouldn't do those things for karma!", and a lot of people wouldn't indeed, but the ones who'd do it would make the place less interesting for everyone.