There is a kind of subtle technical problem with karma, which is (technical) problem of trust. I haven't looked at the Lemmy spec, much less the source, but decentralized systems always have an issue of, who can you trust?
Let's say I want to have a user account with inflated karma for some reason. I can spin up an instance and simply assert what my karma is—and if I need to, I can create a bunch of fake accounts on my instance and create a bunch of fake posts and comments and assign fake karma scores to them, so that it can be audited.
Now if other instance owners get wise to what I'm up to, they can defederate me. But this creates a few immediate problems, including the problem of adding more administrative load to instance owners generally. The bigger problem is the witch hunting that could ensue, if a culture of karma were to develop as it has on Reddit.