Some right wing hate groups have adopted the character as part of their tribal language. For some this was enough to tarnish the character in its entirety via guilt by association. Beyond simply using the common meme format within their communities, however, it has been redrawn many times with aged racist stereotypes (such as overly exaggerated lips or noses)--in a sense there are explicitly bigoted "rare Pepes."

All in all the original meme is not hate speech, and many of its uses and references across diverse communities are not intrinsically hateful, but in hate filled communities it is used extensively and absolutely has racist caricatures associated with it. It's a dog whistle.

Pepe was sort of chosen by the staunchest Trump supporters as their symbol in the wake of his first term. That certainly sparked a controversy, because some folks were only exposed to assholes using Pepe, so they associated you with assholes, if you used Pepe. But other folks liked Pepe as a meme and decided to use it even more, because they did not want it to be appropriated by assholes. So yeah, that's why this question exists.

There's more details here: https://en.wikipedia.org/wiki/Pepe_the_Frog#Use_by_the_alt-right