If they're being violent themselves, or actively advocating for it (as in: in a way that could reasonably cause others to be violent). I'm also not gonna try to stop anyone for punching someone throwing out slurs, though I don't think it's a great response. If it's just "i know this person is a nazi for whatever reason but they act like a normal person" I'm clearly against it and think the punching person is also in the wrong (to be clear, both are). Advocating violence against a group for their beliefs is just something I never consider okay, even if I think those beliefs make them the scum of the earth.

And even with all that I'd probably still press the magic button that makes all nazis drop dead, but mainly because I believe that would probably improve society quite a bit rather than because I think it is justified against them (since I would argue that really isn't any different from genocide even if it doesn't quite fit the definiton). That might make me a bit of a hypocrite, but it's not like that button will ever exist.