The problem is that letting fascist types run rampant and hoping they'll just go away historically doesn't tend to work out well.

I live in Portland. Up until 2021 or so fascists, proud boys, Patriot prayer, etc. All ran rampant in cahoots with local law enforcement. People were menaced, paintballed, attacked and all that. It sucked. The sniveling mayor begged those groups not to come in (they're never local) and act like that. Shockingly they did anyway.

Portland fought back

It was unpleasant, but it worked. Those assholes haven't tried to have their gatherings here in a couple of years. Self described antifa groups are still on edge but have largely diminished in activity. You don't see antifa rallies designed to menace going out to the surrounding small towns where those assholes largely come from. Unless you're sporting some obviously fash flair pretty much everyone gets left alone in town.

Both sides are not the same