Politics are one thing, but there are servers that without-a-doubt bad. Some for example literally have "lolicon" in the URL. Not doing that also may or may not invite trouble, depending on your local laws.

You dont't have to deal a_troll_and_their_2_or_3_friends.lol, but there are lists of known-bad servers (#FediBlock brings up some results). Im talking about defederating those, that can almost universially be considered bad.