Freedom to associate is part of freedom of expression. You are allowed to say and believe a lot of things. And everyone you try to say them to is allowed to say they would prefer you never interact with you ever again (blocking). Further, if you openly associate with people who express opinions that people find shitty, those peioke are allowed to say they don't ever want to interact with you or anyone you associate with ever again (defederation)