I just want the freedom of choice for everyone. I'm tired of living in a world where there are people who think they know whats best for everyone else.

I hear you <3 I think we share the same goals, and might even agree on our ideal world (the best solution IMO would be to let users individually defederate from instances as a first line of defense and then have instances defederate from each other after the community approves of it only as a last line of defense). But I think practically speaking, there are places I disagree with you.

(Also, I apologize for the length of this comment, but I think we are in the early days of something extremely important with these federated and decentralized networks, and hashing out the culture and pros and cons and technical features of these things is really important to do carefully and intentionally to lay the groundwork.)

I think being able to defederate from hate groups and rogue instances is a very important feature of the network, because it allows communities to avoid other hostile communities instead of being forced into one giant, common, one-size-fits-all compromise social sphere where they are forced to coexist with communities who hate them or even want them dead.

A lack of this ability was one half of the problem with corporate social media, the other being that to solve this they had to use centralized moderation that, when banning a user, utterly banished them, instead if just separating two groups but allowing them to remain in a common network and have indirect connections, as in Lemmy.

Mods on one instance aren't able to moderate the comments and posts of people on other instances, so the only way for them to properly deal with users and communities that consistently refuse to moderate their own members is to defederate. If there's no way to defederate, there's no way for communities to essentially moderate their interactions with other communities, so if a group wants to do a mass harassment campaign against e.g. trans people or black people, all they need to do is start an instance where they'll never actually be banned or muted or have their comments removed, and they can then harass people on other instances with complete impunity, consequence-free, with the harassed people having no recourse but to individually choose to block the harassing instance.

Preventing that sort of mess where everyone has to fend for themselves with personal blocklists is the whole reason mods were invented in the first place, to be a first line of defense for everyone else, so we don't have to deal with hate and nonsense constantly. They're essentially a community defense organization. It would make lemmy basically unusable for marginalized people who face a ton of hate and harassment and this information and concerning directed at them to be left completely to their own devices on this front. Yes, they may be a somewhat centralized locus of power, but would you object to moderators doing their other functions such as banning or muting users, removing comments, etc too? Because this is very much in line with those things.

Anyway, independent of what the mechanisms are for deciding when to defederate, I think you have to look at the cost/benefit analysis, instead of just declaring it bad, and I have personally experienced the benefits. Many Mastodon instances regularly defederate from many other Mastodon instances, and yet that network has not turned into anything like what you fear, and I've directly felt the benefits of such defederation as certain never ending sources of problems and hate are effectively quarantined from the people that don't want to deal with them.

Moreover your assumption that allowing defederation will cause a degenerate network condition is verging on a slippery slope argument. Defederation will never be widespread enough to turn the Fediverse into just an array of mostly centralized hubs like corpo social media because it's a very extreme move you only do to cesspools of hate and fascism, so the network will just be a decentralized mesh network that isn't completely directly connected, which might be a small sacrifice in some abstract metric, but has direct benefits in making communities more liveable for people that aren't okay with being on something like 4chan.

Now to answer your main point. The decision to defederate one instance from another is the decision of the instance's owner(s), and so may be unilateral in that sense, yes, but thanks to the overall interconnectedness of the network, unless your current instance literally defederates from ALL other instances, if the mods on an instance do something you don't like, you'll always be able to find or make a new instance that is connected to all the people on the old instance and all the instances you disagreed about the blocking of. That's the most crucial aspect of the federated network — freedom of association and freedom from network effects. That freedom of association means there's competition between instances (and little barrier to entry for making new ones or switching), which will incentivize mods to implement collective decision-making such as polls. Additionally, most instances have a mission statement or description of the attitudes and goals of the instance and so it can be assumed that if people join that instance then they agree for the most part with that ethos and so as long as the mods act in line with that then it's fine.