From a "don't care" position, Elon is probably the only one who has impacted them. For Spez a majority of users probably just use the desktop site, or official app and would be more annoyed with the mods impacting their experience than Spez for making changes. Zuck did a bunch of behind the scenes manipulations, but again the don't cares wouldn't have noticed.

The fediverse itself might be resistant to overall control, but you are still tied to an instance, so a rouge admin, or some spam in activity pub could still cause uprooting.