I've stumbled upon this Github repo listing Lemmy instances and their differences

I think those are all fair points. Reddit did duplicate communities too, sometimes because some communities wanted to focus on specific elements of the topic they were covering, sometimes because of splits and disagreements, and sometimes just because it happened over time. People tend to find their niche, as do communities, but there will usually be a main one with the most members and activity.

Regarding individual instances, the way Mastodon has tried to manage that is by asking the people running instances to commit to a set of rules, one of which is giving appropriate notice should they wish to shut it down. This has been adhered to for the most part, and instances that don't voluntarily subscribe to those rules can get degenerated, or more likely just not promoted through the various explorer tools. So long as there's notice, there's opportunity to migrate to another instance and copy over data. It would be good to see something similar on Lemmy, if it's not already there (this is my first day!).

I'm commenting here from another instance (feddit.de) where I'm logged in and this community is available at https://feddit.de/c/lemmyworld@lemmy.world

Similarly, you can just access communities from other instances here, e.g. https://lemmy.world/c/asklemmy@lemmy.ml

For all of Lemmy: https://lemmy.nz/communities/listing_type/All/page/1

For just your instance: https://lemmy.nz/communities/listing_type/Local/page/1

There isn't really a way to search by topic as of right now, but you can search in those menu's with generalized subjects to find related communities