I have a "when I stop being bad at web development" project idea for this, hopefully someone who has a development background can pick it up.

The idea is an open-source onboarding portal that takes all Lemmy instances from awesome-lemmy-instances and Kbin instances from FediDB and lets their admins tag their instances with what the instance is focused on, maybe through a dedicated community or something. This list of instances and tags is public so instances can't cheat the system with fake tags or get secretly blacklisted just because the project maintainer disagrees with them.

Users get directed to the portal and fill out a quiz with questions like "what are your hobbies", "do you prefer strict or lax moderation", and get matched to a list of the closest servers and recommended communities. There will also be a simple load balancing algorithm to make large instances less likely to be recommended. Of course, because it's open source, the algorithm and list of instances can be changed if someone wants to host their own portal.

Basically, something like Spread Mastodon that covers the entire known network and not just a few of the largest instances that are approved by mastodon.social.