That sounds pretty reasonable, and although I do think that you inherently need quite a few users for something like Lemmy to work, you don't really need that many, so federating with them would really only cause problems.

If someone from an unfederated instance wanted to access this community, they could just create an account on it or a federated instance; it's not like it's uncommon to make alts (or new accounts in general). Overall I just think it would lead to more spam content and moderation issues on FMHY.