Part of the panic isn't moral, it's about money. Money in sports corrupts sports, and you can bet that any athlete making money by playing their sport is going to get real upset when they feel cheated out of potential money because they lost "unfairly" to a trans woman (which isn't actually happening).

As for why republicans and some democrats are against trans girls in K-12 sports? Well, they simply don't see trans girls as girls, and that are 'worthy' of equal rights and protection.

But yeah, you are absolutely right.