No I think this genuinely comes from a fear of trans people. Transphobia. But since you're being honest and good faith, and not hateful about it, I can see why it's confusing to be called transphobic.

People are allowed to have whatever sexual/romantic preferences they want, so long as they're not an asshole about it. Other people don't have the right to tell you what you like or dislike when it comes to intimate partners.

And yes, it can even be for stupid or shallow reasons -- as long as you aren't an asshole about it.

Bit of advice: you don't owe people detailed explanations. "I'm just not attracted to them" is fine. You're fine as long as you're truly a decent person in how you treat people you don't want to sleep with.