You are correct in that clothes do not have a gender.

Which is why it is not called crossdressing but not conforming to the gender expressions which society and culture force upon us i.e. Gender Nonconforming. It's not about the clothes or assigned gender per se but more about those cultural ideas and pushing back against them.

As for the sexualisation of clothes we are probably best left out of that conversation. Hope some{one/many} else can help out there, we'd probably just label it as a fetish/kink or as it is in some cases the beginning of exploring gender.