In a philosophical sense of the strict definition, you're correct. I see no good reason to use our language like that though, as it would inevitably hurt trans people. I choose to instead use gender as an identifier assigned by each individual, as it's our colloquial definition and less harmful to trans people. In my opinion, if someone identifies as a woman, she is a woman, regardless of external perception.

And yes, I also agree that gender would be better abolished and relegated to a vibes-based, self-identified label for people that want it.