Transphobes think all trans people are scary rapist men looking for a way past the impenetrable magic barrier that only lets women into women's toilets, and that there's no reason anyone would want to go into the men's toilets, so trans men are fictional and don't need or want any rights, just like the law doesn't give rights to elves and hobgoblins.