Unfortunately there are a lot of people, including other trans people, who will insist that even if it is a trans man. It mostly comes from radfem types that view trans men as 'joining the enemy', and they usually accuse femboy trans men of transitioning in order to selfishly escape society's misogyny.
Source: me, a very femme transmasc person who has had this thrown at me both online and in person, even in queer spaces.
Stay or leave?
Future me would know what I was asking, and be able to answer wkth few words.