In a pluralistic society it is as bad as racisim or prejudice against Muslims or misogyny. If your fear of a given group informs a change of a long-term friendship just from that friend coming out, then yeah, it's not only a social dysfunction but informs what job positions you can have that are public-facing or public-serving.

For instance, transphobes shouldn't be doctors, pharmacists or defense attorneys if their transphobia.migh affect their regard to clients.

Maybe in a police state where citizens are expected to adhere to an extreme level of conformity then being transphobic might not cause additional harm, but no one would want to live in such a society.