You can use pretty much any word in any context, and the context you previously used "homophobia" in was you saying: "homophobia isnt fear or aversion just because it ends on phobia".

That's exactly the reason that homophobia does mean "fear or aversion". It is used in other contexts as well, and mostly when it's used it's used to describe prejudice that is implied to stem from fear or aversion to homosexuality.

I get your point that colloquial use isn't always the same as the strict prescriptive meaning of the word, but do you get that just because someone doesn't know the prescriptive meaning of the word doesn't mean the meaning doesn't exist?

It's the same as the use of the word "literally." Colloquially, you can use "literally" as emphasis, but there is an actual prescriptive meaning, and that meaning is the reason that the word is used in that context to begin with. It was always used as emphasis, but the "correct" way to use it as emphasis is in a context which "literally" can actually be applied to without it sounding weird.

And the use of "literally" as unconnected emphasis is accepted by major linguistic institutions, so it's not wrong per se, but...