We're talking about an irrational fear here. Just because it's generally irrational, that doesn't mean that there isn't a valid base that the irrationality stems from. Evolution has hardwired us to be fearful of certain things, usually that is just a general discomfort and avoidance, but for some people those fears go beyond the rational response. Clowns may seem like a silly thing to be afraid of to someone that's not scared of them, because you have a rational response to that stimulus.

For a more general example, let's look at a different phobia, one that people often reference incorrectly, referencing general discomfort as if it were a phobia, claustrophobia. Humans generally don't like cramped or small confined spaces, it reduces mobility and options, which evolutionary meant survival, but to those with claustrophobia that instead feels like a real life or death situation, regardless of actual safety. You put a "normal" person in a dog crate and they'll be uncomfortable but they know they aren't going to die just by being in there. Put someone claustrophobic in one however and they'll likely feel like they can't breathe and are dying, despite there clearly being enough airflow, and while they know there's enough air, their fear won't let them process that correctly, it's irrational.