I think the vitriol that comes with the term “retarded” is different. People often make faces or try to sound intellectually disabled when they say it. It gets gross in a way that the other terms don’t match.

I imagine it was because none of those terms were attributed to mental disability.

Retarded means slowed, inhibited. One common use is a bread retarder, which a temperature controlled box you put dough in to slow the fermentation process. Retarded was also used scientifically to describe someone with an intellectual disability. While this sort of conception is out dated, they would describe someone's disability in terms of "having the mind of a 9 year old", so the relationship between the disability and time/slowed progress seemed really straightforward. At some point, saying a person was retarded transformed from being a scientific designation without prejudice to an insult for intellectually capable people who had done something foolish, and then eventually any misbegotten situation would be called retarded. There has been a movement both to use better language for the disabled community (hence me using the term intellectual disability) as well as stop using the word retarded as a pejorative.