The biggest difference, and why your comparison falls flat, is that "retarded" has more than one meaning. First off, it's been used medically. It's also used for something that slows or stops something "x retards progress of y", "flame-retardant" etc.

The N word has ALWAYS been used to refer to other people of a particular skin color, and usually negatively. As such, there's never really been a legitimate use case.