The Haradrim swore allegiance to Sauron after long years of oppression by the Númenóreans and conflict with Gondor, during which Sauron worked to seduce them to his side. However, not all of them joined Sauron. Many sided with the blue wizards. You could draw parallels to modern times; not all Russians support Putin, and not all Iranians support Khomeini.

It's worth mentioning Dunlendings also went the same way after excessive Númenórean exploitation of their natural resources and resulting wars. They fought for Saruman (Sauron by proxy) alongside Uruk-hai in Helm's Deep.

Of course the Haradrim were dark-skinned - they were from the southern hemisphere of Arda, the huge landmass south of Gondor and Mordor. Most of it is on or below the equator.