You can't always tell someone's ancestry from their appearance. Plenty of white/white-passing people, especially in the Americas, have recent black ancestors. Take this Appalachian community, for example.

According to genealogical research published by Henry Louis Gates Jr., 17 of Leo's American ancestors were Black, listed in census records as mulatto, Black, Creole or a free person of color. His family tree includes slaveholders and enslaved people, Gates wrote in The New York Times.

Whites have been slaves for a long time. You can start with reading up the history of the Irish.