Slavery benefitted both white and non-white slave owners.

Edit: at least in the short-term, obviously you could argue differently for long-term effects.

This is an incomplete view of the history of slavery. By the numbers, the vast majority of the people benefitting from chattel slavery were white. However, slave ownership reached similar per capita rates among some of the Native American tribes, and there were instances of slaves being owned by free people of color. Notably, there are a large number of black tribal members in the United States. The United States government signed a number of treaties with the tribes in the 1860s that required them to free their slaves and incorporate them as full tribal members.

None of this is said to diminish the fact that the American system of slavery was a product of colonization by European countries. But it is rather reductive to claim that only white property owners benefitted from the atrocity that was chattel slavery.