Seems that they only counted slaves from the South. I bet that including New England slave owners would take it from "many" to "most", if not even "almost all".

While not on an industrial scale like in the South, there was still a lot of slavery in the original colonies and their successor states, which played a big part in enabling generational wealth for any family rich enough to buy a slave at the time.

Maybe if we bring slavery back for a few years, that will even things out a bit.