At the end of the day they're still using that capital to exploit people by being landlords. Even if they earned that initial capital through hard work, the moment they invest some of it into a down payment on a house and begin to extract profit/equity via someone else's labor, it becomes exploitation.