There are multiple kinds of slavery. Forced penal labor is a different kind of slavery than debt slavery, which is different than racial chattel slavery, forced marriage and so on. Most of those "have rights" to some extent. The US, for instance, had slave codes that delineated things like just how much owners were allowed to beat their slaves.

That is to say: When people talk about workers being wage slaves, they aren't saying the modern worker has it just as bad as other slaves, they're talking about institutionalized forced labor.