Well, it's obvious that any public data can be "harvested" by anyone, but with federation there's also the thing that data gets replicated among all the federated platforms so that each server has actually a copy of that data on it.

I wouldn't want my posts being stored on their server as a consequence of being federated with them.

you can consume content from their users without being on their platform

Wanting to avoid them and then go getting their content nonetheless, doesn't seem very coherent to me.