I've got an... overly simplified answer:

To a bigot, your challenges are annoying so they would avoid talking with you. So if you do want to engage with them, constantly (and aggressively) challenging their bigotry will prevent that. But why would you want to interact with bigots unless you absolutely had to? Their bigotry chafes you as much or more than your challenges chafe them. But also, if you live in some backwater place, and constantly seeking out and challenging bigotry means everybody around you wants nothing to do with you, then you're going to have a rough time.

In the same way, a vegan person challenging your dietary choices chafes you, and they may feel (and you may understand) that they have every reason to make the challenge, but it still is likely to prevent you wanting to engage with them. If most people around them are not vegan, and they seek out opportunities to challenge people, they're going to have a bad time.

But I also think there is a big difference between being in the minority and seeking out opportunities to challenge people (e.g. vegans in meat-eating society) and being the majority and seeking out opportunities to challenge people (e.g. religious area and self-righteous pricks starting conversations by asking if you are worried about going to hell).