An argument I've heard against it is that it's overly harmful against non-western religions, specifically Islam. A pretty common tenet in Islam is some kind of head covering for woman. Banning that is a pretty sweeping reform. Christianity and Catholicism don't have anything like that, and if you really wanted to wear a cross you could just hide a necklace under your shirt. And Judaism, most non -orthodox Jews don't wear a yamaka 24/7. So in the end (typical) white religions aren't affected while minorities are.

Personally for me I don't care about wearing a religious symbol as long as you're not pushing your agenda. I don't care if my boss has a Bible on his desk any more than if he had a copy of dragon Ball z.