I am building it! Or, well, not it anymore but a product that is heavily based on it.
I think we as a company recognize that the, like, 95% of AI products right now are shit. And that the default path from now is that power continues to concentrate at the large labs like OpenAI, which haven't been behaving particularly well.
But we also believe that there are good ways to use it. We hope to build one.
The thing your boss is asking you to do is shitty. However, TBQH humanity doesn't really know what LLMs are useful for yet. It's going to be a long process to find that out, and trying it in places where it isn't helpful is part of that.
Lastly, even if LLMs don't turn out to be useful for real work, there is something interesting and worth exploring there. Look at researcher/writers like Nostalgebrist and Janus - they're exploring what LLMs are like as beings. Not that they're conscious, but rather that there's interesting and complex things going on in there that we don't understand yet. The overarching feeling in my workplace is that we're in a frontier time, where clever thinking and exploration will be rewarded.