I thought I was a man for most of my life and I've only ever heard a man refer to another male's upper chest as a breast, in a non-insulting way one time.

It was Mark Twain in some book. It sounded weird and antiquated to my ear like a lot of the language his books.

Its not common in the modern american english I have encountered. You should speak how you want, but I wouldn't recommend it. Not because it would clock you, but because there are plenty of insecure men out there who might take offense to it.

edit: third time trying to rewrite the first sentence probably got it right this time.