Dude your breath probably smells and people are too polite to tell you.

On your tongue there are a lot of anaerobic microbes (on everybody's tongue, it isn't a hygiene thing, they just live there). When you have your mouth closed for a long time, it creates a oxygen poor environment and these microbes thrive in that. They cause bad breath, so after people sleep or simply have their mouth closed for a while, the breath starts to stink. If you look in a mirror and your tongue has a kinda white or brownish tinge, this is because of the microbes. Everybody should brush your teeth and especially your tongue after sleeping. When looking in a mirror the tongue should be entirely pink and look healthy. Clean the tongue as far back as you can, to prevent bad breath.

If you smoke this isn't true, your breath will stink no matter what.

So for your teeth it probably won't matter much not brushing in the mornings, but for your breath it matters a lot.

Genuinely feel for people who have to spend time in enclosed spaces with you.

Congratulations for winning the genetic lottery with the robustness of your teeth, but brushing is also about hygiene. Body odour isn't physically bad for you, but it is for everyone who has to be near you. Same with your breath.