Wear hearing protection. Often. I work and talk to a lot of handyman type people and almost all of them have somewhat poor hearing. Even some close to my age (20's). The one thing in common? No hearing protection.

Even though hand tool woodwork is much quieter I still always wear earplugs. Even when running the vacuum I wear hearing protection. I also take it easy with the music, with a great pair of headphones you don't need to crank it up.

Also wear respirators often when dealing with almost any particulate. If you are working on something that produces a lot of dust, you probably want to wear a respirator. Doesn't matter if it's natural, even wood dust can mess you up. Especially during grinding or sanding.

A lot of shitty people will give you flak, but NEVER let people dictate your safety, specifically when they tell you to use less.