This is why I didn't stay in management. I had the same attitude as you. I didn't mind working late myself and if someone really wanted to because they had an interesting problem I'd usually let them cut out early Friday or come in late the next day. I got too much shit from other management and C levels about how my team never seemed to be around. I'd ask what was not getting done or what they were expecting that they didn't get. The answer was always just that they didn't see the team in the office when other teams were still there. They could have needed something!

I got tired of that shit. One of the reasons that I went into business for myself as a one man company. Now I don't have to justify anything but my own existence to anyone except who I directly report to. And the guy I directly report to likes the big ass penalty clause in my contract for terminating it early or without notice because accounting can't get rid of me without taking a hit or giving me time to finish up my projects. I'm judged solely on what I deliver. And I almost always come in ahead of time and with better results than most other teams because I'm not worked like a rented mule.