It's not like you can use that time freed by automating gear shifting for something else.

It's a tool, yes, but personally, I like having more control over tools I use. I'd choose a cordless drill that I can set the torque control myself over one that doesn't have that option.

Pretty much every consumer in every auto market agrees with you. There are downsides to manual, you can grind a transmission's gears to dust in a couple of days if you do it wrong, you really can't trust someone to drive your car at all, you are much more actively driving, so you're paying more attention, but you're also more stressed, if you're in bumper to bumper traffic, you will have to do the most difficult aspects of driving every few seconds to inch along for a half hour or more and that's REALLY shitty, if you need to stop on any kind of hill, you have to be aware your gonna need half a car length or more to get into gear where your just going to be falling down that hill while you convince yourself you don't need to panic and you will catch the gear before you're past the point of no return. You get better mileage, you get better control, you pay attention more, you focus more, but it's not all roses, the risks usually aren't worth it for modern car buyers.

I totally understand people who like the whole ritual of the manual car. Hell, that's how I feel about music making. But there's something to be said for just getting something to happen without much effort.