Hm, yeah. I guess I was thinking of it in a movie context, where the Director is basically the big boss and everybody on set answers to him (including the Casting Director). Still has to answer to the producers, but they're usually not hands-on, and if they disagree with what they're doing, their only leverage is the threat of firing him (but replacing a Director mid-production is a huge headache, so it's not often done).

But in a TV context, yeah, the Director is often only there for one episode, and it's the Show Runner who's usually the big boss, who controls the project as a whole, and who the Casting Director answers to. One episode's Director wouldn't be in charge of casting. Maybe they'd be able to do casting for a small role that only features in the one episode, but that's it.