As someone else mentioned, some jobs have micromanagers who get pissy if they think you aren't working, and keeping up appearances is draining.

From a different perspective, however, is that when it comes to creative fields specifically, downtime means you aren't improving your skills, creating portfolio work, etc. Due to the contracts creative jobs often have, anything you create on company time (and sometimes outside of company time, not that they can legally enforce it, but they'll try) is typically owned by the company. As such, working on personal projects during downtime is a great way to lose ownership of a passion project you're working on, and no official work means you aren't improving or adding to your portfolio (not that creative fields typically have downtime, usually they're the opposite).

It's speculated that that's why Valve had some major staff members leave the company a few years before Half-Life Alyx; they had nothing to do and were just sitting there spinning their wheels.