Ideally, we wouldn't build infinite compute, just as much as we are actually using, and using it efficiently allows you to build less. We would still need datacenters even without LLMs, but they wouldn't need to be so gargantuan, because even the worst, inefficient, nodejs-based, intern-written server you could ever encounter, would be heaps more efficient (or at least less demanding) than any LLM. This is true even from an economical point of view, or any practical point of view, not just environmental.

To quote Tannenbaum: "You know you have the right computer when you are always using 99% of it. If you are using 100%, you are being limited by the machine. If you are using 98% you have bought more than you need". If datacenters were always running at 40%, we would build bigger ones.