I don't have a lot of knowledge about the technical details of these frontends and I'm probably using the incorrect terminology, but this is mostly an empty threat because

The only way Youtube could [restrict access to Invidious/Piped for good] is by making a Google account required to watch videos. Logged out users on Youtube’s official services like its website and mobile app currently use the same API endpoints these frontends use, so breaking those endpoints would disable its actual services for users without an account as well. You’ll notice that you cannot perform any logged in actions with a Google account on these frontends, and that’s because the frontends only use endpoints that don’t require authentication like watching videos and reading comments. This is the same reason why Twitter hasn’t shut down Nitter (the read-only privacy frontend for Twitter) yet, even though they would really want to.

Invidious/Piped do not use the official developer API provided by Youtube, so they can argue that they're not bound by its TOS. There is the concern that Google can implement severe rate limiting per IP to disable proxies or they'll try to make their unauthenticated API private, but people will probably reverse engineer it.