Some people are reporting it happens when your accounts get flagged by YouTube for blocking ads and that using a private browsing session can be used to bypass it, so it's possible this isn't a blanket thing?

Either way, they can go fuck themselves.

If you're on Firefox and using uBlock Origin (which you should), you can add the following to your filters list to essentially disable the delay:

! Bypass 5 seconds delay added by YouTube
www.youtube.com##+js(nano-stb, resolve(1), 5000, 0.001)

It doesn't fully disable it, just makes it almost instant, because Google has been doing shit like looking at what gets blocked to combat ad blockers recently.