I thought it was driver problems at first because it happened so infrequently, but it has gotten worse this year.

It took ages to narrow it down to memory faults since I've been running stock settings forever. Stumbled upon this tool: https://github.com/GpuZelenograd/memtest_vulkan/

Found a load of errors, which went away after the down clock and it's been stable since. It must be just age related degradation, temps were never high but I know the repeated temp changes have an impact.