AMD is a better and newer card regardless. That would be my pick.
It's not even a contest between the two. The 980 ti is at best 1070-class while the V64 is closer to the 1080 than the 1070ti.
If we extrapolate from Kepler EOL, Nvidia will probably drop Maxwell support in the next year or two. Given there will likely never be a free driver for Maxwell, you'd be stuck using sucky old proprietary drivers in the not too distant future.
With the Vega 64, you've got world-class kernel and userspace driver support.
The biggest downside of the V64 would be that it's quite power hungry but the 980 ti isn't much better and if you limited the V64 power draw to the level of the 980 ti, I'm sure it'd still be faster.
Why in the world would you want the 980 ti, especially for Linux?
Btw, if you ever need to compare GPUs on Linux, use https://openbenchmarking.org/vs/Graphics/
https://openbenchmarking.org/vs/Graphics/AMD+Radeon+RX+64,NVIDIA+GeForce+GTX+980+Ti
The performance between the two -- VRAM aside -- is apparently significantly closer on Windows than in Linux, which I assume why OP is asking.
https://www.videocardbenchmark.net/compare/3218vs3808/GeForce-GTX-980-Ti-vs-Radeon-RX-Vega-64
The AMD card tends to slightly pull ahead on Windows too, but it's a much closer thing there.
In the meta-review at launch of the V64, it was 95-98% of a 1080:
https://www.3dcenter.org/artikel/launch-analyse-amd-radeon-rx-vega/launch-analyse-amd-radeon-rx-vega-seite-2 (German but the table is self-explanatory)
980 ti was about 7% behind a 1070 at its launch; over 20% behind a 1080: https://www.3dcenter.org/artikel/launch-analyse-nvidia-geforce-gtx-1070/launch-analyse-nvidia-geforce-gtx-1070-seite-2
I'm leaning towards declaring PassMark's GPU bench not being a very representative benchmark.
A very safe assumption.
I was using a water cooled VEGA 64 up until last year. I could play Cyberpunk at 3440x1440 med/high settings and pull around 40-50 fps. On a free sync monitor that is plenty for a smooth gameplay experience. But as far as I understand Linux doesn't support freeSync yet which is a shame.
If you can water-cool the VEGA 64, there is a lot of performance that is still left on the table. You don't even need to overclock as the drivers will push for as long as you have power budget and thermal room.
Even if the cards were equal in performance, the 2GB of missing VRAM from the 980 would be the deal breaker.
In linux go for AMD. It's not worth the hassle.
The AMD card is newer, faster, has more VRAM and has better driver support in Linux.
Even on Windows, between those choices, Nvidia is the loser.
A community for gaming on the GNU/Linux distros.