Those packages themselves depend on xz. Pretty much all of them.

What you're suggesting would only make the xz executable not be backdoored anymore but any other application using liblzma would still be as vulnerable as before. That's actually the only currently known attack vector; inject malicious code into SSHD via liblzma.