Have you added your flake.nix to your repo? When its not in the repo it does seem to do some copying, whereas if its checked in its pretty fast.