Hmmm never used xubuntu per se, but XFCE already seems like a good option for a low-spec computer. You could probably chip away at the resource usage some more by building your own desktop environment around a bare window manager, but honestly at this point the gain is negligible. If anything, you might want to look into tiling window managers just because they can offer a much more fluid and customizeable desktop experience as opposed to floating WMs. I'm using BSPWM right now, but considering switching to wayland with hyprland or qtile.

As for choice of distro: Not sure if NixOS would run well on your machine -- my homeserver is also a pretty low-spec computer (dual-core Intel Atom), and nixos-rebuild switch takes ages to run. Otherwise, go for Debian Testing if you want stability, Void if you want to not have systemd. There's also Devuan, which is basically Debian without systemd, but iirc it's not as popular as Void. But honestly if xubuntu works for you, then it's fine.

Also, some miscellaneous tweaks for improved performance:

1. IF YOU BOOT FROM A HARD DRIVE REPLACE IT WITH AN SSD! Solid-state drives are pretty cheap nowadays, and the upgrade from hdd to sdd is the single biggest performance improvement you can do for an old laptop
2. If on x11, disable compositing. On XFCE, there should be an option for it somewhere in the settings. If on a bare window manager, simply don't install any compositing manager (picom, xcompmgr, etc.). The downside is screen tearing and no proper window transparency, but it does put less strain on the CPU.
3. Consider looking into a custom linux kernel? I boot linux-tkg on my main laptop and it gives some pretty good performance improvements. But I'm not so sure whether it would translate well to a low-spec system.
4. Again, not exactly a performance tip, but consider formatting your boot partition as btrfs. Apart from all of the other cool features that you get with BTRFS, transparent file compression can, in some cases, be a win-win-win situation: less disk usage, faster file access, and longer SSD longevity. On low end system tho it may actually be the case that the CPU is the bottleneck as opposed to the disk, so transparent file compression may actually slow things down. Here are the settings I use for btrfs on my laptop (thinkpad with a core i7-5600U, mSATA solid state drive): lazytime,noatime,autodefrag,compress=zstd:3,discard=async,space_cache=v2,ssd. Again, not sure how well these translate to a low-end system, you should do your research.
5. If your system supports uefi, consider using EFISTUB as opposed to Grub. Much faster boot times. Another option is to add two efi entries: one for EFISTUB (and have that be the default), and a second one for Grub, for when you need to change boot options or boot into recovery mode.