I'd recommend OPNsense over PFsense due to multiple shady moves by netgate (the parent company of pfsense), including moving to closed-source:

• pfsense is falsely open-source: https://news.ycombinator.com/item?id=26476030
• pfsense botched/rushed their wireguard implementation: https://forum.endeavouros.com/t/migration-from-pfsense-to-opnsense-drama-about-wireguard/12798
• pfsense squatted on competitor domain and used underhanded/defamatory practices: https://opnsense.org/opnsense-com/

If you don't mind the drama, both PFsense and OPNsense are perfectly competent router OSes.

Regarding hardware:

• OPNsense also sells rack-mountable server hosts.
• OP may not actually need a rack-mounted server -- I have several machines just sitting on a 2u rack-mounted shelf. My opnsense install runs on a cheap protectli box, and there's enough room for a handful of raspberry pis and their power bricks on the shelf next to it.