I actually do own an external usb sound card, this problem doesn't happen there. I don't use it because it has a severe crackling/static noise constantly playing, I could also never figure it out. My luck with audio in Linux is just not there. EDIT: after further testing, it also happens there but I REALLY have to boost both the mic and sound volume. But it happens, which is scary.

I was imprecise in the description of my connection. My headset only has one of 3.5mm who has both micro + headphone combined, like the one you use on your smartphone. I think it's called TRSS. I then use the Y cable that came from with my headphones to split it into two separate 3.5mm connections, one for input and another for output. When I disconnected my headphones, the Y cable remained connected to audio card. It appears that the Y cable is the source of the problem.

If I only use the microphone jack of my Y cable, and connect my speakers to the sound card output, it works fine. The should outputs the the speakers, the input comes from my microphone and it doesn't loopback.

I've tried a second pair of Y cable but the same is happening. Both are oficial Beyerdynamic cables, maybe swapping to another brand could fix it?

At least now I can make some sense out of the problem, even if I'm unable to fix it. I opened a bug report on pipewire as you suggested but I doubt anything will come out of it. I think I'll just buy a non-ancient USB sound card and see how that works out for me.

I use a Beyerdynamic Custom One Pro with a Vmoda Boom ProX microphone. It connects to my internal audio card via a normal 3.5mm jack. The problem happens even after I unplug them.