Maybe try to use the "Gender slider" embeddings : https://civitai.com/models/89709?modelVersionId=95959 I got the same issue with a LORA of a short haired woman who turned into a man one I trained it... Using those embeddings worked well for me ! (The result were close to the generated images with GS-DeMasculate-neg, but she was trained from generated picture so I can't confirm if it will keep your face correctly...)