Xournalpp is what you are looking for.

It has a horrible outdated interface, but it makes layerwise edits and preserves the original data of the doc.

To the best of my knowledge, there isn't a good FOSS PDF editor.

There's LibreOffice Draw, but as you said, it messes up the formatting. There's also Inkscape, but good luck if you have pages of text. You can also try Scribus, but I wouldn't say it's good (or intuitive).

If all you want to do is write over a PDF, put (not edit, nor remove) text, formulae, or images, then Xournal++ is very decent.

If I need to edit a PDF, I use (*gags*) Adobe Acrobat running in Wine (with Lutris is very easy). It's as proprietary and evil as it can be, but it's good at editing PDFs.