Providing binary packages is kind of a futile effort since you'd need one for each distro/version and architecture. I don't like/use Docker either but it is at least a distro-agnostic format. If you want distro packages, ask your distro maintainers to package it.
Are you compiling it directly on the Pi? If so, try cross-compiling, it's going to be a lot faster that way. I just cross-compiled the server in about 5 minutes on my machine as a test.