The goal is to share an http service privately on my tailnet but with an HTTPs connection. It seems others have spent lots of time figuring out and never sharing their solutions. I just got a setup to work satisfactorily so I'll share it. Criticism is welcome. First a few notes:

• I'm using headscale on a VPS behind Caddy.
• Official tailscale allegedly can do this out of the box with tailscale serve or tailscale cert.
• Headscale supports tailscale serve but not with https. Maybe if I removed caddy and let headscale to https directly it would. I haven't tested that yet.
• Yes I know https over wireguard is redundant. This effort is not only to make Firefox shut up but to make some clients that demand https work.

I also have deliberately avoided the "Private CA" because installing the cert of every client on my tailnet sounds like a nightmare. If someone can prove me wrong there, please share.

The context

1. I have a VPS and a public domain with DNS A and AAAA records that point all sub domains *.mydomain.net to that VPS.
2. The VPS runs caddy and headscale and is on the tailnet itself.
3. Caddy route the hs subdomain to headscale.
4. I have numerous devices on my tailnet, many running different http services but only some of them I want public.
5. I can publicly expose a service with https by simply adding an entry to caddy like so,

publicservice.mydomain.net {
    reverse_proxy privatehost:8080
}

Restart caddy and that's it.

The solution

First, I used sub domains of the public domain instead of headscales base domain. eg Use *.ts.mydomain.net instead of ts.net. I made a *.ts.mydomain.net A record pointing to my servers public IP. Caddy will automatically fetch https certificates for any *.mydomain.net domains automatically. It cannot for a domain not routed to it. (DNS01 authentication might circumvent this but I haven't tested that yet).

Second, I restrict caddy to only accept tailscale connections by using the bind directive. Otherwise it will accept and route public traffic. A caddy entry for a private service would look like this,

privateservice.ts.mydomain.net {
    bind 100.64.0.1 [fd7a:115c:a1e0::1]
    reverse_proxy privatehost:8081
}

The IP addresses come from the output of tailscale ip on the caddy/headscale machine.

Now privateservice.ts.mydomain.net routes to the caddy server with https but it gets a default blank 200 response from caddy because its coming from the machine's public IP instead of the tailnet.

The last step is to configure headscale's DNS to route private services to the headscale server on its its tailscale IP instead of the public IP.

# /etc/headscale/config.yaml
# ...
dns:
  magic_dns: true
  # base_domain is irrelevant
  nameservers:
    global: [ whatever ]
    split:
        # required to override the public dns records
        ts.mydomain.net: 100.100.100.100
  extra_records:
     - type: "A"
       name: "privateservice.mydomain.net"
       value: "100.64.0.1"
     - type: "AAAA"
       name: "privateservice.mydomain.net"
       value: "fd7a:115c:a1e0::1"
     # repeat for each service, always the same IPs

You can have base_domain be whatever or make it ts.mydomain.net if you want to be consistent and aren't worried about collisions with your extra records.

I tried using wildcard DNS records in headscale and it didn't work. It felt like it completely broke DNS without any clear warnings or errors. Idk if that's a bug or what. DNS just timed out internally

Limitations

All internal HTTPS traffic is routed through my VPS instead of directly peer to peer, which is a real bummer for internal latency. I think the only way around that is to give each internal host their own caddy server, have the DNS records point directly to them, but then use a private CA and all the hassle that's worth. Maybe DNS01 challenges will work...

Also while I have no public records indicating what private subdomains I have beyond *.ts.mydomain.net for DNS, I do have them for my TLS certificates... somewhere. I'm not super concerned about that though. I think only a private CA will hide those.