Hello, I also run k3s on a single node here.

Yes, it is more complex in some ways. But what I love about kubernetes, is the helm package manager and it's ecosystem, that makes it easier to user packages from other people and organizations.

They can be searched here: https://artifacthub.io/

There are many that you may be interested in, like forgejo.

The big thing I like about helm vs docker-compose, is helm is another layer on docker-compose. If an app receives an update that causes it to need another service/container deployed, that will be reflected in the helm release, which orchestrates containers. But with docker-compose, you would have to manually update the config file format in order to handle such changes.

Because this setup is more resilient, I actually just auto update all my apps to the latest version on kubernetes, and this is one of the big reasons why I use kubernetes instead of docker. Yes, things do occasionally break, but they mostly break in predictable, easy to handle ways like "config option changed" or "database needs migration". Plus backups/snapshots to ensure if there is a bug or something I can simply roll back.

I run k3s on a single node and it’s not really that much more overhead than Docker Compose if you understand k8s. I mostly have a deployment.yaml, service.yaml, ingress.yaml, and network-policy.yaml for each service that I’ve copy / pasted and updated. Here are some of the benefits over Docker Compose for my setup:

• Has a built-in Traefik reverse proxy / ingress controller so I can access my services by domain name instead of by port, like http://jellyfin.lan/, http://forgejo.lan/ (using local dns on my OpenWRT router)

• I use the Calico CNI so I can have network policies for each service to allow them to access only what they need. If a service doesn’t need internet access, it doesn’t get it.

• I use Bitnami Sealed Secrets to store my secrets in YAML files that can be safely stored in git

• ConfigMaps make it easy to manage configuration files

• Easier to have separate YAML files for each service while sharing a network between them. Services connect to each other like http://forgejo.forgejo.svc.cluster.local/

Of course, if you’re looking to load balance across multiple machines, k3s makes even more sense.

Edit:

k8s is the clear industry standard for container orchestration at this point, so if you want something beyond Compose, a lightweight k8s distribution like k3s is an obvious choice.

Regarding different distros: yes, absolutely possible! Considering that swarm development is basically stopped the risk of inconsistencies is lower than back in the days but I’d pin the engine version on all nodes and only explicitly upgrade all at the same time to a newer version.

Regarding swarm itself, even though I understand your desire, I’d recommend to have a look at something else. Have you had a look at Nomad for instance? It’s not compose compatible but I guess these days it’s a simple prompt to convert the compose files, it’s supposed to be simpler than K8s and still offers many perks (like the failover you are looking for).

Also, from what I just read, Proxmox is working on improving the container support to allow “native” containers (inside VMs similar to Kata containers). They’re not there yet but I am wondering if it’s worth the effort from your side right now 😅

Either way, have fun! A fellow homelab buddy 😄

I'm running dokploy in swarm mode on 3 nodes.

The only downside is the development of swarm is basically halted and some features are missing (like passing /dev devices to a container, you have to use dirty workarounds) but otherwise it just works.