https://kairos.io/docs/reference/faq/ I was wondering how it compares to Talos and now I'm confused. Is it basically a bootloader+host kernel shim and nothing else?
That is also what I came here to find out. Would love to hear from the creators of the project how it compares and contrasts to Talos. We've been running Talos for a few bare-metal and air-gapped cluster deployments with pretty good success but do have some pain-points.
Not OP, but when we tested it out it was painful to handle usb disks. The reason being that if you have two they get named sda/sdb randomly. We managed to overwrite the usb we were using to install talos since that one was named sda one boot and sdb the next. This lead ut to develop the “pullout technique” when installing…
This mostly only happened because it was a test cluster where we used usb disks, probably not a problem when one properly provisions.
Otherwise it was great! But it does feel akward not booting into an environment where you have a terminal at first
This does sound like it could be solved with better installDiskSelectors[0]. Talos has done a fair bit of work in improving this and UserVolumeConfigs in the last couple of 1.x revisions.
Alternatively, network booting in some fashion is an option. [1]
It's an opinionated vertical platform; if you run into an edge case, bug, or functionality you don't like, you are have to open a discussion Github and wait for a new release to fix or change things. The devs are very responsive, but the same as any open source tool, it's their project.
It perhaps depends on how much customization you want to do - GPUs and drivers, custom CNI, very specific disk settings. I've had more trouble with bare metal systems with varied hardware vs their supported cloud platforms, which are approved and tested.
I'm pretty positive toward Talos but if you stray from the happy path, by choice or accident, it can become challenging technically. And then you have sunk costs around choosing this platform and how hard it would be to restart from scratch.
It's very different from k3s. With k3s, you have a Linux installation like Debian or Fedora and you install k3s on it. You can SSH into the box, install any other Linux program not running in Kubernetes, etc. It also means that you need to run security updates and all the other stuff that goes along with administering a Linux box.
With Talos, it's just Kubernetes running on the box. There's no SSH or anything. Yes, it's a Linux kernel running, but you don't have a way of running stuff on the box outside of Kubernetes.
For me, Talos is great. If I'm setting up some boxes for K8s, I don't want to have to deal with admin'ing a Linux box. I don't want to login to the box and run some non-K8s service on it. I just want a K8s node and that's what Talos gives me. I think that's also the experience most people want. It's why people pay AWS, GCS, and Azure tons of money to get hosted K8s nodes rather than a Linux box they need to admin.
Been successful using Kairos to build and deploy an immutable RHEL9 to environments without internet access. Appreciate the flexibility to work with different distros and the maintainers provide great support.
RH also provides immutable OSes (Kairo uses some of their stuff) via Fedora Atomic and Fedora CoreOS, so for a single RHEL-like immutable instance, i'd just go CoreOS or Kinoite
The main reasons to use Kairo are perhaps more around the P2P mesh, using Cloudinit, deb vs rpm etc.
I'm (still) very interested about this project. Two years ago, I came across this when setting up a Raspberry Pi based "home lab" Kubernetes cluster (https://github.com/tantalic/shaving-yaks). I ran into some issues (I believe all Raspberry Pi specific) and ended up going with k0s. While I am happy with k0s, I do wish Kairos worked out for me and I might have to give it a try again soon.
We faced a similar need and wanted something more "familiar". We built EtchaOS as an in-memory, immutable variant of popular distributions like Debian, bundled with Docker and containerd, and powered by an imperative, stateful configuration management tool (Etcha):
https://kairos.io/docs/reference/faq/ I was wondering how it compares to Talos and now I'm confused. Is it basically a bootloader+host kernel shim and nothing else?
That is also what I came here to find out. Would love to hear from the creators of the project how it compares and contrasts to Talos. We've been running Talos for a few bare-metal and air-gapped cluster deployments with pretty good success but do have some pain-points.
Would love to hear about the pain points: Please elaborate, as I am currently in the decision phase and Talos as of now the top contender.
Not OP, but when we tested it out it was painful to handle usb disks. The reason being that if you have two they get named sda/sdb randomly. We managed to overwrite the usb we were using to install talos since that one was named sda one boot and sdb the next. This lead ut to develop the “pullout technique” when installing…
This mostly only happened because it was a test cluster where we used usb disks, probably not a problem when one properly provisions.
Otherwise it was great! But it does feel akward not booting into an environment where you have a terminal at first
This does sound like it could be solved with better installDiskSelectors[0]. Talos has done a fair bit of work in improving this and UserVolumeConfigs in the last couple of 1.x revisions.
Alternatively, network booting in some fashion is an option. [1]
[0] https://www.talos.dev/v1.11/reference/configuration/v1alpha1...
[1] https://www.talos.dev/v1.11/talos-guides/install/bare-metal-...
I recognize this from my bare-metal homelab setup. But at work we used VMs for Talos nodes so this was not an issue.
And if I had to deploy it on bare-metal at work I'd most likely use PXE booting instead of USB.
It's an opinionated vertical platform; if you run into an edge case, bug, or functionality you don't like, you are have to open a discussion Github and wait for a new release to fix or change things. The devs are very responsive, but the same as any open source tool, it's their project. It perhaps depends on how much customization you want to do - GPUs and drivers, custom CNI, very specific disk settings. I've had more trouble with bare metal systems with varied hardware vs their supported cloud platforms, which are approved and tested.
I'm pretty positive toward Talos but if you stray from the happy path, by choice or accident, it can become challenging technically. And then you have sunk costs around choosing this platform and how hard it would be to restart from scratch.
Is it better than k3s?
It's very different from k3s. With k3s, you have a Linux installation like Debian or Fedora and you install k3s on it. You can SSH into the box, install any other Linux program not running in Kubernetes, etc. It also means that you need to run security updates and all the other stuff that goes along with administering a Linux box.
With Talos, it's just Kubernetes running on the box. There's no SSH or anything. Yes, it's a Linux kernel running, but you don't have a way of running stuff on the box outside of Kubernetes.
For me, Talos is great. If I'm setting up some boxes for K8s, I don't want to have to deal with admin'ing a Linux box. I don't want to login to the box and run some non-K8s service on it. I just want a K8s node and that's what Talos gives me. I think that's also the experience most people want. It's why people pay AWS, GCS, and Azure tons of money to get hosted K8s nodes rather than a Linux box they need to admin.
Been successful using Kairos to build and deploy an immutable RHEL9 to environments without internet access. Appreciate the flexibility to work with different distros and the maintainers provide great support.
RH also provides immutable OSes (Kairo uses some of their stuff) via Fedora Atomic and Fedora CoreOS, so for a single RHEL-like immutable instance, i'd just go CoreOS or Kinoite
The main reasons to use Kairo are perhaps more around the P2P mesh, using Cloudinit, deb vs rpm etc.
I'm (still) very interested about this project. Two years ago, I came across this when setting up a Raspberry Pi based "home lab" Kubernetes cluster (https://github.com/tantalic/shaving-yaks). I ran into some issues (I believe all Raspberry Pi specific) and ended up going with k0s. While I am happy with k0s, I do wish Kairos worked out for me and I might have to give it a try again soon.
We faced a similar need and wanted something more "familiar". We built EtchaOS as an in-memory, immutable variant of popular distributions like Debian, bundled with Docker and containerd, and powered by an imperative, stateful configuration management tool (Etcha):
https://etcha.dev/etchaos/
I know balena etcher is a different product, but balena do operate in your space too so it seems like an unfortunate name clash.