For those not keeping count, total hardware spend is in the 13k-20k USD ballpark, by my count.
The thing that I like about this post is that it touches on many of the difficulties of running a homelab with many physical hosts. You might not need all or most of this setup, but at least you have an idea of how this particular design (a decent one!) scales after reading this.
- Array of off-the-shell compute servers with console access + networking + power
- ArgoCD + GitOps makes a K8 cluster declarative
- Talos makes the physical hosts that provide the K8 cluster be declarative
- Dedicated machines for storage, Control Plan, and Networking isolate the infrequently-changing stateful parts
This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab. I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
A single machine with a lot of CPU and a LOT of memory/storage is typically what people want---so that projects they're setting up are fast and having lots of old/idling projects is fine. My old recommendation was a mini-ITX with 128 GB of ram and a modern AMD cpu should take most people very far. Perhaps a smaller NUC/Beelink computer if you're not storage hungry.
However, keep in mind that a single machine will make it hard to tinker with large parts of the stack. It's harder to play with kernel mod settings if you need to constantly reboot all your projects and possibly nuke your lab. It's harder to test podman vs docker if in involves turning off all your contains. A more complicated home-lab gives you more surface area to tinker with. That's both more fun and makes you a better engineer. Of course, you can get most of this experience for far less money if you budget isn't quite as generous.
I personally prefer a digital nomad aesthetic, so I focus on small & simple on-prem hosts paired with cloud stacks. I'm willing to pay a premium on compute to have less commitment and physical footprint. I've been considering setting up a K8 cluster on Hetzner dedicated machines. In my case, that Mini-ITX box is actually a storage-optimized ATX build for backing up my laptop (daily-driver) and media server.
> This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab.
I kept waiting for the description of what it would be used for, but there was only a passing reference to learning how to run AI workloads.
For some people, buying and assembling hardware is the hobby. This gets old fast, especially when you add up how much was spent on hardware that's now sitting idle while it becomes more outdated year over year.
I agree that for typical learning cases the best solution is a single, cheap consumer CPU paired with a lot of RAM. For the $8000 spent on those 8 mini PCs, you could build a 256GB RAM box with 2 or even 3 nVidia 5090 GPUs and be in a different league of performance. It's also much easier to resell big nVidia consumer GPUs and recoup some of your money for the next upgrade.
It does look fun to assemble all of this into a rack and make it all work together. However, it's an extremely expensive means to an end. If you just want to experiment with distributed systems you can pair 128GB of RAM with a 16-core consumer CPU and run dozens or even 100 small VMs without issue. If you want to do GPU work you can even use PCIe passthrough to assign GPUs to VMs.
> For those not keeping count, total hardware spend is in the 13k-20k USD ballpark, by my count.
Yep! Right around the $13.5k mark.
> This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab.
Very compute focused for the specific projects that I intend to work on.
> I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
1gbps symmetric + a failover Starlink connection. Not a ton of large output workloads at the moment.
> However, keep in mind that a single machine will make it hard to tinker with large parts of the stack.
Very much in agreement here. This is one of the reasons I went with multiple machines.
> I'm willing to pay a premium on compute to have less commitment and physical footprint.
I also like this mindset, but my other hobbies are piano (and I'm very sensitive to the way that the keys are weighted, so I prefer playing a real grand piano vs a portable/mini electronic piano) and woodworking (even more bulky equipment), so I'm already pretty committed to putting down roots.
In my experience, yes and no. Homelabs seem work because they let you experiment while striving towards goals (normally hosting something). That means that the organic problems that arise are the best teachers, since you're honestly motivated to overcome them.
Getting your homelab to boot up and run containers is an honest problem to solve. Figuring out the kernel modules to let you pass through your GPU or run ZFS on root are actual blockers to hosting a gitlab instance.
Running gitlab on multiple nodes to get high-availability is an honest problem to solve. Trying to do it in multiple VMs just to see how it works might teach you something, but it can feel pointless unless it's serving a real goal. I think that choosing a more complicated setup is good because it is hard, and forces you to learn more to achieve the same goal (and ultimately, some of those skills will hopefully be useful).
Additionally, there used to be some limitations on consumer CPUs around nested virtualization, which made it difficult to run VMs in VMs. When I was hosting apps in VMs on a machine, I would want to play around with the machine's configs, but risked disrupting my hosted apps. If I broke something, I didn't have my hosted apps until I got around to fixing my machine. Having multiple machines ensured that I could tinker with one while relying on the other. The process, by accident, gave me an intuition for planning upgrades to infrastructure. This intuition bubbles back into the design process.
I don't often own infrastructure for multiple upgrade cycles professionally, so it is a good way to earn some wisdom on my own.
The thing that I like about this post is that it touches on many of the difficulties of running a homelab with many physical hosts. You might not need all or most of this setup, but at least you have an idea of how this particular design (a decent one!) scales after reading this.
- Array of off-the-shell compute servers with console access + networking + power
- ArgoCD + GitOps makes a K8 cluster declarative
- Talos makes the physical hosts that provide the K8 cluster be declarative
- Dedicated machines for storage, Control Plan, and Networking isolate the infrequently-changing stateful parts
This homelab does seem compute focused, which might be right for OP but is normally a mistake that people make when they build their first homelab. I'm wondering what OP's home internet bandwidth is. It seems odd to have so much compute behind a network bottleneck unless OP has high-compute-small-output workloads (ml training, data searching/analysis, NOT video encoding)
A single machine with a lot of CPU and a LOT of memory/storage is typically what people want---so that projects they're setting up are fast and having lots of old/idling projects is fine. My old recommendation was a mini-ITX with 128 GB of ram and a modern AMD cpu should take most people very far. Perhaps a smaller NUC/Beelink computer if you're not storage hungry.
However, keep in mind that a single machine will make it hard to tinker with large parts of the stack. It's harder to play with kernel mod settings if you need to constantly reboot all your projects and possibly nuke your lab. It's harder to test podman vs docker if in involves turning off all your contains. A more complicated home-lab gives you more surface area to tinker with. That's both more fun and makes you a better engineer. Of course, you can get most of this experience for far less money if you budget isn't quite as generous.
I personally prefer a digital nomad aesthetic, so I focus on small & simple on-prem hosts paired with cloud stacks. I'm willing to pay a premium on compute to have less commitment and physical footprint. I've been considering setting up a K8 cluster on Hetzner dedicated machines. In my case, that Mini-ITX box is actually a storage-optimized ATX build for backing up my laptop (daily-driver) and media server.