Home Lab Chronicles
Step 1 is About Choices
First, a little background information. I just shut down my startup, NeoBloom. That’s sad, but the timing just wasn’t right. Because of my two year foray into entrepreneurship, I have ~20 single board computers (SBC) laying around. Some of them have never left the box, but I have the hardware necessary to set each of them up with 1/2 TB of storage, usually super-fast NVME. They’re mostly Arm64 chips, but a few are ArmHF and even x86. A few years ago, I wired my house with beaucoup Cat 6. Most rooms have 2–5 cables terminating there with a central patch panel in the garage. I’ve got a couple of 48 port GbE switches. I’ve repurposed an old PC to run PFSense so my network gateway and firewall are strong. All in all, I’m about as well set up to do a garage tech startup as anyone.
I’m pretty good with Docker Swarm, but Kubernetes is where all the interest lies right now. Consequently, I have more than enough hardware to set up a good Kubernetes cluster. All in all, I should end up with 60+ CPU cores, 60+ GB RAM, and about 10 TB of storage.
Twenty machines gets tedious to maintain so I have to standardize and automate. The first thing is operating system. I’m usually a Debian guy, but I am super impressed with Ubuntu 20.04. I’m not running desktops on these SBCes, but I do run a Linux laptop and they have drastically improved desktop performance with this release. Nonetheless, 20.04 is a Long Term Support release so I know security updates will be available until at least 2025. Since we’re talking SBCes, I have to install this manually on each one. Luckily, that’s usually just me burning an emmc or sd card.
Storage is kind of tricky for clusters. You really don’t want to control which server an app gets put on by hand. So your storage needs to be distributed. In enterprise situations, you have dedicated machines for storage with redundancy both intra and extra-chassis. I can’t do that with my home lab. I’ve got lots of CPU cores and pretty good network bandwidth for a relatively small cluster. All the storage is flash. So we’ll just run an open source distributed storage system. It comes down to Ceph or GlusterFS. I’m old, I guess. I think of storage in terms of files. GlusterFS presents a paradigm I’m familiar with, so that’s what we’ll do.
Now, for automation. I don’t want to log into 20 machines, change the hostname, update the OS, add a user, change some passwords, create and mount a filesystem, etc. Ansible is the tool I’ve heard about the most and it meets my needs. Yes, I will have to manually watch my pfSense UI and set up static DHCP leases. That’s fine, because after that’s done, I can just tell Ansible to run a playbook to get everything in order.
For bare metal monitoring, I really like Cockpit from Redhat. I used to think Webmin was cool, but it’s showing its age. It gives me graphs, etc. I can terminal into any machine from a web browser. So that’s the choice.
Finally, what version of Kubernetes do I run? I’ve admired the guys at Rancher from afar for several years now. Plus, we’re running on small machines without a lot of RAM. Consequently, I will give K3s a go.
This is really high level so far. There is physical stuff I have to do like build a rack mount for 20 SBCes. I’m actually using wood for that and I may throw a couple of fans onto it, too. My rack in the garage is earthquake braced. Also, I’m starting with three nodes on my desk. Before I run Ansible, I will test at small scale. This will be fun.
