Modernizing my homelab

I recently started a new role on a systems and development team. My previous job was much more of a standard Linux sysadmin position, however this one is much more DevOps focused. Because of this, I decided to redo my lab using modern IaC principals, and I have learned so much from it in a relatively short period of time. Check out my repo at https://github.com/PhDbuilds/home.lab and the project page at https://github.com/users/PhDbuilds/projects/2 to stay up to date, as I am keeping all of this and everything I do public. I’m doing this so that others can see the decisions and progress that I make throughout, as I have many more plans for this lab and it will likely be much different by the time you are reading this.

What I’ve done

The foundation of the lab is a Proxmox hypervisor using Open vSwitch bridges for all internal networking. Rather than clicking through the Proxmox GUI to spin things up, the entire VM lifecycle is automated through a three-tool IaC pipeline: Packer builds a minimal AlmaLinux 9 golden image using a Kickstart file, Terraform clones and provisions VMs from that template using the bpg/proxmox provider, and Ansible handles all post-boot configuration. Secrets for all three tools are managed via direnv, keeping credentials out of the codebase entirely. I set up Packer Vault at one point, but it ended up being too much overhead for what I’m doing here. I might set it back up later though once my lab grows, or just to learn more about it.

The network is segmented into five zones — a LAN uplink to my home router, and four internal networks: Management (10.0.0.0/24), Prod (10.10.0.0/24), Test (10.20.0.0/24), and SecLab (airgapped network) (10.30.0.0/24). All routing and firewalling between these segments is handled by OPNsense running as a VM (polaris), with four VirtIO NICs mapped to each bridge. Getting OPNsense set up how I wanted involved working through firewall rule ordering issues, a WAN “block private networks” setting that was silently dropping RFC1918 traffic, and a stale DNS resolution problem caused by a leftover /etc/hosts entry from a previous build, to name a few. I also migrated off of dnsmasq, replacing it with Kea DHCPv4 for address management and Unbound for DNS resolution.

The main current deployed hosts are sirius (VM 101), a jumphost and Ansible control node on the Management network, and pulsar (VM 900), a PXE boot server running TFTP and httpd via syslinux (which I set up to develop a deeper understanding of the Anaconda installer and kernel command line internals rather than out of necessity). I’ve also got test machines set up on each network, and test machines for other things, like Grafana, etc. I built a Linux From Scratch machine on here too, so be on the lookout for a writeup on that soon too. Hosts follow an astronomy-themed naming scheme. Mostly.

Again, everything is tracked publicly in the repo and on the project board.

What’s next

Standing up a k3s cluster, applying CIS benchmarks via Ansible, and wiring it all together with GitOps. I’ve also got plans for some Grafana dashboards (currently just a test machine is set up with this), and more kubernetes stuff inspired by https://www.youtube.com/@mischavandenburg