Over the last couple months my lab grew from a handful of services to a small production environment. Yet every time I set up a new VM I was rediscovering the wheel.
And with scale came other colorful problems. I used full backups to test changes. My documentation became scattered. There was no single source of truth. I was starting to hear the spirit of my old boss whispering in my ear: “No unicorns” — I needed to purge the unicorns. So I revamped my system, and my wish list had three items:
- Version control.
For configs and testing changes with branches. - Runbooks and documentation.
Accessible online for authorized users. - A professional recovery path.
If a service breaks, I should be able to rollback git history, recreate my stack from an offsite location, or — worst case — fully restore the VM from a backup.
And of course, it should all be self-hosted. I landed on a stack that I’ll walk through in detail in the next two posts:
- Gitea as a self-hosted Git server. Lightweight, enough for my lab. Has CI/CD and web UI.
- Docs and runbooks via Quartz 4 Markdown files for VM & service setup, recovery, and configuration.
- GitHub mirroring + offsite backups. I want a copy of everything somewhere outside my house. Github holds the config files; offsite peer lab holds the full backups.
This isn’t a novel architecture. But for years I’ve rolled my eyes at the term “GitOps” (which frankly was justified). I didn’t need it — at least not yet. I think its worth actually bumping into the problem you’re trying to solve before solving it. Knowledge can be taught; but wisdom must come from experience. Before, shoving GitOps down my throat would’ve been ineffective. Now I’m electing to learn it…
More on how I did it in the next post — GitOps — For the Impatient