I often hear people complain how much work it is to keep a homelab up to date. Once you deploy more than one service, you'll eventually loose sight of versions, updates and changes. I used to think this was just a thing I had to deal with.
Enter Immich. It's a very neat piece of software that allows you to backup & organize pictures. When I first set this up, the APIs weren't super stable and updates to the app sometimes broke the functionality entirely as the server didn't know about the new API endpoints yet. This was very annoying so I invested some time into building a flow that'd allow me to keep track of things to update with ease.
The results are heavily inspired by the way we deploy at Grafana, but substituting for "homelab-scale" where it makes sense.
If you don't want to read on and rather just explore the code yourself, the
source can be found in the infrastructure repository.
The goal
In an ideal world, I get notified of a new update and can deploy the new version with the click of a button. Auto upgrades aren't feasible as sometimes breaking changes slip through and require manual intervention.
Everything in my homelab runs in Kubernetes so I needed to find a way to continuously check for updates to container images and apply the changes to the manifests.
Running this directly against the Kubernetes API sounds like a bad idea so I instead opted for a GitOps based approach in which updates are reflected as PRs to my infrastructure repository.
The cast
For this to succeed, I decided on the following three tools:
- Renovate - to find available updates & create PRs
From my research, it's the most flexible solution out there and allows for custom regular expression based managers. I use the CLI variant to keep it simple and not set up yet another service. - Flux - to automatically apply the changes
I could have opted for ArgoCD as well, but the simplicity of flux is a real advantage in a resource constrained homelab environment. - Forgejo Actions - to run Renovate & render my manifests
This could have really been any CI but as I already use Forgejo, it was the path of least resistance.
You'd be right to ask why I need to "render" my Kubernetes manifests but the reason is actually pretty simple: I use jsonnet and tanka to define my cluster manifests. This allows me to reuse components and more importantly: deal with helm in a sane way.
The downside being that flux doesn't support jsonnet natively. To solve this, I opted to choose the same approach we use at Grafana Labs: render the resulting manifest to a separate repository.
Setting things up
To set up Flux, I simply followed official documentation using the flux CLI.
Setting up Renovate was relatively simple as well. I opted to run it on a schedule using forgejo actions. The resulting workflow definition is very small and easy to maintain.
For the image rendering, I opted to run tk export in a loop. This also allows me
to use tankas built-in environment filtering to toggle flux on/off for
specific environments.
Pro tip: make sure you disable pruning on your flux resources and separate your flux installation repo from your rendered manifests. Otherwise you'll loose everything in case you accidentally delete your flux environment. Don't ask me how I know…
Matching images
Renovate obviously doesn't support my arcane setup out of the box. So I had to teach it how to detect and update image versions. In the end, I came up with these two matchers:
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"prHourlyLimit": 0,
"prConcurrentLimit": 0,
"packageRules": [],
"customManagers": [
{
"customType": "regex",
"fileMatch": ["\\.libsonnet"],
"datasourceTemplate": "docker",
"matchStrings": [
"image=[\"'](?<depName>[\\w./]+):(?<currentValue>[\\w+.]*)",
"image:\\s[\"'](?<depName>[\\w./]+):(?<currentValue>[\\w+.]*)"
]
},
{
"customType": "regex",
"fileMatch": ["\\.libsonnet"],
"matchStrings": [
"\\/\\/\\s*renovate: datasource=(?<datasource>[a-z-.]+?) depName=(?<depName>[^\\s]+?)\\s+tag:\\s*[\"']?(?<currentValue>[\\w+\\.\\-]*)"
]
},
]
}This will match image: "..." references in objects as well as image="..." calls
in functions. For cases where this doesn't work (specifically version arguments
in helm charts), the second matcher allows me to control dependencies with
comments:
{
image: {
// renovate: datasource=github-releases depName=immich-app/immich
tag: 'v1.141.1',
},
}And that's really it! I haven't set up a manager for helm charts in tankas
chartfile.yaml yet, but this is on my ever-growing todo list.
Rendering manifests in pull requests
By rendering the diff on every PR, I can avoid catastrophic failure (like deleting the flux-system namespace) early if the diff is suspicious. To do this, a workflow runs on every PR, renders all environments and comments the diff.
The tricky part here, was getting the caches right but once this was set up, rendering takes less than a minute.
Managing secrets
Before setting up flux, I used git-crypt to keep my secrets safe. While it works
great when applying changes locally, it is very cumbersome to use in CI/CD
systems. Instead I opted for the flux sops integration. It decrypts SOPS YAML
when applying manifests so I just need to keep all secrets in SOPS managed
files. Since SOPS encrypts contents while keeping the data format, I can import
the secrets files in jsonnet using importstr and export them as-is.
The final result
With everything set up, I am greeted with nice PRs like infrastructure#205. Without leaving the browser tab, I get the release notes and the exact updates to the rendered manifest.
Obviously this isn't the end. Once I have a more stable messaging setup, I'd love to receive notifications about new updates & state changes directly but that'll have to wait until I get around to setting up Matrix again.