I'm up for my hardware refresh at work and as such, have been looking into NixOS again. I'm currently using it for my home-manager configuration but this has been broken for a while on my system. I suspected it was a fluke during which upstream packages weren't done building yet but when I tried to bootstrap a fresh config, I realized that my nix installation was broken 😅. I re-installed nix and everything is working now!
On the home-cluster side, I sucessfully set up a SeaweedFS cluster. Initially I started with the seaweedfs operator but this didn't allow me to deploy it exactly how I needed so I rolled my own deployments. This also helped me understand the architecture better and I really appreciate the though that went into it. The SeaweedFS documentation is a bit hard to find (it's hidden in the github wiki) but it's very detailed and answered (almost) all of my questions.
I ran benchmarks against both the current piraeus setup and seaweedfs with postgres for metadata storage and sadly it's quite a bit slower. I'll try out using the redis backend (which is supposedly the fastest) and also try running juicefs against the seaweedfs s3 api next.