GitHub Actions self-hosted runners for NixOS where every job runs in its own throwaway microVM, and the number of VMs follows the host's spare CPU and memory.
Built on cloud-hypervisor, microvm.nix and the actions/scaleset client. No Kubernetes.
Self-hosted runners usually share one long-lived machine. Anything a job
leaves behind (Docker images, caches, tokens in $HOME, a root-equivalent
Docker socket) is there for the next job, including jobs from pull requests.
Here each job gets:
- a fresh VM with a fresh disk, destroyed when the job ends;
- its own rootful Docker daemon (with buildx, compose and QEMU binfmt), since the VM is the boundary;
- a single-use just-in-time runner config. The GitHub App key stays on the host and never enters a VM;
- Nix through the host's daemon as an untrusted user: builds and their cache
land in the host store, while the VM sees
/nix/storeread-only; - a network that reaches the internet but not the host or private ranges.
microvm-scaleset registers a runner scale set
in every organization listed in urls and tells GitHub how many jobs the host
can take right now. All organizations share the host: an idle one holds
nothing, and when several have jobs waiting, they take turns. Each assigned job
gets one cloud-hypervisor VM.
Nothing is sized per host. Every VM sees all host CPUs, boots with 2 GiB of memory and grows from there: the guest reports its available memory and memory pressure every two seconds, and the controller plugs memory in 1 GiB steps (virtio-mem) while the guest is under pressure and unplugs it once the guest has had plenty to spare for a while. Memory a job frees goes back to the host through free page reporting. Under CPU contention VMs share the CPUs equally.
A new VM boots only while:
- the 1-minute load average per CPU is below
maxLoadPerCpu, - available host memory, minus the reserve and VMs still booting, covers the boot memory,
- host memory, minus the reserve and what running VMs were given plus one 1 GiB plug each, covers the boot memory and its first plug, so jobs wait in the queue instead of starving running ones, and
- the
workDirfile system has more free space than its reserve.
The memory reserve is memReservePercent (default 10%) of host RAM, at least
2 GiB; no VM grows into it either, so when the host is full a guest under
pressure may run out of memory, never the host. The disk reserve is
diskReservePercent (default 10%). Each VM's disk is an empty sparse file the
guest formats on boot, and only takes the space the job writes. /tmp lives
on that disk too, so large temporary files cost disk, not memory.
When the host is busy it boots no new VMs; the jobs it accepted wait for one instead of overloading the machine.
{
inputs.microvm-github-runners.url = "github:OSSystems/microvm-github-runners";
# in your NixOS configuration
imports = [ inputs.microvm-github-runners.nixosModules.default ];
services.microvm-github-runners = {
enable = true;
urls = [ # all share the host
"https://github.com/my-org"
"https://github.com/my-other-org"
];
githubApp = {
id = "123456";
privateKeyFile = "/run/secrets/github-app-key";
};
network.externalInterface = "enp6s0";
# Anything jobs need on top of the base image.
guestModules = [
({ pkgs, ... }: {
microvm-github-runner.packages = with pkgs; [ gcc gnumake jq ];
})
];
};
}The scale set is named after the host (name defaults to
networking.hostName) and workflows can target it with runs-on: <hostname>.
To run a fleet, import the module on every machine and give them all one
extra label, which workflows target instead:
services.microvm-github-runners.labels = [ config.networking.hostName "nix-vm" ];runs-on: nix-vmAdding a machine is then importing the module there; its jobs come from the same queue.
The GitHub App needs organization Self-hosted runners: read & write
permission and must be installed on every organization in urls.
The host needs KVM. VMs use a /24 bridge with NAT (network.subnet,
default 10.231.0), addresses .10 to .254, and vsock CIDs 3 and up.
sudo journalctl -u microvm-scalesetA controller restart kills every running VM, so deploys leave
microvm-scaleset running: a changed guest image, controller or flag takes
effect at its next restart. Restart it when the host is idle, or accept that
its running jobs fail:
sudo systemctl restart microvm-scalesetThis holds for the deploy that moves a host from QEMU to cloud-hypervisor
too: the new unit says not to restart, so the old QEMU controller keeps
running against slots that no longer exist. Restart microvm-scaleset once
after that first deploy; no cloud-hypervisor VM exists yet, so it costs
nothing.
backend = "qemu" keeps the earlier fixed pool: poolSize QEMU VMs declared
through microvm.nix, each sized by vm.memGiB, vm.vcpu and vm.diskGiB,
admitted against memReserveGiB and memCommitRatio, with optional balloon
reclamation (balloon.*). These options apply only to it; setting them with
the default backend fails evaluation.
services.microvm-github-runners = {
backend = "qemu";
poolSize = 8;
vm = {
memGiB = 10;
initialMemGiB = 10; # lower to start smaller and grow through QMP
vcpu = 8;
diskGiB = 100;
cpuWeight = 100; # elastic CPU sharing under contention
cpuQuotaPercent = null; # optional hard ceiling; 400 permits four CPUs
};
memReserveGiB = 8;
memCommitRatio = 1.0; # 2.0 permits 200% of usable host RAM
balloon = {
enable = false; # deploy telemetry first, then enable a canary
floorGiB = 8;
headroomGiB = 2;
canarySlot = null; # e.g. "my-host-1" during rollout
};
};The controller writes /run/microvm-scaleset/status.json with host headroom,
commitment, the admission gate and per-guest balloon statistics; completed
runners log a runner memory summary. Start with reclamation disabled, check
ownership recovery across a controller restart, then enable it for one slot
with balloon.canarySlot. Avoid cgroup MemoryMax below a VM's boot ceiling:
it can kill the VM instead of reclaiming guest cache.
Early, but running real CI. The scale set client is a public preview, so its API may still change.