Managing a multi-machine fleet with NixOS across physical workstations, Incus virtual machines, and development containers usually runs into two persistent architectural dilemmas:

  1. Module & Dotfile Sprawl: Monolithic flakes where host hardware, system services, and user dotfiles get tangled together in a fragile import graph.
  2. The Secret Deployment Paradox: You want root and user passwords locked behind a physical hardware token (YubiKey Age identity), but you also need headless servers and Incus containers to boot, rebuild, and activate unattended without an administrator’s physical key present.

Over the past week, I rebuilt my entire infrastructure repository from scratch to solve these exact problems. The project is called r3x.

Here is how the architecture works, how secrets move through the pipeline, and the hardware-first security tricks that make it pleasant to use every day.

1. Dendritic Architecture with Den

Most NixOS repositories rely on a single directory hierarchy where hosts import modules directly. Refactoring or adding a new machine usually means editing a huge list of imports = [ ... ] lines across multiple files.

In r3x, we organize modules using Den (Dendritic design), splitting concerns into two clean namespaces:

  • r3x namespace: Reusable infrastructure aspects—system roles (desktop, devcontainer, live installer), disk partitioning schemes (Btrfs root + Impermanence via Disko), security services, and desktop environments (System76 COSMIC Desktop and Niri).
  • users namespace: Isolated personal configurations, dotfiles, desktop settings, and Home Manager modules.

Regional Settings & Host Templates

Hosts live under regions/<region>/hosts/<host>/. A top-level settings.json defines regional defaults, while a small per-host settings.json customizes the machine:

{
  "template": "desktop",
  "disk": "/dev/nvme0n1",
  "users": {
    "r3j0": {
      "groups": ["wheel", "networkmanager", "video", "incus-admin"]
    }
  }
}

A dynamic evaluation engine in modules/hosts.nix walks these JSON definitions and generates the NixOS system derivations automatically. If a container host like dev01 needs user r3j0 to have no sudo privileges while a secondary account r3j0-2 gets wheel, that distinction is declared explicitly in JSON without touching Nix code.

2. Solving Secrets with Vaultix

Secret management in declarative systems has always been tricky:

  • Decrypting secrets at Nix evaluation time leaks plaintext into world-readable /nix/store derivations.
  • Decrypting secrets at runtime usually requires either an online secrets manager (Vault) or having private keys stored permanently on every node’s disk.
  • Encrypting strictly to physical YubiKeys breaks unattended reboots and automated deployments.

To solve this, we deeply integrated Vaultix into a two-tier cryptographic pipeline:

[Admin Workstation]
       │
       ▼
Master Secret (encrypted with YubiKey Age identity: age-plugin-yubikey)
       │
       ▼ (vaultix-renc: batch re-encryption using evaluated Nix manifest)
Encrypted Host Cache (secrets/cache/<host>/, encrypted with host SSH public key)
       │
       ▼ (Committed to Git)
[Target Host Boot / Activation]
       │
       ▼ (vaultix-activate.service decrypts cache using /persist/etc/ssh/ssh_host_ed25519_key)
RAM-Only Plaintext (/run/vaultix-for-user/<name>)
       │
       ▼
services.userborn (sets user password hashes before login target)

The Workflow in Practice:

  1. Master Secret Authoring: Regional secrets (like shadow.age) are authored using vaultix-edit. They are encrypted strictly against the administrator’s physical YubiKey Age recipients (age1yubikey1...).
  2. Batch Re-Encryption (vaultix-renc): When a secret or host profile changes, vaultix-renc inspects vaultix-manifest.json (evaluated directly by Nix) and re-encrypts the master secret for each target host’s public SSH host key (ssh_host_ed25519_key.pub).
  3. Committed Encrypted Cache: The per-host secrets are stored in secrets/cache/<host>/ and tracked in Git.
  4. Offline Host Decryption: When a target machine boots or rebuilds, a systemd service (vaultix-activate.service) uses the machine’s local private SSH key (/persist/etc/ssh/ssh_host_ed25519_key) to decrypt its secrets directly into volatile RAM (/run/vaultix-for-user/<name>).
  5. Race-Free User Initialization: The decrypted password hashes in RAM are consumed immediately by services.userborn before user login services start.

Target nodes can rebuild, reboot, and deploy completely offline. The admin YubiKey is never required during remote deployment or machine restarts.

3. Context-Aware PAM Privilege Escalation

Hardware security keys should improve security without breaking standard workflows. A notorious issue with pam_u2f for passwordless sudo on Linux is SSH: if you configure PAM to demand a U2F touch, running sudo over an SSH session causes PAM to block indefinitely, waiting for a physical button press on a machine in another room.

In modules/r3x/yubikey.nix, we implemented a lightweight PAM helper (check_not_ssh) that inspects the process ancestry and /proc/environ:

                  ┌──────────────────────┐
                  │      sudo cmd        │
                  └──────────┬───────────┘
                             │
                  ┌──────────▼───────────┐
                  │   check_not_ssh      │
                  │ (inspect /proc/environ)
                  └──────────┬───────────┘
                             │
              ┌──────────────┴──────────────┐
              │                             │
    [Local Session (exit 0)]       [SSH Session (exit 1)]
              │                             │
    Skip pam_unix (1 jump)         Run pam_unix (requisite)
              │                             │
    ┌─────────▼──────────┐         ┌────────▼───────────┐
    │     pam_u2f        │         │   Password Prompt  │
    │  (Touch YubiKey)   │         │  (Unix credentials)│
    └────────────────────┘         └────────────────────┘

  • Local Session: The helper exits 0. With [success=1 default=ignore], PAM skips pam_unix and jumps straight to pam_u2f. You touch the key, and you’re in.

  • SSH Session: The helper detects the SSH session and exits 1. PAM bypasses the jump, evaluates pam_unix, and prompts for your Unix password immediately.

  • Physical Walk-Away Protection (and Threat Model Reality): A udev rule monitors the USB bus:

    ACTION=="remove", ENV{ID_BUS}=="usb", ENV{ID_VENDOR_ID}=="1050", RUN+="loginctl lock-sessions"

    The millisecond the YubiKey is pulled from the USB port, all desktop sessions lock instantly.

    (Before the security purists chime in: yes, I don’t have full disk encryption configured on this workstation yet. If someone physically breaks in and runs off with the bare NVMe drive, no amount of PAM U2F or session locking is going to save me. Then again, I also don’t own a weapon to defend the SSD in person, so LUKS remains comfortably on the post-1.0 roadmap.)

4. Hardware FIDO2 SSH Keys Alongside Proton Pass

We configure OpenSSH client to support resident FIDO2 keys (sk-ssh-ed25519@openssh.com) alongside Proton Pass (pass-cli ssh-agent via ~/.ssh/proton-pass-agent.sock).

Rather than fighting over the SSH agent socket:

  • Cloud-managed software keys are served via the standard SSH_AUTH_SOCK.
  • Hardware-backed ssh-sk keys are queried directly by OpenSSH via libfido2 without agent contention.
  • Keys are enrolled with resident credentials (-O resident) and touch-only verification (no PIN required during use). If you move to a fresh machine, running ssh-keygen -K pulls the resident key handles straight out of the YubiKey hardware.

5. Incus Orchestration with an nftables Trick

All virtual machines and development containers in r3x run on Incus:

  • devcontainer: Lightweight LXC container for fast, unprivileged terminal development.
  • desktop-vm: Full virtual machine with Wayland display integration.

A Justfile recipe compiles native squashfs images (containers) or qcow2 images (VMs) directly from Nix expressions and uploads them to the Incus remote, while OpenTofu provisions the instance and mounts a dedicated persistent storage volume to /persist.

The Incus + NixOS Firewall Solution

Running Incus on a NixOS host often creates firewall headaches. NixOS’s firewall (nixos-fw) drops unmatched incoming and forwarded packets, while Incus manages dynamic bridges in table inet incus under the @bridges set.

Instead of hardcoding bridge interface names or disabling the host firewall, we attached early prerouting and forward hooks:

networking.nftables.tables."incus" = {
  family = "inet";
  content = ''
    set bridges {
      type ifname
    }
    chain incus_bridge_mark_prerouting {
      type filter hook prerouting priority -150;
      iifname @bridges meta mark set 0x494e43
    }
    chain incus_bridge_mark_forward {
      type filter hook forward priority -150;
      oifname @bridges meta mark set 0x494e43
    }
  '';
};

networking.firewall = {
  extraInputRules = ''
    meta mark 0x494e43 accept comment "trust all incus managed bridges via @bridges"
  '';
  extraForwardRules = ''
    meta mark 0x494e43 accept comment "allow traffic for all incus managed bridges via @bridges"
  '';
};

Any packet arriving on or routed through an Incus-managed bridge is marked with 0x494e43 (ASCII for INC) at priority -150. The NixOS firewall automatically trusts all Incus bridges dynamically without needing manual rule reloads.

6. Built in a Week: Systems Engineering with AI

Putting this entire setup together took about a week of intense iteration. I heavily used AI coding assistants as a force multiplier for systems engineering:

  • Writing and testing the custom C PAM helper (check_not_ssh)
  • Packaging custom derivations for Vaultix batch re-encryption tools
  • Diagnosing early-boot systemd ordering and Userborn integration
  • Refactoring the module hierarchy cleanly into Den namespaces

Having an AI assistant handle boilerplate, C plumbing, and Nix syntax allowed me to focus entirely on systems architecture, security boundaries, and operational ergonomics.

Community & Acknowledgments

My journey into NixOS began at LinuxDay Vorarlberg organized by LUGV (Linux User Group Vorarlberg in Dornbirn, Austria), where the folks from BWI Suisse AG first introduced me to the power and beauty of declarative systems. That initial spark eventually evolved into the architecture powering this fleet.

Local user groups like LUGV are indispensable for fostering open-source knowledge, community, and giving engineers the space to discover tools that fundamentally change how they build infrastructure.