nix

Nix

Stay frosty like Tony

A home for my system configurations and home lab using Nix Flakes.

Be warned, I’m still learning and experimenting.

As-is network and systems diagram

Features and Todo

Diagram of Earth's layers

Bedrock (Networking)

Features:

Todo:

Substratum (Virtualization and Systems)

Features:

Todo:

Subsoil (Foundational Services)

Features:

Todo:

Topsoil (Kubernetes)

Features:

Todo:

foolish mortals

Organics (Applications and nice-to-haves)

Implementation Notes

See also:

Bedrock

Substratum

Pre-requisites:

HP EliteDesk 800 G3 Micro/Mini.

  1. Mash F10/Esc to hit the bios (this was a thowback and a pain to do). Or just use systemctl reboot --firmware-setup ~ Future Ariel.
  2. Update the BIOS
  3. Load the settings from HpSetup.txt OR follow along the rest.
  4. Configure the following
    • Ensure legacy boot is enabled.
    • I disabled secure boot and MS certificate in case
    • Turn off fast boot (might be optional)
    • Add boot delay 5 seconds (purely QoL)
    • Ensure USB takes priority over local disk
    • I disabled prompt on memory change so if I add RAM later I don’t have to displace the system.
    • I disabled Intel’s sgx or whatnot. Don’t trust it after the RST debacle.
  5. Save and reboot
  6. Hit escape to select boot option of USB (esc maybe not required)
  7. Follow the instructions to install NixOS
    • 23.05 (but higher is fine)
    • User nixos
    • Same password for root
    • Auto login (QoL but consult your threat model)
  8. Use nix-shell to obtain Git and Helix
  9. Clone this flake repo from github
  10. Copy the machine-specific disk config from /etc/nixos/hardware-configuration.nix. Place it in the machine’s hardware-configuration.nix in the flake repo. (This step may no longer be necessary)
  11. Nix rebuild switch to the flake’s config.
  12. Confirm SSH remote access is working.
  13. Reboot and enter bios.
    • Turn fast boot back on
    • Set boot delay to 0
    • Disable UEFI boot priority. If we need to boot from USB we’ll reenter the BIOS.
  14. Save BIOS changes and one last confirmation that the system boots and is remotable.
  15. Move the machine to it’s final home.
  16. Remotely retrieve the hardware configuration and commit it to the flake repo.

Topton N100 (CW-AL-4L-V1.0 N100)

  1. Download BIOS update and place on Ventoy USB.
  2. Mash F10 to enter BIOS, boot update. Enter 1 and it should update.
  3. Reset and wait, it will beep and hang and reboot but eventually it should come good.
  4. Set USB boot precendence above internal drive/s
  5. Boot Proxmox installer and walk through Set static IP with netmask as same as router’s DHCP netmask. Best I can tell this is required to send traffic back to origin.
  6. Highly recommended but optionally, trust your SSH keys. curl https://github.com/arichtman.keys >> ~/.ssh/authorized_keys
  7. Optionally, add static DHCP lease to the router. If you do this, you can also optionally remove the fixed interface configuration. Edit /etc/network/interfaces and switch the virtual bridge network configuration from manual to dhcp.
  8. Optionally, install trusted certificates. Instructions are on my blog.
  9. Run some of the proxmox helper scripts At least the post install one to fix sources. I also ran the microcode update, CPU scaling governor, and kernel cleanup (since I had been operating for a while).
  10. Enable IOMMU. First, check GRUB/systemd efibootmgr -v. If GRUB, sed -i -r -e 's/(GRUB_CMDLINE_LINUX_DEFAULT=")(.*)"/\1\2 intel_iommu=on"/' /etc/default/grub
  11. echo 'vfio vfio_iommu_type1 vfio_pci vfio_virqfd' >> /etc/modules
  12. Reboot to check config
  13. Set BIOS settings:
    • Boot:
      • Disable beep
      • Enable fast boot
      • Enable network stack
    • Chipset:
      • PCH-IO:
        • Enable Wake on lan and BT
        • Enable TCO timer
  14. Install Prometheus node exporter, apt install prometheus-node-exporter.
  15. Install Avahi daemon to enable mDNS, apt install avahi-daemon.
  16. Install grub package so actual grub binaries get updates, apt install grub-efi-amd64.
  17. Optionally comment out the Cron job on reboot that sets it to power save.
  18. Disable IPMI service since we don’t have support, systemctl disable openipmi.
  19. Optionally install the PVE Prometheus exporter.

If I check /etc/grub.d/000_ proxmox whatever it says update-grub isn’t the way and to use proxmox-boot-tool refresh. It also looks like there’s a specific proxmox grub config file under /etc/default/grub.d/proxmox-ve.cfg. I don’t expect it hurts much to have iommu on as a machine default, and we’re not booting anything else… Might tidy up the sed config command though. Looking at the systemd we could probably do both without harm. That one’s also using the official proxmox command.

References:

Proxmox Disk Setup

We did run mkfs -t ext4 but it didn’t allow us to use the disk in the GUI. So using GUI we wiped disk and initialized with GPT.

For the USB rust bucket we found the device name with fdisk -l. Then we mkfs -t ext4 /dev/sdb, followed by a mount /dev/sdb /media/backup. Never mind, same dance with the GUI, followed by heading to Node > Disks > Directory and creating one.

Use blkid to pull details and populate a line in /etc/fstab for auto remount of backup disk. Ref

/etc/fstab:

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=C61A-7940 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
UUID=b35130d3-6351-4010-87dd-6f2dac34cfba /mnt/pve/Backup ext4 defaults,nofail,x-systemd.device-timeout=5 0 2

Re-IDing a Proxmox VM

I used this to shift OPNsense to 999 and any templates to >=1000.

  1. Stop VM
  2. Get storage group name lvs -a
  3. Rename disk lvrename prod vm-100-disk-0 vm-999-disk-0
  4. Enter /etc/pve/nodes/proxmox/qemu-server
  5. Edit conf file to use renamed disk.
  6. Move conf file to new id

Virtual node disk resize

nix-shell -p cloud-utils
growpart /dev/sda 1
resize2fs /dev/sda1

Opnsense

VM Setup
  1. Download iso and unpack
  2. Move iso to /var/lib/vz/template/iso
  3. Create VM with adjustments: I’m trying 2 cores now, utilization was low but we had spikes which I suspect were system stuff
    • Start at boot
    • SSD emulation, 48GiB
    • 1 socket+ 2 cores, NUMA enabled
    • 2048 MiB RAM
  4. Use proxmox under datacenter to configure a backup schedule. The following should keep a rolling 4-weekly history.
    • Sunday 0100
    • Notify only on fail
    • Keep weekly 4
  5. Boot machine and follow installer
  6. Add PCIe ethernet controllers
Base OS Setup
  1. Boot system and root login
  2. Assign WAN and LAN interfaces to ethernet controllers
  3. Check for updates either opkg update or maybe system > firmware
  4. Add static DHCP leases for any machines using static IPs so Upbound will serve records for them
  5. Install an intermediate cert and it’s corrosponding bundle under system > trust
  6. Switch to using the TLS certificate under System > Settings > Administration
  7. Set both interfaces to delete protected and IPv6 SLAAC
  8. Under System > General:
    • set hostname
    • set domanin
    • configure DNS servers
    • Disallow DNS override on WAN
  9. Reporting > netflow set capture on
  10. System > settings > cron
    • once daily to update the block lists
    • once weekly after the backup is taken (this ensures we can restore)
Tuning
DNS Configuration
  1. Configure Upbound DNS service
    • enable DNSSEC
    • enable DHCP lease registration
    • Disallow system nameservers in DoT and add records with blank domains+port 853
    • Enable blocklists
      • OIDS Ads
      • Steven Black
      • Hagezi Multi Pro++
    • Enable data capture
Firewall Configuration
  1. Firewall
    • Add aliases for static boxes, localhost
    • Create a NAT port-forward:
      • LAN interface
      • IPv4+6
      • TCP+UDP
      • Invert
      • Destination LAN net
      • from dns to dns
      • Redirect target Localhost:53
  2. Test
    • DNS redirection:
      • Unbound host override bing.com to something
      • Check this returns the override dig +trace @4.4.4.4 bing.com
    • Ad blocking https://d3ward.github.io/toolz/adblock.html
  3. Block bad IPs
    1. Add aliases for IP lists (see references)
    2. Add firewall rules: Either floating destination FireHOL level 3 + Spamhaus OR WAN outbound FireHOL Level 1 + WAN inbound all bad IPs
OpenVPN

Follow one of the 6000 tutorials AKA yes, I forgot to document it.

WireGuard

Follow tutorial AKA forgot to document it. See also wg0.conf in this repo.

Alacritty terminal

/usr/local/etc/rc.syshook.d/update/99-alacritty-terminal:

#!/bin/sh
# Configures terminal for Alacritty

curl -sSL https://raw.githubusercontent.com/alacritty/alacritty/master/extra/alacritty.info -o alacritty.info
infotocap alacritty.info >> /usr/share/misc/termcap
cap_mkdb /usr/share/misc/termcap
rm alacritty.info
Nginx for Kanidm

Nginx configuration on OPNsense requires modification for Kanidm Oauth to work!

  1. Inspect the generated configuration file, either via SSH or Web console
  2. Locate the server block for the Kanidm HTTP server
  3. There should be a line like include $UUID_post/*.conf, note the directory name.
  4. Create such a directory in /usr/local/etc/nginx, and add a .conf file with the following: proxy_pass_header X-KANIDM-OPID;
  5. Restart the nginx service
Plugins

Notes:

I will revisit the resources supplied after running the box for a bit.

CPU seems fine, spikey with what I think are Python runtime startups from the control layer. RAM looks consistently under about 1Gb so I’ll trim that back from the recommended minimum 2Gb. We’re doing pretty well on space too but I’m less short on that.

References:

Disaster Recovery / Restoring from Backup

  1. Access Proxmox directly
  2. Visit Backup storage and confirm thre is an intact and appropriate snapshot image to restore.
  3. Disable VM protection and delete it.
  4. Select backup image and restore to VM ID 999. Enable start on boot and disable unique feature.

Virtualized NixOS Node Bootstrap

  1. Use GUI installer to deploy, the CLI one’s a pain, 12GiB storage minimum, 4 cores/8GiB ram (min 4+ GiB). We’ll scale it down later, it bombs completely without plenty of RAM.
  2. Thing’s a pain to bootstrap and the web console is limited Sudo edit /etc/nixos/configuration.nix
    • Enable the openssh service
    • security.sudo.wheelNeedsPassword = false
    • Disable OSprober by removing the line
    • Set disk source to /dev/sda
    • I’m not sure those last 2 make much of a difference once the machine is under flake control
  3. Rebuild
  4. Bounce the machine so it releases using the new hostname
  5. Then pull down some keys to get in, it should have already DHCP’d over the bridge network. mkdir ~~/.ssh && curl https://github.com/arichtman.keys -o ~/.ssh/authorized_keys && chmod 600 ~~/.ssh/authorized_keys
  6. Upgrade the system sudo nixos-rebuild switch --upgrade --upgrade-all
  7. Reboot to test
  8. Clear history sudo nix-rebuild list-generations sudo rm /nix/var/nix/profiles/system-#-profile sudo nix-collect-garbage --delete-old
  9. Adjust hardware down to 1/2GiB.
  10. Make template

Subsoil

Trust chain setup

Arguably this mingles with substratum, as PKI/trust/TLS is required or very desirable for VPN/HTTPS etc. SPIFFE/SPIRE will address this somewhat.

  1. Create root CA xkcdpass --delimiter - --numwords 4 > root-ca.pass step certificate create "ariel@richtman.au" ./root-ca.pem ./root-ca-key.pem --profile root-ca --password-file ./root-ca.pass
  2. Distribute the intermediate certificates and keys
  3. Secure the root CA, it’s a bit hidden but Bitwarden does take attachments.
  4. Publish the root CA, with my current setup this meant uploading it to s3.
  5. Update the sha256 for the root certificate fetchUrl call

Garage setup

garage layout assign --zone garage.services.richtman.au --capacity 128GB $(garage node id 2>/dev/null)
garage layout apply --version 1

Kanidm setup

export KANIDM_URL=https://id.richtman.au
export GRAFANA_FQDN=grafana.services.richtman.au
kanidm system oauth2 create grafana $GRAFANA_FQDN https://$GRAFANA_FQDN
kanidm system oauth2 set-landing-url grafana "https://${GRAFANA_FQDN}/login/generic_oauth"

kanidm group create 'grafana_superadmins'
kanidm group create 'grafana_admins'
kanidm group create 'grafana_editors'
kanidm group create 'grafana_users'

kanidm system oauth2 update-scope-map grafana grafana_users email openid profile groups
kanidm system oauth2 enable-pkce grafana

kanidm system oauth2 update-claim-map-join 'grafana' 'grafana_role' array
kanidm system oauth2 update-claim-map 'grafana' 'grafana_role' 'grafana_superadmins' 'GrafanaAdmin'
kanidm system oauth2 update-claim-map 'grafana' 'grafana_role' 'grafana_admins' 'Admin'
kanidm system oauth2 update-claim-map 'grafana' 'grafana_role' 'grafana_editors' 'Editor'

# Note: I'm not sure I need *all* of these
kanidm group add-members grafana_superadmins arichtman
kanidm group add-members grafana_admins arichtman
kanidm group add-members grafana_editors arichtman
kanidm group add-members grafana_users arichtman

kanidm system oauth2 get grafana
kanidm system oauth2 show-basic-secret grafana

Grafana side: Official docs examples

Topsoil

Cluster access bootstrap
# Create a client certificate with admin
step certificate create cluster-admin cluster-admin.pem cluster-admin-key.pem \
  --ca ca.pem --ca-key ca-key.pem --insecure --no-password --template granular-dn-leaf.tpl --set-file dn-defaults.json --not-after 8760h \
  --set organization=system:masters
# Construct the kubeconfig file
# Here we're embedding certificates to avoid breaking stuff if we move or remove cert files
kubectl config set-cluster home --server https://fat-controller.systems.richtman.au:6443 --certificate-authority ca.pem --embed-certs=true
kubectl config set-credentials home-admin --client-certificate cluster-admin.pem --client-key cluster-admin-key.pem --embed-certs=true
kubectl config set-context --user home-admin --cluster home home-admin

Cluster access (normal)

  1. Create private key openssl genpkey -out klient-key.pem -algorithm ed25519
  2. Create CSR openssl req -new -config klient.csr.conf -key klient-key.pem -out klient.csr
  3. export KLIENT_CSR=$(base64 klient.csr | tr -d "\n")
  4. Submit the CSR to the cluster envsubst -i klient-csr.yaml | kubectly apply -f -
  5. Approve the request kubectl certificate user approve

Cluster node roles

For security reasons, it’s not possible for nodes to self-select roles. We can label our nodes using label.sh.

hmmm, deleting the nodes (reasonably) removes labels. …and since they can’t self-identify, we have to relabel every time. I expect taints would work the same way, so we couldn’t use a daemonset or spread topology with labeling privileges since it wouldn’t know what to label the node. Unless… we deploy it with a configMap? That’s kinda lame. I suppose all the nodes that need this are dynamic, ergo ephemeral and workers, so we could make something like that. Heck, a static pod would work fine for this and be simple as. But then it’d be a pod, which is a continuous workload, which we really don’t need. A job would suit better, but then it’s like, why even run this on the nodes themselves? Have the node self-delete (it’ll self-register again anyway), and have the admin box worry about admin like labelling. I wonder if there’s any better way security-wise to have nodes be trusted with certain labels. Already they need apiServer-trusted client certificates, it’d be cool if the metadata on those could determine labels.

There is a way to tell the Kubelet to register with labels but it’s limited to a specific group. I doubt the Kubelet has an option to open that up and since we’re getting denied even starting the binary it’s probably not settable on the APIserver.

Node CSRs piling up

kubectl get csr --no-headers -o jsonpath='{.items[*].metadata.name}' | xargs -r kubectl certificate approve

Notes

Checking builds manually: nix build .#nixosConfigurations.fat-controller.config.system.build.toplevel Minimal install ~3.2 gigs Lab-node with master node about 3.2 gb also, so will want more headroom.

Add to nomicon

Android phone setup

Using tasker

Profile: AutoPrivateDNS
         State: Wifi Connected [ SSID:sugar_monster_house MAC:* IP:* Active:Any ]
     Enter: Anon
         A1: Custom Setting [ Type:Global Name:private_dns_mode Value:off Use Root:Off Read Setting To: ]
     Exit: Anon
         A1: Custom Setting [ Type:Global Name:private_dns_mode Value:hostname Use Root:Off Read Setting To: ]

Secure setting accessibility_display_daltonizer_enabled to 0 or 1 for color toggle.

nix shell nixpkgs#android-tools -c adb shell pm grant net.dinglisch.android.taskerm android.permission.WRITE_SECURE_SETTINGS

References:

Desktops

Mac

Trust chain system install: sudo security add-trusted-cert -r trustRoot -k /Library/Keychains/System.keychain -d ~/Downloads/root-ca.pem

Old MBP setup

OPNsense/openssl’s ciphers are too new, to install client certificate you may need to pkcs12 bundle legacy. openssl pkcs12 -export -legacy -out Certificate.p12 -in certificate.pem -inkey key.pem

MBP M2 setup

  1. Update everything softwareupdate -ia
  2. Optionally install rosetta softwareupdate --install-rosetta --agree-to-license I didn’t explicitly install it but it’s on there somehow now. There was some mention that it auto-installs if you try running x86_64 binaries.
  3. Determinant systems install nix
  4. Until this is resolved https://github.com/LnL7/nix-darwin/issues/149 sudo mv /etc/nix/nix.conf /etc/nix/.nix-darwin.bkp.nix.conf
  5. Nix-Darwin build and run installer
nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer
./result/bin/darwin-installer

edit default configuration.nix? n
# Accept the option to manage nix-darwin using nix-channel or else it bombs
manage using channels? y
add to bashrc y
add to zshrc? y
create /run? y
# a nix-channel call will now fail

Bootstrapping:

  1. do the xcode-install method
  2. Build manually once nix build github:arichtman/nix#darwinConfigurations.macbook-pro-work.system
  3. Switch manually once ./result/sw/bin/darwin-rebuild switch --flake .#macbook-pro-work
  4. If bootstrapped, build according to flake ./result/sw/bin/darwin-rebuild switch --flake github:arichtman/nix

To do: look into Nix VMs on Mac

Universal Blue

some very wip notes about the desktop.

Desktop Todo

Nix References

This site is open source. Improve this page.