Repository files navigation

Okay, your system is completely broken, and you need to umount/ or something like that.What do you do?

One of our servers had a broken root filesystem (btrfs, don't judge me).Online recovery was not possible, so the filesystem needed to be unmounted which is not possible for the root fs.Additionally, as errors were detected, the kernel decided to mount it read only and didn't let me remount it asrw.IPMI? Yes, I had the password in my password store but not the username.So the only logical solution was to kexec into an emergency system.This code is what I used.It recovers all IP addresses as well as SSH host and user keys from the old system and kexecs into a new one - entirely in-memory.

Theemergency script (found in the repository root) will SSH over and execute the following things:

1. Build the recovery image (a.tar.xz with a small nix store and akexec script) from the files in this repository locally on the machine you're executing this code on
   1. The system configuration is found inconfiguration.nix
   2. Somekexec-related features are imported fromkexec.nix
   3. The scripts will be included to be used in thekexec script (see below)
2. Try tomkdir/nix and/tmp. If the don't already exist and your root fs is read-only, you have a problem this project can't fix
3. Mount a freshtmpfs on/tmp because there might not be one already
4. scp the emergency image over and extract it
5. Mount the nix store from the emergency image over/nix usingoverlayfs
6. Run the kexec script

Thekexec script (found inkexec.nix) will do the following:

1. Prepare a second initrd
2. Put your SSH host keys into the initrd
3. Put all of your SSH user keys into the initrd
4. Fetch all your IP addresses and routes and put them into the initrd
5. Pack the second initrd and append it to the default NixOS initrd from the emergency image
6. kexec into the kernel from the emergency image while using the new initrd
7. In case you didn't already notice:This will crash your currently running system, so maybe it's a good idea to gracefully shut down remaining daemons if that's still possible

The script that is packed into the initrd of the new system will do the following:

1. Place the SSH host key
2. Place the SSH user keys
3. Place a script for the IP addresses which will be executed usingnetworking.localCommands so the interfaces are available

If you set the environment variableEMERGENCY_DUMP_NETWORK to1, all IPs, routes, and nameservers will be placed in theemergency_ips,emergency_routes, andemergency_nameservers files, respectively.

$ ./emergency root@somehost# or$ ./emergency somebody@somehost

If it doesn't work for you, I'm sorry.I can probably not help you, but if you're able to fix something, feel free to create a PR.

The code is based onclever's kexec nix-test (foundhere).

The code is licensed under theLGPL3.