SmartConfig

M4 plan: the rescue path

Status: approved (2026-10-04, “approved”: every recommendation of section 10). Drafted in the session from the boot research in a throwaway QEMU VM (Appendix A).

0. Summary

1. Goals and non-goals

Goals. M4 is done when all of these hold, plus the sign-off in section 7.

  1. A broken file can be undone from the console without a live USB. One menu entry and one command: sc restore ID.
  2. The rescue path changes nothing by itself. Root stays read-only until the owner remounts it, and sc status only reads.
  3. The menu shows after a failed boot. That covers a boot that never reached multi-user and one that reached it with local-fs.target failed.
  4. sc reads the store on a read-only root, hot journal included, without writing to it.
  5. Boot verdicts are right. “ok” only when multi-user was reached with local-fs.target active, no emergency or rescue mode, and no failed units; “bad” otherwise.
  6. Normal boots do not get noticeably slower. Two oneshot units: one early, one after multi-user, each measured in the lab and on this VM.
  7. All checks pass. gofmt, vet, tests as a user and as root, race, the static build, make m1-compat, and make lab-e2e in BIOS and UEFI mode.

Non-goals (not built, not even as stubs): packaging (M5; M4 installs by hand like scd); LUKS, LVM and btrfs roots; an entry for every kernel; managing a GRUB password; network in rescue; a TUI; the deferred items in section 9.

2. What the lab showed

Question Answer (details in Appendix A)
A bad fstab line, normal boot After a 90 s wait, emergency mode gives a root shell with no password. The outcome varies: stuck in emergency mode, continued to multi-user, or the getty hung up the shell. sc records nothing (scd is not running).
ro fstab=no systemd.unit=rescue.target SYSTEMD_SULOGIN_FORCE=1 Rescue shell about 20 s after systemd starts. Root is read-only, /boot and /boot/efi are not mounted, journald and udevd run. Recommended.
Without fstab=no rescue.target: a 90 s wait, then emergency mode with root read-write. emergency.target: grub-initrd-fallback.service pulls in sysinit, root goes read-write, and the bad mount waits in the background.
The static sc on a read-only root log, diff and cat work. check needs a writable TMPDIR (/run works). restore gives a clean “read-only file system” error until mount -o remount,rw /, then works.
A hot journal on a read-only root Every read fails with attempt to write a readonly database (776). A copy of the database and its journal in /run reads fine; the real store is untouched.
GRUB entry /etc/grub.d/42_smartconfig Generated by update-grub. Works through grub-reboot, the menu (UEFI serial, BIOS screen) and Secure Boot (shim plus signed kernel).
recordfail The 30 s menu returns only when the failed boot never reached multi-user. grub-common.service clears it otherwise.
A report before the shell A drop-in on rescue.service and emergency.service with ExecStartPre=-sc … prints right above the “Press Enter” line. A separate unit’s output scrolls away.
A boot-ok marker boot-complete.target is unused. A unit after multi-user.target that checks local-fs.target tells ok from bad. A boot that never got there leaves only the early “seen” line.
Speed under TCG About 75 s to a login prompt and about 47 s to the rescue shell. Some boots stall for minutes, and two panicked (“IO-APIC + timer”; no_timer_check fixes it).

3. Architecture

GRUB ─ "Ubuntu" (default)
     └ "SmartConfig rescue": ro fstab=no systemd.unit=rescue.target SYSTEMD_SULOGIN_FORCE=1
         └ rescue.service + drop-in: ExecStartPre=-/usr/local/sbin/sc status --console
             └ shell: mount -o remount,rw / ; sc restore ID ; sync ; systemctl reboot

every boot: sc-boot-seen.service (early, after remount-fs)  → boots: "<id> seen", grubenv smartconfig_pending=1
            sc-boot-ok.service (after multi-user.target)   → boots: "<id> ok|bad <rowid> <why>"; ok clears the flag
next boot:  42_smartconfig in grub.cfg: smartconfig_pending=1 → show the menu for 30 s

3.1 sc status

Read-only. It prints, one fact per line, in plain text for an 80×25 console:

--console is the rescue form. It caps itself at 10 s and 20 lines (older changes are counted), and runs the checks with sc’s own rules only when the validators are not there or the root is read-only.

3.2 Boot verdicts ($SC_HOME/boots)

3.3 The menu after a failed boot

3.4 The rescue entry: scripts/42_smartconfig

3.5 sc on a read-only root

3.6 Before a restore: immutable files

sc restore reads the target’s and its directory’s inode flags (FS_IOC_GETFLAGS). It refuses an immutable (+i) or append-only (+a) target before writing anything, and names the chattr to run. This is the M2 plan’s deferred “+i/+a precheck”.

4. CLI

Command What
sc status Section 3.1. Exit 0 healthy, 2 when the last boot failed or a change since the last healthy boot has a blocker or error, 1 when sc itself failed.
sc status --console The rescue form (section 3.1).
sc boot seen, sc boot verdict Hidden; run only by the two units.
sc restore Also refuses +i/+a targets (section 3.6).

5. Security

6. Order of implementation

Step Chunk Commit Tests
1 A docs, check: Ubuntu's emergency mode gives a root shell (fstab explanation, PROJECT_LOG recovery guide, JOURNEY, the film’s console text) rule text test
2 A store: read-only open, hot journal copy a read-only store directory as a user; a hot journal made by a killed writer; no write to the store (checksums)
3 A check: scratch fallback when the store is read-only read-only $SC_HOME and TMPDIR
4 B boot: seen and verdict lines, sc boot (with the runner calls faked in tests) ok, bad (local-fs inactive), never reached multi-user, a rotated file
5 B scripts: sc-boot-seen and sc-boot-ok units systemd-analyze verify; ordering read back
6 B sc status (normal and --console) fabricated boots and rows; a read-only store; the 10 s and 20-line caps
7 C scripts: 42_smartconfig (entry and menu flag) run against fabricated /boot trees; grub-script-check on its output
8 C boot: grubenv flag (set at seen, cleared on ok, skipped where unsupported) faked grub-editenv
9 C scripts: rescue and emergency drop-ins systemd-analyze verify
10 C restore: refuse +i and +a targets as root on a scratch ext4 loop image, or skipped
11 D lab: QEMU rescue lab and make lab-e2e (from the research scripts; the image is cached outside the repo) the owner scenario end to end, BIOS and UEFI
12 E scripts: accept-m4.sh (the non-boot parts on this VM, as root)  
13 E docs: README “When the machine does not boot”, PROJECT_LOG, MILESTONES  

The chunk reviews are after steps 3, 6, 10, 11 and 13; then the sign-off.

7. Acceptance and sign-off

Owner scenario (the one for M4). You add a line for a disk that is not there to /etc/fstab with nano, ignore scd’s warning, and reboot.

make lab-e2e runs exactly that in QEMU, BIOS and UEFI, and passes before anything is installed on this VM.

Sign-off runs, each with your OK:

8. Risks

9. Deferred

10. Questions for you

  1. Scope as in sections 1 and 9. Recommendation: yes.
  2. Show the menu after a failed boot through sc’s own grubenv flag (two writes per boot, like recordfail, skipped where unsupported). Recommendation: yes. Ubuntu’s recordfail misses the fstab failure that reaches multi-user.
  3. No GRUB password by default. The README explains how to add one. Recommendation: yes. Console access already gives root on Ubuntu 24.04.
  4. Install by hand (README), like scd, until the M5 package. Recommendation: yes.
  5. Keep the QEMU lab in the repo (lab/, make lab-e2e, not in CI), with the image cached under ~/.cache/smartconfig-lab. Recommendation: yes. It is the only safe way to test boot breakage.
  6. Boot verdicts in a file, not a store column (section 3.2), so the rescue binaries keep reading the store. This replaces NEXT_STEPS question 3’s boot_id column. Recommendation: yes.
  7. Correct the “no shell” claims first (step 1). Recommendation: yes.

11. Estimate

About 4 working days: chunk A 1 day; B 1 day; C 1 day; D (the lab) half a day, and boots under TCG are slow; E and the sign-off half a day, plus your time for S2 and S3, which need reboots of this VM.

12. Changes after approval

# Date Change Why Commit
C1 2026-10-04 A store at an older schema on a read-only root is not read through its old columns (section 3.5): Open says to remount read-write so sc can upgrade it. Every store sc has written since M2 is at the current schema, and M4 does not change it; reading old columns would be code for a case that cannot occur here. step 2
C2 2026-10-04 With no writable scratch directory at all, sc check fails with one line naming every place it tried; it does not run sc’s own rules alone (section 3.5). netplan and the account checkers write companion files before their validator runs, so a check without a scratch directory would need each checker rewired. On Ubuntu /run (root), the runtime directory (a user) and /dev/shm are tmpfs and writable in the rescue shell. step 3
C3 2026-10-04 The boot verdict does not count failed units: “ok” needs local-fs.target active and neither emergency nor rescue mode (goal 5 and section 3.2 said “no failed units”); the count is recorded in the line. This healthy VM has two failed units (whoopsie.path and .service), and a “bad” verdict keeps the boot menu showing: every boot would show it. The lab’s failure is caught by local-fs.target alone. step 4
C4 2026-10-04 From the chunk B review. sc status --console runs sc’s own rules only, with no time cap; it lists worst first, at most 6 files, and fits 80x25 (section 3.1 said 10 s and 20 lines, validators where present). The last healthy boot includes this one. A verdict with an unreadable store records row -1. sc-boot-seen no longer orders itself before sysinit.target, and has the default timeout. reviews/2026-10-04-m4-chunk-b.md: a time cap could drop a blocker; 20 changes did not fit the screen; the healthy boot after a fix reported failure; the lab killed sc-boot-seen at 10 s. 40288d8
C5 2026-10-04 From the chunk C review. The grubenv flag is kept wherever grubenv is GRUB’s 1024-byte block (section 3.3 named 00_header’s filesystem check): Linux writes it, GRUB only reads it. A /boot that fails to mount leaves the flag unset, and the menu then depends on recordfail. GRUB_DISABLE_RECOVERY=true leaves the rescue entry out, and the menu flag stays. The rescue boot cannot clear the flag, so the first boot after the fix shows the menu once more (section 7); sc status says so. The rescue report goes to every console. sc-boot-seen runs before Ubuntu’s other writers of grubenv, which wait for it. A multipath root’s root= is deferred, with LVM and LUKS. reviews/2026-10-04-m4-chunk-c.md: lost grubenv writes, a screen without the report, an admin’s choice ignored. 683678a
C6 2026-10-05 From the chunk D review. sc status --console stops itself after 60 s and says so on every console, with the command to run from the shell (C4 said no time cap): systemd ends rescue.service and emergency.service, shell and all, when their start, ExecStartPre included, passes 90 s, and the - before the drop-in’s line forgives an exit status, not a timeout. The report goes to each console while that console takes bytes or its output queue drains, given up after 5 s of stalled slices and 10 s per console at most, where a host pause counts as one slice (chunk C gave each console one 2 s write). The emergency report has one more line: at a login: prompt instead of #, log in and put sudo before each command. The hangup is ignored from the start of main, not from the command. reviews/2026-10-05-m4-chunk-d.md: a transient unit with a slow ExecStartPre got no shell; in the lab runs a slow ttyS0 cut the report and a 10 s host pause lost it. 3115b71, 1d25037, c70d886
C7 2026-10-06 From the chunk E review. The rescue report promises the menu once more only while its flag is set: after a failed boot, or in the emergency shell of the failing boot itself (it said so in every rescue boot, also one picked by hand after a healthy boot). The console lists at most 5 files (C4 said 6; changed with chunk C’s fixes, 683678a, and not recorded then). sc status does not say mount /boot (section 8): the README says to mount a separate /boot, /var or /usr/local in the rescue shell. The undo is for the newest blocker, else the newest error, as section 3.1 says (the code took the newest of either). reviews/2026-10-06-m4-chunk-e.md: an older fstab blocker lost its undo to a newer error, and the menu was promised where none comes. 654ee7f
C8 2026-10-07 From the final review (sign-off S4). The two boot units also need a command line without fstab=no: the rescue boot gets no line, even continued to the desktop. A boot with a verdict is not seen again; seen sets the flag before its line; a bad verdict sets it too. The rescue entry keeps every token of GRUB_CMDLINE_LINUX_DEFAULT but quiet splash (3.4) and masks grub-initrd-fallback.service. sc status: a path is new only with proof of absence (“did not exist”); one first seen after the line is compared with its first version, and with no earlier version the report says to fix it by hand; without a healthy boot the comparison is with the version before the rows listed; deleting a flag file or ld.so.preload is no error; with no store in a rescue or emergency boot it says to mount /var; the undo’s paths are shell words. The console: the command’s output gets the report only when no console opens, one deadline, 80 s from the start, ends every write, and the stop removes the /run copies. As built and not recorded before: sc status finds scd in /proc, not by its lock; the undo has systemctl daemon-reload in every rescue or emergency boot; a verdict line holds its time; a boot that cannot write /var gets no line. reviews/2026-10-07-m4-final.md: a stopped sole console held the report, and the shell, past 90 s; an fstab first seen after the line was called new and moved aside by the undo; a rescue boot continued with “exit” was “ok” and hid the next failure’s undo; a nomodeset was dropped. 69ea585, 4a8f7b4, 8c6f30e, 580edca, 03ec9ca, 734b670, d9c8817

Appendix A. Evidence (QEMU lab, 2026-10-04)

Setup:

A1. Boot times (guest-side):

Boot Time
First boot to login 108.6 s
Later boots to login about 74–78 s
Rescue entry, firmware to prompt about 47 s (systemd starts at uptime 21 s)
Emergency mode with fstab=no 29.8 s

Some boots stalled 1–6 minutes; two panicked (“IO-APIC + timer doesn’t work”), and no_timer_check avoids it.

A2. Bad fstab line (UUID=3f6c1e2a-… /mnt/backup ext4 defaults 0 2; sc check flags it as a blocker):

A3. Kernel argument combinations (with the bad line):

Arguments Result
A: ro systemd.unit=rescue.target SYSTEMD_SULOGIN_FORCE=1 90 s, then emergency mode; root read-write; /boot mounted
B: A plus fstab=no rescue in about 20 s; root read-only; /boot and /boot/efi not mounted
C: systemd.unit=emergency.target grub-initrd-fallback.service (Requires=sysinit.target) remounts root read-write and starts the 90 s wait in the background
D: C plus fstab=no shell in about 8 s; root read-only
E: stock “recovery mode” the 90 s wait, then emergency; the recovery menu never showed

A4. sc on a read-only root:

A5. Hot journal:

A6. GRUB:

A7. A report before the shell:

A8. A boot-ok marker: