Date: 2026-10-05, closed 2026-10-06. Two reviewers (AI agents) worked in their own
clones on c7bc566..da706fd. The chunk is:
lab/, make lab-e2e);55d2bde (sc status --console outlives the getty’s hang-up).Reviewer A took the scenario: lab/e2e.py, its checks and verdicts, lab/test_e2e.py
and the README. Reviewer B took the machinery and the Go fix: lab/vm.py,
lab/serialmux.py, lab/console.py, the guest scripts, the Makefile’s lab targets,
55d2bde and its test. Neither started a QEMU: they replayed real console text from
the lab’s runs through the lab’s own functions, and drove the machinery with a
stand-in QEMU (a script serving the serial and QMP sockets).
They found 1 high (narrow), 10 medium, 13 low and 5 cleanups. None of it changes the
PASS of the step 11 check on 56f5359: both runs had no retry, and every [M4] row
passed on its first attempt. What the findings say is that the lab, as a standing
proof, had holes: one check that passed a known product fault, verdict classes that
leaked both ways, retry paths and check conditions with no test.
| Area | Finding | Severity | Fixed in |
|---|---|---|---|
| checks (A, F1) | The recordfail rule from 152ecbe applied to every attempt, so a stale recordfail=1 after a healthy boot (the lost unset chunk C measured) passed 1.1 and 5.1. |
high (narrow) | 9d3df9e: the rule holds only in an attempt after a reset in the kernel; 1.6 and 4.4 hold grubenv to no recordfail=1 |
| verdicts (A, F2) | bios: a menu that never showed was an [M4] FAIL only with style=[hidden]; after outcome a the style is empty, so a missing flag was retried as menu-missed and ended INCONCLUSIVE. |
medium | 9d3df9e: observers still on the screen with timeout=[0] mean no menu was drawn; with the flag set that fails x.1 |
| verdicts (A, F3) | Four product faults in the rescue boot (a menu with no countdown, an entry that does not boot, a report that never ends, no shell after Enter) were LabError: INCONCLUSIVE. |
medium | 9d3df9e: with userspace up and no gap in the lab’s running they fail 3.2, 3.5, 3.7 or x.1 as [M4] |
| product (A, F3 note) | systemd gives rescue.service and emergency.service 90 s to start, ExecStartPre included; the - forgives an exit status, not a timeout. Since C4 sc status --console had no cap, so a slow report would have left no shell at all. Tried with a transient unit: Result=timeout, ExecStart never ran. |
note (the reviewer’s side note; not in the counts) | 3115b71: it stops itself after 60 s, says so on every console, exits 1; plan change C6 |
| verdicts (A, F4) | An ssh call lost (exit 255) or timed out inside 1.3, 1.9, E.2-E.5, 4.2 or 4.6 was recorded as an [M4] FAIL. | medium | 9d3df9e: SshLost is a lab error everywhere; a poll gets back the time the host stood still |
| checks (A, F5) | 2.5 failed on where the getty’s banner landed, or when it read before the shell’s first line. | medium | 9d3df9e: the block ends at whatever follows it (shell line, banner, login prompt), and the check waits for that |
| checks (A, F6) | 2.5 was a W in outcome c or after a hang-up, so a missing emergency report passed a run with goal3=multi-user. The W dated from before 55d2bde. |
medium | 9d3df9e: F in every outcome |
| tests (A, T1) | 115 of 158 mutations of check conditions and glue survived make lab-test. |
medium | fcb1b6d, see below |
| verdicts (A, F7) | A flake in boot 5 and another in 5b left 5.1 SKIP and the run PASS. | low | 9d3df9e: 5.1 is recorded before anything can be retried; never checked is INCONCLUSIVE |
| verdicts (A, F8) | A panic after userspace began kept the old flag, giving false [M4] FAILs on the retry. | low | 9d3df9e: after /init the flag is unknown; only TCG’s panic is retried |
| machinery (A, F9; B, B7) | A boot’s mark was taken in the QMP thread before the mux had read the old boot’s last bytes; reboot: Restarting system could land in the new boot and read as “shutdown”. 0 of 95 real marks showed it. |
low | 9d3df9e: the mark waits for the mux to drain; the boot’s end is read from its kernel’s first line on. The 1 s drain then lost a SHUTDOWN during a serial reconnect: 0ccba27 |
| output (A, F10) | --boot2 reset-at-timeout and dirty=yes were not in make’s last line. |
low | 9d3df9e: boot2=a(forced), dirty=yes |
| checks (A, F11) | 0.5 did not hold the rescue entry to keeping GRUB_CMDLINE_LINUX and console=. |
low | 9d3df9e: every token of the default entry but ro quiet splash |
| checks (A, F12) | 2.4 was registered [M4] H with no failing path. | cleanup | 9d3df9e: a boot 2 with no end in its budget fails it |
| output (A, F13) | Expected WARNs (3.1/4.1 on bios, 2.5 in outcome b) taught the reader to skip WARNs. | cleanup | 9d3df9e: notes. Both runs on 22383a0 have no WARN row |
| docs (A, F14) | lab/README.md said more than the code did (the recordfail rule, “nothing else is retried”, what FAIL and INCONCLUSIVE mean). |
cleanup | 9d3df9e; the one sentence left on 2.5 (and the registry text in lab/e2e.py) in 4ce59a3 |
| product (A, item 6) | In 5 of the lab’s 11 failed boots the owner had a login prompt beside the emergency shell or instead of it (Ubuntu’s race, plan A2). The report said nothing about it. | question | 3115b71: one line in the emergency report, “at a login: prompt instead of #: log in, then put sudo before each” |
| teardown (B, B1) | A second signal cut the teardown, and make lab-e2e delivered every signal twice (timeout --foreground). About half of all interrupts lost T.1, result.txt and the ledger. |
medium | 9d3df9e: the first signal starts the teardown, the rest are ignored; no dump from the guest after an interrupt |
| teardown (B, B2) | After the mode’s deadline failure_dump could wait 360 s with 120 s left before make’s SIGTERM. |
medium | 9d3df9e: the dump keeps within the mode’s time |
| cache (B, B3) | vm.py gc and every other command chmodded and cleaned whatever SC_LAB_CACHE named, no marker, no name check. |
medium | 9d3df9e: the cache must be the lab’s (a marker, or only its names); gc removes only names the lab makes |
| tests (B, B4) | TestConsoleStatusOutlivesHangup sent SIGHUP after the report was written, so the fix moved behind the write passed it. |
medium | 3115b71: the signal comes while the command runs (a hook in the sctest build); that mutant fails |
| product (B, B5) | A SIGHUP in the first 3 to 5 ms of sc status --console still ended sc, silently: the flag was set when cobra reached the command. |
low | 3115b71: main sets it before the handlers are installed. What comes before signal.Notify is the kernel’s default and stays open |
| tests (B, B6) | TestLabGuestFacts read the host’s real process list and failed on a command line with p_ in it. |
low | 3115b71 |
| machinery (B, B8) | install_signal_handlers replaced an inherited SIG_IGN, so nohup did not protect a run. |
low | 9d3df9e: an inherited ignore stays; the README says setsid nohup |
| machinery (B, B9) | “One VM per user” held only within one cache. | low | 9d3df9e: a second lock in XDG_RUNTIME_DIR |
| output (B, B10) | An interpreter error before main exited 1 and make called it an [M4] FAIL. |
low | 9d3df9e: make’s FAIL needs a FAIL summary line |
| verdicts (B, B11) | Every kernel panic was retried as the lab’s; a panic sc’s rescue entry caused each time ended INCONCLUSIVE. | low | 9d3df9e: only IO-APIC + timer is retried; another in the rescue boot fails its check |
| machinery (B, B12) | vm.py image: a signal left curl running, and a stalled download held the lock for good. |
low | 9d3df9e: curl gives up on a stalled download; the signal ends it |
| preflight (B, B13) | OVMF was pinned by path only; ovmf_code_sha256 was written and never compared. |
cleanup | 9d3df9e: P.5 compares it |
| machinery (B, B14) | Three small ones: the gap note used the monotonic clock only, so a suspended host left no note; facts.sh parts report the last pipe command’s status; skipUnless(has(...)) turned a deleted function into a skip. |
cleanup | 9d3df9e for the first and the third. The facts.sh pipe statuses are not fixed (see open items) |
| tests (B) | Mutation survivors in vm.py, serialmux.py, console.py and the guest scripts: 68 of 123 Python and 13 of 23 shell. |
cleanup | a371738, see below |
make lab-e2eThe fixes were run three times end to end. Each run found something.
| Tree | Result | What it found |
|---|---|---|
85e9f59 (3115b71, 9d3df9e) |
FAIL, both modes, 0 retries | UEFI 3.6: the rescue report on ttyS0 ended in the middle of its first change row. sc gave each console one 2 s write; a serial line whose queue is full of boot messages takes no byte for about 4 s at 9600 baud. tty1 got the report whole. Fixed in 1d25037: a console keeps its turn while it takes bytes or its queue drains (TIOCOUTQ), given up after 5 s with neither or 10 s in all. BIOS 3.4: the rescue kernel’s first Command line: lost one byte on serial (oot=UUID=) right after a 379 s host pause; its second print was whole. Fixed in 64911f4: the lab takes the second line where the first differs and the second is what grub.cfg says, with a note. |
fc47ea3 (the tests in) |
UEFI INCONCLUSIVE, BIOS FAIL; every other row passed | 5.2 in both modes, a lab race: the guest powered off and QEMU sent SHUTDOWN and exited, but mux.mark (the QMP listener) drained for 1 s while serial reconnected, and wait_event’s abort returned None before the event was in the list. Fixed in 0ccba27: an abort waits for the reader’s EOF and looks once more. BIOS 2.5, sc’s: no line of the emergency report on ttyS0. Boot 2’s journal (the kept disk booted once through an overlay) shows a 10.3 s host pause as emergency.service started; writeConsole measured its stall in wall-clock time, so the slice over the pause counted as a 10 s stall and ttyS0 was given up with nothing written. Fixed in c70d886: a slice counts at most 500 ms, so one pause is one slice. |
22383a0 (all fixes) |
UEFI PASS, 24m29s, 0 retries, boot2=b goal3=multi-user, boot5=yes, dirty=no. BIOS INCONCLUSIVE at 5.2 only, every [M4] row passed. |
Before the poweroff the lab’s cat boot_id over ssh took 385 s and timed out at 180 s, across a 384 s host pause (mux.log +1306.50). A lab error, correctly not [M4]. |
22383a0, bios only |
BIOS PASS, 45m14s, 2 lab retries, boot2=b goal3=multi-user, boot5=yes, dirty=no. |
Boot 3: one menu-missed retry (no observer line on the VGA screen). Boot 5: the stall retry fired for the first time: the kernel sat at smpboot for 900 s. evidence/stall-5-1.txt says why: mux gaps of 250, 364, 400, 400 and 265 s, and QEMU’s 5 s CPU sample spans 265 s of CPU, so this VM barely ran from about +1140 to +2570 s, 24 minutes. The host, not QEMU or the guest. |
So the PASS on 22383a0 is UEFI from one run and BIOS from a rerun of the same
commit, same sc (a0502d6461db), both with boot 5 and a clean tree. Evidence
(result, log, ledger, mux and marks log per mode, and the stall file) is kept in
~/smartconfig-work/signoff/lab-e2e-22383a0/, with the two make logs beside it.
The UEFI run had host pauses too, 167 s and 318 s, and passed with 0 retries: the fixes for F4 and the console writer held through them.
lab/e2e.py, reviewer A’s method: one condition removed or loosened at a
time): 158 mutations against da706fd, 115 survived. On the fixed code that is 261
mutations. fcb1b6d adds a test for each: execute() end to end with the VM and the
flow stubbed, one fault at a time in every check from 0.5 to 5.1 on the text and facts
of the passing run of 56f5359, and the glue. 261 of 261 killed on 85e9f59; after
64911f4 one mutation (“check_cmdline ignored”) no longer applies, 260 of 260.lab/vm.py, console.py, serialmux.py, the guest scripts, reviewer
B’s harness): 123 Python and 23 shell mutations, of which 68 and 13 had survived.
a371738 kills 123 of 123 and 23 of 23, with stand-ins and never a QEMU, a download
or the real cache. One shell mutant that still passed added a bare : and changed
nothing; it was replaced.check_decision lost
its bios notes (the menu’s notes rebound the same name), so “observers: no ‘sclab:
pre’ line” never reached the row. Fixed in 64911f4.make lab-test: 174 tests at da706fd, 326 now.mux.log, evidence/stall-*.txt). They are the user’s to
look at on the host (sleep or power saving first).fc47ea3 are
in the journal but not on ttyS0. Not understood; not needed for the fix.mark() during a serial reconnect. 0ccba27 makes the
SHUTDOWN safe; the drain itself still waits the whole second while the socket is gone.test_the_first_signal_is_the_only_one (from 9d3df9e) hung its
child past the 60 s limit in 3 of 20 runs with four CPU burners beside it, and once
ended “SystemExit 143” with no teardown; the same script run directly under the same
load had 0 hangs in 120. test_the_loop_measures_its_wakes (from 152ecbe) failed
once right after a go test loop; it expects the late wake to measure exactly 0.2 s.
make lab-test alone passes, three runs in a row.go test ./cmd/sc/ of 14 failed right after three lab-test runs; its output was
not kept, and 13 later runs passed.facts.sh’s p_blame, p_targets, p_dmesg, p_procs and p_vcs report the
last pipe command’s status, and p_consoles that of printf, so a failing
systemd-analyze or systemctl gives an empty block with rc=0. None of these
blocks decides a check; they are evidence and dump output.signal.Notify runs, in the first milliseconds of
sc status --console, still ends sc by the kernel’s default. It can only be closed
outside sc.seen line, the flag and the
counts stay right, check_30 restarts at the attempt’s own mark.Run /init. Reviewer A found
no way for sc to produce either.writeConsoles fans out only when stdout is
/dev/console, and in facts.sh it is a pipe.sha256sum), the boots file and Go’s goldens,
so 2.5 and 3.6 compare the real console with what the unit tests render.PR_SET_PDEATHSIG kills it when the lab dies, SIGKILL
included (8 of 8). The command line: TCG, no display, hostfwd on 127.0.0.1 only,
sockets in the 0700 run directory. The cache 0700, the key 0600, the reference files
0444 and renamed into place after qemu-img check.55d2bde is right and narrow: sc watch is handled first, no other
subcommand sets the flag, children are not affected (the signal is caught, not
ignored, and exec resets it).make lab-test 174 -> 326 (9d3df9e, fcb1b6d, a371738, 0ccba27). Each
lab fix from a failed run has a test that failed on the old code (64911f4’s second
command line, 0ccba27’s SHUTDOWN under a slow listener).TestConsoleStatusOutlivesHangup sends SIGHUP during the command; the 60 s limit
(a report that hangs gives way, one in time is written once, plain sc status has no
limit); writeConsole with a fake console that is slow, drains, stops, trickles or
fails, and a real pty; TestWriteConsoleHostPause (0 of 1260 bytes on the old
writer). All pass as a user, as root and under race.