SmartConfig

The SmartConfig journey

How the project started, which path we chose at each turn and why, and where we have moved away from the plan. For the day-to-day log see WORKLOG.md; for decisions and recovery, PROJECT_LOG.md. Every fact here comes from those files, the reviews and git history.

Last updated: 2026-10-09, M5 done. Sections 1 to 7 tell M1 and M2 in full, as written on 2026-10-01 (section 7 as it stood then); section 8 tells M3 to M5 in brief, and section 9 says where we are now.

1. The idea

A safety net for the Linux config files that take a machine down (fstab, GRUB, sshd, sudoers, PAM). Snapshot a file before it is edited, warn before a bad edit is applied, explain what broke, and restore one file from a rescue shell. Target: Ubuntu 24.04, on a VirtualBox VM used as the lab.

It is built in seven milestones, strictly one at a time and never ahead of the current one, not even as stubs (MILESTONES.md): M1 store and CLI, M2 watcher, M3 checkers, M4 rescue path, M5 .deb package, M6 incident set, M7 local model.

Ground rules chosen at the start (CLAUDE.md), and why:

Choice	Why
Go, one static binary (CGO_ENABLED=0)	It must run from a broken system with nothing else working.
SQLite through modernc.org/sqlite (pure Go)	Keeps the binary static; no C library needed.
Write a config file back only by temp file, fsync, rename	A crash can never leave a half-written fstab.
Tests never touch /etc or the real store	The lab machine must stay bootable.
Small commits, WORKLOG updated and pushed after every step	Anyone, human or Claude, can pick the work up cold.
Stop and ask before touching the real system, adding a dependency or changing an approved plan	The owner decides anything that is hard to undo.

2. M1: the store and the sc command (2026-09-27 to 28)

• Built in one sitting as seven commits: file helpers, store, CLI, Makefile, smoke test, milestones. sc init, snapshot, log, cat, diff, restore.
• First decision taken to the owner: the spec’s snapshot id collides when the same file is saved twice in one second, which the acceptance run itself does. We stopped and asked, then chose “hash again with a counter” (PROJECT_LOG D1).
• Proved on the real VM: a one-character UUID typo in /etc/fstab, found with sc diff and undone with sc restore, byte for byte. Research with systemd’s own fstab generator showed which typos really stop a boot: a data disk line does (90 s wait, then emergency mode); the root line most likely does not. We first thought emergency mode gave no shell, as root is locked; the M4 lab showed that Ubuntu’s sulogin opens one at the console anyway.
• Published: GitHub repo, a docs site, a system map and narrated films.

Hardening: four review rounds, five rounds of fixes. Instead of calling M1 done after its tests passed, it was attacked on purpose. Each round, AI reviewers ran the code and tried to break it, and two skeptics had to confirm each finding before it counted.

Round	Findings	What they were about
1	14	Concurrency and security of the first two follow-up fixes
2	15	Transactions, restore atomicity
3 (gate)	18	Restore logic, signals, spec conformance
4	7	Ctrl-C handling; led to a second design of signal handling

Round 2 found one way to lose data: a restore that failed or was killed right after replacing the file lost its undo point (finding U1, a regression of a round-2 fix); round 3 fixed it. The gate and signal reviews found no data loss. The fixes made sc safe to run next to the future watcher: every write is one locked transaction, a restore records an undo point before it touches the file, and Ctrl-C always tells the truth about whether the file changed. M1 was tagged m1 on 2026-09-28, and backups of the binary, the store, /etc and /boot/grub were made.

3. Planning M2: the watcher (2026-09-27 to 28)

• Research first, no code. M2_WATCHLIST.md studied what on this VM changes, what must be watched and what is noise. It found that /etc holds 791 symlinks (systemd enable/mask), so symlinks had to be in scope.
• Six decisions by the owner (2026-09-27): all of /etc minus an exclude list; symlinks supported; deletions and creations recorded; SSH host keys fingerprint-only; authorized_keys and /boot/grub outside /etc; the two M1 fixes first.
• Roadmap with 13 questions (NEXT_STEPS.md), answered “yes to every recommendation” on 2026-09-28: CI, git hooks against secrets, a supported Go, make race, tiers only as log priority in M2, boot_id deferred to M4, reboot and 24-hour soak as part of acceptance.
• The plan itself was written three times independently (lean, traps, admin), judged, the lean draft chosen and the best ideas of the other two merged in, then checked by a critic (one blocker, six major issues, all applied). Approved 2026-09-28 with all recommendations: 17 steps in five reviewed chunks (M2_PLAN.md).

4. Building M2

Overnight run, 2026-09-28 to 29 (unattended). Setup first: local hooks that refuse secrets, make race, Go 1.26.8 (the old 1.22.2 had three known vulnerabilities), CLAUDE.md edits. A very slow disk that night exposed a real M1 race (two lines after a double Ctrl-C) and two timing-dependent tests; all three were fixed. Then:

• Step 1: history ordered by insert order, not the clock, which can step back.
• Step 2: the database created readable by root only (0600).
• Step 3 was left half done: the run stopped with the code uncommitted and no note why.

Resumed interactively, 2026-09-30.

• Step 3 (the schema upgrade that adds row kinds) was checked, finished and pushed. CI went live once the GitHub token had the workflow scope, and the VM snapshot m1-frozen was taken.
• The owner asked for strictly one step at a time. From then on no review runs while the next chunk is built.
• Chunk A review found a serious bug in step 3. The backup of an M1 database read the file through a second descriptor; closing it silently dropped SQLite’s lock for the whole process. With several processes opening an old store at once (sc and scd at boot), 13 of 20 failed and an index could be corrupted. Fixed by reading the backup through SQLite itself: 0 of 80 failed afterwards. The tests had missed it because they used goroutines, which share one process; the new tests use real processes.
• Steps 4 and 5: the pattern language and the default scope. Checked against this VM’s real /etc (read-only): the numbers matched the plan’s research prototype exactly.

• First scope change, after the owner reviewed what is excluded: /etc/alternatives is now recorded (an iptables legacy/nft switch was invisible), and TLS private keys and PPP secrets are fingerprinted instead of ignored (a replaced key is now seen; secrets are still never stored).

• Chunk C review found no secret leak for the named paths, but two medium gaps: ssh-keygen writes a new private host key under a temporary name first, which would have been stored with its content; and scripts that systemd generators and run-parts directories execute were hidden by the noise rules. Both fixed, plus four small ones, each with a test that fails on the old code.

• Steps 6-10 (chunk B): safe reads that never follow a symlink, row kinds for links, deletions and fingerprints, restores of links and deletions, a kind-aware CLI, and proof that the M1 binary still works on an upgraded store. Its review (two reviewers) found one medium bug that both found independently: a host key or machine-id reached through a symlinked directory was stored with its content. Fixed, with four small fixes and ten test gaps closed.

• Steps 11-13 (chunk D), the watcher engine: inotify readers, a debounced worker, proof of absence, rescans, overflow handling (tested against a real kernel overflow), disk and user-file limits, and a crash test that kills a watcher mid-baseline. Its full review (three reviewers) found the most serious bug so far: as a normal user, a directory named authorized_keys in ~/.ssh and a symlink swap made the root watcher store a root-only file. It is fixed twice over (scope and an O_NOFOLLOW read at every level), and the attack is now a permanent test. The same review found an endless loop, a crash, a hang after a panic and a rescan race, all fixed with tests.

• Steps 14-16 (chunk E): sc watch, the scd.service unit, the 19-step acceptance script and the docs. Its review found that the acceptance script, started as the worklog then said, would have restored root’s own authorized_keys under nested sudo, and could have stopped itself before cleaning up. Both were fixed before it was ever run, which is why reviews come before anything touches the real system.

5. Where we moved away from the plan

All changes since approval, with reasons, are kept in one place: M2_PLAN.md Appendix C. In short:

	Change	Why
C1	Overnight run stopped in step 3	Unknown; finished interactively
C2	No overlapping reviews	Owner: one step at a time
C3	Light review by one reviewer agent plus Claude as skeptic	Enough for a light review, far cheaper; same method
C4	Backup read through SQLite; process-level lock tests	Review finding D1 (high)
C5	New known limit for never-migrated stores on read-only disks	Accepted: the real store is migrated long before M4
C6	SC_HOME excluded by a path check, not a glob	Glob characters in a path cannot misfire
C7	Record alternatives; fingerprint private keys and secrets	Owner, after reviewing the excluded list
C8	Fingerprint host-key temp names; more consumer directories in block D	Chunk C review findings D1-D6
C9	Record ~/.ssh/rc and environment; fingerprint dropbear, WireGuard, apt auth and LUKS key files	Owner, on two questions from the chunk C review
C10	Step 6 reuses M1’s Meta.Stamp and LstatStamp	No second copy of an API the store already uses
C11	Computed intents only for watcher rows	sc snapshot -m stays what the user typed
C12	Obs.Suffix; the CLI turns log priorities into journald or time prefixes	Plan had no field for the suffixes
C13	A rescan for a moved root waits Cap	A root moved away and back was dropped for an hour
C14	SIGHUP rescans; unit reload and remote-fs ordering; acceptance fixes	Chunk E review
C15	Digest rows get a random id, not one derived from their fingerprint	Final review: a journal reader could test guesses at a short secret
C16	A row budget per system file: 20 rows, then one every 5 min	Final review: a file rewritten every second gave about 86,000 rows a day

6. Lessons so far

• A test that passes on the first run proves little. Every step now gets mutation checks: break the code on purpose and confirm a test fails. This found weak or missing tests in steps 3, 4 and 5 before they were committed.
• Concurrency must be tested the way it happens. Goroutines in one process hid a lock bug that separate processes showed at once.
• Independent review pays. Every review round has found something the author’s own tests did not; the chunk D security review found a way for a normal user to read root’s files through the watcher.
• Attack your own tests. Several tests passed whatever the code did (a stall placed too late, a barrier that missed moved directories, a panic test whose reader was never blocked). Mutation checks found each.
• A check that cannot fail is no check. A pipe through tail once hid a failing test; the checks now run with pipefail.
• Check the real machine, read-only. Running the scope against the real /etc listing is what makes the exclude list trustworthy.

7. Where we were on 2026-10-01

M1 done. M2: built (steps 1-16 of 17) and every chunk reviewed and closed. What is left is the owner’s part: the acceptance run, the M1 smoke run, the real store (S2), the owner scenario, a reboot, failure paths and a 24-hour soak, each with a VM snapshot first. Step 17 (tag m2) closes M2.

8. M2’s end to M5, in brief (2026-10-02 to 10-09)

	When	What happened, and the turns that mattered	Changes from the plan
M2	tag m2 2026-10-02	Signed off on the dev VM: the acceptance and smoke runs, the real store migrated, a reboot, kill -9 in a dpkg burst. The 24-hour soak was closed after 5.6 h (your call, 2026-10-03).	M2 Appendix C, C15-C16 above
Order	2026-10-03	The roadmap’s question 1 was answered: M3 before M4. M4’s boot tests need QEMU without KVM, which is slow on this VM, and M3 could be built while the soak ran.
M3	plan approved and tag m3 2026-10-03	Checkers, sc check, sc edit, sc scope. Probing first changed the design: a stock Ubuntu already fails some validators, so an edit is judged by the findings it adds. netplan generate asks systemd to reload, so the generator binary is used instead. findmnt misses a misspelt option, so sc has rules of its own. Four agents built chunk C’s checkers in parallel worktrees (your call). The final review found two high findings (files that lock the owner out or stop the boot with no word from sc); both were fixed before the tag.	M3 section 15, C1-C9
M4	plan approved 2026-10-04, tag m4 2026-10-07	The rescue path: a GRUB entry, sc status above the rescue shell’s prompt, boot verdicts, the menu back after a failed boot. The QEMU lab (lab/, make lab-e2e) proves it end to end under TCG. The lab showed that Ubuntu 24.04’s sulogin opens a root shell at the console even with root locked (PROJECT_LOG D6). Host stalls cost many lab runs their verdict, so the lab gained verdict classes (PASS, FAIL, INCONCLUSIVE) and retries a boot only for known lab flakes (a stall, TCG’s own panic).	M4 section 12, C1-C8
Follow-ups	2026-10-07 to 08	The M3 and M4 follow-up lists, in three reviewed chunks (F, G, H). Signed off with the lab in both modes (with the GRUB password recipe) and accept-m2/m3/m4 as root.	MILESTONES
M5	plan drafted 2026-10-08, tag m5 2026-10-09	The package, built with dpkg-deb rather than nfpm (no new tool). Your call was that M5 starts whether or not you had read the plan, so its questions ran on the recommended answers. On the dev VM, the image’s policy-rc.d stopped every package script from restarting scd. It was found in the sign-off and moved aside on your answer.	M5 section 9, C1-C8

Lessons from these milestones, added to section 6’s:

• Probe the real tool before trusting it. Every validator form was run on this VM before sc used it, and one “check” (netplan generate) turned out to change the system.
• A test rig must tell its own failures from the product’s. The lab’s verdicts separate a failed SmartConfig check (FAIL) from a lab or host problem (INCONCLUSIVE), and only known lab flakes are retried.

9. Where we are (2026-10-09)

M1 to M5 are done and tagged; this VM runs the package, 0.5.0. Next is M6, the incident factory and eval set. Its plan is drafted (~/smartconfig-work/M6_PLAN.draft.md) and being revised after a review of the whole plan on 2026-10-09; nothing of M6 is built. M7, a local model (sc why), is not planned yet. That review also found work that no milestone owns, among it retention, secrets kept in the store, and a restore that writes without a check. It is listed at the end of MILESTONES.md for your decision.