SmartConfig

M2 watch list: what scd watches, ignores, and must beware of

Research for milestone 2 (the scd watcher), done on the development VM (Ubuntu 24.04.5, kernel 7.0.0-34, systemd 255, GRUB 2.12) on 2026-09-27. Six research agents surveyed boot, access, network/apt, a census of /etc, inotify behaviour (real experiments with sed, nano, install, mv, visudo, dpkg-style renames and sc restore), and prior art (etckeeper, AIDE, CIS audit rules). A synthesis merged them; 8 key claims were re-tested by independent verifiers trying to refute them; a completeness critic looked for gaps. Nothing under /etc or /boot was modified (only inotify-tools was installed, which rebuilt /etc/ld.so.cache).

Status: research only. No M2 code exists. The decisions in section 8 come first.

Contents

  1. Tiers: what to watch
  2. Exclude list
  3. Special handling
  4. Design rules for scd
  5. Gaps found by the critic (to fold in)
  6. Verification of key claims
  7. Changes needed in M1 code before M2
  8. Decisions to make
  9. Facts about this VM

1. Tiers: what to watch

Tier 1: boot

A mistake stops the machine booting or reaching a usable state. Watch by default and alert loudly. Many of these only take effect after a regeneration command, which unattended upgrades run with no human present.

Path Why it matters Takes effect On this VM
/etc/fstab A line without nofail whose device is missing becomes a hard requirement of local-fs.target: boot waits 90 s, then OnFailure=emergency.target drops to the emergency shell (generator test: dead UUID -> local-fs.target.requires). The root device comes from root=UUID on the kernel command line; the / line only supplies remount options and fsck pass. Note: The initrd carries a 0-byte fstab, so update-initramfs is not needed. Also watch /etc/fstab.d/*.fstab if it appears (the initramfs fsck hook reads it). next boot; also at any systemctl daemon-reload (the fstab generator re-runs, and apt triggers reloads) or mount -a exists, regular 0644 root:root 446 B, one by-uuid line, not package-owned; last changed 2026-09-27 08:26 (film demo sed -i then sc restore)
/etc/crypttab On encrypted installs a wrong entry blocks unlocking root at boot (not verified here). Creating it must be seen. Note: Needs a non-recursive watch on /etc filtered to this name. next boot or daemon-reload; on LUKS systems also update-initramfs (not verified) missing; cryptsetup not installed
/etc/default/grub Bad GRUB_CMDLINE_LINUX(_DEFAULT) root=/init= or a wrong GRUB_DEFAULT passes grub-script-check and breaks the next boot. Shell syntax errors abort grub-mkconfig, so the old grub.cfg stays. The menu is hidden, so the user cannot easily pick another entry. Note: A bad edit lies dormant until grub.cfg is regenerated. A grub-pc upgrade can rewrite it via ucf and debconf. only after update-grub, which runs automatically on every kernel install or removal via /etc/kernel/postinst.d/zz-update-grub, including unattended security upgrades exists, regular 0644 root:root 1550 B, ucf-managed by grub-pc (not a dpkg conffile), mtime 2025-10-30
/etc/default/grub.d/ grub-mkconfig sources every *.cfg here after /etc/default/grub, so a drop-in silently overrides the main file. Note: Only *.cfg is active; a rename to *.cfg is itself an activation. after update-grub (manual or at kernel install) exists, empty dir 0755 root:root
/etc/grub.d/ Generator scripts for grub.cfg. They run only if executable and not garbage-named. chmod -x 10_linux gives a grub.cfg with no Linux entries at the next update-grub, and emits only one IN_ATTRIB event. Note: The exec bit is content (see special handling). after update-grub (manual or at kernel install) exists: 12 scripts 0755 root:root (00_header .. 41_custom, grub-common) plus README 0644
/boot/grub/custom.cfg 41_custom sources it at BOOT time, so creating or editing it changes the next boot with no update-grub. Note: Needs a non-recursive watch on /boot/grub that acts only on custom.cfg (plus grub.cfg as the applied marker). next boot missing
/etc/initramfs-tools/ Shapes the initrd. MODULES=dep or list on the wrong hardware can drop the disk driver, so boot stops at the (initramfs) shell. A failing hook fails the kernel postinst. scripts/* run in early boot. update_initramfs=no silently turns update-initramfs -u into a no-op. Note: Watch recursively. The exec bit on hooks/ and scripts/ matters. update-initramfs -u (dpkg triggers) or -c on every kernel install exists: initramfs.conf 1502 B (MODULES=most, COMPRESS=zstd), modules 246 B (comments only, unowned), update-initramfs.conf 378 B; 14 dirs, hooks/ conf.d/ scripts/* all empty
/etc/modprobe.d/ Read by every modprobe. A blacklist or ‘install X /bin/false’ for ahci, ext4 or e1000 (this VM’s only NIC, a loadable module) removes root or the network. Only *.conf is read. Note: Initramfs-fed source. immediately for the next module load; next boot; early boot only after update-initramfs (all *.conf are copied into the initrd) exists: 11 files 0644 (154-2507 B) plus blacklist-oss.conf -> /lib/linux-sound-base/noOSS.modprobe.conf (symlink)
/etc/modules Modules loaded at boot, also reached through /etc/modules-load.d/modules.conf -> ../modules. A bad name makes systemd-modules-load fail, and boot continues (general systemd behaviour, not verified). Note: Snapshot the real file here; record the modules-load.d link as a link. next boot or systemctl restart systemd-modules-load exists, 0644 212 B, comments only, not package-owned
/etc/modules-load.d/ *.conf lists modules loaded at boot. A same-name file overrides /usr/lib/modules-load.d. next boot or systemctl restart systemd-modules-load exists: cups-filters.conf 119 B plus modules.conf -> ../modules (symlink)
/etc/sysctl.conf Applied last at boot through /etc/sysctl.d/99-sysctl.conf -> ../sysctl.conf. Valid but harmful values (kernel.modules_disabled=1, vm.overcommit_memory=2 with a low ratio, a tiny pid_max) make the system unusable. Unknown keys are only logged. next boot (systemd-sysctl), or immediately with sysctl –system or sysctl -p exists, 0644 2209 B, no active lines, dpkg conffile
/etc/sysctl.d/ Same effects as sysctl.conf; only *.conf is read. A same-name file replaces /usr/lib/sysctl.d/ (for example 10-apparmor.conf, 99-protect-links.conf, 50-pid-max.conf), so an empty one silently drops those settings. next boot or sysctl –system exists: 9 *.conf plus README.sysctl plus 99-sysctl.conf -> ../sysctl.conf (symlink)
/etc/systemd/system/ Local units, drop-ins, enable links (.wants/.requires), masks (-> /dev/null), default.target and aliases. Masking journald, dbus, logind or getty, a bad drop-in on a sysinit unit, or default.target pointing at rescue or a missing target stops or degrades boot. Note: Watch recursively. New .d/ dirs appear from systemctl edit (add the watch, then scan). About 90% of entries are symlinks, which sc refuses today. snap* entries are excluded. daemon-reload or next boot; enable/disable links at next boot exists: 28 dirs, 149 symlinks, 17 regular files, all snapd-generated (16 snap-*.mount plus snap.mesa-2404.component-monitor.service); no admin units or *.d drop-ins today; display-manager.service -> /lib/systemd/system/gdm3.service; default.target absent (graphical via /usr/lib)
/etc/systemd/system.{control,attached}/ Both are in the unit search path, and system.control has higher precedence than /etc/systemd/system. systemctl set-property writes persistent drop-ins into system.control (general systemd knowledge, not verified). Note: Needs a non-recursive watch on /etc/systemd for their creation; recurse once they appear. daemon-reload or next boot both missing (systemd-analyze unit-paths lists them)
/etc/systemd/system-generators/ Executables here run as root at every boot and every daemon-reload, before units load. A link to /dev/null masks the same-named /usr/lib generator. Note: Link records only. next boot or daemon-reload exists: systemd-gpt-auto-generator -> /dev/null (symlink, the only mask in /etc)
/etc/systemd/system.conf PID 1 defaults for every service (DefaultTimeoutStartSec, DefaultLimitNOFILE, DefaultTasksMax, CPUAffinity, DefaultEnvironment). Also covers /etc/systemd/system.conf.d/*.conf. next boot; daemon-reexec (not verified) exists, 0644 2318 B, [Manager] header only, conffile; system.conf.d missing
/etc/udev/rules.d/ A same-name file REPLACES the /usr/lib rule. For 10 core names (50-udev-default, 60-block, 60-persistent-storage, 80-drivers, 80-net-setup-link, …) the initramfs udev hook copies the /etc version into the initrd. So an empty 60-persistent-storage.rules removes /dev/disk/by-uuid, which both root=UUID and fstab need. NAME= rules rename NICs, so netplan’s enp0s3 no longer matches. Note: 70-snap.* is excluded. The initrd copy of 70-snap.firefox.rules is already stale vs /etc (proof that initrd copies freeze). next boot, or udevadm control –reload plus udevadm trigger; the initrd copy only after update-initramfs exists: 5 files, all snapd-generated 70-snap.*.rules (2226-63357 B); no admin rules
/etc/udev/udev.conf udevd settings (log level, event timeout), copied into the initrd. Low risk. next boot; initrd copy after update-initramfs exists, 0644 305 B
/etc/tmpfiles.d/ Processed at every boot: an ‘R’ or ‘D’ line deletes data. A same-name file masks /usr/lib/tmpfiles.d (for example the one creating /run/sshd). General systemd behaviour, not exercised here. Note: A naive ‘/etc/tmp*’ exclude would hide this directory. every boot, systemd-tmpfiles –create, and the daily clean timer exists, empty dir
/etc/ld.so.conf Input to ldconfig. Losing the include line, or adding a dir of incompatible libraries, changes the libraries every dynamically linked program loads. Note: Applied marker: /etc/ld.so.cache MOVED_TO. after ldconfig, which the libc-bin dpkg trigger runs after most apt installs (17 times in dpkg.log, twice today) exists, 0644 34 B ‘include /etc/ld.so.conf.d/*.conf’, conffile; also copied into the initrd
/etc/ld.so.conf.d/ Same as ld.so.conf; only *.conf is included; packages add files here. after the next ldconfig (any package install) exists: fakeroot-x86_64-linux-gnu.conf 38 B (added 2026-09-27 00:00), libc.conf 44 B, x86_64-linux-gnu.conf 100 B
/etc/ld.so.preload The loader reads it for EVERY new dynamically linked process: no apply step, no reboot. A bad library breaks sudo, login, bash and every service start. Only static binaries survive (bin/sc, busybox). Note: Highest severity: alert on creation (non-recursive /etc watch). scd must stay static. immediately missing
/etc/kernel/ postinst.d, postrm.d and preinst.d run via run-parts –exit-on-error on every kernel install. zz-update-grub and initramfs-tools there make a new kernel bootable. chmod -x, a rename to *.sh, or a delete silently stops initrd or grub.cfg generation for the next kernel. A non-zero exit leaves the kernel half-configured. Note: Mode and name are content. next kernel install or removal (unattended security upgrades) exists: postinst.d = initramfs-tools, unattended-upgrades, update-notifier (symlink -> /usr/share/…), xx-update-initrd-links, zz-update-grub (all 0755); postrm.d 2; preinst.d 1; install.d empty; /etc/kernel/cmdline absent
/etc/apparmor.d/{tunables,abstractions,abi}/ Shared includes: tunables/global alone is included by 112 profiles. One syntax error makes every profile fail to parse at the next apparmor.service run. Confinement is lost and snap-confine’s profile fails, so snaps such as Firefox likely stop launching (snap effect not verified). Boot itself continues. Note: Borderline tier 1 (reaching a usable state, not booting). The rest of /etc/apparmor.d is tier 4. next boot (apparmor.service, Before=sysinit.target) or systemctl reload apparmor exists: tunables/global 897 B, abstractions/base 7093 B; aa-status: 157 profiles loaded

Tier 2: access

A mistake locks people out: sudo, console login, ssh, desktop login. Most bite immediately (next sudo, next login). The ssh and GDM files bite only at the next service start, which is a delayed bomb.

Path Why it matters Takes effect On this VM
/etc/sudoers sudo re-reads it on every run. If it is not owned by uid 0 or is world-writable, sudo refuses for everyone (verified in a private mount namespace). An empty file, or one missing ‘@includedir /etc/sudoers.d’, drops every rule. A bad line is skipped by sudo 1.9.15. Note: visudo -c -f passed a 0666 copy, so owner and mode must be checked separately. Restore exactly 0440 root:root. immediately (next sudo) exists, 0440 root:root 1800 B, sudo conffile, unmodified
/etc/sudoers.d/ Every name without ‘.’ and without a trailing ‘~’ is live policy the moment it appears, including README. A broken drop-in is skipped, which loses what it granted: here vboxuser loses NOPASSWD and sudo -n fails. Note: The NOPASSWD rule hides most sudoers breakage on this VM; test with a user that has no NOPASSWD rule. immediately exists: README 0440 1068 B (parsed), 90-vboxuser-nopasswd 0440 33 B ‘vboxuser ALL=(ALL) NOPASSWD: ALL’ (created 2026-09-27 00:48, unowned)
/etc/sudo.conf Read on every sudo run. A Plugin line pointing at a missing .so blocks sudo completely. Junk lines or an empty file are tolerated. A non-root owner makes sudo ignore the file. immediately exists, 0644 4343 B, all comments, conffile
/etc/pam.d/ A missing required module in common-account blocks sudo, su, ssh, console and GDM for everyone (verified: even root’s su failed). A missing required module in common-session-noninteractive kills sudo. Every file name is a PAM service name. Note: common-* is a group (special handling). The stock login file already references a missing pam_lastlog.so on an optional line. immediately (next PAM transaction) exists: 30 files plus gdm-smartcard -> /etc/alternatives/gdm-smartcard (symlink); common-* are unowned and generated by pam-auth-update (rewritten 2026-09-23 17:04:22)
/etc/security/ pam_limits is required in sudo, sshd, login, su and gdm: ‘vboxuser hard nofile 2’ in limits.conf made ‘su - vboxuser’ fail. limits.d/*.conf behaves the same. access.conf, faillock.conf, time.conf and namespace.conf are inert today but lock everyone out once their module is enabled. Note: opasswd is excluded. next login or session exists: 15 files incl. limits.conf 2752 B, limits.d/{10-gamemode,25-pw-rlimits}.conf, access.conf 4564 B, faillock.conf 2234 B, pam_env.conf 2971 B, pwquality.conf; no pam_access or pam_faillock in any active stack
/etc/{passwd,group,shadow,gshadow} Every new process reads them. A broken line or lost entry locks users out. group decides admin rights (%sudo in sudoers, unix-group:sudo in polkit). shadow holds password hashes and locks. Note: Handled as one set with subuid/subgid (special handling). shadow and gshadow are secret. immediately passwd 0644 2916 B (50 lines), group 0644 1077 B, shadow 0640 root:shadow 1327 B (rewritten 2026-09-26 23:57 by passwd), gshadow 0640 root:shadow 904 B; none package-owned
/etc/nsswitch.conf ‘passwd: fiels …’ made every user vanish (id, su, sudo, ssh, GDM all fail). A group: line without ‘files’ drops sudo membership. ‘hosts: files’ kills DNS. An empty file is safe (glibc defaults). Note: Also copied into the initrd. Also matters for tier 3 (DNS). immediately for every new process exists, 0644 594 B, not package-owned; passwd: files systemd sss; hosts: files mdns4_minimal [NOTFOUND=return] dns
/etc/login.defs UID ranges, umask, ENCRYPT_METHOD and password aging for shadow-utils and login. Tolerant: junk lines or an empty file did not break su or sudo. next useradd or login exists, 0644 12345 B, conffile
/etc/shells chsh requires the new shell to be listed (pam_shells). gnome-session re-execs through a login shell (which sources /etc/profile) only if $SHELL is listed. immediately exists, 0644 118 B, 7 shells, not package-owned (add-shell writes shells.tmp + mv)
/etc/environment pam_env readenv=1 loads it into sudo, sshd, login and gdm sessions. A broken PATH makes commands ‘not found’ in new sessions. next login or sudo exists, 0644 106 B, single PATH= line, not package-owned
/etc/profile Sourced by every login shell (ssh, console, ‘su -‘, and GNOME via its login-shell re-exec). An ‘exit’ or a hang kills the login right after authentication. Plain syntax errors are survived. next login exists, 0644 582 B, not a conffile
/etc/profile.d/ /etc/profile sources every *.sh here. One file containing ‘exit 1’ made ‘su -‘ fail (verified). Third-party installers drop files here. next login exists: 11 files (mostly 0644, Z99-*.sh 0755) plus debuginfod.{sh,csh} -> /usr/share/libdebuginfod-common/ (symlinks)
/etc/bash.bashrc Sourced by interactive bash and by login shells via /etc/profile. An ‘exit’ kills interactive ssh and terminal sessions. next interactive shell exists, 0644 2319 B, conffile
/etc/ssh/sshd_config A delayed bomb: the running listener never re-reads it. At the next ssh.service start, ExecStartPre ‘sshd -t’ fails with status 255 and RestartPreventExitStatus=255 stops retries, so SSH is gone. ‘PasswordAuthentication no’ without keys, AllowUsers or a bad Match block pass sshd -t and still lock out. Note: Warn at snapshot time (sshd -t belongs in M3). next ssh.service start (reboot, restart, or an apt-triggered restart) exists, 0644 3517 B, ucf-managed by openssh-server; ‘Include /etc/ssh/sshd_config.d/*.conf’ at line 12
/etc/ssh/sshd_config.d/ Included first, and the first match wins in sshd, so a new *.conf here overrides the main file. A missing include glob is not an error. next ssh.service start exists, empty dir 0755
/etc/default/ssh EnvironmentFile of ssh.service; SSHD_OPTS is appended to the sshd command line, so a bad flag stops sshd at its next start. next ssh.service start exists, 0644 133 B ‘SSHD_OPTS=’
/etc/hosts.{allow,deny} sshd links libwrap.so.0, so a line like ‘sshd: ALL’ or ‘ALL: ALL’ in hosts.deny refuses new SSH connections. libwrap reads the files per connection. Note: Braces here mean two files. immediately for new connections (effect on a live connection not verified) hosts.allow 0644 411 B and hosts.deny 0644 711 B, comments only, not package-owned
/etc/polkit-1/rules.d/ Local rules override /usr/share/polkit-1/rules.d and decide GUI admin actions (updates, network, users). sudo is unaffected. Note: A restore must keep group polkitd (GID 987, dynamically allocated). immediately (polkitd PID 841 holds an inotify watch on this dir) exists, empty, dir 0750 root:polkitd
/etc/gdm3/custom.conf Autologin, Wayland and debug settings. Restoring an old copy can silently re-enable autologin. How GDM handles a malformed file was not verified. Note: Warn when a change toggles AutomaticLoginEnable. next gdm start (next boot) exists, 0644 554 B, section headers only, gdm3 conffile
/etc/gdm3/{Init,PostLogin,PreSession,PostSession,Prime,PrimeOff}/, /etc/gdm3/Xsession Scripts GDM runs as root around every login. Per GDM docs a non-zero PostLogin aborts the login (not verified). The exec bit must survive a restore. next login exist, 0755 root:root: Init/Default 2795 B, PreSession/Default 258 B, PostSession/Default 18 B, Prime/Default 97 B, PrimeOff/Default 93 B, Xsession 7025 B; PostLogin holds only Default.sample
/etc/systemd/logind.conf NAutoVTs=0 disables the tty2-6 gettys, so if GDM breaks there is no text login. HandlePowerKey or HandleLidSwitch can surprise the user. Also covers /etc/systemd/logind.conf.d/. next boot (logind start) exists, 0644 1663 B, [Login] header only, conffile; logind.conf.d absent (vendor drop-in in /usr/lib)
/etc/default/{keyboard,console-setup} Copied into the initrd and applied by keyboard-setup.service (sysinit). A wrong XKBLAYOUT means passwords cannot be typed in the initramfs shell, the rescue shell or on the VTs, which is exactly where sc restore is run. Note: Initramfs-fed sources. The generated console-setup/cached_* files are excluded. next boot; the initrd copy after update-initramfs keyboard 0644 150 B (pc105, us), console-setup 0644 285 B, both debconf-generated and unowned; /etc/vconsole.conf -> default/keyboard (symlink)
/root/.ssh/ sshd reads .ssh/authorized_keys and authorized_keys2, and PermitRootLogin is without-password. An emptied file loses key-based root login; an injected key is a backdoor. Note: Outside /etc (open question). immediately for new connections exists, dir 0700; authorized_keys 0600 root 0 B (2026-05-18)
/home/*/.ssh/ Users’ authorized_keys; a bad edit locks that user out of SSH. Note: Outside /etc and user-owned (open question). Take home dirs from /etc/passwd, not from a /home glob. immediately for new connections /home/vboxuser/.ssh/authorized_keys 0600 vboxuser 0 B; only root and vboxuser have login shells

Tier 3: network and updates

A mistake cuts the machine off or breaks apt. Several of these are not package-owned, so apt –reinstall cannot restore them, and a broken apt config blocks the apt command that would fix it.

Path Why it matters Takes effect On this VM
/etc/netplan/ On a YAML error or unknown key the generator exits 1 and writes nothing. The running system keeps its old /run output, but at the next boot there is no NM connection and no /run override of /usr/lib’s unmanaged-devices=,except:type:wifi…, so enp0s3 is unmanaged and there is no network. Any extra *.yaml (even foo.yaml.bak.yaml) is merged in lexical order; a same-name file in /run/netplan shadows it. Note: Can hold secrets (NM writes 90-NM-.yaml with Wi-Fi PSKs). Keep the recorded mode; do not silently tighten 0644. Also check /usr/lib/netplan/00-network-manager-all.yaml when explaining. next boot, netplan apply, or any systemctl daemon-reload (apt triggers reloads; the generator reran at 08:26:41) 01-network-manager-all.yaml 0644 104 B (renderer NetworkManager), 50-cloud-init.yaml 0600 65 B (enp0s3 dhcp4); neither package-owned
/etc/NetworkManager/{NetworkManager.conf,conf.d/} A syntax error makes NM refuse to start (exit 7) and loop under Restart=on-failure, so there is no network. A conf.d/10-globally-managed-devices.conf with unmanaged-devices=* drops enp0s3; dns=none freezes DNS. Only *.conf in conf.d is read. next NM start, boot, or nmcli general reload NetworkManager.conf 0644 98 B (plugins=ifupdown,keyfile; [ifupdown] managed=false), conffile; conf.d/default-wifi-powersave-on.conf 32 B
/etc/NetworkManager/system-connections/ NM keyfiles hold PSKs, VPN and 802.1x secrets, and NM ignores any keyfile that is not 0600 root. A restore with the wrong mode makes the connection vanish. Note: Secret. next NM reload or boot exists, empty, dir 0755 (ethernet comes from netplan)
/etc/resolv.conf Every lookup goes through it (nsswitch uses dns; there is no nss-resolve module). An empty file made getent exit 2 immediately. Added nameserver lines become global DNS, because resolved reads the file in ‘foreign’ mode. Note: Special handling (type can flip between file and symlink). immediately regular file 0644 920 B, byte-identical to the resolved stub (nameserver 127.0.0.53), born 2025-10-30 19:01:17 by the installer, not package-owned; on stock 24.04 it is a symlink into /run
/etc/systemd/resolved.conf, /etc/systemd/resolved.conf.d/ A bad DNS=, Domains=~., DNSSEC=yes or DNSOverTLS=yes against a router that cannot do them, or DNSStubListener=no (which kills 127.0.0.53, the only DNS path here), breaks name resolution for the whole machine. Note: Watch /etc/systemd for creation of the .d dir. next systemd-resolved restart or boot resolved.conf 0644 1551 B, [Resolve] header only, conffile; resolved.conf.d missing
/etc/hosts Read on every lookup: ‘127.0.0.1 archive.ubuntu.com’ redirected apt in a test. An empty file is survivable because resolved synthesizes localhost and the hostname. immediately exists, 0644 273 B, not package-owned, no 127.0.1.1 hostname line; new inode 2026-09-27 17:16:41 (scripts/smoke.sh restore)
/etc/systemd/network/ *.link files are applied by udev even with networkd disabled, and the udev initramfs hook copies them into the initrd. A rename of enp0s3 breaks netplan’s match. *.network and *.netdev are inert while networkd is disabled. next boot or udev trigger; the initrd copy after update-initramfs exists, empty dir; systemd-networkd inactive and disabled
/etc/network/interfaces NM loads the ifupdown plugin with managed=false, so any interface listed in this file is ignored by NM. ifupdown is not installed, so nobody configures it and there is no network. Note: Needs a watch on /etc/network (also for interfaces.d/). next NM restart or boot missing; ifupdown not installed
/etc/ufw/, /etc/default/ufw ENABLED=yes with DEFAULT_INPUT_POLICY=DROP and no ssh rule cuts remote access. DEFAULT_OUTPUT_POLICY=DROP cuts apt and DNS. Dropping the DHCP or loopback accepts in before.rules breaks DHCP and 127.0.0.53. Note: 0640 must survive a restore. user.rules are rewritten by ‘ufw allow’ and ‘ufw deny’. ufw enable or ufw reload, or next boot once enabled ufw.conf 0644 312 B (ENABLED=no, inactive); user.rules, user6.rules, before.rules, after.rules and *.init are 0640 root:root; default/ufw 0644 1897 B (INPUT DROP, OUTPUT ACCEPT)
/etc/nftables.conf Inert while nftables.service is disabled. If it is enabled, its ‘flush ruleset’ wipes other rules at boot, and a DROP policy cuts the network. Note: Mode 0755 (shebang). Check-only validator is nft -c -f. next boot, if nftables.service is enabled exists, 0755 243 B (flush ruleset plus empty chains), conffile; service disabled
/etc/iptables/ rules.v4 and rules.v6 are restored at boot by netfilter-persistent; a bad ruleset cuts the network. Note: Needs a watch on /etc for creation. next boot missing; netfilter-persistent not installed
/etc/apt/sources.list Parsed by every apt command: one malformed line aborts all apt with exit 100, so apt cannot even install a fix. immediately (next apt command) exists, 0644 57 B (commented cdrom line), not package-owned
/etc/apt/sources.list.d/ A malformed stanza or conflicting Signed-By aborts all apt (exit 100). A truncated ubuntu.sources is worse: apt exits 0 with zero Ubuntu repos, so security updates stop silently. Only *.list and *.sources are read. Note: Warn when the number of Ubuntu origins drops. immediately ubuntu.sources 386 B (installer), google-chrome.sources 334 B (rewritten daily by cron), claude-desktop.list 154 B, original.list 70 B, ubuntu.sources.curtin.orig 2552 B (ignored by apt)
/etc/apt/apt.conf.d/ A syntax error in any parsed file makes every apt command (and apt-config) exit 100. Extensionless names are parsed, so vim’s ‘4913’ probe or an extensionless temp breaks apt while it exists. A dead proxy only prints W: lines. Note: Never exclude a name because it ‘looks temporary’. immediately 18 files 0644, incl. 20auto-upgrades and 50unattended-upgrades (ucf-managed) and zzzz-temp-installer-unattended-upgrade 1031 B (an installer leftover that apt still reads)
/etc/apt/preferences.d/ A bad pin makes apt exit 100. A wrong priority silently holds back or downgrades packages. immediately ubuntu-pro-esm-apps 437 B, ubuntu-pro-esm-infra 429 B (conffiles)
/etc/apt/{trusted.gpg.d,keyrings}/ A missing or corrupt key makes its repo fail with ‘NO_PUBKEY … is not signed’. Adding a key trusts a new repo. Note: Binary files: show a fingerprint change, not a text diff. Signed-By targets in /usr/share/keyrings are outside scope. next apt update trusted.gpg.d: ubuntu-keyring-2012-cdimage.gpg 1165 B and ubuntu-keyring-2018-archive.gpg 1167 B (dpkg-owned, not conffiles); keyrings/ empty
/etc/apt/auth.conf.d/ Credentials for private repos (Ubuntu Pro/ESM); a broken file gives 401s for those repos. Note: Secret; preserve 0600. next apt update exists, empty
/etc/dpkg/dpkg.cfg, /etc/dpkg/dpkg.cfg.d/ path-exclude, force-* and –pre/–post-invoke options change every package operation. next dpkg run dpkg.cfg 446 B; dpkg.cfg.d empty
/etc/ca-certificates.conf Chooses which CAs go into /etc/ssl/certs/ca-certificates.crt, which apt’s https method (GnuTLS) uses. Deselecting a root breaks TLS for sites that chain to it (dl.google.com, downloads.claude.ai here). The Ubuntu repos are http plus gpg and are unaffected. Note: The generated /etc/ssl/certs is excluded; its source is watched here. after the next update-ca-certificates (typically a ca-certificates upgrade) exists, 0644 6862 B, not package-owned, rewritten by the ca-certificates postinst 2026-09-23 17:04; a .dpkg-old sits next to it

Tier 4: services, optional

Worth history, lower stakes; candidates for opt-in. The last entry applies only if the user chooses to sweep all of /etc.

Path Why it matters Takes effect On this VM
/etc/systemd/journald.conf Does not stop boot, but Storage=none, volatile or a tiny SystemMaxUse destroys the previous-boot logs SmartConfig needs to explain what broke. Note: Cheap; consider default-on. next boot or journald restart exists, 0644 1406 B, header only, conffile; journal holds 5 boots
/etc/systemd/timesyncd.conf, /etc/systemd/timesyncd.conf.d/ timesyncd is the only time source. Bad NTP= stops syncing, and days of drift would break TLS and apt Release validity (not verified). next timesyncd restart or boot timesyncd.conf 0644 1003 B header only; timesyncd.conf.d/cloud-init.conf 166 B (NTP=0-3.ubuntu.pool.ntp.org)
/etc/systemd/{user,networkd,oomd,sleep,pstore}.conf Per-daemon settings; none blocks boot; networkd is disabled here. next restart of each daemon or boot all exist, header-only conffiles (879-1768 B)
/etc/environment.d/ Read by the systemd user environment generator; affects user sessions only. next login 90atk-adaptor.conf 57 B, 90qt-a11y.conf 19 B
/etc/default/ The remaining service environment files, read when each service starts (the grub, keyboard, console-setup, ssh and ufw files are tiered above). next start of the owning service about 19 files (alsa, anacron, apport, cron, dbus, google-chrome, qemu-kvm, rsync, sssd, useradd, …); google-chrome rewritten daily; locale -> ../locale.conf (symlink)
/etc/apparmor.d/ A broken or over-strict single profile affects only that program after a reload. disable/ and force-complain/ links switch enforcement. local/ holds admin overrides. Note: The tunables/abstractions/abi subdirs are tier 1. next boot or systemctl reload apparmor 274 files, 13 dirs, 1 symlink (force-complain/usr.sbin.sssd); claude-desktop profile added by a package 2026-09-23; conffile nautilus deleted
/etc/crontab, /etc/anacrontab, /etc/cron.d/, /etc/cron.{hourly,daily,weekly,monthly}/ Root jobs can do anything but rarely take the machine down. next scheduled run crontab 0644 1136 B; cron.d 4 files; cron.daily 8 files plus google-chrome -> /opt/google/chrome/cron/google-chrome (symlink out of /etc)
/var/spool/cron/crontabs/ Per-user crontabs, root’s included. Note: Outside /etc. crontab(1)’s temp pattern not verified. next scheduled run exists, dir 1730 root:crontab, empty
/etc/cups/{cupsd,cups-files,cups-browsed,snmp}.conf Printing daemon config; cannot lock anyone out. Note: cupsd’s own state files are excluded. next cupsd restart cupsd.conf 6795 B, cups-files.conf 2939 B, cups-browsed.conf 29545 B
/etc/cloud/ If cloud-init.disabled is removed, cloud-init runs at the next boot and can regenerate netplan, hostname and timesyncd drop-ins over hand edits. Note: 99-installer.cfg is excluded (secret). next boot 64 files; cloud-init.disabled 132 B; cloud.cfg.d/90-installer-network.cfg 0600 117 B
/etc/hostname The hostname is set at boot or via hostnamectl. It does not cut the network here, because resolved synthesizes the name. next boot or hostnamectl exists, 0644 7 B ‘Ubuntu’, not package-owned
/etc/{subuid,subgid} Subordinate ID ranges for rootless containers; useradd writes them in the same transaction as passwd. Note: Part of the account set. next newuidmap use both 0644 22 B ‘vboxuser:100000:65536’
/etc/{adduser.conf,deluser.conf} Defaults for future accounts only. next adduser or deluser adduser.conf 3444 B, deluser.conf 1706 B, conffiles
/etc/{host.conf,gai.conf,services,protocols,networks} Resolver options, address-family preference and name tables. An empty services file did not break apt. immediately for new processes host.conf 92 B, gai.conf 2584 B, services 12813 B, protocols 3144 B, networks 91 B
/etc/ssh/{ssh_config,ssh_config.d/,moduli,ssh_host_*_key.pub} Client config, DH moduli and public host keys; none locks anyone out. The .pub files show host-key regeneration. Note: The .pub files pair with the private keys (special handling). next ssh client run or sshd start ssh_config 1649 B; ssh_config.d empty; moduli 620042 B (the largest kept file); .pub 93-565 B
/etc/{locale.conf,timezone} Locale (read by pam_env via /etc/default/locale) and the legacy timezone name. next login locale.conf 0644 17 B ‘LANG=en_US.UTF-8’; timezone 8 B ‘Etc/UTC’ (rewritten by the tzdata postinst)
/etc/udev/hwdb.d/, /etc/udev/iocost.conf, /etc/binfmt.d/ Hardware database overrides, IO cost config and binfmt registrations; low boot risk. systemd-hwdb update, next boot hwdb.d empty; iocost.conf 865 B; binfmt.d empty
/etc/init.d/ Legacy SysV scripts, mostly wrapped by systemd. next service start 30 files, all conffiles
/etc/NetworkManager/dispatcher.d/ Executable hooks run on link events; they can run arbitrary commands. next link event 3 empty subdirs
/etc/{fwupd,bluetooth}/ Firmware update remotes and Bluetooth daemon config; not boot-critical. Note: Preserve fwupd.conf’s 0640. next daemon start fwupd.conf 0640 51 B plus remotes.d/*.conf; bluetooth main.conf 12455 B, input.conf, network.conf
/etc/sssd/ pam_sss is in the common-* stacks, but sssd is inactive. It becomes tier 2 if someone configures sssd. Note: A future sssd.conf may hold bind passwords (secret). next sssd start dir 0711 with only an empty conf.d/; sssd inactive
/usr/local/etc/, /usr/local/share/ca-certificates/ Config for locally built software, and admin-added CAs (an input to update-ca-certificates). Note: Outside /etc. varies; CAs after update-ca-certificates both exist, empty
/var/lib/polkit-1/localauthority/ Legacy .pkla authorizations granting desktop admin rights (50-local.d is the admin dir). Note: Outside /etc. immediately 10-vendor.d holds 3 dpkg-owned .pkla files; dir /var/lib/polkit-1 is 0750 root:polkitd
/etc/** (everything else not excluded) Every prior-art tool (etckeeper, AIDE, Tripwire, Samhain) covers the whole /etc tree minus a volatile list. Note: Only if the sweep option is chosen (open question 1). varies applying the exclude list keeps 1144 of 1663 regular files (2.46 MB) in 324 of 389 dirs

2. Exclude list

Excludes are a noise filter, not the correctness mechanism (see rule 4). They must not apply in directories whose consumer executes or loads those names (see critic finding on /etc/grub.d, initramfs hooks, apparmor).

Pattern Reason
/var/lib/smartconfig/** ($SC_HOME), plus scd's own log and temp dirs Feedback loop: an inotifywait logging into its own watched dir grew to 29 MB in seconds, and a fanotify probe logging to the watched fs got 248613 self-events in 10 s against 39 real ones. The store also holds copies of secrets.
**/.*.sc-tmp-* sc’s own atomic-write temp (restore and blob store); it gets CLOSE_WRITE before being renamed over the target.
**/*.dpkg-{new,tmp,old,dist,bak,remove,backup} dpkg staging and leftovers. The change is the MOVED_TO onto the final name. A .dpkg-new can sit in /etc between unpack and configure. .dpkg-old/.dpkg-dist are worth reporting as a pending merge, not snapshotting. On VM: /etc/ca-certificates.conf.dpkg-old.
**/*.{ucf-new,ucf-old,ucf-dist,merge-error} ucf equivalents of dpkg’s staging names (7 files here are ucf-managed, incl. /etc/default/grub and sshd_config).
**/*.pam-{new,old} pam-auth-update writes common-.pam-new, then renames it into place.
**/{*~,.*.sw[a-p],.*.swx,.*.swpx,4913,#*#,.#*,DEADJOE,.goutputstream-*} Editor backups, swap and lock files, vim’s write probe and the GIO temp. Swap files contain the buffer, including secrets, of files like /etc/shadow. None exist in /etc now; all were observed transiently in scratch tests.
**/{sed??????,XX??????,*.tmp} Temps from sed -i, perl -pi, visudo (sudoers.tmp), add-shell (shells.tmp) and chrome’s cron (NAME.$$.tmp). All get CLOSE_WRITE before being renamed away. ln -sf, patch and rsync use unpredictable names, so the structural ‘still exists after quiet period’ rule is the real guard.
/etc/{passwd,shadow,group,gshadow,subuid,subgid}{.lock,+,.[0-9]*,.edit}, /etc/nshadow, /etc/npasswd, /etc/.pwd??????, /etc/.pwd.lock shadow-utils and pam_unix lock and staging names (useradd: passwd. -> passwd.lock, passwd+ -> rename). shadow+ and nshadow briefly hold password hashes.
/etc/{passwd,shadow,group,gshadow,subuid,subgid}- One-step backups rewritten in place on account changes. They are not a reliable previous version: pam_unix password changes never refresh shadow-, which still holds the May hash here. shadow- and gshadow- are secret. Never delete them.
/etc/security/opasswd PAM password-history state (old hashes), 0600, 0 B, unused; not config and cannot fix a lockout.
**/*.{old,orig,bak,save,rej,distUpgrade,disabled}, /etc/cups/*.O Backups that consumers ignore (apt’s Ignore-Files-Silently lists ~ .disabled .bak .save .orig .distUpgrade). On VM: /etc/xml/*.old x5, ubuntu.sources.curtin.orig, /etc/.resolv.conf.systemd-resolved.bak, cups *.O. Renaming a live name to .disabled is still recorded, as a deletion of the live name.
/etc/apt/preferences.d.save/** Image-build copy of preferences.d; apt does not read it.
/etc/tmp[a-z0-9_]{8}/** (e.g. /etc/tmpxdvph4n_) Installer mkdtemp leftover (0700, born 2025-10-30 19:01:17, holds a dangling resolv.conf symlink). Must NOT be written as /etc/tmp*, which would also hide the boot-relevant /etc/tmpfiles.d.
/etc/ssl/certs/** Generated by update-ca-certificates: 243 symlinks plus a 182140 B bundle; 29 entries changed in one upgrade. Watch /etc/ca-certificates.conf instead; the recovery is update-ca-certificates –fresh.
/etc/alternatives/** 146 symlinks managed by update-alternatives (26 relinked by one dev-tools install); the source of truth is /var/lib/dpkg/alternatives.
/etc/rc?.d/** 133 symlinks managed by update-rc.d; enablement is systemd’s job on 24.04.
/etc/console-setup/cached_* setupcon output, rewritten together on upgrades; watch /etc/default/{keyboard,console-setup} instead.
/etc/xml/** XML catalogs regenerated by update-xmlcatalog on package installs; all 10 files are unowned tool output.
/etc/dconf/db/* (compiled files directly in db/ only) Output of dconf update. Keep the sources /etc/dconf/db/.d/ and /etc/dconf/profile/; a find-style ‘’ crossing ‘/’ wrongly dropped ibus.d/00-upstream-settings in the census test.
/etc/systemd/system/{snap-*.mount,snap.*}, /etc/systemd/system/*.wants/snap*, /etc/systemd/system/snapd.mounts.target.wants/**, /etc/systemd/user/snap.*, /etc/udev/rules.d/70-snap.*.rules snapd writes and deletes these on every snap install or refresh (change 7 at 23:57-23:58 matched the mtimes); restoring them fights snapd. Names contain literal backslashes (snap-firmware\x2dupdater-258.mount). Open question: record with origin=snapd instead.
/etc/cups/{printers,classes,subscriptions}.conf*, /etc/cups/ssl/**, /etc/printcap cupsd runtime state: subscriptions.conf is rewritten about every 8 h here. printers.conf is 0600 because DeviceURI can hold credentials. printcap is a symlink into /run.
/etc/mtab Symlink to ../proc/self/mounts; inotify never reports procfs changes.
/etc/{machine-id,.updated,adjtime} machine-id is generated once and must never be restored (clone clash for journal and DHCP). .updated is a systemd-update-done timestamp. adjtime is hwclock state (absent here).
/etc/brltty/** (keep /etc/brltty.conf) 448 braille translation tables, 27% of all /etc files; static data that changes only on upgrade.
/etc/{ssl/private,credstore,credstore.encrypted}/**, /etc/brlapi.key, /etc/cloud/cloud.cfg.d/99-installer.cfg, /etc/ppp/*-secrets Secrets with no rescue value; opt-in only. 99-installer.cfg still holds two install-time $6$ password hashes.
/etc/.git/**, /etc/.etckeeper, /etc/.hg/**, /etc/.bzr/**, /etc/_darcs/** VCS metadata if etckeeper is ever installed: one commit plus checkout gave 345 events, and the repo contains secret copies. Not present.
/boot/grub/grubenv GRUB runtime state rewritten on every boot (mtime 15 s after kernel start, by grub-common.service). Restoring it could force or clear recordfail. Read it (never write it) to learn whether the last boot failed.
/boot/grub/{grub.cfg.new,unicode.pf2,gfxblacklist.txt}, /boot/grub/{i386-pc,fonts,locale}/** grub-mkconfig’s temp, plus GRUB modules, fonts (unicode.pf2 2.4 MB, twice) and locales installed by grub-install.
/run/** tmpfs, regenerated every boot: netplan’s /run/NetworkManager output, resolved’s stub, /run/netplan. Nothing there survives to snapshot.
/usr/** (vendor dirs) except /usr/local/etc and /usr/local/share/ca-certificates dpkg-owned and replaced on upgrade (/usr/lib/systemd alone has 680 files and 121 links); admins override in /etc. Read these only when explaining overrides. dpkg records them as /lib/… under usrmerge.
/var/cache/apparmor/**, /var/lib/snapd/**, /var/lib/NetworkManager/**, /var/lib/bluetooth/**, /var/lib/AccountsService/**, /var/tmp/** Compiled policy caches (22 MB), snapd-generated profiles, DHCP leases, pairing keys, accounts-daemon state, and sudoedit working copies (which hold root-only content while being edited).

3. Special handling

/etc/{passwd,group,shadow,gshadow,subuid,subgid} (account set)
Problem: useradd/usermod rewrite several files in one locked transaction (FILE. -> FILE.lock, FILE- backup, FILE+ -> rename), so a snapshot taken mid-way is inconsistent. A single-file restore desyncs the set: the older passwd- lacks the sshd user and 'sshd -t' then fails; the older group- drops vboxuser from sudo; an older shadow reverts the 2026-09-26 password and the GNOME keyring. shadow and gshadow hold hashes. pam_unix password changes (nshadow -> rename) touch only shadow. Handling: Debounce the set until no /etc/.lock exists, then snapshot all changed members as one change group. Redact field 2 of shadow and gshadow in diff, log and journal output ('password hash changed for USER'). Keep modes and owners exactly: shadow and gshadow 0640 root:shadow (0600 root:root broke unix_chkpwd, so the lock screen cannot unlock). A later restore works on the set only, validated with pwck -r and grpck -r, holding the shadow-utils locks, with a per-user summary.

/etc/pam.d/common-{auth,account,password,session,session-noninteractive} with /var/lib/pam/
Problem: Generated by pam-auth-update (all five rewritten within 16 ms at 2026-09-23 17:04:22 during apt), which keeps its state in /var/lib/pam/. Restoring common- without matching state makes the next pam-auth-update treat the difference as local edits.
Handling: Group the five files as one change. Ignore .pam-new/.pam-old. Record /var/lib/pam/* hashes as read-only metadata, never restore them automatically, and warn when they disagree.

/boot/grub/grub.cfg
Problem: This is the file GRUB actually reads (0600 root, 10431 B). It is generated from /etc/default/grub, /etc/default/grub.d and /etc/grub.d, and rewritten IN PLACE (‘cat grub.cfg.new > grub.cfg’), so the first MODIFY is a truncation to 0 bytes. Hand edits are overwritten at the next update-grub, and an old copy may reference kernels removed since.
Handling: Act only after CLOSE_WRITE plus the quiet period. Record (hash, mtime) as the ‘bootloader applied’ marker for its sources. Whether to also keep content as a last-good rescue copy is open question 6. The normal fix is to restore the sources and run update-grub.

/boot/initrd.img-*, /boot/vmlinuz-*, /etc/ld.so.cache
Problem: Generated outputs. The initrds are 80-82 MB (over the 8 MB cap) and are written as .new then ‘mv -f’ (MOVED_TO in /boot). ld.so.cache is written as ld.so.cache~ then renamed by ldconfig after most apt installs (08:02:49 and 19:15:41 today).
Handling: Never snapshot their content. Watch /boot and /etc non-recursively for MOVED_TO of these names and record (name, size, mtime, sha256) as ‘initramfs applied’ and ‘ld.so.conf applied’ markers. Recovery is to rerun update-initramfs -u or ldconfig.

Initramfs-fed sources: /etc/initramfs-tools/**, /etc/modprobe.d/*.conf, /etc/udev/rules.d/*.rules, /etc/udev/udev.conf, /etc/systemd/network/*.link, /etc/default/{keyboard,console-setup}, /etc/ld.so.conf(.d), /etc/nsswitch.conf
Problem: The initrd holds frozen copies taken at update-initramfs time. Restoring the /etc file does not fix early boot, and the initrd can still hold a bad version after /etc is fixed (the initrd’s 70-snap.firefox.rules already differs from live).
Handling: Tag snapshots of these paths as ‘initramfs source’. Restore output must say ‘also run: update-initramfs -u’. Later, compare snapshot time with the initrd mtime to say which version early boot uses.

Symlinks: /etc/systemd/system/**/*.{wants,requires}/*, masks to /dev/null, /etc/systemd/system/{default.target,display-manager.service}, /etc/systemd/system-generators/*, /etc/localtime, /etc/default/locale, /etc/vconsole.conf, /etc/modules-load.d/modules.conf, /etc/sysctl.d/99-sysctl.conf, /etc/modprobe.d/blacklist-oss.conf, /etc/pam.d/gdm-smartcard, /etc/profile.d/debuginfod.{sh,csh}, /etc/lighttpd/conf-enabled/*, /etc/xdg/systemd/user
Problem: Enable, disable, mask, default target and timezone are pure symlink state, and creating a link emits only IN_CREATE. sc refuses symlinks today; /etc has 791 of them. /etc/xdg/systemd/user is a directory symlink that would double-watch /etc/systemd/user.
Handling: lstat and never follow. Record a link record (target string, lstat owner); flag a link to /dev/null as ‘masked’. For links whose target is inside /etc, also watch the real file. Restore is symlink-to-temp plus rename. Never add a watch through /etc/xdg/systemd/user. Whether this is M2 is open question 3.

/etc/resolv.conf
Problem: On this VM it is a regular file (installer copy of the stub); on stock 24.04 it is a symlink into /run/systemd/resolve whose content churns on every DHCP event. Writes through the symlink are invisible to an /etc watch, and sed -i turns the symlink into a regular file.
Handling: Record the type. As a symlink, record only the target and never follow it into /run; as a regular file, snapshot the content. Treat a file-to-symlink flip (or the reverse) as a notable event. Restoring the stock symlink is a type change M1’s restore cannot express.

/etc/ssh/ssh_host_{rsa,ecdsa,ed25519}_key (with .pub)
Problem: Private host keys (0600 root, 399-2590 B). At mode 0644 sshd ignores a key, and with no usable key it exits ‘no hostkeys available’. Restoring an old key after a rotation flips the fingerprint every client sees and can resurrect a compromised key.
Handling: Always record hash, mode and owner, and alert on any ATTRIB mode change. Whether to store content (0600 blobs) is open question 5. Snapshot and restore each private key together with its .pub; never auto-restore.

Secret-bearing watched files: /etc/shadow, /etc/gshadow, /etc/netplan/*.yaml, /etc/NetworkManager/system-connections/*, /etc/apt/auth.conf.d/*, /etc/sssd/sssd.conf
Problem: Snapshots copy secrets (hashes, PSKs, repo credentials), and sc diff, sc cat and log output print content on the console.
Handling: Store blobs 0600 in a 0700 store (already true). Mark files secret by path list, with ‘not world-readable’ as a hint. Redact hash, psk=, password= and private-key values in diff, log and scd journal messages. Restore the exact mode and owner (NM ignores keyfiles that are not 0600 root).

Existence flags: /etc/ld.so.preload, /etc/nologin, /etc/ssh/sshd_not_to_be_run
Problem: Their mere existence changes behaviour: ld.so.preload affects every new process; nologin makes pam_nologin block every non-root login (login, sshd, gdm-*); sshd_not_to_be_run stops ssh.socket and ssh.service at their next start. All are absent today. ‘Restore’ means delete, which sc cannot express. /run/nologin is created by systemd at shutdown, so ignore that one.
Handling: Non-recursive watches on /etc and /etc/ssh filtered to these names, firing on CREATE and MOVED_TO. Record a ‘did not exist’ baseline and raise an alert, ld.so.preload at the highest severity. Snapshot ld.so.preload content; the other two are flags only.

Mode-is-content: /etc/grub.d/*, /etc/kernel/{postinst,postrm,preinst}.d/*, /etc/initramfs-tools/{hooks,scripts}/**, /etc/gdm3/*/Default
Problem: The exec bit decides whether grub-mkconfig or run-parts runs a script at all. chmod -x 10linux gives no Linux entries at the next update-grub. Renaming zz-update-grub to zz-update-grub.sh makes run-parts skip it. Both emit only ATTRIB or a rename, with no content change.
Handling: IN_ATTRIB triggers a metadata comparison and a metadata-only row. Raise a specific alert when an exec bit disappears, or when a hook is renamed to a name run-parts skips (a dot, or a character outside [A-Za-z0-9-]).

Vendor overrides: /etc/udev/rules.d/*.rules, /etc/sysctl.d/*.conf, /etc/tmpfiles.d/*.conf, /etc/modules-load.d/*.conf, /etc/NetworkManager/conf.d/*.conf, /etc/netplan/*.yaml
Problem: An /etc file with the same basename as the /usr/lib one replaces it, and an empty file or a /dev/null link disables it. For udev, the initramfs hook also prefers /etc/udev/rules.d/60-persistent-storage.rules.
Handling: On CREATE, check /usr/lib/

/ (also the /lib form, because dpkg records /lib paths under usrmerge) and label the event 'overrides vendor file' with raised severity. Keep a critical-name list: 50-udev-default, 60-block, 60-persistent-storage, 80-drivers, 80-net-setup-link, 10-apparmor, 99-protect-links, 10-globally-managed-devices.

/etc/apt/sources.list.d/google-chrome.sources, /etc/default/google-chrome
Problem: /etc/cron.daily/google-chrome (a symlink to /opt) rewrites both every day with identical content (tmp + mv, sed -i), which would produce one snapshot per day forever.
Handling: Content-plus-metadata dedup yields no new row. Optionally tag rows from this job as origin=cron.

ucf- and dpkg-managed files: /etc/default/grub, /etc/ssh/sshd_config, /etc/apt/apt.conf.d/{20auto-upgrades,50unattended-upgrades}, and the 1486 dpkg conffiles
Problem: Package upgrades legitimately rewrite them. Restoring an older version makes dpkg or ucf see the file as locally modified, so the next upgrade prompts, keeps the stale copy or merges oddly.
Handling: Record the owning package and the dpkg/ucf md5 in snapshot metadata. Attribute apt-time changes to apt. After a restore, warn when the file no longer matches the package version.

/root/.ssh/, /home/*/.ssh/
Problem: Outside /etc. Files are user-owned in 0750 homes, sshd StrictModes rejects group- or world-writable files, and the set of homes changes with /etc/passwd.
Handling: Derive homes from /etc/passwd login shells, not a /home glob. Add or remove watches when passwd changes. Restore must chown back to the user and keep 0600.

4. Design rules for scd

  1. Watch directories, never individual files. Filter by name in user space and key records by path.
    Because: inotify watches inodes. Every rename-over writer (sed -i, mv, dpkg, visudo, useradd, sc restore) kills a file watch, and later edits are lost. vim’s rename-style save moves the watch to FILE~.

  2. Use the mask IN_CLOSE_WRITE|IN_MOVED_TO|IN_MOVED_FROM|IN_CREATE|IN_DELETE|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF with IN_DONT_FOLLOW|IN_ONLYDIR. Treat every one of them as ‘path is dirty’. Use IN_MODIFY only to extend the debounce. Never subscribe to OPEN, ACCESS or CLOSE_NOWRITE.
    Because: No single event means ‘final’. In-place writers end with CLOSE_WRITE; rename writers end with MOVED_TO, and a mv from outside the dir gives only MOVED_TO. install and vim set the mode by ATTRIB after close. gio and visudo emit CLOSE_WRITE after MOVED_TO. chmod and chown give only ATTRIB, and a new symlink gives only CREATE. Read events would make scd trigger itself.

  3. Keep a per-path quiet period (start at about 500 ms, cap at a few seconds). Then lstat, read and hash, fstat again, and accept only if size, mtime, ctime and inode are unchanged; otherwise retry, and at the cap record the row marked ‘unstable’. Never snapshot on MODIFY.
    Because: nano, cp, tee, ‘>’, sudoedit write-back, ucf (cp -f) and grub-mkconfig all truncate first and write in place, so a MODIFY-time read sees 0 bytes or partial content.

  4. Apply a structural temp-file rule: when a path’s quiet period ends, snapshot it only if it still exists, is a regular file or symlink, and passes include/exclude; drop everything else. Exclude globs are a noise filter, not the correctness mechanism.
    Because: Temp names are unpredictable (ln -sf ‘Cu0SWPQ0’, patch ‘a.conf.oEjj3Ws’, rsync ‘.a.conf.O4dxr3’) and most get CLOSE_WRITE before being renamed away. Some live files look temporary (apt.conf.d/zzzz-temp-installer-unattended-upgrade).

  5. Treat DELETE or MOVED_FROM of a tracked name as a deletion only if the path is still absent when the quiet period ends. Then write a deletion row that points at the last blob.
    Because: vim, cp -b, install, rm plus recreate, and git checkout all remove the name before the new file appears. Real deletions are real config changes, such as a deleted drop-in or conffile.

  6. Deduplicate every origin on (sha256, mode, uid, gid, file type, link target) against the latest row. An mtime-only change is not a change.
    Because: touch, no-op chown, ed, gio, dpkg’s fsync pass, shadow-utils opening group read-write, grub-editenv and chrome’s daily cron all emit write events without changing anything. The current store dedups only OriginManual, and only on the blob, so auto rows would flood and mode-only changes would be missed.

  7. Make metadata-only changes first-class: on ATTRIB, compare mode, uid and gid, and record a metadata-only row that points at the same blob.
    Because: Several lockouts and boot failures are ATTRIB-only: chmod -x /etc/grub.d/10_linux leaves no Linux entries at the next update-grub; /etc/sudoers at 0666 makes sudo refuse; /etc/shadow at 0600 root:root breaks unix_chkpwd; a host key at 0644 is ignored.

  8. On CREATE|ISDIR or MOVED_TO|ISDIR, add the watch first, then readdir recursively and mark every entry dirty.
    Because: Files written into a new directory before its watch exists can produce no event. dpkg creates directories as NAME.dpkg-new and renames them. A populated directory moved in reports only itself. systemctl edit creates .d/ and then override.conf immediately.

  9. Keep a live wd-to-path map. Rewrite prefixes on MOVED_FROM/MOVED_TO|ISDIR cookie pairs, drop watches on an unpaired MOVED_FROM|ISDIR and on IN_IGNORED, and rescan the subtree when unsure.
    Because: After a directory rename its wd keeps reporting bare names, so a map built when the watch was added produces wrong absolute paths.

  10. Drain the inotify fd in a dedicated goroutine into an in-memory dirty set, so SQLite fsyncs never block reading. On IN_Q_OVERFLOW, run a full rescan of every watch root against the latest rows.
    Because: The queue holds 16384 events per inotify instance (root is not exempt). When it fills, the kernel keeps the queued events, drops one continuous stretch of new ones, and adds a single IN_Q_OVERFLOW marker (wd=-1) that does not say which paths were affected, including other, quiet watches. Events carry no content, so the final state is still on disk: a rescan comparing disk with the last snapshot recovers it. Intermediate versions written during the lost stretch cannot be recovered. (Verified on this VM.)

  11. Run a full rescan at every scd start (baseline-snapshot every included file; dedup keeps repeats cheap) and a daily backstop. Compare hash plus inode and ctime, never mtime alone. Stamp rows with wall-clock time and /proc/sys/kernel/random/boot_id, never uptime.
    Because: Nothing reports changes made while scd was down, at boot or pre-OS (grubenv, netplan generator), from the rescue shell, through hard links, or behind symlinks. inotify reports after the fact, so the ‘before’ version must already be stored. cp -p and rsync -t reset mtime. uptime lies on a paused VM.

  12. Make restores announce themselves. sc restore commits an intent marker (path, blob, pid, expiry) or its row before the rename, or scd treats a change whose hash equals a recent restore row for that path as origin=restore. Ignore ..sc-tmp- names. Test this with an in-process watcher.
    Because: store.Restore renames the file (store.go:341) before it commits its ‘restore’ row (store.go:349). Measured gap on this VM: 9.9-61.7 ms, median 19.5 ms, over 30 runs. A watcher that acts on MOVED_TO right away recorded an extra unattributed row in 20 of 20 trials. A settle delay only makes it unlikely (1/20 at 25 ms across processes). The deterministic fixes: restore records its intent (or row) before the rename, or restore holds the SQLite write lock across rename and insert while scd checks and inserts in one IMMEDIATE transaction.

  13. Refuse to start if $SC_HOME, scd’s log file or its temp dir lies under any watch root. Never write per-event lines under /etc.
    Because: Writing inside a watched tree feeds back on itself.

  14. Use one inotify instance with directory watches only, and never recurse into /usr, /var or /run. On ENOSPC from inotify_add_watch, log it loudly, move that subtree to periodic rescans, and show it in sc status.
    Because: max_user_watches (65536) and max_user_instances (128) are per uid and shared with every root process. 65536 comes from a desktop package, so a server install may differ. /etc needs about 330-400 watches.

  15. Never follow symlinks: walk with lstat semantics (filepath.WalkDir), add watches with IN_DONT_FOLLOW|IN_ONLYDIR, and record symlinks as link records.
    Because: /etc has 791 symlinks, and systemd enable/mask, localtime and alternatives are link state. /etc/xdg/systemd/user is a directory symlink. Links point into /proc, /run, /opt and /dev/null.

  16. Group package operations. Guarded apt hooks (DPkg::Pre-Invoke / DPkg::Post-Invoke, ‘if [ -x … ]; then …; fi’ plus ‘|| true’) can bracket a change set with origin=apt and snapshot pending human edits at the start. Fall back to a time window while /var/lib/dpkg/lock(-frontend) is held, with a stale-marker timeout.
    Because: One apt upgrade touches about 110 /etc entries in 10 minutes. Bare dpkg -i, dpkg triggers (libc-bin ldconfig), snapd refreshes and cron jobs bypass the apt hooks. A failing hook aborts apt.

  17. Label every name active or inert for its directory’s consumer. Record everything in scope, alert only on active names, and report activations and deactivations caused by renames.
    Because: sudoers.d parses names without ‘.’ or a trailing ‘~’ (so README is parsed). apt reads extensionless and .conf files. netplan reads non-hidden *.yaml. modprobe.d, sysctl.d, modules-load.d, tmpfiles.d and ld.so.conf.d read only *.conf; udev reads *.rules; default/grub.d reads *.cfg; run-parts requires [A-Za-z0-9_-]. A rename can be the whole edit.

  18. *Keep sc’s temp name dot-prefixed and containing a dot (‘..sc-tmp-’). Never create an extensionless, .conf or *.yaml temp name inside /etc.**
    Because: That name is ignored by sudoers.d, apt.conf.d, sources.list.d, preferences.d, netplan and every *.conf reader. An extensionless temp in apt.conf.d breaks every apt command while it exists. Exception found by verification: logrotate reads hidden files in /etc/logrotate.d (its tabooext does not cover .sc-tmp-    ), so a restore there briefly exposes a duplicate stanza. grub-mkconfig is safe because its ‘*’ glob skips dotfiles. Consider a temp suffix that logrotate’s tabooext skips.

  19. scd is static (CGO_ENABLED=0), runs as root, does not shell out on its hot path, and does no NSS lookups (numeric uid/gid only).
    Because: /etc/ld.so.preload or a bad ld.so.cache breaks every dynamically linked binary immediately. A broken nsswitch.conf makes users vanish. 34 /etc files are not world-readable.

  20. Define scd’s glob language explicitly: ‘’ is one segment, ‘**’ is any depth, braces are expanded by scd, char classes are allowed. Ship a test that runs every default exclude against a real /etc listing and asserts that no tier-1 or tier-2 path is excluded. Treat paths as opaque bytes, escape \ * ? [ when turning a literal path into a pattern, and print paths with %q.**
    Because: find’s -path ‘    ’ crosses ‘/’, while Go’s filepath.Match does not and has no ‘    ’ or braces. ‘/etc/tmp*’ would hide /etc/tmpfiles.d. 30 snap unit paths contain literal backslashes, and one cert name is UTF-8.

  21. Track ‘edited’ vs ‘applied’. Record generated outputs as markers (grub.cfg CLOSE_WRITE, /boot/initrd.img-* MOVED_TO, /etc/ld.so.cache MOVED_TO). A source newer than its output is ‘pending until ’.
    Because: Most boot files take effect only after update-grub, update-initramfs, ldconfig or daemon-reload. Unattended upgrades run these with nobody present, so ‘the machine still works after my edit’ proves nothing.

  22. Do not promise ‘who changed it’ in M2. Attribute only to sc (its own marker), apt (the dpkg lock and log window) and known generators by name.
    Because: struct inotify_event carries only wd, mask, cookie, len and name. fanotify gives a pid, but short-lived writers such as sed -i are gone by the time it is read. auditd is not installed.

  23. When validators arrive (M3), run only check-only forms on scratch copies, and never synchronously inside the event loop.
    Because: Some commands that look like checks apply changes. Safe check-only forms exist for most consumers.

5. Gaps found by the critic (to fold in)

The completeness critic checked the plan above against this VM. These are additions and corrections, not yet merged into sections 1-4.

Missing paths

Path Tier Why
/etc/dbus-1/system-local.conf, /etc/dbus-1/system.conf (both are single <include ignore_missing>), /etc/dbus-1/system.d/*.conf, /etc/dbus-1/session.d/ 1 Every system service (logind, NetworkManager, polkit, GDM, accounts-daemon) talks over the system bus. A malformed file in one of the two single includes stops dbus-daemon from starting at all, so at the next boot there is no bus (the boot effect was not verified). Files in system.d/ go live immediately with no signal, because the running bus has an inotify watch on that directory, so a valid policy cuts a service off at once. A malformed system.d file is only skipped. Not in any tier today; only the /etc/** sweep would see it.
/etc/rc.local 1 An existence flag plus the exec bit. If the file exists and is executable, it runs as root at every boot. GDM and the VT gettys are ordered after rc-local.service, and that unit has no start timeout, so a script that blocks (a foreground loop, a wait for a network that never comes) means no graphical login and no console login, forever. SSH is not ordered after it. Needs the same CREATE/MOVED_TO/ATTRIB watch on /etc as ld.so.preload and nologin, with an alert when it appears or gains +x.
/var/cache/debconf/config.dat -> key grub-pc/install_devices (read the key only, never snapshot or restore the DB) 1 This VM boots via BIOS with grub-pc. On every grub-pc upgrade (unattended upgrades included), the postinst runs grub-install on the devices named in this debconf key. A wrong or stale value (a disk that was replaced or renamed, or an empty value) updates /boot/grub/i386-pc modules without updating the boot code on the real boot disk. That is the classic ‘symbol not found’ GRUB failure at the next boot (general knowledge, not reproduced). This is boot configuration that lives outside /etc and /boot, so nothing in the plan sees it. Read the value at start and after each apt run; alert on a change or when a listed device node is missing.
/etc/depmod.d/ 1 depmod config decides which module files go into modules.dep. One valid line, ‘exclude kernel’, makes depmod index zero in-tree modules. The kernel and modules postinst run depmod on every kernel install (and dkms builds do too), so modprobe of the only NIC (e1000) and of the disk and fs modules would fail after the next kernel update, including in the initrd built from it (the boot effect was not verified). This is an initramfs-fed source like modprobe.d.
/etc/apparmor/parser.conf 1 (same level as apparmor.d/tunables) apparmor_parser reads it on every run, including the boot-time load of every profile. A valid-looking option with a bad value makes every profile fail to load, which is the same blast radius the plan gives tunables/global. Unknown options only warn.
/usr/local/lib/systemd/{system,system-generators,system-environment-generators}/, /usr/local/lib/{sysctl.d,tmpfiles.d,modules-load.d,binfmt.d,udev/rules.d}/ 1 These are admin override dirs that systemd searches. /usr/local/lib/systemd/system has higher precedence than /usr/lib/systemd/system, so a same-name unit REPLACES the vendor unit (for example systemd-journald.service). Generators there run as root at every boot and daemon-reload. The plan’s ‘/usr/**’ exclude hides all of them. npm and pip already write into /usr/local/lib on this VM. Watch /usr/local/lib and /usr/local/lib/systemd non-recursively for their creation.
/etc/apt/apt.conf, /etc/apt/preferences, /etc/apt/trusted.gpg, /etc/apt/auth.conf 3 These are apt’s four top-level files (not the .d dirs). A syntax error in apt.conf or preferences makes every apt command exit 100, like the .d dirs. trusted.gpg is the legacy keyring (adding a key trusts a repo); auth.conf holds credentials (secret). All four are absent today, so their creation must be seen; the plan lists only sources.list and the *.d dirs.
/etc/ssl/openssl.cnf 3 It is read by every OpenSSL client at start. One valid line breaks TLS for curl, wget and Python (ubuntu-pro client, software-properties, update tooling) immediately. apt’s https (GnuTLS) and snapd (Go) are unaffected. The plan excludes /etc/ssl/certs and /etc/ssl/private; this file is covered only by the optional sweep.
/etc/systemd/user/ (whole dir), /etc/systemd/{system-environment-generators,user-generators,user-environment-generators,system-preset}/ 2 /etc/systemd/user is in every user manager’s unit path above /usr/lib/systemd/user and holds the enable links that start GNOME keyring, gcr-ssh-agent, IBus and PipeWire parts. A mask or bad drop-in here applies to every user’s desktop session (not verified). The plan lists only user.conf and a snap exclude. Env generators run as root at every boot and daemon-reload. system-preset drives first-boot enablement (see the machine-id exclude). All four sibling dirs are searched by systemd but absent, so they need creation watches on /etc/systemd.
/etc/ssh/sshd_config + /etc/ssh/sshd_config.d/*.conf: Port/ListenAddress lines (correction to takes_effect) 2 On 24.04 port 22 is held by ssh.socket, not sshd. sshd-socket-generator parses sshd_config at every boot and every daemon-reload (apt triggers these) and writes ListenStream= into /run/systemd/generator/ssh.socket.d/addresses.conf. So a Port/ListenAddress change goes live at the next boot (or ssh.socket restart) regardless of when ssh.service restarts. The ‘applied’ marker is that /run file, which the plan excludes. It also combines with ufw rules into a reboot-time lockout. The plan’s ‘next ssh.service start’ is wrong for these keys.
/boot/efi/EFI/ubuntu/grub.cfg (plus shimx64.efi, grubx64.efi) on UEFI installs 1 (not present on this VM) On UEFI (most real 24.04 installs) this stub grub.cfg on the ESP holds the search.fs_uuid and configfile lines that find /boot/grub; a wrong UUID gives a grub> prompt. The ESP is a separate vfat mount, so: (1) per-file uid/gid/mode do not exist and M1’s restore chown/chmod (fsutil.go:91) would fail; (2) a watch added before /boot/efi is mounted sits on the covered directory. Not verifiable here.

Excluded but should not be (fully)

Watched but should not be (as proposed)

Missing design rules

6. Verification of key claims

Each claim was re-tested on this VM by an agent told to refute it.

Claim Verdict What is actually true
On IN_Q_OVERFLOW the kernel keeps the oldest 16384 events and silently drops the newer ones, which are exactly the final states. scd must full-rescan on overflow. partly-true What holds on this VM: the queue limit is 16384 events per inotify instance. When the queue is full, the kernel keeps the oldest events and drops the new ones. It then adds exactly one IN_Q_OVERFLOW event (wd=-1, mask=0x4000, len=0) after the 16384 kept events. Root is not exempt. The limit counts events, not bytes. scd does need to rescan on overflow, and the rescan must cover every watched path, because the marker has wd=-1 and events for a different, quiet watch are lost too (twowd test). What is wrong or needs precision: (a) The drop is not silent. The kernel sends one IN_Q_OVERFLOW event. What it does not say is which events, watches or paths were lost. (b) “Keeps the oldest 16384” is only true while the queue stays full. As soon as the reader frees space, new events are queued again after the marker (refill test: 15 after-drain events arrived after the marker). What gets lost is one continuous stretch of events, from the moment the queue fills until the reader catches up. Those are the final states only if the burst ends while the reader is still stalled, as in this probe. (c) Events are lost, not states. inotify events carry no file content, so the final content is still on disk. That is why a rescan can recover it: compare what is on disk now with the last snapshot. A file with an event before the cutoff gets its final content read when scd handles that event (alt test). The real gap is files with no surviving event at all. (d) Design consequences: on the marker, keep reading. Handle the events after the marker normally. Then rescan every watched file and snapshot any whose content, mode or owner differs from its last snapshot. Intermediate versions written during the lost stretch cannot be recovered. Also verified: many writes on one open fd merge into one MODIFY plus one CLOSE_WRITE, so they do not overflow. Alternating open/write/close cycles do not merge. Not verified on this VM: that max_queued_events is read when the instance is created, so changing the sysctl only affects new instances. I did not change sysctls.
No single event marks ‘content final’: a same-fs mv from outside the dir gives only MOVED_TO, chmod only ATTRIB, install an ATTRIB after CLOSE_WRITE, and a new symlink only CREATE. In-place writers expose a 0-byte file at the first MODIFY. partly-true The first sentence is confirmed on this VM, and the problem is broader than the claim says. Same-fs mv gives only MOVED_TO. chmod gives only ATTRIB. A new symlink gives only CREATE, and so does a new hardlink, which has full content. install gives DELETE, CREATE, OPEN, MODIFY, ATTRIB, CLOSE_WRITE, ATTRIB, and at CLOSE_WRITE the file is still mode 0600 because fchmodat(0644) comes after close. Even CLOSE_WRITE is not final: a writer that opens the file twice gives two CLOSE_WRITEs. mmap writes give no MODIFY at all. The second sentence is only partly true. Writers that open with O_TRUNC (python ‘w’, shell ‘>’, cp onto an existing file) truncate the file at open, and the first MODIFY happens while it is 0 bytes. A slow reader or a writer that pauses reliably shows size=0 there, and later MODIFYs show partial sizes (10, then 27). But: (a) In-place writers that do not truncate (open r+, write, ftruncate at the end) never show 0 bytes. Their first MODIFY shows the old size with mixed new and old content (41 bytes: “bbbb\n” plus the old A’s). That is arguably worse, because it looks like a valid file. (b) For fast O_TRUNC writers the truncate and write MODIFYs can merge into one, and the reader saw the final size=2. The 0-byte state is a race window, not a reliable observation. Accurate statement: at any MODIFY, the content may be empty, partial, or a mix of old and new bytes. Only a settle or debounce after the last event, re-checking with lstat and the content, is safe. CLOSE_WRITE alone is not enough (it misses mv, hardlink and install’s final chmod), and neither is MOVED_TO alone. Not verified on this VM: cross-filesystem mv (it would need a mount outside scratch) and writers running as root with chown.
Files created inside a brand-new directory before scd adds its watch can be missed (a timing race), so add-watch-then-readdir is required. confirmed The claim holds, with these points made precise: (a) The race is not rare and does not need load. With no deliberate load (background load average about 2), a shell mkdir-then-write missed 6–18% of files. A single process creating the dir and file back to back, like tar, cp -r or dpkg, missed 81–95%. The watcher simply cannot add the watch before the file exists. (b) The order must be add_watch first, then readdir. Scanning first and watching second still loses files: 32–52 of 200 in these runs. (c) The scan must be recursive, adding watches on subdirectories it finds (per inotify(7)). Nested directories were not tested here. (d) Files can show up both in the scan and as events, so the daemon must deduplicate. Some files also get IN_MODIFY/IN_CLOSE_WRITE without IN_CREATE, so the daemon should not rely on IN_CREATE alone. (e) “Required” applies to a per-directory inotify design. A fanotify filesystem mark (FAN_MARK_FILESYSTEM with FAN_REPORT_DFID_NAME, as root) would avoid per-directory watches altogether; that alternative is not verified on this VM. (f) How much this affects real dpkg installs into new /etc/*.d dirs is not verified on this VM. dpkg unpacks to *.dpkg-new and then renames, so the later IN_MOVED_TO may still be seen.
A per-file inotify watch dies at the first rename-over save and misses every later edit, while a directory watch sees every writer. scd must therefore watch directories only. partly-true What holds on this VM: - A per-file watch is attached to an inode. After a rename-over save (sed -i, mv, install, perl -pi, cp –remove-destination) it stays on the old inode and never sees another edit to the path. - A watch on the parent directory sees the MOVED_TO and the later writes. - So a directory watch is the right main mechanism for inotify. The claim overstates three things. 1. “Dies” is not guaranteed. - If the file has a second hardlink, DELETE_SELF never comes. The watch silently goes stale and reports writes to the other link under the watched name. - If any process still has the old inode open or mmapped, only ATTRIB arrives at the rename. DELETE_SELF and IGNORED wait until it closes; ibus holds /etc/dconf/db/ibus this way on this VM. So re-arming a watch on DELETE_SELF is unreliable. Re-arm on ATTRIB and check that the path’s inode matches instead. - Writers that edit in place keep the per-file watch alive: nano 7.2 (the Ubuntu default editor), echo », printf >, cp and tee. Vim is not installed here, so its behaviour is not verified on this VM. 2. “Sees every writer” is false. A directory watch misses: - writes made through a hardlink in another directory (/etc has 0 hardlinked files here, so low practical impact); - mmap writes: there is no MODIFY, and CLOSE_WRITE can come before the change; - every event lost to a queue overflow at 16384 events, which leaves only Q_OVERFLOW; - everything inside a subdirectory that was replaced (its watch gets IGNORED); - the first write in a newly created subdirectory (5 of 50 in the recursive test); - writes to symlink targets outside /etc (by reasoning, not event-tested). 3. “Must watch directories only” is too strong. - Directory watches are necessary but not enough. scd also needs to: - re-add the watch and rescan when a directory’s watch gets IGNORED; - scan a new subdirectory after adding its watch; - do a full rescan on Q_OVERFLOW; - reconcile hashes at startup and on a timer. - fanotify is a real alternative for a root daemon (CONFIG_FANOTIFY=y here). A mount mark caught the hardlink write that the directory watch missed. Catching renames with fanotify needs FAN_MARK_FILESYSTEM plus FAN_REPORT_DFID_NAME (kernel 5.9 or later); that part is not verified on this VM.
chmod -x on /etc/grub.d/10_linux (or on /etc/kernel/postinst.d/zz-update-grub) emits only IN_ATTRIB with no content change, yet removes every Linux menu entry, or stops grub.cfg updates, for the next kernel. ATTRIB must therefore trigger a metadata snapshot. confirmed Confirmed as stated.
sc restore renames the file about 24 ms before its ‘restore’ row is committed, so an in-process scd would record a spurious unattributed auto row unless restore announces itself first. partly-true Confirmed on this VM: restore renames the file before it commits the ‘restore’ row (store.go:341 comes before :349). A watcher that acts on the MOVED_TO event right away, whether in-process or as a separate daemon, records an extra origin=auto row with the restored content and no intent in 20/20 cases. Not exact: “about 24 ms” is roughly the median, not a constant. The gap depends on fsync latency: 9.9 to 61.7 ms (p50 19.5 ms) over 30 runs of the real binary on ext4, and 11.5 to 28 ms in single traced runs. It is made up of the directory fsync plus four SQLite commit fsyncs and the journal unlink. Overstated: “unless restore announces itself first”. Announcing is not the only way to avoid the extra row. A settle delay after the event removed it: 0/20 at 100 ms in both setups and 0/20 at 25 ms in-process, but still 1/20 at 25 ms cross-process and 3/20 at 5 ms. A delay is only probabilistic, though, because the gap reached 61.7 ms with no load. The deterministic fixes are: (a) restore records or announces its intent before the rename, or (b) restore holds the SQLite write lock (BEGIN IMMEDIATE) across the rename and the insert, while scd does its check and insert inside one IMMEDIATE transaction, so it blocks and then sees the restore row. A check-then-insert in scd alone does not fix it, because restore’s transaction has not started at event time. scd must also ignore the ..sc-tmp-* temp files, which produce CREATE, MODIFY, ATTRIB and CLOSE_WRITE events in the watched directory.
Unattended security upgrades run update-initramfs and update-grub on kernel updates with nobody present, so a dormant bad edit to /etc/default/grub, /etc/grub.d or the initramfs sources goes live overnight. partly-true What is true on this VM: unattended-upgrades runs daily with nobody present, between 06:00 and 07:00 UTC or at boot catch-up via Persistent=true. It is allowed to install kernel packages from noble-security; its own is_allowed_origin() accepts linux-generic-hwe-24.04 and linux-image-7.0.0-34-generic. A kernel install runs /etc/kernel/postinst.d, which runs update-initramfs -c -k NEWVER and update-grub. So a dormant edit to /etc/default/grub, /etc/grub.d or an initramfs source (including /etc/fstab, /etc/modprobe.d and /etc/default/keyboard) can be baked into /boot unattended. For GRUB this reaches every menu entry, including recovery and the old kernel. Corrections: (1) This has not happened here. unattended-upgrades never installed a kernel on this VM; 7.0.0-34 came from a manual apt upgrade -y by vboxuser on 2026-09-23. (2) Nothing goes live overnight. Automatic-Reboot defaults to False, so the change is only written to /boot overnight and takes effect at the next reboot, which may be days later. (3) Only edits that are still valid syntax get armed. A shell syntax error in /etc/default/grub or a grub-script error makes grub-mkconfig exit non-zero and leave grub.cfg unchanged, and the kernel postinst then fails. (4) The trigger set is wider than kernel updates. Kernel removals, which are on by default, run update-grub. A udev security update, or any package that activates the update-initramfs trigger, rebuilds the newest initrd in place with backup_initramfs=no. Some non-kernel packages (grub-pc, plymouth-theme-spinner, friendly-recovery, memtest86+) call update-grub directly, though today those come only through noble-updates, which is not an allowed origin.
sc’s temp name ‘..sc-tmp-*’ is ignored by every consumer of the watched directories (sudoers.d, apt.conf.d, sources.list.d, preferences.d, netplan, *.conf readers), whereas an extensionless temp name in apt.conf.d makes every apt command fail while it exists. partly-true First half: true for the listed consumers, which all ignore .<base>.sc-tmp-N. That covers APT apt.conf.d, sources.list.d and preferences.d, sudo @includedir, netplan’s *.yaml, and the *.conf readers tested (sshd Include, rsyslog $IncludeConfig, systemd conf.d/sysctl.d, modprobe.d). AppArmor ignores it too. But “every consumer” is not true in general. - logrotate reads hidden files in /etc/logrotate.d. When sc restores e.g. /etc/logrotate.d/rsyslog, logrotate can read .rsyslog.sc-tmp-N while it exists or if it is left behind. A full copy then gives “duplicate log entry” and exit 1, and the stanza is skipped. - Which part of the name protects the file depends on the consumer: - APT, netplan and *.conf globs are saved by the .sc-tmp-N suffix. - sudo is saved by any dot. - AppArmor is saved only by the leading dot. - logrotate is saved by neither. Its tabooext list does not cover .sc-tmp-*. Second half: overstated. An extensionless, dot-free name in apt.conf.d (for example sed -i’s sedXXXXXX) is parsed by every APT frontend, including apt-get --version and apt --help. Every one of them exits 100 only while the file’s content is not valid APT syntax, such as junk or a write cut off mid-statement. If the file is empty or holds complete valid content, APT exits 0 and silently merges it. List options then get applied twice: a copy of 70debconf makes the Pre-Install-Pkgs hook appear twice.

7. Changes needed in M1 code before M2

Found during this research and confirmed by reading the code. None of them affects M1 as used today (root runs sc by hand on root-owned files), but each must be fixed before a daemon runs sc’s code automatically.

  1. Fixed in c720f22. Read race in fsutil.ReadWithMeta. It calls os.Lstat(path), then os.Open(path) again by name, which follows symlinks, and does not compare device and inode. If the file is swapped for a symlink in between, root copies the link’s target (for example /etc/shadow) into the store; a FIFO swapped in makes the read hang. Fix: open with O_NOFOLLOW|O_NONBLOCK, compare lstat and fstat dev/inode, reject non-regular files after open.
  2. Fixed in ec15ece. Restore writes before it records. store.Restore renames the file (store.go:341) before inserting its row (store.go:349), a 10-60 ms window in which a watcher sees an unexplained change. Fix: record the intent before the rename, or hold the write lock across both (see rule 12).
  3. Tested, left as is. Temp name and logrotate. Confirmed with logrotate -d: it reads .<name>.sc-tmp-N, and also names ending in ~, .dpkg-tmp or .swp, so no suffix avoids it. Impact: one skipped duplicate stanza if logrotate runs during a restore’s few milliseconds. .<name>.sc-tmp-N is ignored by apt, sudo, netplan, systemd and other *.conf readers, but logrotate reads hidden files in /etc/logrotate.d. Consider a suffix logrotate skips.
  4. Symlinks and metadata. sc refuses symlinks, but /etc has 791 of them (systemd enable/mask links, localtime); sc also ignores immutable flags and ACLs, which the critic shows can lock people out. These are M2 scope decisions (section 8).

8. Decisions to make

  1. Scope: watch only the tier 1-3 paths listed here, or sweep all of /etc minus the exclude list (about 324 dirs, 1144 files, 2.46 MB) and use tiers only for alert priority?
  2. Tier 4: watched by default, or opt-in per path?
  3. Symlinks: add a link-record type in M2 (needed for systemd enable/mask, default.target and localtime), or leave symlinks unrecorded until later?
  4. Deletions and creations: record ‘deleted’ and ‘did not exist’ rows in M2 (restore then means delete or recreate), or record content changes only?
  5. SSH host private keys (and other secrets beyond shadow): store full content as 0600 blobs, or only hash plus mode and owner?
  6. /boot/grub/grub.cfg: keep full copies as rescue material (origin=generated), or record only its hash as the ‘applied’ marker?
  7. Outside /etc: also watch /root/.ssh, each login user’s ~/.ssh, /boot/grub and /var/spool/cron/crontabs in M2?
  8. apt grouping: ship DPkg::Pre-Invoke/Post-Invoke hooks in M2 (scd would add a file to /etc/apt/apt.conf.d), or use only dpkg-lock time-window grouping until the M5 .deb?
  9. Restore self-trigger: change store.Restore to write its row or an intent marker before the rename, or have scd match restores by hash within N seconds?
  10. Applied markers (grub.cfg, initrd, ld.so.cache) and ‘pending until update-grub’ state: build in M2, or defer to M3’s file graph?
  11. snapd-generated units and udev rules: exclude entirely, or record them with origin=snapd and never alert?
  12. Account set (passwd, shadow, group, gshadow): in M2 only group their changes into one change set, or also block single-file restore of them now?

From the critic and verification:

  1. Fix the two M1 issues in section 7 as the first commits of M2, or as a small M1 follow-up now?
  2. ~/.ssh: watch only authorized_keys (and authorized_keys2) rather than whole .ssh directories, which hold private keys?
  3. Disk budget for the store: free-space floor, per-path rate limit, retention for automatic rows?

9. Facts about this VM