tests: cover polkit rule, offline-update timer, preseed generation and ISO build

Adds T16-T20 to the Debian harness (103 passed, 0 failed, 2 skipped):

T16  the polkit rule is installed, mode 644, covers the two update actions,
     does not over-grant package-remove/reinstall/untrusted, and is idempotent.
T17  the real decision, in both directions: a sudo-group user is authorised
     (pkcheck rc=0) with the rule, and still prompted (rc=2) without it. The
     control is the point - without it the test cannot fail.
T18  vdi-offline-update: bash -n, unit verification, the throttle stamp, that
     a second run is skipped, that a stale stamp lets it run, that pkcon exit 5
     is reported as 'no updates available' rather than a download failure, and
     that /system-update is never armed without a prepared payload.
T19  the preseed still generates a script that parses, delegates to
     vdi-offline-update instead of an inline pkcon sequence, and enables cron
     and the timer.
T20  make-iso.sh: refuses to overwrite its input or an existing output, then
     checks the built image - preseed byte-identical, all 40 kernel lines
     patched, gtk.cfg patched, no doubled parameters, El Torito present with
     both BIOS and UEFI images, no source file missing, and size within 16MB of
     the original (the bloat regression).

Skips stay honest: T12 needs gpg, T13's ks.cfg assertion is vacuous because
the file was never in the repo.
This commit is contained in:
Mistercoolertyper committed 2026-11-04 10:30:30 +01:00
1 parent 15d65183dc
commit 402fea66fd
2 files changed
+282

No files matched your search

+81
View File
@@ -39,6 +39,7 @@ not a pass**, and the summary says so.
| T17 | A real sudo-group user is authorised **without a password** (pkcheck), with a control |
| T18 | Offline-update script/units: bash -n, `systemd-analyze verify`, timer enabled, throttle, no dangling `/system-update` |
| T19 | The preseed still generates a working `/root/vdi-post.sh` that delegates updates |
| T20 | `make-iso.sh` builds a bootable ISO with the preseed baked in (skips without a netinst ISO) |
| T15 | Nothing was left behind: no source file, no cached indexes, no installs, no `__pycache__`, no polkit rule or unit residue |
### The repo build it tests
@@ -117,6 +118,12 @@ Being explicit matters more than the green summary.
`pkcon refresh` uses) and that `package-remove` is *not* granted. Installing a
package from a desktop session without a password is not exercised.
11. **Actually booting the built ISO.** T20 proves the image is structurally
correct and the preseed is on it, but it never boots it in a VM. Whether the
installer runs unattended end to end, and whether the inventory prompt behaves,
is untested. `xorriso` says the boot record is intact; only a real (virtual)
machine says the firmware agrees.
## Two spec'd checks that cannot fail, and what replaced them
This is the most important thing in this file. Two checks that look like tests
@@ -322,6 +329,75 @@ before the run, so "it did not exist and must be gone" is distinguishable from
symlink - it never recurses into a directory. The same restores run in the
`EXIT` trap, so an aborted run still cleans up.
## T20: the ISO builder
T20 needs a real Debian netinst ISO and **does not download one** - a test that
fetches 792 MB before it can start is a test nobody runs. It looks for
`/root/trixie-netinst.iso` (or `NETINST_ISO=/path`), and without one it skips
and lists what it would have checked.
With an ISO present it runs `make-iso.sh` into a temp directory (about 3 seconds)
and asserts:
- It **refuses** to overwrite an existing output, and refuses to write over its
own input. Both are tested by expecting a non-zero exit, not by reading the code.
- `/preseed.cfg` on the output is byte-identical to `debian/preseed.cfg`.
Extracted to a real file first - `-extract ... /dev/stdout` does not work.
- **Every** kernel line under `/isolinux` and `/boot/grub` carries
`preseed/file=/cdrom/preseed.cfg`. Counted, not sampled: on 13.7.0 it is 40
lines (20 BIOS + 20 UEFI), and the assertion is `patched == total`. A
`> 0` check would pass while half the entries still prompt.
- `isolinux/gtk.cfg` specifically is patched. That file holds the BIOS *default*
entry; `isolinux.cfg` is only `include` and `default vesamenu.c32`, so patching
the obvious files leaves the default boot prompting. That is the bug this test
exists to catch, and it is asserted by name rather than left implicit.
- No kernel line carries the parameter **twice** (a re-run of the patch, or a
`-map` applied on top of an already-patched file).
- Both a BIOS and a UEFI El Torito image are present.
- No file from the source ISO is missing from the output (1010 files on 13.7.0).
- Output size is within 16 MB of the source. The measured delta is -704 KB. The
naive extract-and-rebuild implementation bloats 792 MB -> 988 MB because
Debian's ISO shares identical file extents between paths; this guards that
regression.
### `grep -c` exits 1 on zero, and that will corrupt your count
While writing T20 this produced a wrong total: `grep -c ... || echo 0` prints
`0` **and then** appends another `0` when there are no matches, because `grep -c`
prints the count and *also* exits 1. The result is `0\n0`, and `$((total + 0\n0))`
is a syntax error. The harness swallows the exit status only (`|| true`) and
validates the value with a `case` before doing arithmetic.
## Two corrections that are now asserted
**`subject.local` must not come back into the polkit rule.** An earlier version
of the rule included `subject.local == true` to restrict it to a seat physically
in front of the machine. It never matches: `pkcheck` returns 2 with the clause
and 0 without it, for the same user. If it returns, passwordless updates stop
working while every other T16 assertion still passes - so T16 now asserts the
clause is **absent**, with the reason in a comment.
**"the timer is enabled" is not a stable assertion on its own.** Enabling is done
by `systemctl enable vdi-offline-update.timer` inside
`install_debian_update_policy()`, and that only persists when systemd is the
running init. T18 therefore drives the installer function first, then asserts,
and **skips** the assertion with a reason when systemd is not PID 1.
## A killed run leaves apt permanently broken, and the lock cannot help
The harness removes its own apt source in an `EXIT` trap. A run that is killed
(the turn is cancelled, the terminal is closed, `SIGKILL`) never reaches that
trap, so `/etc/apt/sources.list.d/zz-vdi-test-<pid>.list` survives pointing at a
temp directory that no longer exists. Every subsequent `apt-get update` then
fails with `Failed to fetch ... File not found`, and T06/T07 report that as a
port defect. This was observed: a leftover `zz-vdi-test-250.list` from another
run broke three T06 assertions while the port was fine.
The `flock` does not prevent it, because there is no concurrent process - just
residue. So at startup, under the lock, the harness sweeps any
`zz-vdi-test-*.list` whose target directory is gone, and says so. That is safe
precisely because the lock guarantees no peer run is alive at that moment.
## Requirements
- Debian 13 (trixie), root.
@@ -333,6 +409,11 @@ symlink - it never recurses into a directory. The same restores run in the
machine, which is why the harness treats its absence as a skip rather than
installing it itself.
- `gpg` **for T12 only**, and only if you want to re-check the signing decision.
- `xorriso` **for T20 only**. Without it T20 skips with a reason.
- A Debian netinst ISO **for T20 only**, at `/root/trixie-netinst.iso` or
`NETINST_ISO=/path/to/netinst.iso`. It is never downloaded. The build itself
takes about 3 seconds and needs ~1.6 GB free in `TMPDIR` (source plus output;
the source is not copied, only written to the output).
## Safety
+201
View File
@@ -236,6 +236,26 @@ for tool in apt-ftparchive apt-get apt-cache dpkg-deb python3 gzip; do
done
# Clear indexes from any earlier run before doing anything else.
#
# A run that was killed (SIGKILL, a cancelled turn, a closed terminal) never
# reaches its EXIT trap, so its /etc/apt/sources.list.d/zz-vdi-test-<pid>.list
# survives pointing at a temp directory that no longer exists. Every later
# `apt-get update` then fails with "Failed to fetch ... File not found" and
# T06/T07 blame the port for the wreckage - measured: exactly that happened
# with a leftover zz-vdi-test-250.list from another run.
#
# This is safe to sweep because flock above guarantees no other run is alive,
# so any zz-vdi-test-* source file at this point is residue, not a peer.
for stale_list in /etc/apt/sources.list.d/zz-vdi-test-*.list; do
[ -e "$stale_list" ] || continue
stale_dir=$(sed -n 's|.*file:\([^ ]*\).*|\1|p' "$stale_list" 2>/dev/null)
if [ -z "$stale_dir" ] || [ ! -d "$stale_dir" ]; then
rm -f "$stale_list"
printf ' removed stale apt source from a killed run: %s\n' \
"$(basename "$stale_list")"
fi
done
purge_all_harness_lists
CODENAME=""
@@ -1546,6 +1566,187 @@ PYEOF
fi
fi
# ---------------------------------------------------------------------------
# T20: make-iso.sh produces a bootable, correctly-preseeded ISO
# ---------------------------------------------------------------------------
# Needs a real Debian netinst ISO. It is NOT downloaded here: a test that
# fetches 792 MB from the network is a test nobody runs. Without the ISO this
# skips and says what it would have checked.
head2 "T20 make-iso.sh builds a preseeded bootable ISO"
MAKE_ISO="$REPO_ROOT/debian/make-iso.sh"
NETINST=""
for candidate in "${NETINST_ISO:-}" /root/trixie-netinst.iso /root/debian-netinst.iso; do
[ -n "$candidate" ] && [ -f "$candidate" ] && { NETINST="$candidate"; break; }
done
if [ ! -f "$MAKE_ISO" ]; then
skip "T20 no debian/make-iso.sh in this repo (owned by another change)"
elif ! command -v xorriso >/dev/null 2>&1; then
skip "T20 xorriso is not installed - make-iso.sh cannot run"
info "apt-get install xorriso"
elif [ -z "$NETINST" ]; then
skip "T20 no Debian netinst ISO found (looked for /root/trixie-netinst.iso)"
info "would have checked: preseed on the image is byte-identical, every kernel"
info "line carries preseed/file=, gtk.cfg is patched, both El Torito images"
info "exist, no source file is missing, size is within a few MB of the source."
info "Point NETINST_ISO=/path/to/netinst.iso at one to run it."
else
info "source ISO: $NETINST ($(du -h "$NETINST" | cut -f1))"
if bash -n "$MAKE_ISO" 2>"$TMP_DIR/isoerr"; then
pass "T20 make-iso.sh passes bash -n"
else
fail "T20 make-iso.sh fails bash -n: $(cat "$TMP_DIR/isoerr" | tr '\n' ' ')"
fi
ISOWORK="$TMP_DIR/iso"
mkdir -p "$ISOWORK"
OUT_ISO="$ISOWORK/out.iso"
# Guard: it must refuse to overwrite an existing output, and refuse to
# write over its own input. Both are destructive if they get it wrong.
: > "$OUT_ISO"
if bash "$MAKE_ISO" "$NETINST" "$OUT_ISO" >/dev/null 2>&1; then
fail "T20 make-iso.sh overwrote an existing output file (should refuse)"
else
pass "T20 make-iso.sh refuses to overwrite an existing output"
fi
rm -f "$OUT_ISO"
if bash "$MAKE_ISO" "$NETINST" "$NETINST" >/dev/null 2>&1; then
fail "T20 make-iso.sh accepted output == input (would destroy the source)"
else
pass "T20 make-iso.sh refuses to write over its input ISO"
fi
# The real build.
build_start=$(date +%s)
if build_out=$(bash "$MAKE_ISO" "$NETINST" "$OUT_ISO" 2>&1); then
build_rc=0
else
build_rc=$?
fi
build_secs=$(( $(date +%s) - build_start ))
if [ "$build_rc" -eq 0 ] && [ -s "$OUT_ISO" ]; then
pass "T20 make-iso.sh built an ISO (${build_secs}s, $(du -h "$OUT_ISO" | cut -f1))"
else
fail "T20 make-iso.sh failed (rc=$build_rc)"
printf '%s\n' "$build_out" | tail -12 | sed 's/^/ /'
fi
if [ -s "$OUT_ISO" ]; then
# ---- preseed is on the image and byte-identical ----
# Note: `-extract /preseed.cfg /dev/stdout` does not work; extract to a
# real file.
if xorriso -osirrox on -indev "$OUT_ISO" -extract /preseed.cfg \
"$ISOWORK/preseed.on.iso" >/dev/null 2>&1; then
if cmp -s "$ISOWORK/preseed.on.iso" "$REPO_ROOT/debian/preseed.cfg"; then
pass "T20 /preseed.cfg on the ISO is byte-identical to debian/preseed.cfg"
else
fail "T20 /preseed.cfg on the ISO differs from debian/preseed.cfg"
fi
else
fail "T20 /preseed.cfg is not present on the output ISO"
fi
# ---- every kernel line is patched, counted not sampled ----
menu_list=$( for dir in /isolinux /boot/grub; do
xorriso -indev "$NETINST" -find "$dir" -name '*.cfg' 2>/dev/null
done | tr -d "'" | grep -v '^$' | sort -u )
total_k=0; patched_k=0; unpatched_list=""; dup_list=""
for menu in $menu_list; do
local_file="$ISOWORK/menu_$(printf '%s' "$menu" | tr '/' '_')"
xorriso -osirrox on -indev "$OUT_ISO" -extract "/$menu" "$local_file" \
>/dev/null 2>&1 || continue
# grep -c prints 0 AND exits 1 when there are no matches, so
# `|| echo 0` would append a second zero. Swallow the status only.
k=$(grep -cE '^[[:space:]]*(append|linux)[[:space:]]' "$local_file" 2>/dev/null || true)
case "$k" in ''|*[!0-9]*) k=0 ;; esac
p=$(grep -E '^[[:space:]]*(append|linux)[[:space:]]' "$local_file" 2>/dev/null \
| grep -cF 'preseed/file=/cdrom/preseed.cfg' || true)
case "$p" in ''|*[!0-9]*) p=0 ;; esac
total_k=$((total_k + k))
patched_k=$((patched_k + p))
[ "$k" -gt 0 ] && [ "$p" -ne "$k" ] && unpatched_list="$unpatched_list $menu($p/$k)"
# The parameter must appear at most once per kernel line.
dups=$(grep -E '^[[:space:]]*(append|linux)[[:space:]]' "$local_file" 2>/dev/null \
| awk '{n=gsub(/preseed\/file=/,"&"); if (n>1) print}' | wc -l)
[ "$dups" -gt 0 ] && dup_list="$dup_list $menu"
done
if [ "$total_k" -gt 0 ] && [ "$patched_k" -eq "$total_k" ]; then
pass "T20 all $total_k kernel lines carry preseed/file= ($patched_k patched)"
else
fail "T20 only $patched_k of $total_k kernel lines are patched"
[ -n "$unpatched_list" ] && info "unpatched:$unpatched_list"
fi
# The BIOS *default* entry lives in gtk.cfg, not isolinux.cfg. Patching
# only the obvious files leaves the machine prompting, which is the bug
# this test exists to catch.
gtk_file="$ISOWORK/menu__isolinux_gtk.cfg"
if [ -f "$gtk_file" ]; then
gtk_k=$(grep -cE '^[[:space:]]*(append|linux)[[:space:]]' "$gtk_file" 2>/dev/null || true)
case "$gtk_k" in ''|*[!0-9]*) gtk_k=0 ;; esac
if [ "$gtk_k" -gt 0 ] && grep -qE '^[[:space:]]*(append|linux)[[:space:]].*preseed/file=' "$gtk_file"; then
pass "T20 isolinux/gtk.cfg is patched (the BIOS default entry)"
else
fail "T20 isolinux/gtk.cfg is NOT patched - the BIOS default would still prompt"
fi
else
info "T20 isolinux/gtk.cfg not present on this ISO (layout differs)"
fi
if [ -z "$dup_list" ]; then
pass "T20 no kernel line carries the parameter twice"
else
fail "T20 duplicated parameter in:$dup_list"
fi
# ---- both El Torito images survive ----
torito=$(xorriso -indev "$OUT_ISO" -report_el_torito plain 2>&1)
if printf '%s' "$torito" | grep -q 'El Torito'; then
pass "T20 output ISO has an El Torito boot record"
else
fail "T20 output ISO has no El Torito boot record"
fi
if printf '%s' "$torito" | grep -q 'BIOS' && printf '%s' "$torito" | grep -q 'UEFI'; then
pass "T20 both BIOS and UEFI boot images are present"
else
fail "T20 only one firmware type is bootable in the output ISO"
fi
# ---- no source file is missing ----
xorriso -indev "$NETINST" -find / -type f -exec lsdl 2>/dev/null \
| awk '{print $NF}' | sort > "$ISOWORK/src.list"
xorriso -indev "$OUT_ISO" -find / -type f -exec lsdl 2>/dev/null \
| awk '{print $NF}' | sort > "$ISOWORK/out.list"
missing_n=$(comm -23 "$ISOWORK/src.list" "$ISOWORK/out.list" | wc -l)
src_n=$(wc -l < "$ISOWORK/src.list")
if [ "$src_n" -gt 0 ] && [ "$missing_n" -eq 0 ]; then
pass "T20 all $src_n source files are present in the output"
else
fail "T20 $missing_n of $src_n source files are missing from the output"
comm -23 "$ISOWORK/src.list" "$ISOWORK/out.list" | head -5 | sed 's/^/ /'
fi
# ---- size guard: the extract-and-rebuild approach bloated 792->988 MB ----
src_bytes=$(stat -c '%s' "$NETINST")
out_bytes=$(stat -c '%s' "$OUT_ISO")
delta=$(( out_bytes - src_bytes ))
if [ "$delta" -lt 0 ]; then abs_delta=$(( -delta )); else abs_delta=$delta; fi
if [ "$abs_delta" -le 16777216 ]; then
pass "T20 output size is within 16 MB of the source (delta ${delta} bytes)"
else
fail "T20 output size differs by $delta bytes - the extent-dedup regression is back"
fi
fi
fi
# ---------------------------------------------------------------------------
# T15: the harness left nothing behind
# ---------------------------------------------------------------------------