T17's package-remove assertion could not fire: it used a sleeper that had already exited, so pkcheck returned 127 and the check printed 'info' rather than a verdict. A negative assertion that cannot fail is worse than none. It now owns a sleeper (setsid + exec -a, pid confirmed via /proc before pkcheck is trusted), and 127 or any unexpected code is a FAIL rather than a pass. Verified capable of failing by injecting package-remove into the rule: pkcheck went rc=2 -> rc=0 and the assertion went red; the rule was then restored (sha256 unchanged). T21 asserts both scripts are recorded as 100755 in the index. core.fileMode=false means a Windows checkout silently drops the executable bit, so a fresh Linux clone got non-executable files. T19 now reads update_local_repo.OFFLINE_UPDATE_SCRIPT and asserts the preseed names the same path. Both files hard-code it, and nothing tied them together, so a rename would leave auto_update.sh printing 'not installed yet' with everything else green. Final: 107 passed, 0 failed, 2 skipped. The skips stay honest (T12 needs gpg; T13's ks.cfg assertion is vacuous because the file is not in the repo).
25 KiB
Debian port test harness
run-tests.sh proves that the Debian side of the VDI-Linux port actually works,
and that the Fedora side has not regressed. It is a real harness: it builds a
real apt repository from the real Citrix .deb packages, adds a real apt
source, and asks real apt whether it can install the result. Nothing is
simulated except the parts that would change the machine.
sudo bash debian/test/run-tests.sh
Run as root. It needs the two Citrix .deb files (see Fixtures).
Exit status is 0 when no test FAILED, 1 otherwise. A SKIP does not fail the run, but every skip prints why it skipped and what it would have checked. A skip is not a pass, and the summary says so.
What it tests
| Test | What it proves |
|---|---|
| T00 | Environment: root, required tools, trixie, fixtures present |
| T01 | The .deb fixtures really are version 26.08.0.225 |
| T02 | python3 -m py_compile passes for update_local_repo.py and configure_kde.py |
| T03 | Both modules import cleanly on Debian (no tqdm, no pip) |
| T04 | The tqdm fallback is usable, not merely importable |
| T05 | The flat repo builds; Packages has exactly 2 stanzas at the right version |
| T06 | apt-get update exits 0 with no E:, no Err: and no warnings for the source |
| T07 | apt can see the package: correct candidate, correct origin, install resolves |
| T08 | Rebuilding twice is byte-identical and produces no duplicate stanzas |
| T09 | sync_debian prunes stale .deb files (version bump cannot leave two versions) |
| T10 | A corrupted index is rejected even with trusted=yes (control for T06/T07) |
| T11 | The production local-citrix-repo.list has the shape the harness verified |
| T12 | GPG signed-by path (design check; skips without a signing tool) |
| T13 | Fedora path unregressed: tracked files identical, sync_fedora command sequence identical |
| T14 | bash -n on every shell script the port adds |
| T16 | polkit rule installed, mode 644, correct actions, no over-grant, balanced, idempotent |
| T17 | A real sudo-group user is authorised without a password (pkcheck), with a control |
| T18 | Offline-update script/units: bash -n, systemd-analyze verify, timer enabled, throttle, no dangling /system-update |
| T19 | The preseed still generates a working /root/vdi-post.sh that delegates updates |
| T20 | make-iso.sh builds a bootable ISO with the preseed baked in (skips without a netinst ISO) |
| T21 | The executable bit is recorded as 100755 in git, not just on disk |
| T15 | Nothing was left behind: no source file, no cached indexes, no installs, no __pycache__, no polkit rule or unit residue |
The repo build it tests
The exact production sequence, on a copy of the real .deb files in a temp
directory:
cd "$repo"
apt-ftparchive packages . > Packages
gzip -c Packages > Packages.gz
apt-ftparchive release . > Release
and the exact production source line:
deb [trusted=yes] file:/var/local/citrix-repo ./
with the path substituted for the temp directory. A flat ("trivial") archive:
no dists/ tree, suite ./.
What it does NOT test, and cannot
Being explicit matters more than the green summary.
-
A real d-i install. The harness never boots an installer.
debian/preseed.cfgis owned and validated by another change; this harness does not test it at all. Note thatdebconf-set-selectionsvalidates line shape only - it silently accepts an invented question name - so it is not evidence that directives exist. -
An actual GUI session. Nothing here starts Plasma, and no test asserts that
configure_kde.pyproduces a working desktop. T03 proves it imports; whether the touchpad and power settings take effect on a real session is untested.kwriteconfig6is not installed here, soconfigure_kde.KWRITEisNoneand every write would no-op. That path needs a real Plasma machine. -
The Citrix preinst/postinst maintainer scripts.
apt-get install --simulateresolves dependencies; it does not run the package's scripts. Whethericaclient's postinst succeeds on a real machine is untested. This is the single biggest gap: a dependency-resolvable package can still fail to configure. -
A real d-i/target filesystem for the repo path. The harness builds the repo in a temp directory, never at
/var/local/citrix-repo. It asserts the shape is identical (T11), not that the production path is writable in a target environment. -
The network path. The Gitea release API and the download code are not exercised. T09 redirects the source files to a local directory, so
download_fileis only exercised indirectly. A changed release asset name would not be caught here. -
apt-get updateinside a d-i chroot. The preseed'slate_commandruns in anin-targetcontext where apt behaves differently (no_aptuser, different permissions). Untested. -
Fedora at runtime. T13 compares source and command sequences; it does not run
createrepo_c/dnf/rpmon Fedora. The command sequence matching is strong evidence but it is not an execution test. A Fedora 44 WSL distro exists on the development machine and was not used. -
A real desktop session. T17 proves polkit authorises a sudo-group user without a password, which is the actual requirement, but it does not open a GUI and click an update.
pkcon refreshfrom a live Plasma session, with an authentication agent running, is not exercised. -
An actual offline update reboot. T18 checks that
/system-updateis armed with an existing target and thatsystemd-analyze verifyis clean. It does not reboot and letpackagekit-offline-update.serviceapply anything. Whether the update actually installs at the next boot is untested - only that it was correctly prepared. -
package-installend to end. The polkit rule also grantspackage-install, but T17 only checkssystem-sources-refresh(the actionpkcon refreshuses) and thatpackage-removeis not granted. Installing a package from a desktop session without a password is not exercised. -
Actually booting the built ISO. T20 proves the image is structurally correct and the preseed is on it, but it never boots it in a VM. Whether the installer runs unattended end to end, and whether the inventory prompt behaves, is untested.
xorrisosays the boot record is intact; only a real (virtual) machine says the firmware agrees.
Two spec'd checks that cannot fail, and what replaced them
This is the most important thing in this file. Two checks that look like tests are not tests.
apt-get update exiting 0 with no E: lines
Measured on Debian 13: a repo whose Packages is unreadable produces
Err:4 file:/tmp/.../repo ./ Packages
Could not open file .../Packages - open (13: Permission denied)
and apt-get update still exits 0 with no E: lines. The same is true of a
repo with no Release file at all, and of a repo with duplicate stanzas.
So "exit 0 and no E:" passes on a broken repo.
The harness runs the spec'd check as T06 and the stronger check separately: no
Err: lines, no W: lines for that source, and then T07 requires apt to name the
exact version from that exact directory. T10 is the control that proves the
strong checks can fail: it corrupts Packages after Release was built and
asserts apt rejects it with a hash mismatch. Without T10, T06/T07 could be
passing vacuously.
git diff main -- ks.cfg
ks.cfg is not in the repository. It is not in main, not in debian-port,
and git log --all -- ks.cfg is empty. It lives one level above the repo, in the
parent directory, untracked.
So git diff main -- ks.cfg exits 0 and prints nothing - because the path does
not exist on either side, not because the file is unchanged. The harness detects
this with git cat-file -e main:ks.cfg and reports it as SKIP, explicitly
labelled VACUOUS, rather than as a pass. It then prints the sha256 of the real
out-of-tree file so a change to it is at least visible.
Line endings will fool you
This repo is checked out on Windows. Every tracked file in the working tree has
CRLF line endings; every blob in git has LF. A plain git diff main therefore
reports all five files as completely rewritten, including
RPM-GPG-KEY-citrix-local with 104 changed lines - the key itself is byte-identical.
The harness compares modulo CRLF (tr -d '\r' | cmp). If you see a huge diff,
check line endings before believing it.
The apt index cache will fool you
apt-cache policy and apt-get update read cached indexes in
/var/lib/apt/lists. During development, a wrong GPG key appeared to be
accepted - exit 0, no E:, package still visible - purely because a good index
was cached from an earlier run.
Every case that must fail purges that repo's cached index files first
(purge_apt_lists_for). Any test that checks "can apt see the package" without
purging is testing the cache, not apt.
Related: /tmp is wiped between separate wsl invocations on this machine
(systemd-tmpfiles). Anything expected to survive across invocations must live
in /root.
GPG signing: why trusted=yes is the right call
Production does not sign the Debian repo, and T12 exists to check that decision
rather than assume it. A stock trixie ships no gpg - only /usr/bin/sqv,
the Sequoia verifier that apt calls internally. So signing would add a gpg
dependency that Debian 13 does not have by default, in an installer environment
where new dependencies are the fragile thing.
The signing path was measured out of band (a throwaway key generated with Fedora
WSL's gpg, because Debian has none), with the apt cache purged between cases:
| Keyring | Result |
|---|---|
armored .asc, correct key |
exit 0, no E:, package visible |
armored .asc, wrong key |
exit 100, E:, Missing key, not visible |
binary .gpg, correct key |
exit 0, no E:, package visible |
binary .gpg, wrong key |
exit 100, E:, Missing key, not visible |
trusted=yes, no signature |
exit 0, no E:, package visible |
Two conclusions:
signed-byworks correctly on a flatfile:repo on trixie, for both keyring formats. It is not broken; it is simply not worth it here.trusted=yesdoes not mean "no integrity checking". T10 shows a tamperedPackagesis rejected withE: ... Hash Sum mismatchregardless. Signing would only add proof of authorship, which is meaningless for a repository the machine builds from packages it just downloaded and verified itself.
T12 skips loudly when no signing tool is present, which is the normal case on a
stock trixie. Install gpg to make it run.
Fixtures
The two real packages are expected next to the repo, not inside it:
LinuxISO/
icaclient_26.08.0.225_amd64.deb (256 MB)
ctxusb_26.08.0.225_amd64.deb (178 KB)
VDI-Linux/ <- the repo
debian/test/run-tests.sh <- this harness
Override with DEB_DIR=/path/to/debs sudo -E bash debian/test/run-tests.sh.
The 256 MB file is copied into a temp directory, which is fast on native
filesystem and slow off /mnt/c. If /tmp is small, set TMPDIR to somewhere
with ~600 MB free.
T17 is the test that matters, and why it needs a control
The user's requirement is that a desktop user can update without an admin
password. Asserting that the rule file contains the right action names proves
nothing about whether polkit acts on it - a rule can be syntactically perfect,
in the right directory, and still never match (which is exactly what happened
during development with a subject.local == true clause).
So T17 asks polkit itself:
- Start a process owned by a real member of group
sudo(setsid, so nothing blocks), thenpkcheck --action-id ...system-sources-refresh --process <pid>as root. Assert rc=0. - The control: move the rule away, restart polkit, repeat. Assert rc=2 (authentication required). Then restore.
Without step 2 the test cannot fail: a polkit configured to answer YES to everything would also give rc=0. The control is what makes step 1 evidence.
It also asserts the rule does not grant package-remove, package-reinstall
or package-install-untrusted - a user who can remove packages without a
password can break the machine.
That refusal is checked three ways, because a negative assertion is easy to write in a way that cannot fail:
- The rule text must not mention the action (T16).
- polkit itself must refuse it: T17 starts its own detached sleeper owned by
the sudo-group user, proves
/proc/<pid>is owned by that user, then askspkcheck --action-id ...package-remove --process <pid>and requires rc=2. A missing process or any unexpected rc is a FAIL, never a pass - a 127 must not be able to masquerade as "denied". - The check was proven able to fail. With
package-removeadded to the rule by hand and polkit restarted,pkcheckreturned rc=0 (granted), so assertion 2 goes red. Restored afterwards and verified by sha256. Without this step the assertion would only be known to pass, not known to be capable of failing.
pkcheck is only conclusive because it consults the live polkitd. If polkit is
not installed, T16 and T17 skip with a reason rather than passing quietly.
T18: the dangerous failure mode
A dangling /system-update symlink fails the next boot. The offline-update
script arms that symlink, so the harness asserts, after every run, that if
/system-update exists then its readlink -f target exists too. It is checked
both immediately after the run and again after the forced (stamp=0) run, which
is the one most likely to arm it.
Also asserted, because it is a real behaviour and not a detail:
- Running the script twice in a row is throttled (prints "skipping").
- Setting the stamp to
0defeats the throttle - so it is time-based, not a permanent latch. pkconexit 5 means "no packages to update" and must not be reported asdownload failed. On an up-to-date box the output must not contain "download failed".
T19: why it never executes the generated script
The preseed's late_command writes /root/vdi-post.sh and then runs it with
in-target. That script writes /etc/hostname, /etc/hosts, /etc/crontab
and systemd units by absolute path - running it here would reconfigure the
machine the tests are running on.
T19 therefore only regenerates the script by running the preseed's own
printf into a temp file, then checks it: bash -n, that it delegates to
/usr/local/sbin/vdi-offline-update instead of duplicating the pkcon
sequence (comments stripped first, so the explanatory comment does not produce
a false positive), and that it still enables cron and
vdi-offline-update.timer.
Extracting it is fiddly, and the naive approaches are all wrong in ways that look like preseed bugs:
- A regex that crosses
;captures the firefoxprintftoo, which then tries to write/target/etc/firefoxand fails. - Splitting on
;breaks, because the payload contains;and>inside quoted arguments. - Leaving the leading
\of the first continuation line makessh -cskip theprintfentirely, producing a "No such file" that looks exactly like a preseed defect.
The harness anchors on two unambiguous markers instead: the payload's own first
argument '#!/bin/bash' and the > /target/root/vdi-post.sh redirect. It also
asserts the captured text does not contain ManagedBookmarks, so a future
change to the preseed fails loudly rather than silently testing the wrong
block.
Tests that mutate the machine, and how that is undone
T16-T19 install a polkit rule and systemd units and can arm /system-update.
T15 asserts no collateral damage, so every mutated path is backed up before
the test and restored at teardown:
| Path | How it is handled |
|---|---|
/etc/polkit-1/rules.d/60-vdi-no-password-updates.rules |
backed up, restored |
/usr/local/sbin/vdi-offline-update |
backed up, restored |
/etc/systemd/system/vdi-offline-update.{service,timer} |
backed up, restored |
/var/lib/vdi-offline-update.last |
backed up, restored |
/system-update |
backed up as a symlink target, restored; never rm -rf |
restore_path records whether a path was present, absent or a symlink
before the run, so "it did not exist and must be gone" is distinguishable from
"the backup failed". For an absent path it removes only a regular file or
symlink - it never recurses into a directory. The same restores run in the
EXIT trap, so an aborted run still cleans up.
T20: the ISO builder
T20 needs a real Debian netinst ISO and does not download one - a test that
fetches 792 MB before it can start is a test nobody runs. It looks for
/root/trixie-netinst.iso (or NETINST_ISO=/path), and without one it skips
and lists what it would have checked.
With an ISO present it runs make-iso.sh into a temp directory (about 3 seconds)
and asserts:
- It refuses to overwrite an existing output, and refuses to write over its own input. Both are tested by expecting a non-zero exit, not by reading the code.
/preseed.cfgon the output is byte-identical todebian/preseed.cfg. Extracted to a real file first --extract ... /dev/stdoutdoes not work.- Every kernel line under
/isolinuxand/boot/grubcarriespreseed/file=/cdrom/preseed.cfg. Counted, not sampled: on 13.7.0 it is 40 lines (20 BIOS + 20 UEFI), and the assertion ispatched == total. A> 0check would pass while half the entries still prompt. isolinux/gtk.cfgspecifically is patched. That file holds the BIOS default entry;isolinux.cfgis onlyincludeanddefault vesamenu.c32, so patching the obvious files leaves the default boot prompting. That is the bug this test exists to catch, and it is asserted by name rather than left implicit.- No kernel line carries the parameter twice (a re-run of the patch, or a
-mapapplied on top of an already-patched file). - Both a BIOS and a UEFI El Torito image are present.
- No file from the source ISO is missing from the output (1010 files on 13.7.0).
- Output size is within 16 MB of the source. The measured delta is -704 KB. The naive extract-and-rebuild implementation bloats 792 MB -> 988 MB because Debian's ISO shares identical file extents between paths; this guards that regression.
grep -c exits 1 on zero, and that will corrupt your count
While writing T20 this produced a wrong total: grep -c ... || echo 0 prints
0 and then appends another 0 when there are no matches, because grep -c
prints the count and also exits 1. The result is 0\n0, and $((total + 0\n0))
is a syntax error. The harness swallows the exit status only (|| true) and
validates the value with a case before doing arithmetic.
Two corrections that are now asserted
subject.local must not come back into the polkit rule. An earlier version
of the rule included subject.local == true to restrict it to a seat physically
in front of the machine. It never matches: pkcheck returns 2 with the clause
and 0 without it, for the same user. If it returns, passwordless updates stop
working while every other T16 assertion still passes - so T16 now asserts the
clause is absent, with the reason in a comment.
"the timer is enabled" is not a stable assertion on its own. Enabling is done
by systemctl enable vdi-offline-update.timer inside
install_debian_update_policy(), and that only persists when systemd is the
running init. T18 therefore drives the installer function first, then asserts,
and skips the assertion with a reason when systemd is not PID 1.
A killed run leaves apt permanently broken, and the lock cannot help
The harness removes its own apt source in an EXIT trap. A run that is killed
(the turn is cancelled, the terminal is closed, SIGKILL) never reaches that
trap, so /etc/apt/sources.list.d/zz-vdi-test-<pid>.list survives pointing at a
temp directory that no longer exists. Every subsequent apt-get update then
fails with Failed to fetch ... File not found, and T06/T07 report that as a
port defect. This was observed: a leftover zz-vdi-test-250.list from another
run broke three T06 assertions while the port was fine.
The flock does not prevent it, because there is no concurrent process - just
residue. So at startup, under the lock, the harness sweeps any
zz-vdi-test-*.list whose target directory is gone, and says so. That is safe
precisely because the lock guarantees no peer run is alive at that moment.
T21: executable on disk is not executable in a clone
This repo is developed on a Windows checkout with core.fileMode=false, so git
records mode 100644 regardless of the mode on disk. Both scripts were
executable locally and non-executable in the index - a fresh Linux clone gets
Permission denied running ./debian/test/run-tests.sh.
T21 asserts git ls-files -s reports 100755 for debian/test/run-tests.sh
and debian/make-iso.sh. It skips honestly when git is unavailable or the
directory is not a checkout. Fix a failure with:
git update-index --chmod=+x <path>
T19 cross-checks the preseed against the installer
T19 asserts the generated auto_update.sh delegates to
/usr/local/sbin/vdi-offline-update. That string is hard-coded in the preseed,
while the installer of that path lives in update_local_repo.py - two
independent files with nothing tying them together. If the installer's path ever
changed, the preseed would keep pointing at the old one and auto_update.sh
would degrade to printing vdi-offline-update not installed yet:
gracefully, silently, with every other assertion still green.
So T19 reads update_local_repo.OFFLINE_UPDATE_SCRIPT and asserts the generated
script names that same path. It skips if that constant does not exist.
What T19 still does not prove: that the helper exists and runs on a real
target. It is installed by update_local_repo.py at first cron run, so on a
fresh install auto_update.sh may legitimately find it absent. That ordering is
by design, not a defect, but it means T19 is evidence about the contract
between the two files, not about the end state of an installed machine.
Requirements
- Debian 13 (trixie), root.
apt-ftparchive,apt-get,apt-cache,dpkg-deb,python3,gzip- all present in a stock install.gitfor T13 only. It is not part of a stock trixie image; without it T13 skips and says so. Install it if you want the Fedora regression checks to run:apt-get install -y --no-install-recommends git. Note this changes the test machine, which is why the harness treats its absence as a skip rather than installing it itself.gpgfor T12 only, and only if you want to re-check the signing decision.xorrisofor T20 only. Without it T20 skips with a reason.- A Debian netinst ISO for T20 only, at
/root/trixie-netinst.isoorNETINST_ISO=/path/to/netinst.iso. It is never downloaded. The build itself takes about 3 seconds and needs ~1.6 GB free inTMPDIR(source plus output; the source is not copied, only written to the output).
Safety
The harness is designed to be safe to run on a machine you care about.
- It never writes to
/var/local/citrix-repo. It records that directory's mtime before the run and asserts it is unchanged in T15. - It never installs anything. The only
apt-get installis--simulate. T15 compares the installed package count before and after. - It adds exactly one apt source file,
/etc/apt/sources.list.d/zz-vdi-test-$$.list, and removes it. T15 verifies the removal happened. - It removes only the apt index files it created, matched by the encoded repo path,
never
/var/lib/apt/listswholesale. - It removes any keyring it installed, matched by the
zz-vdi-test-*prefix, and onlyrmdirs/etc/apt/keyrings(which fails harmlessly if it was already there with other keys in it). - It never writes
__pycache__into the repo:PYTHONDONTWRITEBYTECODE=1andPYTHONPYCACHEPREFIXare set, and T15 asserts the repo is clean. - All scratch work happens in one
mktemp -ddirectory, removed at the end. - A failing test does not abort the run, so you get the whole picture in one pass.
Interpreting a failure
- T05/T06/T07 failing together - the repo build is broken. Look at T05 first.
- T07 fails but T05 passes - the repo is fine but apt cannot consume it; check
directory permissions (apt drops to
_apt, so the repo path must be traversable). - T09 fails -
sync_debianis leaving old versions in the published directory. On Debian, filenames carry the version, so copying without deleting leaves every previous release indexed and makes rollback impossible. - T13 fails on the command sequence - compare the printed
MAIN-CMDSandFEDORA-CMDSlists. Both sides are normalised to<REPO>; if they differ, the Fedora path genuinely changed. - T15 fails on residue - the teardown did not finish. Check the messages immediately above it.