Pester was installed with -MinimumVersion 5.0 and no ceiling in the CI job
image, both install steps in unit-tests.yml, and both Import-Module calls, plus
the suite's own import inside the container. The image is rebuilt on every CI
run and Pester is installed fresh on every unit-test run, so PSGallery chose the
version — a new major could reach the required PR checks with no commit here,
surfacing as unexplained test breakage on whichever PR ran next.
It had already happened. Steps named "Install Pester 5" were resolving 6.1.0 on
both legs, because Pester 6 declares PowerShellVersion 5.1 and so installs on
Windows PowerShell too. Nothing broke — the suite uses only constructs common to
5 and 6, and runs 1566/0 under 6.1.0 with no deprecation warnings — but nobody
chose it. The step names are corrected; they had been describing an install that
stopped happening some time ago.
Pinning the install alone is not enough, in two ways review found:
An unset variable does not fail. -RequiredVersion accepts an empty value and
degrades to "latest" for Install-Module and to "any" for Import-Module, both
exiting 0, so a renamed or dropped env key would silently restore the float this
commit removes. A guard step now fails the job instead.
The point of use was still floored. run-integration.sh imported the suite's
Pester with -MinimumVersion 5.0, so a second Pester reaching PSModulePath would
win regardless of what was installed. The Dockerfile now promotes the ARG to ENV
so the version is discoverable at runtime, and that import is pinned to it.
The pin lives in two files, so shell-selfchecks asserts they agree — split-brain
between the workflow and the image is precisely the unexplained breakage this is
meant to prevent. CONTRIBUTING.md and CLAUDE.md are updated too; the contributor
instructions were a third floating install site.
Recorded as an amendment to D017 — the same principle as the nested PVE package
pin, applied to the lane's own tooling.
Two filed issues in one rewrite of preflight-cleanup.sh's ISO block, because
they are the same twenty lines.
#111 — ISO_FILENAME was interpolated into python3 -c PROGRAM TEXT inside a
single-quoted literal, so a quote in the value escaped it and executed
arbitrary Python in a container holding PVE_API_TOKEN, PVE_PASSWORD, the
Terraform state and the storage VM's SSH key. It now arrives through the
environment and is read with os.environ. The volid is passed to urllib's
quote() via argv for the same reason, and an empty encode result now skips
the volume instead of issuing a DELETE against the bare collection URL.
#105 — generated ISOs embed a hash of first-boot.sh, so every change to that
script mints a new filename. Deleting only the exact current name orphaned
each earlier ISO on the storage permanently, because force-cleanup wipes the
Terraform state that could otherwise reclaim it. The family is now swept by
rebuilding the full generated shape: the captured prefix plus twelve hex
characters plus .iso. A prefix test alone would also have matched a longer
FQDN's family and any hand-uploaded "-manual-backup.iso" sibling, which in a
script whose job is deletion is worse than the leak it fixes.
Multi-delete applies only to that family. A name that is not generated — the
storage VM's cloud image — keeps the original one-shot behaviour, since a
basename can repeat across content namespaces and a plain name carries nothing
that identifies a family.
Adds preflight-cleanup.test.sh, wired into shell-selfchecks. The script had no
coverage at all. It stubs curl and sleep, then asserts on the DELETEs issued:
the family goes, the pinned base ISO and unrelated uploads stay, a non-hash
sibling stays, the cloud image takes only itself, a quoted payload is data
rather than code, and unset storage skips only the ISO branch. Every case also
asserts the script ran to completion and removed the Terraform state, so a path
that dies early cannot pass by having issued the right DELETEs first.
Expands the Rosetta note into a working recipe, after running the whole
provision -> test -> cleanup lifecycle on an Apple Silicon Mac against the real
parent cluster.
Compose is the wrong entry point on a Mac: its dev-infra service builds rather
than pulls, and bind-mounts /opt/pve-integration, which does not exist there.
The macOS path pulls the image CI already built and drives run-integration.sh
with docker run. Records that GHCR needs a classic PAT, since fine-grained
tokens cannot reach it at all and the failure gives no hint why.
Restores the x86 compose instructions, which the previous commit's rewrite
consumed, and hoists the fixed-VMID warning out of the macOS section — 5080,
5091 and 5092 are shared with CI on the same parent cluster whatever host you
run from, so a local run during a CI run collides, and a skipped force-cleanup
fails the next run's headroom guard.
Also warns that emulation runs the suite ~40% slower and so loses the
qemu-server flock race (#113) that CI wins: Reset-PveVm, clone and
Set-PveVmConfig fail locally on a tree that is green in CI. Provisioning and
cleanup are unaffected.
"x86 only" was too strong. The image is amd64-only — proxmox-auto-install-assistant
and the HashiCorp apt repo publish no arm64 — but it builds and runs on Apple Silicon
once Docker Desktop's Rosetta emulation is on.
Under the default qemu translation pwsh starts and reports its version, then segfaults
on module discovery (uncaught target signal 11), which fails the build at
Install-Module Pester and would fail Pester at test time. With Rosetta enabled the same
Dockerfile builds to within 150 bytes of the image CI pushed for this commit, and
Invoke-Pester runs.
Worth stating explicitly because the failure is silent: the build step exits 1 with no
diagnostic output, which reads as a Dockerfile defect rather than an emulation problem.
The local dev environment had drifted badly from CI. Remove the parts that no
longer describe anything real, and make the rest match how CI actually runs.
Delete tests/dev.ps1. It wrapped run-integration.sh, which CI calls directly,
and duplicated the module build that script already performs internally. As a
second entry point it drifted: it still offered the PVE 8 leg retired in #88,
mounted the Docker socket for storage containers replaced by the storage VM in
#87, and pointed its remote-host examples at a runner decommissioned in the ARC
migration. All four documents describing it used a positional syntax that bound
the bare word to -Tests and then fell through to -Shell, so every documented
command silently opened a container shell. Recorded as D019.
Delete tests/infrastructure/runner/, a self-hosted-runner-in-Docker superseded
by Actions Runner Controller.
Make disk_storage and iso_storage required. Their defaults named a NAS that the
lab replaced with Ceph, and CI overrides both from repository variables, so the
defaults only ever misled local runs. require_env now fails at the top of a run
rather than at terraform apply, and the descriptions point at tests/.env.test
because cmd_provision deletes terraform.tfvars before applying.
preflight-cleanup.sh no longer falls back to the literal "local" storage. An
unset TF_VAR_iso_storage now skips only the ISO branch, leaving VM destroy and
state cleanup intact, and emits a workflow annotation: force-cleanup is the
only cleanup CI runs and it wipes Terraform state, so a silent skip strands the
uploaded ISO with nothing left to reclaim it.
Drop docker-ce-cli and the /var/run/docker.sock mount. Nothing in the container
has called docker since #87 moved storage into a VM; the remaining docker calls
run inside that VM over SSH. The CI job image is built from the same target, so
this also removes a third-party apt repository from its supply chain.
Rewrite tests/.env.test.example against what the code now requires, and fix the
documented commands in CLAUDE.md, README.md, copilot-instructions.md and the
integration README.
Review finding on #97: `statusReached` was sticky, so a guest that reached the
expected status once and then drifted away from it would still be reported as a
success at the deadline, contradicting the fallback's own meaning.
Track the most recent observation instead. A guest that drifts away and never
re-matches now raises PveTaskTimeoutException as it should. A failed poll leaves
the previous observation standing, so an API blip is not read as divergence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci: test against the PVE 9.2 ISO
The nested nodes were pinned to proxmox-ve_9.1-1.iso while production runs 9.2.
Validated by runs 183/184; the cluster suite passed in both, and run 184 was
fully green at 148 passed / 0 failed / 5 skipped.
PVE publishes a guest's new status while the operation still holds
/var/lock/qemu-server/lock-<vmid>.conf, so WaitForStatusTransition could return
while the guest was still locked and the caller's next request would fail with
"can't lock file ... got timeout".
Integration run 183 failed four tests from this one cause: Restart-PveVm -Wait
returned after 4.1s having seen "running", the following Stop-PveVm spent exactly
10.0s failing to take the lock, and that cascaded into the template convert,
clone, and remove tests. Run 184 - same commit, re-run - passed because its status
poll happened to take 10.1s, by which point the lock had cleared. The same
settling happens either way; the only variable is whether the wait absorbs it or
the next caller does.
The check goes in WaitForStatusTransition because all nine lifecycle call sites
(Start/Stop/Restart/Reset/Resume across VMs and containers) route through it.
`lock` comes from the status/current response the poll already fetches - present
on both qemu and lxc since PVE 5.4, below the module's 7.0 floor - so it costs no
extra request.
If the status is reached but the lock outlasts -Timeout the cmdlet still returns
success, so a call that succeeded before this change cannot become an exception
after it.
Recorded as D015, the guest-lock sibling of D014.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix: New-PveCluster -Wait blocks until quorate; make cluster failures diagnosable
Adds ClusterConfigService.WaitForQuorum with a -Timeout parameter (D014), so
Add-PveClusterMember no longer hits "cluster not ready - no quorum?" after a
completed create task. Adds diagnose-cluster.sh, which captures corosync and
pmxcfs state plus PVE task logs from both nodes before cleanup destroys them.
goodolclint-claude[bot]