Expands the Rosetta note into a working recipe, after running the whole
provision -> test -> cleanup lifecycle on an Apple Silicon Mac against the real
parent cluster.
Compose is the wrong entry point on a Mac: its dev-infra service builds rather
than pulls, and bind-mounts /opt/pve-integration, which does not exist there.
The macOS path pulls the image CI already built and drives run-integration.sh
with docker run. Records that GHCR needs a classic PAT, since fine-grained
tokens cannot reach it at all and the failure gives no hint why.
Restores the x86 compose instructions, which the previous commit's rewrite
consumed, and hoists the fixed-VMID warning out of the macOS section — 5080,
5091 and 5092 are shared with CI on the same parent cluster whatever host you
run from, so a local run during a CI run collides, and a skipped force-cleanup
fails the next run's headroom guard.
Also warns that emulation runs the suite ~40% slower and so loses the
qemu-server flock race (#113) that CI wins: Reset-PveVm, clone and
Set-PveVmConfig fail locally on a tree that is green in CI. Provisioning and
cleanup are unaffected.
"x86 only" was too strong. The image is amd64-only — proxmox-auto-install-assistant
and the HashiCorp apt repo publish no arm64 — but it builds and runs on Apple Silicon
once Docker Desktop's Rosetta emulation is on.
Under the default qemu translation pwsh starts and reports its version, then segfaults
on module discovery (uncaught target signal 11), which fails the build at
Install-Module Pester and would fail Pester at test time. With Rosetta enabled the same
Dockerfile builds to within 150 bytes of the image CI pushed for this commit, and
Invoke-Pester runs.
Worth stating explicitly because the failure is silent: the build step exits 1 with no
diagnostic output, which reads as a Dockerfile defect rather than an emulation problem.
The local dev environment had drifted badly from CI. Remove the parts that no
longer describe anything real, and make the rest match how CI actually runs.
Delete tests/dev.ps1. It wrapped run-integration.sh, which CI calls directly,
and duplicated the module build that script already performs internally. As a
second entry point it drifted: it still offered the PVE 8 leg retired in #88,
mounted the Docker socket for storage containers replaced by the storage VM in
#87, and pointed its remote-host examples at a runner decommissioned in the ARC
migration. All four documents describing it used a positional syntax that bound
the bare word to -Tests and then fell through to -Shell, so every documented
command silently opened a container shell. Recorded as D019.
Delete tests/infrastructure/runner/, a self-hosted-runner-in-Docker superseded
by Actions Runner Controller.
Make disk_storage and iso_storage required. Their defaults named a NAS that the
lab replaced with Ceph, and CI overrides both from repository variables, so the
defaults only ever misled local runs. require_env now fails at the top of a run
rather than at terraform apply, and the descriptions point at tests/.env.test
because cmd_provision deletes terraform.tfvars before applying.
preflight-cleanup.sh no longer falls back to the literal "local" storage. An
unset TF_VAR_iso_storage now skips only the ISO branch, leaving VM destroy and
state cleanup intact, and emits a workflow annotation: force-cleanup is the
only cleanup CI runs and it wipes Terraform state, so a silent skip strands the
uploaded ISO with nothing left to reclaim it.
Drop docker-ce-cli and the /var/run/docker.sock mount. Nothing in the container
has called docker since #87 moved storage into a VM; the remaining docker calls
run inside that VM over SSH. The CI job image is built from the same target, so
this also removes a third-party apt repository from its supply chain.
Rewrite tests/.env.test.example against what the code now requires, and fix the
documented commands in CLAUDE.md, README.md, copilot-instructions.md and the
integration README.
Review finding on #97: `statusReached` was sticky, so a guest that reached the
expected status once and then drifted away from it would still be reported as a
success at the deadline, contradicting the fallback's own meaning.
Track the most recent observation instead. A guest that drifts away and never
re-matches now raises PveTaskTimeoutException as it should. A failed poll leaves
the previous observation standing, so an API blip is not read as divergence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci: test against the PVE 9.2 ISO
The nested nodes were pinned to proxmox-ve_9.1-1.iso while production runs 9.2.
Validated by runs 183/184; the cluster suite passed in both, and run 184 was
fully green at 148 passed / 0 failed / 5 skipped.
PVE publishes a guest's new status while the operation still holds
/var/lock/qemu-server/lock-<vmid>.conf, so WaitForStatusTransition could return
while the guest was still locked and the caller's next request would fail with
"can't lock file ... got timeout".
Integration run 183 failed four tests from this one cause: Restart-PveVm -Wait
returned after 4.1s having seen "running", the following Stop-PveVm spent exactly
10.0s failing to take the lock, and that cascaded into the template convert,
clone, and remove tests. Run 184 - same commit, re-run - passed because its status
poll happened to take 10.1s, by which point the lock had cleared. The same
settling happens either way; the only variable is whether the wait absorbs it or
the next caller does.
The check goes in WaitForStatusTransition because all nine lifecycle call sites
(Start/Stop/Restart/Reset/Resume across VMs and containers) route through it.
`lock` comes from the status/current response the poll already fetches - present
on both qemu and lxc since PVE 5.4, below the module's 7.0 floor - so it costs no
extra request.
If the status is reached but the lock outlasts -Timeout the cmdlet still returns
success, so a call that succeeded before this change cannot become an exception
after it.
Recorded as D015, the guest-lock sibling of D014.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix: New-PveCluster -Wait blocks until quorate; make cluster failures diagnosable
Adds ClusterConfigService.WaitForQuorum with a -Timeout parameter (D014), so
Add-PveClusterMember no longer hits "cluster not ready - no quorum?" after a
completed create task. Adds diagnose-cluster.sh, which captures corosync and
pmxcfs state plus PVE task logs from both nodes before cleanup destroys them.
The nested nodes were pinned to proxmox-ve_9.1-1.iso while production runs
9.2, so the suite was validating a PVE nobody uses. run-integration.sh already
reads PVE9_ISO with the 9.1 name only as a fallback, so this needs no script
change; setting it at the workflow level keeps provision and cleanup agreeing
on the ISO name, which force-cleanup needs to find the right file to delete.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`apt-get upgrade` holds back any package whose upgrade needs new dependencies,
so it neither pins nor updates — it produces whatever partial set the
dependency graph allows that day. On these nodes it upgraded pve-cluster to
9.1.6 while leaving libpve-cluster-api-perl at 9.1.0.
Those two ship the halves of the join: cfs_backup_database() in
PVE/Cluster.pm (pve-cluster) and finish_join() in PVE/Cluster/Setup.pm
(libpve-cluster-api-perl). Upstream removed `return $dbfile` from the former
and stopped relying on it in the latter, both at 9.1.1 — 9.1.6's finish_join
calls cfs_unlink_db_unsafe() instead. The 9.1.0 caller against the 9.1.6
callee unlinks an empty string, so the standalone config.db survives the join,
pmxcfs restarts in local mode, and the node reports online=0 forever while
corosync forms a healthy 2-node membership. That is the "2 nodes online"
failure, and it is not reachable on any coherent install.
The ISO is the pin, so drop the upgrade and install only what the harness
needs. Upgrades belong in a separate currency lane that records the package
set it tested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 173 left node B with healthy corosync (2-member primary component, both
links connected) but no /etc/pve/corosync.conf, no dcdb/status journal lines,
and pvecm status reporting it is not part of a cluster. That file is
database-backed: pmxcfs creates it only when it starts with no config.db and
imports /etc/corosync/corosync.conf, so a surviving standalone config.db would
mean silent local mode.
Capture the package versions, pmxcfs command line, /etc/pve mount, .members,
the config.db and its backup dir, whether the database holds a corosync.conf
row, and the CPG group membership. Read-only; the sqlite3 CLI is not guaranteed
on a PVE node, so fall back to strings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Also documents the two distinct timeout conventions in the module so the
-Wait vs HTTP-client distinction does not have to be re-derived.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
New-PveCluster -Wait now guarantees quorum, so the Start-Sleep -Seconds 5
that lost the race is replaced by an assertion on the new contract.
The link0 pin is dropped: the join-abort and the never-a-member modes both
occurred with and without it, so it was never implicated, and -Links is
already covered at the service level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
goodolclint-claude[bot]