PVE publishes a guest's new status while the operation still holds
/var/lock/qemu-server/lock-<vmid>.conf, so WaitForStatusTransition could return
while the guest was still locked and the caller's next request would fail with
"can't lock file ... got timeout".
Integration run 183 failed four tests from this one cause: Restart-PveVm -Wait
returned after 4.1s having seen "running", the following Stop-PveVm spent exactly
10.0s failing to take the lock, and that cascaded into the template convert,
clone, and remove tests. Run 184 - same commit, re-run - passed because its status
poll happened to take 10.1s, by which point the lock had cleared. The same
settling happens either way; the only variable is whether the wait absorbs it or
the next caller does.
The check goes in WaitForStatusTransition because all nine lifecycle call sites
(Start/Stop/Restart/Reset/Resume across VMs and containers) route through it.
`lock` comes from the status/current response the poll already fetches - present
on both qemu and lxc since PVE 5.4, below the module's 7.0 floor - so it costs no
extra request.
If the status is reached but the lock outlasts -Timeout the cmdlet still returns
success, so a call that succeeded before this change cannot become an exception
after it.
Recorded as D015, the guest-lock sibling of D014.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`apt-get upgrade` holds back any package whose upgrade needs new dependencies,
so it neither pins nor updates — it produces whatever partial set the
dependency graph allows that day. On these nodes it upgraded pve-cluster to
9.1.6 while leaving libpve-cluster-api-perl at 9.1.0.
Those two ship the halves of the join: cfs_backup_database() in
PVE/Cluster.pm (pve-cluster) and finish_join() in PVE/Cluster/Setup.pm
(libpve-cluster-api-perl). Upstream removed `return $dbfile` from the former
and stopped relying on it in the latter, both at 9.1.1 — 9.1.6's finish_join
calls cfs_unlink_db_unsafe() instead. The 9.1.0 caller against the 9.1.6
callee unlinks an empty string, so the standalone config.db survives the join,
pmxcfs restarts in local mode, and the node reports online=0 forever while
corosync forms a healthy 2-node membership. That is the "2 nodes online"
failure, and it is not reachable on any coherent install.
The ISO is the pin, so drop the upgrade and install only what the harness
needs. Upgrades belong in a separate currency lane that records the package
set it tested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 173 left node B with healthy corosync (2-member primary component, both
links connected) but no /etc/pve/corosync.conf, no dcdb/status journal lines,
and pvecm status reporting it is not part of a cluster. That file is
database-backed: pmxcfs creates it only when it starts with no config.db and
imports /etc/corosync/corosync.conf, so a surviving standalone config.db would
mean silent local mode.
Capture the package versions, pmxcfs command line, /etc/pve mount, .members,
the config.db and its backup dir, whether the database holds a corosync.conf
row, and the CPG group membership. Read-only; the sqlite3 CLI is not guaranteed
on a PVE node, so fall back to strings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
New-PveCluster -Wait now guarantees quorum, so the Start-Sleep -Seconds 5
that lost the race is replaced by an assertion on the new contract.
The link0 pin is dropped: the join-abort and the never-a-member modes both
occurred with and without it, so it was never implicated, and -Links is
already covered at the service level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Returns once quorate, throws TimeoutException when quorum never arrives,
and keeps polling through a PveApiException from the restarting API.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Cluster join aborted!" is PVE's generic wrapper; the reason lives only in
the task log on the joining node. Run 172's log said "An error occurred on
the cluster node: cluster not ready - no quorum?", which is what identified
the race. Capture it so the evidence survives cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The API reports a joined-but-offline node as online=0 with no further
detail, and the cleanup job destroys the nodes minutes later, so the
reason corosync membership never forms has never reached a log. Read
corosync.conf, corosync-cfgtool, pvecm status and the corosync journal
off both nodes while they are still alive. Best-effort: never fails the
caller.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "2 nodes online" check has never passed — run 159 shows the identical
failure while the job reported success (swallowed Pester exit code), and
run 170's 60 s poll expires with node B still online=0. Node B reaches
corosync.conf (Get-PveClusterConfigNode passes) but never becomes a
corosync member, and the suite captures nothing about why.
Pass link0 explicitly so ring0 is the address the harness verified node B
answers on, rather than whatever node B resolves its own DHCP-assigned
.test.local hostname to. Dump quorate and each node's ring0/online/local
unconditionally so the next run distinguishes a wrong ring0 address from
a working address with no corosync transport.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The join task completing does not mean corosync membership has reached
the status endpoint; the instant assertion failed intermittently (runs
159 and 169) while every other cluster check passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo stops holding opinions about CI VLAN addressing: the VM boots
via DHCP with hostname pvetest-storage (deterministic MAC for an
optional reservation), registers in the operator's CI DNS zone, and
everything addresses it by STORAGE_VM_FQDN. Replaces the static-IP +
explicit-DNS variables.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 168 burned 13 minutes to report 'unable to locate package tgt' when
the real failure was dead egress from the CI VLAN (apt lists never
updated). The setup script now probes gateway/internet/DNS from inside
the VM and prints the verdicts, fails apt fast with the real error, and
marks each phase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
run-integration.sh: two-phase apply (storage VM configured before the
PVE guests boot), Pester PassThru with explicit failed/zero-test checks,
auto-ISO cache names embed the answer host, force-cleanup covers the
storage VM and imported cloud image, docker usage removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nested PVE guests live on an isolated VLAN with no route to the
runner network (ADR 0032, homelab repo), and the runner-hosted Docker
services died with each ephemeral ARC pod anyway. A small cloud-image VM
in the ci pool now serves NFS, iSCSI, and the auto-install answer files
from inside the sandbox. Includes fixes from Codex and in-house review:
bpg provider >= 0.79.0, serial console for the resized cloud image,
key-based SSH (cloud images refuse password auth), CIDR validation,
storage VM in the headroom check, docker socket mounts removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PVE_TARGET_NODE becomes optional: unset (or 'auto') queries
/cluster/resources for the online node with the most free memory and
refuses to provision when no node has vm_count x 8GiB + 8GiB headroom
free. The chosen node persists on the shared mount so cleanup and
force-cleanup aim at the node provision picked.
2026-08-31: a provision onto the pinned node OOM-killed a 16GiB
services-plane VM on the parent hypervisor; the pin predated a memory
bump that consumed the headroom it silently relied on. Needs PVEAuditor
on /nodes for the token's user (a non-privsep token ignores token ACLs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- VmService.ExecuteGuestCommand: guard against null elements in -Args.
The old JSON-serialization tolerated nulls (as "null"); the repeated-key
path would NRE in EncodeFormValue. Throw a clear ArgumentException instead.
- findings.json: refresh the stale counters block (untouched since F085) to
the actual ledger state — next_id 92, resolved 83 — and bump last_updated
to 2026-05-22. last_scan_date stays 2026-03-26 (F086–F091 came from issue
triage, not a formal review scan).
- VmServiceTests: add empty-array (single command entry) and null-element
(throws) cases.
Note: F091 is the correct next ID — F086–F090 already exist from prior
merged PRs (#60/#61/#66/#67); only the counters were lagging.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ExecuteGuestCommand JSON-serialized the args array into the agent/exec
'input-data' field — which is the process's STDIN, not its arguments. So
guest commands ran with no argv: cmd.exe started interactively and the
JSON blob ['/c','echo',...] arrived at its prompt.
PVE's agent/exec 'command' parameter is itself an array (element 0 = the
executable, the rest = argv) sent as repeated form keys. The low-level
client couldn't express repeated keys (Dictionary<string,string> only),
so:
- Add PostAsync(string, IEnumerable<KeyValuePair<string,string>>) to
IPveHttpClient/PveHttpClient; BuildFormContent now emits one key=value
field per pair, so a key may repeat.
- ExecuteGuestCommand builds command = [exe] + args as repeated 'command'
fields and no longer touches input-data.
Tests: form-encoder repeated-key + per-value encoding cases; VmService
tests asserting the command array, order, and absence of input-data; an
integration regression guard that echoes an arg and checks it round-trips
as stdout.
Tracked as F091. Closes#68.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- HasDiskOptions(): include -ScsiHardware so the "disk options ignored"
warning fires when -ScsiHardware is passed without -DiskStorage/-DiskSize.
- PveVmConfig.AdditionalProperties: lazy-init a backing field so the native
dictionary is built once rather than reallocated on every property access
(matters when iterating many configs in a pipeline). Safe because the model
is effectively immutable after deserialization.
- New-PveVm.Tests.ps1: add a case asserting -DiskIoThread on scsi with a
wrong -ScsiHardware (virtio-scsi-pci) is rejected, covering the validator's
"!= virtio-scsi-single" branch (not just the null case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New-PveVm (F089):
- Add -DiskBus (virtio/scsi/sata/ide, default virtio), -ScsiHardware (scsihw),
-DiskIoThread, -DiskAio, -DiskSsd, -DiskDiscard, -DiskCache so a tuned disk
(e.g. virtio-scsi-single + scsi0,iothread=1,aio=native,ssd=1,discard=on) can
be created in one call instead of diskless + a hand-built Set-PveVmConfig string.
- Disk spec built via BuildDiskSpec; ValidateDiskOptions runs before ShouldProcess
and rejects ssd on virtio and iothread on sata/ide or scsi-without-virtio-scsi-single
with clear errors, instead of letting PVE fail at VM start.
Get-PveVmConfig (F090):
- PveVmConfig was a fixed allow-list, silently dropping keys like scsihw, efidisk0,
tpmstate0, hostpci0. Add typed scsihw/efidisk0/tpmstate0 plus a [JsonExtensionData]
catch-all exposed as AdditionalProperties (native types via JsonHelper.ToNative,
per D013 — no JToken leakage). Makes the disk tuning above verifiable by reading
the config back.
Closes#65.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PveHttpClient.EncodeFormValue encoded &, =, +, space, and % but left ';'
literal. PVE's application/x-www-form-urlencoded parser treats a raw ';'
as a field separator (the historical alternative to '&'), so a value like
boot=order=scsi0;ide2
was split into 'boot=order=scsi0' plus an empty 'ide2' field, and PVE
rejected the PUT with "ide2: unable to parse drive options". This broke
any multi-device boot order set via Set-PveVmConfig -AdditionalConfig,
and any other value containing ';'.
Encode ';' as %3B. Safe under the existing minimal-encoding policy that
keeps ':' and '!' literal for cluster-join: cluster-join payloads never
contain ';', and PVE url-decodes config form values.
Tracked as F088. Closes#64.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When HttpClient.Timeout elapses, .NET throws TaskCanceledException — not
HttpRequestException — so the existing catch in PveHttpClient.SendAsync
missed it and callers got a raw stack trace. With -TimeoutSeconds now
configurable and documented, this gap became user-visible.
In .NET 5+ HttpClient surfaces transport timeouts as TaskCanceledException
with a TimeoutException inner; user-driven token cancellation does not.
Catch by that inner-type signature and rethrow as PveApiException with
HttpStatusCode.RequestTimeout, the resource path, and a message that
reports the configured timeout.
Adds SendAsync_TimeoutFires_ThrowsPveApiExceptionWithRequestTimeout which
swaps in a delaying HttpMessageHandler with a 50ms timeout to exercise
the path deterministically. Drops the redundant
DefaultSessionTimeoutIs100Seconds test (covered by
PveSessionTests.Timeout_DefaultIs100Seconds and the existing flow-through
test).
Addresses PR #61 review feedback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves findings.json conflict — F086 (from #60, merged into main) and
F087 (this branch) both append to the trailing findings array. Kept both
entries, in numeric order.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PveHttpClient was constructed without setting HttpClient.Timeout, so
.NET's 100s default applied to every request. Multi-GB ISO uploads via
Send-PveFile on a real LAN reliably tripped this with TaskCanceledException
after 100 seconds, and there was no way to override it.
- PveSession gains a Timeout (TimeSpan) property, defaulting to 100s.
- PveHttpClient accepts an optional per-instance timeout override that
takes precedence over the session timeout.
- Connect-PveServer exposes -TimeoutSeconds to set the session default.
- Send-PveFile and Invoke-PveStorageDownload expose -TimeoutSeconds with
a 30-minute implicit default so large uploads/downloads do not trip
the 100s default. -TimeoutSeconds 0 means Timeout.InfiniteTimeSpan.
Tracked as F087. Closes#59.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- SizeParser: wrap TB-suffix overflow in try/catch so callers get
ArgumentException with the parameter name rather than OverflowException.
- New-PveVm/New-PveContainer: validate -DiskSize/-RootFsSize before
ShouldProcess so typos like "512M" are rejected even with -WhatIf and
even when the matching -DiskStorage/-RootFsStorage is omitted.
- Add Pester tests for the new DiskSize and RootFsSize validation paths,
including a new New-PveContainer.Tests.ps1.
- Add SizeParserTests coverage for the TB overflow path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Disk and rootfs size strings were interpolated directly into the disk
spec as "<storage>:<size>", so "60G" produced "local-lvm:60G". On
LVM/LVM-thin storages PVE parses the value after the colon as a volume
name unless it is a bare integer, returning "unable to parse lvm volume
name '60G'". File-backed storages mask this by accepting either form.
SizeParser.NormalizeToGibibytes() now strips G/GB/T/TB suffixes and
returns a bare GiB integer string, so the documented "32G" call shape
works on every storage type. Sub-GB units are rejected with a clear
error rather than being silently truncated.
Tracked as F086. Closes#58.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace single-version fixture with per-version specs from pve-api.
Tests now validate enum values against PVE 7 (best-effort), 8, and 9.
Key findings from version-specific specs:
- VM.Monitor: valid in PVE 7+8, removed in PVE 9
- VM.Replicate: added in PVE 9 only
- VM.GuestAgent.*: added in PVE 9 only
- Mapping.*: added in PVE 8+
- glusterfs: not in any version (removed before PVE 7)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 70 xUnit tests that validate every ValidateSet in the module
against the PVE OpenAPI spec. Three bugs found and fixed:
- Storage: remove `glusterfs` (dropped in PVE 9), add `btrfs`, `esxi`
- Backup compression: `none` → `0` (PVE uses "0" not "none")
- Cluster resources: remove `lxc` filter (PVE uses `vm` for both)
The pve-api-enums.json fixture (199KB) is extracted from the full
OpenAPI spec and contains parameter enum values for 302 API paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The ValidateSet attribute was intentionally removed from the Type
parameter during merge review to allow future rule types without
code changes. Updated test to only assert Type is mandatory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVE 9 API requires 'type' on PUT /cluster/ha/rules/{rule} even for
updates. Added mandatory Type parameter with ValidateSet for
node-affinity and resource-affinity.
Updated integration test and Pester unit tests to pass -Type.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rename ZZ.ClusterConfig.Integration.Tests.ps1 → 16_Cluster.Tests.ps1
- Replace manual env var setup with _IntegrationHelper.ps1 + Connect-TestPve
- Use credential auth (root@pam) throughout — no API tokens
- Use $script:PasswordB for node B auth (from helper)
- Access JoinInfo.Nodelist as List<Dictionary> (native types from D013)
- Keep cluster-specific skip helpers (Skip-IfNoCluster, Skip-IfPve9HaGroups)
- Remove redundant Connection context (Connect-TestPve handles it)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVE 9 rejects rules referencing unmanaged resources. Test now creates
a minimal VM, registers it as a disabled HA resource, then creates
the node-affinity rule. Cleanup removes rule, resource, and VM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVE 9 rejects 'affinity' as an unexpected property on POST
cluster/ha/rules — it's implied by the node-affinity type.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Full lifecycle: create node-affinity rule, list, get by ID, update
comment, delete, and verify deletion. Uses vm:99999 as a synthetic
resource SID. Tests skip on PVE 8 (rules not available).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cluster tests create a 2-node cluster that cannot be torn down via API
(quorum loss). Prefix with ZZ so they execute after all other
integration tests that assume standalone nodes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clint Branham