Commit Graph

204 Commits

Author SHA1 Message Date
goodolclint-claude[bot] def8dc6b67 fix: verify checksums for downloaded ISOs/images, keep sshpass off argv (#166)
* fix: verify checksums for downloaded ISOs/images, keep sshpass off argv

ensure-base-iso.sh downloaded the PVE install ISO over plain HTTP with no
checksum, caching it on the persistent /opt/pve-integration mount and
booting it as the nested trust root the integration suite relies on.
ensure-cloud-images.sh fetched the Ubuntu cloud image and OVA over HTTPS
but never checked them either. prepare-test-environment.sh and
diagnose-cluster.sh passed the nested root password to sshpass via -p,
putting it in the process table. create-api-token.sh, unused anywhere in
the repo, minted a privsep=0 root token and echoed the secret unmasked.

- ensure-base-iso.sh now downloads from https://enterprise.proxmox.com/iso
  and verifies against its SHA256SUMS on every run, including a cache hit.
  download.proxmox.com's own TLS cert does not list download.proxmox.com in
  its SAN (confirmed with curl/openssl from this environment), so https to
  that name fails certificate validation; enterprise.proxmox.com serves the
  identical ISO tree over a valid cert. Verification happens before the
  downloaded file is moved to its canonical cache path.
- ensure-cloud-images.sh verifies the cloud image and OVA against Ubuntu's
  published SHA256SUMS the same way, matching by upstream filename since
  the cloud image is cached locally under a different extension (.img
  upstream, .qcow2 cached — the bytes are already qcow2-formatted).
- prepare-test-environment.sh and diagnose-cluster.sh now export SSHPASS
  and call sshpass -e, keeping the password out of argv/ps. This also fixes
  a latent bug: the old unquoted `sshpass -p ${ROOT_PASS}` word-split any
  password containing whitespace.
- create-api-token.sh deleted; grep across the repo found no caller.

Reviewers (codex:codex-rescue, correctness-reviewer, security-reviewer) all
independently found the same blocking bug in the first pass: when a cached
file failed verification and the subsequent redownload then failed,
ensure-cloud-images.sh fell through to a "keep the stale copy" branch and
returned that same known-bad file with exit 0 — verification could be
bypassed by inducing one failed redownload. Fixed by deleting the file
immediately on a failed verification, before the redownload is attempted,
so the later "is there a safe stale copy" check can no longer find it.
Added a test case (case 5) that reproduces this exact sequence and
mutation-tested it against the unfixed code. The three reviews also
flagged a real but separate bug already fixed in this same change: `trap
... RETURN` inside a function nested in another function is not scoped to
that function in bash — it re-fires on the OUTER function's return,
referencing an out-of-scope local. Both verify_checksum() helpers now
clean up their temp file explicitly instead of via trap.

Findings not acted on, judged out of scope for this fix:
- SHA256SUMS-fetch failures are treated the same as a checksum mismatch
  (delete + fail) rather than left untouched — a transient network blip
  destroys a good multi-GB cached ISO. This is the safer failure direction
  (never silently trust unverified bytes) and was a deliberate trade-off,
  not a defect.
- ensure-cloud-images.sh's 7-day cache window can span an upstream
  republish of noble/current, causing a legitimate re-verification churn
  (not a security issue, a cache-hit-rate one). Pre-existing cache design,
  unrelated to adding verification.
- wait-for-pve.sh (curl -d with the password on argv) and
  prepare-test-environment.sh's own positional password argument (from
  run-integration.sh) carry the same password-on-argv pattern this issue
  targeted in create-api-token.sh, sshpass -p and diagnose-cluster.sh, but
  neither script nor run-integration.sh was named in the issue. Left
  untouched per scope; worth a follow-up issue.
- GPG/detached-signature verification of the upstream SHA256SUMS was not
  added — the new checks defend against cache poisoning and transit
  corruption, not a compromised origin. Worth a follow-up issue.
- The two new self-checks (ensure-base-iso.test.sh,
  ensure-cloud-images.test.sh) are not wired into
  .github/workflows/unit-tests.yml's shell-selfchecks job. That file is
  code-owned and out of scope for this change; needs an operator follow-up.

Password rotation (the Testpass123! value from before it moved to a
secret) is unaddressed here per the contract — flagged for the operator.

Mutation-tested: broke the post-download checksum check in
ensure-base-iso.sh, confirmed the affected test cases failed, restored it.
Broke the sshpass -e change back to -p, confirmed the new assertions in
prepare-test-environment.test.sh failed, restored it. Broke the fail-open
fix in ensure-cloud-images.sh, confirmed case 5 failed, restored it.

Closes #149

* fix: also verify the stale-by-age fallback copy in ensure-cloud-images.sh

PR review on #166 (COMMENTED, non-blocking) found the sibling of the
fail-open bug already fixed in this branch: when the cached cloud image
is stale by *age* (>= 7 days) rather than failed verification, the
redownload-failure fallback could hand back that file with exit 0
without ever re-verifying it in this run. A file that failed the
earlier verification is already deleted by the time the fallback runs,
but a stale-by-age file skips verification entirely on the way in.

Fixed by verifying the stale-by-age file at the point of actual
fallback use — after the redownload has failed, not proactively before
it's attempted, so a copy the redownload was about to replace anyway
isn't deleted along a path that would have succeeded. Added two test
cases (6, 7): a still-verifying stale-by-age copy is used as a
fallback; one that no longer verifies is not. Mutation-tested by
reverting to the unfixed fallback and confirming case 7 fails, then
restored.

---------

Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
2026-09-02 17:20:36 +00:00
goodolclint-claude[bot] 1bc46567f5 fix: inject the guest-lock retry delay so tests no longer race wall clock (#167)
PveHttpClientLockRetryTests set GuestLockRetry's retry budget via reflection
on a private field with no production writer, and scripted two lock
failures to land inside a 400ms window. A cold or loaded CI runner's
first-attempt JIT and scheduling could burn past 400ms before the second
attempt started, failing the test though the retry itself was correct
(#134).

GuestLockRetry.ExecuteAsync gains an internal overload that takes the
inter-attempt delay as a Func<TimeSpan, Task>; the public overload keeps
defaulting to Task.Delay, so production behaviour (45s window, budget/4
capped at 2s) is unchanged. PveHttpClient gains a matching internal
constructor seam (window, handler, delay), replacing the reflection the
tests used for both the private HttpClient and the retry window. Tests
now pass a no-op delay, so the retry loop's real elapsed time drops to
microseconds and the production 45s window can never be exhausted by
runner speed.

Two tests pin that production still waits for real: one records the
delay invocations through the internal seam and asserts the computed
interval, the other drives the public overload with a small window and
asserts wall-clock time actually advances. Both were mutation-tested
against a no-op-default regression and fail without the fix.

The give-up test was renamed (PutAsync_DoesNotReissueWhenTheRetryWindowIsAlreadySpent)
to describe what TimeSpan.Zero actually proves: the client never attempts
a reissue once the budget reads spent, not a multi-attempt exhaustion
sequence — a review finding on the original name.

Out of scope, noted for follow-up: GuestLockRetryTests.cs's synchronous
Execute() tests still use a 400ms ShortWindow with real Thread.Sleep,
which is the same flake shape on the sync path; Execute() has no delay
seam. PveHttpClientTimeoutTests.cs and PveHttpClientFormEncodingTests.cs
still reflect on the private _httpClient field, which the new handler
seam could also retire.

Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
2026-09-02 17:10:52 +00:00
goodolclint-claude[bot] 1bf7483a4e fix: escape dynamic path segments in three Remove-* cmdlets (#161)
* fix: escape dynamic path segments in three Remove-* cmdlets

Wrap Storage, Vnet, and Zone parameters with Uri.EscapeDataString() in
RemovePveStorageCmdlet, RemovePveSdnVnetCmdlet, and RemovePveSdnZoneCmdlet
to prevent path traversal attacks via the API path.

Add xUnit test demonstrating that escaped paths preserve percent-encoding
(preventing path collapse) while unescaped paths allow segment traversal.

Fixes #145

* fix: route Remove-Pve{Storage,SdnVnet,SdnZone} through their services

Delete the private PveHttpClient construction and inline
Uri.EscapeDataString call in RemovePveStorageCmdlet,
RemovePveSdnVnetCmdlet and RemovePveSdnZoneCmdlet; call
StorageService.RemoveStorage / NetworkService.RemoveSdnZone /
NetworkService.RemoveSdnVnet instead, which already escape the
identifier identically and are now the single place doing so.

Add ValidatePattern on the Storage/Vnet/Zone parameters as defense
in depth, anchored with \A/\z so a trailing newline cannot slip a
disallowed character past the gate.

Replace PveHttpClientPathEscapingTests with a version that actually
regression-tests the real Uri parser (asserts both that %2F survives
and that the unescaped form is absent), and add
StorageServiceTests/NetworkServiceTests cases that mock
IPveHttpClient and verify the exact escaped DELETE path for a
traversal-attempt name.

---------

Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
2026-09-02 16:59:57 +00:00
goodolclint-claude[bot] b90791e2bf docs: migrate DECISIONS.md to house-format ADRs, and retire the review folder (#131)
D001-D021 become ADR 0001-0021 in docs/decisions/, one decision per file.
D017's PESTER_VERSION amendment was a second decision in one entry and becomes
ADR 0022. ADR 0023 records the migration and reverses the lane2-change-plan
ruling that deliberately kept DECISIONS.md until the CI lane work landed.

DECISIONS.md is reduced to a stub with a D-to-ADR redirect table, so the four
released CHANGELOG entries and older issue bodies that cite it degrade to a
redirect rather than a dead reference.

docs/review/ and docs/lane2-change-plan.md are deleted (ADR 0024). Of 91
findings, 83 were resolved and six of the seven still open were already GitHub
issues; F021 was the exception and is now #130.

CLAUDE.md's Key Conventions list gains the two rules it was missing and becomes
the checklist, with the ADRs carrying rationale.
2026-09-02 14:49:07 +00:00
goodolclint-claude[bot] 2dff02f2bd feat: VLAN-aware bridges via New-PveNetwork and Set-PveNetwork
PveNetwork already deserialised bridge_vlan_aware as BridgeVlanAware, so a
VLAN-aware bridge could be read back but never created or changed. Both write
paths now take a -BridgeVlanAware switch.

Clearing the flag does not use bridge_vlan_aware=0. PVE merges the supplied
keys onto the stored stanza and accepts that 0 without acting on it, so the
obvious form is a silent no-op: an integration run against PVE 9 issued it and
Get-PveNetwork still reported 1. The endpoint's delete list is what actually
removes the key. The API schema advertises a plain boolean and gives no hint of
this, which is why the behaviour is pinned by an integration test rather than
inferred.

Set-PveNetwork guards the switch on BoundParameters so an update that omits it
leaves the flag alone; the create path follows the existing -Autostart form.
Only bridge_vlan_aware is added. bridge_vids is an independent parameter that
PVE defaults to 2-4094, and the issue asks only for the flag.

Coverage: the integration suite pins create, disable, re-enable, and that an
unrelated Set leaves the flag alone -- that last one kills a mutant that drops
the BoundParameters guard, which every other test survives. A model test pins
the read path the assertions depend on. The Pester unit tests assert only
parameter metadata; the defect is server-side, so nothing offline can catch it.

Closes #92
2026-09-01 22:53:26 -05:00
goodolclint-claude[bot] 1e94d4188c fix: report each flock reissue, and scale the gap to the retry budget
Two non-blocking review observations.

A 45s retry is indistinguishable from a hang with nothing on the wire, so
GuestLockRetry.Execute takes an onRetry hook and InvokeGuestTask reports
each reissue through WriteVerbose.

The gap between attempts now scales with the budget, capped at the 2s
production value. A caller passing a short window wants a fast answer
rather than one long sleep, which also takes the retrying unit tests off
a real 2s sleep each: the xUnit run drops from 8s to 4s. PveHttpClient's
window becomes a field so those tests can shorten it too.
2026-09-01 21:49:00 -05:00
goodolclint-claude[bot] db416a3f03 fix: retry the qemu-server flock instead of predicting it
WaitForStatusTransition refused to return while snapshot.Locked, and its
comment quoted the exact error it was meant to prevent. Locked reads the
guest config's lock: property; the failure is the flock on
/var/lock/qemu-server/lock-<vmid>.conf, which PVE exposes nowhere.

The flock cannot be observed, so it is retried. GuestLockRetry reissues an
operation for a bounded 45s while PVE reports failing to enter lock_config
for a guest, which it raises before doing any work.

Two seams, because the failure has two surfaces. PveHttpClient.SendAsync
retries the request for operations PVE serialises in the API handler; it
takes a request factory because an HttpRequestMessage cannot be resent.
PveCmdletBase.InvokeGuestTask reissues the call and re-waits its task for
operations serialised in the forked worker, where the POST returns 200 and
only the task fails. WaitForStatusTransition routes through the latter,
hence Func<PveTask>.

The predicate is path-specific and anchored at the start of what PVE said:
lock_file uses identical wording for storage, LVM and HA locks, and a
qmclone that fails after allocating disks must not be reissued into
"VM already exists". That requires the raw text, so it reads
PveTaskFailedException.ExitStatus and PveApiException.ApiMessage.

The Locked check stays — it is correct for the config lock — with a comment
that says so.

Closes #113
2026-09-01 21:19:05 -05:00
goodolclint-claude[bot] d287ea8e26 Merge branch 'main' into ci/pin-pester
# Conflicts:
#	.github/workflows/unit-tests.yml
2026-09-01 18:31:09 -05:00
goodolclint-claude[bot] fc073e7e2a ci: pin Pester by exact version everywhere it is installed or imported
Pester was installed with -MinimumVersion 5.0 and no ceiling in the CI job
image, both install steps in unit-tests.yml, and both Import-Module calls, plus
the suite's own import inside the container. The image is rebuilt on every CI
run and Pester is installed fresh on every unit-test run, so PSGallery chose the
version — a new major could reach the required PR checks with no commit here,
surfacing as unexplained test breakage on whichever PR ran next.

It had already happened. Steps named "Install Pester 5" were resolving 6.1.0 on
both legs, because Pester 6 declares PowerShellVersion 5.1 and so installs on
Windows PowerShell too. Nothing broke — the suite uses only constructs common to
5 and 6, and runs 1566/0 under 6.1.0 with no deprecation warnings — but nobody
chose it. The step names are corrected; they had been describing an install that
stopped happening some time ago.

Pinning the install alone is not enough, in two ways review found:

An unset variable does not fail. -RequiredVersion accepts an empty value and
degrades to "latest" for Install-Module and to "any" for Import-Module, both
exiting 0, so a renamed or dropped env key would silently restore the float this
commit removes. A guard step now fails the job instead.

The point of use was still floored. run-integration.sh imported the suite's
Pester with -MinimumVersion 5.0, so a second Pester reaching PSModulePath would
win regardless of what was installed. The Dockerfile now promotes the ARG to ENV
so the version is discoverable at runtime, and that import is pinned to it.

The pin lives in two files, so shell-selfchecks asserts they agree — split-brain
between the workflow and the image is precisely the unexplained breakage this is
meant to prevent. CONTRIBUTING.md and CLAUDE.md are updated too; the contributor
instructions were a third floating install site.

Recorded as an amendment to D017 — the same principle as the nested PVE package
pin, applied to the lane's own tooling.
2026-09-01 18:08:02 -05:00
goodolclint-claude[bot] 122e79407c fix: reap the whole generated-ISO family without over-matching, and stop building python from the filename
Two filed issues in one rewrite of preflight-cleanup.sh's ISO block, because
they are the same twenty lines.

#111 — ISO_FILENAME was interpolated into python3 -c PROGRAM TEXT inside a
single-quoted literal, so a quote in the value escaped it and executed
arbitrary Python in a container holding PVE_API_TOKEN, PVE_PASSWORD, the
Terraform state and the storage VM's SSH key. It now arrives through the
environment and is read with os.environ. The volid is passed to urllib's
quote() via argv for the same reason, and an empty encode result now skips
the volume instead of issuing a DELETE against the bare collection URL.

#105 — generated ISOs embed a hash of first-boot.sh, so every change to that
script mints a new filename. Deleting only the exact current name orphaned
each earlier ISO on the storage permanently, because force-cleanup wipes the
Terraform state that could otherwise reclaim it. The family is now swept by
rebuilding the full generated shape: the captured prefix plus twelve hex
characters plus .iso. A prefix test alone would also have matched a longer
FQDN's family and any hand-uploaded "-manual-backup.iso" sibling, which in a
script whose job is deletion is worse than the leak it fixes.

Multi-delete applies only to that family. A name that is not generated — the
storage VM's cloud image — keeps the original one-shot behaviour, since a
basename can repeat across content namespaces and a plain name carries nothing
that identifies a family.

Adds preflight-cleanup.test.sh, wired into shell-selfchecks. The script had no
coverage at all. It stubs curl and sleep, then asserts on the DELETEs issued:
the family goes, the pinned base ISO and unrelated uploads stay, a non-hash
sibling stays, the cloud image takes only itself, a quoted payload is data
rather than code, and unset storage skips only the ISO branch. Every case also
asserts the script ran to completion and removed the Terraform state, so a path
that dies early cannot pass by having issued the right DELETEs first.
2026-09-01 18:05:03 -05:00
goodolclint-claude[bot] 08ee3ae249 chore: repair the local dev path and delete its dead scaffolding
The local dev environment had drifted badly from CI. Remove the parts that no
longer describe anything real, and make the rest match how CI actually runs.

Delete tests/dev.ps1. It wrapped run-integration.sh, which CI calls directly,
and duplicated the module build that script already performs internally. As a
second entry point it drifted: it still offered the PVE 8 leg retired in #88,
mounted the Docker socket for storage containers replaced by the storage VM in
#87, and pointed its remote-host examples at a runner decommissioned in the ARC
migration. All four documents describing it used a positional syntax that bound
the bare word to -Tests and then fell through to -Shell, so every documented
command silently opened a container shell. Recorded as D019.

Delete tests/infrastructure/runner/, a self-hosted-runner-in-Docker superseded
by Actions Runner Controller.

Make disk_storage and iso_storage required. Their defaults named a NAS that the
lab replaced with Ceph, and CI overrides both from repository variables, so the
defaults only ever misled local runs. require_env now fails at the top of a run
rather than at terraform apply, and the descriptions point at tests/.env.test
because cmd_provision deletes terraform.tfvars before applying.

preflight-cleanup.sh no longer falls back to the literal "local" storage. An
unset TF_VAR_iso_storage now skips only the ISO branch, leaving VM destroy and
state cleanup intact, and emits a workflow annotation: force-cleanup is the
only cleanup CI runs and it wipes Terraform state, so a silent skip strands the
uploaded ISO with nothing left to reclaim it.

Drop docker-ce-cli and the /var/run/docker.sock mount. Nothing in the container
has called docker since #87 moved storage into a VM; the remaining docker calls
run inside that VM over SSH. The CI job image is built from the same target, so
this also removes a third-party apt repository from its supply chain.

Rewrite tests/.env.test.example against what the code now requires, and fix the
documented commands in CLAUDE.md, README.md, copilot-instructions.md and the
integration README.
2026-09-01 16:27:10 -05:00
goodolclint-claude[bot] e7f8460ff7 ci: report package currency to a rolling issue and a data branch
Acts on pre-push review findings from codex + correctness/security subagents.
2026-09-01 15:21:48 -05:00
goodolclint-claude[bot] e5fa905ee2 ci: verify the reboot, split machinery failures from test failures
Acts on pre-push review findings from codex + correctness/security subagents.
2026-09-01 14:22:43 -05:00
goodolclint-claude[bot] a5dab58592 ci: add the package-currency workflow (lane 2) 2026-09-01 14:09:58 -05:00
goodolclint-claude[bot] 7e66dfc051 ci: opt-in dist-upgrade and reboot for the currency lane 2026-09-01 14:02:15 -05:00
goodolclint-claude[bot] 959b62c86f test: assert node B joined by name, not an online-node count
Closes #94
2026-09-01 18:37:22 +00:00
Clint Branham c15820b5aa ci: hash first-boot.sh into the cached auto-install ISO name
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 13:24:36 -05:00
goodolclint-claude[bot] c5e73f4fc9 fix: Restart-PveVm uses PVE's native reboot endpoint (cmdlet + tests) 2026-09-01 17:46:21 +00:00
goodolclint-claude[bot] 3f63ddcad8 fix: lifecycle -Wait blocks until the guest config lock clears
PVE publishes a guest's new status while the operation still holds
/var/lock/qemu-server/lock-<vmid>.conf, so WaitForStatusTransition could return
while the guest was still locked and the caller's next request would fail with
"can't lock file ... got timeout".

Integration run 183 failed four tests from this one cause: Restart-PveVm -Wait
returned after 4.1s having seen "running", the following Stop-PveVm spent exactly
10.0s failing to take the lock, and that cascaded into the template convert,
clone, and remove tests. Run 184 - same commit, re-run - passed because its status
poll happened to take 10.1s, by which point the lock had cleared. The same
settling happens either way; the only variable is whether the wait absorbs it or
the next caller does.

The check goes in WaitForStatusTransition because all nine lifecycle call sites
(Start/Stop/Restart/Reset/Resume across VMs and containers) route through it.
`lock` comes from the status/current response the poll already fetches - present
on both qemu and lxc since PVE 5.4, below the module's 7.0 floor - so it costs no
extra request.

If the status is reached but the lock outlasts -Timeout the cmdlet still returns
success, so a call that succeeded before this change cannot become an exception
after it.

Recorded as D015, the guest-lock sibling of D014.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 16:33:47 +00:00
goodolclint-claude[bot] 5e046e3d31 ci: stop half-upgrading the nested nodes
`apt-get upgrade` holds back any package whose upgrade needs new dependencies,
so it neither pins nor updates — it produces whatever partial set the
dependency graph allows that day. On these nodes it upgraded pve-cluster to
9.1.6 while leaving libpve-cluster-api-perl at 9.1.0.

Those two ship the halves of the join: cfs_backup_database() in
PVE/Cluster.pm (pve-cluster) and finish_join() in PVE/Cluster/Setup.pm
(libpve-cluster-api-perl). Upstream removed `return $dbfile` from the former
and stopped relying on it in the latter, both at 9.1.1 — 9.1.6's finish_join
calls cfs_unlink_db_unsafe() instead. The 9.1.0 caller against the 9.1.6
callee unlinks an empty string, so the standalone config.db survives the join,
pmxcfs restarts in local mode, and the node reports online=0 forever while
corosync forms a healthy 2-node membership. That is the "2 nodes online"
failure, and it is not reachable on any coherent install.

The ISO is the pin, so drop the upgrade and install only what the harness
needs. Upgrades belong in a separate currency lane that records the package
set it tested.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 14:25:50 +00:00
goodolclint-claude[bot] 5c6cd7f40f ci: probe whether pmxcfs came up clustered or in local mode
Run 173 left node B with healthy corosync (2-member primary component, both
links connected) but no /etc/pve/corosync.conf, no dcdb/status journal lines,
and pvecm status reporting it is not part of a cluster. That file is
database-backed: pmxcfs creates it only when it starts with no config.db and
imports /etc/corosync/corosync.conf, so a surviving standalone config.db would
mean silent local mode.

Capture the package versions, pmxcfs command line, /etc/pve mount, .members,
the config.db and its backup dir, whether the database holds a corosync.conf
row, and the CPG group membership. Read-only; the sqlite3 CLI is not guaranteed
on a PVE node, so fall back to strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 07:27:31 +00:00
goodolclint-claude[bot] 1df1b05876 test: drop the fixed sleep and the link0 pin from the cluster tests
New-PveCluster -Wait now guarantees quorum, so the Start-Sleep -Seconds 5
that lost the race is replaced by an assertion on the new contract.

The link0 pin is dropped: the join-abort and the never-a-member modes both
occurred with and without it, so it was never implicated, and -Links is
already covered at the service level.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 06:24:11 +00:00
goodolclint-claude[bot] 2df9f47e2d test: cover ClusterConfigService.WaitForQuorum
Returns once quorate, throws TimeoutException when quorum never arrives,
and keeps polling through a PveApiException from the restarting API.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 06:22:55 +00:00
goodolclint-claude[bot] db93af27b2 ci: capture cluster task logs in the diagnostics
"Cluster join aborted!" is PVE's generic wrapper; the reason lives only in
the task log on the joining node. Run 172's log said "An error occurred on
the cluster node: cluster not ready - no quorum?", which is what identified
the race. Capture it so the evidence survives cleanup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 06:18:48 +00:00
goodolclint-claude[bot] 44101e39c7 ci: dump corosync state from both nested nodes on cluster test failure
The API reports a joined-but-offline node as online=0 with no further
detail, and the cleanup job destroys the nodes minutes later, so the
reason corosync membership never forms has never reached a log. Read
corosync.conf, corosync-cfgtool, pvecm status and the corosync journal
off both nodes while they are still alive. Best-effort: never fails the
caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 05:38:08 +00:00
goodolclint-claude[bot] e0960d9c18 test: pin node B ring0 at join and dump cluster status on the online check
The "2 nodes online" check has never passed — run 159 shows the identical
failure while the job reported success (swallowed Pester exit code), and
run 170's 60 s poll expires with node B still online=0. Node B reaches
corosync.conf (Get-PveClusterConfigNode passes) but never becomes a
corosync member, and the suite captures nothing about why.

Pass link0 explicitly so ring0 is the address the harness verified node B
answers on, rather than whatever node B resolves its own DHCP-assigned
.test.local hostname to. Dump quorate and each node's ring0/online/local
unconditionally so the next run distinguishes a wrong ring0 address from
a working address with no corosync transport.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 05:02:53 +00:00
goodolclint-claude[bot] f758275668 test: poll cluster status for node B online instead of asserting instantly
The join task completing does not mean corosync membership has reached
the status endpoint; the instant assertion failed intermittently (runs
159 and 169) while every other cluster check passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 23:00:49 +00:00
goodolclint-claude[bot] 59a47e3fab ci: storage VM moves to DHCP and is addressed by FQDN (2/2)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 20:54:14 +00:00
goodolclint-claude[bot] 25746a94e3 ci: storage VM moves to DHCP and is addressed by FQDN (1/2)
The repo stops holding opinions about CI VLAN addressing: the VM boots
via DHCP with hostname pvetest-storage (deterministic MAC for an
optional reservation), registers in the operator's CI DNS zone, and
everything addresses it by STORAGE_VM_FQDN. Replaces the static-IP +
explicit-DNS variables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 20:49:48 +00:00
goodolclint-claude[bot] 7872f08102 ci: storage VM uses public DNS — the CI VLAN gateway runs no resolver
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 20:42:07 +00:00
goodolclint-claude[bot] 5d0b384508 ci: storage VM setup diagnoses its own network and logs phases
Run 168 burned 13 minutes to report 'unable to locate package tgt' when
the real failure was dead egress from the CI VLAN (apt lists never
updated). The setup script now probes gateway/internet/DNS from inside
the VM and prints the verdicts, fails apt fast with the real error, and
marks each phase.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 20:39:02 +00:00
goodolclint-claude[bot] 77c6510788 ci: default PVE_VERSIONS to 9, count storage VM in headroom message
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 20:15:50 +00:00
goodolclint-claude[bot] 4d3ea81bd4 ci: remove docker-compose storage definition (moved to storage VM) 2026-08-31 19:56:16 +00:00
goodolclint-claude[bot] 999bac6bdc ci: remove runner-side docker storage resources (moved to storage VM) 2026-08-31 19:56:10 +00:00
goodolclint-claude[bot] f4e5760080 ci: storage and answer services move to a VM inside the CI sandbox (2/2)
run-integration.sh: two-phase apply (storage VM configured before the
PVE guests boot), Pester PassThru with explicit failed/zero-test checks,
auto-ISO cache names embed the answer host, force-cleanup covers the
storage VM and imported cloud image, docker usage removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 19:56:02 +00:00
goodolclint-claude[bot] f48f0ea9ae ci: storage and answer services move to a VM inside the CI sandbox (1/2)
The nested PVE guests live on an isolated VLAN with no route to the
runner network (ADR 0032, homelab repo), and the runner-hosted Docker
services died with each ephemeral ARC pod anyway. A small cloud-image VM
in the ci pool now serves NFS, iSCSI, and the auto-install answer files
from inside the sandbox. Includes fixes from Codex and in-house review:
bpg provider >= 0.79.0, serial console for the resized cloud image,
key-based SSH (cloud images refuse password auth), CIDR validation,
storage VM in the headroom check, docker socket mounts removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 19:52:36 +00:00
goodolclint-claude[bot] d105f51246 ci: provision picks the least-loaded node and guards host memory
PVE_TARGET_NODE becomes optional: unset (or 'auto') queries
/cluster/resources for the online node with the most free memory and
refuses to provision when no node has vm_count x 8GiB + 8GiB headroom
free. The chosen node persists on the shared mount so cleanup and
force-cleanup aim at the node provision picked.

2026-08-31: a provision onto the pinned node OOM-killed a 16GiB
services-plane VM on the parent hypervisor; the pin predated a memory
bump that consumed the headroom it silently relied on. Needs PVEAuditor
on /nodes for the token's user (a non-privsep token ignores token ACLs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-31 18:23:23 +00:00
goodolclint-claude[bot] 3c97f0237a Merge branch 'main' into dependabot/nuget/tests/PSProxmoxVE.Core.Tests/main/xunit.runner.visualstudio-4.0.0 2026-08-31 17:19:58 +00:00
dependabot[bot] aae3764227 Bump xunit.runner.visualstudio from 3.1.5 to 4.0.0
---
updated-dependencies:
- dependency-name: xunit.runner.visualstudio
  dependency-version: 4.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-31 17:10:23 +00:00
dependabot[bot] eacd570922 Bump Microsoft.NET.Test.Sdk from 18.5.1 to 18.9.0
---
updated-dependencies:
- dependency-name: Microsoft.NET.Test.Sdk
  dependency-version: 18.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-31 17:10:15 +00:00
Clint Branham 2897feb43e ci: cleanup honours PVE_TARGET_NODE and the ISO storage var
A pool-scoped token cannot list /nodes, so node discovery fell back to 'pve'
and force-cleanup silently found no VMs to destroy.
2026-08-23 09:27:27 -05:00
Clint Branham d3f05187dd ci: nested PVE guests land in the ci pool with VMIDs 5081-5092
The runner token is pool-scoped (PVEVMAdmin on /pool/ci only), so VM.Allocate
succeeds only with pool_id set; VMIDs move into the 5000-5999 CI range.
2026-08-23 09:06:11 -05:00
Clint Branham 56dcf22cea fix: address PR #69 review feedback
- VmService.ExecuteGuestCommand: guard against null elements in -Args.
  The old JSON-serialization tolerated nulls (as "null"); the repeated-key
  path would NRE in EncodeFormValue. Throw a clear ArgumentException instead.
- findings.json: refresh the stale counters block (untouched since F085) to
  the actual ledger state — next_id 92, resolved 83 — and bump last_updated
  to 2026-05-22. last_scan_date stays 2026-03-26 (F086–F091 came from issue
  triage, not a formal review scan).
- VmServiceTests: add empty-array (single command entry) and null-element
  (throws) cases.

Note: F091 is the correct next ID — F086–F090 already exist from prior
merged PRs (#60/#61/#66/#67); only the counters were lagging.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 14:38:54 -05:00
Clint Branham bc71ed4a12 fix: deliver Invoke-PveVmGuestExec -Args to the guest as argv
ExecuteGuestCommand JSON-serialized the args array into the agent/exec
'input-data' field — which is the process's STDIN, not its arguments. So
guest commands ran with no argv: cmd.exe started interactively and the
JSON blob ['/c','echo',...] arrived at its prompt.

PVE's agent/exec 'command' parameter is itself an array (element 0 = the
executable, the rest = argv) sent as repeated form keys. The low-level
client couldn't express repeated keys (Dictionary<string,string> only),
so:

- Add PostAsync(string, IEnumerable<KeyValuePair<string,string>>) to
  IPveHttpClient/PveHttpClient; BuildFormContent now emits one key=value
  field per pair, so a key may repeat.
- ExecuteGuestCommand builds command = [exe] + args as repeated 'command'
  fields and no longer touches input-data.

Tests: form-encoder repeated-key + per-value encoding cases; VmService
tests asserting the command array, order, and absence of input-data; an
integration regression guard that echoes an arg and checks it round-trips
as stdout.

Tracked as F091. Closes #68.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 14:32:04 -05:00
Clint Branham 80b70cdaf6 fix: address PR #67 review feedback
- HasDiskOptions(): include -ScsiHardware so the "disk options ignored"
  warning fires when -ScsiHardware is passed without -DiskStorage/-DiskSize.
- PveVmConfig.AdditionalProperties: lazy-init a backing field so the native
  dictionary is built once rather than reallocated on every property access
  (matters when iterating many configs in a pipeline). Safe because the model
  is effectively immutable after deserialization.
- New-PveVm.Tests.ps1: add a case asserting -DiskIoThread on scsi with a
  wrong -ScsiHardware (virtio-scsi-pci) is rejected, covering the validator's
  "!= virtio-scsi-single" branch (not just the null case).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 14:18:32 -05:00
Clint Branham 63ee16a9e7 Merge remote-tracking branch 'origin/main' into feat/new-pvevm-disk-options
# Conflicts:
#	docs/review/findings.json
2026-05-22 14:15:20 -05:00
Clint Branham c1714048b4 feat: disk controller/IO options on New-PveVm + surface all Get-PveVmConfig keys
New-PveVm (F089):
- Add -DiskBus (virtio/scsi/sata/ide, default virtio), -ScsiHardware (scsihw),
  -DiskIoThread, -DiskAio, -DiskSsd, -DiskDiscard, -DiskCache so a tuned disk
  (e.g. virtio-scsi-single + scsi0,iothread=1,aio=native,ssd=1,discard=on) can
  be created in one call instead of diskless + a hand-built Set-PveVmConfig string.
- Disk spec built via BuildDiskSpec; ValidateDiskOptions runs before ShouldProcess
  and rejects ssd on virtio and iothread on sata/ide or scsi-without-virtio-scsi-single
  with clear errors, instead of letting PVE fail at VM start.

Get-PveVmConfig (F090):
- PveVmConfig was a fixed allow-list, silently dropping keys like scsihw, efidisk0,
  tpmstate0, hostpci0. Add typed scsihw/efidisk0/tpmstate0 plus a [JsonExtensionData]
  catch-all exposed as AdditionalProperties (native types via JsonHelper.ToNative,
  per D013 — no JToken leakage). Makes the disk tuning above verifiable by reading
  the config back.

Closes #65.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 14:10:14 -05:00
Clint Branham 3fd770d84c fix: percent-encode ';' in form values
PveHttpClient.EncodeFormValue encoded &, =, +, space, and % but left ';'
literal. PVE's application/x-www-form-urlencoded parser treats a raw ';'
as a field separator (the historical alternative to '&'), so a value like

    boot=order=scsi0;ide2

was split into 'boot=order=scsi0' plus an empty 'ide2' field, and PVE
rejected the PUT with "ide2: unable to parse drive options". This broke
any multi-device boot order set via Set-PveVmConfig -AdditionalConfig,
and any other value containing ';'.

Encode ';' as %3B. Safe under the existing minimal-encoding policy that
keeps ':' and '!' literal for cluster-join: cluster-join payloads never
contain ';', and PVE url-decodes config form values.

Tracked as F088. Closes #64.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 14:04:31 -05:00
Clint Branham a1d9550d83 fix: surface HttpClient timeouts as PveApiException(RequestTimeout)
When HttpClient.Timeout elapses, .NET throws TaskCanceledException — not
HttpRequestException — so the existing catch in PveHttpClient.SendAsync
missed it and callers got a raw stack trace. With -TimeoutSeconds now
configurable and documented, this gap became user-visible.

In .NET 5+ HttpClient surfaces transport timeouts as TaskCanceledException
with a TimeoutException inner; user-driven token cancellation does not.
Catch by that inner-type signature and rethrow as PveApiException with
HttpStatusCode.RequestTimeout, the resource path, and a message that
reports the configured timeout.

Adds SendAsync_TimeoutFires_ThrowsPveApiExceptionWithRequestTimeout which
swaps in a delaying HttpMessageHandler with a 50ms timeout to exercise
the path deterministically. Drops the redundant
DefaultSessionTimeoutIs100Seconds test (covered by
PveSessionTests.Timeout_DefaultIs100Seconds and the existing flow-through
test).

Addresses PR #61 review feedback.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 18:17:31 -05:00
Clint Branham ea2bcdc336 Merge main into fix/http-client-timeout
Resolves findings.json conflict — F086 (from #60, merged into main) and
F087 (this branch) both append to the trailing findings array. Kept both
entries, in numeric order.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 18:17:20 -05:00