* fix: verify checksums for downloaded ISOs/images, keep sshpass off argv
ensure-base-iso.sh downloaded the PVE install ISO over plain HTTP with no
checksum, caching it on the persistent /opt/pve-integration mount and
booting it as the nested trust root the integration suite relies on.
ensure-cloud-images.sh fetched the Ubuntu cloud image and OVA over HTTPS
but never checked them either. prepare-test-environment.sh and
diagnose-cluster.sh passed the nested root password to sshpass via -p,
putting it in the process table. create-api-token.sh, unused anywhere in
the repo, minted a privsep=0 root token and echoed the secret unmasked.
- ensure-base-iso.sh now downloads from https://enterprise.proxmox.com/iso
and verifies against its SHA256SUMS on every run, including a cache hit.
download.proxmox.com's own TLS cert does not list download.proxmox.com in
its SAN (confirmed with curl/openssl from this environment), so https to
that name fails certificate validation; enterprise.proxmox.com serves the
identical ISO tree over a valid cert. Verification happens before the
downloaded file is moved to its canonical cache path.
- ensure-cloud-images.sh verifies the cloud image and OVA against Ubuntu's
published SHA256SUMS the same way, matching by upstream filename since
the cloud image is cached locally under a different extension (.img
upstream, .qcow2 cached — the bytes are already qcow2-formatted).
- prepare-test-environment.sh and diagnose-cluster.sh now export SSHPASS
and call sshpass -e, keeping the password out of argv/ps. This also fixes
a latent bug: the old unquoted `sshpass -p ${ROOT_PASS}` word-split any
password containing whitespace.
- create-api-token.sh deleted; grep across the repo found no caller.
Reviewers (codex:codex-rescue, correctness-reviewer, security-reviewer) all
independently found the same blocking bug in the first pass: when a cached
file failed verification and the subsequent redownload then failed,
ensure-cloud-images.sh fell through to a "keep the stale copy" branch and
returned that same known-bad file with exit 0 — verification could be
bypassed by inducing one failed redownload. Fixed by deleting the file
immediately on a failed verification, before the redownload is attempted,
so the later "is there a safe stale copy" check can no longer find it.
Added a test case (case 5) that reproduces this exact sequence and
mutation-tested it against the unfixed code. The three reviews also
flagged a real but separate bug already fixed in this same change: `trap
... RETURN` inside a function nested in another function is not scoped to
that function in bash — it re-fires on the OUTER function's return,
referencing an out-of-scope local. Both verify_checksum() helpers now
clean up their temp file explicitly instead of via trap.
Findings not acted on, judged out of scope for this fix:
- SHA256SUMS-fetch failures are treated the same as a checksum mismatch
(delete + fail) rather than left untouched — a transient network blip
destroys a good multi-GB cached ISO. This is the safer failure direction
(never silently trust unverified bytes) and was a deliberate trade-off,
not a defect.
- ensure-cloud-images.sh's 7-day cache window can span an upstream
republish of noble/current, causing a legitimate re-verification churn
(not a security issue, a cache-hit-rate one). Pre-existing cache design,
unrelated to adding verification.
- wait-for-pve.sh (curl -d with the password on argv) and
prepare-test-environment.sh's own positional password argument (from
run-integration.sh) carry the same password-on-argv pattern this issue
targeted in create-api-token.sh, sshpass -p and diagnose-cluster.sh, but
neither script nor run-integration.sh was named in the issue. Left
untouched per scope; worth a follow-up issue.
- GPG/detached-signature verification of the upstream SHA256SUMS was not
added — the new checks defend against cache poisoning and transit
corruption, not a compromised origin. Worth a follow-up issue.
- The two new self-checks (ensure-base-iso.test.sh,
ensure-cloud-images.test.sh) are not wired into
.github/workflows/unit-tests.yml's shell-selfchecks job. That file is
code-owned and out of scope for this change; needs an operator follow-up.
Password rotation (the Testpass123! value from before it moved to a
secret) is unaddressed here per the contract — flagged for the operator.
Mutation-tested: broke the post-download checksum check in
ensure-base-iso.sh, confirmed the affected test cases failed, restored it.
Broke the sshpass -e change back to -p, confirmed the new assertions in
prepare-test-environment.test.sh failed, restored it. Broke the fail-open
fix in ensure-cloud-images.sh, confirmed case 5 failed, restored it.
Closes#149
* fix: also verify the stale-by-age fallback copy in ensure-cloud-images.sh
PR review on #166 (COMMENTED, non-blocking) found the sibling of the
fail-open bug already fixed in this branch: when the cached cloud image
is stale by *age* (>= 7 days) rather than failed verification, the
redownload-failure fallback could hand back that file with exit 0
without ever re-verifying it in this run. A file that failed the
earlier verification is already deleted by the time the fallback runs,
but a stale-by-age file skips verification entirely on the way in.
Fixed by verifying the stale-by-age file at the point of actual
fallback use — after the redownload has failed, not proactively before
it's attempted, so a copy the redownload was about to replace anyway
isn't deleted along a path that would have succeeded. Added two test
cases (6, 7): a still-verifying stale-by-age copy is used as a
fallback; one that no longer verifies is not. Mutation-tested by
reverting to the unfixed fallback and confirming case 7 fails, then
restored.
---------
Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
Restructures the integration test workflow from a monolithic sequential
job into separate provision → test → cleanup stages:
- provision: creates ALL nested PVE VMs in a single terraform apply
(parallel), waits for APIs, creates tokens, passes outputs to tests
- test: matrix [pve9, pve8] consumes provision outputs, no provisioning
- cleanup: always runs, API-only teardown for all VMs
Terraform refactored to for_each with pve_instances map variable,
enabling parallel ISO upload and VM creation.
New caching scripts reduce redundant downloads:
- ensure-base-iso.sh: downloads PVE ISOs to /opt/pve-isos if missing
- ensure-cloud-images.sh: caches cloud image + OVA with 7-day TTL
- prepare-auto-iso.sh: --cache-dir flag with hash-based skip
Runner no longer needs manual ISO provisioning (zero-touch setup).
cleanup-images bumped to min-versions-to-keep: 3 to survive overlapping
runs. All jobs gated against dependabot.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All VM and container lifecycle cmdlets (Start, Stop, Restart, Suspend,
Resume, Reset) now poll VM/container status when -Wait is specified,
blocking until the expected state is reached or -Timeout (default 60s)
is exceeded.
Implementation:
- PveCmdletBase.WaitForStatusTransition() — shared helper that waits
for PVE task completion then polls status via API
- 9 cmdlets updated: 6 VM (Start, Stop, Restart, Suspend, Resume,
Reset) + 3 container (Start, Stop, Restart)
- -Timeout parameter with [ValidateRange(1, 3600)] on each
Integration tests:
- Replace all manual Start-Sleep + polling loops with -Wait -Timeout
- Switch to real Ubuntu cloud OVA (571 MB) for Import-PveOva testing
- OVA test verifies full import + VM start
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Suspend/Resume: move to Linux VM lifecycle context (empty VMs may
not reliably transition to paused state)
- SDN subnet model: dhcp-range is an array, not string
- SDN subnet delete: use PVE-format ID (vnet-ip-prefix) not CIDR
- Container config: trim trailing newline from PVE description
- OVA: use qemu-img to create valid sparse VMDK (add qemu-utils to
test container Dockerfile)
- Enable 'images' content type on local storage for OVA import
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SDN: zone ID cannot contain hyphens — rename 'pester-zone' to 'pesterz'
- SDN: subnet removal uses original CIDR, not transformed ID
- Containers: New-PveContainer now auto-assigns VM ID via /cluster/nextid
when -VmId is omitted (was missing, unlike New-PveVm)
- OVA: enable 'images' content type on local storage for import-from
- Suspend: add delay before status check, use Should -Not -Throw pattern
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVE upload API validates file extensions per content type. The
content=import type does not accept .img extension. The Ubuntu cloud
image is qcow2 format, so rename from .img to .qcow2 during download.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Send-PveFile replaces Send-PveIso with a -ContentType parameter
(iso, vztmpl, import). This is a breaking rename — the module has
not been released yet.
Test infrastructure refactored:
- prepare-test-vm.sh replaced with prepare-test-environment.sh
(only SSH ops: enable content types, upload cloud-init snippet,
curl download cloud image)
- VM provisioning moved into integration tests as "Linux VM —
Provisioning" context: Send-PveFile, New-PveVm, Import-PveVmDisk,
Set-PveVmConfig, Set-PveCloudInitConfig, Start-PveVm,
Test-PveVmGuestAgent — all tested as part of the test suite
- PVETEST_LINUX_VMID env var removed (VM created by tests)
- PVETEST_CLOUD_IMAGE_PATH + PVETEST_PASSWORD added
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
goodolclint-claude[bot]