* fix: verify checksums for downloaded ISOs/images, keep sshpass off argv
ensure-base-iso.sh downloaded the PVE install ISO over plain HTTP with no
checksum, caching it on the persistent /opt/pve-integration mount and
booting it as the nested trust root the integration suite relies on.
ensure-cloud-images.sh fetched the Ubuntu cloud image and OVA over HTTPS
but never checked them either. prepare-test-environment.sh and
diagnose-cluster.sh passed the nested root password to sshpass via -p,
putting it in the process table. create-api-token.sh, unused anywhere in
the repo, minted a privsep=0 root token and echoed the secret unmasked.
- ensure-base-iso.sh now downloads from https://enterprise.proxmox.com/iso
and verifies against its SHA256SUMS on every run, including a cache hit.
download.proxmox.com's own TLS cert does not list download.proxmox.com in
its SAN (confirmed with curl/openssl from this environment), so https to
that name fails certificate validation; enterprise.proxmox.com serves the
identical ISO tree over a valid cert. Verification happens before the
downloaded file is moved to its canonical cache path.
- ensure-cloud-images.sh verifies the cloud image and OVA against Ubuntu's
published SHA256SUMS the same way, matching by upstream filename since
the cloud image is cached locally under a different extension (.img
upstream, .qcow2 cached — the bytes are already qcow2-formatted).
- prepare-test-environment.sh and diagnose-cluster.sh now export SSHPASS
and call sshpass -e, keeping the password out of argv/ps. This also fixes
a latent bug: the old unquoted `sshpass -p ${ROOT_PASS}` word-split any
password containing whitespace.
- create-api-token.sh deleted; grep across the repo found no caller.
Reviewers (codex:codex-rescue, correctness-reviewer, security-reviewer) all
independently found the same blocking bug in the first pass: when a cached
file failed verification and the subsequent redownload then failed,
ensure-cloud-images.sh fell through to a "keep the stale copy" branch and
returned that same known-bad file with exit 0 — verification could be
bypassed by inducing one failed redownload. Fixed by deleting the file
immediately on a failed verification, before the redownload is attempted,
so the later "is there a safe stale copy" check can no longer find it.
Added a test case (case 5) that reproduces this exact sequence and
mutation-tested it against the unfixed code. The three reviews also
flagged a real but separate bug already fixed in this same change: `trap
... RETURN` inside a function nested in another function is not scoped to
that function in bash — it re-fires on the OUTER function's return,
referencing an out-of-scope local. Both verify_checksum() helpers now
clean up their temp file explicitly instead of via trap.
Findings not acted on, judged out of scope for this fix:
- SHA256SUMS-fetch failures are treated the same as a checksum mismatch
(delete + fail) rather than left untouched — a transient network blip
destroys a good multi-GB cached ISO. This is the safer failure direction
(never silently trust unverified bytes) and was a deliberate trade-off,
not a defect.
- ensure-cloud-images.sh's 7-day cache window can span an upstream
republish of noble/current, causing a legitimate re-verification churn
(not a security issue, a cache-hit-rate one). Pre-existing cache design,
unrelated to adding verification.
- wait-for-pve.sh (curl -d with the password on argv) and
prepare-test-environment.sh's own positional password argument (from
run-integration.sh) carry the same password-on-argv pattern this issue
targeted in create-api-token.sh, sshpass -p and diagnose-cluster.sh, but
neither script nor run-integration.sh was named in the issue. Left
untouched per scope; worth a follow-up issue.
- GPG/detached-signature verification of the upstream SHA256SUMS was not
added — the new checks defend against cache poisoning and transit
corruption, not a compromised origin. Worth a follow-up issue.
- The two new self-checks (ensure-base-iso.test.sh,
ensure-cloud-images.test.sh) are not wired into
.github/workflows/unit-tests.yml's shell-selfchecks job. That file is
code-owned and out of scope for this change; needs an operator follow-up.
Password rotation (the Testpass123! value from before it moved to a
secret) is unaddressed here per the contract — flagged for the operator.
Mutation-tested: broke the post-download checksum check in
ensure-base-iso.sh, confirmed the affected test cases failed, restored it.
Broke the sshpass -e change back to -p, confirmed the new assertions in
prepare-test-environment.test.sh failed, restored it. Broke the fail-open
fix in ensure-cloud-images.sh, confirmed case 5 failed, restored it.
Closes#149
* fix: also verify the stale-by-age fallback copy in ensure-cloud-images.sh
PR review on #166 (COMMENTED, non-blocking) found the sibling of the
fail-open bug already fixed in this branch: when the cached cloud image
is stale by *age* (>= 7 days) rather than failed verification, the
redownload-failure fallback could hand back that file with exit 0
without ever re-verifying it in this run. A file that failed the
earlier verification is already deleted by the time the fallback runs,
but a stale-by-age file skips verification entirely on the way in.
Fixed by verifying the stale-by-age file at the point of actual
fallback use — after the redownload has failed, not proactively before
it's attempted, so a copy the redownload was about to replace anyway
isn't deleted along a path that would have succeeded. Added two test
cases (6, 7): a still-verifying stale-by-age copy is used as a
fallback; one that no longer verifies is not. Mutation-tested by
reverting to the unfixed fallback and confirming case 7 fails, then
restored.
---------
Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
Pester was installed with -MinimumVersion 5.0 and no ceiling in the CI job
image, both install steps in unit-tests.yml, and both Import-Module calls, plus
the suite's own import inside the container. The image is rebuilt on every CI
run and Pester is installed fresh on every unit-test run, so PSGallery chose the
version — a new major could reach the required PR checks with no commit here,
surfacing as unexplained test breakage on whichever PR ran next.
It had already happened. Steps named "Install Pester 5" were resolving 6.1.0 on
both legs, because Pester 6 declares PowerShellVersion 5.1 and so installs on
Windows PowerShell too. Nothing broke — the suite uses only constructs common to
5 and 6, and runs 1566/0 under 6.1.0 with no deprecation warnings — but nobody
chose it. The step names are corrected; they had been describing an install that
stopped happening some time ago.
Pinning the install alone is not enough, in two ways review found:
An unset variable does not fail. -RequiredVersion accepts an empty value and
degrades to "latest" for Install-Module and to "any" for Import-Module, both
exiting 0, so a renamed or dropped env key would silently restore the float this
commit removes. A guard step now fails the job instead.
The point of use was still floored. run-integration.sh imported the suite's
Pester with -MinimumVersion 5.0, so a second Pester reaching PSModulePath would
win regardless of what was installed. The Dockerfile now promotes the ARG to ENV
so the version is discoverable at runtime, and that import is pinned to it.
The pin lives in two files, so shell-selfchecks asserts they agree — split-brain
between the workflow and the image is precisely the unexplained breakage this is
meant to prevent. CONTRIBUTING.md and CLAUDE.md are updated too; the contributor
instructions were a third floating install site.
Recorded as an amendment to D017 — the same principle as the nested PVE package
pin, applied to the lane's own tooling.
Two filed issues in one rewrite of preflight-cleanup.sh's ISO block, because
they are the same twenty lines.
#111 — ISO_FILENAME was interpolated into python3 -c PROGRAM TEXT inside a
single-quoted literal, so a quote in the value escaped it and executed
arbitrary Python in a container holding PVE_API_TOKEN, PVE_PASSWORD, the
Terraform state and the storage VM's SSH key. It now arrives through the
environment and is read with os.environ. The volid is passed to urllib's
quote() via argv for the same reason, and an empty encode result now skips
the volume instead of issuing a DELETE against the bare collection URL.
#105 — generated ISOs embed a hash of first-boot.sh, so every change to that
script mints a new filename. Deleting only the exact current name orphaned
each earlier ISO on the storage permanently, because force-cleanup wipes the
Terraform state that could otherwise reclaim it. The family is now swept by
rebuilding the full generated shape: the captured prefix plus twelve hex
characters plus .iso. A prefix test alone would also have matched a longer
FQDN's family and any hand-uploaded "-manual-backup.iso" sibling, which in a
script whose job is deletion is worse than the leak it fixes.
Multi-delete applies only to that family. A name that is not generated — the
storage VM's cloud image — keeps the original one-shot behaviour, since a
basename can repeat across content namespaces and a plain name carries nothing
that identifies a family.
Adds preflight-cleanup.test.sh, wired into shell-selfchecks. The script had no
coverage at all. It stubs curl and sleep, then asserts on the DELETEs issued:
the family goes, the pinned base ISO and unrelated uploads stay, a non-hash
sibling stays, the cloud image takes only itself, a quoted payload is data
rather than code, and unset storage skips only the ISO branch. Every case also
asserts the script ran to completion and removed the Terraform state, so a path
that dies early cannot pass by having issued the right DELETEs first.
The local dev environment had drifted badly from CI. Remove the parts that no
longer describe anything real, and make the rest match how CI actually runs.
Delete tests/dev.ps1. It wrapped run-integration.sh, which CI calls directly,
and duplicated the module build that script already performs internally. As a
second entry point it drifted: it still offered the PVE 8 leg retired in #88,
mounted the Docker socket for storage containers replaced by the storage VM in
#87, and pointed its remote-host examples at a runner decommissioned in the ARC
migration. All four documents describing it used a positional syntax that bound
the bare word to -Tests and then fell through to -Shell, so every documented
command silently opened a container shell. Recorded as D019.
Delete tests/infrastructure/runner/, a self-hosted-runner-in-Docker superseded
by Actions Runner Controller.
Make disk_storage and iso_storage required. Their defaults named a NAS that the
lab replaced with Ceph, and CI overrides both from repository variables, so the
defaults only ever misled local runs. require_env now fails at the top of a run
rather than at terraform apply, and the descriptions point at tests/.env.test
because cmd_provision deletes terraform.tfvars before applying.
preflight-cleanup.sh no longer falls back to the literal "local" storage. An
unset TF_VAR_iso_storage now skips only the ISO branch, leaving VM destroy and
state cleanup intact, and emits a workflow annotation: force-cleanup is the
only cleanup CI runs and it wipes Terraform state, so a silent skip strands the
uploaded ISO with nothing left to reclaim it.
Drop docker-ce-cli and the /var/run/docker.sock mount. Nothing in the container
has called docker since #87 moved storage into a VM; the remaining docker calls
run inside that VM over SSH. The CI job image is built from the same target, so
this also removes a third-party apt repository from its supply chain.
Rewrite tests/.env.test.example against what the code now requires, and fix the
documented commands in CLAUDE.md, README.md, copilot-instructions.md and the
integration README.
`apt-get upgrade` holds back any package whose upgrade needs new dependencies,
so it neither pins nor updates — it produces whatever partial set the
dependency graph allows that day. On these nodes it upgraded pve-cluster to
9.1.6 while leaving libpve-cluster-api-perl at 9.1.0.
Those two ship the halves of the join: cfs_backup_database() in
PVE/Cluster.pm (pve-cluster) and finish_join() in PVE/Cluster/Setup.pm
(libpve-cluster-api-perl). Upstream removed `return $dbfile` from the former
and stopped relying on it in the latter, both at 9.1.1 — 9.1.6's finish_join
calls cfs_unlink_db_unsafe() instead. The 9.1.0 caller against the 9.1.6
callee unlinks an empty string, so the standalone config.db survives the join,
pmxcfs restarts in local mode, and the node reports online=0 forever while
corosync forms a healthy 2-node membership. That is the "2 nodes online"
failure, and it is not reachable on any coherent install.
The ISO is the pin, so drop the upgrade and install only what the harness
needs. Upgrades belong in a separate currency lane that records the package
set it tested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 173 left node B with healthy corosync (2-member primary component, both
links connected) but no /etc/pve/corosync.conf, no dcdb/status journal lines,
and pvecm status reporting it is not part of a cluster. That file is
database-backed: pmxcfs creates it only when it starts with no config.db and
imports /etc/corosync/corosync.conf, so a surviving standalone config.db would
mean silent local mode.
Capture the package versions, pmxcfs command line, /etc/pve mount, .members,
the config.db and its backup dir, whether the database holds a corosync.conf
row, and the CPG group membership. Read-only; the sqlite3 CLI is not guaranteed
on a PVE node, so fall back to strings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Cluster join aborted!" is PVE's generic wrapper; the reason lives only in
the task log on the joining node. Run 172's log said "An error occurred on
the cluster node: cluster not ready - no quorum?", which is what identified
the race. Capture it so the evidence survives cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The API reports a joined-but-offline node as online=0 with no further
detail, and the cleanup job destroys the nodes minutes later, so the
reason corosync membership never forms has never reached a log. Read
corosync.conf, corosync-cfgtool, pvecm status and the corosync journal
off both nodes while they are still alive. Best-effort: never fails the
caller.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo stops holding opinions about CI VLAN addressing: the VM boots
via DHCP with hostname pvetest-storage (deterministic MAC for an
optional reservation), registers in the operator's CI DNS zone, and
everything addresses it by STORAGE_VM_FQDN. Replaces the static-IP +
explicit-DNS variables.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 168 burned 13 minutes to report 'unable to locate package tgt' when
the real failure was dead egress from the CI VLAN (apt lists never
updated). The setup script now probes gateway/internet/DNS from inside
the VM and prints the verdicts, fails apt fast with the real error, and
marks each phase.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
run-integration.sh: two-phase apply (storage VM configured before the
PVE guests boot), Pester PassThru with explicit failed/zero-test checks,
auto-ISO cache names embed the answer host, force-cleanup covers the
storage VM and imported cloud image, docker usage removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nested PVE guests live on an isolated VLAN with no route to the
runner network (ADR 0032, homelab repo), and the runner-hosted Docker
services died with each ephemeral ARC pod anyway. A small cloud-image VM
in the ci pool now serves NFS, iSCSI, and the auto-install answer files
from inside the sandbox. Includes fixes from Codex and in-house review:
bpg provider >= 0.79.0, serial console for the resized cloud image,
key-based SSH (cloud images refuse password auth), CIDR validation,
storage VM in the headroom check, docker socket mounts removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PVE_TARGET_NODE becomes optional: unset (or 'auto') queries
/cluster/resources for the online node with the most free memory and
refuses to provision when no node has vm_count x 8GiB + 8GiB headroom
free. The chosen node persists on the shared mount so cleanup and
force-cleanup aim at the node provision picked.
2026-08-31: a provision onto the pinned node OOM-killed a 16GiB
services-plane VM on the parent hypervisor; the pin predated a memory
bump that consumed the headroom it silently relied on. Needs PVEAuditor
on /nodes for the token's user (a non-privsep token ignores token ACLs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RUNNER_TEMP (/__w/_temp in CI) is container-local and invisible to
the Docker host. Files written there can't be bind-mounted into
sibling containers (answer server, storage). This was the root cause
of the "not a directory" mount failures in CI.
Removed RUNNER_TEMP from the WORK_DIR fallback chain. WORK_DIR now
always defaults to CACHE_DIR/work (/opt/pve-integration/work/) which
is on the shared mount visible to both the CI container and Docker host.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Docker bind mount fails when mounting a file that doesn't exist on
the host (creates a directory instead). In CI, the container's
$WORK_DIR path differs from the host path, so the file mount failed.
Changed to mount a single answer-server/ directory containing both
default.toml and answers/ subdirectory. Replaced two TF variables
(answer_files_dir, default_answer_file) with one (answer_server_dir).
Layout: $WORK_DIR/answer-server/default.toml + answers/<mac>.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Terraform state was stored in the container's working directory
(fresh checkout), so it was lost between CI jobs. The cleanup job
couldn't destroy resources because it had no state.
Now stores state at /opt/pve-integration/work/terraform.tfstate via
-state flag on all terraform commands. This persists across the
provision → test → cleanup job chain in GitHub Actions.
Also:
- Force cleanup now removes state from both local and shared paths
- Added -reconfigure to terraform init (avoids backend mismatch errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix bare Skip-IfNoTarget calls in 13_Firewall and 14_Backup
(missing if/return pattern caused tests to run when they should skip)
- Validate modifier-only switches in dev.ps1 (-Force/-Reprovision
without an action switch now errors instead of defaulting to -Shell)
- Add force-cleanup to usage text in run-integration.sh
- Add --connect-timeout/--max-time to guest agent curl in wait-for-pve.sh
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update header comments: PVETEST_APITOKEN → PVETEST_PASSWORD,
CACHE_DIR default → /opt/pve-integration
- Pin ubuntu and NFS server Docker images to SHA256 digests
- Fix cmd_all to pass version to provision and cleanup
- Keep .terraform.lock.hcl in force cleanup (provider reproducibility)
- Remove || true from terraform destroy in cleanup (propagate errors;
use -Force for best-effort recovery)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add ValidateSet('8','9','all') to dev.ps1 -Version parameter
- Fix Shell warning to reference $DevContainer not $InfraContainer
- Fix Skip-IfNoNodeB to check $PasswordB not $Password
- Pass PVE_TARGET_NODE to wait-for-pve.sh instead of auto-discovering
- Add error default cases to all pve_* helper functions
- Lowercase MAC addresses for answer server matching
- Create answer file paths before terraform destroy in cleanup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot review fixes:
- Pin answer server Docker image to SHA256 digest instead of :latest
for reproducible builds
- Fix cmd_taint: ISO resources are keyed by version ("9") not node
("9a"), so taint was no-op. Now taints ISOs by version and VMs
by node separately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ensures nested PVE nodes are fully patched before integration tests
run. Adds ~5-10min to first provision but gives more realistic test
results against current PVE releases.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WORK_DIR now defaults to /opt/pve-integration/work (under the shared
host mount) instead of /tmp/pve-integration (container-local).
This fixes the answer server seeing empty answer files — the dev-infra
container writes answer files to WORK_DIR, and the answer server
container (a sibling) needs to read them from the same host path.
Also:
- Renamed mount from /opt/pve-isos to /opt/pve-integration
- Added must_run=true, start=true to answer server container
- Updated docker-compose.test.yml and dev.ps1 remote override
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changed Terraform ISO resource from for_each=pve_instances (per-node)
to for_each=pve_isos (per-version). With HTTP answer server, both
nodes of the same version share the same generic ISO.
New pve_isos variable maps version to ISO path. pve_instances now
has pve_version field instead of iso_local_path. VMs reference
their version's ISO via auto_iso[each.value.pve_version].
Updated run-integration.sh tfvars generation and -target flags
for both provision and cleanup to use the new schema.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Force cleanup must remove pvetest-* Docker containers unconditionally,
not just when cleaning all versions. Stale containers cause Terraform
to fail on next provision ("container already exists") since the state
was also wiped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace 4 per-node auto-install ISOs (~4GB) with 2 generic ISOs
(~2GB) served by an HTTP answer server that routes per-node configs
by MAC address.
New flow:
1. Deterministic MAC addresses assigned per node (AA:BB:CC:00:VV:NN)
2. Per-MAC answer.toml files generated in answers/ directory
3. HTTP answer server (slothcroissant/proxmox-auto-installer-server)
managed by Terraform, serves answer files on port 8000
4. Generic ISOs prepared with --fetch-from http --url pointing to
the answer server
5. PVE installer POSTs system info, server matches MAC to answer file
Benefits:
- 50% reduction in ISO disk/tmp usage (2 ISOs instead of 4)
- Faster ISO preparation (2 builds instead of 4)
- Answer files can be updated without rebuilding ISOs
- First-boot script embedded in generic ISO via --on-first-boot
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Terraform state is corrupted (e.g. interrupted provision),
-Cleanup -Force bypasses Terraform and:
1. Destroys VMs via direct PVE API calls (preflight-cleanup.sh)
2. Force-removes Docker storage containers and volumes
3. Deletes Terraform state files so next provision starts clean
Usage: dev.ps1 -Cleanup -Force -DockerHost 172.16.40.113
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When provisioning a subset of versions (-Version 9), the -target flags
only included PVE VM resources. Docker storage containers (iSCSI, NFS)
were skipped because they weren't targeted. Now always includes all
Docker resources in the target list since storage is shared across
all PVE versions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major provisioning pipeline changes:
Terraform:
- Add kreuzwerker/docker provider to manage iSCSI and NFS storage
containers alongside PVE VMs in a single Terraform config
- New storage.tf with Docker container, image, and volume resources
- docker_host_ip variable for PVE nodes to reach storage services
Provisioning:
- Replace docker-compose storage management with Terraform
- Replace create-api-token.sh with wait-for-pve.sh (IP discovery +
API readiness + auth verification only — no token creation)
- Tests use root@pam credentials, not API tokens
Cleanup:
- Replace preflight-cleanup.sh loop with terraform destroy
- Supports version filtering: cleanup 9 destroys only PVE 9 resources
- Full cleanup also removes config.json and tfvars
New commands:
- taint [8|9|all]: marks VMs for recreation on next provision
- dev.ps1 -Reprovision: runs taint before provision to force VM rebuild
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When provisioning a subset of versions (e.g. -Version 9), use
terraform -target to apply only the requested nodes. This prevents
Terraform from destroying VMs for other versions that exist in state.
Key changes:
- Provision: ISOs, answer files, and API token creation only run for
requested version nodes. Tfvars always include ALL versions for
state consistency. -target limits what Terraform applies.
- Cleanup: only destroys VMs for the requested version. Storage
containers only stopped when cleaning all versions.
- Config: merges with existing config.json to preserve entries from
previously provisioned versions.
Based on -target pattern from ~/Source/homelab Makefile.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Provision and cleanup now accept a version argument (8, 9, or all)
to operate on a subset of PVE nodes:
dev.ps1 -Provision -Version 9 -DockerHost ... # only PVE 9 nodes
dev.ps1 -Cleanup -Version 8 -DockerHost ... # only PVE 8 nodes
run-integration.sh provision/cleanup also accept the version arg,
overriding PVE_VERSIONS and ALL_NODES for that invocation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PowerShell's -Command doesn't support the -- argument separator for
param() blocks. Inject $PveVersion and $TestFilter as variable
assignments at the top of the script string instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config file now defaults to $WORK_DIR/config.json instead of
$CACHE_DIR/test-config.json. This keeps test artifacts in
/tmp/pve-integration alongside other build/work files, separate
from the ISO cache in /opt/pve-isos.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cmd_test() now accepts an optional second argument for filtering
integration tests by area name. Comma-separated names are matched
against test filenames via glob (e.g. *Connection*.Tests.ps1).
Usage: run-integration.sh test 9 Connection,VMs
The Pester invocation builds a path array from matched files when
a filter is specified, otherwise runs the entire Integration directory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix storage_ip derivation to use default route first, not Swarm RemoteManagers
- Make iSCSI container entrypoint idempotent (check before create for target/LUN/bind)
- Pin erichough/nfs-server from 'latest' to '2.2.1' for reproducibility
- Replace curl|sh Docker install with apt repo in Dockerfile.test
- Add docker.sock security warnings in docker-compose.test.yml and CI workflow
- Update stale 'storage VM' references in comments and .env.test.example"
Co-authored-by: GoodOlClint <151449+GoodOlClint@users.noreply.github.com>
Agent-Logs-Url: https://github.com/GoodOlClint/PSProxmoxVE/sessions/534c0a63-2988-496c-b0ae-e291fcfa050b
Provision two PVE nodes per version (a/b) for future cluster testing,
plus Docker-based iSCSI target and NFS server for shared storage tests.
Multi-node changes:
- Each PVE version gets two nodes: 9a/9b and 8a/8b (4 VMs total)
- Parameterized answer.toml FQDN for unique hostnames per node
- Per-node auto-install ISOs with unique answer files
- Node name discovered from FQDN and included in test config
- API token creation handles pre-existing tokens (delete + recreate)
- New test env vars: PVETEST_HOST_B, PVETEST_APITOKEN_B
- Removed preflight cleanup from provision (use explicit cleanup instead)
Docker storage services:
- New docker-compose.storage.yml with iSCSI (tgt) and NFS containers
- Host networking so PVE nodes can reach storage services
- Docker socket mounted into dev-infra container for host Docker access
- Docker CLI added to dev-infra Dockerfile stage
- New test env vars: PVETEST_STORAGE_VM_IP, PVETEST_ISCSI_IQN, PVETEST_NFS_EXPORT
Other fixes:
- first-boot.sh installs open-iscsi on PVE nodes
- preflight-cleanup.sh handles empty ISO filename gracefully
- TMPDIR set to work dir to avoid /tmp overflow during ISO uploads
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single source of truth for the provision → test → cleanup lifecycle,
callable from both CI workflows and local dev containers.
Subcommands: provision, test [8|9|all], cleanup, all
Detects CI vs local via $GITHUB_ACTIONS and adjusts output accordingly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CI workflow: PVE_PASSWORD now reads from secrets.PVE_TEST_PASSWORD
- variables.tf: removed default password, requires TF_VAR env var
- Integration README: examples use <your-test-password> placeholder
- create-api-token.sh: example IP changed to pve.example.com
- .gitignore: added .env/.env.* exclusion
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restructures the integration test workflow from a monolithic sequential
job into separate provision → test → cleanup stages:
- provision: creates ALL nested PVE VMs in a single terraform apply
(parallel), waits for APIs, creates tokens, passes outputs to tests
- test: matrix [pve9, pve8] consumes provision outputs, no provisioning
- cleanup: always runs, API-only teardown for all VMs
Terraform refactored to for_each with pve_instances map variable,
enabling parallel ISO upload and VM creation.
New caching scripts reduce redundant downloads:
- ensure-base-iso.sh: downloads PVE ISOs to /opt/pve-isos if missing
- ensure-cloud-images.sh: caches cloud image + OVA with 7-day TTL
- prepare-auto-iso.sh: --cache-dir flag with hash-based skip
Runner no longer needs manual ISO provisioning (zero-touch setup).
cleanup-images bumped to min-versions-to-keep: 3 to survive overlapping
runs. All jobs gated against dependabot.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All VM and container lifecycle cmdlets (Start, Stop, Restart, Suspend,
Resume, Reset) now poll VM/container status when -Wait is specified,
blocking until the expected state is reached or -Timeout (default 60s)
is exceeded.
Implementation:
- PveCmdletBase.WaitForStatusTransition() — shared helper that waits
for PVE task completion then polls status via API
- 9 cmdlets updated: 6 VM (Start, Stop, Restart, Suspend, Resume,
Reset) + 3 container (Start, Stop, Restart)
- -Timeout parameter with [ValidateRange(1, 3600)] on each
Integration tests:
- Replace all manual Start-Sleep + polling loops with -Wait -Timeout
- Switch to real Ubuntu cloud OVA (571 MB) for Import-PveOva testing
- OVA test verifies full import + VM start
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
goodolclint-claude[bot]