RUNNER_TEMP (/__w/_temp in CI) is container-local and invisible to
the Docker host. Files written there can't be bind-mounted into
sibling containers (answer server, storage). This was the root cause
of the "not a directory" mount failures in CI.
Removed RUNNER_TEMP from the WORK_DIR fallback chain. WORK_DIR now
always defaults to CACHE_DIR/work (/opt/pve-integration/work/) which
is on the shared mount visible to both the CI container and Docker host.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Docker bind mount fails when mounting a file that doesn't exist on
the host (creates a directory instead). In CI, the container's
$WORK_DIR path differs from the host path, so the file mount failed.
Changed to mount a single answer-server/ directory containing both
default.toml and answers/ subdirectory. Replaced two TF variables
(answer_files_dir, default_answer_file) with one (answer_server_dir).
Layout: $WORK_DIR/answer-server/default.toml + answers/<mac>.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Terraform state was stored in the container's working directory
(fresh checkout), so it was lost between CI jobs. The cleanup job
couldn't destroy resources because it had no state.
Now stores state at /opt/pve-integration/work/terraform.tfstate via
-state flag on all terraform commands. This persists across the
provision → test → cleanup job chain in GitHub Actions.
Also:
- Force cleanup now removes state from both local and shared paths
- Added -reconfigure to terraform init (avoids backend mismatch errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix bare Skip-IfNoTarget calls in 13_Firewall and 14_Backup
(missing if/return pattern caused tests to run when they should skip)
- Validate modifier-only switches in dev.ps1 (-Force/-Reprovision
without an action switch now errors instead of defaulting to -Shell)
- Add force-cleanup to usage text in run-integration.sh
- Add --connect-timeout/--max-time to guest agent curl in wait-for-pve.sh
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update header comments: PVETEST_APITOKEN → PVETEST_PASSWORD,
CACHE_DIR default → /opt/pve-integration
- Pin ubuntu and NFS server Docker images to SHA256 digests
- Fix cmd_all to pass version to provision and cleanup
- Keep .terraform.lock.hcl in force cleanup (provider reproducibility)
- Remove || true from terraform destroy in cleanup (propagate errors;
use -Force for best-effort recovery)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add ValidateSet('8','9','all') to dev.ps1 -Version parameter
- Fix Shell warning to reference $DevContainer not $InfraContainer
- Fix Skip-IfNoNodeB to check $PasswordB not $Password
- Pass PVE_TARGET_NODE to wait-for-pve.sh instead of auto-discovering
- Add error default cases to all pve_* helper functions
- Lowercase MAC addresses for answer server matching
- Create answer file paths before terraform destroy in cleanup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot review fixes:
- Pin answer server Docker image to SHA256 digest instead of :latest
for reproducible builds
- Fix cmd_taint: ISO resources are keyed by version ("9") not node
("9a"), so taint was no-op. Now taints ISOs by version and VMs
by node separately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ensures nested PVE nodes are fully patched before integration tests
run. Adds ~5-10min to first provision but gives more realistic test
results against current PVE releases.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WORK_DIR now defaults to /opt/pve-integration/work (under the shared
host mount) instead of /tmp/pve-integration (container-local).
This fixes the answer server seeing empty answer files — the dev-infra
container writes answer files to WORK_DIR, and the answer server
container (a sibling) needs to read them from the same host path.
Also:
- Renamed mount from /opt/pve-isos to /opt/pve-integration
- Added must_run=true, start=true to answer server container
- Updated docker-compose.test.yml and dev.ps1 remote override
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changed Terraform ISO resource from for_each=pve_instances (per-node)
to for_each=pve_isos (per-version). With HTTP answer server, both
nodes of the same version share the same generic ISO.
New pve_isos variable maps version to ISO path. pve_instances now
has pve_version field instead of iso_local_path. VMs reference
their version's ISO via auto_iso[each.value.pve_version].
Updated run-integration.sh tfvars generation and -target flags
for both provision and cleanup to use the new schema.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Force cleanup must remove pvetest-* Docker containers unconditionally,
not just when cleaning all versions. Stale containers cause Terraform
to fail on next provision ("container already exists") since the state
was also wiped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace 4 per-node auto-install ISOs (~4GB) with 2 generic ISOs
(~2GB) served by an HTTP answer server that routes per-node configs
by MAC address.
New flow:
1. Deterministic MAC addresses assigned per node (AA:BB:CC:00:VV:NN)
2. Per-MAC answer.toml files generated in answers/ directory
3. HTTP answer server (slothcroissant/proxmox-auto-installer-server)
managed by Terraform, serves answer files on port 8000
4. Generic ISOs prepared with --fetch-from http --url pointing to
the answer server
5. PVE installer POSTs system info, server matches MAC to answer file
Benefits:
- 50% reduction in ISO disk/tmp usage (2 ISOs instead of 4)
- Faster ISO preparation (2 builds instead of 4)
- Answer files can be updated without rebuilding ISOs
- First-boot script embedded in generic ISO via --on-first-boot
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Terraform state is corrupted (e.g. interrupted provision),
-Cleanup -Force bypasses Terraform and:
1. Destroys VMs via direct PVE API calls (preflight-cleanup.sh)
2. Force-removes Docker storage containers and volumes
3. Deletes Terraform state files so next provision starts clean
Usage: dev.ps1 -Cleanup -Force -DockerHost 172.16.40.113
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When provisioning a subset of versions (-Version 9), the -target flags
only included PVE VM resources. Docker storage containers (iSCSI, NFS)
were skipped because they weren't targeted. Now always includes all
Docker resources in the target list since storage is shared across
all PVE versions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major provisioning pipeline changes:
Terraform:
- Add kreuzwerker/docker provider to manage iSCSI and NFS storage
containers alongside PVE VMs in a single Terraform config
- New storage.tf with Docker container, image, and volume resources
- docker_host_ip variable for PVE nodes to reach storage services
Provisioning:
- Replace docker-compose storage management with Terraform
- Replace create-api-token.sh with wait-for-pve.sh (IP discovery +
API readiness + auth verification only — no token creation)
- Tests use root@pam credentials, not API tokens
Cleanup:
- Replace preflight-cleanup.sh loop with terraform destroy
- Supports version filtering: cleanup 9 destroys only PVE 9 resources
- Full cleanup also removes config.json and tfvars
New commands:
- taint [8|9|all]: marks VMs for recreation on next provision
- dev.ps1 -Reprovision: runs taint before provision to force VM rebuild
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When provisioning a subset of versions (e.g. -Version 9), use
terraform -target to apply only the requested nodes. This prevents
Terraform from destroying VMs for other versions that exist in state.
Key changes:
- Provision: ISOs, answer files, and API token creation only run for
requested version nodes. Tfvars always include ALL versions for
state consistency. -target limits what Terraform applies.
- Cleanup: only destroys VMs for the requested version. Storage
containers only stopped when cleaning all versions.
- Config: merges with existing config.json to preserve entries from
previously provisioned versions.
Based on -target pattern from ~/Source/homelab Makefile.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Provision and cleanup now accept a version argument (8, 9, or all)
to operate on a subset of PVE nodes:
dev.ps1 -Provision -Version 9 -DockerHost ... # only PVE 9 nodes
dev.ps1 -Cleanup -Version 8 -DockerHost ... # only PVE 8 nodes
run-integration.sh provision/cleanup also accept the version arg,
overriding PVE_VERSIONS and ALL_NODES for that invocation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PowerShell's -Command doesn't support the -- argument separator for
param() blocks. Inject $PveVersion and $TestFilter as variable
assignments at the top of the script string instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config file now defaults to $WORK_DIR/config.json instead of
$CACHE_DIR/test-config.json. This keeps test artifacts in
/tmp/pve-integration alongside other build/work files, separate
from the ISO cache in /opt/pve-isos.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cmd_test() now accepts an optional second argument for filtering
integration tests by area name. Comma-separated names are matched
against test filenames via glob (e.g. *Connection*.Tests.ps1).
Usage: run-integration.sh test 9 Connection,VMs
The Pester invocation builds a path array from matched files when
a filter is specified, otherwise runs the entire Integration directory.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix storage_ip derivation to use default route first, not Swarm RemoteManagers
- Make iSCSI container entrypoint idempotent (check before create for target/LUN/bind)
- Pin erichough/nfs-server from 'latest' to '2.2.1' for reproducibility
- Replace curl|sh Docker install with apt repo in Dockerfile.test
- Add docker.sock security warnings in docker-compose.test.yml and CI workflow
- Update stale 'storage VM' references in comments and .env.test.example"
Co-authored-by: GoodOlClint <151449+GoodOlClint@users.noreply.github.com>
Agent-Logs-Url: https://github.com/GoodOlClint/PSProxmoxVE/sessions/534c0a63-2988-496c-b0ae-e291fcfa050b
Provision two PVE nodes per version (a/b) for future cluster testing,
plus Docker-based iSCSI target and NFS server for shared storage tests.
Multi-node changes:
- Each PVE version gets two nodes: 9a/9b and 8a/8b (4 VMs total)
- Parameterized answer.toml FQDN for unique hostnames per node
- Per-node auto-install ISOs with unique answer files
- Node name discovered from FQDN and included in test config
- API token creation handles pre-existing tokens (delete + recreate)
- New test env vars: PVETEST_HOST_B, PVETEST_APITOKEN_B
- Removed preflight cleanup from provision (use explicit cleanup instead)
Docker storage services:
- New docker-compose.storage.yml with iSCSI (tgt) and NFS containers
- Host networking so PVE nodes can reach storage services
- Docker socket mounted into dev-infra container for host Docker access
- Docker CLI added to dev-infra Dockerfile stage
- New test env vars: PVETEST_STORAGE_VM_IP, PVETEST_ISCSI_IQN, PVETEST_NFS_EXPORT
Other fixes:
- first-boot.sh installs open-iscsi on PVE nodes
- preflight-cleanup.sh handles empty ISO filename gracefully
- TMPDIR set to work dir to avoid /tmp overflow during ISO uploads
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single source of truth for the provision → test → cleanup lifecycle,
callable from both CI workflows and local dev containers.
Subcommands: provision, test [8|9|all], cleanup, all
Detects CI vs local via $GITHUB_ACTIONS and adjusts output accordingly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CI workflow: PVE_PASSWORD now reads from secrets.PVE_TEST_PASSWORD
- variables.tf: removed default password, requires TF_VAR env var
- Integration README: examples use <your-test-password> placeholder
- create-api-token.sh: example IP changed to pve.example.com
- .gitignore: added .env/.env.* exclusion
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restructures the integration test workflow from a monolithic sequential
job into separate provision → test → cleanup stages:
- provision: creates ALL nested PVE VMs in a single terraform apply
(parallel), waits for APIs, creates tokens, passes outputs to tests
- test: matrix [pve9, pve8] consumes provision outputs, no provisioning
- cleanup: always runs, API-only teardown for all VMs
Terraform refactored to for_each with pve_instances map variable,
enabling parallel ISO upload and VM creation.
New caching scripts reduce redundant downloads:
- ensure-base-iso.sh: downloads PVE ISOs to /opt/pve-isos if missing
- ensure-cloud-images.sh: caches cloud image + OVA with 7-day TTL
- prepare-auto-iso.sh: --cache-dir flag with hash-based skip
Runner no longer needs manual ISO provisioning (zero-touch setup).
cleanup-images bumped to min-versions-to-keep: 3 to survive overlapping
runs. All jobs gated against dependabot.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All VM and container lifecycle cmdlets (Start, Stop, Restart, Suspend,
Resume, Reset) now poll VM/container status when -Wait is specified,
blocking until the expected state is reached or -Timeout (default 60s)
is exceeded.
Implementation:
- PveCmdletBase.WaitForStatusTransition() — shared helper that waits
for PVE task completion then polls status via API
- 9 cmdlets updated: 6 VM (Start, Stop, Restart, Suspend, Resume,
Reset) + 3 container (Start, Stop, Restart)
- -Timeout parameter with [ValidateRange(1, 3600)] on each
Integration tests:
- Replace all manual Start-Sleep + polling loops with -Wait -Timeout
- Switch to real Ubuntu cloud OVA (571 MB) for Import-PveOva testing
- OVA test verifies full import + VM start
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Suspend/Resume: move to Linux VM lifecycle context (empty VMs may
not reliably transition to paused state)
- SDN subnet model: dhcp-range is an array, not string
- SDN subnet delete: use PVE-format ID (vnet-ip-prefix) not CIDR
- Container config: trim trailing newline from PVE description
- OVA: use qemu-img to create valid sparse VMDK (add qemu-utils to
test container Dockerfile)
- Enable 'images' content type on local storage for OVA import
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SDN: zone ID cannot contain hyphens — rename 'pester-zone' to 'pesterz'
- SDN: subnet removal uses original CIDR, not transformed ID
- Containers: New-PveContainer now auto-assigns VM ID via /cluster/nextid
when -VmId is omitted (was missing, unlike New-PveVm)
- OVA: enable 'images' content type on local storage for import-from
- Suspend: add delay before status check, use Should -Not -Throw pattern
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVE upload API validates file extensions per content type. The
content=import type does not accept .img extension. The Ubuntu cloud
image is qcow2 format, so rename from .img to .qcow2 during download.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Send-PveFile replaces Send-PveIso with a -ContentType parameter
(iso, vztmpl, import). This is a breaking rename — the module has
not been released yet.
Test infrastructure refactored:
- prepare-test-vm.sh replaced with prepare-test-environment.sh
(only SSH ops: enable content types, upload cloud-init snippet,
curl download cloud image)
- VM provisioning moved into integration tests as "Linux VM —
Provisioning" context: Send-PveFile, New-PveVm, Import-PveVmDisk,
Set-PveVmConfig, Set-PveCloudInitConfig, Start-PveVm,
Test-PveVmGuestAgent — all tested as part of the test suite
- PVETEST_LINUX_VMID env var removed (VM created by tests)
- PVETEST_CLOUD_IMAGE_PATH + PVETEST_PASSWORD added
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Absolute filesystem paths are rejected by PVE API when using API token
auth (only root ticket auth allows arbitrary paths). Instead:
- Enable 'import' content type on local storage alongside snippets
- Download cloud image with content=import (not iso)
- Reference as local:import/filename in Import-PveVmDisk
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The cloud image is downloaded as content=iso, but import-from requires
content type 'images' or 'import'. Use the absolute node path
(/var/lib/vz/template/iso/) instead of the storage volume reference
(local:iso/) to work around this constraint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Imports disk images (qcow2, raw, vmdk) and OVA archives into VMs
via the PVE API using POST config with import-from syntax.
- VmService.ImportDisk() method using POST /nodes/{node}/qemu/{vmid}/config
- Import-PveVmDisk cmdlet with -Source, -Disk, -TargetStorage, -Format, -Wait
- Pester unit tests
- prepare-test-vm.sh updated to use Import-PveVmDisk instead of SSH qm importdisk
- Set-PveNetwork: add required Type parameter (fixes PVE API 400 error)
- Integration test fix: pass -Type bridge to Set-PveNetwork
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The cmdlet calls the /cloudinit/dump endpoint which returns YAML
content, not a task UPID. This gets passed to WaitForTask causing a
501 error. PVE regenerates cloud-init automatically on VM start, so
the explicit regenerate call is not needed for test VM setup.
Bug to fix: RegenerateCloudInitImage should PUT to /cloudinit instead
of GET from /cloudinit/dump.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Debian .qcow2 extension was rejected by PVE's download-url API.
Ubuntu's noble-server-cloudimg-amd64.img has a .img extension that
PVE accepts, and cloud-init + qemu-guest-agent work the same way.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace raw qm/SSH commands with module cmdlets where possible:
- Invoke-PveStorageDownload for cloud image download
- New-PveVm for VM creation
- Set-PveCloudInitConfig for cloud-init user/password/IP
- Invoke-PveCloudInitRegenerate to apply config
- Start-PveVm to boot the VM
SSH/SCP retained only for operations without API support:
- Snippet upload (no snippet API)
- pvesm set for snippets content type
- qm importdisk (no importdisk cmdlet)
- qm set for --scsi0, --cicustom, --agent, --boot (not in Set-PveVmConfig)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace virt-customize with a cloud-init custom user-data snippet that
installs qemu-guest-agent on first boot. The snippet is SCP'd to the
nested PVE's /var/lib/vz/snippets/ and referenced via --cicustom.
This eliminates the need for libguestfs-tools entirely — no install
on nested PVE, no extra Docker image, no large dependencies.
Timeout increased to 300s to allow for cloud-init package installation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- prepare-test-vm.sh: install libguestfs-tools if not present on the
fresh nested PVE before running virt-customize
- Mask PVE_PASSWORD in logs via ::add-mask:: to prevent leaking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move virt-customize to run on the nested PVE host (which already has
libguestfs) instead of inside the CI container. This removes
libguestfs-tools, linux-image-generic, and qemu-utils from the
Dockerfile, shrinking the image from ~1.2GB to ~770MB.
- prepare-test-vm.sh: downloads Debian cloud image on nested PVE,
runs virt-customize there to install qemu-guest-agent, then creates
VM with imported disk + cloud-init
- Remove build-test-image.sh (no longer needed)
- Remove /opt/pve-images volume mount
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Infrastructure:
- build-test-image.sh: Downloads Alpine cloud image, uses virt-customize
to install qemu-guest-agent, caches at /opt/pve-images/
- prepare-test-vm.sh: SCPs image to nested PVE, creates VM via
qm importdisk with cloud-init and agent enabled, waits for agent
Workflow:
- Build Alpine image step (cached on runner, only built once)
- Deploy test VM step after provisioning, passes LINUX_VMID to tests
Tests:
- Guest Agent VM lifecycle: verify running, ACPI restart, ACPI stop
- Templates: convert Linux VM to template, clone from template
(New-PveTemplate, New-PveVmFromTemplate)
- Fix Get-PveTask: use .Upid property instead of whole PveTask object
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVE 8 (Bookworm) may not have /etc/apt/sources.list.d/debian.sources,
causing the grep fallback to use 'trixie' which is invalid for PVE 8.
Use /etc/os-release VERSION_CODENAME which works on both PVE 8 and 9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
/tmp on the self-hosted runner has sticky bit — files created by
previous runs (or different users) can't be overwritten. RUNNER_TEMP
is per-run and owned by the runner user.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The proxmox-auto-install-assistant defaults --tmp to the source ISO's
directory (/opt/pve-isos/), which the runner user can't write to.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- first-boot.sh: Remove .sources files (PVE 9 DEB822 format) not just
.list, and detect suite (trixie vs bookworm) from debian.sources
- create-api-token.sh: Accept full endpoint URL (fixes missing port),
use /access/domains for API check (no auth needed on PVE 9),
avoid pipefail on curl|python3 pipe
- preflight-cleanup.sh: New script that runs at start and end of every
provisioning run — cleans orphaned VMs, ISOs, and stale terraform state
- Workflow: Add pre-flight and final cleanup steps so failed runs can't
leave the runner/host in a broken state
All changes validated manually on the self-hosted runner.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use proxmox_virtual_environment_file resource to upload the auto-install
ISO via the API. terraform destroy now cleans up both the VM and ISO.
- Remove manual upload-to-pve.sh and cleanup-pve-storage.sh scripts.
- Commit .terraform.lock.hcl for reproducible CI builds.
- Replace iso_file_id variable with iso_local_path and iso_storage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Workflow now provisions a throwaway nested PVE VM via Terraform, runs
integration tests against it, then destroys it. Uses proxmox-auto-install-assistant
to bake the answer file and a first-boot script (installs qemu-guest-agent)
directly into the ISO. IP is discovered via the QEMU guest agent on the
parent PVE, eliminating the need for static IP configuration.
Supports both PVE 8.x and 9.x via workflow_dispatch version selector.
Skip provisioning with skip_provision=true to test against a pre-existing PVE.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Provisions a throwaway nested Proxmox VE instance on an existing host
using bpg/proxmox provider. Includes unattended install via answer file,
API wait script, and automatic API token creation. Designed for CI-driven
integration test environments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clint Branham