PVE 9 rejects rules referencing unmanaged resources. Test now creates
a minimal VM, registers it as a disabled HA resource, then creates
the node-affinity rule. Cleanup removes rule, resource, and VM.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PVE 9 rejects 'affinity' as an unexpected property on POST
cluster/ha/rules — it's implied by the node-affinity type.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Full lifecycle: create node-affinity rule, list, get by ID, update
comment, delete, and verify deletion. Uses vm:99999 as a synthetic
resource SID. Tests skip on PVE 8 (rules not available).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cluster tests create a 2-node cluster that cannot be torn down via API
(quorum loss). Prefix with ZZ so they execute after all other
integration tests that assume standalone nodes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removing a node from a 2-node PVE cluster via REST API is not supported
— the remaining node loses quorum mid-operation, causing "no quorum!"
errors. PVE requires stopping corosync on the departing node first
(pvecm expected 1), which is not available via the REST API.
Replace removal tests with a final cluster state verification.
Test infrastructure handles cleanup via reprovisioning.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove-PveClusterConfigNode requires root@pam (not API token).
Fixed both the test context and AfterAll cleanup to connect with
root@pam credentials before attempting node removal.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
.NET's FormUrlEncodedContent over-encodes characters like : and !
(%3A, %21) in form values. PVE's internal API consumers (specifically
the cluster join process) do not URL-decode these values before using
them, causing fingerprint comparison failures and password mismatches.
Replace with BuildFormContent/EncodeFormValue that only encodes
characters that break form parsing (&, =, +, space, %). This matches
curl's -d behavior and fixes cluster join "Cluster join aborted!" errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract re-auth retry into ReauthenticateWithRetry helper (fixes
cognitive complexity warning)
- Retry auth up to 10x with 3s delay — node B's auth services need
time to restart after joining the cluster
- Wrap join in try/finally so the test always reconnects to node A,
even if the join fails
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cluster join restarts auth services on the joining node, which
invalidates the PVE ticket mid-poll. When WaitForTask gets a 401,
catch it and re-authenticate with the password already available
in the cmdlet, then retry the wait.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All cluster operations (create, status, join, options, cleanup) require
root@pam ticket auth — API tokens lack Sys.Audit and Sys.Modify on /.
Changed test to connect as root@pam once during cluster creation and
stay on that session for the entire lifecycle.
Removed redundant Connect-PveServer calls that switched between
root@pam and API token between contexts.
Added 5s stabilization sleep after cluster creation for corosync.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Convert JoinInfo.Nodelist to native PS array via ConvertFrom-Json
(JArray .Item() still hits IEnumerator error in PowerShell)
- Use root@pam for cluster options context (delete requires Sys.Modify)
- Reconnect with API token after options restore
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
xUnit (47 new tests, 429 total):
- ClusterConfigServiceTests: 25 tests covering all 14 service methods
including URL encoding, null guards, and error responses
- HaServiceTests: 22 tests covering resources, groups, status, and rules
with URI encoding verification (vm:100 → vm%3A100)
Pester (187 new tests, 1525 total):
- ClusterConfigCmdlets.Tests.ps1: 11 cmdlets tested
- HaCmdlets.Tests.ps1: 14 cmdlets tested
Integration (ClusterConfig.Integration.Tests.ps1):
- Full 2-node cluster lifecycle with -Wait for task completion
- Uses root@pam ticket auth for cluster create/join operations
- HA group tests skip on PVE 9.0+ (groups migrated to rules)
- JArray indexing uses .Item() for PowerShell compatibility
Cmdlet improvements:
- New-PveCluster, Add-PveClusterConfigNode, Add-PveClusterMember now
support -Wait switch to block until task completes (via TaskService)
- GetClusterConfig returns JToken to handle array responses on standalone
- OutputType updated to PveTask for task-returning cmdlets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RUNNER_TEMP (/__w/_temp in CI) is container-local and invisible to
the Docker host. Files written there can't be bind-mounted into
sibling containers (answer server, storage). This was the root cause
of the "not a directory" mount failures in CI.
Removed RUNNER_TEMP from the WORK_DIR fallback chain. WORK_DIR now
always defaults to CACHE_DIR/work (/opt/pve-integration/work/) which
is on the shared mount visible to both the CI container and Docker host.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mounting the whole directory to /app overwrites server.py inside the
container, causing "can't open file '/app/server.py'" errors.
Mount the two paths individually instead:
- answer_server_dir/answers → /app/answers
- answer_server_dir/default.toml → /app/default.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Docker bind mount fails when mounting a file that doesn't exist on
the host (creates a directory instead). In CI, the container's
$WORK_DIR path differs from the host path, so the file mount failed.
Changed to mount a single answer-server/ directory containing both
default.toml and answers/ subdirectory. Replaced two TF variables
(answer_files_dir, default_answer_file) with one (answer_server_dir).
Layout: $WORK_DIR/answer-server/default.toml + answers/<mac>.toml
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI cleanup should always succeed regardless of Terraform state
(cancelled runs, timeouts, partial failures). force-cleanup uses
direct API calls + docker rm instead of terraform destroy.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Terraform state was stored in the container's working directory
(fresh checkout), so it was lost between CI jobs. The cleanup job
couldn't destroy resources because it had no state.
Now stores state at /opt/pve-integration/work/terraform.tfstate via
-state flag on all terraform commands. This persists across the
provision → test → cleanup job chain in GitHub Actions.
Also:
- Force cleanup now removes state from both local and shared paths
- Added -reconfigure to terraform init (avoids backend mismatch errors)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Terraform creates resources in parallel. Without depends_on, PVE VMs
could boot and start the auto-installer before the HTTP answer server
container is running, causing "could not find answer file" errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix bare Skip-IfNoTarget calls in 13_Firewall and 14_Backup
(missing if/return pattern caused tests to run when they should skip)
- Validate modifier-only switches in dev.ps1 (-Force/-Reprovision
without an action switch now errors instead of defaulting to -Shell)
- Add force-cleanup to usage text in run-integration.sh
- Add --connect-timeout/--max-time to guest agent curl in wait-for-pve.sh
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update header comments: PVETEST_APITOKEN → PVETEST_PASSWORD,
CACHE_DIR default → /opt/pve-integration
- Pin ubuntu and NFS server Docker images to SHA256 digests
- Fix cmd_all to pass version to provision and cleanup
- Keep .terraform.lock.hcl in force cleanup (provider reproducibility)
- Remove || true from terraform destroy in cleanup (propagate errors;
use -Force for best-effort recovery)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add ValidateSet('8','9','all') to dev.ps1 -Version parameter
- Fix Shell warning to reference $DevContainer not $InfraContainer
- Fix Skip-IfNoNodeB to check $PasswordB not $Password
- Pass PVE_TARGET_NODE to wait-for-pve.sh instead of auto-discovering
- Add error default cases to all pve_* helper functions
- Lowercase MAC addresses for answer server matching
- Create answer file paths before terraform destroy in cleanup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot review fixes:
- Pin answer server Docker image to SHA256 digest instead of :latest
for reproducible builds
- Fix cmd_taint: ISO resources are keyed by version ("9") not node
("9a"), so taint was no-op. Now taints ISOs by version and VMs
by node separately.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test was still expecting GetAsync on cloudinit/dump but the service
now calls PutAsync on cloudinit (regenerate endpoint). Updated mock
setup and assertions to match.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ensures nested PVE nodes are fully patched before integration tests
run. Adds ~5-10min to first provision but gives more realistic test
results against current PVE releases.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
WORK_DIR now defaults to /opt/pve-integration/work (under the shared
host mount) instead of /tmp/pve-integration (container-local).
This fixes the answer server seeing empty answer files — the dev-infra
container writes answer files to WORK_DIR, and the answer server
container (a sibling) needs to read them from the same host path.
Also:
- Renamed mount from /opt/pve-isos to /opt/pve-integration
- Added must_run=true, start=true to answer server container
- Updated docker-compose.test.yml and dev.ps1 remote override
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changed Terraform ISO resource from for_each=pve_instances (per-node)
to for_each=pve_isos (per-version). With HTTP answer server, both
nodes of the same version share the same generic ISO.
New pve_isos variable maps version to ISO path. pve_instances now
has pve_version field instead of iso_local_path. VMs reference
their version's ISO via auto_iso[each.value.pve_version].
Updated run-integration.sh tfvars generation and -target flags
for both provision and cleanup to use the new schema.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Force cleanup wipes all Terraform state, so filtering by version
would leave an inconsistent state where subsequent provisions fail.
Error early with a clear message instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Force cleanup must remove pvetest-* Docker containers unconditionally,
not just when cleaning all versions. Stale containers cause Terraform
to fail on next provision ("container already exists") since the state
was also wiped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace 4 per-node auto-install ISOs (~4GB) with 2 generic ISOs
(~2GB) served by an HTTP answer server that routes per-node configs
by MAC address.
New flow:
1. Deterministic MAC addresses assigned per node (AA:BB:CC:00:VV:NN)
2. Per-MAC answer.toml files generated in answers/ directory
3. HTTP answer server (slothcroissant/proxmox-auto-installer-server)
managed by Terraform, serves answer files on port 8000
4. Generic ISOs prepared with --fetch-from http --url pointing to
the answer server
5. PVE installer POSTs system info, server matches MAC to answer file
Benefits:
- 50% reduction in ISO disk/tmp usage (2 ISOs instead of 4)
- Faster ISO preparation (2 builds instead of 4)
- Answer files can be updated without rebuilding ISOs
- First-boot script embedded in generic ISO via --on-first-boot
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Terraform state is corrupted (e.g. interrupted provision),
-Cleanup -Force bypasses Terraform and:
1. Destroys VMs via direct PVE API calls (preflight-cleanup.sh)
2. Force-removes Docker storage containers and volumes
3. Deletes Terraform state files so next provision starts clean
Usage: dev.ps1 -Cleanup -Force -DockerHost 172.16.40.113
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When provisioning a subset of versions (-Version 9), the -target flags
only included PVE VM resources. Docker storage containers (iSCSI, NFS)
were skipped because they weren't targeted. Now always includes all
Docker resources in the target list since storage is shared across
all PVE versions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Major provisioning pipeline changes:
Terraform:
- Add kreuzwerker/docker provider to manage iSCSI and NFS storage
containers alongside PVE VMs in a single Terraform config
- New storage.tf with Docker container, image, and volume resources
- docker_host_ip variable for PVE nodes to reach storage services
Provisioning:
- Replace docker-compose storage management with Terraform
- Replace create-api-token.sh with wait-for-pve.sh (IP discovery +
API readiness + auth verification only — no token creation)
- Tests use root@pam credentials, not API tokens
Cleanup:
- Replace preflight-cleanup.sh loop with terraform destroy
- Supports version filtering: cleanup 9 destroys only PVE 9 resources
- Full cleanup also removes config.json and tfvars
New commands:
- taint [8|9|all]: marks VMs for recreation on next provision
- dev.ps1 -Reprovision: runs taint before provision to force VM rebuild
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RegenerateCloudInitImage was calling GET /cloudinit/dump?type=user
which returns the cloud-init YAML content, not a task UPID. The
cmdlet then passed this YAML string to WaitForTask, causing a 501
error trying to poll a URI like "GET nodes/.../tasks/%23cloud-config..."
Fixed to call PUT /nodes/{node}/qemu/{vmid}/cloudinit which is the
correct regeneration endpoint that returns a UPID.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When provisioning a subset of versions (e.g. -Version 9), use
terraform -target to apply only the requested nodes. This prevents
Terraform from destroying VMs for other versions that exist in state.
Key changes:
- Provision: ISOs, answer files, and API token creation only run for
requested version nodes. Tfvars always include ALL versions for
state consistency. -target limits what Terraform applies.
- Cleanup: only destroys VMs for the requested version. Storage
containers only stopped when cleaning all versions.
- Config: merges with existing config.json to preserve entries from
previously provisioned versions.
Based on -target pattern from ~/Source/homelab Makefile.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Provision and cleanup now accept a version argument (8, 9, or all)
to operate on a subset of PVE nodes:
dev.ps1 -Provision -Version 9 -DockerHost ... # only PVE 9 nodes
dev.ps1 -Cleanup -Version 8 -DockerHost ... # only PVE 8 nodes
run-integration.sh provision/cleanup also accept the version arg,
overriding PVE_VERSIONS and ALL_NODES for that invocation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each test file now creates and cleans up its own resources:
- 06_VMs: restores own AfterAll cleanup for pester-test-vm
- 07_Snapshots: creates pester-snap-vm, tests snapshots, cleans up
- 09_CloudInit: creates pester-ci-vm with cloud-init drive, cleans up
- 15_Tasks: creates pester-task-vm, tests task CRUD, cleans up
Removed cross-file dependency helpers (Find-TestVm, Register-TestResource)
from _IntegrationHelper.ps1 — no longer needed.
Each file can now run independently via -Tests filter. If a Setup
context fails, subsequent tests in the same file skip gracefully.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
06_VMs AfterAll was deleting pester-test-vm, causing 07_Snapshots,
09_CloudInit, and 15_Tasks to fail with VMID 100 (wrong VM or
nonexistent). Now 99_Cleanup handles all pester-* resource removal.
Also fixed cleanup timing:
- Use -Wait on Stop-PveVm/Container before Remove
- Add sleep after removal for API propagation
- 10_Containers AfterAll uses -Wait on stop
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PowerShell's -Command doesn't support the -- argument separator for
param() blocks. Inject $PveVersion and $TestFilter as variable
assignments at the top of the script string instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config file now defaults to $WORK_DIR/config.json instead of
$CACHE_DIR/test-config.json. This keeps test artifacts in
/tmp/pve-integration alongside other build/work files, separate
from the ISO cache in /opt/pve-isos.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Config file is at /opt/pve-isos/test-config.json (CACHE_DIR default),
not /tmp/pve-integration/config.json.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clint Branham