PSProxmoxVE/docs/review/PLAN-integration-refactor.md

Plan: Integration Test Workflow Refactoring

Status: Planned (not started) Created: 2026-03-23 Priority: Medium — improves CI speed and reliability but not blocking

Goals

1. Parallel VM provisioning — provision PVE 8 and PVE 9 VMs simultaneously instead of sequentially
2. File caching — cache ISOs, cloud images, and OVA files on the self-hosted runner
3. Zero-touch runner setup — runner no longer needs manual ISO provisioning
4. Extensibility — easy to add cluster peers, storage VMs in the future

Proposed Job Graph

build ─────────────┐
                    ├──→ provision ──→ test-pve8 ──┐
container-image ───┘     (all VMs      test-pve9 ──┤──→ cleanup
                          parallel)    test-cluster─┘    (always)
                                       (future)

Changes Required

1. Terraform: for_each multi-VM provisioning

• Refactor main.tf from single-VM to pve_instances map variable
• Both ISOs upload and both VMs create in a single terraform apply
• Update variables.tf and outputs.tf

2. New caching scripts

• ensure-base-iso.sh — download PVE base ISO if not cached in /opt/pve-isos
• ensure-cloud-images.sh — download cloud image + OVA with ETag-based 7-day TTL
• Modify prepare-auto-iso.sh — add --cache-dir flag with hash-based skip

3. New provision job (self-hosted)

• Preflight cleanup for ALL VM IDs
• Prepare both auto-install ISOs (cached)
• Download cloud images once (cached)
• Single terraform apply
• Wait for both PVE installs + create API tokens
• Expose connection details as job outputs

4. Test jobs consume provision outputs

• Matrix with max-parallel: 2
• Tests get PVETEST_HOST, PVETEST_APITOKEN from needs.provision.outputs.*
• No provisioning in test jobs

5. Cleanup job with if: always()

• API-only cleanup via preflight-cleanup.sh (stateless)
• Terraform state as artifact for belt-and-suspenders

Estimated Time Savings

Phase	Current	Proposed
Build + container	5-10 min	5-10 min
Provision (sequential → parallel)	20-30 min	10-15 min
Tests (sequential → parallel if runner allows)	10-20 min	5-10 min
Teardown	4-6 min	3-5 min
Total	~40-60 min	~25-40 min

Dependabot / CI Isolation Lessons (2026-03-23)

During scan-6 we discovered two issues that this refactoring must account for:

1. cleanup-images must be gated against dependabot

The cleanup-images job uses actions/delete-package-versions with if: always() to prune old GHCR container images. When a dependabot PR ran, the integration job was skipped (correctly), but cleanup-images still fired and deleted the container image that the main branch integration run was actively using, causing PVE 8 to fail with "image not found".

Fix already applied: All jobs now have if: github.actor != 'dependabot[bot]'.

For the refactoring: The new cleanup job (Terraform destroy + VM cleanup) must also be gated. Use if: always() && github.actor != 'dependabot[bot]' on all jobs that touch shared resources (self-hosted runner, GHCR, PVE host).

2. Container image tags must survive concurrent cleanup

The current cleanup-images job deletes all but 1 container image version. If two workflow runs overlap (e.g., a push to main while a prior run is still testing), the cleanup from the first run can delete the image needed by the second.

For the refactoring: Consider one of:

• Tag images by run ID instead of commit SHA, and only delete images older than the current run
• Pin min-versions-to-keep: 3 to survive overlapping runs
• Move cleanup to a scheduled workflow (weekly) instead of per-run

3. Self-hosted runner disk space

The runner ran out of disk space during an integration test run, leaving it in a broken state that required manual rebuilding. The caching strategy must account for this:

• Set a maximum cache size or file count in /opt/pve-isos
• Auto-prune ISOs not referenced by the current workflow matrix
• The cleanup job should clean up uploaded ISOs from the PVE host, not just VMs
• Consider a periodic runner maintenance script that frees disk space

Implementation Order

1. Refactor Terraform (main.tf, variables.tf, outputs.tf) → for_each
2. Create ensure-base-iso.sh and ensure-cloud-images.sh
3. Add --cache-dir to prepare-auto-iso.sh
4. Create parallel wait wrapper for create-api-token.sh
5. Restructure workflow into provision → test → cleanup jobs
6. Update prepare-test-environment.sh for cache dir
7. Ensure all jobs gated with github.actor != 'dependabot[bot]'
8. Address container image cleanup race condition
9. Test on runner via workflow_dispatch

Files to Modify

• .github/workflows/integration-tests.yml
• tests/infrastructure/main.tf
• tests/infrastructure/variables.tf
• tests/infrastructure/outputs.tf
• tests/infrastructure/scripts/prepare-auto-iso.sh
• tests/infrastructure/scripts/prepare-test-environment.sh
• tests/infrastructure/scripts/create-api-token.sh (wrapper)
• New: tests/infrastructure/scripts/ensure-base-iso.sh
• New: tests/infrastructure/scripts/ensure-cloud-images.sh