fix: verify checksums for downloaded ISOs/images, keep sshpass off argv (#166)

* fix: verify checksums for downloaded ISOs/images, keep sshpass off argv

ensure-base-iso.sh downloaded the PVE install ISO over plain HTTP with no
checksum, caching it on the persistent /opt/pve-integration mount and
booting it as the nested trust root the integration suite relies on.
ensure-cloud-images.sh fetched the Ubuntu cloud image and OVA over HTTPS
but never checked them either. prepare-test-environment.sh and
diagnose-cluster.sh passed the nested root password to sshpass via -p,
putting it in the process table. create-api-token.sh, unused anywhere in
the repo, minted a privsep=0 root token and echoed the secret unmasked.

- ensure-base-iso.sh now downloads from https://enterprise.proxmox.com/iso
  and verifies against its SHA256SUMS on every run, including a cache hit.
  download.proxmox.com's own TLS cert does not list download.proxmox.com in
  its SAN (confirmed with curl/openssl from this environment), so https to
  that name fails certificate validation; enterprise.proxmox.com serves the
  identical ISO tree over a valid cert. Verification happens before the
  downloaded file is moved to its canonical cache path.
- ensure-cloud-images.sh verifies the cloud image and OVA against Ubuntu's
  published SHA256SUMS the same way, matching by upstream filename since
  the cloud image is cached locally under a different extension (.img
  upstream, .qcow2 cached — the bytes are already qcow2-formatted).
- prepare-test-environment.sh and diagnose-cluster.sh now export SSHPASS
  and call sshpass -e, keeping the password out of argv/ps. This also fixes
  a latent bug: the old unquoted `sshpass -p ${ROOT_PASS}` word-split any
  password containing whitespace.
- create-api-token.sh deleted; grep across the repo found no caller.

Reviewers (codex:codex-rescue, correctness-reviewer, security-reviewer) all
independently found the same blocking bug in the first pass: when a cached
file failed verification and the subsequent redownload then failed,
ensure-cloud-images.sh fell through to a "keep the stale copy" branch and
returned that same known-bad file with exit 0 — verification could be
bypassed by inducing one failed redownload. Fixed by deleting the file
immediately on a failed verification, before the redownload is attempted,
so the later "is there a safe stale copy" check can no longer find it.
Added a test case (case 5) that reproduces this exact sequence and
mutation-tested it against the unfixed code. The three reviews also
flagged a real but separate bug already fixed in this same change: `trap
... RETURN` inside a function nested in another function is not scoped to
that function in bash — it re-fires on the OUTER function's return,
referencing an out-of-scope local. Both verify_checksum() helpers now
clean up their temp file explicitly instead of via trap.

Findings not acted on, judged out of scope for this fix:
- SHA256SUMS-fetch failures are treated the same as a checksum mismatch
  (delete + fail) rather than left untouched — a transient network blip
  destroys a good multi-GB cached ISO. This is the safer failure direction
  (never silently trust unverified bytes) and was a deliberate trade-off,
  not a defect.
- ensure-cloud-images.sh's 7-day cache window can span an upstream
  republish of noble/current, causing a legitimate re-verification churn
  (not a security issue, a cache-hit-rate one). Pre-existing cache design,
  unrelated to adding verification.
- wait-for-pve.sh (curl -d with the password on argv) and
  prepare-test-environment.sh's own positional password argument (from
  run-integration.sh) carry the same password-on-argv pattern this issue
  targeted in create-api-token.sh, sshpass -p and diagnose-cluster.sh, but
  neither script nor run-integration.sh was named in the issue. Left
  untouched per scope; worth a follow-up issue.
- GPG/detached-signature verification of the upstream SHA256SUMS was not
  added — the new checks defend against cache poisoning and transit
  corruption, not a compromised origin. Worth a follow-up issue.
- The two new self-checks (ensure-base-iso.test.sh,
  ensure-cloud-images.test.sh) are not wired into
  .github/workflows/unit-tests.yml's shell-selfchecks job. That file is
  code-owned and out of scope for this change; needs an operator follow-up.

Password rotation (the Testpass123! value from before it moved to a
secret) is unaddressed here per the contract — flagged for the operator.

Mutation-tested: broke the post-download checksum check in
ensure-base-iso.sh, confirmed the affected test cases failed, restored it.
Broke the sshpass -e change back to -p, confirmed the new assertions in
prepare-test-environment.test.sh failed, restored it. Broke the fail-open
fix in ensure-cloud-images.sh, confirmed case 5 failed, restored it.

Closes #149

* fix: also verify the stale-by-age fallback copy in ensure-cloud-images.sh

PR review on #166 (COMMENTED, non-blocking) found the sibling of the
fail-open bug already fixed in this branch: when the cached cloud image
is stale by *age* (>= 7 days) rather than failed verification, the
redownload-failure fallback could hand back that file with exit 0
without ever re-verifying it in this run. A file that failed the
earlier verification is already deleted by the time the fallback runs,
but a stale-by-age file skips verification entirely on the way in.

Fixed by verifying the stale-by-age file at the point of actual
fallback use — after the redownload has failed, not proactively before
it's attempted, so a copy the redownload was about to replace anyway
isn't deleted along a path that would have succeeded. Added two test
cases (6, 7): a still-verifying stale-by-age copy is used as a
fallback; one that no longer verifies is not. Mutation-tested by
reverting to the unfixed fallback and confirming case 7 fails, then
restored.

---------

Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>

tests/infrastructure/scripts/create-api-token.sh

@@ -1,113 +0,0 @@
-#!/usr/bin/env bash
-# Wait for a fresh nested PVE instance to boot, discover its IP via the QEMU guest agent,
-# then wait for the PVE API and create an API token.
-#
-# Usage: create-api-token.sh <parent-pve-endpoint> <parent-api-token> <vm-id> <root-password> [max-wait-seconds]
-#   parent-pve-endpoint: Full URL e.g. https://pve.example.com:8006
-#   Outputs two lines:
-#     IP=<discovered-ip>
-#     TOKEN=root@pam!integration=<secret>
-set -euo pipefail
-
-PARENT_ENDPOINT="${1%/}"
-PARENT_TOKEN="$2"
-VM_ID="$3"
-ROOT_PASSWORD="$4"
-MAX_WAIT="${5:-600}"
-INTERVAL=10
-PARENT_API="${PARENT_ENDPOINT}/api2/json"
-NODES_JSON=$(curl -sk -H "Authorization: PVEAPIToken=${PARENT_TOKEN}" \
-    "${PARENT_API}/nodes")
-PARENT_NODE=$(echo "$NODES_JSON" | python3 -c "import json,sys; print(json.load(sys.stdin)['data'][0]['node'])")
-
-# --- Phase 1: Discover IP via QEMU guest agent ---
-echo "Waiting for guest agent on VM ${VM_ID} (node: ${PARENT_NODE})..."
-VM_IP=""
-elapsed=0
-while [ $elapsed -lt $MAX_WAIT ]; do
-    AGENT_RESPONSE=$(curl -sk \
-        -H "Authorization: PVEAPIToken=${PARENT_TOKEN}" \
-        "${PARENT_API}/nodes/${PARENT_NODE}/qemu/${VM_ID}/agent/network-get-interfaces" 2>/dev/null || true)
-
-    VM_IP=$(echo "$AGENT_RESPONSE" | python3 -c "
-import json, sys
-try:
-    data = json.load(sys.stdin).get('data', {}).get('result', [])
-    for iface in data:
-        if iface.get('name') == 'lo':
-            continue
-        for addr in iface.get('ip-addresses', []):
-            if addr.get('ip-address-type') == 'ipv4' and not addr['ip-address'].startswith('127.'):
-                print(addr['ip-address'])
-                sys.exit(0)
-except:
-    pass
-" 2>/dev/null || true)
-
-    if [ -n "$VM_IP" ]; then
-        echo "Discovered VM IP: $VM_IP (after ${elapsed}s)"
-        break
-    fi
-    echo "  Guest agent not ready yet (${elapsed}s elapsed)..."
-    sleep $INTERVAL
-    elapsed=$((elapsed + INTERVAL))
-done
-
-if [ -z "$VM_IP" ]; then
-    echo "ERROR: Could not discover VM IP via guest agent after ${MAX_WAIT}s" >&2
-    exit 1
-fi
-
-# --- Phase 2: Wait for PVE API on the nested instance ---
-NESTED_API="https://${VM_IP}:8006/api2/json"
-echo "Waiting for nested PVE API at ${NESTED_API}..."
-while [ $elapsed -lt $MAX_WAIT ]; do
-    if curl -sk --connect-timeout 5 "${NESTED_API}/access/domains" 2>/dev/null | grep -q '"realm"'; then
-        echo "Nested PVE API is responsive after ${elapsed}s"
-        break
-    fi
-    echo "  API not ready yet (${elapsed}s elapsed)..."
-    sleep $INTERVAL
-    elapsed=$((elapsed + INTERVAL))
-done
-
-if [ $elapsed -ge $MAX_WAIT ]; then
-    echo "ERROR: Nested PVE API not responsive after ${MAX_WAIT}s" >&2
-    exit 1
-fi
-
-# --- Phase 3: Authenticate and create API token ---
-echo "Authenticating as root@pam on nested PVE..."
-AUTH_RESPONSE=$(curl -sk -d "username=root@pam&password=${ROOT_PASSWORD}" \
-    "${NESTED_API}/access/ticket")
-TICKET=$(echo "$AUTH_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['ticket'])" 2>/dev/null || true)
-CSRF=$(echo "$AUTH_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['CSRFPreventionToken'])" 2>/dev/null || true)
-
-if [ -z "$TICKET" ] || [ -z "$CSRF" ]; then
-    echo "ERROR: Authentication failed. Response: $AUTH_RESPONSE" >&2
-    exit 1
-fi
-
-# Delete existing token if present, then create fresh
-echo "Creating API token root@pam!integration..."
-curl -sk \
-    -b "PVEAuthCookie=${TICKET}" \
-    -H "CSRFPreventionToken: ${CSRF}" \
-    -X DELETE \
-    "${NESTED_API}/access/users/root@pam/token/integration" >/dev/null 2>&1 || true
-
-TOKEN_RESPONSE=$(curl -sk \
-    -b "PVEAuthCookie=${TICKET}" \
-    -H "CSRFPreventionToken: ${CSRF}" \
-    -d "privsep=0" \
-    "${NESTED_API}/access/users/root@pam/token/integration")
-
-TOKEN_VALUE=$(echo "$TOKEN_RESPONSE" | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['value'])" 2>/dev/null || true)
-
-if [ -z "$TOKEN_VALUE" ]; then
-    echo "ERROR: Token creation failed. Response: $TOKEN_RESPONSE" >&2
-    exit 1
-fi
-
-echo "IP=${VM_IP}"
-echo "TOKEN=root@pam!integration=${TOKEN_VALUE}"

tests/infrastructure/scripts/diagnose-cluster.sh

@@ -29,6 +29,7 @@ if [[ -z "${PVE_PASSWORD:-}" ]]; then
 fi
 
 SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=10)
+export SSHPASS="$PVE_PASSWORD"
 
 dump_node() {
     local label="$1" ip="$2"
@@ -39,7 +40,7 @@ dump_node() {
         return
     fi
 
-    sshpass -p "$PVE_PASSWORD" ssh "${SSH_OPTS[@]}" "root@${ip}" bash -s <<'REMOTE' 2>&1 || echo "  ssh to $ip failed (rc=$?)"
+    sshpass -e ssh "${SSH_OPTS[@]}" "root@${ip}" bash -s <<'REMOTE' 2>&1 || echo "  ssh to $ip failed (rc=$?)"
 set +e
 echo "--- hostname / resolution ---"
 hostname -f

tests/infrastructure/scripts/ensure-base-iso.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
-# Downloads a PVE base ISO to the cache directory if not already present.
+# Downloads a PVE base ISO to the cache directory if not already present,
+# and verifies it against the upstream SHA256SUMS on every run — including a
+# cache hit, since the cache lives on a persistent mount nothing else audits.
 #
 # Usage: ensure-base-iso.sh <iso-filename> <cache-dir>
 #   iso-filename: e.g. proxmox-ve_9.1-1.iso
@@ -11,22 +13,70 @@ CACHE_DIR="${2:?Cache directory required}"
 
 CACHED_PATH="${CACHE_DIR}/${ISO_FILENAME}"
 
-if [ -f "${CACHED_PATH}" ] && [ -s "${CACHED_PATH}" ]; then
-    echo "Base ISO already cached: ${CACHED_PATH} ($(du -h "${CACHED_PATH}" | cut -f1))"
-    exit 0
-fi
+# download.proxmox.com's own TLS cert does not list download.proxmox.com in
+# its SAN — only the regional *.cdn.proxmox.com aliases and
+# enterprise.proxmox.com — so https to that name fails certificate
+# validation. enterprise.proxmox.com serves byte-identical ISOs and
+# SHA256SUMS over a valid cert.
+BASE_URL="https://enterprise.proxmox.com/iso"
+DOWNLOAD_URL="${BASE_URL}/${ISO_FILENAME}"
+SUMS_URL="${BASE_URL}/SHA256SUMS"
 
-# Ensure cache directory exists and is writable
 mkdir -p "${CACHE_DIR}"
 
-DOWNLOAD_URL="http://download.proxmox.com/iso/${ISO_FILENAME}"
+verify_checksum() {
+    local filepath="$1"
+    local sums_file hash dir base
+
+    sums_file="$(mktemp)"
+
+    if ! curl -fsSL -o "${sums_file}" "${SUMS_URL}"; then
+        echo "ERROR: failed to download ${SUMS_URL}" >&2
+        rm -f "${sums_file}"
+        return 1
+    fi
+
+    hash="$(awk -v f="${ISO_FILENAME}" '$2 == f || $2 == "*" f {print $1; exit}' "${sums_file}")"
+    rm -f "${sums_file}"
+
+    if [ -z "${hash}" ]; then
+        echo "ERROR: ${ISO_FILENAME} not listed in ${SUMS_URL}" >&2
+        return 1
+    fi
+
+    dir="$(dirname "${filepath}")"
+    base="$(basename "${filepath}")"
+
+    if ! printf '%s  %s\n' "${hash}" "${base}" | (cd "${dir}" && sha256sum -c -); then
+        echo "ERROR: checksum mismatch for ${filepath}" >&2
+        return 1
+    fi
+}
+
+if [ -f "${CACHED_PATH}" ] && [ -s "${CACHED_PATH}" ]; then
+    echo "Base ISO already cached: ${CACHED_PATH} ($(du -h "${CACHED_PATH}" | cut -f1))"
+    if verify_checksum "${CACHED_PATH}"; then
+        echo "Checksum verified: ${CACHED_PATH}"
+        exit 0
+    fi
+    echo "Cached ISO failed checksum verification; removing and re-downloading." >&2
+    rm -f "${CACHED_PATH}"
+fi
+
 TMP_PATH="${CACHED_PATH}.downloading"
 
 echo "Downloading PVE base ISO: ${DOWNLOAD_URL}"
 echo "  Target: ${CACHED_PATH}"
 
-# Download to temp file, then atomic move
+# Download to a temp file, verify it, then atomic move — never let unverified
+# bytes sit at the canonical cache path, where a concurrent run's cache-hit
+# check could pick them up.
 if curl -fSL --progress-bar -o "${TMP_PATH}" "${DOWNLOAD_URL}"; then
+    if ! verify_checksum "${TMP_PATH}"; then
+        rm -f "${TMP_PATH}"
+        echo "ERROR: downloaded ISO failed checksum verification; removed" >&2
+        exit 1
+    fi
     mv "${TMP_PATH}" "${CACHED_PATH}"
     echo "Downloaded: ${CACHED_PATH} ($(du -h "${CACHED_PATH}" | cut -f1))"
 else
@@ -34,3 +84,5 @@ else
     echo "ERROR: Failed to download ${DOWNLOAD_URL}" >&2
     exit 1
 fi
+
+echo "Checksum verified: ${CACHED_PATH}"

tests/infrastructure/scripts/ensure-base-iso.test.sh

@@ -0,0 +1,142 @@
+#!/usr/bin/env bash
+# Self-check for ensure-base-iso.sh's checksum verification.
+#
+# Stubs curl on PATH so every path runs offline in ~0s; sha256sum is the real
+# binary, so the checksum comparisons are genuine. The fake upstream is a
+# small file whose real sha256 is embedded in a fake SHA256SUMS.
+#
+# Run: bash tests/infrastructure/scripts/ensure-base-iso.test.sh
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TARGET="$SCRIPT_DIR/ensure-base-iso.sh"
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+
+mkdir -p "$TMP/bin" "$TMP/cache" "$TMP/upstream"
+
+ISO_NAME="proxmox-ve_9.1-1.iso"
+GOOD_CONTENT="$TMP/upstream/good.iso"
+printf 'good iso bytes' > "$GOOD_CONTENT"
+GOOD_SHA="$(sha256sum "$GOOD_CONTENT" | cut -d' ' -f1)"
+
+BAD_CONTENT="$TMP/upstream/bad.iso"
+printf 'corrupted bytes' > "$BAD_CONTENT"
+
+SUMS_FILE="$TMP/upstream/SHA256SUMS"
+printf '%s  %s\n' "$GOOD_SHA" "$ISO_NAME" > "$SUMS_FILE"
+
+SUMS_FILE_MISSING="$TMP/upstream/SHA256SUMS_MISSING"
+printf '%s  %s\n' "$GOOD_SHA" "some-other.iso" > "$SUMS_FILE_MISSING"
+
+# Fake curl. Serves the SHA256SUMS fixture pointed to by $SUMS_SOURCE, and the
+# ISO bytes pointed to by $ISO_SOURCE, to whatever -o path was requested.
+# CURL_FAIL_SUMS / CURL_FAIL_ISO make the corresponding fetch fail.
+cat > "$TMP/bin/curl" <<'STUB'
+#!/usr/bin/env bash
+echo "curl $*" >> "$STUB_LOG"
+url="${!#}"
+out=""
+prev=""
+for a in "$@"; do
+    if [[ "$prev" == "-o" ]]; then
+        out="$a"
+    fi
+    prev="$a"
+done
+
+case "$url" in
+    *SHA256SUMS*)
+        [[ "${CURL_FAIL_SUMS:-0}" == "1" ]] && exit 22
+        cp "$SUMS_SOURCE" "$out"
+        ;;
+    *)
+        [[ "${CURL_FAIL_ISO:-0}" == "1" ]] && exit 22
+        cp "$ISO_SOURCE" "$out"
+        ;;
+esac
+exit 0
+STUB
+chmod +x "$TMP/bin/"*
+export PATH="$TMP/bin:$PATH"
+
+fail=0
+pass() { echo "  ok: $1"; }
+fatal() { echo "  FAIL: $1"; fail=1; }
+
+run() {
+    export STUB_LOG="$TMP/log"
+    : > "$STUB_LOG"
+    rm -rf "$TMP/cache"
+    mkdir -p "$TMP/cache"
+    "$@"
+}
+
+echo "case 1: nothing cached, good download — must verify and succeed"
+export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
+if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out1" 2>&1; then
+    pass "exits 0 on a verified download"
+    [[ -f "$TMP/cache/$ISO_NAME" ]] && pass "ISO left in cache" || fatal "ISO missing from cache"
+    grep -q "Checksum verified" "$TMP/out1" && pass "reports verification" || fatal "silent on verification"
+else
+    fatal "exited non-zero on a good download: $(cat "$TMP/out1")"
+fi
+
+echo "case 2: already cached with a matching checksum — must not re-download"
+export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
+export STUB_LOG="$TMP/log2"
+: > "$STUB_LOG"
+if bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out2" 2>&1; then
+    pass "exits 0 on a fresh, matching cache hit"
+    grep -q "SHA256SUMS" "$STUB_LOG" && pass "still re-verifies the cache hit" || fatal "skipped re-verification"
+    grep -q "$ISO_NAME\$" "$STUB_LOG" && fatal "re-downloaded the ISO on a cache hit" || pass "did not re-download the ISO"
+else
+    fatal "exited non-zero on a valid cache hit: $(cat "$TMP/out2")"
+fi
+
+echo "case 3: cached copy is corrupted — must delete, re-download, and verify"
+export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
+run cp "$BAD_CONTENT" "$TMP/cache/$ISO_NAME"
+if bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out3" 2>&1; then
+    pass "exits 0 after replacing a corrupted cached copy"
+    actual_sha="$(sha256sum "$TMP/cache/$ISO_NAME" | cut -d' ' -f1)"
+    [[ "$actual_sha" == "$GOOD_SHA" ]] && pass "cache now holds the good bytes" || fatal "cache still holds bad bytes"
+    grep -q "failed checksum verification" "$TMP/out3" && pass "names the corrupted-cache path" || fatal "silent about the corrupted cache"
+else
+    fatal "did not recover from a corrupted cache: $(cat "$TMP/out3")"
+fi
+
+echo "case 4: downloaded bytes do not match the checksum — must fail and not leave a bad file behind"
+export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$BAD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
+if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out4" 2>&1; then
+    fatal "exited 0 despite a checksum mismatch"
+else
+    pass "checksum mismatch fails the run"
+    [[ -f "$TMP/cache/$ISO_NAME" ]] && fatal "bad ISO left in cache" || pass "bad ISO removed from cache"
+fi
+
+echo "case 5: upstream SHA256SUMS does not list this ISO — must fail"
+export SUMS_SOURCE="$SUMS_FILE_MISSING" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
+if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out5" 2>&1; then
+    fatal "exited 0 despite the ISO being absent from SHA256SUMS"
+else
+    pass "missing SHA256SUMS entry fails the run"
+    grep -q "not listed" "$TMP/out5" && pass "names the cause" || fatal "failure message does not mention the missing entry"
+fi
+
+echo "case 6: SHA256SUMS fetch itself fails — must fail rather than skip verification"
+export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=1 CURL_FAIL_ISO=0
+if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out6" 2>&1; then
+    fatal "exited 0 despite being unable to fetch SHA256SUMS"
+else
+    pass "a failed SHA256SUMS fetch fails the run (verification cannot be silently skipped)"
+fi
+
+if [[ "$fail" -eq 0 ]]; then
+    echo "PASS"
+else
+    echo "FAILED"
+    exit 1
+fi

tests/infrastructure/scripts/ensure-cloud-images.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 # Downloads cloud image and OVA to the cache directory if not already present
-# or if the cached copy is older than 7 days.
+# or if the cached copy is older than 7 days. Verifies both against the
+# upstream Ubuntu SHA256SUMS on every check, including a fresh cache hit.
 #
 # Usage: ensure-cloud-images.sh <cache-dir>
 #
@@ -13,49 +14,107 @@ CACHE_DIR="${1:?Usage: ensure-cloud-images.sh <cache-dir>}"
 MAX_AGE_DAYS=7
 
 CLOUD_IMAGE_URL="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
+CLOUD_IMAGE_SUMS_URL="https://cloud-images.ubuntu.com/noble/current/SHA256SUMS"
 CLOUD_IMAGE_FILENAME="noble-server-cloudimg-amd64.qcow2"
 
 OVA_URL="https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.ova"
+OVA_SUMS_URL="https://cloud-images.ubuntu.com/releases/24.04/release/SHA256SUMS"
 OVA_FILENAME="ubuntu-24.04-server-cloudimg-amd64.ova"
 
 mkdir -p "${CACHE_DIR}"
 
+# Ubuntu's SHA256SUMS lists the upstream filename, which is not always the
+# name we cache under (the cloud image is published as .img and cached as
+# .qcow2 — Ubuntu's .img is already qcow2-formatted). Match by upstream name,
+# then check the bytes under the name they actually have on disk.
+verify_checksum() {
+    local filepath="$1" sums_url="$2" upstream_name="$3"
+    local sums_file hash dir base
+
+    sums_file="$(mktemp)"
+
+    if ! curl -fsSL -o "${sums_file}" "${sums_url}"; then
+        echo "  ERROR: failed to download ${sums_url}" >&2
+        rm -f "${sums_file}"
+        return 1
+    fi
+
+    hash="$(awk -v f="${upstream_name}" '$2 == f || $2 == "*" f {print $1; exit}' "${sums_file}")"
+    rm -f "${sums_file}"
+
+    if [ -z "${hash}" ]; then
+        echo "  ERROR: ${upstream_name} not listed in ${sums_url}" >&2
+        return 1
+    fi
+
+    dir="$(dirname "${filepath}")"
+    base="$(basename "${filepath}")"
+
+    if ! printf '%s  %s\n' "${hash}" "${base}" | (cd "${dir}" && sha256sum -c -); then
+        echo "  ERROR: checksum mismatch for ${filepath}" >&2
+        return 1
+    fi
+}
+
 download_if_stale() {
     local url="$1"
     local filepath="$2"
     local description="$3"
+    local sums_url="$4"
+    local upstream_name
+    upstream_name="$(basename "${url}")"
 
     if [ -f "${filepath}" ] && [ -s "${filepath}" ]; then
         # Check age
         local age_days
         age_days=$(( ( $(date +%s) - $(stat -c %Y "${filepath}" 2>/dev/null || stat -f %m "${filepath}" 2>/dev/null) ) / 86400 ))
         if [ "${age_days}" -lt "${MAX_AGE_DAYS}" ]; then
-            echo "${description} cached and fresh (${age_days}d old): ${filepath}"
-            return 0
+            if verify_checksum "${filepath}" "${sums_url}" "${upstream_name}"; then
+                echo "${description} cached and fresh (${age_days}d old): ${filepath}"
+                return 0
+            fi
+            # Remove it now, not just on a redownload's own failure below —
+            # otherwise a redownload that then fails falls through to the
+            # "keep the stale copy" branch and hands back these same
+            # known-bad bytes with exit 0.
+            echo "${description} cached copy failed checksum verification, removing and re-downloading..." >&2
+            rm -f "${filepath}"
+        else
+            echo "${description} is ${age_days}d old, re-downloading..."
         fi
-        echo "${description} is ${age_days}d old, re-downloading..."
     else
         echo "Downloading ${description}..."
     fi
 
     local tmp_path="${filepath}.downloading"
     if curl -fSL --progress-bar -o "${tmp_path}" "${url}"; then
+        if ! verify_checksum "${tmp_path}" "${sums_url}" "${upstream_name}"; then
+            rm -f "${tmp_path}"
+            echo "ERROR: ${description} failed checksum verification" >&2
+            return 1
+        fi
         mv "${tmp_path}" "${filepath}"
         echo "Downloaded ${description}: $(du -h "${filepath}" | cut -f1)"
     else
         rm -f "${tmp_path}"
-        # If we have a stale copy, keep using it
-        if [ -f "${filepath}" ]; then
+        # A copy that failed verification above is already gone by this
+        # point; a copy that's here because it was merely stale-by-age was
+        # never re-verified this run. Verify it now, at the point we'd
+        # actually hand it back — checking only here, not proactively before
+        # the redownload attempt, avoids deleting a copy the redownload was
+        # about to replace anyway.
+        if [ -f "${filepath}" ] && verify_checksum "${filepath}" "${sums_url}" "${upstream_name}"; then
             echo "WARNING: Download failed, using stale cached copy" >&2
             return 0
         fi
+        rm -f "${filepath}"
         echo "ERROR: Failed to download ${description}" >&2
         return 1
     fi
 }
 
-download_if_stale "${CLOUD_IMAGE_URL}" "${CACHE_DIR}/${CLOUD_IMAGE_FILENAME}" "Ubuntu cloud image"
-download_if_stale "${OVA_URL}" "${CACHE_DIR}/${OVA_FILENAME}" "Ubuntu OVA"
+download_if_stale "${CLOUD_IMAGE_URL}" "${CACHE_DIR}/${CLOUD_IMAGE_FILENAME}" "Ubuntu cloud image" "${CLOUD_IMAGE_SUMS_URL}"
+download_if_stale "${OVA_URL}" "${CACHE_DIR}/${OVA_FILENAME}" "Ubuntu OVA" "${OVA_SUMS_URL}"
 
 echo "CLOUD_IMAGE_PATH=${CACHE_DIR}/${CLOUD_IMAGE_FILENAME}"
 echo "OVA_PATH=${CACHE_DIR}/${OVA_FILENAME}"

tests/infrastructure/scripts/ensure-cloud-images.test.sh

@@ -0,0 +1,211 @@
+#!/usr/bin/env bash
+# Self-check for ensure-cloud-images.sh's checksum verification.
+#
+# Stubs curl, date and stat on PATH so every path runs offline in ~0s;
+# sha256sum is the real binary, so the comparisons are genuine. The fake
+# cloud image is served under its Ubuntu upstream name (.img) but cached
+# under a different local name (.qcow2) — the checksum must still match,
+# because ensure-cloud-images.sh renames the SHA256SUMS entry, not the
+# bytes.
+#
+# Run: bash tests/infrastructure/scripts/ensure-cloud-images.test.sh
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TARGET="$SCRIPT_DIR/ensure-cloud-images.sh"
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+
+mkdir -p "$TMP/bin" "$TMP/cache" "$TMP/upstream"
+
+UPSTREAM_IMG_NAME="noble-server-cloudimg-amd64.img"
+LOCAL_IMG_NAME="noble-server-cloudimg-amd64.qcow2"
+OVA_NAME="ubuntu-24.04-server-cloudimg-amd64.ova"
+
+GOOD_IMG="$TMP/upstream/good.img"
+printf 'good cloud image bytes' > "$GOOD_IMG"
+GOOD_IMG_SHA="$(sha256sum "$GOOD_IMG" | cut -d' ' -f1)"
+
+GOOD_OVA="$TMP/upstream/good.ova"
+printf 'good ova bytes' > "$GOOD_OVA"
+GOOD_OVA_SHA="$(sha256sum "$GOOD_OVA" | cut -d' ' -f1)"
+
+BAD_CONTENT="$TMP/upstream/bad"
+printf 'corrupted bytes' > "$BAD_CONTENT"
+
+# Ubuntu publishes the binary-mode "*filename" marker.
+IMG_SUMS="$TMP/upstream/img-sums"
+printf '%s *%s\n' "$GOOD_IMG_SHA" "$UPSTREAM_IMG_NAME" > "$IMG_SUMS"
+OVA_SUMS="$TMP/upstream/ova-sums"
+printf '%s *%s\n' "$GOOD_OVA_SHA" "$OVA_NAME" > "$OVA_SUMS"
+
+# Fake curl: serves $IMG_SUMS/$OVA_SUMS for their SHA256SUMS URLs, and the
+# fixture pointed to by IMG_SOURCE/OVA_SOURCE for the image/OVA URLs.
+cat > "$TMP/bin/curl" <<'STUB'
+#!/usr/bin/env bash
+echo "curl $*" >> "$STUB_LOG"
+url="${!#}"
+out=""
+prev=""
+for a in "$@"; do
+    if [[ "$prev" == "-o" ]]; then
+        out="$a"
+    fi
+    prev="$a"
+done
+
+case "$url" in
+    */noble/current/SHA256SUMS) cp "$IMG_SUMS" "$out" ;;
+    */releases/24.04/release/SHA256SUMS) cp "$OVA_SUMS" "$out" ;;
+    *.img)
+        [[ "${CURL_FAIL_IMG:-0}" == "1" ]] && exit 22
+        cp "$IMG_SOURCE" "$out"
+        ;;
+    *.ova)
+        [[ "${CURL_FAIL_OVA:-0}" == "1" ]] && exit 22
+        cp "$OVA_SOURCE" "$out"
+        ;;
+esac
+exit 0
+STUB
+
+# Fixed "now" and an mtime helper so age math is deterministic without
+# touching the real clock: files pre-dated via $TMP/bin/touch-old.
+cat > "$TMP/bin/date" <<'STUB'
+#!/usr/bin/env bash
+if [[ "$1" == "+%s" ]]; then
+    echo 2000000000
+else
+    exec /usr/bin/date "$@"
+fi
+STUB
+cat > "$TMP/bin/stat" <<'STUB'
+#!/usr/bin/env bash
+# Only the two forms ensure-cloud-images.sh calls are stubbed.
+for a in "$@"; do
+    :
+done
+target="${!#}"
+if [[ -f "${target}.age_days" ]]; then
+    age="$(cat "${target}.age_days")"
+    echo $(( 2000000000 - age * 86400 ))
+else
+    echo 2000000000
+fi
+STUB
+
+chmod +x "$TMP/bin/"*
+export PATH="$TMP/bin:$PATH"
+export IMG_SUMS OVA_SUMS
+
+fail=0
+pass() { echo "  ok: $1"; }
+fatal() { echo "  FAIL: $1"; fail=1; }
+
+reset_cache() {
+    rm -rf "$TMP/cache"
+    mkdir -p "$TMP/cache"
+}
+
+echo "case 1: nothing cached — both files download and verify"
+reset_cache
+export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
+export STUB_LOG="$TMP/log1"; : > "$STUB_LOG"
+if bash "$TARGET" "$TMP/cache" > "$TMP/out1" 2>&1; then
+    pass "exits 0"
+    [[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && pass "cloud image cached" || fatal "cloud image missing"
+    [[ -f "$TMP/cache/$OVA_NAME" ]] && pass "OVA cached" || fatal "OVA missing"
+    grep -q "CLOUD_IMAGE_PATH=" "$TMP/out1" && pass "emits CLOUD_IMAGE_PATH" || fatal "missing CLOUD_IMAGE_PATH output"
+else
+    fatal "exited non-zero on a clean run: $(cat "$TMP/out1")"
+fi
+
+echo "case 2: fresh cache with matching checksums — must re-verify, not re-download"
+reset_cache
+cp "$GOOD_IMG" "$TMP/cache/$LOCAL_IMG_NAME"
+cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
+export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
+export STUB_LOG="$TMP/log2"; : > "$STUB_LOG"
+if bash "$TARGET" "$TMP/cache" > "$TMP/out2" 2>&1; then
+    pass "exits 0 on a fresh, matching cache"
+    grep -q "SHA256SUMS" "$STUB_LOG" && pass "re-verifies the cache hit" || fatal "skipped re-verification"
+    grep -q '\.img$' "$STUB_LOG" && fatal "re-downloaded the cloud image on a cache hit" || pass "did not re-download the cloud image"
+    grep -q '\.ova$' "$STUB_LOG" && fatal "re-downloaded the OVA on a cache hit" || pass "did not re-download the OVA"
+else
+    fatal "exited non-zero on a valid fresh cache: $(cat "$TMP/out2")"
+fi
+
+echo "case 3: fresh cache but corrupted bytes — must re-download and fix it"
+reset_cache
+cp "$BAD_CONTENT" "$TMP/cache/$LOCAL_IMG_NAME"
+cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
+export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
+export STUB_LOG="$TMP/log3"; : > "$STUB_LOG"
+if bash "$TARGET" "$TMP/cache" > "$TMP/out3" 2>&1; then
+    actual="$(sha256sum "$TMP/cache/$LOCAL_IMG_NAME" | cut -d' ' -f1)"
+    [[ "$actual" == "$GOOD_IMG_SHA" ]] && pass "corrupted cloud image replaced with good bytes" || fatal "corrupted cloud image not fixed"
+else
+    fatal "did not recover from a corrupted fresh cache: $(cat "$TMP/out3")"
+fi
+
+echo "case 4: downloaded bytes do not match upstream checksum — must fail"
+reset_cache
+export IMG_SOURCE="$BAD_CONTENT" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
+export STUB_LOG="$TMP/log4"; : > "$STUB_LOG"
+if bash "$TARGET" "$TMP/cache" > "$TMP/out4" 2>&1; then
+    fatal "exited 0 despite a checksum mismatch on the cloud image"
+else
+    pass "checksum mismatch fails the run"
+    [[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && fatal "bad cloud image left in cache" || pass "bad cloud image not left in cache"
+fi
+
+echo "case 5: corrupted fresh cache AND the redownload fails — must not hand back the corrupt file"
+reset_cache
+cp "$BAD_CONTENT" "$TMP/cache/$LOCAL_IMG_NAME"
+cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
+export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=1 CURL_FAIL_OVA=0
+export STUB_LOG="$TMP/log5"; : > "$STUB_LOG"
+if bash "$TARGET" "$TMP/cache" > "$TMP/out5" 2>&1; then
+    fatal "exited 0 despite a corrupted cache whose redownload failed: $(cat "$TMP/out5")"
+else
+    pass "fails the run rather than falling back to the corrupt file"
+    [[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && fatal "corrupt cloud image left in cache" || pass "corrupt cloud image removed, not handed back"
+fi
+
+echo "case 6: stale-by-age cache still verifies AND the redownload fails — must fall back to it"
+reset_cache
+cp "$GOOD_IMG" "$TMP/cache/$LOCAL_IMG_NAME"
+echo 10 > "$TMP/cache/$LOCAL_IMG_NAME.age_days"
+cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
+export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=1 CURL_FAIL_OVA=0
+export STUB_LOG="$TMP/log6"; : > "$STUB_LOG"
+if bash "$TARGET" "$TMP/cache" > "$TMP/out6" 2>&1; then
+    pass "falls back to the still-good stale-by-age copy"
+    grep -q "using stale cached copy" "$TMP/out6" && pass "reports the fallback" || fatal "silent about the fallback"
+    [[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && pass "verified stale copy kept" || fatal "verified stale copy removed"
+else
+    fatal "exited non-zero despite a stale-by-age copy that still verifies: $(cat "$TMP/out6")"
+fi
+
+echo "case 7: stale-by-age cache no longer verifies AND the redownload fails — must not hand it back"
+reset_cache
+cp "$BAD_CONTENT" "$TMP/cache/$LOCAL_IMG_NAME"
+echo 10 > "$TMP/cache/$LOCAL_IMG_NAME.age_days"
+cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
+export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=1 CURL_FAIL_OVA=0
+export STUB_LOG="$TMP/log7"; : > "$STUB_LOG"
+if bash "$TARGET" "$TMP/cache" > "$TMP/out7" 2>&1; then
+    fatal "exited 0 despite a stale-by-age copy that no longer verifies: $(cat "$TMP/out7")"
+else
+    pass "fails the run rather than falling back to an unverifiable stale-by-age copy"
+    [[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && fatal "unverifiable stale-by-age copy left in cache" || pass "unverifiable stale-by-age copy removed"
+fi
+
+if [[ "$fail" -eq 0 ]]; then
+    echo "PASS"
+else
+    echo "FAILED"
+    exit 1
+fi

tests/infrastructure/scripts/prepare-test-environment.sh

@@ -20,8 +20,9 @@ PKG_OUT="${4:-}"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
 SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"
-SSH_CMD="sshpass -p ${ROOT_PASS} ssh ${SSH_OPTS} root@${NESTED_IP}"
-SCP_CMD="sshpass -p ${ROOT_PASS} scp ${SSH_OPTS}"
+export SSHPASS="${ROOT_PASS}"
+SSH_CMD="sshpass -e ssh ${SSH_OPTS} root@${NESTED_IP}"
+SCP_CMD="sshpass -e scp ${SSH_OPTS}"
 
 echo "=== Preparing test environment on ${NESTED_IP} ==="
 

tests/infrastructure/scripts/prepare-test-environment.test.sh

@@ -24,7 +24,7 @@ mkdir -p "$TMP/bin"
 # never rebooted.
 cat > "$TMP/bin/sshpass" <<'STUB'
 #!/usr/bin/env bash
-echo "$*" >> "$STUB_LOG"
+echo "SSHPASS_ENV=${SSHPASS:-<unset>} ARGS=$*" >> "$STUB_LOG"
 case "$*" in
     *boot_id*)
         if [[ "${BOOT_ID_STUCK:-0}" == "1" ]]; then
@@ -93,6 +93,9 @@ check "no reboot issued"            "$STUB_LOG" "systemctl reboot" no
 check "no package set recorded"     "$STUB_LOG" "dpkg-query"      no
 check "no boot_id probe"            "$STUB_LOG" "boot_id"         no
 check "storage still configured"    "$STUB_LOG" "pvesm set local" yes
+check "password not passed as -p arg" "$STUB_LOG" "ARGS=-p"       no
+check "sshpass invoked with -e"     "$STUB_LOG" "ARGS=-e"          yes
+check "password reaches sshpass via SSHPASS env" "$STUB_LOG" "SSHPASS_ENV=secret" yes
 
 echo "case 2: dist-upgrade requested"
 export STUB_LOG="$TMP/log2"