Files
PSProxmoxVE/tests/infrastructure/scripts/ensure-base-iso.test.sh
T
goodolclint-claude[bot] def8dc6b67 fix: verify checksums for downloaded ISOs/images, keep sshpass off argv (#166)
* fix: verify checksums for downloaded ISOs/images, keep sshpass off argv

ensure-base-iso.sh downloaded the PVE install ISO over plain HTTP with no
checksum, caching it on the persistent /opt/pve-integration mount and
booting it as the nested trust root the integration suite relies on.
ensure-cloud-images.sh fetched the Ubuntu cloud image and OVA over HTTPS
but never checked them either. prepare-test-environment.sh and
diagnose-cluster.sh passed the nested root password to sshpass via -p,
putting it in the process table. create-api-token.sh, unused anywhere in
the repo, minted a privsep=0 root token and echoed the secret unmasked.

- ensure-base-iso.sh now downloads from https://enterprise.proxmox.com/iso
  and verifies against its SHA256SUMS on every run, including a cache hit.
  download.proxmox.com's own TLS cert does not list download.proxmox.com in
  its SAN (confirmed with curl/openssl from this environment), so https to
  that name fails certificate validation; enterprise.proxmox.com serves the
  identical ISO tree over a valid cert. Verification happens before the
  downloaded file is moved to its canonical cache path.
- ensure-cloud-images.sh verifies the cloud image and OVA against Ubuntu's
  published SHA256SUMS the same way, matching by upstream filename since
  the cloud image is cached locally under a different extension (.img
  upstream, .qcow2 cached — the bytes are already qcow2-formatted).
- prepare-test-environment.sh and diagnose-cluster.sh now export SSHPASS
  and call sshpass -e, keeping the password out of argv/ps. This also fixes
  a latent bug: the old unquoted `sshpass -p ${ROOT_PASS}` word-split any
  password containing whitespace.
- create-api-token.sh deleted; grep across the repo found no caller.

Reviewers (codex:codex-rescue, correctness-reviewer, security-reviewer) all
independently found the same blocking bug in the first pass: when a cached
file failed verification and the subsequent redownload then failed,
ensure-cloud-images.sh fell through to a "keep the stale copy" branch and
returned that same known-bad file with exit 0 — verification could be
bypassed by inducing one failed redownload. Fixed by deleting the file
immediately on a failed verification, before the redownload is attempted,
so the later "is there a safe stale copy" check can no longer find it.
Added a test case (case 5) that reproduces this exact sequence and
mutation-tested it against the unfixed code. The three reviews also
flagged a real but separate bug already fixed in this same change: `trap
... RETURN` inside a function nested in another function is not scoped to
that function in bash — it re-fires on the OUTER function's return,
referencing an out-of-scope local. Both verify_checksum() helpers now
clean up their temp file explicitly instead of via trap.

Findings not acted on, judged out of scope for this fix:
- SHA256SUMS-fetch failures are treated the same as a checksum mismatch
  (delete + fail) rather than left untouched — a transient network blip
  destroys a good multi-GB cached ISO. This is the safer failure direction
  (never silently trust unverified bytes) and was a deliberate trade-off,
  not a defect.
- ensure-cloud-images.sh's 7-day cache window can span an upstream
  republish of noble/current, causing a legitimate re-verification churn
  (not a security issue, a cache-hit-rate one). Pre-existing cache design,
  unrelated to adding verification.
- wait-for-pve.sh (curl -d with the password on argv) and
  prepare-test-environment.sh's own positional password argument (from
  run-integration.sh) carry the same password-on-argv pattern this issue
  targeted in create-api-token.sh, sshpass -p and diagnose-cluster.sh, but
  neither script nor run-integration.sh was named in the issue. Left
  untouched per scope; worth a follow-up issue.
- GPG/detached-signature verification of the upstream SHA256SUMS was not
  added — the new checks defend against cache poisoning and transit
  corruption, not a compromised origin. Worth a follow-up issue.
- The two new self-checks (ensure-base-iso.test.sh,
  ensure-cloud-images.test.sh) are not wired into
  .github/workflows/unit-tests.yml's shell-selfchecks job. That file is
  code-owned and out of scope for this change; needs an operator follow-up.

Password rotation (the Testpass123! value from before it moved to a
secret) is unaddressed here per the contract — flagged for the operator.

Mutation-tested: broke the post-download checksum check in
ensure-base-iso.sh, confirmed the affected test cases failed, restored it.
Broke the sshpass -e change back to -p, confirmed the new assertions in
prepare-test-environment.test.sh failed, restored it. Broke the fail-open
fix in ensure-cloud-images.sh, confirmed case 5 failed, restored it.

Closes #149

* fix: also verify the stale-by-age fallback copy in ensure-cloud-images.sh

PR review on #166 (COMMENTED, non-blocking) found the sibling of the
fail-open bug already fixed in this branch: when the cached cloud image
is stale by *age* (>= 7 days) rather than failed verification, the
redownload-failure fallback could hand back that file with exit 0
without ever re-verifying it in this run. A file that failed the
earlier verification is already deleted by the time the fallback runs,
but a stale-by-age file skips verification entirely on the way in.

Fixed by verifying the stale-by-age file at the point of actual
fallback use — after the redownload has failed, not proactively before
it's attempted, so a copy the redownload was about to replace anyway
isn't deleted along a path that would have succeeded. Added two test
cases (6, 7): a still-verifying stale-by-age copy is used as a
fallback; one that no longer verifies is not. Mutation-tested by
reverting to the unfixed fallback and confirming case 7 fails, then
restored.

---------

Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
2026-09-02 17:20:36 +00:00

143 lines
5.3 KiB
Bash
Executable File

#!/usr/bin/env bash
# Self-check for ensure-base-iso.sh's checksum verification.
#
# Stubs curl on PATH so every path runs offline in ~0s; sha256sum is the real
# binary, so the checksum comparisons are genuine. The fake upstream is a
# small file whose real sha256 is embedded in a fake SHA256SUMS.
#
# Run: bash tests/infrastructure/scripts/ensure-base-iso.test.sh
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TARGET="$SCRIPT_DIR/ensure-base-iso.sh"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
mkdir -p "$TMP/bin" "$TMP/cache" "$TMP/upstream"
ISO_NAME="proxmox-ve_9.1-1.iso"
GOOD_CONTENT="$TMP/upstream/good.iso"
printf 'good iso bytes' > "$GOOD_CONTENT"
GOOD_SHA="$(sha256sum "$GOOD_CONTENT" | cut -d' ' -f1)"
BAD_CONTENT="$TMP/upstream/bad.iso"
printf 'corrupted bytes' > "$BAD_CONTENT"
SUMS_FILE="$TMP/upstream/SHA256SUMS"
printf '%s %s\n' "$GOOD_SHA" "$ISO_NAME" > "$SUMS_FILE"
SUMS_FILE_MISSING="$TMP/upstream/SHA256SUMS_MISSING"
printf '%s %s\n' "$GOOD_SHA" "some-other.iso" > "$SUMS_FILE_MISSING"
# Fake curl. Serves the SHA256SUMS fixture pointed to by $SUMS_SOURCE, and the
# ISO bytes pointed to by $ISO_SOURCE, to whatever -o path was requested.
# CURL_FAIL_SUMS / CURL_FAIL_ISO make the corresponding fetch fail.
cat > "$TMP/bin/curl" <<'STUB'
#!/usr/bin/env bash
echo "curl $*" >> "$STUB_LOG"
url="${!#}"
out=""
prev=""
for a in "$@"; do
if [[ "$prev" == "-o" ]]; then
out="$a"
fi
prev="$a"
done
case "$url" in
*SHA256SUMS*)
[[ "${CURL_FAIL_SUMS:-0}" == "1" ]] && exit 22
cp "$SUMS_SOURCE" "$out"
;;
*)
[[ "${CURL_FAIL_ISO:-0}" == "1" ]] && exit 22
cp "$ISO_SOURCE" "$out"
;;
esac
exit 0
STUB
chmod +x "$TMP/bin/"*
export PATH="$TMP/bin:$PATH"
fail=0
pass() { echo " ok: $1"; }
fatal() { echo " FAIL: $1"; fail=1; }
run() {
export STUB_LOG="$TMP/log"
: > "$STUB_LOG"
rm -rf "$TMP/cache"
mkdir -p "$TMP/cache"
"$@"
}
echo "case 1: nothing cached, good download — must verify and succeed"
export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out1" 2>&1; then
pass "exits 0 on a verified download"
[[ -f "$TMP/cache/$ISO_NAME" ]] && pass "ISO left in cache" || fatal "ISO missing from cache"
grep -q "Checksum verified" "$TMP/out1" && pass "reports verification" || fatal "silent on verification"
else
fatal "exited non-zero on a good download: $(cat "$TMP/out1")"
fi
echo "case 2: already cached with a matching checksum — must not re-download"
export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
export STUB_LOG="$TMP/log2"
: > "$STUB_LOG"
if bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out2" 2>&1; then
pass "exits 0 on a fresh, matching cache hit"
grep -q "SHA256SUMS" "$STUB_LOG" && pass "still re-verifies the cache hit" || fatal "skipped re-verification"
grep -q "$ISO_NAME\$" "$STUB_LOG" && fatal "re-downloaded the ISO on a cache hit" || pass "did not re-download the ISO"
else
fatal "exited non-zero on a valid cache hit: $(cat "$TMP/out2")"
fi
echo "case 3: cached copy is corrupted — must delete, re-download, and verify"
export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
run cp "$BAD_CONTENT" "$TMP/cache/$ISO_NAME"
if bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out3" 2>&1; then
pass "exits 0 after replacing a corrupted cached copy"
actual_sha="$(sha256sum "$TMP/cache/$ISO_NAME" | cut -d' ' -f1)"
[[ "$actual_sha" == "$GOOD_SHA" ]] && pass "cache now holds the good bytes" || fatal "cache still holds bad bytes"
grep -q "failed checksum verification" "$TMP/out3" && pass "names the corrupted-cache path" || fatal "silent about the corrupted cache"
else
fatal "did not recover from a corrupted cache: $(cat "$TMP/out3")"
fi
echo "case 4: downloaded bytes do not match the checksum — must fail and not leave a bad file behind"
export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$BAD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out4" 2>&1; then
fatal "exited 0 despite a checksum mismatch"
else
pass "checksum mismatch fails the run"
[[ -f "$TMP/cache/$ISO_NAME" ]] && fatal "bad ISO left in cache" || pass "bad ISO removed from cache"
fi
echo "case 5: upstream SHA256SUMS does not list this ISO — must fail"
export SUMS_SOURCE="$SUMS_FILE_MISSING" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=0 CURL_FAIL_ISO=0
if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out5" 2>&1; then
fatal "exited 0 despite the ISO being absent from SHA256SUMS"
else
pass "missing SHA256SUMS entry fails the run"
grep -q "not listed" "$TMP/out5" && pass "names the cause" || fatal "failure message does not mention the missing entry"
fi
echo "case 6: SHA256SUMS fetch itself fails — must fail rather than skip verification"
export SUMS_SOURCE="$SUMS_FILE" ISO_SOURCE="$GOOD_CONTENT" CURL_FAIL_SUMS=1 CURL_FAIL_ISO=0
if run bash "$TARGET" "$ISO_NAME" "$TMP/cache" > "$TMP/out6" 2>&1; then
fatal "exited 0 despite being unable to fetch SHA256SUMS"
else
pass "a failed SHA256SUMS fetch fails the run (verification cannot be silently skipped)"
fi
if [[ "$fail" -eq 0 ]]; then
echo "PASS"
else
echo "FAILED"
exit 1
fi