mirror of
https://github.com/GoodOlClint/PSProxmoxVE.git
synced 2026-09-04 19:25:36 +00:00
def8dc6b67
* fix: verify checksums for downloaded ISOs/images, keep sshpass off argv ensure-base-iso.sh downloaded the PVE install ISO over plain HTTP with no checksum, caching it on the persistent /opt/pve-integration mount and booting it as the nested trust root the integration suite relies on. ensure-cloud-images.sh fetched the Ubuntu cloud image and OVA over HTTPS but never checked them either. prepare-test-environment.sh and diagnose-cluster.sh passed the nested root password to sshpass via -p, putting it in the process table. create-api-token.sh, unused anywhere in the repo, minted a privsep=0 root token and echoed the secret unmasked. - ensure-base-iso.sh now downloads from https://enterprise.proxmox.com/iso and verifies against its SHA256SUMS on every run, including a cache hit. download.proxmox.com's own TLS cert does not list download.proxmox.com in its SAN (confirmed with curl/openssl from this environment), so https to that name fails certificate validation; enterprise.proxmox.com serves the identical ISO tree over a valid cert. Verification happens before the downloaded file is moved to its canonical cache path. - ensure-cloud-images.sh verifies the cloud image and OVA against Ubuntu's published SHA256SUMS the same way, matching by upstream filename since the cloud image is cached locally under a different extension (.img upstream, .qcow2 cached — the bytes are already qcow2-formatted). - prepare-test-environment.sh and diagnose-cluster.sh now export SSHPASS and call sshpass -e, keeping the password out of argv/ps. This also fixes a latent bug: the old unquoted `sshpass -p ${ROOT_PASS}` word-split any password containing whitespace. - create-api-token.sh deleted; grep across the repo found no caller. Reviewers (codex:codex-rescue, correctness-reviewer, security-reviewer) all independently found the same blocking bug in the first pass: when a cached file failed verification and the subsequent redownload then failed, ensure-cloud-images.sh fell through to a "keep the stale copy" branch and returned that same known-bad file with exit 0 — verification could be bypassed by inducing one failed redownload. Fixed by deleting the file immediately on a failed verification, before the redownload is attempted, so the later "is there a safe stale copy" check can no longer find it. Added a test case (case 5) that reproduces this exact sequence and mutation-tested it against the unfixed code. The three reviews also flagged a real but separate bug already fixed in this same change: `trap ... RETURN` inside a function nested in another function is not scoped to that function in bash — it re-fires on the OUTER function's return, referencing an out-of-scope local. Both verify_checksum() helpers now clean up their temp file explicitly instead of via trap. Findings not acted on, judged out of scope for this fix: - SHA256SUMS-fetch failures are treated the same as a checksum mismatch (delete + fail) rather than left untouched — a transient network blip destroys a good multi-GB cached ISO. This is the safer failure direction (never silently trust unverified bytes) and was a deliberate trade-off, not a defect. - ensure-cloud-images.sh's 7-day cache window can span an upstream republish of noble/current, causing a legitimate re-verification churn (not a security issue, a cache-hit-rate one). Pre-existing cache design, unrelated to adding verification. - wait-for-pve.sh (curl -d with the password on argv) and prepare-test-environment.sh's own positional password argument (from run-integration.sh) carry the same password-on-argv pattern this issue targeted in create-api-token.sh, sshpass -p and diagnose-cluster.sh, but neither script nor run-integration.sh was named in the issue. Left untouched per scope; worth a follow-up issue. - GPG/detached-signature verification of the upstream SHA256SUMS was not added — the new checks defend against cache poisoning and transit corruption, not a compromised origin. Worth a follow-up issue. - The two new self-checks (ensure-base-iso.test.sh, ensure-cloud-images.test.sh) are not wired into .github/workflows/unit-tests.yml's shell-selfchecks job. That file is code-owned and out of scope for this change; needs an operator follow-up. Password rotation (the Testpass123! value from before it moved to a secret) is unaddressed here per the contract — flagged for the operator. Mutation-tested: broke the post-download checksum check in ensure-base-iso.sh, confirmed the affected test cases failed, restored it. Broke the sshpass -e change back to -p, confirmed the new assertions in prepare-test-environment.test.sh failed, restored it. Broke the fail-open fix in ensure-cloud-images.sh, confirmed case 5 failed, restored it. Closes #149 * fix: also verify the stale-by-age fallback copy in ensure-cloud-images.sh PR review on #166 (COMMENTED, non-blocking) found the sibling of the fail-open bug already fixed in this branch: when the cached cloud image is stale by *age* (>= 7 days) rather than failed verification, the redownload-failure fallback could hand back that file with exit 0 without ever re-verifying it in this run. A file that failed the earlier verification is already deleted by the time the fallback runs, but a stale-by-age file skips verification entirely on the way in. Fixed by verifying the stale-by-age file at the point of actual fallback use — after the redownload has failed, not proactively before it's attempted, so a copy the redownload was about to replace anyway isn't deleted along a path that would have succeeded. Added two test cases (6, 7): a still-verifying stale-by-age copy is used as a fallback; one that no longer verifies is not. Mutation-tested by reverting to the unfixed fallback and confirming case 7 fails, then restored. --------- Co-authored-by: goodolclint-claude[bot] <323206664+goodolclint-claude[bot]@users.noreply.github.com>
212 lines
7.9 KiB
Bash
Executable File
212 lines
7.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Self-check for ensure-cloud-images.sh's checksum verification.
|
|
#
|
|
# Stubs curl, date and stat on PATH so every path runs offline in ~0s;
|
|
# sha256sum is the real binary, so the comparisons are genuine. The fake
|
|
# cloud image is served under its Ubuntu upstream name (.img) but cached
|
|
# under a different local name (.qcow2) — the checksum must still match,
|
|
# because ensure-cloud-images.sh renames the SHA256SUMS entry, not the
|
|
# bytes.
|
|
#
|
|
# Run: bash tests/infrastructure/scripts/ensure-cloud-images.test.sh
|
|
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
TARGET="$SCRIPT_DIR/ensure-cloud-images.sh"
|
|
|
|
TMP="$(mktemp -d)"
|
|
trap 'rm -rf "$TMP"' EXIT
|
|
|
|
mkdir -p "$TMP/bin" "$TMP/cache" "$TMP/upstream"
|
|
|
|
UPSTREAM_IMG_NAME="noble-server-cloudimg-amd64.img"
|
|
LOCAL_IMG_NAME="noble-server-cloudimg-amd64.qcow2"
|
|
OVA_NAME="ubuntu-24.04-server-cloudimg-amd64.ova"
|
|
|
|
GOOD_IMG="$TMP/upstream/good.img"
|
|
printf 'good cloud image bytes' > "$GOOD_IMG"
|
|
GOOD_IMG_SHA="$(sha256sum "$GOOD_IMG" | cut -d' ' -f1)"
|
|
|
|
GOOD_OVA="$TMP/upstream/good.ova"
|
|
printf 'good ova bytes' > "$GOOD_OVA"
|
|
GOOD_OVA_SHA="$(sha256sum "$GOOD_OVA" | cut -d' ' -f1)"
|
|
|
|
BAD_CONTENT="$TMP/upstream/bad"
|
|
printf 'corrupted bytes' > "$BAD_CONTENT"
|
|
|
|
# Ubuntu publishes the binary-mode "*filename" marker.
|
|
IMG_SUMS="$TMP/upstream/img-sums"
|
|
printf '%s *%s\n' "$GOOD_IMG_SHA" "$UPSTREAM_IMG_NAME" > "$IMG_SUMS"
|
|
OVA_SUMS="$TMP/upstream/ova-sums"
|
|
printf '%s *%s\n' "$GOOD_OVA_SHA" "$OVA_NAME" > "$OVA_SUMS"
|
|
|
|
# Fake curl: serves $IMG_SUMS/$OVA_SUMS for their SHA256SUMS URLs, and the
|
|
# fixture pointed to by IMG_SOURCE/OVA_SOURCE for the image/OVA URLs.
|
|
cat > "$TMP/bin/curl" <<'STUB'
|
|
#!/usr/bin/env bash
|
|
echo "curl $*" >> "$STUB_LOG"
|
|
url="${!#}"
|
|
out=""
|
|
prev=""
|
|
for a in "$@"; do
|
|
if [[ "$prev" == "-o" ]]; then
|
|
out="$a"
|
|
fi
|
|
prev="$a"
|
|
done
|
|
|
|
case "$url" in
|
|
*/noble/current/SHA256SUMS) cp "$IMG_SUMS" "$out" ;;
|
|
*/releases/24.04/release/SHA256SUMS) cp "$OVA_SUMS" "$out" ;;
|
|
*.img)
|
|
[[ "${CURL_FAIL_IMG:-0}" == "1" ]] && exit 22
|
|
cp "$IMG_SOURCE" "$out"
|
|
;;
|
|
*.ova)
|
|
[[ "${CURL_FAIL_OVA:-0}" == "1" ]] && exit 22
|
|
cp "$OVA_SOURCE" "$out"
|
|
;;
|
|
esac
|
|
exit 0
|
|
STUB
|
|
|
|
# Fixed "now" and an mtime helper so age math is deterministic without
|
|
# touching the real clock: files pre-dated via $TMP/bin/touch-old.
|
|
cat > "$TMP/bin/date" <<'STUB'
|
|
#!/usr/bin/env bash
|
|
if [[ "$1" == "+%s" ]]; then
|
|
echo 2000000000
|
|
else
|
|
exec /usr/bin/date "$@"
|
|
fi
|
|
STUB
|
|
cat > "$TMP/bin/stat" <<'STUB'
|
|
#!/usr/bin/env bash
|
|
# Only the two forms ensure-cloud-images.sh calls are stubbed.
|
|
for a in "$@"; do
|
|
:
|
|
done
|
|
target="${!#}"
|
|
if [[ -f "${target}.age_days" ]]; then
|
|
age="$(cat "${target}.age_days")"
|
|
echo $(( 2000000000 - age * 86400 ))
|
|
else
|
|
echo 2000000000
|
|
fi
|
|
STUB
|
|
|
|
chmod +x "$TMP/bin/"*
|
|
export PATH="$TMP/bin:$PATH"
|
|
export IMG_SUMS OVA_SUMS
|
|
|
|
fail=0
|
|
pass() { echo " ok: $1"; }
|
|
fatal() { echo " FAIL: $1"; fail=1; }
|
|
|
|
reset_cache() {
|
|
rm -rf "$TMP/cache"
|
|
mkdir -p "$TMP/cache"
|
|
}
|
|
|
|
echo "case 1: nothing cached — both files download and verify"
|
|
reset_cache
|
|
export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
|
|
export STUB_LOG="$TMP/log1"; : > "$STUB_LOG"
|
|
if bash "$TARGET" "$TMP/cache" > "$TMP/out1" 2>&1; then
|
|
pass "exits 0"
|
|
[[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && pass "cloud image cached" || fatal "cloud image missing"
|
|
[[ -f "$TMP/cache/$OVA_NAME" ]] && pass "OVA cached" || fatal "OVA missing"
|
|
grep -q "CLOUD_IMAGE_PATH=" "$TMP/out1" && pass "emits CLOUD_IMAGE_PATH" || fatal "missing CLOUD_IMAGE_PATH output"
|
|
else
|
|
fatal "exited non-zero on a clean run: $(cat "$TMP/out1")"
|
|
fi
|
|
|
|
echo "case 2: fresh cache with matching checksums — must re-verify, not re-download"
|
|
reset_cache
|
|
cp "$GOOD_IMG" "$TMP/cache/$LOCAL_IMG_NAME"
|
|
cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
|
|
export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
|
|
export STUB_LOG="$TMP/log2"; : > "$STUB_LOG"
|
|
if bash "$TARGET" "$TMP/cache" > "$TMP/out2" 2>&1; then
|
|
pass "exits 0 on a fresh, matching cache"
|
|
grep -q "SHA256SUMS" "$STUB_LOG" && pass "re-verifies the cache hit" || fatal "skipped re-verification"
|
|
grep -q '\.img$' "$STUB_LOG" && fatal "re-downloaded the cloud image on a cache hit" || pass "did not re-download the cloud image"
|
|
grep -q '\.ova$' "$STUB_LOG" && fatal "re-downloaded the OVA on a cache hit" || pass "did not re-download the OVA"
|
|
else
|
|
fatal "exited non-zero on a valid fresh cache: $(cat "$TMP/out2")"
|
|
fi
|
|
|
|
echo "case 3: fresh cache but corrupted bytes — must re-download and fix it"
|
|
reset_cache
|
|
cp "$BAD_CONTENT" "$TMP/cache/$LOCAL_IMG_NAME"
|
|
cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
|
|
export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
|
|
export STUB_LOG="$TMP/log3"; : > "$STUB_LOG"
|
|
if bash "$TARGET" "$TMP/cache" > "$TMP/out3" 2>&1; then
|
|
actual="$(sha256sum "$TMP/cache/$LOCAL_IMG_NAME" | cut -d' ' -f1)"
|
|
[[ "$actual" == "$GOOD_IMG_SHA" ]] && pass "corrupted cloud image replaced with good bytes" || fatal "corrupted cloud image not fixed"
|
|
else
|
|
fatal "did not recover from a corrupted fresh cache: $(cat "$TMP/out3")"
|
|
fi
|
|
|
|
echo "case 4: downloaded bytes do not match upstream checksum — must fail"
|
|
reset_cache
|
|
export IMG_SOURCE="$BAD_CONTENT" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=0 CURL_FAIL_OVA=0
|
|
export STUB_LOG="$TMP/log4"; : > "$STUB_LOG"
|
|
if bash "$TARGET" "$TMP/cache" > "$TMP/out4" 2>&1; then
|
|
fatal "exited 0 despite a checksum mismatch on the cloud image"
|
|
else
|
|
pass "checksum mismatch fails the run"
|
|
[[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && fatal "bad cloud image left in cache" || pass "bad cloud image not left in cache"
|
|
fi
|
|
|
|
echo "case 5: corrupted fresh cache AND the redownload fails — must not hand back the corrupt file"
|
|
reset_cache
|
|
cp "$BAD_CONTENT" "$TMP/cache/$LOCAL_IMG_NAME"
|
|
cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
|
|
export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=1 CURL_FAIL_OVA=0
|
|
export STUB_LOG="$TMP/log5"; : > "$STUB_LOG"
|
|
if bash "$TARGET" "$TMP/cache" > "$TMP/out5" 2>&1; then
|
|
fatal "exited 0 despite a corrupted cache whose redownload failed: $(cat "$TMP/out5")"
|
|
else
|
|
pass "fails the run rather than falling back to the corrupt file"
|
|
[[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && fatal "corrupt cloud image left in cache" || pass "corrupt cloud image removed, not handed back"
|
|
fi
|
|
|
|
echo "case 6: stale-by-age cache still verifies AND the redownload fails — must fall back to it"
|
|
reset_cache
|
|
cp "$GOOD_IMG" "$TMP/cache/$LOCAL_IMG_NAME"
|
|
echo 10 > "$TMP/cache/$LOCAL_IMG_NAME.age_days"
|
|
cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
|
|
export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=1 CURL_FAIL_OVA=0
|
|
export STUB_LOG="$TMP/log6"; : > "$STUB_LOG"
|
|
if bash "$TARGET" "$TMP/cache" > "$TMP/out6" 2>&1; then
|
|
pass "falls back to the still-good stale-by-age copy"
|
|
grep -q "using stale cached copy" "$TMP/out6" && pass "reports the fallback" || fatal "silent about the fallback"
|
|
[[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && pass "verified stale copy kept" || fatal "verified stale copy removed"
|
|
else
|
|
fatal "exited non-zero despite a stale-by-age copy that still verifies: $(cat "$TMP/out6")"
|
|
fi
|
|
|
|
echo "case 7: stale-by-age cache no longer verifies AND the redownload fails — must not hand it back"
|
|
reset_cache
|
|
cp "$BAD_CONTENT" "$TMP/cache/$LOCAL_IMG_NAME"
|
|
echo 10 > "$TMP/cache/$LOCAL_IMG_NAME.age_days"
|
|
cp "$GOOD_OVA" "$TMP/cache/$OVA_NAME"
|
|
export IMG_SOURCE="$GOOD_IMG" OVA_SOURCE="$GOOD_OVA" CURL_FAIL_IMG=1 CURL_FAIL_OVA=0
|
|
export STUB_LOG="$TMP/log7"; : > "$STUB_LOG"
|
|
if bash "$TARGET" "$TMP/cache" > "$TMP/out7" 2>&1; then
|
|
fatal "exited 0 despite a stale-by-age copy that no longer verifies: $(cat "$TMP/out7")"
|
|
else
|
|
pass "fails the run rather than falling back to an unverifiable stale-by-age copy"
|
|
[[ -f "$TMP/cache/$LOCAL_IMG_NAME" ]] && fatal "unverifiable stale-by-age copy left in cache" || pass "unverifiable stale-by-age copy removed"
|
|
fi
|
|
|
|
if [[ "$fail" -eq 0 ]]; then
|
|
echo "PASS"
|
|
else
|
|
echo "FAILED"
|
|
exit 1
|
|
fi
|