feat: DDIL E2E test harness (mulga-siv-5) (#160)

* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client) Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the tests/e2e/ddil/ package tree with the three harness primitives required by every sibling scenario/fault bead: - harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv - harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh, interface so dry-run and tests can stub it - harness/daemon_client.go — thin HTTPS wrapper over /local/status, /local/instances (placeholder types until daemon-local-autonomy lands), plus /health for scaffolding validation Also adds doc.go and README.md (run instructions + env-var contract). All harness files carry //go:build e2e so default builds skip them. Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1. * feat: add DDIL harness fault injection and link profiles Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault injection and netem primitives consumed by Scenarios A–F: - harness/fault.go — PartitionNode (per-peer iptables DROP with orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS, RestartDaemonOnly, WaitForMode (polls /local/status for typed DaemonMode enum) - harness/netem.go — ApplyNetem (tc qdisc replace, idempotent), ClearNetem (idempotent clear) - harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping constants (unvalidated approximations per plan §2 table; validation tracking lives in TEST_COVERAGE.md) All files carry //go:build e2e. Flapping profile is defined but ApplyNetem rejects it — flap orchestration lands with the scenarios that need it. Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2. * feat: add DDIL harness witness VM helpers LaunchWitnessVM launches a counter VM via the cluster's AWS gateway, locates its QEMU host via ps, and captures a baseline counter reading. ReadCounter/AssertProgressed tunnel SSH through the host's QEMU hostfwd port to verify the workload progressed across a fault. README documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars. * feat: add DDIL harness snapshot and state assertions TakeSnapshot / AssertPreserved compare /local/instances pre and post fault, failing if IDs disappear or a running/pending instance regresses to a terminal state. LocalStateRevision / AssertMonotonic read the local state file revision from /local/status so scenarios can detect the daemon rewinding its persisted state. * feat: add DDIL harness reset, retry, and quarantine ResetAllNodes brings every node back to a clean baseline (iptables -F, tc netem clear on any interface carrying one, spinifex-nats and spinifex-daemon started) then waits for cluster health — daemon /health 200 plus NATS routez peer count matching len(nodes)-1, mirroring the shell suite's verify_nats_cluster predicate. AssertCleanState is a pre-scenario baseline check that recovers a dirty state via ResetAllNodes instead of failing outright, since a dirty baseline usually means a previous scenario's cleanup was skipped. Run wraps each scenario with quarantine handling (DDIL_QUARANTINED skip) and a single reset-and-retry on failure. First-attempt failures are not hidden; the retry is a diagnostic aid rather than a flake mask. bd: mulga-siv-5.5 * feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift) Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency. TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned scenarios. TestCoverageDrift AST-parses sibling *_test.go files and reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3 flips cannot silently diverge doc from code. TestHarnessSmoke covers the AssertCleanState + ResetAllNodes round trip for the "full run on real cluster" acceptance bullet. * feat: add DDIL E2E CI workflow (workflow_dispatch only) Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's provision/bootstrap/teardown shape against the same ci-multi runner and tofu-cluster, but runs the Go harness on the runner (outside the cluster) so its SSH path survives the peer-traffic iptables rules that Scenario C installs on a cluster node. Triggers are workflow_dispatch only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4 scenarios are asserting. results.json (go test -json) uploads as an artifact so flake-rate visibility can be built on gh run view + structured output, per the design doc.
2026-10-20 11:21:49 +00:00 · 2026-04-23 22:38:29 +10:00
parent 604693ea0c
commit 068aef5589
25 changed files with 2240 additions and 1 deletions
@@ -0,0 +1,140 @@
+name: Spinifex DDIL E2E
+
+# Phase 1 trigger: workflow_dispatch only. Nightly cron + on-main-merge
+# land in mulga-siv-5.8 once ≥4 scenarios are asserting. See
+# docs/development/improvements/ddil-e2e-test-harness.md §4 / §Dependencies.
+on:
+  workflow_dispatch:
+    inputs:
+      quarantined:
+        description: "Comma-separated scenario letters to quarantine (A..F)"
+        required: false
+        default: ""
+
+env:
+  GIT_BRANCH: ${{ github.ref_name }}
+
+jobs:
+  ddil:
+    name: DDIL Real Multi-Node
+    runs-on: [ self-hosted, ci-multi ]
+    # Budget: ~45m go test + tofu provision/bootstrap/teardown on ci-multi.
+    # In Phase 1 the scenarios all t.Skip, so the real-world runtime is
+    # dominated by provisioning; the 60m ceiling leaves headroom for
+    # Phase 2/3 when assertions come online.
+    timeout-minutes: 60
+    env:
+      TOFU_LOG: /tmp/e2e-tofu-ddil-${{ github.run_id }}.log
+    steps:
+      - name: Clean Workspace
+        run: rm -rf "$GITHUB_WORKSPACE"/*
+
+      - uses: actions/create-github-app-token@v3
+        id: app-token
+        with:
+          client-id: ${{ secrets.APP_ID }}
+          private-key: ${{ secrets.APP_PRIVATE_KEY }}
+          repositories: mulga
+
+      - name: Resolve Mulga Branch
+        id: mulga-branch
+        env:
+          GH_TOKEN: ${{ steps.app-token.outputs.token }}
+          BRANCH: ${{ github.ref_name }}
+        run: |
+          if gh api "repos/mulgadc/mulga/branches/${BRANCH}" >/dev/null 2>&1; then
+            echo "ref=${BRANCH}" >> "$GITHUB_OUTPUT"
+            echo "mulga: using branch ${BRANCH}"
+          else
+            echo "ref=main" >> "$GITHUB_OUTPUT"
+            echo "mulga: branch ${BRANCH} not found, falling back to main"
+          fi
+
+      # Full spinifex checkout: the Go harness must run on the runner
+      # (outside the cluster) so its SSH path survives scenario partitions
+      # that drop peer-to-peer cluster traffic. Running inside the cluster
+      # would self-sever when Scenario C isolates a node.
+      - uses: actions/checkout@v6
+        name: Checkout Spinifex
+        with:
+          path: spinifex
+
+      - uses: actions/checkout@v6
+        name: Checkout Mulga (tofu-cluster)
+        with:
+          repository: mulgadc/mulga
+          ref: ${{ steps.mulga-branch.outputs.ref }}
+          token: ${{ steps.app-token.outputs.token }}
+          sparse-checkout: scripts/tofu-cluster
+          path: mulga
+
+      - name: Build Install Artifacts
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          ./build-install-artifacts.sh "${GIT_BRANCH}" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Provision VMs
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          export TF_VAR_git_branch="${GIT_BRANCH}"
+          tofu init -input=false >> "$TOFU_LOG" 2>&1
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1 || true
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true
+          tofu apply -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+
+      - name: Bootstrap via Binary Install
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          SPX_EXTERNAL_POOL="${SPINIFEX_CI_POOL}" \
+          ./bootstrap-install.sh "${SPINIFEX_CI_ENV}" \
+            "${RUNNER_TEMP}/setup.sh" \
+            "${RUNNER_TEMP}/spinifex-distro.tar.gz" \
+            --test-tar "${RUNNER_TEMP}/spinifex-tests.tar"
+
+      - name: Run DDIL Scenarios
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          set -o pipefail
+          INVENTORY=$(tofu output -json inventory)
+          SSH_KEY=$(echo "$INVENTORY" | jq -r '.ssh_key_path')
+          SSH_KEY="${SSH_KEY/#\~/$HOME}"
+          SSH_USER=$(echo "$INVENTORY" | jq -r '.ssh_user')
+          REGION=$(echo "$INVENTORY" | jq -r '.region')
+          # DDIL_NODES must be the cluster-traffic IPs: PartitionNode
+          # installs iptables DROP rules for each peer, so the runner's
+          # SSH source IP must differ from these. On single-NIC
+          # tofu-cluster, wan_ip == cluster IP and the runner's outbound
+          # IP is distinct — see e2e-ddil-test-harness.md §3 Scenario C.
+          NODE_IPS=$(echo "$INVENTORY" | jq -r '[.nodes[].wan_ip] | join(",")')
+          echo "DDIL_NODES=${NODE_IPS}"
+
+          mkdir -p "${RUNNER_TEMP}/ddil-results"
+          cd "${GITHUB_WORKSPACE}/spinifex"
+          DDIL_NODES="${NODE_IPS}" \
+          DDIL_SSH_USER="${SSH_USER}" \
+          DDIL_SSH_KEY="${SSH_KEY}" \
+          DDIL_QUARANTINED="${{ inputs.quarantined }}" \
+          AWS_REGION="${REGION}" \
+            go test -tags=e2e -timeout 45m -json \
+              ./tests/e2e/ddil/scenarios/... \
+              | tee "${RUNNER_TEMP}/ddil-results/results.json"
+
+      - name: Upload DDIL Results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: ddil-results-${{ github.run_id }}
+          path: ${{ runner.temp }}/ddil-results
+          retention-days: 30
+
+      - name: Teardown
+        if: always()
+        working-directory: mulga/scripts/tofu-cluster
+        run: |
+          tofu workspace select -or-create "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1
+          tofu destroy -var-file=envs/${SPINIFEX_CI_ENV}.tfvars -auto-approve -input=false >> "$TOFU_LOG" 2>&1
+          ./scrub-env.sh "${SPINIFEX_CI_ENV}" >> "$TOFU_LOG" 2>&1 || true