joshsiv-mulga
964f4038e0
CI run 26016840819 surfaced that ScenarioC_CleanPartition silently depends on predastore-ddil-hardening §2 (write-quorum with deferred parity repair). Root cause from node3 daemon log: predastore: write udp ...->192.168.0.22:9991: sendmsg: operation not permitted predastore: putObjectViaQUIC: put failed spx QMP: BLOCK_IO_ERROR instanceId=<witness> spx: StopOrTerminateInstance: Terminating spx: Instance state transition running → shutting-down The 'sendmsg: operation not permitted' is the EPERM Linux returns when iptables drops an outbound packet — exactly the OUTPUT DROP rules PartitionNode installs. The witness VM's block I/O is backed by distributed predastore, so the same partition that severs node3↔peer NATS traffic also severs the witness disk's cross-shard QUIC writes. QEMU power-downs the guest after BLOCK_IO_ERROR, the daemon's reconciler terminates the instance, and the next AssertProgressed dials a sshd that's already gone (hence the 'handshake failed: EOF' the test reports). Quarantine by collapsing C to the same scenarioSkip stub D/E/F already use, citing predastore-ddil-hardening §2 as the dependency. Flip C back to ENABLED in the same PR that lands §2 — that's when the witness disk can survive a peer partition without QEMU halting. TEST_COVERAGE.md row C flipped ENABLED → SKIPPED; added a quarantine note explaining the predastore dependency.
DDIL E2E Test Harness
Go-based E2E harness that exercises Spinifex and Predastore against DDIL (Denied, Disrupted, Intermittent, Limited) failure modes — NATS kill, daemon-without-NATS restart, cluster partition, degraded links, Predastore writes under partition, Raft under SATCOM latency.
Design: docs/development/improvements/ddil-e2e-test-harness.md
(in the mulga monorepo).
Layout
tests/e2e/ddil/
├── doc.go # package doc
├── README.md # this file
├── harness/ # typed primitives (cluster, SSH, daemon client, ...)
└── scenarios/ # TestScenarioA..F, build tag e2e
All test and helper files use //go:build e2e so go build ./... skips them.
Run with -tags=e2e.
Running
Against a pre-provisioned 3-node Proxmox cluster:
DDIL_NODES=10.0.0.1,10.0.0.2,10.0.0.3 \
DDIL_SSH_USER=ubuntu \
DDIL_SSH_KEY=$HOME/.ssh/tf-user-ap-southeast-2 \
AWS_REGION=ap-southeast-2 \
go test -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
Run a single scenario:
go test -tags=e2e -run TestScenarioA ./tests/e2e/ddil/scenarios/...
Dry-run mode (no cluster ops; verifies SSH reachability, tool availability on
remote nodes, and runs TestCoverageDrift only):
DDIL_DRY_RUN=1 go test -tags=e2e ./tests/e2e/ddil/scenarios/...
Quarantine listed scenarios (failures downgrade to t.Skip with a tagged
reason; quarantined scenarios must have a linked bead):
DDIL_QUARANTINED=D,F go test -tags=e2e ./tests/e2e/ddil/scenarios/...
Environment variables
| Var | Required | Purpose |
|---|---|---|
DDIL_NODES |
yes (unless dry-run) | Comma-separated cluster node IPs, order matches scenario references (node1, node2, node3). |
DDIL_SSH_USER |
yes (unless dry-run) | SSH login user on each node. |
DDIL_SSH_KEY |
yes (unless dry-run) | Path to SSH private key. |
DDIL_DRY_RUN |
no | 1 skips cluster ops, runs TestCoverageDrift only. |
DDIL_QUARANTINED |
no | Comma-separated scenario letters (A..F) to quarantine. |
AWS_REGION |
yes for witness scenarios | Region used by the AWS SDK when launching witness VMs via the Spinifex AWS gateway. |
DDIL_WITNESS_AMI |
no | Specific AMI ID for witness VMs. If unset, the harness picks the first ami-ubuntu-* image registered in the cluster. |
DDIL_WITNESS_INSTANCE_TYPE |
no (default t2.micro) |
EC2 instance type for witness VMs. |
DDIL_WITNESS_KEY_NAME |
no (default spinifex-key) |
EC2 key pair name attached to witness VMs; must match an existing registered key. |
DDIL_GUEST_SSH_USER |
no (default ubuntu) |
SSH login user inside the witness guest. |
DDIL_GUEST_SSH_KEY |
no (defaults to DDIL_SSH_KEY) |
Private key for guest SSH; override when the guest image uses a different key than the host. |
Scenario status
Scenario-level status (SKIPPED / ENABLED / QUARANTINED) plus profile
validation tracking lives in tests/e2e/TEST_COVERAGE.md.
Scenarios begin as t.Skip("requires <dep>") and are flipped to real
assertions by the hardening epics they exercise.