Go-based E2E harness that exercises Spinifex and Predastore against DDIL (Denied, Disrupted, Intermittent, Limited) failure modes — NATS kill, daemon-without-NATS restart, cluster partition, degraded links, Predastore writes under partition, Raft under SATCOM latency.

Design: docs/development/improvements/ddil-e2e-test-harness.md (in the mulga monorepo).

Layout

tests/e2e/ddil/
├── doc.go                  # package doc
├── README.md               # this file
├── harness/                # typed primitives (cluster, SSH, daemon client, ...)
└── scenarios/              # TestScenarioA..F, build tag e2e

All test and helper files use //go:build e2e so go build ./... skips them. Run with -tags=e2e.

Running

Against a pre-provisioned 3-node Proxmox cluster:

DDIL_NODES=10.0.0.1,10.0.0.2,10.0.0.3 \
DDIL_SSH_USER=ubuntu \
DDIL_SSH_KEY=$HOME/.ssh/tf-user-ap-southeast-2 \
AWS_REGION=ap-southeast-2 \
go test -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...

Run a single scenario:

go test -tags=e2e -run TestScenarioA ./tests/e2e/ddil/scenarios/...

Dry-run mode (no cluster ops; verifies SSH reachability, tool availability on remote nodes, and runs TestCoverageDrift only):

DDIL_DRY_RUN=1 go test -tags=e2e ./tests/e2e/ddil/scenarios/...

Quarantine listed scenarios (failures downgrade to t.Skip with a tagged reason; quarantined scenarios must have a linked bead):

DDIL_QUARANTINED=D,F go test -tags=e2e ./tests/e2e/ddil/scenarios/...

Environment variables

Var	Required	Purpose
`DDIL_NODES`	yes (unless dry-run)	Comma-separated cluster node IPs, order matches scenario references (node1, node2, node3).
`DDIL_SSH_USER`	yes (unless dry-run)	SSH login user on each node.
`DDIL_SSH_KEY`	yes (unless dry-run)	Path to SSH private key.
`DDIL_DRY_RUN`	no	`1` skips cluster ops, runs `TestCoverageDrift` only.
`DDIL_QUARANTINED`	no	Comma-separated scenario letters (A..F) to quarantine.
`AWS_REGION`	yes for witness scenarios	Region used by the AWS SDK when launching witness VMs via the Spinifex AWS gateway.
`DDIL_WITNESS_AMI`	no	Specific AMI ID for witness VMs. If unset, the harness picks the first `ami-ubuntu-*` image registered in the cluster.
`DDIL_WITNESS_INSTANCE_TYPE`	no (default `t2.micro`)	EC2 instance type for witness VMs.
`DDIL_WITNESS_KEY_NAME`	no (default `spinifex-key`)	EC2 key pair name attached to witness VMs; must match an existing registered key.
`DDIL_GUEST_SSH_USER`	no (default `ubuntu`)	SSH login user inside the witness guest.
`DDIL_GUEST_SSH_KEY`	no (defaults to `DDIL_SSH_KEY`)	Private key for guest SSH; override when the guest image uses a different key than the host.

Scenario status

Scenario-level status (SKIPPED / ENABLED / QUARANTINED) plus profile validation tracking lives in tests/e2e/TEST_COVERAGE.md. Scenarios begin as t.Skip("requires <dep>") and are flipped to real assertions by the hardening epics they exercise.