mirror of https://github.com/mulgadc/spinifex.git synced 2026-10-15 17:11:30 +00:00

Files

T

Josh Siviour 068aef5589 feat: DDIL E2E test harness (mulga-siv-5) (#160 )

* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)

Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:

- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
  interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
  /local/instances (placeholder types until daemon-local-autonomy lands),
  plus /health for scaffolding validation

Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.

* feat: add DDIL harness fault injection and link profiles

Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:

- harness/fault.go — PartitionNode (per-peer iptables DROP with
  orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
  RestartDaemonOnly, WaitForMode (polls /local/status for typed
  DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
  ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
  SATCOM, HFData, Flapping constants (unvalidated approximations per
  plan §2 table; validation tracking lives in TEST_COVERAGE.md)

All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.

* feat: add DDIL harness witness VM helpers

LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.

* feat: add DDIL harness snapshot and state assertions

TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.

* feat: add DDIL harness reset, retry, and quarantine

ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.

AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.

Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.

bd: mulga-siv-5.5

* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)

Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.

* feat: add DDIL E2E CI workflow (workflow_dispatch only)

Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.

2026-04-23 22:38:29 +10:00

3.2 KiB

Raw Blame History

DDIL E2E Test Harness

Go-based E2E harness that exercises Spinifex and Predastore against DDIL (Denied, Disrupted, Intermittent, Limited) failure modes — NATS kill, daemon-without-NATS restart, cluster partition, degraded links, Predastore writes under partition, Raft under SATCOM latency.

Design: docs/development/improvements/ddil-e2e-test-harness.md (in the mulga monorepo).

Layout

tests/e2e/ddil/
├── doc.go                  # package doc
├── README.md               # this file
├── harness/                # typed primitives (cluster, SSH, daemon client, ...)
└── scenarios/              # TestScenarioA..F, build tag e2e

All test and helper files use //go:build e2e so go build ./... skips them. Run with -tags=e2e.

Running

Against a pre-provisioned 3-node Proxmox cluster:

DDIL_NODES=10.0.0.1,10.0.0.2,10.0.0.3 \
DDIL_SSH_USER=ubuntu \
DDIL_SSH_KEY=$HOME/.ssh/tf-user-ap-southeast-2 \
AWS_REGION=ap-southeast-2 \
go test -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...

Run a single scenario:

go test -tags=e2e -run TestScenarioA ./tests/e2e/ddil/scenarios/...

Dry-run mode (no cluster ops; verifies SSH reachability, tool availability on remote nodes, and runs TestCoverageDrift only):

DDIL_DRY_RUN=1 go test -tags=e2e ./tests/e2e/ddil/scenarios/...

Quarantine listed scenarios (failures downgrade to t.Skip with a tagged reason; quarantined scenarios must have a linked bead):

DDIL_QUARANTINED=D,F go test -tags=e2e ./tests/e2e/ddil/scenarios/...

Environment variables

Var	Required	Purpose
`DDIL_NODES`	yes (unless dry-run)	Comma-separated cluster node IPs, order matches scenario references (node1, node2, node3).
`DDIL_SSH_USER`	yes (unless dry-run)	SSH login user on each node.
`DDIL_SSH_KEY`	yes (unless dry-run)	Path to SSH private key.
`DDIL_DRY_RUN`	no	`1` skips cluster ops, runs `TestCoverageDrift` only.
`DDIL_QUARANTINED`	no	Comma-separated scenario letters (A..F) to quarantine.
`AWS_REGION`	yes for witness scenarios	Region used by the AWS SDK when launching witness VMs via the Spinifex AWS gateway.
`DDIL_WITNESS_AMI`	no	Specific AMI ID for witness VMs. If unset, the harness picks the first `ami-ubuntu-*` image registered in the cluster.
`DDIL_WITNESS_INSTANCE_TYPE`	no (default `t2.micro`)	EC2 instance type for witness VMs.
`DDIL_WITNESS_KEY_NAME`	no (default `spinifex-key`)	EC2 key pair name attached to witness VMs; must match an existing registered key.
`DDIL_GUEST_SSH_USER`	no (default `ubuntu`)	SSH login user inside the witness guest.
`DDIL_GUEST_SSH_KEY`	no (defaults to `DDIL_SSH_KEY`)	Private key for guest SSH; override when the guest image uses a different key than the host.

Scenario status

Scenario-level status (SKIPPED / ENABLED / QUARANTINED) plus profile validation tracking lives in tests/e2e/TEST_COVERAGE.md. Scenarios begin as t.Skip("requires <dep>") and are flipped to real assertions by the hardening epics they exercise.

3.2 KiB Raw Blame History