mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-20 03:11:46 +00:00
Josh Siviour
068aef5589
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client) Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the tests/e2e/ddil/ package tree with the three harness primitives required by every sibling scenario/fault bead: - harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv - harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh, interface so dry-run and tests can stub it - harness/daemon_client.go — thin HTTPS wrapper over /local/status, /local/instances (placeholder types until daemon-local-autonomy lands), plus /health for scaffolding validation Also adds doc.go and README.md (run instructions + env-var contract). All harness files carry //go:build e2e so default builds skip them. Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1. * feat: add DDIL harness fault injection and link profiles Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault injection and netem primitives consumed by Scenarios A–F: - harness/fault.go — PartitionNode (per-peer iptables DROP with orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS, RestartDaemonOnly, WaitForMode (polls /local/status for typed DaemonMode enum) - harness/netem.go — ApplyNetem (tc qdisc replace, idempotent), ClearNetem (idempotent clear) - harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping constants (unvalidated approximations per plan §2 table; validation tracking lives in TEST_COVERAGE.md) All files carry //go:build e2e. Flapping profile is defined but ApplyNetem rejects it — flap orchestration lands with the scenarios that need it. Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2. * feat: add DDIL harness witness VM helpers LaunchWitnessVM launches a counter VM via the cluster's AWS gateway, locates its QEMU host via ps, and captures a baseline counter reading. ReadCounter/AssertProgressed tunnel SSH through the host's QEMU hostfwd port to verify the workload progressed across a fault. README documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars. * feat: add DDIL harness snapshot and state assertions TakeSnapshot / AssertPreserved compare /local/instances pre and post fault, failing if IDs disappear or a running/pending instance regresses to a terminal state. LocalStateRevision / AssertMonotonic read the local state file revision from /local/status so scenarios can detect the daemon rewinding its persisted state. * feat: add DDIL harness reset, retry, and quarantine ResetAllNodes brings every node back to a clean baseline (iptables -F, tc netem clear on any interface carrying one, spinifex-nats and spinifex-daemon started) then waits for cluster health — daemon /health 200 plus NATS routez peer count matching len(nodes)-1, mirroring the shell suite's verify_nats_cluster predicate. AssertCleanState is a pre-scenario baseline check that recovers a dirty state via ResetAllNodes instead of failing outright, since a dirty baseline usually means a previous scenario's cleanup was skipped. Run wraps each scenario with quarantine handling (DDIL_QUARANTINED skip) and a single reset-and-retry on failure. First-attempt failures are not hidden; the retry is a diagnostic aid rather than a flake mask. bd: mulga-siv-5.5 * feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift) Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency. TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned scenarios. TestCoverageDrift AST-parses sibling *_test.go files and reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3 flips cannot silently diverge doc from code. TestHarnessSmoke covers the AssertCleanState + ResetAllNodes round trip for the "full run on real cluster" acceptance bullet. * feat: add DDIL E2E CI workflow (workflow_dispatch only) Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's provision/bootstrap/teardown shape against the same ci-multi runner and tofu-cluster, but runs the Go harness on the runner (outside the cluster) so its SSH path survives the peer-traffic iptables rules that Scenario C installs on a cluster node. Triggers are workflow_dispatch only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4 scenarios are asserting. results.json (go test -json) uploads as an artifact so flake-rate visibility can be built on gh run view + structured output, per the design doc.
78 lines
3.2 KiB
Markdown
78 lines
3.2 KiB
Markdown
# DDIL E2E Test Harness
|
|
|
|
Go-based E2E harness that exercises Spinifex and Predastore against DDIL
|
|
(Denied, Disrupted, Intermittent, Limited) failure modes — NATS kill,
|
|
daemon-without-NATS restart, cluster partition, degraded links, Predastore
|
|
writes under partition, Raft under SATCOM latency.
|
|
|
|
Design: [`docs/development/improvements/ddil-e2e-test-harness.md`](../../../../docs/development/improvements/ddil-e2e-test-harness.md)
|
|
(in the mulga monorepo).
|
|
|
|
## Layout
|
|
|
|
```
|
|
tests/e2e/ddil/
|
|
├── doc.go # package doc
|
|
├── README.md # this file
|
|
├── harness/ # typed primitives (cluster, SSH, daemon client, ...)
|
|
└── scenarios/ # TestScenarioA..F, build tag e2e
|
|
```
|
|
|
|
All test and helper files use `//go:build e2e` so `go build ./...` skips them.
|
|
Run with `-tags=e2e`.
|
|
|
|
## Running
|
|
|
|
Against a pre-provisioned 3-node Proxmox cluster:
|
|
|
|
```bash
|
|
DDIL_NODES=10.0.0.1,10.0.0.2,10.0.0.3 \
|
|
DDIL_SSH_USER=ubuntu \
|
|
DDIL_SSH_KEY=$HOME/.ssh/tf-user-ap-southeast-2 \
|
|
AWS_REGION=ap-southeast-2 \
|
|
go test -tags=e2e -timeout 45m ./tests/e2e/ddil/scenarios/...
|
|
```
|
|
|
|
Run a single scenario:
|
|
|
|
```bash
|
|
go test -tags=e2e -run TestScenarioA ./tests/e2e/ddil/scenarios/...
|
|
```
|
|
|
|
Dry-run mode (no cluster ops; verifies SSH reachability, tool availability on
|
|
remote nodes, and runs `TestCoverageDrift` only):
|
|
|
|
```bash
|
|
DDIL_DRY_RUN=1 go test -tags=e2e ./tests/e2e/ddil/scenarios/...
|
|
```
|
|
|
|
Quarantine listed scenarios (failures downgrade to `t.Skip` with a tagged
|
|
reason; quarantined scenarios must have a linked bead):
|
|
|
|
```bash
|
|
DDIL_QUARANTINED=D,F go test -tags=e2e ./tests/e2e/ddil/scenarios/...
|
|
```
|
|
|
|
## Environment variables
|
|
|
|
| Var | Required | Purpose |
|
|
|-----|----------|---------|
|
|
| `DDIL_NODES` | yes (unless dry-run) | Comma-separated cluster node IPs, order matches scenario references (node1, node2, node3). |
|
|
| `DDIL_SSH_USER` | yes (unless dry-run) | SSH login user on each node. |
|
|
| `DDIL_SSH_KEY` | yes (unless dry-run) | Path to SSH private key. |
|
|
| `DDIL_DRY_RUN` | no | `1` skips cluster ops, runs `TestCoverageDrift` only. |
|
|
| `DDIL_QUARANTINED` | no | Comma-separated scenario letters (A..F) to quarantine. |
|
|
| `AWS_REGION` | yes for witness scenarios | Region used by the AWS SDK when launching witness VMs via the Spinifex AWS gateway. |
|
|
| `DDIL_WITNESS_AMI` | no | Specific AMI ID for witness VMs. If unset, the harness picks the first `ami-ubuntu-*` image registered in the cluster. |
|
|
| `DDIL_WITNESS_INSTANCE_TYPE` | no (default `t2.micro`) | EC2 instance type for witness VMs. |
|
|
| `DDIL_WITNESS_KEY_NAME` | no (default `spinifex-key`) | EC2 key pair name attached to witness VMs; must match an existing registered key. |
|
|
| `DDIL_GUEST_SSH_USER` | no (default `ubuntu`) | SSH login user inside the witness guest. |
|
|
| `DDIL_GUEST_SSH_KEY` | no (defaults to `DDIL_SSH_KEY`) | Private key for guest SSH; override when the guest image uses a different key than the host. |
|
|
|
|
## Scenario status
|
|
|
|
Scenario-level status (SKIPPED / ENABLED / QUARANTINED) plus profile
|
|
validation tracking lives in [`tests/e2e/TEST_COVERAGE.md`](../TEST_COVERAGE.md).
|
|
Scenarios begin as `t.Skip("requires <dep>")` and are flipped to real
|
|
assertions by the hardening epics they exercise.
|