mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-22 12:16:52 +00:00
068aef5589
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client) Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the tests/e2e/ddil/ package tree with the three harness primitives required by every sibling scenario/fault bead: - harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv - harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh, interface so dry-run and tests can stub it - harness/daemon_client.go — thin HTTPS wrapper over /local/status, /local/instances (placeholder types until daemon-local-autonomy lands), plus /health for scaffolding validation Also adds doc.go and README.md (run instructions + env-var contract). All harness files carry //go:build e2e so default builds skip them. Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1. * feat: add DDIL harness fault injection and link profiles Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault injection and netem primitives consumed by Scenarios A–F: - harness/fault.go — PartitionNode (per-peer iptables DROP with orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS, RestartDaemonOnly, WaitForMode (polls /local/status for typed DaemonMode enum) - harness/netem.go — ApplyNetem (tc qdisc replace, idempotent), ClearNetem (idempotent clear) - harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping constants (unvalidated approximations per plan §2 table; validation tracking lives in TEST_COVERAGE.md) All files carry //go:build e2e. Flapping profile is defined but ApplyNetem rejects it — flap orchestration lands with the scenarios that need it. Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2. * feat: add DDIL harness witness VM helpers LaunchWitnessVM launches a counter VM via the cluster's AWS gateway, locates its QEMU host via ps, and captures a baseline counter reading. ReadCounter/AssertProgressed tunnel SSH through the host's QEMU hostfwd port to verify the workload progressed across a fault. README documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars. * feat: add DDIL harness snapshot and state assertions TakeSnapshot / AssertPreserved compare /local/instances pre and post fault, failing if IDs disappear or a running/pending instance regresses to a terminal state. LocalStateRevision / AssertMonotonic read the local state file revision from /local/status so scenarios can detect the daemon rewinding its persisted state. * feat: add DDIL harness reset, retry, and quarantine ResetAllNodes brings every node back to a clean baseline (iptables -F, tc netem clear on any interface carrying one, spinifex-nats and spinifex-daemon started) then waits for cluster health — daemon /health 200 plus NATS routez peer count matching len(nodes)-1, mirroring the shell suite's verify_nats_cluster predicate. AssertCleanState is a pre-scenario baseline check that recovers a dirty state via ResetAllNodes instead of failing outright, since a dirty baseline usually means a previous scenario's cleanup was skipped. Run wraps each scenario with quarantine handling (DDIL_QUARANTINED skip) and a single reset-and-retry on failure. First-attempt failures are not hidden; the retry is a diagnostic aid rather than a flake mask. bd: mulga-siv-5.5 * feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift) Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency. TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned scenarios. TestCoverageDrift AST-parses sibling *_test.go files and reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3 flips cannot silently diverge doc from code. TestHarnessSmoke covers the AssertCleanState + ResetAllNodes round trip for the "full run on real cluster" acceptance bullet. * feat: add DDIL E2E CI workflow (workflow_dispatch only) Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's provision/bootstrap/teardown shape against the same ci-multi runner and tofu-cluster, but runs the Go harness on the runner (outside the cluster) so its SSH path survives the peer-traffic iptables rules that Scenario C installs on a cluster node. Triggers are workflow_dispatch only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4 scenarios are asserting. results.json (go test -json) uploads as an artifact so flake-rate visibility can be built on gh run view + structured output, per the design doc.
80 lines
2.2 KiB
Go
80 lines
2.2 KiB
Go
//go:build e2e
|
|
|
|
package harness
|
|
|
|
import "time"
|
|
|
|
// LinkProfile describes a tc netem configuration approximating a real-world
|
|
// tactical link. Values are unvalidated approximations derived from public
|
|
// specifications; see docs/development/improvements/ddil-e2e-test-harness.md
|
|
// §2 and spinifex/tests/e2e/TEST_COVERAGE.md for validation tracking.
|
|
type LinkProfile struct {
|
|
Name string
|
|
Delay time.Duration
|
|
Jitter time.Duration
|
|
Loss float64 // percent, 0..100
|
|
Bandwidth string // tc-compatible rate, e.g. "512kbit"; empty means unshaped
|
|
Flapping *FlapSpec
|
|
}
|
|
|
|
// FlapSpec toggles link up/down for profiles that simulate jammed or
|
|
// intermittent RF. The flap loop itself is not driven by ApplyNetem — it is
|
|
// orchestrated by a higher-level helper (added with the scenarios that use
|
|
// it) so scenarios keep full control over when the flap stops.
|
|
type FlapSpec struct {
|
|
Up time.Duration
|
|
Down time.Duration
|
|
}
|
|
|
|
// Authoritative DDIL link profile values. See the §2 table in the design doc
|
|
// for the rationale behind each row. Replace with measured values when
|
|
// real-hardware access is available.
|
|
var (
|
|
LAN = LinkProfile{
|
|
Name: "LAN",
|
|
Delay: 1 * time.Millisecond,
|
|
Jitter: 0,
|
|
Loss: 0,
|
|
Bandwidth: "1gbit",
|
|
}
|
|
WAN = LinkProfile{
|
|
Name: "WAN",
|
|
Delay: 50 * time.Millisecond,
|
|
Jitter: 10 * time.Millisecond,
|
|
Loss: 0.1,
|
|
Bandwidth: "100mbit",
|
|
}
|
|
LTEDegraded = LinkProfile{
|
|
Name: "LTEDegraded",
|
|
Delay: 100 * time.Millisecond,
|
|
Jitter: 30 * time.Millisecond,
|
|
Loss: 5,
|
|
Bandwidth: "1mbit",
|
|
}
|
|
SATCOM = LinkProfile{
|
|
Name: "SATCOM",
|
|
Delay: 600 * time.Millisecond,
|
|
Jitter: 50 * time.Millisecond,
|
|
Loss: 2,
|
|
Bandwidth: "512kbit",
|
|
}
|
|
HFData = LinkProfile{
|
|
Name: "HFData",
|
|
Delay: 2000 * time.Millisecond,
|
|
Jitter: 200 * time.Millisecond,
|
|
Loss: 10,
|
|
Bandwidth: "9600bit",
|
|
}
|
|
Flapping = LinkProfile{
|
|
Name: "Flapping",
|
|
Flapping: &FlapSpec{
|
|
Up: 10 * time.Second,
|
|
Down: 5 * time.Second,
|
|
},
|
|
}
|
|
)
|
|
|
|
// AllProfiles lists every predefined profile. Useful for doc-drift checks and
|
|
// for the TEST_COVERAGE.md profile-validation table.
|
|
var AllProfiles = []LinkProfile{LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping}
|