Files
spinifex/go.mod
T
Josh Siviour 068aef5589 feat: DDIL E2E test harness (mulga-siv-5) (#160)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)

Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:

- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
  interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
  /local/instances (placeholder types until daemon-local-autonomy lands),
  plus /health for scaffolding validation

Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.

* feat: add DDIL harness fault injection and link profiles

Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:

- harness/fault.go — PartitionNode (per-peer iptables DROP with
  orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
  RestartDaemonOnly, WaitForMode (polls /local/status for typed
  DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
  ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
  SATCOM, HFData, Flapping constants (unvalidated approximations per
  plan §2 table; validation tracking lives in TEST_COVERAGE.md)

All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.

* feat: add DDIL harness witness VM helpers

LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.

* feat: add DDIL harness snapshot and state assertions

TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.

* feat: add DDIL harness reset, retry, and quarantine

ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.

AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.

Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.

bd: mulga-siv-5.5

* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)

Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.

* feat: add DDIL E2E CI workflow (workflow_dispatch only)

Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
2026-04-23 22:38:29 +10:00

114 lines
4.8 KiB
Modula-2

module github.com/mulgadc/spinifex
go 1.26.2
require (
github.com/aws/aws-sdk-go v1.55.8
github.com/go-chi/chi/v5 v5.2.5
github.com/google/uuid v1.6.0
github.com/kdomanski/iso9660 v0.4.0
github.com/klauspost/cpuid/v2 v2.3.0
github.com/mulgadc/predastore v1.1.6
github.com/mulgadc/viperblock v1.1.6
github.com/nats-io/nats-server/v2 v2.12.7
github.com/nats-io/nats.go v1.51.0
github.com/ovn-kubernetes/libovsdb v0.8.1
github.com/pelletier/go-toml/v2 v2.3.0
github.com/pterm/pterm v0.12.83
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
github.com/stretchr/testify v1.11.1
golang.org/x/crypto v0.49.0
gopkg.in/ini.v1 v1.67.1
)
require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.0 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boltdb/bolt v1.3.1 // indirect
github.com/buraksezer/consistent v0.10.0 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cenkalti/hub v1.0.2 // indirect
github.com/cenkalti/rpc2 v1.0.5 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/containerd/console v1.0.5 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgraph-io/badger/v4 v4.9.1 // indirect
github.com/dgraph-io/ristretto/v2 v2.4.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/fatih/color v1.19.0 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/google/flatbuffers v25.12.19+incompatible // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/gookit/color v1.6.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.4 // indirect
github.com/hashicorp/go-msgpack/v2 v2.1.5 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/raft v1.7.3 // indirect
github.com/hashicorp/raft-boltdb/v2 v2.3.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/klauspost/reedsolomon v1.13.3 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.15 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/quic-go/quic-go v0.59.0 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.etcd.io/bbolt v1.4.3 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel v1.42.0 // indirect
go.opentelemetry.io/otel/metric v1.42.0 // indirect
go.opentelemetry.io/otel/trace v1.42.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/net v0.52.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.42.0 // indirect
golang.org/x/telemetry v0.0.0-20260304144227-18da59047661 // indirect
golang.org/x/term v0.41.0 // indirect
golang.org/x/text v0.35.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.42.0 // indirect
golang.org/x/vuln v1.1.4 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
tool golang.org/x/vuln/cmd/govulncheck