Files
spinifex/tests/e2e/ddil/harness/daemon_client.go
T
Josh Siviour 068aef5589 feat: DDIL E2E test harness (mulga-siv-5) (#160)
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client)

Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the
tests/e2e/ddil/ package tree with the three harness primitives required
by every sibling scenario/fault bead:

- harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv
- harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh,
  interface so dry-run and tests can stub it
- harness/daemon_client.go — thin HTTPS wrapper over /local/status,
  /local/instances (placeholder types until daemon-local-autonomy lands),
  plus /health for scaffolding validation

Also adds doc.go and README.md (run instructions + env-var contract). All
harness files carry //go:build e2e so default builds skip them.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1.

* feat: add DDIL harness fault injection and link profiles

Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault
injection and netem primitives consumed by Scenarios A–F:

- harness/fault.go — PartitionNode (per-peer iptables DROP with
  orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS,
  RestartDaemonOnly, WaitForMode (polls /local/status for typed
  DaemonMode enum)
- harness/netem.go — ApplyNetem (tc qdisc replace, idempotent),
  ClearNetem (idempotent clear)
- harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded,
  SATCOM, HFData, Flapping constants (unvalidated approximations per
  plan §2 table; validation tracking lives in TEST_COVERAGE.md)

All files carry //go:build e2e. Flapping profile is defined but
ApplyNetem rejects it — flap orchestration lands with the scenarios
that need it.

Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2.

* feat: add DDIL harness witness VM helpers

LaunchWitnessVM launches a counter VM via the cluster's AWS gateway,
locates its QEMU host via ps, and captures a baseline counter reading.
ReadCounter/AssertProgressed tunnel SSH through the host's QEMU
hostfwd port to verify the workload progressed across a fault. README
documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars.

* feat: add DDIL harness snapshot and state assertions

TakeSnapshot / AssertPreserved compare /local/instances pre and post
fault, failing if IDs disappear or a running/pending instance regresses
to a terminal state. LocalStateRevision / AssertMonotonic read the
local state file revision from /local/status so scenarios can detect
the daemon rewinding its persisted state.

* feat: add DDIL harness reset, retry, and quarantine

ResetAllNodes brings every node back to a clean baseline (iptables -F,
tc netem clear on any interface carrying one, spinifex-nats and
spinifex-daemon started) then waits for cluster health — daemon /health
200 plus NATS routez peer count matching len(nodes)-1, mirroring the
shell suite's verify_nats_cluster predicate.

AssertCleanState is a pre-scenario baseline check that recovers a dirty
state via ResetAllNodes instead of failing outright, since a dirty
baseline usually means a previous scenario's cleanup was skipped.

Run wraps each scenario with quarantine handling (DDIL_QUARANTINED
skip) and a single reset-and-retry on failure. First-attempt failures
are not hidden; the retry is a diagnostic aid rather than a flake mask.

bd: mulga-siv-5.5

* feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift)

Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as
TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency.
TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before
exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned
scenarios. TestCoverageDrift AST-parses sibling *_test.go files and
reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3
flips cannot silently diverge doc from code. TestHarnessSmoke covers the
AssertCleanState + ResetAllNodes round trip for the "full run on real
cluster" acceptance bullet.

* feat: add DDIL E2E CI workflow (workflow_dispatch only)

Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's
provision/bootstrap/teardown shape against the same ci-multi runner and
tofu-cluster, but runs the Go harness on the runner (outside the
cluster) so its SSH path survives the peer-traffic iptables rules that
Scenario C installs on a cluster node. Triggers are workflow_dispatch
only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4
scenarios are asserting. results.json (go test -json) uploads as an
artifact so flake-rate visibility can be built on gh run view +
structured output, per the design doc.
2026-04-23 22:38:29 +10:00

140 lines
4.4 KiB
Go

//go:build e2e
package harness
import (
"context"
"crypto/tls"
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"strconv"
"time"
"github.com/mulgadc/spinifex/spinifex/types"
)
// daemonPort is the cluster-manager HTTPS port (see
// spinifex/daemon/daemon.go ClusterManager). The harness targets the same
// port the production daemon exposes.
const daemonPort = 8443
// DaemonMode mirrors the daemon's operating mode reported by /local/status.
//
// The typed enum avoids stringly-typed assertions in scenarios. The concrete
// endpoint and JSON schema land with daemon-local-autonomy §1b; this file
// predates that PR and its types will be swapped for the daemon's own when
// it merges.
type DaemonMode string
const (
ModeUnknown DaemonMode = ""
ModeCluster DaemonMode = "cluster"
ModeStandalone DaemonMode = "standalone"
)
// LocalStatus is the response shape from GET /local/status.
//
// Placeholder fields only — the authoritative struct ships with
// daemon-local-autonomy §1b. Scenarios that depend on these fields remain
// t.Skip until that epic replaces this definition with an import from the
// daemon package.
type LocalStatus struct {
Node string `json:"node"`
Mode DaemonMode `json:"mode"`
Revision uint64 `json:"revision"`
NATS string `json:"nats"` // "connected" | "disconnected"
}
// LocalInstance is one row of GET /local/instances. Placeholder — see
// LocalStatus.
type LocalInstance struct {
InstanceID string `json:"instance_id"`
State string `json:"state"`
PID int `json:"pid,omitempty"`
}
// DaemonClient is a thin HTTPS client targeting a single daemon's local
// endpoints. It uses one *http.Client per node so connection reuse works
// across calls within a scenario.
type DaemonClient struct {
http *http.Client
}
// NewDaemonClient returns a DaemonClient with TLS verification disabled
// (the tofu-cluster uses self-signed per-node certs). Timeouts are set to
// 5s so a wedged daemon fails fast enough for scenario retry logic.
func NewDaemonClient() *DaemonClient {
return &DaemonClient{
http: &http.Client{
Timeout: 5 * time.Second,
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // self-signed test certs
},
},
}
}
func (d *DaemonClient) url(node Node, path string) string {
return "https://" + net.JoinHostPort(node.Addr, strconv.Itoa(daemonPort)) + path
}
func (d *DaemonClient) getJSON(ctx context.Context, node Node, path string, out any) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, d.url(node, path), nil)
if err != nil {
return fmt.Errorf("ddil harness: daemon %s %s: %w", node.Name, path, err)
}
resp, err := d.http.Do(req)
if err != nil {
return fmt.Errorf("ddil harness: daemon %s %s: %w", node.Name, path, err)
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return fmt.Errorf("ddil harness: daemon %s %s: status %d: %s",
node.Name, path, resp.StatusCode, string(body))
}
if err := json.NewDecoder(resp.Body).Decode(out); err != nil {
return fmt.Errorf("ddil harness: daemon %s %s decode: %w", node.Name, path, err)
}
return nil
}
// Status returns the daemon's self-reported local status.
//
// Depends on daemon-local-autonomy §1b. Until that lands, the daemon will
// return 404 and callers should t.Skip.
func (d *DaemonClient) Status(ctx context.Context, node Node) (LocalStatus, error) {
var s LocalStatus
if err := d.getJSON(ctx, node, "/local/status", &s); err != nil {
return LocalStatus{}, err
}
return s, nil
}
// Instances returns the daemon's locally-known instance list.
//
// Depends on daemon-local-autonomy §1a. Until that lands, the daemon will
// return 404 and callers should t.Skip.
func (d *DaemonClient) Instances(ctx context.Context, node Node) ([]LocalInstance, error) {
var xs []LocalInstance
if err := d.getJSON(ctx, node, "/local/instances", &xs); err != nil {
return nil, err
}
return xs, nil
}
// Health hits the daemon's existing /health endpoint. Useful during
// scaffolding validation before the /local/* endpoints exist — confirms
// the harness can reach the daemon at all.
func (d *DaemonClient) Health(ctx context.Context, node Node) (types.NodeHealthResponse, error) {
var h types.NodeHealthResponse
if err := d.getJSON(ctx, node, "/health", &h); err != nil {
return types.NodeHealthResponse{}, err
}
return h, nil
}