mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-21 19:56:33 +00:00
068aef5589
* feat: scaffold DDIL E2E harness (cluster, SSH, daemon client) Phase 1 of the DDIL E2E test harness (mulga-siv-5.1). Adds the tests/e2e/ddil/ package tree with the three harness primitives required by every sibling scenario/fault bead: - harness/cluster.go — Cluster, Node, ClusterFromEnv, NodeFromEnv - harness/ssh.go — typed SSH transport backed by golang.org/x/crypto/ssh, interface so dry-run and tests can stub it - harness/daemon_client.go — thin HTTPS wrapper over /local/status, /local/instances (placeholder types until daemon-local-autonomy lands), plus /health for scaffolding validation Also adds doc.go and README.md (run instructions + env-var contract). All harness files carry //go:build e2e so default builds skip them. Plan: docs/development/improvements/ddil-e2e-test-harness.md §0, §1. * feat: add DDIL harness fault injection and link profiles Phase 2 of the DDIL E2E test harness (mulga-siv-5.2). Adds the fault injection and netem primitives consumed by Scenarios A–F: - harness/fault.go — PartitionNode (per-peer iptables DROP with orchestrator-SSH sanity check), HealNode, KillNATS, StartNATS, RestartDaemonOnly, WaitForMode (polls /local/status for typed DaemonMode enum) - harness/netem.go — ApplyNetem (tc qdisc replace, idempotent), ClearNetem (idempotent clear) - harness/profiles.go — LinkProfile struct plus LAN, WAN, LTEDegraded, SATCOM, HFData, Flapping constants (unvalidated approximations per plan §2 table; validation tracking lives in TEST_COVERAGE.md) All files carry //go:build e2e. Flapping profile is defined but ApplyNetem rejects it — flap orchestration lands with the scenarios that need it. Plan: docs/development/improvements/ddil-e2e-test-harness.md §1, §2. * feat: add DDIL harness witness VM helpers LaunchWitnessVM launches a counter VM via the cluster's AWS gateway, locates its QEMU host via ps, and captures a baseline counter reading. ReadCounter/AssertProgressed tunnel SSH through the host's QEMU hostfwd port to verify the workload progressed across a fault. README documents the new DDIL_WITNESS_* and DDIL_GUEST_SSH_* env vars. * feat: add DDIL harness snapshot and state assertions TakeSnapshot / AssertPreserved compare /local/instances pre and post fault, failing if IDs disappear or a running/pending instance regresses to a terminal state. LocalStateRevision / AssertMonotonic read the local state file revision from /local/status so scenarios can detect the daemon rewinding its persisted state. * feat: add DDIL harness reset, retry, and quarantine ResetAllNodes brings every node back to a clean baseline (iptables -F, tc netem clear on any interface carrying one, spinifex-nats and spinifex-daemon started) then waits for cluster health — daemon /health 200 plus NATS routez peer count matching len(nodes)-1, mirroring the shell suite's verify_nats_cluster predicate. AssertCleanState is a pre-scenario baseline check that recovers a dirty state via ResetAllNodes instead of failing outright, since a dirty baseline usually means a previous scenario's cleanup was skipped. Run wraps each scenario with quarantine handling (DDIL_QUARANTINED skip) and a single reset-and-retry on failure. First-attempt failures are not hidden; the retry is a diagnostic aid rather than a flake mask. bd: mulga-siv-5.5 * feat: DDIL scenarios skeleton (6x t.Skip + TestMain + coverage drift) Phase 1 of mulga-siv-5 (DDIL E2E harness): all six scenarios land as TestScenario<L>_... stubs that t.Skipf on their hardening-epic dependency. TestMain installs a SIGINT/SIGTERM trap that ResetAllNodes before exiting; DDIL_DRY_RUN=1 short-circuits cluster setup and logs the planned scenarios. TestCoverageDrift AST-parses sibling *_test.go files and reconciles them against the DDIL table in TEST_COVERAGE.md so Phase 2/3 flips cannot silently diverge doc from code. TestHarnessSmoke covers the AssertCleanState + ResetAllNodes round trip for the "full run on real cluster" acceptance bullet. * feat: add DDIL E2E CI workflow (workflow_dispatch only) Phase 1 CI wiring for mulga-siv-5. Mirrors e2e_real_multi's provision/bootstrap/teardown shape against the same ci-multi runner and tofu-cluster, but runs the Go harness on the runner (outside the cluster) so its SSH path survives the peer-traffic iptables rules that Scenario C installs on a cluster node. Triggers are workflow_dispatch only — nightly cron and on-main-merge come in mulga-siv-5.8 once ≥4 scenarios are asserting. results.json (go test -json) uploads as an artifact so flake-rate visibility can be built on gh run view + structured output, per the design doc.
140 lines
4.4 KiB
Go
140 lines
4.4 KiB
Go
//go:build e2e
|
|
|
|
package harness
|
|
|
|
import (
|
|
"context"
|
|
"crypto/tls"
|
|
"encoding/json"
|
|
"fmt"
|
|
"io"
|
|
"net"
|
|
"net/http"
|
|
"strconv"
|
|
"time"
|
|
|
|
"github.com/mulgadc/spinifex/spinifex/types"
|
|
)
|
|
|
|
// daemonPort is the cluster-manager HTTPS port (see
|
|
// spinifex/daemon/daemon.go ClusterManager). The harness targets the same
|
|
// port the production daemon exposes.
|
|
const daemonPort = 8443
|
|
|
|
// DaemonMode mirrors the daemon's operating mode reported by /local/status.
|
|
//
|
|
// The typed enum avoids stringly-typed assertions in scenarios. The concrete
|
|
// endpoint and JSON schema land with daemon-local-autonomy §1b; this file
|
|
// predates that PR and its types will be swapped for the daemon's own when
|
|
// it merges.
|
|
type DaemonMode string
|
|
|
|
const (
|
|
ModeUnknown DaemonMode = ""
|
|
ModeCluster DaemonMode = "cluster"
|
|
ModeStandalone DaemonMode = "standalone"
|
|
)
|
|
|
|
// LocalStatus is the response shape from GET /local/status.
|
|
//
|
|
// Placeholder fields only — the authoritative struct ships with
|
|
// daemon-local-autonomy §1b. Scenarios that depend on these fields remain
|
|
// t.Skip until that epic replaces this definition with an import from the
|
|
// daemon package.
|
|
type LocalStatus struct {
|
|
Node string `json:"node"`
|
|
Mode DaemonMode `json:"mode"`
|
|
Revision uint64 `json:"revision"`
|
|
NATS string `json:"nats"` // "connected" | "disconnected"
|
|
}
|
|
|
|
// LocalInstance is one row of GET /local/instances. Placeholder — see
|
|
// LocalStatus.
|
|
type LocalInstance struct {
|
|
InstanceID string `json:"instance_id"`
|
|
State string `json:"state"`
|
|
PID int `json:"pid,omitempty"`
|
|
}
|
|
|
|
// DaemonClient is a thin HTTPS client targeting a single daemon's local
|
|
// endpoints. It uses one *http.Client per node so connection reuse works
|
|
// across calls within a scenario.
|
|
type DaemonClient struct {
|
|
http *http.Client
|
|
}
|
|
|
|
// NewDaemonClient returns a DaemonClient with TLS verification disabled
|
|
// (the tofu-cluster uses self-signed per-node certs). Timeouts are set to
|
|
// 5s so a wedged daemon fails fast enough for scenario retry logic.
|
|
func NewDaemonClient() *DaemonClient {
|
|
return &DaemonClient{
|
|
http: &http.Client{
|
|
Timeout: 5 * time.Second,
|
|
Transport: &http.Transport{
|
|
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // self-signed test certs
|
|
},
|
|
},
|
|
}
|
|
}
|
|
|
|
func (d *DaemonClient) url(node Node, path string) string {
|
|
return "https://" + net.JoinHostPort(node.Addr, strconv.Itoa(daemonPort)) + path
|
|
}
|
|
|
|
func (d *DaemonClient) getJSON(ctx context.Context, node Node, path string, out any) error {
|
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, d.url(node, path), nil)
|
|
if err != nil {
|
|
return fmt.Errorf("ddil harness: daemon %s %s: %w", node.Name, path, err)
|
|
}
|
|
resp, err := d.http.Do(req)
|
|
if err != nil {
|
|
return fmt.Errorf("ddil harness: daemon %s %s: %w", node.Name, path, err)
|
|
}
|
|
defer func() { _ = resp.Body.Close() }()
|
|
|
|
if resp.StatusCode != http.StatusOK {
|
|
body, _ := io.ReadAll(resp.Body)
|
|
return fmt.Errorf("ddil harness: daemon %s %s: status %d: %s",
|
|
node.Name, path, resp.StatusCode, string(body))
|
|
}
|
|
if err := json.NewDecoder(resp.Body).Decode(out); err != nil {
|
|
return fmt.Errorf("ddil harness: daemon %s %s decode: %w", node.Name, path, err)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// Status returns the daemon's self-reported local status.
|
|
//
|
|
// Depends on daemon-local-autonomy §1b. Until that lands, the daemon will
|
|
// return 404 and callers should t.Skip.
|
|
func (d *DaemonClient) Status(ctx context.Context, node Node) (LocalStatus, error) {
|
|
var s LocalStatus
|
|
if err := d.getJSON(ctx, node, "/local/status", &s); err != nil {
|
|
return LocalStatus{}, err
|
|
}
|
|
return s, nil
|
|
}
|
|
|
|
// Instances returns the daemon's locally-known instance list.
|
|
//
|
|
// Depends on daemon-local-autonomy §1a. Until that lands, the daemon will
|
|
// return 404 and callers should t.Skip.
|
|
func (d *DaemonClient) Instances(ctx context.Context, node Node) ([]LocalInstance, error) {
|
|
var xs []LocalInstance
|
|
if err := d.getJSON(ctx, node, "/local/instances", &xs); err != nil {
|
|
return nil, err
|
|
}
|
|
return xs, nil
|
|
}
|
|
|
|
// Health hits the daemon's existing /health endpoint. Useful during
|
|
// scaffolding validation before the /local/* endpoints exist — confirms
|
|
// the harness can reach the daemon at all.
|
|
func (d *DaemonClient) Health(ctx context.Context, node Node) (types.NodeHealthResponse, error) {
|
|
var h types.NodeHealthResponse
|
|
if err := d.getJSON(ctx, node, "/health", &h); err != nil {
|
|
return types.NodeHealthResponse{}, err
|
|
}
|
|
return h, nil
|
|
}
|