Files
spinifex/tests/e2e/single/tests_test.go
T
Bryn Mailer 15dede64ad feat: predastore compaction E2E (#407)
* feat: add predastore compaction E2E

Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.

* fix: compaction E2E gate writes churn to its own bucket

The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.

Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.

* test: run predastore compaction gate before destructive node tests

The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.

* fix: race-free SSH output capture + empty-tolerant du probe

The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:

- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
  which x/crypto/ssh copies in two concurrent goroutines — a data race on a
  non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
  combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
  A missing sentinel is reported as output lost in SSH retrieval; a present
  sentinel with no number is treated as a 0 measurement (anomalous, logged)
  instead of crashing the gate.

The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.

* test(e2e): replace predastore du-reclaim gate with object lifecycle contract

The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.

Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.

* test(e2e): drop orphaned multinode TestPredastoreCompaction

Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
2026-06-23 16:19:10 +10:00

200 lines
5.5 KiB
Go

//go:build e2e
package single
import "testing"
// Top-level Test* entry points. Each delegates to a runX function in the
// matching <name>_test.go file. Every run* self-bootstraps prerequisites so
// `go test -run TestX` works in isolation.
//
// Parallel bucket (t.Parallel): read-only or own-everything tests that never
// mutate the singleton instance, default VPC, or default SG.
// Sequential: tests that boot, mutate, or snapshot the singleton VM; IAM tests
// that share package-scoped IAM state; TestFinalClusterStats (last gate).
// --- Bucket #1: parallel-safe ---
func TestEnvironment(t *testing.T) {
t.Parallel()
runEnvironment(t, requireSingleNodeFixture(t))
}
func TestDiscovery(t *testing.T) {
t.Parallel()
runDiscovery(t, requireSingleNodeFixture(t))
}
func TestSerialConsoleAccess(t *testing.T) {
t.Parallel()
runSerialConsoleAccess(t, requireSingleNodeFixture(t))
}
func TestKeyPairs(t *testing.T) {
t.Parallel()
runKeyPairs(t, requireSingleNodeFixture(t))
}
func TestImage(t *testing.T) {
t.Parallel()
runImage(t, requireSingleNodeFixture(t))
}
func TestTagManagement(t *testing.T) {
t.Parallel()
runTagManagement(t, requireSingleNodeFixture(t))
}
func TestAccountScoping(t *testing.T) {
t.Parallel()
runAccountScoping(t, requireSingleNodeFixture(t))
}
// TestPredastoreObjectLifecycle owns a dedicated bucket and never touches the
// singleton VM or default VPC/SG, so it is parallel-safe.
func TestPredastoreObjectLifecycle(t *testing.T) {
t.Parallel()
runPredastoreObjectLifecycle(t, requireSingleNodeFixture(t))
}
func TestVPCSubnetE2E(t *testing.T) {
t.Parallel()
runVPCSubnetE2E(t, requireSingleNodeFixture(t))
}
func TestRouteTableValidation(t *testing.T) {
t.Parallel()
runRouteTableValidation(t, requireSingleNodeFixture(t))
}
// TestReplaceRouteConvergence owns its own scratch VPCs/IGWs end to end and
// never touches the singleton or default VPC, so it is parallel-safe.
func TestReplaceRouteConvergence(t *testing.T) {
t.Parallel()
runReplaceRouteConvergence(t, requireSingleNodeFixture(t))
}
// --- Sequential: singleton VM lifecycle ---
// TestClusterStatsCLI exercises the spx cluster CLI; its get-vms baseline is
// concurrency-tolerant (other suites may have VMs on the node).
func TestClusterStatsCLI(t *testing.T) {
runClusterStatsCLI(t, requireSingleNodeFixture(t))
}
// TestDefaultSGReachabilityBaseline, TestNewVPCEgressBaseline, and TestSameSGComms
// own all mutable resources and run before the singleton launch.
func TestDefaultSGReachabilityBaseline(t *testing.T) {
runDefaultSGReachabilityBaseline(t, requireSingleNodeFixture(t))
}
func TestNewVPCEgressBaseline(t *testing.T) {
runNewVPCEgressBaseline(t, requireSingleNodeFixture(t))
}
func TestSameSGComms(t *testing.T) {
runSameSGComms(t, requireSingleNodeFixture(t))
}
func TestInstanceLaunch(t *testing.T) {
runInstanceLaunch(t, requireSingleNodeFixture(t))
}
func TestInstanceClusterStats(t *testing.T) {
runInstanceClusterStats(t, requireSingleNodeFixture(t))
}
func TestInstanceMetadata(t *testing.T) {
runInstanceMetadata(t, requireSingleNodeFixture(t))
}
func TestSSHProbe(t *testing.T) {
runSSHProbe(t, requireSingleNodeFixture(t))
}
func TestConsoleOutput(t *testing.T) {
runConsoleOutput(t, requireSingleNodeFixture(t))
}
// TestENIHotplug hot-plugs a secondary ENI onto the freshly-booted singleton
// and asserts the NIC reaches the guest, then restores it. Placed before the
// stop/start churn so the singleton is known running + SSH-healthy.
func TestENIHotplug(t *testing.T) {
runENIHotplug(t, requireSingleNodeFixture(t))
}
func TestVolumeLifecycle(t *testing.T) {
runVolumeLifecycle(t, requireSingleNodeFixture(t))
}
func TestVolumeStatus(t *testing.T) {
runVolumeStatus(t, requireSingleNodeFixture(t))
}
// TestVolumeDurability stops/starts the singleton; sequential, leaves it running.
func TestVolumeDurability(t *testing.T) {
runVolumeDurability(t, requireSingleNodeFixture(t))
}
func TestSnapshotLifecycle(t *testing.T) {
runSnapshotLifecycle(t, requireSingleNodeFixture(t))
}
func TestSnapshotBackedLaunch(t *testing.T) {
runSnapshotBackedLaunch(t, requireSingleNodeFixture(t))
}
func TestCreateImage(t *testing.T) {
runCreateImage(t, requireSingleNodeFixture(t))
}
func TestSecurityGroupEgress(t *testing.T) {
runSecurityGroupEgress(t, requireSingleNodeFixture(t))
}
func TestStopStart(t *testing.T) {
runStopStart(t, requireSingleNodeFixture(t))
}
func TestAttachToStoppedError(t *testing.T) {
runAttachToStoppedError(t, requireSingleNodeFixture(t))
}
func TestModifyInstanceAttribute(t *testing.T) {
runModifyInstanceAttribute(t, requireSingleNodeFixture(t))
}
func TestRebootInstance(t *testing.T) {
runRebootInstance(t, requireSingleNodeFixture(t))
}
func TestRunInstancesMultiCount(t *testing.T) {
runRunInstancesMultiCount(t, requireSingleNodeFixture(t))
}
// --- Sequential: shared-state / sub-test-parallel Tests ---
func TestNegativeErrorPaths(t *testing.T) {
runNegativeErrorPaths(t, requireSingleNodeFixture(t))
}
func TestNATGateway(t *testing.T) {
runNATGateway(t, requireSingleNodeFixture(t))
}
// TestInstanceEIP allocates an Elastic IP, associates it to a throwaway VM,
// and asserts the EIP datapath flips on/off with the association. Sequential:
// it authorizes ingress on the shared default SG.
func TestInstanceEIP(t *testing.T) {
runInstanceEIP(t, requireSingleNodeFixture(t))
}
func TestSGToSGDatapath(t *testing.T) {
runSGToSGDatapath(t, requireSingleNodeFixture(t))
}
// TestFinalClusterStats runs as the last sequential test.
func TestFinalClusterStats(t *testing.T) {
runFinalClusterStats(t, requireSingleNodeFixture(t))
}