mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 19:01:42 +00:00
15dede64ad
* feat: add predastore compaction E2E
Inject a short compaction interval into test-cluster predastore config via a
new --predastore-compaction-interval flag on admin init/join, gated so
production output stays byte-identical when unset. Add a harness helper that
drives write-then-delete churn against predastore S3, brackets it with
du-over-SSH usage, and asserts end-of-churn usage settles below 70% of the
churn peak; wire it into the multinode suite as TestPredastoreCompaction.
* fix: compaction E2E gate writes churn to its own bucket
The reclaim gate signed PutObject against the config-defined system
'predastore' bucket, which belongs to account 000000000000 with only a
static [[auth]] grant. The harness admin identity is a different IAM
account, so predastore returned 403 AccessDenied cross-account.
Create a per-run bucket the admin identity owns (AdministratorAccess)
and churn into it. Objects still erasure-code into distributed/nodes/
node-* so the du-based reclaim measurement is unchanged. The gateway
(:9999) does not proxy S3, so the client stays predastore-direct.
* test: run predastore compaction gate before destructive node tests
The compaction reclaim gate was declared last in the multinode suite, so it
measured shard-store usage after NodeFailure/NodeRecovery had torn down and
restarted node2. NodeRecovery only gates on NATS/gateway/EC2 readiness, not
predastore, so node2's shard store was still settling and the du baseline came
back empty. Move the test before the destructive node tests so it runs against
a pristine, fully-converged cluster primed by the volume tests.
* fix: race-free SSH output capture + empty-tolerant du probe
The compaction reclaim gate crashed with ParseInt("") when node1's du
probe returned empty combined output. Harden the probe and the SSH layer:
- ssh.Run used one bytes.Buffer for both session.Stdout and session.Stderr,
which x/crypto/ssh copies in two concurrent goroutines — a data race on a
non-concurrent-safe type. Switch to session.CombinedOutput, which guards the
combined stream with a mutex-wrapped singleWriter.
- shardStoreUsageBytes now terminates the remote command with a sentinel echo.
A missing sentinel is reported as output lost in SSH retrieval; a present
sentinel with no number is treated as a 0 measurement (anomalous, logged)
instead of crashing the gate.
The sentinel lets the next run distinguish a transport-layer drop from a
genuinely empty remote measurement.
* test(e2e): replace predastore du-reclaim gate with object lifecycle contract
The multinode compaction gate measured shard-store disk usage over SSH (du)
and asserted reclaim after churn. It could never pass: segments only seal at
the 4 GiB roll threshold, so test-scale churn produces no compactable segment,
and the du probe read 0 bytes before and after. Reclaim correctness is already
covered deterministically by predastore's store-level unit tests.
Delete AssertPredastoreCompaction and its du/churn/poll/sentinel machinery,
the harness unit tests that exercised them, and the multinode TestPredastore
Compaction entry. Add a single-node TestPredastoreObjectLifecycle that checks
the user-visible contract: an object is readable after PutObject, gone after
DeleteObject (GET returns NoSuchKey and ListObjectsV2 is empty), and the store
keeps serving fresh writes. newPredastoreS3 is retained, refactored to take a
host so it works outside the multinode cluster type.
* test(e2e): drop orphaned multinode TestPredastoreCompaction
Complete the du-reclaim -> object-lifecycle refactor: remove the
multinode TestPredastoreCompaction caller (and its now-unused context
and harness imports) that still referenced the deleted
harness.AssertPredastoreCompaction, which broke e2e-build. Wire the
single-node TestPredastoreObjectLifecycle entry that replaces it.
200 lines
5.5 KiB
Go
200 lines
5.5 KiB
Go
//go:build e2e
|
|
|
|
package single
|
|
|
|
import "testing"
|
|
|
|
// Top-level Test* entry points. Each delegates to a runX function in the
|
|
// matching <name>_test.go file. Every run* self-bootstraps prerequisites so
|
|
// `go test -run TestX` works in isolation.
|
|
//
|
|
// Parallel bucket (t.Parallel): read-only or own-everything tests that never
|
|
// mutate the singleton instance, default VPC, or default SG.
|
|
// Sequential: tests that boot, mutate, or snapshot the singleton VM; IAM tests
|
|
// that share package-scoped IAM state; TestFinalClusterStats (last gate).
|
|
|
|
// --- Bucket #1: parallel-safe ---
|
|
|
|
func TestEnvironment(t *testing.T) {
|
|
t.Parallel()
|
|
runEnvironment(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestDiscovery(t *testing.T) {
|
|
t.Parallel()
|
|
runDiscovery(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestSerialConsoleAccess(t *testing.T) {
|
|
t.Parallel()
|
|
runSerialConsoleAccess(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestKeyPairs(t *testing.T) {
|
|
t.Parallel()
|
|
runKeyPairs(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestImage(t *testing.T) {
|
|
t.Parallel()
|
|
runImage(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestTagManagement(t *testing.T) {
|
|
t.Parallel()
|
|
runTagManagement(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestAccountScoping(t *testing.T) {
|
|
t.Parallel()
|
|
runAccountScoping(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// TestPredastoreObjectLifecycle owns a dedicated bucket and never touches the
|
|
// singleton VM or default VPC/SG, so it is parallel-safe.
|
|
func TestPredastoreObjectLifecycle(t *testing.T) {
|
|
t.Parallel()
|
|
runPredastoreObjectLifecycle(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestVPCSubnetE2E(t *testing.T) {
|
|
t.Parallel()
|
|
runVPCSubnetE2E(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestRouteTableValidation(t *testing.T) {
|
|
t.Parallel()
|
|
runRouteTableValidation(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// TestReplaceRouteConvergence owns its own scratch VPCs/IGWs end to end and
|
|
// never touches the singleton or default VPC, so it is parallel-safe.
|
|
func TestReplaceRouteConvergence(t *testing.T) {
|
|
t.Parallel()
|
|
runReplaceRouteConvergence(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// --- Sequential: singleton VM lifecycle ---
|
|
|
|
// TestClusterStatsCLI exercises the spx cluster CLI; its get-vms baseline is
|
|
// concurrency-tolerant (other suites may have VMs on the node).
|
|
func TestClusterStatsCLI(t *testing.T) {
|
|
runClusterStatsCLI(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// TestDefaultSGReachabilityBaseline, TestNewVPCEgressBaseline, and TestSameSGComms
|
|
// own all mutable resources and run before the singleton launch.
|
|
func TestDefaultSGReachabilityBaseline(t *testing.T) {
|
|
runDefaultSGReachabilityBaseline(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestNewVPCEgressBaseline(t *testing.T) {
|
|
runNewVPCEgressBaseline(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestSameSGComms(t *testing.T) {
|
|
runSameSGComms(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestInstanceLaunch(t *testing.T) {
|
|
runInstanceLaunch(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestInstanceClusterStats(t *testing.T) {
|
|
runInstanceClusterStats(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestInstanceMetadata(t *testing.T) {
|
|
runInstanceMetadata(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestSSHProbe(t *testing.T) {
|
|
runSSHProbe(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestConsoleOutput(t *testing.T) {
|
|
runConsoleOutput(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// TestENIHotplug hot-plugs a secondary ENI onto the freshly-booted singleton
|
|
// and asserts the NIC reaches the guest, then restores it. Placed before the
|
|
// stop/start churn so the singleton is known running + SSH-healthy.
|
|
func TestENIHotplug(t *testing.T) {
|
|
runENIHotplug(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestVolumeLifecycle(t *testing.T) {
|
|
runVolumeLifecycle(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestVolumeStatus(t *testing.T) {
|
|
runVolumeStatus(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// TestVolumeDurability stops/starts the singleton; sequential, leaves it running.
|
|
func TestVolumeDurability(t *testing.T) {
|
|
runVolumeDurability(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestSnapshotLifecycle(t *testing.T) {
|
|
runSnapshotLifecycle(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestSnapshotBackedLaunch(t *testing.T) {
|
|
runSnapshotBackedLaunch(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestCreateImage(t *testing.T) {
|
|
runCreateImage(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestSecurityGroupEgress(t *testing.T) {
|
|
runSecurityGroupEgress(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestStopStart(t *testing.T) {
|
|
runStopStart(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestAttachToStoppedError(t *testing.T) {
|
|
runAttachToStoppedError(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestModifyInstanceAttribute(t *testing.T) {
|
|
runModifyInstanceAttribute(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestRebootInstance(t *testing.T) {
|
|
runRebootInstance(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestRunInstancesMultiCount(t *testing.T) {
|
|
runRunInstancesMultiCount(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// --- Sequential: shared-state / sub-test-parallel Tests ---
|
|
|
|
func TestNegativeErrorPaths(t *testing.T) {
|
|
runNegativeErrorPaths(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestNATGateway(t *testing.T) {
|
|
runNATGateway(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// TestInstanceEIP allocates an Elastic IP, associates it to a throwaway VM,
|
|
// and asserts the EIP datapath flips on/off with the association. Sequential:
|
|
// it authorizes ingress on the shared default SG.
|
|
func TestInstanceEIP(t *testing.T) {
|
|
runInstanceEIP(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
func TestSGToSGDatapath(t *testing.T) {
|
|
runSGToSGDatapath(t, requireSingleNodeFixture(t))
|
|
}
|
|
|
|
// TestFinalClusterStats runs as the last sequential test.
|
|
func TestFinalClusterStats(t *testing.T) {
|
|
runFinalClusterStats(t, requireSingleNodeFixture(t))
|
|
}
|