mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-22 12:16:52 +00:00
Josh Siviour
98ec781629
* feat: introduce the EBSProvider interface and control-plane metadata store
Phase 1 of decoupling the control plane from viperblock. Adds a
provider-neutral block-storage contract that spinifex owns end to end.
ebsprovider defines EBSProvider, whose operation set and idempotency rules
follow the CSI controller service, with NATS request/reply as the transport
rather than CSI's gRPC. Every request and response carries an explicit schema
version so wire skew fails loudly at the boundary instead of silently decoding
a changed payload. The package deliberately exposes no viperblock types:
provider-specific state is an opaque handle. Ships with a NATS client against
the ebs.provider.v1.* subjects and an in-memory fake for tests.
ebsmetadata gives spinifex its own volume and AMI documents under a spinifex
key prefix, so the control plane can answer DescribeVolumes and DescribeImages
without reading the storage engine's persisted state.
The EC2 volume and snapshot handlers gain provider-backed paths for create,
describe, modify, attach-state and delete. Provider injection is optional for
now, so the existing engine path stays the default until the viperblockd side
serves the new subjects and the metadata migration lands.
vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol
it used and is now a local helper.
* feat: serve the EBS provider contract from viperblockd
Gives the ebsprovider NATS client a server. Volume create, describe, expand
and delete, plus snapshot create and delete, are now handled by the daemon
that owns BaseDir and the mounted-volume registry, which is where viperblock
engine construction belongs.
Every handler prefers an already-mounted engine over opening a second one on
the same volume, and expand refuses outright while the volume is mounted.
Opening a second engine on a live volume is the double-writer hazard this
decoupling exists to remove, so the boundary enforces it rather than trusting
callers. Requests carrying an unrecognised schema version are rejected at the
boundary instead of being decoded on a best-effort basis.
Snapshot creation replies immediately with an operation ID and completes
asynchronously on a per-snapshot subject, since a large volume will outrun the
request timeout. Draining the volume stays with the caller: it coordinates
with the guest and is not something the storage boundary can reach.
The snapshot create subject gains an explicit create verb. Addressing it as
ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject
and left the wildcard unsubscribable.
Publish and unpublish are not served yet; the mount and unmount logic they
would front is still inline in launchService and needs extracting first.
Also adds a conformance suite that runs one set of contract assertions against
both the in-memory fake and the NATS client, so each side is checked against
the contract rather than against the other's structs.
* feat: let an operator select the EBS provider
Adds an [ebs] provider key and wires the NATS provider into the EC2 volume,
snapshot, image and instance services at the daemon's composition root. Until
now nothing injected a provider, so the provider-backed code paths in those
services were unreachable.
The key is provider-neutral rather than nested under [viperblock], because it
names the boundary rather than one provider's settings; a second provider will
not need it renamed. An empty value resolves to "embedded", so deployed config
files that predate the key keep today's behaviour, and an unrecognised value
is a startup error rather than a silent fallback.
Startup probes the provider's capability verb and warns when nothing answers.
The daemon and viperblockd start independently, so an unanswered probe is not
fatal, but without it a misconfigured node fails much later as an unexplained
request timeout on the first EBS call.
"viperblockd" stays experimental: under it volume metadata lives in the
control plane's own document, and the leak reaper, GC and admin paths still
read viperblock's config.json, so they cannot see tenant or tag data for
volumes created that way.
* fix: make every volume-metadata consumer provider-aware
Several readers and writers of volume metadata were never given a provider
branch, so under the viperblockd provider they silently used the wrong store.
Each of these is a correctness bug rather than a missing feature, because the
control plane had already stopped writing the document they read.
Modifications had nowhere to live: ModifyVolume built its result in memory and
dropped it, and DescribeVolumesModifications read it back out of viperblock's
config.json, so a describe straight after a successful modify returned nothing.
Volume metadata now carries its own modification record.
The leak reaper read attachment and tags from config.json and wrote orphan
markers to tags.json, neither of which exists for a provider-managed volume, so
a leaked volume was invisible to it and any mark it did write could not be read
back. Tag mutations had the same split: a tag applied after creation never
reached the document DescribeVolumes reads.
Snapshot deletion could not see a provider-managed clone, because a clone
records its source only in the control-plane document. That let a delete strip
chunks a live clone still reads, so it is fixed here rather than left for the
migration.
Volume enumeration and DescribeVolumeStatus were likewise scanning object
prefixes that a provider-managed volume does not create.
Also adds an encryption flag to the volume document, which DescribeVolumes had
been reporting as false unconditionally and snapshots could not carry at all,
and a list operation for AMIs so the image path can follow later.
The legacy path is unchanged; its invariant tests pass unmodified.
* feat: backfill ebsmetadata documents from legacy volume and AMI state
DescribeVolumes under the viperblockd provider enumerates only ebsmetadata
documents, so a volume created before the switch had no document and was
invisible rather than merely slow to read. A per-object read fallback cannot
fix enumeration, so both a backfill and a fallback are needed.
Add an object-store migration kind to the migrate framework. Predastore has no
conditional write, so an object-based version stamp would be last-writer-wins
across nodes; RunObject stamps progress in a caller-supplied JetStream KV
bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a
read-then-write rather than a compare-and-swap, so correctness rests on every
ObjectMigration.Run being safe to re-execute.
The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the
numbered config migrations: it needs viperblock's VBState schema to decode the
legacy config.json, and migrate is imported by every handler that stamps a KV
bucket version. It converts what it can reach and skips unreadable objects with
a log, so one corrupt volume cannot hide the rest of the fleet.
ebsmetadata gains the legacy read fallback for the window before the migration
has run everywhere. It stays free of viperblock by taking the decode as an
injected function value, wired at the composition root, and the fallback is
retired by leaving it unset. A fallback read never writes the converted value
back, which would race the migration.
Prefix enumeration pages to exhaustion. Stopping at the first page would let
the backfill stamp itself complete having converted only the volumes it
happened to see, reintroducing the invisible-volume failure it exists to
prevent.
The migration runs from configureEBSProvider, so only when the provider is
selected; backfilling while the embedded engine is still authoritative would
produce documents that immediately go stale. Failure is fatal rather than
advisory, unlike the capability probe beside it. This makes the switch to
viperblockd one-way, which is now recorded on the config field.
* feat: serve volume publish and unpublish from viperblockd
PublishVolume and UnpublishVolume were the last unserved verbs on the provider
contract, and they are the ones that attach a volume to a VM. Until they are
served the provider path cannot boot an instance, so provider = "viperblockd"
cannot be verified end to end.
The mount and unmount logic was two large anonymous handler bodies inline in
launchService. Rather than write a second mount path, extract each into a named
function and have both the legacy ebs.mount/ebs.unmount handlers and the new
provider handlers call it. Two divergent implementations of nbdkit spawning and
the seal decision would be far worse than the inline bodies. The extraction
relocates code without rewriting it; response publishing stays in each handler
so the provider never publishes to ebs.mount.response.
Publish is idempotent: a volume this node already has mounted returns its
existing attachment rather than starting a second nbdkit against it, which is
the double-writer hazard the boundary exists to prevent. Unpublish preserves
the invariant that a failed seal leaves the volume mounted so a retry can
re-attempt it, and reports an error rather than success.
Both verbs are node-addressed, so they subscribe directly rather than through
the spinifex-workers queue group, and are registered only when NodeName is set.
There is no queue-group fallback: a subject that already names one node has
nothing sensible to fall back to.
Read-only attachment is refused rather than silently granted read-write, since
nbdkit is started read-write or not at all.
Also fixes a latent panic carried by the extracted mount path: viperblock.New
returns a nil VB alongside its error, and SetCacheSize was called on it before
the error was checked, so a failing New would have taken down the handler.
* feat: read and write AMI metadata through ebsmetadata
The backfill migration wrote ebsmetadata AMI documents that nothing read: every
AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at
{imageID}/config.json. Give the image service provider branches for the AMI
read, write and enumerate paths, with enumeration going through ListAMIs rather
than an object-prefix scan, as DescribeVolumes already does.
AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a
viperblock type on an interface boundary in the instance package. It now returns
ebsmetadata.AMI on both paths, the embedded one converting before it returns, so
the boundary names no viperblock type whichever provider is selected.
The legacy-to-document mapping had accumulated three copies. Consolidate them in
ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata
itself stays free of viperblock, and cover it with a reflective test that fails
when a field is added to either schema without being wired through. Sixteen
fields copied by hand in three places would have dropped one silently.
The admin image paths read whichever representation exists and write back to
that same one, so a provider-managed AMI never gains a spurious config.json.
FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan:
missing a dependent there deletes an AMI a live volume still needs, the same
class of bug as the snapshot clone path fixed earlier on this branch.
That union is built by hand rather than by wiring Store.ListVolumes' fallback,
which swallows per-ID read errors. This path deliberately fails closed on a
transport error so a delete cannot proceed on a partial picture, and the
fallback would have quietly turned that into failing open.
The CLI image import now also writes the document, best-effort, so an AMI
imported after the migration is not left reachable only through the fallback.
* fix(ec2): resolve volume config through ebsmetadata when a provider is configured
A provider-created volume has no vol-*/config.json — its metadata lives in
the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy
key unconditionally, so AttachVolume reported a volume that DescribeVolumes
had just listed as available as InvalidVolume.NotFound.
Routing through ebsmetadata.Store also puts this path behind the legacy
fallback, so a volume created before the provider switch stays attachable.
The NoSuchKey remap now names the key it missed; its silence made the
original failure invisible in the daemon logs on every node.
* test(bench): add an EC2 control-plane latency benchmark
Measures the EBS control-plane operations either side of the provider
boundary, so the cost of the NATS hop can be stated rather than guessed.
Reports median and p95 with a bootstrap CI on the median difference, and
labels a delta indicative unless that CI excludes zero.
* fix(ec2): drain the volume before a provider snapshot
The legacy CreateSnapshot branch drains first so the snapshot reads a
current checkpoint; the provider branch did not, and viperblockd documents
the drain as the caller's responsibility. Both sides deferred to the other,
so an attached volume snapshotted stale data with no error raised.
DrainVolume and volumeAttachment now take the fallback state and instance
as strings rather than a viperblock.VolumeMetadata, removing a viperblock
type from an exported signature.
* feat(ec2): route the instance root volume through the EBS provider
RunInstances built its own viperblock engine to clone the AMI snapshot into
the root volume, which made every launch a second writer against a volume
viperblockd already serves. The launch now calls the provider's CreateVolume
with the AMI's snapshot and source volume, and records the result in
ebsmetadata so DescribeVolumes and attach can see it.
A snapshot clone needs both the snapshot ID and its source volume ID to
resolve base blocks, so CreateVolume in the volume handler now sends
SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected
every create-volume --snapshot-id outright.
The gp3 IOPS and throughput constants move to spinifex/types so the instance
and volume handlers share one definition rather than duplicating literals.
* feat(ebsprovider): add a bounded volume seed to CreateVolume
The EFI variable store must be initialised from the launching node's firmware
VARS template, byte for byte. Passing a host path would not work: CreateVolume
is queue-subscribed, so any node can service it, and a foreign node's OVMF
package would bind the guest's pflash region to the wrong firmware build.
CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd
writes them at offset 0 in the engine it already builds. The seed applies only
when the create actually allocates the volume, so a relaunch never overwrites
a BootOrder the guest has since written.
JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes
bounds the seed well inside the NATS max_payload the cluster runs with and
both ends reject an oversized seed with a clear error rather than letting the
publish fail. VolumeSeeding advertises the behaviour so a provider that cannot
seed says so instead of silently dropping the bytes.
* feat(ec2): route the EFI variable store through the EBS provider
prepareEFIVolume built a control-plane viperblock engine to seed the variable
store from the firmware VARS template, making every UEFI launch a second
writer against a volume viperblockd already serves. It now calls the
provider's CreateVolume with the template as seed bytes, so the store is
allocated and written by the daemon that owns the volume.
The template is still read on the launching node and shipped with the request.
CreateVolume is queue-subscribed, so any node can service it, and a VARS
region seeded from a foreign node's OVMF build would bind the guest's pflash
to the wrong firmware.
pflash refuses a VARS region that is not byte-exact, so the returned capacity
is checked against the template length; a provider that rounded up to a GiB
boundary would otherwise yield a guest that cannot boot.
The engine path stays behind a nil-provider guard until the legacy branch goes.
* feat(ec2): route image snapshots through the EBS provider
snapshotRunningVolume and snapshotStoppedVolume each built a control-plane
viperblock engine to take the snapshot backing a new AMI, making CreateImage
a second writer against a volume viperblockd already serves. Both now call
the provider's CreateSnapshot with the volume's provider handle.
The running path keeps its drain, which is what makes the snapshot current
rather than one predating a guest-triggered write. The stopped path needs no
drain, and the daemon already prefers a live mounted engine over opening a
second one, so it handles both cases itself.
getVolumeConfig read the legacy config.json, which a provider-created volume
does not have, so every CreateImage against one failed. It now reads the
ebsmetadata document under a provider, the same way attach does.
* fix(daemon): only the owning node creates an image for a stopped instance
handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which
only the node holding the VM passes, so a running instance is imaged once.
A stopped instance lives in the cluster-replicated KV instead, which every
node reads successfully, so all of them ran the full snapshot and AMI
creation. The caller saw the first reply and the rest leaked: on a three-node
cluster one create-image produced three AMIs and three snapshots.
A stopped instance records its owner in LastNode, so a node that is not the
owner now declines. Gather tallies error frames and keeps waiting, returning
only on a success, so declining cannot preempt the owner's reply.
An instance stopped by a build that never set LastNode elects an owner
through the KV's compare-and-swap, since having every node decline would make
create-image fail permanently for those records.
* perf(ec2): serve DescribeVolumes by ID from the volume documents
The provider branch called ListVolumes unconditionally and filtered in
memory, so asking for one volume read and decoded every volume document in
the bucket. The non-provider branch already had the equivalent fast path.
Explicit IDs now fetch their documents directly. The tenant check moves with
them: a document belonging to another account reports not-found rather than
being returned, matching what the enumeration filter did.
Measured against a memory object store, the fast path stays flat near 20us
regardless of bucket size while the enumeration grows linearly, reaching
4.6ms at 500 volumes.
* chore(lint): ban the viperblock import from the EC2 control plane
The control plane reaches block storage through ebsprovider now, but nothing
stopped a handler from importing the engine again. That is not hypothetical:
the instance handler first imported viperblock for a config struct alone, and
the engine construction followed thirteen days later.
depguard denies the import across handlers/, which matters because
viperblock.New starts a chunk uploader — every construction is a writer
against a volume viperblockd already serves. A types-only import is denied
too, since that is exactly how the last one started.
The four files still carrying the pre-provider engine path behind a
nil-provider guard are exempted by name. The exemption list is the remaining
work: it shrinks to nothing when that path goes.
* fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot
CopySnapshot wrote a control-plane snapshot document under the new ID and
nothing else. The copy described correctly and CreateVolume against it
succeeded, so it looked like it worked, but the new ID had no viperblock
snapshot metadata behind it and the volume failed when the guest attached it.
Producing the metadata means re-sealing it under the destination ID. On an
encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the
sealing helpers are unexported, so spinifex cannot write it. Copying only the
unencrypted form would be dead code on any cluster with encryption enabled.
Refusing with UnsupportedOperation fails at the call the caller can act on
rather than at attach time. Source lookup and the ownership check still run
first, so a bad ID or another tenant's snapshot reports that instead.
* test(ec2): give handler test fixtures their own legacy config.json shape
Handler tests built the legacy config.json fixture by importing
viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly.
Mirror that shape in ebsmetadata/vblegacy, next to the production code
that already reads it (ebsmetadatabackfill), and switch the
fixture-only usages across the volume/snapshot/image handler tests to
it. A byte-for-byte equivalence test against the real viperblock types
guards the mirror from drifting out of sync.
Call sites that hand a viperblock type into putVolumeConfig,
mergeVolumeConfig, or buildVBConfig are untouched, since those
functions still name the real type in their signature.
* feat(ebsprovider): add CopySnapshot to the EBS provider contract
Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy
NATS subject, MemoryProvider and NATSProvider implementations, conformance
coverage) and a viperblockd handler that duplicates a snapshot's metadata via
viperblock.CopySnapshotMeta on the source volume's VB.
* feat(ec2): serve CopySnapshot through the EBS provider
CopySnapshot has been refusing with UnsupportedOperation because a copy needs
its own snapshot metadata on the backend under the new ID, and on an encrypted
volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the
control plane cannot re-seal it. The provider contract now carries a copy
operation, so the storage engine does it.
The provider path generates a new snapshot ID, has the provider duplicate the
backend metadata under it, then writes the control-plane document. Rollback
mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot
rather than leaving one nothing references. Ownership is still checked against
the source before anything is created.
The no-provider path keeps refusing. It has no way to produce the backend
metadata, and the copy it used to write described fine and then failed at
attach — a resource that fails as far as possible from the call that made it.
Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot
reached the provider rather than existing only as a control-plane document.
That assertion is what catches the old behaviour.
* refactor: remove the embedded EBS engine path from the EC2 control plane
viperblockd is now the only EBS provider. The pre-provider path that
constructed a viperblock engine in-process is deleted, and with it every
github.com/mulgadc/viperblock import under spinifex/handlers/ (tests
included). The depguard ec2-control-plane rule drops all four file
exemptions and the !$test carve-out, so the rule now reads simply as
"nothing under spinifex/handlers/** may import viperblock".
viperblock.VolumeConfig was also a data carrier on the provider path. It
is replaced by the spinifex-native ebsmetadata.Volume rather than the
vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is
decoding old objects, while ebsmetadata.Volume already carries every field
the call sites read, so the conversion step disappears instead of being
reimplemented against a legacy shape.
The embedded-only helpers (volumeConfigWrapper, putVolumeConfig,
mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached,
deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json
prefix scan) had exactly one production caller chain each, all through a
nil-provider branch, and are removed. The snapshot-dependency guard on
DeleteVolume is provider-agnostic and is kept, now running on the provider
path where it previously did not.
config: viperblockd is the default and an empty Provider resolves to it.
An explicit provider = "embedded" is rejected at config load rather than
silently upgraded, because starting anyway would migrate a node's volumes
to ebsmetadata one-way without the operator asking. The generated
spinifex.toml template emits viperblockd.
* fix(images): write the imported AMI document from the config the import wrote
ImportDiskImage creates the AMI's snapshot and records it on its own
*viperblock.VB, which viperblock.New allocates fresh, so the value lands only
in {imageID}/config.json. The import command was persisting the ebsmetadata
document from its pre-import manifest copy, whose SnapshotID is still empty.
Store.GetAMI only consults the legacy config.json fallback when the document
is missing, so that empty-SnapshotID document permanently shadowed the real
value and every launch from an imported AMI failed the zero-copy clone check
with ServerInternal.
Rebuild the document by reading back the config.json the import just wrote. A
missing or unreadable config is skipped rather than written from the stale
manifest: no document still resolves correctly through the legacy fallback.
* test(e2e): read NODE from spx get vms instead of VCPU in the spread check
spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to
field 8. The spread placement check still read field 6, so every instance
reported its vCPU count instead of its node, the unique-node count always
collapsed to 1, and the check failed on clusters that had spread correctly.
Read field 8 and strip the ANSI colour codes wrapping each cell.
* feat(viperblockd): add a --debug flag so viperblock debug logging is reachable
viperblockd hardcoded Info, and viperblock's VB.logger() falls back to
slog.Default() because vb.log is only ever set by SetDebug. That left the
engine's debug records, including the chunk uploader's start and disabled
lines emitted during New(), unreachable in a deployed environment.
Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG
rather than the bare debug key, which several commands share. Also sets the
Config.Debug field, which existed but was set only by tests.
* fix(viperblockd): release the chunk uploader when openVolumeVB fails
viperblock.New starts a chunk uploader and a WAL syncer before the backend is
touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the
caller while leaving both running with no handle to stop them. Repeated backend
failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders
on a 30s ticker for the life of the process.
Adds invariant tests for both halves of the ownership rule: a mounted volume's
snapshot create and copy reuse the live engine and log zero volume opens, an
unmounted volume opens exactly one, and a failed open leaves no uploader behind.
The open counts are read from the same log record a production alarm watches.
* fix(ebsprovider): route requests to the node that owns the mount
Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one
spinifex-workers group, so NATS delivered each request to a random node. Only
the node holding the mount can reuse the live engine through findMountedVolume;
any other node constructed a second viperblock engine, with a live 30s chunk
uploader, against a volume nbdkit was serving on a different host. Measured on
a three-node cluster, this was roughly two snapshots in three.
Rather than resolve the owner, let the owner answer. A node subscribes
ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts
a volume and drops it on unmount, mirroring the per-volume ebs.config
subscription. Clients try that subject first and fall back to the queue group
only on ErrNoResponders, which means nothing is mounted and opening an engine
is correct. A timeout deliberately does not fall back, since the owner may
still be running the operation.
Applies to snapshot create and copy, expand and describe, the four verbs that
reuse a mounted engine. Delete keeps queue dispatch.
* fix: rebuild viperblockd mounted-volume registry after a restart
Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a
viperblockd restart while a VM keeps running leaves nbdkit serving a volume
the daemon has forgotten. Owner routing then infers from ErrNoResponders that
nothing is mounted and falls back to the queue group, which opens a second
viperblock engine, with a live chunk uploader, on another node. The same empty
registry also defeats handlePublishVolume's double-mount guard, so a republish
could start a second nbdkit.
Discover surviving nbdkit processes from /proc at startup and rebuild the
registry from them, following orphan_scan.go's precedent: match comm exactly,
parse the argv for volume=<id> and --unix/-p, and adopt a PID only when
nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or
killed; anything that fails corroboration is logged and skipped. Recovery runs
before the first subscription so no handler can race it.
Extract mountVolume's engine construction into constructMountedVB so mount and
recovery cannot drift, and release the WAL syncer viperblock.New starts when
construction fails after it.
* fix: corroborate recovered nbdkit processes from argv, not a pidfile
nbdkit is launched with -f (required for the Go plugin) and never writes the
--pidfile it is given, so no such file exists on any node; the daemon's own
/tmp is a PrivateTmp namespace besides. Corroborating against it rejected
every genuine mount, making recovery a no-op in production.
Corroborate against what the kernel actually has: the process must serve our
plugin, name this daemon's base dir, and still hold its socket. Adopt a volume
once, so two live nbdkits for it leave one registry entry and a warning rather
than two conflicting ones.
* feat: add a second EBS provider backed by qcow2 and qemu-nbd
The decoupling was justified by swappability that nothing demonstrated: the
conformance suite ran against MemoryProvider and against a NATSProvider wired
to a MemoryProvider, so both arms were the same implementation behind
different transports. A fake is not a second implementation.
Add three pieces. natsserve serves the ebs.provider.v1.* contract from any
EBSProvider, depending on nothing but the interface, and replaces the
test-only server twin so conformance now exercises production code. The
conformance suite moves to its own package and is exported so implementations
outside ebsprovider can run it, with no assertion changed. qemunbdd is the
second implementation: one qcow2 file per volume, qemu-nbd for publish, real
snapshot copies instead of chunk references, and every exec behind a runner
seam so its tests need no qemu.
Owner-subject routing is deliberately not served neutrally: it needs a
per-node mounted-volume registry the interface does not model, so it stays in
viperblockd and a neutral server's clients correctly fall back to the queue
group.
* test: run the conformance suite against the qcow2 provider
The suite passes unmodified against an implementation sharing no code with
viperblock, both in-process and through natsserve over the real wire
contract, which is the swappability claim the provider boundary was built to
support.
Two real defects it caught. ExpandVolume and GetVolume inspected the image
before consulting the publication map, so qemu-img could not open a volume
qemu-nbd held the write lock on and the refusal surfaced as an internal
error instead of ErrVolumeInUse; qemu-img info now runs with --force-share,
so describing a volume no longer depends on whether it is attached. Publish
also failed with qemu-nbd's raw exit status when the socket path exceeded the
108-byte AF_UNIX limit, and now reports that reason itself.
* feat: make the EBS provider selectable by config
Add the qemunbd selector alongside viperblockd, a spx service command that
serves the qcow2 provider over natsserve, and service registration. Config
validation accepts the new value everywhere viperblockd is accepted, while
embedded stays rejected by name.
The daemon logged a hard-coded viperblockd constant as the active provider
rather than the resolved one, in configureEBSProvider and in both of
probeEBSProvider's lines; all three now report what is actually configured.
Both daemons answer the same subjects, so exactly one may run on a cluster.
* test: add a live lifecycle check for the qcow2 provider
Drives create, publish, write, snapshot, restore and read-back entirely
through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a
cluster node, so the client under test is the same shape the control plane
uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE
is set, so it stays out of preflight.
* fix: publish NBD URIs in the form standard clients accept
PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy
pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright
with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever
consume what the EBS provider boundary hands back.
Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit
all accept. ParseNBDURI now takes both that and the TCP form, and still
accepts the legacy nbd:unix: syntax for state written by older daemons, so
a conformant third-party provider's URI is no longer rejected on the way
in either.
Verified against qemu 10.0.11: qemu-img and a real -drive option string
both open the URI form, and qemu reports both forms as nbd+unix://?socket=,
so it already treats the URI as canonical.
* test: validate published exports with libnbd's client tools
Every other assertion about the EBS provider goes through our own Go
client, so nothing has ever checked whether the export we publish is usable
by an NBD client that knows nothing about this codebase. That is the layer
neither the conformance suite nor a CSI-shaped contract can reach.
RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume:
the URI connects, the export advertises its size and block-size
constraints, data written externally reads back byte for byte, a ReadOnly
publish sets the read-only transmission flag, and UnpublishVolume tears the
export down. It skips when libnbd-bin is absent rather than passing
vacuously.
Wired to qemunbdd, where all five failed before the URI fix. The publish
assertion in the main suite now requires a dialable URI instead of merely
a non-empty string, which is what let the wrong form through.
* test: run the external NBD client suite against a live viperblockd
The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit
export is the comparison worth having: one external client, two providers it
cannot tell apart, and every difference is a candidate leak in the boundary.
RunNBDClientSuiteWithConfig takes the node name and volume prefix a live
cluster needs, and publishForNBD now unpublishes and deletes what it created
so a run against a real cluster leaves nothing behind. Subtests dial a
normalised URI so a provider emitting the legacy syntax still gets its export
tested rather than every subtest failing on the same string; whether the URI
itself is standard is now its own subtest.
Adds a reconnect check, since publication holds the volume rather than any one
connection. It retries the real operation because a cheaper liveness probe is
itself a connection and re-triggers whatever refused it.
* feat: make the conformance suite capability-driven
The suite asserted a fixed Capabilities literal, so it could only ever pass
against a provider advertising exactly that set. Every implementation had to
match the suite rather than the suite testing the contract. RunSuite now reads
GetCapabilities from the provider under test and gates optional behaviour on
what it advertises, and the fixed literal is demoted to ReferenceCapabilities,
a convenience for callers constructing a MemoryProvider.
Add ReadOnlyPublish to Capabilities. A provider that leaves it false must
refuse a ReadOnly publish rather than hand back a writable export, which is
what viperblockd already does; it now has a way to say so up front.
Add ErrUnsupportedCapability and the unsupported_capability error code. A
request for optional behaviour a provider does not advertise is well formed,
so classifying it as invalid_argument told the caller to fix a request that
was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly.
Declared capabilities now match observed behaviour: qemunbdd advertises
SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with
nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises
neither: the nbdkit Go binding exposes no extents callback, and the mount path
never passes -r.
The external NBD client suite gains a sparse-extent subtest that checks the
export's advertised contexts against the capability, so SparseExtentReporting
is falsifiable from outside rather than taken on trust, and its read-only
subtest now branches on the capability instead of assuming refusal.
Cover the new branches with a MemoryProvider advertising no optional
capabilities and one advertising OnlineExpansion, the branch no provider we
ship exercises.
* refactor: rename SourceVolumeID to SourceSnapshotVolumeID
CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd
required it whenever SourceSnapshotID was set and used it to resolve the
snapshot's chunk references, so there it named the snapshot's origin volume.
MemoryProvider ignored it entirely. qemunbdd implemented it as an independent
volume-to-volume clone, which is what the name and doc comment implied. The
conformance suite could not catch the divergence because the reference
provider had no branch for the field at all.
The contract already has a field meaning "the volume this snapshot came from":
Snapshot.SourceVolumeID, which every provider populates. So the request field
is the same fact travelling back the other way, and its name was the problem.
Rename it to SourceSnapshotVolumeID, document it as the source snapshot's
origin volume, and state that it is required with SourceSnapshotID and
meaningless without it.
All three providers now enforce one rule: a source snapshot must be
accompanied by its origin volume, or the request is invalid_argument.
MemoryProvider additionally rejects an origin that does not match the
snapshot's own SourceVolumeID, so a wrong name fails instead of resolving
blocks against a volume the snapshot never came from. qemunbdd loses its
volume-to-volume clone branch, which implemented behaviour the contract no
longer offers and no caller ever requested.
The suite pins both the requirement and a full create-snapshot-restore round
trip, so the field is exercised end to end rather than only on the error path.
* feat: make the conformance suite runnable against a live provider
The suite had only ever run against MemoryProvider and qemunbdd, both of which
are discarded with the test binary. viperblockd, the provider production
actually uses, was the one implementation the suite never judged, because the
suite assumed each subtest got fresh state: hard-coded volume IDs collide with
an earlier run's leftovers on a store that keeps them, and a hard-coded node ID
addresses a node that does not exist.
SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records
every volume, snapshot and publication the suite creates and tears them down in
dependency order, logging rather than failing so a teardown problem cannot mask
the assertion a subtest was making. Two new tests cover this by running the
whole suite twice against one shared provider instance, which passes only
because cleanup works, and once per prefix to cover concurrent runs sharing a
daemon.
TestLive_ViperblockdConformance wires this to a real cluster behind
SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite.
* fix(viperblockd): close the four contract gaps the live conformance run found
Unpublishing a volume this node has no mount for now succeeds instead of
returning not_found, matching the legacy unmount path: nothing mounted here is
already the state the request asks for.
CreateSnapshot refuses a snapshot ID that already exists against a different
source volume with already_exists, rather than overwriting it.
DeleteVolume refuses a published volume with volume_in_use instead of tearing
the mount down, on this node or on whichever node answers the volume's owner
subject. The export and the guest writing through it survive the metadata
deletion, so the old behaviour surfaced later as corruption.
CreateVolume reads the source snapshot before writing the volume, returning
not_found when it is absent and invalid_argument when it was taken from a
different volume than the request names. Previously both produced a blank
volume and a successful response.
CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument
now that it carries a sentinel.
* feat(ebsprovider): pin the source snapshot's origin volume in the contract
The suite only required source_snapshot_volume_id to be present, so a caller
naming a volume the snapshot never came from was accepted everywhere. The
snapshot records its own origin, so this is checkable: creating from a snapshot
with a mismatched source volume is now invalid_argument for every provider,
enforced by qemunbdd against its snapshot metadata and by viperblockd against
the snapshot's persisted SourceVolumeName.
* test(ebsprovider): run the contract suite against viperblockd in CI
CI installed neither qemu nor nbdkit, so every conformance test that needs a
real implementation skipped silently and MemoryProvider was the only provider
the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and
sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails
instead of skipping.
TestViperblockdConformance stands the provider handlers up over an embedded
NATS and a real predastore and runs the same suite against them, which is what
caught CopySnapshot accepting a foreign volume's snapshot on unencrypted
volumes.
* feat(viperblockd): honour read-only publish and advertise block size constraints
ReadOnlyPublish was advertised false and read_only=true was rejected,
so the capability was true only by omission. nbdkit -r now starts the
export when the request asks for it, which sets the NBD read-only
transmission flag and makes the plugin refuse writes. Access mode is
fixed when nbdkit starts, so a republish asking for the other mode is
refused with VolumeInUse rather than handed the running export.
Block size constraints were absent entirely: the nbdkit Go binding has
no block-size callback, so the plugin cannot advertise them. The
blocksize-policy filter can, without a binding callback. Minimum 1
because viperblock's WriteAt handles unaligned requests with a
read-modify-write, preferred 4096 to match its block size, maximum 32M
to match what qemu-nbd advertises.
* test(conformance): judge sparse reporting by the allocation map, not the context
The subtest asserted that base:allocation was offered. nbdkit offers
that context whether or not the plugin can compute extents, answering
'all allocated' when it cannot, so the assertion passed for a server
with no extent support at all — it could not fail. What separates the
two is the answer: a volume with nothing written to it is entirely
holes to a server that really tracks allocation.
Read the map with nbdinfo --map --json and compare holes against the
advertised capability. qemunbdd reports holes and advertises true;
viperblockd reports none and advertises false.
* test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime
The predastore fixture is one daemon per test binary, but its cert was
exported with t.Setenv, which reverts when the first test that called
Start finishes. In-process clients did not notice because crypto/x509
caches the pool behind a sync.Once, loaded while the variable was still
set. Child processes did: nbdkit spawned by a later test inherited an
environment with no SSL_CERT_FILE and could not verify the daemon, so
the whole NBD client suite failed when the package ran as a whole while
passing when run alone.
Also install libnbd-bin in CI, so the NBD client suite runs there
instead of failing on missing tools.
* ci: install nbdkit-plugin-dev so the viperblock plugin can build
* test(volume): give the snapshot fixture an owner
CreateVolume now fails closed on a snapshot whose owner it cannot match,
including an unset owner_id. The fixture predates that and wrote none,
so the provider test failed with InvalidSnapshot.NotFound before ever
reaching the provider.
* test(integration): give the volume harness a real provider
The control plane no longer builds volumes itself, so CreateVolume in
the integration harness failed with "no EBS provider configured": the
harness wired a VolumeServiceImpl with nothing behind it. It now serves
viperblockd's provider subjects against the same predastore fixture, so
the blocks are still written by real viperblock to a real daemon.
Three persisted-state assertions moved with it. TenantID, VolumeType
and AvailabilityZone are control-plane facts, held in the ebsmetadata
document rather than viperblock's own state, so the storage oracle now
asserts their absence and DescribeVolumes -- whose answer is
account-scoped -- is what checks them.
* refactor: route image import through the EBS provider and drop the legacy metadata path
The control plane could still only run against viperblock: the AMI import and
the ochre weights import both linked the storage library directly, and the
ebsmetadata store kept a read fallback onto viperblock's config.json.
Import now goes through the provider contract with no new verb: CreateVolume,
PublishVolume, a bulk copy over the published NBD export with qemu-img, then
UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS,
whose request size the contract caps far below a disk image.
The migration scaffolding goes with it. ebsmetadata documents are now the only
thing the control plane reads: no legacy prefix scan, no config.json fallback,
no backfill on daemon start. config.json belongs to the provider, which is the
single writer of it.
ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is
right for DescribeImages, where one corrupt document must not hide the fleet,
and wrong for a name-uniqueness check, which cannot rule out a name it failed
to read.
spinifex and cmd no longer import viperblock outside services/viperblockd and
testutil/vbscan.
* feat: add a volume enumeration verb so a lost document cannot strand blocks
Making ebsmetadata documents the sole index of volumes left one failure with
no recovery: a volume whose document is gone disappears from DescribeVolumes,
from the leak reaper (which enumerates documents), and from the provider,
whose contract had no verb that asks what it holds.
ListVolumes closes that. It is capability-gated on VolumeEnumeration and
paginated, following CSI's optional LIST_VOLUMES rather than inventing a
shape. Refs are deliberately not Volumes: enumeration answers what exists,
and a provider must not open every volume's engine to say so.
Each implementation reads what actually exists rather than what it remembers.
qemunbdd walks its volumes directory, not its in-memory map, which does not
survive a restart; viperblockd walks the bucket's top-level prefixes. A
volume the provider has forgotten is precisely what a caller is looking for.
The consumer is a read-only orphan report: provider volumes minus documented
ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is
excused only when its base volume is documented, so a stranded pair is still
reported. Listing documents stays strict, because reporting a volume as
orphaned on the strength of a document that failed to decode would point an
operator at live data.
* fix: stop the orphan scan accusing storage of holding things it does not
A live env19 run reported three orphans against a healthy cluster: an AMI's
backing volume, a snapshot, and the cluster key store. All three were wrong,
and a report that cries wolf on a clean cluster is worse than no report.
Enumeration treated every top-level bucket prefix as a volume. Snapshots sit
beside volumes at that level and have a config of their own, so presence of a
config cannot separate them; the snap- name is what does, which is already how
the AMI removal path reads them back. keys/ and spinifex/ are not volumes at
all.
An AMI's blocks are a genuine provider volume, so enumeration keeps reporting
them. What was missing is on the reading side: an AMI is recorded as an image
document, not a volume one, so the scan now treats a documented image as
known. Listing images is strict for the same reason listing volumes is —
unreadable and absent are different answers, and only one means orphaned.
* fix: let an orphaned image's blocks be reclaimed
The orphan report tells an operator that reclaiming a stranded volume is their
decision, and images remove --force is the tool it points them at. On env19
that tool failed on exactly those volumes: removal drops the ebsmetadata
document first, and predastore answers NoSuchKey for a document that is
already gone, which is the definition of an orphan.
Deleting the document stays first and stays fatal. That ordering is the
barrier that stops a launch landing on blocks about to be deleted, and an AMI
still advertised after its blocks are gone is the worse failure. Only an
already-absent document is now tolerated: the barrier is satisfied, and the
blocks are the sole thing left to reclaim.
S3ObjectStore.DeleteObject normalises a missing key the way GetObject already
does, so this is recognised rather than matched on error text.
The obvious unit test passed without the fix, because the in-memory store is
silently tolerant on delete and real S3 answers 204 — neither behaves like the
backend this runs against. The test now drives a store that reports the miss,
and fails without the fix.
* feat: add per-verb OTel spans to the EBS provider contract
The ten provider verbs cross a single NATS seam and nothing measured it.
The only spans on the storage path were the legacy ebs.mount/ebs.unmount
pair, so control-plane storage latency could not be attributed to a verb
or compared across provider implementations.
NATSProvider.request is the one client-side funnel every verb passes
through, including both legs of an owner-subject fallback, so a single
span there covers the whole contract and measures what the control plane
experiences regardless of which provider answers.
Span names come from a closed verb set rather than the subject. Owner
and snapshot-create subjects embed a volume ID and the mount/unmount
subjects embed a node ID; naming spans after them would give every
volume a span name of its own. The IDs move to attributes and an
unrecognised subject collapses to a single unknown verb.
Trace context rides the message headers, so the serving provider joins
the caller's trace instead of rooting one of its own. The carrier is a
copy of the one in spinifex/utils rather than an import: that package
pulls in the aws-sdk and the cluster key store, which the wire contract
must not depend on.
Responses carry no shared accessor for their error field, so both sides
re-read just that field from the encoded reply, and only when the span
is recording. Without it a failed verb looks like a successful round
trip.
viperblockd serves the contract with its own handlers rather than
natsserve, so its handlers now take the per-message context and its
publish/unpublish pair drops the subject-named consumer span it opened
for itself.
* feat: add null and hostile providers, and a per-verb bench suite
The provider boundary made it possible to put something other than
viperblock behind the control plane, and nothing yet did.
nullprovider is the floor. It is not a new type: MemoryProvider already
answers every verb from a map and natsserve already serves any
EBSProvider, so the package names that composition and states what it
measures. On loopback with no storage behind it a round trip costs
around 195us against 74ns in process, which is the seam's own price and
is not attributable to any storage engine.
conformance.RunBenchSuite prices one round trip per verb and drives
every implementation through the same calls in the same order, so two
providers' numbers are comparable. It measures control operations only:
guest I/O never crosses these verbs.
hostile wraps any provider in one that misbehaves. Faults derive from
the seed and a per-target sequence rather than a global counter, so
concurrent callers on different volumes cannot reorder each other's
faults and a failure found deep into a soak run replays exactly. Error
and error-after-work are separate faults on purpose: the second reports
failure after the work has already happened, which is how an orphan is
made, and is what makes an enumeration scan the right oracle for a soak.
Enumeration itself is never faulted, since an oracle that can be told
what to say cannot falsify anything.
The soak asserts that every volume the provider still holds is
attributable to an injected fault. A leftover with nothing behind it is
the control plane losing track of storage on its own.
Two bugs were found by falsifying the new tests rather than by running
them. The FNV-derived draw clustered badly enough that two fault bands
were never reachable in 1200 draws, so the draw now goes through a
SplitMix64 finalizer. And the first version of the lie test was
worthless: MemoryProvider returns copies, so a lie could not corrupt it
whether or not the injector copied, and the test now drives a provider
that hands out pointers into its own state.
* test: bench the provider contract against qemunbdd and a live viperblockd
RunBenchSuite was only ever pointed at the null provider, so the numbers
priced the seam and nothing else. Point it at the two real implementations.
BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a
provider that stores nothing and wrong against one that does, so qemunbdd and
viperblockd are both measured at 64 MiB.
liveProvider splits into a testing.TB flavour. Benchmarks need the same
connection and the same node and cannot be handed a *testing.T.
* fix(viperblockd): carry the trace context into every viperblock engine call
Both S3 clients only emit a span when the request already carries one, so
calling the no-context Init, LoadState, SaveState, LoadBlockState and Close
left the whole engine open untraceable. Every provider verb traced as one flat
server span, and the 342ms describe measured on env19 could not be attributed
to anything.
viperblock already exposes the Ctx variants and Backend already declares
InitCtx, so this is threading, not new instrumentation. openVolumeVB,
openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine,
copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and
mount/unmount keep the context their own span derives instead of dropping it.
completeCreateSnapshot gets WithoutCancel: the background half of
accept-then-publish has to outlive the request it was accepted by and still be
attributable to it.
The per-volume ebs.config handler opens a consumer span like the other legacy
subjects, so a config update is no longer the one path with no trace at all.
* perf: describe a volume without opening an engine on it
describeVolumeEngine fell through to openVolumeVB for any volume not mounted
on the node serving the request, which constructed a full viperblock engine
to read one integer. On env19 that decomposed as a 22 ms reachability list, a
3 ms state read and a 16 ms PutObject of the volume's own config.json — the
write coming from the SeqNum bootstrap a read-write open performs.
Use viperblock's read-only state load instead. A describe is now the 3 ms
read: no background goroutines, no reachability probe, no SeqNum window
claimed, and no write issued by a worker that does not own the volume.
Falsified: restoring the openVolumeVB call fails
TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the
list, with the fake backend answering each so neither assertion hides behind
the other.
* perf: share one HTTP client across every viperblock backend
Four sites built the same S3Config by hand, each leaving HTTPClient unset, so
every engine open and state read got its own connection pool and paid its own
TLS handshake. That was invisible while Backend.Init issued a whole-bucket
list — the list paid for the connection and the state read rode it — and it
became the dominant cost once describe was reduced to a single request.
Route all four through cfg.volumeS3Config, which builds the client once.
Falsified: dropping the HTTPClient field takes
TestDescribeVolume_ReusesOneConnection from one connection to three.
* perf: wait for nbdkit to be ready instead of waiting a fixed second
mountVolume spawned nbdkit and then ran a confirm loop until a one-second
deadline passed. The loop only broke early when the process exited, so the
fast-failure path worked and the success path always cost the full second.
That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so
the timings never contradicted the assumption that it was process spawn.
Wait for the endpoint to accept a connection instead. That is also the
question the caller needs answered, since it is about to hand the URI to
something that will connect to it — where 'still alive after a second' only
correlates with it. A nbdkit that exits during startup still fails the mount
immediately, and one that never comes up now fails on a deadline rather than
being assumed healthy.
Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness
timing assertion and the fast-failure one.
* test: measure the data path, not just the control verbs
The conformance benchmarks price create, publish and describe. Guest
I/O never crosses those verbs, so none of those numbers say anything
about what a guest experiences. This drives the published export with
qemu-img bench, which sits on the same block layer virtio-blk does.
It asserts only that the I/O completes and logs a table: what a given
store is capable of is not a contract, and pinning a number here would
only produce a flaky test.
Read profiles stay inside the region the write profiles covered, or
they would be timing how fast the export answers with zeroes, and one
write profile flushes, because an unflushed write measures how fast the
WAL accepts data rather than how fast the store takes it.
Profiles are spaced apart because the plugin tears an engine down per
disconnect and removes the volume's local directory as it goes, so a
connection opened while the previous one is closing fails its
handshake.
* test: run data-path profiles back to back
The spacing was a workaround for a volume lock that did not hold across
close. With that fixed, consecutive profiles are worth more than the
workaround was: each is its own NBD connection, so a run that completes
has also shown that an arriving connection waits out the previous one's
close rather than racing it.
* fix: refuse a delete that cannot establish whether a volume is published
volumeMountedElsewhere returned false on every failure path, including a
request timeout, so a busy or partitioned owner was indistinguishable
from no owner at all and the delete proceeded. Deleting a published
volume leaves the nbdkit export and the guest writing through it alive
over storage that is going away.
Report the three outcomes separately. NATS distinguishes no-responders
from no-answer, so only the former means unmounted; an unanswered probe
is refused and the caller may retry once the owner is reachable.
* feat: exclude a second viperblock engine with a JetStream volume lease
Nothing in the stack stopped two viperblock engines opening one volume from
different nodes. For an encrypted volume that is not just a consistency risk:
the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte,
and the SeqNum window claim is a read-then-write with no CAS, so two engines
opening concurrently issue overlapping nonces under the same key.
Every engine open now takes a lease first, keyed by volume in a JetStream KV
bucket. A lease is a KV create, which JetStream serialises, so a second
claimant loses instead of both winning. The revision that won is the
generation: monotonic across the bucket, so a later opener always carries a
higher one than the writer it displaced. Entries carry a TTL and a renewal
loop, so a node that dies mid-mount does not wedge its volumes forever.
Losing a lease is logged, not enforced. Until the object store can refuse a
stale writer's PUT, failing live guest I/O on lease loss would add a failure
mode without stopping the zombie it is aimed at.
A daemon with no lease store refuses to open rather than opening blind, so
nbdkit recovery binds the store before it adopts surviving exports. Repeat
acquisitions on one node share a lease: the unmount seal opens a detached
engine while the mount being torn down still holds it, and cross-node
exclusion is what the lease is for. A restart reclaims this node's own stale
entries, since the exports that outlived the daemon are still running.
* refactor: consume masterkey from bluebottle in the test this branch added
* feat(ebsprovider): classify refusals a caller may retry
Every ErrorCode said what went wrong and none said whether the same
request could succeed later, so a caller could not tell a permanent
refusal from a transient one. Transient conditions shipped as internal,
which tells a caller to give up when it should retry.
Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(),
which callers branch on instead of on the code itself. DeleteVolume's
"could not establish whether the volume is published" refusal moves off
internal onto it: the owner that did not answer may answer later.
TestErrorCodeRetryable names every code, so adding one without deciding
its retry meaning fails rather than silently defaulting to permanent.
TestClassifyError pins the sentinel-to-wire-code mapping, which had no
coverage at all; a sentinel missing from classifyError falls through to
internal and loses both the reason and the retry signal.
* fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for
Four providers advertised the capability and nothing consumed it: there
is no clone verb on the interface. CreateVolume's SourceSnapshotID
restores from a snapshot and CopySnapshot duplicates snapshot metadata;
neither is a volume clone, and the field documenting SourceSnapshotID
says so explicitly. CSI expresses clone as a volume_content_source with
a volume source, which this contract does not implement.
Advertising it told a caller a feature was available and then gave them
no way to use it. Removed rather than backed by a method: adding a clone
verb is a feature, not a contract fix, and the capability can come back
alongside it. The Capabilities doc comment now states the rule, and
qemunbdd's stale comment about sparse extents is corrected while there.
* feat(ebsprovider): add ListSnapshots so snapshots are reconcilable
The contract could create, copy and delete snapshots but never say which
ones it held. That left the control-plane metadata document as the only
index of them, so losing one stranded the data it pointed at with no way
to find it again. This is the same exposure ListVolumes closed for
volumes, and it stayed invisible because ListSnapshots is optional in
CSI too.
ListSnapshots pages exactly as ListVolumes does, gated on a new
SnapshotEnumeration capability. SnapshotRef is deliberately not a
Snapshot: enumeration answers what exists, and reading a snapshot's size
or state can cost far more than listing it.
viperblockd walks the bucket's snap- prefixes, sharing the top-level
walk with ListVolumes. It leaves SourceVolumeID empty because the link
lives inside the snapshot's own encrypted state and reading it would
mean a fetch per snapshot listed. qemunbdd walks its snapshots
directory rather than its in-memory map, so a snapshot that outlived the
process still appears; its SourceVolumeID is empty in that case for the
same reason CreateSnapshot refuses to reuse the ID.
Five conformance subtests cover it, and they run green against
MemoryProvider, qemunbdd and viperblockd over a real predastore.
* feat(ebsprovider): state the exclusion guarantee in the contract
A caller could not tell which providers prevent two writers from holding
one volume, at what scope, or for how long. The guarantee existed in
viperblockd and was invisible from the outside, which is why the
conformance suite had no concurrency dimension at all.
Capabilities.Exclusion states it, and deliberately separates three
questions the contract used to run together: how far exclusion reaches
(none, node, cluster), whether a dead owner's claim ever expires, and
whether a writer that lost its claim is actually stopped. That last one
matters most and is the least flattering: exclusion refuses the second
opener, fencing stops the first. viperblockd advertises cluster scope
with a 45s TTL and FencesLostClaim false, because a partitioned node
whose lease lapsed keeps writing until the object store can refuse it.
Four new conformance subtests. Two required a second viperblockd node in
the harness: cluster exclusion cannot be exercised by one node, because
the second publish goes to a subject nobody serves and times out rather
than being refused. Both nodes share the NATS the leases live in and the
bucket the volumes live in, so a refusal comes from the lease.
Standing that second node up immediately found a real gap, in a test
that had been silently skipping for want of one. A publish refused by
another node's lease came back as internal rather than volume_in_use, so
a caller was told something had broken when the answer was that someone
else held the volume. Mapped in handlePublishVolume.
* build: pin viperblock to the release carrying the provider's dependencies
The branch has been building against a local replace pointing at a
viperblock worktree, so its committed pin never actually built: it named
a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch,
NewStateReader or s3.NewHTTPClient, all of which this branch calls.
Points at the merged viperblock dev instead and drops the replace.
GOWORK=off now builds clean, which is the deploy path ansible uses.
* fix: let viperblockd's delete sweep tolerate an object another sweep removed
deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and
give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it
returned on any DeleteObject error. Two overlapping deletes for one volume left
the loser reporting an internal failure once the winner had removed a chunk
between the list and the delete.
The control-plane copy that carried this fix is gone with the embedded engine
path, so the provider path reintroduced the bug it had just fixed.
* fix: stop CreateVolume leaving local state on whichever node served it
A state read prefers this node's local copy over the backend, and provider RPCs
are not owner-routed, so the node that happened to serve a create kept a copy
that answered describe as if the volume still existed after another node
deleted it. The two-node conformance harness caught it as DeleteVolume/success
failing about one run in three.
The seeded path already removed its local files after closing. The unseeded
path saved state and left them, which was the only difference between the two.
* fix: guard the mount path itself against a second export of one volume
handlePublishVolume refused a volume this node already exported, but
mountVolume did not, and the legacy ebs.<node>.mount subject is the route
production takes for instance launch and hot-attach. A retried attach racing a
fresh one started two real nbdkit against one volume, with no refusal and no
warning: the volume lease only refuses a different node, and a same-node
re-acquire finds the volume already held and increments its refcount.
That also made the advertised exclusion inaccurate. viperblockd claims
cluster-scoped exclusion on the grounds that a second opener is refused, while
the conformance subtests reach it only through PublishVolume, so they passed
with the legacy route unguarded. Moving the check into mountVolume gives both
entry points the same behaviour and makes the claim true for the path
production uses.
* fix: tolerate a missing legacy AMI config on deregister
DeregisterImage deletes the AMI document, then the pre-provider
{imageID}/config.json. Predastore answers a delete of an absent key with
NoSuchKey, and that was treated as fatal, so every AMI created since the
metadata store landed failed to deregister after its document was already
gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had
just been told it could not delete.
The regression test uses a store that answers a missing-key delete the way
Predastore does; MemoryObjectStore silently succeeds, which is why the unit
suite stayed green.
* fix: reap ENIs left behind by an abandoned instance launch
A launch creates the instance's ENI and then attaches it. When the daemon
dies in between, the record survives with no instance and nothing ever looks
at it again. The security group dependency check counts an ENI that lists the
group, so the group cannot be deleted for as long as the record exists.
The in-process launch path already rolls back on error; a process that dies
cannot. This adds a cluster-wide sweep over records that are launch-created,
never attached, and older than the launch RPC timeout by a wide margin. The
description prefix is now a shared constant because it is the only thing
separating an abandoned launch from a standalone ENI a caller asked for.
1831 lines
72 KiB
Bash
Executable File
1831 lines
72 KiB
Bash
Executable File
#!/bin/bash
|
|
set -euo pipefail
|
|
|
|
# Multi-Node E2E Test Suite — bare-metal variant.
|
|
# Adapted from run-multinode-e2e.sh for N≥2 bare-metal nodes bootstrapped by
|
|
# bm-bootstrap.sh. SSH key and user are parameterised via env vars.
|
|
#
|
|
# Usage: run-bm-multinode-e2e.sh <node1_ip> <node2_ip> [node3_ip ...]
|
|
# Env: PEER_SSH_KEY path to private key for peer_ssh (default: ~/bm-ci key)
|
|
# PEER_SSH_USER SSH username for peer nodes (default: spinifex)
|
|
|
|
# Ensure Go is on PATH (SSH non-interactive shells don't source .bashrc)
|
|
export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"
|
|
|
|
# ==========================================================================
|
|
# Argument parsing
|
|
# ==========================================================================
|
|
if [ $# -lt 2 ]; then
|
|
echo "Usage: $0 <node1_ip> <node2_ip> [node3_ip ...]"
|
|
echo " At least 2 node WAN IPs are required."
|
|
exit 1
|
|
fi
|
|
|
|
NODE_IPS=("$@")
|
|
NODE_COUNT=${#NODE_IPS[@]}
|
|
LOCAL_IP="${NODE_IPS[0]}"
|
|
NODE2_IP="${NODE_IPS[1]}"
|
|
NODE3_IP="${NODE_IPS[2]:-}"
|
|
|
|
# ==========================================================================
|
|
# Constants
|
|
# ==========================================================================
|
|
NATS_MONITOR_PORT=8222
|
|
PREDASTORE_PORT=8443
|
|
AWSGW_PORT=9999
|
|
SSH_KEY_PATH="${PEER_SSH_KEY:-$HOME/.ssh/bm-ci}"
|
|
PEER_USER="${PEER_SSH_USER:-spinifex}"
|
|
SPINIFEX_BIN="spx"
|
|
|
|
# Use Spinifex profile for AWS CLI
|
|
export AWS_PROFILE=spinifex
|
|
# Trust Spinifex CA for AWS CLI v2 (bundles its own Python/certifi, ignores system CA store)
|
|
export AWS_CA_BUNDLE="/etc/spinifex/ca.pem"
|
|
|
|
# Track test results
|
|
TESTS_PASSED=0
|
|
TESTS_FAILED=0
|
|
TESTS_SKIPPED=0
|
|
FAILED_TESTS=()
|
|
SKIPPED_TESTS=()
|
|
|
|
# Cluster size target: phases that depend on multi-instance density (3+
|
|
# t3.nano instances spread across the cluster) require NODE_COUNT >= DENSITY_NODE_TARGET.
|
|
# On smaller clusters the smallest node (e.g. a 4-vCPU NUC) can hold only one
|
|
# t3.nano after the daemon's host reserve, so any phase that puts >1 instance
|
|
# on it — directly via spread or indirectly via StartStoppedInstance returning
|
|
# instances to their original node — legitimately exhausts capacity.
|
|
DENSITY_NODE_TARGET=3
|
|
|
|
# ==========================================================================
|
|
# Helper functions
|
|
# ==========================================================================
|
|
|
|
# SSH to a peer node
|
|
peer_ssh() {
|
|
local ip="$1"; shift
|
|
ssh -i "$SSH_KEY_PATH" \
|
|
-o StrictHostKeyChecking=no \
|
|
-o UserKnownHostsFile=/dev/null \
|
|
-o ConnectTimeout=10 \
|
|
-o LogLevel=ERROR \
|
|
"${PEER_USER}@${ip}" "$@"
|
|
}
|
|
|
|
# Run AWS CLI against a specific node's gateway
|
|
aws_via() {
|
|
local node_ip="$1"; shift
|
|
aws --endpoint-url "https://${node_ip}:${AWSGW_PORT}" "$@"
|
|
}
|
|
|
|
# Retry wrapper for aws_via — retries on transient failures (e.g. NATS cluster reformation)
|
|
# Usage: aws_via_retry <max_attempts> <node_ip> <aws args...>
|
|
aws_via_retry() {
|
|
local max_attempts="$1"; shift
|
|
local node_ip="$1"; shift
|
|
local attempt=0
|
|
local result
|
|
|
|
while [ $attempt -lt $max_attempts ]; do
|
|
result=$(aws_via "$node_ip" "$@" 2>/dev/null) && {
|
|
if [ -n "$result" ] && [ "$result" != "None" ]; then
|
|
echo "$result"
|
|
return 0
|
|
fi
|
|
}
|
|
attempt=$((attempt + 1))
|
|
echo " Retry $attempt/$max_attempts..." >&2
|
|
sleep 1
|
|
done
|
|
return 1
|
|
}
|
|
|
|
# Default AWS CLI shorthand (via local node)
|
|
AWS_EC2="aws --endpoint-url https://${LOCAL_IP}:${AWSGW_PORT} ec2"
|
|
|
|
# Dump SSH-related diagnostics when a guest SSH attempt fails.
|
|
# Args: <instance_id> <host_ip> <ssh_host> [ssh_port]
|
|
# ssh_host is the address we tried (EIP or node IP), ssh_port is hostfwd port if applicable.
|
|
dump_guest_ssh_diagnostics() {
|
|
local instance_id="$1" host_ip="$2" ssh_host="$3" ssh_port="${4:-}"
|
|
echo ""
|
|
echo " === Diagnostics for ${instance_id} (host=${host_ip}, target=${ssh_host}:${ssh_port:-22}) ==="
|
|
|
|
echo " --- describe-instances ---"
|
|
$AWS_EC2 describe-instances --instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].[InstanceId,State.Name,PrivateIpAddress,PublicIpAddress,SecurityGroups[].GroupId,NetworkInterfaces[0].MacAddress,VpcId,SubnetId]' \
|
|
--output json 2>&1 || true
|
|
|
|
local sg_ids priv_ip mac vpc_id
|
|
sg_ids=$($AWS_EC2 describe-instances --instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].SecurityGroups[].GroupId' --output text 2>/dev/null || true)
|
|
priv_ip=$($AWS_EC2 describe-instances --instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].PrivateIpAddress' --output text 2>/dev/null || true)
|
|
mac=$($AWS_EC2 describe-instances --instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].NetworkInterfaces[0].MacAddress' --output text 2>/dev/null || true)
|
|
vpc_id=$($AWS_EC2 describe-instances --instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].VpcId' --output text 2>/dev/null || true)
|
|
|
|
if [ -n "$sg_ids" ] && [ "$sg_ids" != "None" ]; then
|
|
echo " --- describe-security-groups (${sg_ids}) ---"
|
|
# shellcheck disable=SC2086
|
|
$AWS_EC2 describe-security-groups --group-ids $sg_ids \
|
|
--query 'SecurityGroups[].[GroupId,GroupName,IpPermissions]' --output json 2>&1 || true
|
|
fi
|
|
|
|
echo " --- local ARP for ${ssh_host} ---"
|
|
ip neigh show "$ssh_host" 2>&1 || true
|
|
|
|
echo " --- local ping ${ssh_host} (2 pkts) ---"
|
|
ping -c 2 -W 2 "$ssh_host" 2>&1 || true
|
|
|
|
if [ -n "$host_ip" ] && [ "$host_ip" != "unknown" ]; then
|
|
echo " --- on hosting node ${host_ip}: QEMU process ---"
|
|
peer_ssh "$host_ip" "pgrep -af 'qemu.*${instance_id}' || echo '(no qemu process)'" 2>&1 || true
|
|
|
|
if [ -n "$priv_ip" ] && [ "$priv_ip" != "None" ]; then
|
|
echo " --- on hosting node ${host_ip}: ping VM private IP ${priv_ip} ---"
|
|
peer_ssh "$host_ip" "ping -c 2 -W 2 ${priv_ip}" 2>&1 || true
|
|
fi
|
|
|
|
echo " --- on hosting node ${host_ip}: OVN chassis + port bindings ---"
|
|
peer_ssh "$host_ip" "sudo ovs-vsctl get open_vswitch . external_ids:system-id 2>/dev/null || true" 2>&1 || true
|
|
fi
|
|
|
|
if [ -n "$vpc_id" ] && [ "$vpc_id" != "None" ]; then
|
|
echo " --- OVN NAT rules for vpc-${vpc_id} (from primary) ---"
|
|
peer_ssh "$LOCAL_IP" "sudo ovn-nbctl lr-nat-list vpc-${vpc_id} 2>&1 || true" 2>&1 || true
|
|
fi
|
|
|
|
if [ -n "$mac" ] && [ "$mac" != "None" ]; then
|
|
# ovn-sbctl find parses 'mac~=...' as a column name. mac is a set
|
|
# column on Port_Binding so list+grep is the simplest correct form.
|
|
echo " --- OVN port binding for MAC ${mac} (from primary) ---"
|
|
peer_ssh "$LOCAL_IP" "sudo ovn-sbctl --bare --columns=logical_port,chassis,mac list Port_Binding 2>&1 | grep -F '${mac}' || echo '(no Port_Binding matched MAC ${mac})'" 2>&1 || true
|
|
fi
|
|
|
|
# ----- Cross-chassis dataplane diagnostics -----
|
|
echo " --- OVN SB chassis registrations (from primary) ---"
|
|
peer_ssh "$LOCAL_IP" "sudo ovn-sbctl show 2>&1 || true" 2>&1 || true
|
|
|
|
echo " --- OVN SB port_binding chassis claims (from primary) ---"
|
|
peer_ssh "$LOCAL_IP" "sudo ovn-sbctl --bare --columns=logical_port,chassis,up list Port_Binding 2>&1 | head -80 || true" 2>&1 || true
|
|
|
|
echo " --- OVN NB gateway_chassis (from primary) ---"
|
|
peer_ssh "$LOCAL_IP" "sudo ovn-nbctl --bare --columns=name,chassis_name,priority list Gateway_Chassis 2>&1 || true" 2>&1 || true
|
|
|
|
if [ -n "$host_ip" ] && [ "$host_ip" != "unknown" ]; then
|
|
echo " --- on hosting node ${host_ip}: ovs-vsctl show (taps + geneve) ---"
|
|
peer_ssh "$host_ip" "sudo ovs-vsctl show 2>&1 | grep -E 'Bridge|Port|Interface|tap-|geneve|external_ids|iface-id|attached-mac|remote_ip' | head -80 || true" 2>&1 || true
|
|
|
|
echo " --- on hosting node ${host_ip}: ovn-controller status + system-id ---"
|
|
peer_ssh "$host_ip" "sudo ovs-vsctl get open_vswitch . external_ids:system-id 2>&1; \
|
|
sudo ovs-vsctl get open_vswitch . external_ids:ovn-encap-ip 2>&1; \
|
|
sudo systemctl is-active ovn-controller 2>&1 || true" 2>&1 || true
|
|
|
|
echo " --- on hosting node ${host_ip}: Geneve tunnel ports (br-int) ---"
|
|
peer_ssh "$host_ip" "sudo ovs-vsctl --columns=name,type,options find Interface type=geneve 2>&1 || true" 2>&1 || true
|
|
|
|
echo " --- on hosting node ${host_ip}: br-int output flows (tunnel) ---"
|
|
peer_ssh "$host_ip" "sudo ovs-ofctl dump-flows br-int 2>/dev/null | grep -E 'tun_dst|output:.*tun|table=33|table=37' | head -40 || true" 2>&1 || true
|
|
fi
|
|
|
|
echo " --- cloud-init / sshd from hosting node (via QEMU console log if available) ---"
|
|
peer_ssh "$host_ip" "for f in /run/spinifex/console-${instance_id}.log /tmp/spinifex/vms/${instance_id}/console.log; do \
|
|
[ -f \"\$f\" ] && echo \"--- \$f (tail 50) ---\" && sudo tail -50 \"\$f\"; \
|
|
done" 2>&1 || true
|
|
|
|
echo " === end diagnostics for ${instance_id} ==="
|
|
echo ""
|
|
}
|
|
|
|
# Dump logs from ALL nodes on failure
|
|
dump_all_node_logs() {
|
|
echo ""
|
|
echo "=========================================="
|
|
echo "DUMPING LOGS FROM ALL NODES"
|
|
echo "=========================================="
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
local ip="${NODE_IPS[$i]}"
|
|
echo ""
|
|
echo "=== Node $((i+1)) ($ip) ==="
|
|
for svc in spinifex-nats spinifex-predastore spinifex-viperblock \
|
|
spinifex-daemon spinifex-awsgw spinifex-vpcd; do
|
|
echo "--- ${svc} (last 50 lines) ---"
|
|
if [ "$ip" = "$LOCAL_IP" ]; then
|
|
sudo journalctl -u "$svc" --no-pager -n 50 2>/dev/null || echo "(not found)"
|
|
else
|
|
peer_ssh "$ip" "sudo journalctl -u $svc --no-pager -n 50" 2>/dev/null || echo "(node unreachable)"
|
|
fi
|
|
done
|
|
done
|
|
echo ""
|
|
echo "=========================================="
|
|
echo "END OF LOG DUMP"
|
|
echo "=========================================="
|
|
}
|
|
|
|
# Wait for a specific instance state
|
|
# Usage: wait_for_instance_state <instance_id> <target_state> [max_attempts] [gateway_ip]
|
|
wait_for_instance_state() {
|
|
local instance_id="$1"
|
|
local target_state="$2"
|
|
local max_attempts="${3:-60}"
|
|
local gw_ip="${4:-$LOCAL_IP}"
|
|
local attempt=0
|
|
|
|
echo " Waiting for $instance_id to reach state: $target_state..."
|
|
|
|
while [ $attempt -lt $max_attempts ]; do
|
|
local state
|
|
state=$(aws_via "$gw_ip" ec2 describe-instances \
|
|
--instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].State.Name' \
|
|
--output text 2>/dev/null) || {
|
|
sleep 1
|
|
attempt=$((attempt + 1))
|
|
continue
|
|
}
|
|
|
|
if [ "$state" == "$target_state" ]; then
|
|
echo " Instance reached state: $target_state"
|
|
return 0
|
|
fi
|
|
|
|
if [ "$state" == "terminated" ] && [ "$target_state" != "terminated" ]; then
|
|
echo " ERROR: Instance terminated unexpectedly"
|
|
return 1
|
|
fi
|
|
|
|
sleep 1
|
|
attempt=$((attempt + 1))
|
|
done
|
|
|
|
echo " ERROR: Instance did not reach $target_state within $max_attempts attempts"
|
|
return 1
|
|
}
|
|
|
|
# Find which node runs a QEMU instance (by checking ps on each node)
|
|
# Usage: find_instance_node <instance_id>
|
|
# Returns: the WAN IP of the hosting node
|
|
find_instance_node() {
|
|
local instance_id="$1"
|
|
|
|
for ip in "${NODE_IPS[@]}"; do
|
|
local found
|
|
if [ "$ip" = "$LOCAL_IP" ]; then
|
|
found=$(ps auxw | grep "$instance_id" | grep qemu-system | grep -v grep || true)
|
|
else
|
|
found=$(peer_ssh "$ip" "ps auxw | grep '$instance_id' | grep qemu-system | grep -v grep" 2>/dev/null || true)
|
|
fi
|
|
if [ -n "$found" ]; then
|
|
echo "$ip"
|
|
return 0
|
|
fi
|
|
done
|
|
|
|
return 1
|
|
}
|
|
|
|
# Extract the SSH hostfwd port for an instance from the QEMU process on a remote node
|
|
# Usage: get_remote_ssh_port <node_ip> <instance_id> [max_attempts]
|
|
get_remote_ssh_port() {
|
|
local node_ip="$1"
|
|
local instance_id="$2"
|
|
local max_attempts="${3:-30}"
|
|
local attempt=0
|
|
|
|
while [ $attempt -lt $max_attempts ]; do
|
|
local qemu_cmd
|
|
if [ "$node_ip" = "$LOCAL_IP" ]; then
|
|
qemu_cmd=$(ps auxw | grep "$instance_id" | grep qemu-system | grep -v grep || true)
|
|
else
|
|
qemu_cmd=$(peer_ssh "$node_ip" "ps auxw | grep '$instance_id' | grep qemu-system | grep -v grep" 2>/dev/null || true)
|
|
fi
|
|
|
|
if [ -n "$qemu_cmd" ]; then
|
|
local ssh_port
|
|
ssh_port=$(echo "$qemu_cmd" | sed -n 's/.*hostfwd=tcp:[^:]*:\([0-9]*\)-:22.*/\1/p')
|
|
if [ -n "$ssh_port" ]; then
|
|
echo "$ssh_port"
|
|
return 0
|
|
fi
|
|
fi
|
|
|
|
attempt=$((attempt + 1))
|
|
[ $attempt -lt $max_attempts ] && sleep 1
|
|
done
|
|
|
|
return 1
|
|
}
|
|
|
|
# Expect an AWS CLI command to fail with a specific error code
|
|
# Usage: expect_error "ErrorCode" aws ec2 some-command --args...
|
|
expect_error() {
|
|
local expected_error="$1"
|
|
shift
|
|
|
|
set +e
|
|
local output
|
|
output=$("$@" 2>&1)
|
|
local exit_code=$?
|
|
set -e
|
|
|
|
if [ $exit_code -eq 0 ]; then
|
|
echo " FAIL: Expected error '$expected_error' but command succeeded"
|
|
echo " Output: $output"
|
|
return 1
|
|
fi
|
|
|
|
if echo "$output" | grep -q "$expected_error"; then
|
|
echo " Got expected error: $expected_error"
|
|
return 0
|
|
else
|
|
echo " FAIL: Expected error '$expected_error' but got different error"
|
|
echo " Output: $output"
|
|
return 1
|
|
fi
|
|
}
|
|
|
|
# Terminate instances and wait for terminated state
|
|
terminate_and_wait() {
|
|
local ids=("$@")
|
|
|
|
for instance_id in "${ids[@]}"; do
|
|
local state
|
|
state=$($AWS_EC2 describe-instances --instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].State.Name' --output text 2>/dev/null || echo "unknown")
|
|
if [ "$state" != "terminated" ] && [ "$state" != "unknown" ]; then
|
|
echo " Terminating $instance_id (state: $state)..."
|
|
$AWS_EC2 terminate-instances --instance-ids "$instance_id" > /dev/null 2>&1 || true
|
|
fi
|
|
done
|
|
|
|
local failed=0
|
|
for instance_id in "${ids[@]}"; do
|
|
if ! wait_for_instance_state "$instance_id" "terminated" 60; then
|
|
echo " WARNING: Failed to confirm termination of $instance_id"
|
|
failed=1
|
|
fi
|
|
done
|
|
|
|
return $failed
|
|
}
|
|
|
|
# Record test result
|
|
pass_test() {
|
|
local name="$1"
|
|
echo " $name PASSED"
|
|
TESTS_PASSED=$((TESTS_PASSED + 1))
|
|
}
|
|
|
|
fail_test() {
|
|
local name="$1"
|
|
echo " $name FAILED"
|
|
TESTS_FAILED=$((TESTS_FAILED + 1))
|
|
FAILED_TESTS+=("$name")
|
|
}
|
|
|
|
skip_test() {
|
|
local name="$1"
|
|
local reason="${2:-density skip}"
|
|
echo " $name SKIPPED ($reason)"
|
|
TESTS_SKIPPED=$((TESTS_SKIPPED + 1))
|
|
SKIPPED_TESTS+=("$name ($reason)")
|
|
}
|
|
|
|
# is_density_skip returns 0 (true) when $1 contains InsufficientInstanceCapacity
|
|
# AND the cluster is smaller than the design target (3 nodes). On larger clusters
|
|
# capacity exhaustion is a real failure, not a hardware-sizing artefact.
|
|
is_density_skip() {
|
|
[ "$NODE_COUNT" -lt "$DENSITY_NODE_TARGET" ] || return 1
|
|
echo "$1" | grep -q "InsufficientInstanceCapacity"
|
|
}
|
|
|
|
# ==========================================================================
|
|
# EXIT trap — dump logs on failure
|
|
# ==========================================================================
|
|
trap 'EXIT_CODE=$?; if [ $EXIT_CODE -ne 0 ]; then dump_all_node_logs; fi; exit $EXIT_CODE' EXIT
|
|
|
|
# Track instance IDs for cleanup
|
|
INSTANCE_IDS=()
|
|
|
|
echo "========================================"
|
|
echo "Real Multi-Node E2E Test Suite"
|
|
echo "========================================"
|
|
echo "Nodes: ${NODE_IPS[*]}"
|
|
echo "Local: $LOCAL_IP"
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 1: Pre-flight Validation
|
|
# ==========================================================================
|
|
echo "Phase 1: Pre-flight Validation"
|
|
echo "========================================"
|
|
|
|
# Check KVM support
|
|
echo "Checking KVM support..."
|
|
if [ -e /dev/kvm ] && [ -w /dev/kvm ]; then
|
|
echo " /dev/kvm exists and is writable"
|
|
else
|
|
echo " ERROR: /dev/kvm missing or not writable"
|
|
exit 1
|
|
fi
|
|
|
|
# Verify SSH to peer nodes
|
|
echo "Verifying SSH connectivity to peer nodes..."
|
|
for i in $(seq 1 $((NODE_COUNT - 1))); do
|
|
local_idx=$((i))
|
|
ip="${NODE_IPS[$local_idx]}"
|
|
echo -n " SSH to node$((local_idx + 1)) ($ip)..."
|
|
if peer_ssh "$ip" "hostname" > /dev/null 2>&1; then
|
|
echo " OK"
|
|
else
|
|
echo " FAILED"
|
|
echo " ERROR: Cannot SSH to $ip"
|
|
exit 1
|
|
fi
|
|
done
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 2: Cluster Health
|
|
# ==========================================================================
|
|
echo "Phase 2: Cluster Health"
|
|
echo "========================================"
|
|
|
|
# NATS cluster: verify 2 unique peers from node1
|
|
echo "Checking NATS cluster..."
|
|
NATS_INFO=$(curl -s "http://127.0.0.1:${NATS_MONITOR_PORT}/routez" 2>/dev/null) || {
|
|
echo " ERROR: Cannot reach NATS monitoring endpoint"
|
|
exit 1
|
|
}
|
|
UNIQUE_PEERS=$(echo "$NATS_INFO" | jq -r '[.routes[].remote_name] | unique | length')
|
|
EXPECTED_PEERS=$((NODE_COUNT - 1))
|
|
echo " NATS unique peers: $UNIQUE_PEERS (expected: $EXPECTED_PEERS)"
|
|
if [ "$UNIQUE_PEERS" -ge "$EXPECTED_PEERS" ]; then
|
|
pass_test "NATS quorum"
|
|
else
|
|
echo " ERROR: NATS cluster not fully formed"
|
|
exit 1
|
|
fi
|
|
|
|
# Predastore: check each node
|
|
echo "Checking Predastore on all nodes..."
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
ip="${NODE_IPS[$i]}"
|
|
if curl -k -s "https://${ip}:${PREDASTORE_PORT}" > /dev/null 2>&1; then
|
|
echo " Node$((i+1)) ($ip): Predastore reachable"
|
|
else
|
|
echo " ERROR: Predastore not reachable on node$((i+1)) ($ip)"
|
|
exit 1
|
|
fi
|
|
done
|
|
pass_test "Predastore cluster"
|
|
|
|
# Gateway: check each node
|
|
echo "Checking gateway on all nodes..."
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
ip="${NODE_IPS[$i]}"
|
|
if curl -k -s "https://${ip}:${AWSGW_PORT}" > /dev/null 2>&1; then
|
|
echo " Node$((i+1)) ($ip): Gateway reachable"
|
|
else
|
|
echo " ERROR: Gateway not reachable on node$((i+1)) ($ip)"
|
|
exit 1
|
|
fi
|
|
done
|
|
pass_test "All gateways"
|
|
|
|
# Daemon readiness: describe-instance-types must return results
|
|
echo "Checking daemon readiness..."
|
|
ATTEMPT=0
|
|
while [ $ATTEMPT -lt 30 ]; do
|
|
TYPES=$($AWS_EC2 describe-instance-types \
|
|
--query 'InstanceTypes[*].InstanceType' --output text 2>/dev/null || true)
|
|
if [ -n "$TYPES" ] && [ "$TYPES" != "None" ]; then
|
|
echo " Daemon ready (instance types: $TYPES)"
|
|
break
|
|
fi
|
|
echo " Waiting for daemon... ($((ATTEMPT + 1))/30)"
|
|
sleep 1
|
|
ATTEMPT=$((ATTEMPT + 1))
|
|
done
|
|
if [ -z "$TYPES" ] || [ "$TYPES" == "None" ]; then
|
|
echo " ERROR: Daemon not ready"
|
|
exit 1
|
|
fi
|
|
pass_test "Daemon readiness"
|
|
|
|
# Spinifex CLI: get nodes
|
|
echo "Checking spx get nodes..."
|
|
GET_NODES_OUTPUT=$($SPINIFEX_BIN get nodes --timeout 5s 2>/dev/null)
|
|
echo "$GET_NODES_OUTPUT"
|
|
READY_COUNT=$(echo "$GET_NODES_OUTPUT" | grep -c "Ready" || true)
|
|
if [ "$READY_COUNT" -ge "$NODE_COUNT" ]; then
|
|
pass_test "spx get nodes ($READY_COUNT Ready)"
|
|
else
|
|
echo " WARNING: spx get nodes shows $READY_COUNT Ready nodes (expected $NODE_COUNT)"
|
|
fail_test "spx get nodes"
|
|
fi
|
|
|
|
# Spinifex CLI: get vms (should be empty)
|
|
echo "Checking spx get vms (empty)..."
|
|
GET_VMS_OUTPUT=$($SPINIFEX_BIN get vms --timeout 5s 2>/dev/null)
|
|
echo "$GET_VMS_OUTPUT"
|
|
pass_test "spx get vms (empty)"
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 3: Instance Lifecycle + Distribution
|
|
# ==========================================================================
|
|
echo "Phase 3: Instance Lifecycle + Distribution"
|
|
echo "========================================"
|
|
|
|
# Discover instance type
|
|
INSTANCE_TYPE=$(echo $TYPES | tr ' ' '\n' | grep -m1 'nano')
|
|
if [ -z "$INSTANCE_TYPE" ]; then
|
|
echo "ERROR: No nano instance type found"
|
|
exit 1
|
|
fi
|
|
echo "Using instance type: $INSTANCE_TYPE"
|
|
|
|
# Discover Ubuntu AMI (filter to avoid picking up the LB Alpine image)
|
|
AMI_ID=$($AWS_EC2 describe-images --filters "Name=name,Values=ami-ubuntu-*" --query 'Images[0].ImageId' --output text)
|
|
if [ -z "$AMI_ID" ] || [ "$AMI_ID" == "None" ]; then
|
|
echo "ERROR: No AMI found (bootstrap should have imported one)"
|
|
exit 1
|
|
fi
|
|
echo "Using AMI: $AMI_ID"
|
|
|
|
# Authorize SSH on the default VPC's default SG before any run-instances. AWS
|
|
# default SGs allow ingress only from members of the same SG; the Phase 4 SSH
|
|
# probe comes from the test runner's IP and would be dropped by the OVN
|
|
# port-group ACL otherwise.
|
|
DEFAULT_VPC_PHASE3=$($AWS_EC2 describe-vpcs \
|
|
--query 'Vpcs[?IsDefault==`true`].VpcId | [0]' --output text)
|
|
DEFAULT_SG_PHASE3=$($AWS_EC2 describe-security-groups \
|
|
--filters "Name=vpc-id,Values=$DEFAULT_VPC_PHASE3" "Name=group-name,Values=default" \
|
|
--query 'SecurityGroups[0].GroupId' --output text)
|
|
echo "Default VPC: $DEFAULT_VPC_PHASE3; default SG: $DEFAULT_SG_PHASE3"
|
|
# Tolerate InvalidPermission.Duplicate on re-runs (idempotent).
|
|
set +e
|
|
AUTH_OUTPUT=$($AWS_EC2 authorize-security-group-ingress \
|
|
--group-id "$DEFAULT_SG_PHASE3" \
|
|
--protocol tcp --port 22 --cidr 0.0.0.0/0 2>&1)
|
|
AUTH_EXIT=$?
|
|
set -e
|
|
if [ $AUTH_EXIT -ne 0 ] && ! echo "$AUTH_OUTPUT" | grep -q 'InvalidPermission.Duplicate'; then
|
|
echo "Failed to authorize SSH ingress on default SG: $AUTH_OUTPUT"
|
|
exit 1
|
|
fi
|
|
echo " Default SG ingress: tcp/22 from 0.0.0.0/0"
|
|
|
|
# Launch 3 instances with stagger to encourage distribution
|
|
echo "Launching 3 instances..."
|
|
PHASE3_DENSITY_SKIP=false
|
|
for i in 1 2 3; do
|
|
echo " Launching instance $i..."
|
|
set +e
|
|
RUN_OUTPUT=$($AWS_EC2 run-instances \
|
|
--image-id "$AMI_ID" \
|
|
--instance-type "$INSTANCE_TYPE" \
|
|
--key-name spinifex-key 2>&1)
|
|
RUN_RC=$?
|
|
set -e
|
|
|
|
if [ $RUN_RC -ne 0 ]; then
|
|
if is_density_skip "$RUN_OUTPUT"; then
|
|
skip_test "Phase 3 launch instance $i" "InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT"
|
|
PHASE3_DENSITY_SKIP=true
|
|
break
|
|
fi
|
|
echo " ERROR: Failed to launch instance $i"
|
|
echo " Output: $RUN_OUTPUT"
|
|
exit 1
|
|
fi
|
|
|
|
INSTANCE_ID=$(echo "$RUN_OUTPUT" | jq -r '.Instances[0].InstanceId')
|
|
if [ -z "$INSTANCE_ID" ] || [ "$INSTANCE_ID" == "null" ]; then
|
|
echo " ERROR: Failed to launch instance $i"
|
|
echo " Output: $RUN_OUTPUT"
|
|
exit 1
|
|
fi
|
|
echo " Launched: $INSTANCE_ID"
|
|
INSTANCE_IDS+=("$INSTANCE_ID")
|
|
|
|
# Stagger launches to encourage distribution across nodes
|
|
[ $i -lt 3 ] && sleep 2
|
|
done
|
|
|
|
if [ "$PHASE3_DENSITY_SKIP" = true ] && [ ${#INSTANCE_IDS[@]} -eq 0 ]; then
|
|
echo " Phase 3 launched zero instances on undersized cluster; skipping dependent phases gracefully."
|
|
fi
|
|
|
|
# Wait for all instances to be running. On a sub-DENSITY_NODE_TARGET cluster
|
|
# an instance that failed to start (capacity, scheduler race) is downgraded
|
|
# to a skip rather than a hard exit so the rest of the suite still runs.
|
|
echo ""
|
|
echo "Waiting for instances to reach running state..."
|
|
RUNNING_INSTANCE_IDS=()
|
|
for instance_id in "${INSTANCE_IDS[@]}"; do
|
|
if wait_for_instance_state "$instance_id" "running" 60; then
|
|
RUNNING_INSTANCE_IDS+=("$instance_id")
|
|
continue
|
|
fi
|
|
if [ "$NODE_COUNT" -lt "$DENSITY_NODE_TARGET" ]; then
|
|
skip_test "Phase 3 wait-running $instance_id" "instance never reached running on NODE_COUNT=$NODE_COUNT"
|
|
continue
|
|
fi
|
|
echo "ERROR: Instance $instance_id failed to start"
|
|
exit 1
|
|
done
|
|
INSTANCE_IDS=("${RUNNING_INSTANCE_IDS[@]}")
|
|
|
|
# Check distribution via spx get vms or QEMU process check
|
|
echo ""
|
|
echo "Checking instance distribution across nodes..."
|
|
declare -A NODE_INSTANCE_COUNT
|
|
HOSTING_NODES=()
|
|
for instance_id in "${INSTANCE_IDS[@]}"; do
|
|
HOST_IP=$(find_instance_node "$instance_id" || echo "unknown")
|
|
echo " $instance_id -> $HOST_IP"
|
|
HOSTING_NODES+=("$HOST_IP")
|
|
NODE_INSTANCE_COUNT[$HOST_IP]=$(( ${NODE_INSTANCE_COUNT[$HOST_IP]:-0} + 1 ))
|
|
done
|
|
|
|
# Count unique hosting nodes
|
|
UNIQUE_HOSTS=$(printf '%s\n' "${HOSTING_NODES[@]}" | sort -u | wc -l)
|
|
echo " Instances on $UNIQUE_HOSTS different nodes"
|
|
if [ "$UNIQUE_HOSTS" -ge 2 ]; then
|
|
pass_test "Instance distribution (>= 2 nodes)"
|
|
else
|
|
echo " WARNING: All instances on same node (distribution not guaranteed, non-fatal)"
|
|
pass_test "Instance distribution (non-deterministic)"
|
|
fi
|
|
|
|
# Verify spx get vms shows all instances
|
|
echo ""
|
|
echo "Verifying spx get vms..."
|
|
GET_VMS_OUTPUT=$($SPINIFEX_BIN get vms --timeout 5s 2>/dev/null)
|
|
echo "$GET_VMS_OUTPUT"
|
|
for instance_id in "${INSTANCE_IDS[@]}"; do
|
|
if ! echo "$GET_VMS_OUTPUT" | grep -q "$instance_id"; then
|
|
echo " WARNING: spx get vms did not show $instance_id"
|
|
fi
|
|
done
|
|
pass_test "spx get vms (with instances)"
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 4: SSH into Guest VMs
|
|
# ==========================================================================
|
|
echo "Phase 4: SSH into Guest VMs"
|
|
echo "========================================"
|
|
|
|
for idx in "${!INSTANCE_IDS[@]}"; do
|
|
instance_id="${INSTANCE_IDS[$idx]}"
|
|
host_ip="${HOSTING_NODES[$idx]}"
|
|
echo ""
|
|
echo " Instance $((idx + 1)): $instance_id (on $host_ip)"
|
|
|
|
# Determine SSH endpoint: use public IP if available, else QEMU hostfwd
|
|
INST_PUB_IP=$($AWS_EC2 describe-instances \
|
|
--instance-ids "$instance_id" \
|
|
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text 2>/dev/null || echo "None")
|
|
|
|
if [ -n "$INST_PUB_IP" ] && [ "$INST_PUB_IP" != "None" ] && [ "$INST_PUB_IP" != "null" ]; then
|
|
SSH_HOST="$INST_PUB_IP"
|
|
SSH_PORT=22
|
|
SSH_KEY="$HOME/.ssh/spinifex-key"
|
|
echo " SSH via public IP: $SSH_HOST:$SSH_PORT"
|
|
else
|
|
SSH_HOST="$host_ip"
|
|
# set -e would abort on $() returning non-zero, hiding the diagnostic below
|
|
if ! SSH_PORT=$(get_remote_ssh_port "$host_ip" "$instance_id" 30); then
|
|
SSH_PORT=""
|
|
fi
|
|
SSH_KEY="$HOME/.ssh/spinifex-key"
|
|
if [ -z "$SSH_PORT" ]; then
|
|
echo " ERROR: Failed to get SSH port for $instance_id on $host_ip"
|
|
dump_guest_ssh_diagnostics "$instance_id" "$host_ip" "$SSH_HOST" ""
|
|
fail_test "Guest SSH ($instance_id)"
|
|
continue
|
|
fi
|
|
echo " SSH endpoint: $SSH_HOST:$SSH_PORT"
|
|
fi
|
|
|
|
# Wait for SSH to be ready (VM boot + cloud-init)
|
|
# Bare-metal NBD-backed VMs take longer than tofu-cluster VMs to boot.
|
|
SSH_MAX_ATTEMPTS=120
|
|
echo " Waiting for SSH to be ready..."
|
|
ATTEMPT=0
|
|
SSH_READY=false
|
|
while [ $ATTEMPT -lt $SSH_MAX_ATTEMPTS ]; do
|
|
if ssh -o StrictHostKeyChecking=no \
|
|
-o UserKnownHostsFile=/dev/null \
|
|
-o ConnectTimeout=2 \
|
|
-o BatchMode=yes \
|
|
-o LogLevel=ERROR \
|
|
-p "$SSH_PORT" \
|
|
-i "$SSH_KEY" \
|
|
ubuntu@"$SSH_HOST" 'echo ready' > /dev/null 2>&1; then
|
|
SSH_READY=true
|
|
break
|
|
fi
|
|
ATTEMPT=$((ATTEMPT + 1))
|
|
[ $((ATTEMPT % 10)) -eq 0 ] && echo " Waiting for SSH... ($ATTEMPT/$SSH_MAX_ATTEMPTS)"
|
|
sleep 1
|
|
done
|
|
|
|
if [ "$SSH_READY" = false ]; then
|
|
echo " ERROR: SSH not ready after $SSH_MAX_ATTEMPTS attempts"
|
|
dump_guest_ssh_diagnostics "$instance_id" "$host_ip" "$SSH_HOST" "$SSH_PORT"
|
|
fail_test "Guest SSH ($instance_id)"
|
|
continue
|
|
fi
|
|
|
|
# Test SSH connectivity
|
|
ID_OUTPUT=$(ssh -o StrictHostKeyChecking=no \
|
|
-o UserKnownHostsFile=/dev/null \
|
|
-o ConnectTimeout=5 \
|
|
-o BatchMode=yes \
|
|
-o LogLevel=ERROR \
|
|
-p "$SSH_PORT" \
|
|
-i "$SSH_KEY" \
|
|
ubuntu@"$SSH_HOST" 'id' 2>&1) || {
|
|
echo " ERROR: SSH 'id' command failed"
|
|
fail_test "Guest SSH ($instance_id)"
|
|
continue
|
|
}
|
|
|
|
echo " SSH 'id' output: $ID_OUTPUT"
|
|
if echo "$ID_OUTPUT" | grep -q "ubuntu"; then
|
|
echo " ubuntu confirmed"
|
|
else
|
|
echo " ERROR: Expected 'ubuntu' in id output"
|
|
fail_test "Guest SSH ($instance_id)"
|
|
continue
|
|
fi
|
|
|
|
# Verify block device
|
|
LSBLK_OUTPUT=$(ssh -o StrictHostKeyChecking=no \
|
|
-o UserKnownHostsFile=/dev/null \
|
|
-o ConnectTimeout=5 \
|
|
-o BatchMode=yes \
|
|
-o LogLevel=ERROR \
|
|
-p "$SSH_PORT" \
|
|
-i "$SSH_KEY" \
|
|
ubuntu@"$SSH_HOST" 'lsblk' 2>&1) || true
|
|
echo " lsblk: $(echo "$LSBLK_OUTPUT" | head -5)"
|
|
|
|
pass_test "Guest SSH ($instance_id)"
|
|
done
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 5: Volume Lifecycle
|
|
# ==========================================================================
|
|
echo "Phase 5: Volume Lifecycle"
|
|
echo "========================================"
|
|
|
|
# Discover AZ
|
|
SPINIFEX_AZ=$($AWS_EC2 describe-availability-zones --query 'AvailabilityZones[0].ZoneName' --output text)
|
|
echo "AZ: $SPINIFEX_AZ"
|
|
|
|
if [ ${#INSTANCE_IDS[@]} -eq 0 ]; then
|
|
skip_test "Volume lifecycle" "no Phase 3 instances available for attach"
|
|
else
|
|
|
|
# Create volume
|
|
echo "Creating 10GB test volume..."
|
|
CREATE_OUTPUT=$($AWS_EC2 create-volume --size 10 --availability-zone "$SPINIFEX_AZ")
|
|
TEST_VOLUME_ID=$(echo "$CREATE_OUTPUT" | jq -r '.VolumeId')
|
|
if [ -z "$TEST_VOLUME_ID" ] || [ "$TEST_VOLUME_ID" == "null" ]; then
|
|
echo " ERROR: Failed to create volume"
|
|
fail_test "Volume create"
|
|
else
|
|
echo " Created: $TEST_VOLUME_ID"
|
|
pass_test "Volume create"
|
|
|
|
# Attach to first instance
|
|
echo " Attaching to ${INSTANCE_IDS[0]}..."
|
|
$AWS_EC2 attach-volume --volume-id "$TEST_VOLUME_ID" \
|
|
--instance-id "${INSTANCE_IDS[0]}" --device /dev/sdf > /dev/null
|
|
|
|
# Wait for attachment
|
|
COUNT=0
|
|
while [ $COUNT -lt 30 ]; do
|
|
ATTACH_STATE=$($AWS_EC2 describe-volumes --volume-ids "$TEST_VOLUME_ID" \
|
|
--query 'Volumes[0].Attachments[0].State' --output text)
|
|
VOL_STATE=$($AWS_EC2 describe-volumes --volume-ids "$TEST_VOLUME_ID" \
|
|
--query 'Volumes[0].State' --output text)
|
|
if [ "$VOL_STATE" == "in-use" ] && [ "$ATTACH_STATE" == "attached" ]; then
|
|
echo " Volume attached"
|
|
break
|
|
fi
|
|
sleep 1
|
|
COUNT=$((COUNT + 1))
|
|
done
|
|
|
|
if [ "$ATTACH_STATE" == "attached" ]; then
|
|
pass_test "Volume attach"
|
|
else
|
|
fail_test "Volume attach"
|
|
fi
|
|
|
|
# Detach
|
|
echo " Detaching volume..."
|
|
$AWS_EC2 detach-volume --volume-id "$TEST_VOLUME_ID" > /dev/null
|
|
|
|
COUNT=0
|
|
while [ $COUNT -lt 30 ]; do
|
|
VOL_STATE=$($AWS_EC2 describe-volumes --volume-ids "$TEST_VOLUME_ID" \
|
|
--query 'Volumes[0].State' --output text)
|
|
if [ "$VOL_STATE" == "available" ]; then
|
|
echo " Volume detached"
|
|
break
|
|
fi
|
|
sleep 1
|
|
COUNT=$((COUNT + 1))
|
|
done
|
|
|
|
if [ "$VOL_STATE" == "available" ]; then
|
|
pass_test "Volume detach"
|
|
else
|
|
fail_test "Volume detach"
|
|
fi
|
|
|
|
# Delete
|
|
echo " Deleting volume..."
|
|
$AWS_EC2 delete-volume --volume-id "$TEST_VOLUME_ID"
|
|
|
|
COUNT=0
|
|
while [ $COUNT -lt 30 ]; do
|
|
set +e
|
|
VOL_CHECK=$($AWS_EC2 describe-volumes --volume-ids "$TEST_VOLUME_ID" \
|
|
--query 'Volumes[0].VolumeId' --output text 2>&1)
|
|
DESCRIBE_EXIT=$?
|
|
set -e
|
|
if [ $DESCRIBE_EXIT -ne 0 ] || [ "$VOL_CHECK" == "None" ] || [ -z "$VOL_CHECK" ]; then
|
|
echo " Volume deleted"
|
|
break
|
|
fi
|
|
sleep 1
|
|
COUNT=$((COUNT + 1))
|
|
done
|
|
|
|
if [ $COUNT -lt 30 ]; then
|
|
pass_test "Volume delete"
|
|
else
|
|
fail_test "Volume delete"
|
|
fi
|
|
fi
|
|
|
|
fi # INSTANCE_IDS non-empty guard
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 6: Cross-Node Gateway Access
|
|
# ==========================================================================
|
|
echo "Phase 6: Cross-Node Gateway Access"
|
|
echo "========================================"
|
|
echo "Verifying describe-instances returns same results via each node's gateway..."
|
|
|
|
BASELINE_COUNT=$($AWS_EC2 describe-instances \
|
|
--query 'length(Reservations[*].Instances[*][])' --output text)
|
|
echo " Baseline (node1): $BASELINE_COUNT instances"
|
|
|
|
for i in $(seq 1 $((NODE_COUNT - 1))); do
|
|
ip="${NODE_IPS[$i]}"
|
|
COUNT=$(aws_via "$ip" ec2 describe-instances \
|
|
--query 'length(Reservations[*].Instances[*][])' --output text 2>/dev/null || echo "0")
|
|
echo " Node$((i+1)) ($ip): $COUNT instances"
|
|
if [ "$COUNT" -eq "$BASELINE_COUNT" ]; then
|
|
pass_test "Cross-node gateway (node$((i+1)))"
|
|
else
|
|
echo " WARNING: Instance count mismatch (expected $BASELINE_COUNT, got $COUNT)"
|
|
fail_test "Cross-node gateway (node$((i+1)))"
|
|
fi
|
|
done
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 7: Cross-Node Operations
|
|
# ==========================================================================
|
|
echo "Phase 7: Cross-Node Operations"
|
|
echo "========================================"
|
|
echo "Testing stop/start via a gateway on a DIFFERENT node than the hosting node..."
|
|
|
|
if [ ${#INSTANCE_IDS[@]} -eq 0 ]; then
|
|
skip_test "Cross-node stop/start" "no Phase 3 instances available"
|
|
else
|
|
# Pick first instance and find its host
|
|
TEST_INSTANCE="${INSTANCE_IDS[0]}"
|
|
INSTANCE_HOST=$(find_instance_node "$TEST_INSTANCE" || echo "$LOCAL_IP")
|
|
echo " Instance $TEST_INSTANCE is on $INSTANCE_HOST"
|
|
|
|
# Pick a different node's gateway for the operation
|
|
OTHER_GW=""
|
|
for ip in "${NODE_IPS[@]}"; do
|
|
if [ "$ip" != "$INSTANCE_HOST" ]; then
|
|
OTHER_GW="$ip"
|
|
break
|
|
fi
|
|
done
|
|
echo " Will operate via gateway on $OTHER_GW"
|
|
|
|
# Stop via other gateway
|
|
echo " Stopping instance via $OTHER_GW..."
|
|
aws_via "$OTHER_GW" ec2 stop-instances --instance-ids "$TEST_INSTANCE" > /dev/null
|
|
wait_for_instance_state "$TEST_INSTANCE" "stopped" 60 "$OTHER_GW"
|
|
|
|
# Pick yet another gateway for start (or same other if only 2 choices)
|
|
THIRD_GW=""
|
|
for ip in "${NODE_IPS[@]}"; do
|
|
if [ "$ip" != "$INSTANCE_HOST" ] && [ "$ip" != "$OTHER_GW" ]; then
|
|
THIRD_GW="$ip"
|
|
break
|
|
fi
|
|
done
|
|
THIRD_GW="${THIRD_GW:-$OTHER_GW}"
|
|
echo " Starting instance via $THIRD_GW..."
|
|
set +e
|
|
START_OUT=$(aws_via "$THIRD_GW" ec2 start-instances --instance-ids "$TEST_INSTANCE" 2>&1)
|
|
START_RC=$?
|
|
set -e
|
|
if [ $START_RC -ne 0 ]; then
|
|
if is_density_skip "$START_OUT"; then
|
|
skip_test "Cross-node stop/start" "InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT"
|
|
else
|
|
echo " ERROR: start-instances failed: $START_OUT"
|
|
fail_test "Cross-node stop/start"
|
|
fi
|
|
else
|
|
wait_for_instance_state "$TEST_INSTANCE" "running" 60 "$THIRD_GW"
|
|
pass_test "Cross-node stop/start"
|
|
fi
|
|
fi
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 8: Node Failure
|
|
# ==========================================================================
|
|
echo "Phase 8: Node Failure"
|
|
echo "========================================"
|
|
echo "Stopping services on node2 ($NODE2_IP) to simulate node failure..."
|
|
|
|
# Stop services on node2 only — SPINIFEX_FORCE_LOCAL_STOP prevents coordinated
|
|
# cluster shutdown which would kill all nodes via NATS
|
|
peer_ssh "$NODE2_IP" "sudo systemctl stop spinifex.target" || {
|
|
echo " WARNING: systemctl stop returned non-zero (may be expected)"
|
|
}
|
|
|
|
# Wait for NATS cluster to detect the failure and reform
|
|
sleep 10
|
|
|
|
# Verify node1 and node3 still serve requests (with retries for NATS reformation)
|
|
echo " Verifying node1 still serves requests..."
|
|
N1_RESULT=$(aws_via_retry 10 "$LOCAL_IP" ec2 describe-instance-types \
|
|
--query 'InstanceTypes[0].InstanceType' --output text) || N1_RESULT="FAIL"
|
|
if [ "$N1_RESULT" != "FAIL" ]; then
|
|
echo " Node1: responding ($N1_RESULT)"
|
|
pass_test "Node1 survives node2 failure"
|
|
else
|
|
echo " ERROR: Node1 not responding after node2 failure"
|
|
fail_test "Node1 survives node2 failure"
|
|
fi
|
|
|
|
if [ -n "$NODE3_IP" ]; then
|
|
echo " Verifying node3 still serves requests..."
|
|
N3_RESULT=$(aws_via_retry 10 "$NODE3_IP" ec2 describe-instance-types \
|
|
--query 'InstanceTypes[0].InstanceType' --output text) || N3_RESULT="FAIL"
|
|
if [ "$N3_RESULT" != "FAIL" ]; then
|
|
echo " Node3: responding ($N3_RESULT)"
|
|
pass_test "Node3 survives node2 failure"
|
|
else
|
|
echo " ERROR: Node3 not responding after node2 failure"
|
|
fail_test "Node3 survives node2 failure"
|
|
fi
|
|
else
|
|
echo " Node3 not present in this cluster — skipping node3 survival check"
|
|
fi
|
|
|
|
# Check NATS degraded state (should have 1 route instead of 2)
|
|
NATS_DEGRADED=$(curl -s "http://127.0.0.1:${NATS_MONITOR_PORT}/routez" 2>/dev/null)
|
|
DEGRADED_PEERS=$(echo "$NATS_DEGRADED" | jq -r '[.routes[].remote_name] | unique | length' 2>/dev/null || echo "0")
|
|
echo " NATS peers during failure: $DEGRADED_PEERS (expected: 1)"
|
|
if [ "$DEGRADED_PEERS" -eq 1 ]; then
|
|
pass_test "NATS degraded mode"
|
|
else
|
|
echo " WARNING: Expected 1 NATS peer during node2 failure, got $DEGRADED_PEERS"
|
|
# Not fatal — NATS might take a moment to detect
|
|
fi
|
|
|
|
# Verify describe-instances still works from surviving nodes
|
|
echo " Verifying describe-instances from surviving nodes..."
|
|
SURVIVING_COUNT=$(aws_via_retry 10 "$LOCAL_IP" ec2 describe-instances \
|
|
--query 'length(Reservations[*].Instances[*][])' --output text) || SURVIVING_COUNT="0"
|
|
echo " Instances visible from node1: $SURVIVING_COUNT"
|
|
if [ "$SURVIVING_COUNT" -gt 0 ]; then
|
|
pass_test "Describe-instances during node failure"
|
|
else
|
|
fail_test "Describe-instances during node failure"
|
|
fi
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 9: Node Recovery
|
|
# ==========================================================================
|
|
echo "Phase 9: Node Recovery"
|
|
echo "========================================"
|
|
echo "Restarting services on node2 ($NODE2_IP)..."
|
|
|
|
peer_ssh "$NODE2_IP" "sudo systemctl start spinifex.target" || {
|
|
echo " ERROR: Failed to restart services on node2"
|
|
fail_test "Node2 restart"
|
|
}
|
|
|
|
# Wait for NATS to reform (2 routes again)
|
|
echo " Waiting for NATS cluster to reform..."
|
|
ATTEMPT=0
|
|
REFORMED=false
|
|
while [ $ATTEMPT -lt 60 ]; do
|
|
NATS_RECOVER=$(curl -s "http://127.0.0.1:${NATS_MONITOR_PORT}/routez" 2>/dev/null)
|
|
RECOVER_PEERS=$(echo "$NATS_RECOVER" | jq -r '[.routes[].remote_name] | unique | length' 2>/dev/null || echo "0")
|
|
if [ "$RECOVER_PEERS" -ge "$((NODE_COUNT - 1))" ]; then
|
|
echo " NATS cluster reformed ($RECOVER_PEERS peers)"
|
|
REFORMED=true
|
|
break
|
|
fi
|
|
echo " Waiting for NATS reform... ($((ATTEMPT + 1))/60, peers: $RECOVER_PEERS)"
|
|
sleep 1
|
|
ATTEMPT=$((ATTEMPT + 1))
|
|
done
|
|
|
|
if [ "$REFORMED" = true ]; then
|
|
pass_test "NATS cluster reform"
|
|
else
|
|
echo " WARNING: NATS did not fully reform within timeout"
|
|
fail_test "NATS cluster reform"
|
|
fi
|
|
|
|
# Verify node2 gateway is back
|
|
echo " Waiting for node2 gateway..."
|
|
ATTEMPT=0
|
|
GW_BACK=false
|
|
while [ $ATTEMPT -lt 30 ]; do
|
|
if curl -k -s "https://${NODE2_IP}:${AWSGW_PORT}" > /dev/null 2>&1; then
|
|
echo " Node2 gateway is back"
|
|
GW_BACK=true
|
|
break
|
|
fi
|
|
sleep 1
|
|
ATTEMPT=$((ATTEMPT + 1))
|
|
done
|
|
|
|
if [ "$GW_BACK" = true ]; then
|
|
pass_test "Node2 gateway recovery"
|
|
else
|
|
fail_test "Node2 gateway recovery"
|
|
fi
|
|
|
|
# Verify spx get nodes shows all nodes Ready again
|
|
echo " Checking spx get nodes after recovery..."
|
|
GET_NODES_RECOVER=$($SPINIFEX_BIN get nodes --timeout 10s 2>/dev/null || echo "")
|
|
echo "$GET_NODES_RECOVER"
|
|
READY_RECOVER=$(echo "$GET_NODES_RECOVER" | grep -c "Ready" || true)
|
|
if [ "$READY_RECOVER" -ge "$NODE_COUNT" ]; then
|
|
pass_test "All nodes Ready after recovery"
|
|
else
|
|
echo " WARNING: Only $READY_RECOVER Ready nodes after recovery (expected $NODE_COUNT)"
|
|
fail_test "All nodes Ready after recovery"
|
|
fi
|
|
|
|
# Verify node2 can serve requests
|
|
echo " Verifying node2 serves requests after recovery..."
|
|
N2_RESULT=$(aws_via "$NODE2_IP" ec2 describe-instance-types \
|
|
--query 'InstanceTypes[0].InstanceType' --output text 2>/dev/null || echo "FAIL")
|
|
if [ "$N2_RESULT" != "FAIL" ] && [ -n "$N2_RESULT" ] && [ "$N2_RESULT" != "None" ]; then
|
|
echo " Node2 is serving requests again ($N2_RESULT)"
|
|
pass_test "Node2 serves requests after recovery"
|
|
else
|
|
fail_test "Node2 serves requests after recovery"
|
|
fi
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 10: VPC Networking
|
|
# ==========================================================================
|
|
echo "Phase 10: VPC Networking"
|
|
echo "========================================"
|
|
echo "Testing VPC instance launch, PrivateIpAddress, and stop/start IP persistence..."
|
|
|
|
# Create a fresh VPC + subnet (don't reuse bootstrap's to avoid interference)
|
|
echo ""
|
|
echo "Creating test VPC + subnet..."
|
|
VPC_OUTPUT=$($AWS_EC2 create-vpc --cidr-block 10.200.0.0/16)
|
|
VPC_ID=$(echo "$VPC_OUTPUT" | jq -r '.Vpc.VpcId')
|
|
if [ -z "$VPC_ID" ] || [ "$VPC_ID" == "null" ]; then
|
|
echo " ERROR: Failed to create VPC"
|
|
fail_test "VPC create"
|
|
else
|
|
echo " Created VPC: $VPC_ID (10.200.0.0/16)"
|
|
pass_test "VPC create"
|
|
|
|
SUBNET_OUTPUT=$($AWS_EC2 create-subnet --vpc-id "$VPC_ID" --cidr-block 10.200.1.0/24)
|
|
SUBNET_ID=$(echo "$SUBNET_OUTPUT" | jq -r '.Subnet.SubnetId')
|
|
if [ -z "$SUBNET_ID" ] || [ "$SUBNET_ID" == "null" ]; then
|
|
echo " ERROR: Failed to create subnet"
|
|
fail_test "Subnet create"
|
|
else
|
|
echo " Created Subnet: $SUBNET_ID (10.200.1.0/24)"
|
|
pass_test "Subnet create"
|
|
|
|
# Brief pause for OVN topology programming
|
|
sleep 2
|
|
|
|
# Launch 3 instances with subnet
|
|
echo ""
|
|
echo "Launching 3 VPC instances..."
|
|
VPC_INSTANCE_IDS=()
|
|
VPC_LAUNCH_OK=true
|
|
VPC_DENSITY_SKIP=false
|
|
for i in 1 2 3; do
|
|
echo " Launching VPC instance $i with subnet $SUBNET_ID..."
|
|
set +e
|
|
RUN_OUTPUT=$($AWS_EC2 run-instances \
|
|
--image-id "$AMI_ID" \
|
|
--instance-type "$INSTANCE_TYPE" \
|
|
--key-name spinifex-key \
|
|
--subnet-id "$SUBNET_ID" 2>&1)
|
|
VPC_RUN_RC=$?
|
|
set -e
|
|
|
|
if [ $VPC_RUN_RC -ne 0 ]; then
|
|
if is_density_skip "$RUN_OUTPUT"; then
|
|
skip_test "Phase 10 VPC instance $i launch" "InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT"
|
|
VPC_DENSITY_SKIP=true
|
|
VPC_LAUNCH_OK=false
|
|
break
|
|
fi
|
|
echo " ERROR: Failed to launch VPC instance $i: $RUN_OUTPUT"
|
|
VPC_LAUNCH_OK=false
|
|
break
|
|
fi
|
|
|
|
VPC_INST_ID=$(echo "$RUN_OUTPUT" | jq -r '.Instances[0].InstanceId')
|
|
VPC_INST_IP=$(echo "$RUN_OUTPUT" | jq -r '.Instances[0].PrivateIpAddress // empty')
|
|
|
|
if [ -z "$VPC_INST_ID" ] || [ "$VPC_INST_ID" == "null" ]; then
|
|
echo " ERROR: Failed to launch VPC instance $i"
|
|
VPC_LAUNCH_OK=false
|
|
break
|
|
fi
|
|
echo " Launched: $VPC_INST_ID (PrivateIpAddress: ${VPC_INST_IP:-not yet assigned})"
|
|
VPC_INSTANCE_IDS+=("$VPC_INST_ID")
|
|
sleep 1
|
|
done
|
|
|
|
if [ "$VPC_LAUNCH_OK" = true ] && [ ${#VPC_INSTANCE_IDS[@]} -eq 3 ]; then
|
|
pass_test "VPC instance launch"
|
|
|
|
# Wait for all VPC instances to be running
|
|
echo ""
|
|
echo "Waiting for VPC instances to reach running state..."
|
|
VPC_ALL_RUNNING=true
|
|
for vpc_inst in "${VPC_INSTANCE_IDS[@]}"; do
|
|
wait_for_instance_state "$vpc_inst" "running" 60 || {
|
|
echo " ERROR: VPC instance $vpc_inst failed to start"
|
|
VPC_ALL_RUNNING=false
|
|
}
|
|
done
|
|
|
|
if [ "$VPC_ALL_RUNNING" = true ]; then
|
|
# Verify PrivateIpAddress in DescribeInstances
|
|
echo ""
|
|
echo "Verifying PrivateIpAddress in DescribeInstances..."
|
|
VPC_PRIVATE_IPS=()
|
|
VPC_IP_OK=true
|
|
for vpc_inst in "${VPC_INSTANCE_IDS[@]}"; do
|
|
DESCRIBE_OUT=$($AWS_EC2 describe-instances --instance-ids "$vpc_inst")
|
|
PRIVATE_IP=$(echo "$DESCRIBE_OUT" | jq -r '.Reservations[0].Instances[0].PrivateIpAddress // empty')
|
|
INST_SUBNET=$(echo "$DESCRIBE_OUT" | jq -r '.Reservations[0].Instances[0].SubnetId // empty')
|
|
INST_VPC=$(echo "$DESCRIBE_OUT" | jq -r '.Reservations[0].Instances[0].VpcId // empty')
|
|
|
|
if [ -z "$PRIVATE_IP" ]; then
|
|
echo " ERROR: $vpc_inst has no PrivateIpAddress"
|
|
VPC_IP_OK=false
|
|
continue
|
|
fi
|
|
|
|
echo " $vpc_inst: IP=$PRIVATE_IP, Subnet=$INST_SUBNET, VPC=$INST_VPC"
|
|
|
|
if [ "$INST_SUBNET" != "$SUBNET_ID" ]; then
|
|
echo " ERROR: SubnetId mismatch (expected $SUBNET_ID, got $INST_SUBNET)"
|
|
VPC_IP_OK=false
|
|
fi
|
|
if [ "$INST_VPC" != "$VPC_ID" ]; then
|
|
echo " ERROR: VpcId mismatch (expected $VPC_ID, got $INST_VPC)"
|
|
VPC_IP_OK=false
|
|
fi
|
|
|
|
VPC_PRIVATE_IPS+=("$PRIVATE_IP")
|
|
done
|
|
|
|
if [ "$VPC_IP_OK" = true ]; then
|
|
# Verify all IPs are unique and in the subnet range (10.200.1.x)
|
|
UNIQUE_IPS=$(printf '%s\n' "${VPC_PRIVATE_IPS[@]}" | sort -u | wc -l)
|
|
IP_RANGE_OK=true
|
|
if [ "$UNIQUE_IPS" -ne "${#VPC_PRIVATE_IPS[@]}" ]; then
|
|
echo " ERROR: Duplicate IPs detected: ${VPC_PRIVATE_IPS[*]}"
|
|
IP_RANGE_OK=false
|
|
fi
|
|
for ip in "${VPC_PRIVATE_IPS[@]}"; do
|
|
if ! echo "$ip" | grep -qE '^10\.200\.1\.[0-9]+$'; then
|
|
echo " ERROR: IP $ip not in expected subnet 10.200.1.0/24"
|
|
IP_RANGE_OK=false
|
|
fi
|
|
done
|
|
|
|
if [ "$IP_RANGE_OK" = true ]; then
|
|
echo " All IPs unique and in correct subnet: ${VPC_PRIVATE_IPS[*]}"
|
|
pass_test "VPC IP allocation"
|
|
else
|
|
fail_test "VPC IP allocation"
|
|
fi
|
|
|
|
# Stop/Start IP persistence
|
|
echo ""
|
|
echo "Testing stop/start IP persistence..."
|
|
echo " IPs before stop: ${VPC_PRIVATE_IPS[*]}"
|
|
|
|
# Stop all VPC instances
|
|
for vpc_inst in "${VPC_INSTANCE_IDS[@]}"; do
|
|
$AWS_EC2 stop-instances --instance-ids "$vpc_inst" > /dev/null
|
|
done
|
|
VPC_ALL_STOPPED=true
|
|
for vpc_inst in "${VPC_INSTANCE_IDS[@]}"; do
|
|
wait_for_instance_state "$vpc_inst" "stopped" 60 || {
|
|
echo " ERROR: VPC instance $vpc_inst failed to stop"
|
|
VPC_ALL_STOPPED=false
|
|
}
|
|
done
|
|
|
|
if [ "$VPC_ALL_STOPPED" = true ]; then
|
|
# Verify IPs persist while stopped
|
|
IP_PERSIST_OK=true
|
|
for idx in "${!VPC_INSTANCE_IDS[@]}"; do
|
|
vpc_inst="${VPC_INSTANCE_IDS[$idx]}"
|
|
expected_ip="${VPC_PRIVATE_IPS[$idx]}"
|
|
STOPPED_IP=$($AWS_EC2 describe-instances --instance-ids "$vpc_inst" \
|
|
--query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)
|
|
if [ "$STOPPED_IP" != "$expected_ip" ]; then
|
|
echo " ERROR: $vpc_inst IP changed while stopped (expected $expected_ip, got $STOPPED_IP)"
|
|
IP_PERSIST_OK=false
|
|
fi
|
|
done
|
|
|
|
# Restart all VPC instances. InsufficientInstanceCapacity here is
|
|
# downgraded to a skip only on sub-DENSITY_NODE_TARGET clusters;
|
|
# on a 3+ node cluster it remains a real failure.
|
|
VPC_RESTART_OK=true
|
|
VPC_RESTART_DENSITY_SKIP=false
|
|
set +e
|
|
for vpc_inst in "${VPC_INSTANCE_IDS[@]}"; do
|
|
START_OUT=$($AWS_EC2 start-instances --instance-ids "$vpc_inst" 2>&1)
|
|
START_RC=$?
|
|
if [ $START_RC -ne 0 ]; then
|
|
if is_density_skip "$START_OUT"; then
|
|
echo " SKIP: $vpc_inst could not restart (InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT)"
|
|
VPC_RESTART_OK=false
|
|
VPC_RESTART_DENSITY_SKIP=true
|
|
else
|
|
echo " ERROR: $vpc_inst start-instances failed: $START_OUT"
|
|
IP_PERSIST_OK=false
|
|
VPC_RESTART_OK=false
|
|
fi
|
|
fi
|
|
done
|
|
set -e
|
|
|
|
if [ "$VPC_RESTART_OK" = true ]; then
|
|
for vpc_inst in "${VPC_INSTANCE_IDS[@]}"; do
|
|
wait_for_instance_state "$vpc_inst" "running" 60 || true
|
|
done
|
|
|
|
# Verify IPs persist after restart
|
|
for idx in "${!VPC_INSTANCE_IDS[@]}"; do
|
|
vpc_inst="${VPC_INSTANCE_IDS[$idx]}"
|
|
expected_ip="${VPC_PRIVATE_IPS[$idx]}"
|
|
RESTARTED_IP=$($AWS_EC2 describe-instances --instance-ids "$vpc_inst" \
|
|
--query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)
|
|
if [ "$RESTARTED_IP" != "$expected_ip" ]; then
|
|
echo " ERROR: $vpc_inst IP changed after restart (expected $expected_ip, got $RESTARTED_IP)"
|
|
IP_PERSIST_OK=false
|
|
else
|
|
echo " $vpc_inst: IP=$RESTARTED_IP (matches pre-stop)"
|
|
fi
|
|
done
|
|
fi
|
|
|
|
if [ "$VPC_RESTART_DENSITY_SKIP" = true ]; then
|
|
skip_test "VPC stop/start IP persistence" "restart blocked by InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT"
|
|
elif [ "$IP_PERSIST_OK" = true ]; then
|
|
pass_test "VPC stop/start IP persistence"
|
|
else
|
|
fail_test "VPC stop/start IP persistence"
|
|
fi
|
|
fi
|
|
else
|
|
fail_test "VPC IP verification"
|
|
fi
|
|
fi
|
|
|
|
# Cleanup VPC instances
|
|
echo ""
|
|
echo "Cleaning up VPC instances..."
|
|
terminate_and_wait "${VPC_INSTANCE_IDS[@]}" || true
|
|
else
|
|
if [ "$VPC_DENSITY_SKIP" = true ]; then
|
|
skip_test "VPC instance launch" "InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT"
|
|
else
|
|
fail_test "VPC instance launch"
|
|
fi
|
|
# Cleanup any partially-launched VPC instances
|
|
if [ ${#VPC_INSTANCE_IDS[@]} -gt 0 ]; then
|
|
echo " Cleaning up partially-launched VPC instances..."
|
|
terminate_and_wait "${VPC_INSTANCE_IDS[@]}" || true
|
|
fi
|
|
fi
|
|
|
|
# Cleanup subnet
|
|
echo " Deleting subnet $SUBNET_ID..."
|
|
$AWS_EC2 delete-subnet --subnet-id "$SUBNET_ID" 2>/dev/null || true
|
|
fi
|
|
|
|
# Cleanup VPC
|
|
echo " Deleting VPC $VPC_ID..."
|
|
$AWS_EC2 delete-vpc --vpc-id "$VPC_ID" 2>/dev/null || true
|
|
fi
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 11: Spread Placement Group + NAT Gateway (multi-node)
|
|
# ==========================================================================
|
|
echo "Phase 11: Spread Placement Group + NAT Gateway (multi-node)"
|
|
echo "========================================"
|
|
|
|
echo "Step 1: Creating VPC infrastructure..."
|
|
NATGW_VPC_ID=$($AWS_EC2 create-vpc --cidr-block 10.100.0.0/16 \
|
|
--query 'Vpc.VpcId' --output text)
|
|
echo " VPC: $NATGW_VPC_ID"
|
|
|
|
NATGW_PUB_SUBNET=$($AWS_EC2 create-subnet --vpc-id "$NATGW_VPC_ID" \
|
|
--cidr-block 10.100.1.0/24 --query 'Subnet.SubnetId' --output text)
|
|
echo " Public subnet: $NATGW_PUB_SUBNET"
|
|
|
|
NATGW_PRIV_SUBNET=$($AWS_EC2 create-subnet --vpc-id "$NATGW_VPC_ID" \
|
|
--cidr-block 10.100.2.0/24 --query 'Subnet.SubnetId' --output text)
|
|
echo " Private subnet: $NATGW_PRIV_SUBNET"
|
|
|
|
# Enable public IPs on public subnet
|
|
$AWS_EC2 modify-subnet-attribute --subnet-id "$NATGW_PUB_SUBNET" \
|
|
--map-public-ip-on-launch 2>/dev/null || true
|
|
|
|
NATGW_IGW_ID=$($AWS_EC2 create-internet-gateway \
|
|
--query 'InternetGateway.InternetGatewayId' --output text)
|
|
$AWS_EC2 attach-internet-gateway --vpc-id "$NATGW_VPC_ID" \
|
|
--internet-gateway-id "$NATGW_IGW_ID"
|
|
echo " IGW: $NATGW_IGW_ID (attached)"
|
|
|
|
# Add default route to IGW on main route table for public subnet
|
|
NATGW_MAIN_RTB=$($AWS_EC2 describe-route-tables \
|
|
--filters "Name=vpc-id,Values=$NATGW_VPC_ID" \
|
|
--query 'RouteTables[0].RouteTableId' --output text)
|
|
$AWS_EC2 create-route --route-table-id "$NATGW_MAIN_RTB" \
|
|
--destination-cidr-block 0.0.0.0/0 --gateway-id "$NATGW_IGW_ID" > /dev/null
|
|
echo " Main route table: 0.0.0.0/0 → $NATGW_IGW_ID"
|
|
|
|
# Security groups: SG enforcement is wired into OVN ACLs, so the default SG
|
|
# (egress-only) drops inbound SSH. Create explicit SGs for bastion and private
|
|
# instances so the bastion is reachable and the private VMs accept SSH from it.
|
|
NATGW_BASTION_SG=$($AWS_EC2 create-security-group \
|
|
--vpc-id "$NATGW_VPC_ID" --group-name natgw-bastion \
|
|
--description "Phase 11 bastion (SSH ingress from anywhere)" \
|
|
--query 'GroupId' --output text)
|
|
$AWS_EC2 authorize-security-group-ingress \
|
|
--group-id "$NATGW_BASTION_SG" --protocol tcp --port 22 --cidr 0.0.0.0/0 > /dev/null
|
|
echo " Bastion SG: $NATGW_BASTION_SG (tcp/22 from 0.0.0.0/0)"
|
|
|
|
NATGW_PRIV_SG=$($AWS_EC2 create-security-group \
|
|
--vpc-id "$NATGW_VPC_ID" --group-name natgw-private \
|
|
--description "Phase 11 private (SSH from bastion-sg, ICMP from VPC)" \
|
|
--query 'GroupId' --output text)
|
|
$AWS_EC2 authorize-security-group-ingress \
|
|
--group-id "$NATGW_PRIV_SG" \
|
|
--ip-permissions "IpProtocol=tcp,FromPort=22,ToPort=22,UserIdGroupPairs=[{GroupId=$NATGW_BASTION_SG}]" > /dev/null
|
|
$AWS_EC2 authorize-security-group-ingress \
|
|
--group-id "$NATGW_PRIV_SG" --protocol icmp --port -1 --cidr 10.100.0.0/16 > /dev/null
|
|
echo " Private SG: $NATGW_PRIV_SG (tcp/22 from bastion-sg, icmp from VPC)"
|
|
|
|
echo ""
|
|
echo "Step 2: Launching bastion in public subnet..."
|
|
set +e
|
|
BASTION_RUN_OUT=$($AWS_EC2 run-instances \
|
|
--image-id "$AMI_ID" --instance-type "$INSTANCE_TYPE" \
|
|
--subnet-id "$NATGW_PUB_SUBNET" --key-name spinifex-key \
|
|
--security-group-ids "$NATGW_BASTION_SG" \
|
|
--count 1 --query 'Instances[0].InstanceId' --output text 2>&1)
|
|
BASTION_RUN_RC=$?
|
|
set -e
|
|
NATGW_DENSITY_SKIP=false
|
|
if [ $BASTION_RUN_RC -ne 0 ]; then
|
|
if is_density_skip "$BASTION_RUN_OUT"; then
|
|
skip_test "NAT GW bastion launch" "InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT"
|
|
NATGW_DENSITY_SKIP=true
|
|
NATGW_BASTION_ID=""
|
|
else
|
|
echo " ERROR: bastion run-instances failed: $BASTION_RUN_OUT"
|
|
fail_test "NAT GW bastion launch"
|
|
NATGW_BASTION_ID=""
|
|
fi
|
|
else
|
|
NATGW_BASTION_ID="$BASTION_RUN_OUT"
|
|
fi
|
|
echo " Bastion: $NATGW_BASTION_ID"
|
|
if [ -z "$NATGW_BASTION_ID" ]; then
|
|
# Density-skip or hard fail at bastion launch: roll back the Phase 11 VPC
|
|
# plumbing and skip the rest of the phase.
|
|
$AWS_EC2 delete-security-group --group-id "$NATGW_PRIV_SG" 2>/dev/null || true
|
|
$AWS_EC2 delete-security-group --group-id "$NATGW_BASTION_SG" 2>/dev/null || true
|
|
$AWS_EC2 delete-subnet --subnet-id "$NATGW_PRIV_SUBNET" 2>/dev/null || true
|
|
$AWS_EC2 delete-subnet --subnet-id "$NATGW_PUB_SUBNET" 2>/dev/null || true
|
|
$AWS_EC2 detach-internet-gateway --vpc-id "$NATGW_VPC_ID" \
|
|
--internet-gateway-id "$NATGW_IGW_ID" 2>/dev/null || true
|
|
$AWS_EC2 delete-internet-gateway --internet-gateway-id "$NATGW_IGW_ID" 2>/dev/null || true
|
|
$AWS_EC2 delete-vpc --vpc-id "$NATGW_VPC_ID" 2>/dev/null || true
|
|
echo " Phase 11 short-circuited after bastion launch failure"
|
|
else
|
|
|
|
wait_for_instance_state "$NATGW_BASTION_ID" "running" 120
|
|
|
|
NATGW_BASTION_PUB_IP=$($AWS_EC2 describe-instances \
|
|
--instance-ids "$NATGW_BASTION_ID" \
|
|
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text)
|
|
echo " Bastion public IP: $NATGW_BASTION_PUB_IP"
|
|
|
|
# Wait for bastion SSH
|
|
echo " Waiting for bastion SSH..."
|
|
BASTION_OK=false
|
|
for attempt in $(seq 1 30); do
|
|
if ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
|
-o LogLevel=ERROR -o ConnectTimeout=5 -o BatchMode=yes \
|
|
-i "$HOME/.ssh/spinifex-key" "ubuntu@$NATGW_BASTION_PUB_IP" "true" 2>/dev/null; then
|
|
BASTION_OK=true
|
|
break
|
|
fi
|
|
sleep 5
|
|
done
|
|
if [ "$BASTION_OK" != true ]; then
|
|
echo " FAIL: Bastion SSH not reachable after 150s"
|
|
fail_test "NAT GW bastion SSH"
|
|
# Best-effort cleanup so the failure doesn't leak the bastion + SGs + VPC.
|
|
terminate_and_wait "$NATGW_BASTION_ID" 2>/dev/null || true
|
|
$AWS_EC2 delete-security-group --group-id "$NATGW_PRIV_SG" 2>/dev/null || true
|
|
$AWS_EC2 delete-security-group --group-id "$NATGW_BASTION_SG" 2>/dev/null || true
|
|
$AWS_EC2 delete-subnet --subnet-id "$NATGW_PRIV_SUBNET" 2>/dev/null || true
|
|
$AWS_EC2 delete-subnet --subnet-id "$NATGW_PUB_SUBNET" 2>/dev/null || true
|
|
$AWS_EC2 detach-internet-gateway --vpc-id "$NATGW_VPC_ID" \
|
|
--internet-gateway-id "$NATGW_IGW_ID" 2>/dev/null || true
|
|
$AWS_EC2 delete-internet-gateway --internet-gateway-id "$NATGW_IGW_ID" 2>/dev/null || true
|
|
$AWS_EC2 delete-vpc --vpc-id "$NATGW_VPC_ID" 2>/dev/null || true
|
|
else
|
|
echo " PASS: Bastion SSH ready"
|
|
|
|
# Copy SSH key to bastion for private instance hops
|
|
scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
|
|
-i "$HOME/.ssh/spinifex-key" "$HOME/.ssh/spinifex-key" \
|
|
"ubuntu@$NATGW_BASTION_PUB_IP:/tmp/key.pem" 2>/dev/null
|
|
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \
|
|
-i "$HOME/.ssh/spinifex-key" "ubuntu@$NATGW_BASTION_PUB_IP" "chmod 600 /tmp/key.pem" 2>/dev/null
|
|
|
|
# Bastion SSH helper
|
|
bastion_ssh() {
|
|
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
|
-o LogLevel=ERROR -o BatchMode=yes -o ConnectTimeout=10 \
|
|
-i "$HOME/.ssh/spinifex-key" "ubuntu@$NATGW_BASTION_PUB_IP" "$@"
|
|
}
|
|
priv_hop_cmd() {
|
|
local priv_ip="$1"; shift
|
|
echo "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=10 -o BatchMode=yes -i /tmp/key.pem ubuntu@${priv_ip} '$*'"
|
|
}
|
|
|
|
echo ""
|
|
echo "Step 3: Creating spread placement group + launching $NODE_COUNT private VMs..."
|
|
$AWS_EC2 create-placement-group --group-name nat-spread --strategy spread > /dev/null
|
|
echo " Placement group: nat-spread (spread)"
|
|
|
|
# Launch NODE_COUNT instances in private subnet with spread placement
|
|
set +e
|
|
NATGW_PRIV_OUTPUT=$($AWS_EC2 run-instances \
|
|
--image-id "$AMI_ID" --instance-type "$INSTANCE_TYPE" \
|
|
--subnet-id "$NATGW_PRIV_SUBNET" --key-name spinifex-key \
|
|
--security-group-ids "$NATGW_PRIV_SG" \
|
|
--count "$NODE_COUNT" --placement "GroupName=nat-spread" --output json 2>&1)
|
|
NATGW_PRIV_RC=$?
|
|
set -e
|
|
NATGW_PRIV_IDS=()
|
|
NATGW_PRIV_DENSITY_SKIP=false
|
|
if [ $NATGW_PRIV_RC -ne 0 ]; then
|
|
if is_density_skip "$NATGW_PRIV_OUTPUT"; then
|
|
skip_test "Spread placement private launch" "InsufficientInstanceCapacity on NODE_COUNT=$NODE_COUNT"
|
|
NATGW_PRIV_DENSITY_SKIP=true
|
|
else
|
|
echo " ERROR: spread run-instances failed: $NATGW_PRIV_OUTPUT"
|
|
fail_test "Spread placement private launch"
|
|
fi
|
|
else
|
|
for i in $(seq 0 $((NODE_COUNT - 1))); do
|
|
NATGW_PRIV_IDS+=("$(echo "$NATGW_PRIV_OUTPUT" | jq -r ".Instances[$i].InstanceId")")
|
|
done
|
|
fi
|
|
echo " Private instances: ${NATGW_PRIV_IDS[*]:-(none)}"
|
|
|
|
# Wait for all running
|
|
ALL_RUNNING=true
|
|
if [ ${#NATGW_PRIV_IDS[@]} -eq 0 ]; then
|
|
# Nothing was launched (density skip or other failure); short-circuit
|
|
# the spread/NAT GW assertions but keep cleanup.
|
|
ALL_RUNNING=false
|
|
else
|
|
for inst_id in "${NATGW_PRIV_IDS[@]}"; do
|
|
if ! wait_for_instance_state "$inst_id" "running" 120; then
|
|
ALL_RUNNING=false
|
|
fi
|
|
done
|
|
fi
|
|
|
|
if [ "$ALL_RUNNING" = true ]; then
|
|
echo ""
|
|
echo "Step 4: Validating spread placement across nodes..."
|
|
|
|
# Use spx get vms to check node assignment
|
|
SPX_VMS=$($SPINIFEX_BIN get vms --timeout 5s 2>/dev/null)
|
|
echo "$SPX_VMS"
|
|
|
|
SPREAD_NODES=()
|
|
for inst_id in "${NATGW_PRIV_IDS[@]}"; do
|
|
# spx get vms is pipe-delimited, with ANSI colour codes around each
|
|
# cell: INSTANCE | STATUS | HEALTH | CRASHES | TYPE | VCPU | MEM |
|
|
# NODE | IP | AGE. NODE is field 8; reading field 6 yields VCPU.
|
|
NODE_NAME=$(echo "$SPX_VMS" | grep "$inst_id" | sed -e 's/\x1b\[[0-9;]*m//g' | awk -F'|' '{gsub(/^[ \t]+|[ \t]+$/, "", $8); print $8}' || echo "unknown")
|
|
if [ -z "$NODE_NAME" ] || [ "$NODE_NAME" = "unknown" ]; then
|
|
NODE_NAME=$(find_instance_node "$inst_id" || echo "unknown")
|
|
fi
|
|
SPREAD_NODES+=("$NODE_NAME")
|
|
echo " $inst_id → $NODE_NAME"
|
|
done
|
|
|
|
# Count unique nodes
|
|
UNIQUE_NODES=$(printf '%s\n' "${SPREAD_NODES[@]}" | sort -u | wc -l)
|
|
echo " Unique hosting nodes: $UNIQUE_NODES / ${#SPREAD_NODES[@]}"
|
|
if [ "$UNIQUE_NODES" -ge "$NODE_COUNT" ]; then
|
|
pass_test "Spread placement ($NODE_COUNT instances on $NODE_COUNT nodes)"
|
|
elif [ "$UNIQUE_NODES" -ge "$((NODE_COUNT - 1))" ]; then
|
|
echo " WARN: Only $UNIQUE_NODES unique nodes (expected $NODE_COUNT) — spread best-effort"
|
|
pass_test "Spread placement (best-effort: $UNIQUE_NODES nodes)"
|
|
else
|
|
fail_test "Spread placement ($UNIQUE_NODES unique nodes, expected $NODE_COUNT)"
|
|
fi
|
|
|
|
# Get private IPs for SSH hop
|
|
NATGW_PRIV_PRIVATE_IPS=()
|
|
for inst_id in "${NATGW_PRIV_IDS[@]}"; do
|
|
PRIV_IP=$($AWS_EC2 describe-instances --instance-ids "$inst_id" \
|
|
--query 'Reservations[0].Instances[0].PrivateIpAddress' --output text)
|
|
NATGW_PRIV_PRIVATE_IPS+=("$PRIV_IP")
|
|
done
|
|
echo " Private IPs: ${NATGW_PRIV_PRIVATE_IPS[*]}"
|
|
|
|
# Wait for cloud-init on private instances via bastion
|
|
echo ""
|
|
echo " Waiting for private instance SSH via bastion..."
|
|
PRIV_SSH_READY=true
|
|
for idx in "${!NATGW_PRIV_IDS[@]}"; do
|
|
priv_ip="${NATGW_PRIV_PRIVATE_IPS[$idx]}"
|
|
inst_id="${NATGW_PRIV_IDS[$idx]}"
|
|
SSH_OK=false
|
|
for attempt in $(seq 1 30); do
|
|
if bastion_ssh "$(priv_hop_cmd "$priv_ip" hostname)" 2>/dev/null; then
|
|
SSH_OK=true
|
|
break
|
|
fi
|
|
sleep 5
|
|
done
|
|
if [ "$SSH_OK" = true ]; then
|
|
echo " $inst_id ($priv_ip): SSH ready"
|
|
else
|
|
echo " $inst_id ($priv_ip): SSH FAILED after 150s"
|
|
PRIV_SSH_READY=false
|
|
fi
|
|
done
|
|
|
|
if [ "$PRIV_SSH_READY" = true ]; then
|
|
echo ""
|
|
echo "Step 5: Verify NO internet (pre-NAT)..."
|
|
PRE_NAT_OK=true
|
|
for idx in "${!NATGW_PRIV_IDS[@]}"; do
|
|
priv_ip="${NATGW_PRIV_PRIVATE_IPS[$idx]}"
|
|
inst_id="${NATGW_PRIV_IDS[$idx]}"
|
|
if bastion_ssh "$(priv_hop_cmd "$priv_ip" 'ping -c 1 -W 3 8.8.8.8')" 2>/dev/null; then
|
|
echo " FAIL: $inst_id can reach internet WITHOUT NAT GW"
|
|
PRE_NAT_OK=false
|
|
else
|
|
echo " PASS: $inst_id has no internet (expected)"
|
|
fi
|
|
done
|
|
|
|
if [ "$PRE_NAT_OK" = true ]; then
|
|
pass_test "Private instances no internet (pre-NAT)"
|
|
|
|
echo ""
|
|
echo "Step 6: Creating NAT Gateway..."
|
|
NATGW_EIP_OUTPUT=$($AWS_EC2 allocate-address --domain vpc --output json)
|
|
NATGW_ALLOC_ID=$(echo "$NATGW_EIP_OUTPUT" | jq -r '.AllocationId')
|
|
NATGW_PUB_IP=$(echo "$NATGW_EIP_OUTPUT" | jq -r '.PublicIp')
|
|
echo " Allocated EIP: $NATGW_PUB_IP ($NATGW_ALLOC_ID)"
|
|
|
|
NATGW_OUTPUT=$($AWS_EC2 create-nat-gateway \
|
|
--subnet-id "$NATGW_PUB_SUBNET" \
|
|
--allocation-id "$NATGW_ALLOC_ID" --output json)
|
|
NATGW_ID=$(echo "$NATGW_OUTPUT" | jq -r '.NatGateway.NatGatewayId')
|
|
echo " NAT Gateway: $NATGW_ID"
|
|
|
|
# Create private route table + NAT GW route
|
|
NATGW_PRIV_RTB=$($AWS_EC2 create-route-table --vpc-id "$NATGW_VPC_ID" \
|
|
--query 'RouteTable.RouteTableId' --output text)
|
|
NATGW_RTB_ASSOC=$($AWS_EC2 associate-route-table \
|
|
--route-table-id "$NATGW_PRIV_RTB" \
|
|
--subnet-id "$NATGW_PRIV_SUBNET" \
|
|
--query 'AssociationId' --output text)
|
|
$AWS_EC2 create-route --route-table-id "$NATGW_PRIV_RTB" \
|
|
--destination-cidr-block 0.0.0.0/0 \
|
|
--nat-gateway-id "$NATGW_ID" > /dev/null
|
|
echo " Route: 0.0.0.0/0 → $NATGW_ID (rtb: $NATGW_PRIV_RTB)"
|
|
|
|
# Poll for OVN SNAT rule
|
|
SNAT_FOUND=false
|
|
for attempt in $(seq 1 30); do
|
|
NAT_RULES=$(sudo ovn-nbctl --no-leader-only lr-nat-list "vpc-${NATGW_VPC_ID}" 2>/dev/null || echo "")
|
|
if echo "$NAT_RULES" | grep -q "snat.*${NATGW_PUB_IP}"; then
|
|
SNAT_FOUND=true
|
|
echo " PASS: OVN SNAT rule created (after ${attempt}s)"
|
|
break
|
|
fi
|
|
sleep 1
|
|
done
|
|
if [ "$SNAT_FOUND" = false ]; then
|
|
echo " WARN: OVN SNAT rule not found after 30s"
|
|
fi
|
|
|
|
echo ""
|
|
echo "Step 7: Verify internet via NAT Gateway (all $NODE_COUNT nodes)..."
|
|
NAT_GW_ALL_OK=true
|
|
for idx in "${!NATGW_PRIV_IDS[@]}"; do
|
|
priv_ip="${NATGW_PRIV_PRIVATE_IPS[$idx]}"
|
|
inst_id="${NATGW_PRIV_IDS[$idx]}"
|
|
node="${SPREAD_NODES[$idx]}"
|
|
CONN_OK=false
|
|
for attempt in $(seq 1 10); do
|
|
if bastion_ssh "$(priv_hop_cmd "$priv_ip" 'ping -c 2 -W 3 8.8.8.8')" 2>/dev/null; then
|
|
CONN_OK=true
|
|
echo " PASS: $inst_id ($node) → internet via NAT GW (attempt $attempt)"
|
|
break
|
|
fi
|
|
sleep 5
|
|
done
|
|
if [ "$CONN_OK" = false ]; then
|
|
echo " FAIL: $inst_id ($node) cannot reach internet via NAT GW"
|
|
NAT_GW_ALL_OK=false
|
|
fi
|
|
done
|
|
|
|
if [ "$NAT_GW_ALL_OK" = true ]; then
|
|
pass_test "NAT Gateway multi-node connectivity (all $NODE_COUNT nodes)"
|
|
else
|
|
fail_test "NAT Gateway multi-node connectivity"
|
|
echo " Dumping OVN NAT rules for debugging:"
|
|
sudo ovn-nbctl --no-leader-only lr-nat-list "vpc-${NATGW_VPC_ID}" 2>/dev/null || true
|
|
fi
|
|
|
|
# Cleanup NAT GW resources
|
|
echo ""
|
|
echo "Step 8: Cleanup NAT Gateway..."
|
|
$AWS_EC2 delete-nat-gateway --nat-gateway-id "$NATGW_ID" > /dev/null 2>&1 || true
|
|
$AWS_EC2 disassociate-route-table --association-id "$NATGW_RTB_ASSOC" 2>/dev/null || true
|
|
$AWS_EC2 delete-route --route-table-id "$NATGW_PRIV_RTB" \
|
|
--destination-cidr-block 0.0.0.0/0 2>/dev/null || true
|
|
$AWS_EC2 delete-route-table --route-table-id "$NATGW_PRIV_RTB" 2>/dev/null || true
|
|
$AWS_EC2 release-address --allocation-id "$NATGW_ALLOC_ID" 2>/dev/null || true
|
|
echo " NAT GW resources cleaned up"
|
|
else
|
|
fail_test "Private instances no internet (pre-NAT)"
|
|
fi
|
|
else
|
|
fail_test "Private instance SSH via bastion"
|
|
fi
|
|
else
|
|
if [ "$NATGW_PRIV_DENSITY_SKIP" = true ]; then
|
|
: # already recorded via skip_test "Spread placement private launch"
|
|
else
|
|
fail_test "Private instance launch (spread)"
|
|
fi
|
|
fi
|
|
|
|
# Cleanup Phase 11 instances
|
|
echo ""
|
|
echo "Cleaning up Phase 11 instances..."
|
|
terminate_and_wait "$NATGW_BASTION_ID" "${NATGW_PRIV_IDS[@]}" 2>/dev/null || true
|
|
$AWS_EC2 delete-placement-group --group-name nat-spread 2>/dev/null || true
|
|
|
|
# Cleanup VPC resources
|
|
$AWS_EC2 delete-security-group --group-id "$NATGW_PRIV_SG" 2>/dev/null || true
|
|
$AWS_EC2 delete-security-group --group-id "$NATGW_BASTION_SG" 2>/dev/null || true
|
|
$AWS_EC2 delete-subnet --subnet-id "$NATGW_PRIV_SUBNET" 2>/dev/null || true
|
|
$AWS_EC2 delete-subnet --subnet-id "$NATGW_PUB_SUBNET" 2>/dev/null || true
|
|
$AWS_EC2 detach-internet-gateway --vpc-id "$NATGW_VPC_ID" \
|
|
--internet-gateway-id "$NATGW_IGW_ID" 2>/dev/null || true
|
|
$AWS_EC2 delete-internet-gateway --internet-gateway-id "$NATGW_IGW_ID" 2>/dev/null || true
|
|
# Delete non-main route tables
|
|
for rtb in $($AWS_EC2 describe-route-tables \
|
|
--filters "Name=vpc-id,Values=$NATGW_VPC_ID" \
|
|
--query 'RouteTables[?Associations[0].Main!=`true`].RouteTableId' --output text 2>/dev/null); do
|
|
$AWS_EC2 delete-route-table --route-table-id "$rtb" 2>/dev/null || true
|
|
done
|
|
$AWS_EC2 delete-vpc --vpc-id "$NATGW_VPC_ID" 2>/dev/null || true
|
|
echo " Phase 11 cleanup complete"
|
|
fi
|
|
|
|
fi # NATGW_BASTION_ID non-empty guard
|
|
|
|
echo ""
|
|
|
|
# ==========================================================================
|
|
# Phase 12: Cleanup
|
|
# ==========================================================================
|
|
echo "Phase 12: Cleanup"
|
|
echo "========================================"
|
|
|
|
echo "Terminating all instances..."
|
|
if terminate_and_wait "${INSTANCE_IDS[@]}"; then
|
|
pass_test "Instance termination"
|
|
else
|
|
fail_test "Instance termination"
|
|
fi
|
|
|
|
# ==========================================================================
|
|
# Summary
|
|
# ==========================================================================
|
|
echo ""
|
|
echo "========================================"
|
|
echo "Real Multi-Node E2E Test Summary"
|
|
echo "========================================"
|
|
echo " Passed: $TESTS_PASSED"
|
|
echo " Failed: $TESTS_FAILED"
|
|
echo " Skipped: $TESTS_SKIPPED"
|
|
if [ ${#FAILED_TESTS[@]} -gt 0 ]; then
|
|
echo " Failed tests:"
|
|
for t in "${FAILED_TESTS[@]}"; do
|
|
echo " - $t"
|
|
done
|
|
fi
|
|
if [ ${#SKIPPED_TESTS[@]} -gt 0 ]; then
|
|
echo " Skipped tests:"
|
|
for t in "${SKIPPED_TESTS[@]}"; do
|
|
echo " - $t"
|
|
done
|
|
fi
|
|
echo "========================================"
|
|
|
|
if [ $TESTS_FAILED -gt 0 ]; then
|
|
echo "SOME TESTS FAILED"
|
|
exit 1
|
|
fi
|
|
|
|
if [ $TESTS_SKIPPED -gt 0 ]; then
|
|
echo "All tests passed (with $TESTS_SKIPPED skipped — undersized cluster)"
|
|
else
|
|
echo "All tests passed!"
|
|
fi
|
|
exit 0
|