mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
feat: decouple the EC2 control plane from the viperblock engine behind an EBSProvider contract (#770)
* feat: introduce the EBSProvider interface and control-plane metadata store
Phase 1 of decoupling the control plane from viperblock. Adds a
provider-neutral block-storage contract that spinifex owns end to end.
ebsprovider defines EBSProvider, whose operation set and idempotency rules
follow the CSI controller service, with NATS request/reply as the transport
rather than CSI's gRPC. Every request and response carries an explicit schema
version so wire skew fails loudly at the boundary instead of silently decoding
a changed payload. The package deliberately exposes no viperblock types:
provider-specific state is an opaque handle. Ships with a NATS client against
the ebs.provider.v1.* subjects and an in-memory fake for tests.
ebsmetadata gives spinifex its own volume and AMI documents under a spinifex
key prefix, so the control plane can answer DescribeVolumes and DescribeImages
without reading the storage engine's persisted state.
The EC2 volume and snapshot handlers gain provider-backed paths for create,
describe, modify, attach-state and delete. Provider injection is optional for
now, so the existing engine path stays the default until the viperblockd side
serves the new subjects and the metadata migration lands.
vm/lifecycle.go no longer imports viperblock; FindFreePort was the only symbol
it used and is now a local helper.
* feat: serve the EBS provider contract from viperblockd
Gives the ebsprovider NATS client a server. Volume create, describe, expand
and delete, plus snapshot create and delete, are now handled by the daemon
that owns BaseDir and the mounted-volume registry, which is where viperblock
engine construction belongs.
Every handler prefers an already-mounted engine over opening a second one on
the same volume, and expand refuses outright while the volume is mounted.
Opening a second engine on a live volume is the double-writer hazard this
decoupling exists to remove, so the boundary enforces it rather than trusting
callers. Requests carrying an unrecognised schema version are rejected at the
boundary instead of being decoded on a best-effort basis.
Snapshot creation replies immediately with an operation ID and completes
asynchronously on a per-snapshot subject, since a large volume will outrun the
request timeout. Draining the volume stays with the caller: it coordinates
with the guest and is not something the storage boundary can reach.
The snapshot create subject gains an explicit create verb. Addressing it as
ebs.provider.v1.snapshot.<volumeID> collided with the snapshot delete subject
and left the wildcard unsubscribable.
Publish and unpublish are not served yet; the mount and unmount logic they
would front is still inline in launchService and needs extracting first.
Also adds a conformance suite that runs one set of contract assertions against
both the in-memory fake and the NATS client, so each side is checked against
the contract rather than against the other's structs.
* feat: let an operator select the EBS provider
Adds an [ebs] provider key and wires the NATS provider into the EC2 volume,
snapshot, image and instance services at the daemon's composition root. Until
now nothing injected a provider, so the provider-backed code paths in those
services were unreachable.
The key is provider-neutral rather than nested under [viperblock], because it
names the boundary rather than one provider's settings; a second provider will
not need it renamed. An empty value resolves to "embedded", so deployed config
files that predate the key keep today's behaviour, and an unrecognised value
is a startup error rather than a silent fallback.
Startup probes the provider's capability verb and warns when nothing answers.
The daemon and viperblockd start independently, so an unanswered probe is not
fatal, but without it a misconfigured node fails much later as an unexplained
request timeout on the first EBS call.
"viperblockd" stays experimental: under it volume metadata lives in the
control plane's own document, and the leak reaper, GC and admin paths still
read viperblock's config.json, so they cannot see tenant or tag data for
volumes created that way.
* fix: make every volume-metadata consumer provider-aware
Several readers and writers of volume metadata were never given a provider
branch, so under the viperblockd provider they silently used the wrong store.
Each of these is a correctness bug rather than a missing feature, because the
control plane had already stopped writing the document they read.
Modifications had nowhere to live: ModifyVolume built its result in memory and
dropped it, and DescribeVolumesModifications read it back out of viperblock's
config.json, so a describe straight after a successful modify returned nothing.
Volume metadata now carries its own modification record.
The leak reaper read attachment and tags from config.json and wrote orphan
markers to tags.json, neither of which exists for a provider-managed volume, so
a leaked volume was invisible to it and any mark it did write could not be read
back. Tag mutations had the same split: a tag applied after creation never
reached the document DescribeVolumes reads.
Snapshot deletion could not see a provider-managed clone, because a clone
records its source only in the control-plane document. That let a delete strip
chunks a live clone still reads, so it is fixed here rather than left for the
migration.
Volume enumeration and DescribeVolumeStatus were likewise scanning object
prefixes that a provider-managed volume does not create.
Also adds an encryption flag to the volume document, which DescribeVolumes had
been reporting as false unconditionally and snapshots could not carry at all,
and a list operation for AMIs so the image path can follow later.
The legacy path is unchanged; its invariant tests pass unmodified.
* feat: backfill ebsmetadata documents from legacy volume and AMI state
DescribeVolumes under the viperblockd provider enumerates only ebsmetadata
documents, so a volume created before the switch had no document and was
invisible rather than merely slow to read. A per-object read fallback cannot
fix enumeration, so both a backfill and a fallback are needed.
Add an object-store migration kind to the migrate framework. Predastore has no
conditional write, so an object-based version stamp would be last-writer-wins
across nodes; RunObject stamps progress in a caller-supplied JetStream KV
bucket instead, exactly as RunKV stamps a bucket's own version. The stamp is a
read-then-write rather than a compare-and-swap, so correctness rests on every
ObjectMigration.Run being safe to re-execute.
The backfill itself lives in migrate/ebsmetadatabackfill rather than beside the
numbered config migrations: it needs viperblock's VBState schema to decode the
legacy config.json, and migrate is imported by every handler that stamps a KV
bucket version. It converts what it can reach and skips unreadable objects with
a log, so one corrupt volume cannot hide the rest of the fleet.
ebsmetadata gains the legacy read fallback for the window before the migration
has run everywhere. It stays free of viperblock by taking the decode as an
injected function value, wired at the composition root, and the fallback is
retired by leaving it unset. A fallback read never writes the converted value
back, which would race the migration.
Prefix enumeration pages to exhaustion. Stopping at the first page would let
the backfill stamp itself complete having converted only the volumes it
happened to see, reintroducing the invisible-volume failure it exists to
prevent.
The migration runs from configureEBSProvider, so only when the provider is
selected; backfilling while the embedded engine is still authoritative would
produce documents that immediately go stale. Failure is fatal rather than
advisory, unlike the capability probe beside it. This makes the switch to
viperblockd one-way, which is now recorded on the config field.
* feat: serve volume publish and unpublish from viperblockd
PublishVolume and UnpublishVolume were the last unserved verbs on the provider
contract, and they are the ones that attach a volume to a VM. Until they are
served the provider path cannot boot an instance, so provider = "viperblockd"
cannot be verified end to end.
The mount and unmount logic was two large anonymous handler bodies inline in
launchService. Rather than write a second mount path, extract each into a named
function and have both the legacy ebs.mount/ebs.unmount handlers and the new
provider handlers call it. Two divergent implementations of nbdkit spawning and
the seal decision would be far worse than the inline bodies. The extraction
relocates code without rewriting it; response publishing stays in each handler
so the provider never publishes to ebs.mount.response.
Publish is idempotent: a volume this node already has mounted returns its
existing attachment rather than starting a second nbdkit against it, which is
the double-writer hazard the boundary exists to prevent. Unpublish preserves
the invariant that a failed seal leaves the volume mounted so a retry can
re-attempt it, and reports an error rather than success.
Both verbs are node-addressed, so they subscribe directly rather than through
the spinifex-workers queue group, and are registered only when NodeName is set.
There is no queue-group fallback: a subject that already names one node has
nothing sensible to fall back to.
Read-only attachment is refused rather than silently granted read-write, since
nbdkit is started read-write or not at all.
Also fixes a latent panic carried by the extracted mount path: viperblock.New
returns a nil VB alongside its error, and SetCacheSize was called on it before
the error was checked, so a failing New would have taken down the handler.
* feat: read and write AMI metadata through ebsmetadata
The backfill migration wrote ebsmetadata AMI documents that nothing read: every
AMI path still went through viperblock.AMIMetadata inside a VBState wrapper at
{imageID}/config.json. Give the image service provider branches for the AMI
read, write and enumerate paths, with enumeration going through ListAMIs rather
than an object-prefix scan, as DescribeVolumes already does.
AMIMetaLoader declared GetAMIConfig returning viperblock.AMIMetadata, putting a
viperblock type on an interface boundary in the instance package. It now returns
ebsmetadata.AMI on both paths, the embedded one converting before it returns, so
the boundary names no viperblock type whichever provider is selected.
The legacy-to-document mapping had accumulated three copies. Consolidate them in
ebsmetadata/vblegacy, a sibling package rather than a member so ebsmetadata
itself stays free of viperblock, and cover it with a reflective test that fails
when a field is added to either schema without being wired through. Sixteen
fields copied by hand in three places would have dropped one silently.
The admin image paths read whichever representation exists and write back to
that same one, so a provider-managed AMI never gains a spurious config.json.
FindAMIDependents now unions ebsmetadata volumes with the legacy prefix scan:
missing a dependent there deletes an AMI a live volume still needs, the same
class of bug as the snapshot clone path fixed earlier on this branch.
That union is built by hand rather than by wiring Store.ListVolumes' fallback,
which swallows per-ID read errors. This path deliberately fails closed on a
transport error so a delete cannot proceed on a partial picture, and the
fallback would have quietly turned that into failing open.
The CLI image import now also writes the document, best-effort, so an AMI
imported after the migration is not left reachable only through the fallback.
* fix(ec2): resolve volume config through ebsmetadata when a provider is configured
A provider-created volume has no vol-*/config.json — its metadata lives in
the ebsmetadata document. getBaseVolumeConfigAndEncryption read the legacy
key unconditionally, so AttachVolume reported a volume that DescribeVolumes
had just listed as available as InvalidVolume.NotFound.
Routing through ebsmetadata.Store also puts this path behind the legacy
fallback, so a volume created before the provider switch stays attachable.
The NoSuchKey remap now names the key it missed; its silence made the
original failure invisible in the daemon logs on every node.
* test(bench): add an EC2 control-plane latency benchmark
Measures the EBS control-plane operations either side of the provider
boundary, so the cost of the NATS hop can be stated rather than guessed.
Reports median and p95 with a bootstrap CI on the median difference, and
labels a delta indicative unless that CI excludes zero.
* fix(ec2): drain the volume before a provider snapshot
The legacy CreateSnapshot branch drains first so the snapshot reads a
current checkpoint; the provider branch did not, and viperblockd documents
the drain as the caller's responsibility. Both sides deferred to the other,
so an attached volume snapshotted stale data with no error raised.
DrainVolume and volumeAttachment now take the fallback state and instance
as strings rather than a viperblock.VolumeMetadata, removing a viperblock
type from an exported signature.
* feat(ec2): route the instance root volume through the EBS provider
RunInstances built its own viperblock engine to clone the AMI snapshot into
the root volume, which made every launch a second writer against a volume
viperblockd already serves. The launch now calls the provider's CreateVolume
with the AMI's snapshot and source volume, and records the result in
ebsmetadata so DescribeVolumes and attach can see it.
A snapshot clone needs both the snapshot ID and its source volume ID to
resolve base blocks, so CreateVolume in the volume handler now sends
SourceVolumeID alongside SourceSnapshotID; without it the daemon rejected
every create-volume --snapshot-id outright.
The gp3 IOPS and throughput constants move to spinifex/types so the instance
and volume handlers share one definition rather than duplicating literals.
* feat(ebsprovider): add a bounded volume seed to CreateVolume
The EFI variable store must be initialised from the launching node's firmware
VARS template, byte for byte. Passing a host path would not work: CreateVolume
is queue-subscribed, so any node can service it, and a foreign node's OVMF
package would bind the guest's pflash region to the wrong firmware build.
CreateVolumeRequest carries the bytes instead, as SeedData, and viperblockd
writes them at offset 0 in the engine it already builds. The seed applies only
when the create actually allocates the volume, so a relaunch never overwrites
a BootOrder the guest has since written.
JSON base64-encodes a []byte, inflating the payload by 4/3, so MaxSeedBytes
bounds the seed well inside the NATS max_payload the cluster runs with and
both ends reject an oversized seed with a clear error rather than letting the
publish fail. VolumeSeeding advertises the behaviour so a provider that cannot
seed says so instead of silently dropping the bytes.
* feat(ec2): route the EFI variable store through the EBS provider
prepareEFIVolume built a control-plane viperblock engine to seed the variable
store from the firmware VARS template, making every UEFI launch a second
writer against a volume viperblockd already serves. It now calls the
provider's CreateVolume with the template as seed bytes, so the store is
allocated and written by the daemon that owns the volume.
The template is still read on the launching node and shipped with the request.
CreateVolume is queue-subscribed, so any node can service it, and a VARS
region seeded from a foreign node's OVMF build would bind the guest's pflash
to the wrong firmware.
pflash refuses a VARS region that is not byte-exact, so the returned capacity
is checked against the template length; a provider that rounded up to a GiB
boundary would otherwise yield a guest that cannot boot.
The engine path stays behind a nil-provider guard until the legacy branch goes.
* feat(ec2): route image snapshots through the EBS provider
snapshotRunningVolume and snapshotStoppedVolume each built a control-plane
viperblock engine to take the snapshot backing a new AMI, making CreateImage
a second writer against a volume viperblockd already serves. Both now call
the provider's CreateSnapshot with the volume's provider handle.
The running path keeps its drain, which is what makes the snapshot current
rather than one predating a guest-triggered write. The stopped path needs no
drain, and the daemon already prefers a live mounted engine over opening a
second one, so it handles both cases itself.
getVolumeConfig read the legacy config.json, which a provider-created volume
does not have, so every CreateImage against one failed. It now reads the
ebsmetadata document under a provider, the same way attach does.
* fix(daemon): only the owning node creates an image for a stopped instance
handleEC2CreateImage is a fan-out. Ownership is established by vmMgr, which
only the node holding the VM passes, so a running instance is imaged once.
A stopped instance lives in the cluster-replicated KV instead, which every
node reads successfully, so all of them ran the full snapshot and AMI
creation. The caller saw the first reply and the rest leaked: on a three-node
cluster one create-image produced three AMIs and three snapshots.
A stopped instance records its owner in LastNode, so a node that is not the
owner now declines. Gather tallies error frames and keeps waiting, returning
only on a success, so declining cannot preempt the owner's reply.
An instance stopped by a build that never set LastNode elects an owner
through the KV's compare-and-swap, since having every node decline would make
create-image fail permanently for those records.
* perf(ec2): serve DescribeVolumes by ID from the volume documents
The provider branch called ListVolumes unconditionally and filtered in
memory, so asking for one volume read and decoded every volume document in
the bucket. The non-provider branch already had the equivalent fast path.
Explicit IDs now fetch their documents directly. The tenant check moves with
them: a document belonging to another account reports not-found rather than
being returned, matching what the enumeration filter did.
Measured against a memory object store, the fast path stays flat near 20us
regardless of bucket size while the enumeration grows linearly, reaching
4.6ms at 500 volumes.
* chore(lint): ban the viperblock import from the EC2 control plane
The control plane reaches block storage through ebsprovider now, but nothing
stopped a handler from importing the engine again. That is not hypothetical:
the instance handler first imported viperblock for a config struct alone, and
the engine construction followed thirteen days later.
depguard denies the import across handlers/, which matters because
viperblock.New starts a chunk uploader — every construction is a writer
against a volume viperblockd already serves. A types-only import is denied
too, since that is exactly how the last one started.
The four files still carrying the pre-provider engine path behind a
nil-provider guard are exempted by name. The exemption list is the remaining
work: it shrinks to nothing when that path goes.
* fix(ec2): refuse CopySnapshot instead of creating an unusable snapshot
CopySnapshot wrote a control-plane snapshot document under the new ID and
nothing else. The copy described correctly and CreateVolume against it
succeeded, so it looked like it worked, but the new ID had no viperblock
snapshot metadata behind it and the volume failed when the guest attached it.
Producing the metadata means re-sealing it under the destination ID. On an
encrypted volume it is AEAD-sealed with the snapshot ID in the AAD and the
sealing helpers are unexported, so spinifex cannot write it. Copying only the
unencrypted form would be dead code on any cluster with encryption enabled.
Refusing with UnsupportedOperation fails at the call the caller can act on
rather than at attach time. Source lookup and the ownership check still run
first, so a bad ID or another tenant's snapshot reports that instead.
* test(ec2): give handler test fixtures their own legacy config.json shape
Handler tests built the legacy config.json fixture by importing
viperblock's VBState/VolumeConfig/VolumeMetadata/AMIMetadata directly.
Mirror that shape in ebsmetadata/vblegacy, next to the production code
that already reads it (ebsmetadatabackfill), and switch the
fixture-only usages across the volume/snapshot/image handler tests to
it. A byte-for-byte equivalence test against the real viperblock types
guards the mirror from drifting out of sync.
Call sites that hand a viperblock type into putVolumeConfig,
mergeVolumeConfig, or buildVBConfig are untouched, since those
functions still name the real type in their signature.
* feat(ebsprovider): add CopySnapshot to the EBS provider contract
Adds a CopySnapshot operation to EBSProvider (types, ebs.provider.v1.snapshot.copy
NATS subject, MemoryProvider and NATSProvider implementations, conformance
coverage) and a viperblockd handler that duplicates a snapshot's metadata via
viperblock.CopySnapshotMeta on the source volume's VB.
* feat(ec2): serve CopySnapshot through the EBS provider
CopySnapshot has been refusing with UnsupportedOperation because a copy needs
its own snapshot metadata on the backend under the new ID, and on an encrypted
volume that metadata is AEAD-sealed with the snapshot ID in the AAD, so the
control plane cannot re-seal it. The provider contract now carries a copy
operation, so the storage engine does it.
The provider path generates a new snapshot ID, has the provider duplicate the
backend metadata under it, then writes the control-plane document. Rollback
mirrors CreateSnapshot: a failure after the copy deletes the provider snapshot
rather than leaving one nothing references. Ownership is still checked against
the source before anything is created.
The no-provider path keeps refusing. It has no way to produce the backend
metadata, and the copy it used to write described fine and then failed at
attach — a resource that fails as far as possible from the call that made it.
Adds MemoryProvider.Snapshot so a control-plane test can assert a snapshot
reached the provider rather than existing only as a control-plane document.
That assertion is what catches the old behaviour.
* refactor: remove the embedded EBS engine path from the EC2 control plane
viperblockd is now the only EBS provider. The pre-provider path that
constructed a viperblock engine in-process is deleted, and with it every
github.com/mulgadc/viperblock import under spinifex/handlers/ (tests
included). The depguard ec2-control-plane rule drops all four file
exemptions and the !$test carve-out, so the rule now reads simply as
"nothing under spinifex/handlers/** may import viperblock".
viperblock.VolumeConfig was also a data carrier on the provider path. It
is replaced by the spinifex-native ebsmetadata.Volume rather than the
vblegacy mirror: vblegacy mirrors a legacy on-disk shape whose only job is
decoding old objects, while ebsmetadata.Volume already carries every field
the call sites read, so the conversion step disappears instead of being
reimplemented against a legacy shape.
The embedded-only helpers (volumeConfigWrapper, putVolumeConfig,
mergeVolumeConfig, configIsEncrypted, putVolumeConfigViaDetached,
deleteS3Prefix, buildVBConfig, snapshotVolume, the ami-*/config.json
prefix scan) had exactly one production caller chain each, all through a
nil-provider branch, and are removed. The snapshot-dependency guard on
DeleteVolume is provider-agnostic and is kept, now running on the provider
path where it previously did not.
config: viperblockd is the default and an empty Provider resolves to it.
An explicit provider = "embedded" is rejected at config load rather than
silently upgraded, because starting anyway would migrate a node's volumes
to ebsmetadata one-way without the operator asking. The generated
spinifex.toml template emits viperblockd.
* fix(images): write the imported AMI document from the config the import wrote
ImportDiskImage creates the AMI's snapshot and records it on its own
*viperblock.VB, which viperblock.New allocates fresh, so the value lands only
in {imageID}/config.json. The import command was persisting the ebsmetadata
document from its pre-import manifest copy, whose SnapshotID is still empty.
Store.GetAMI only consults the legacy config.json fallback when the document
is missing, so that empty-SnapshotID document permanently shadowed the real
value and every launch from an imported AMI failed the zero-copy clone check
with ServerInternal.
Rebuild the document by reading back the config.json the import just wrote. A
missing or unreadable config is skipped rather than written from the stale
manifest: no document still resolves correctly through the legacy fallback.
* test(e2e): read NODE from spx get vms instead of VCPU in the spread check
spx get vms gained HEALTH and CRASHES columns, shifting NODE from field 6 to
field 8. The spread placement check still read field 6, so every instance
reported its vCPU count instead of its node, the unique-node count always
collapsed to 1, and the check failed on clusters that had spread correctly.
Read field 8 and strip the ANSI colour codes wrapping each cell.
* feat(viperblockd): add a --debug flag so viperblock debug logging is reachable
viperblockd hardcoded Info, and viperblock's VB.logger() falls back to
slog.Default() because vb.log is only ever set by SetDebug. That left the
engine's debug records, including the chunk uploader's start and disabled
lines emitted during New(), unreachable in a deployed environment.
Binds the namespaced viper key viperblock-debug and SPINIFEX_VIPERBLOCK_DEBUG
rather than the bare debug key, which several commands share. Also sets the
Config.Debug field, which existed but was set only by tests.
* fix(viperblockd): release the chunk uploader when openVolumeVB fails
viperblock.New starts a chunk uploader and a WAL syncer before the backend is
touched, so a failure in Backend.Init or loadStateWithRetry returned nil to the
caller while leaving both running with no handle to stop them. Repeated backend
failures on CopySnapshot, ExpandVolume or DescribeVolume accumulated uploaders
on a 30s ticker for the life of the process.
Adds invariant tests for both halves of the ownership rule: a mounted volume's
snapshot create and copy reuse the live engine and log zero volume opens, an
unmounted volume opens exactly one, and a failed open leaves no uploader behind.
The open counts are read from the same log record a production alarm watches.
* fix(ebsprovider): route requests to the node that owns the mount
Every viperblockd queue-subscribed the ebs.provider.v1.* verbs in one
spinifex-workers group, so NATS delivered each request to a random node. Only
the node holding the mount can reuse the live engine through findMountedVolume;
any other node constructed a second viperblock engine, with a live 30s chunk
uploader, against a volume nbdkit was serving on a different host. Measured on
a three-node cluster, this was roughly two snapshots in three.
Rather than resolve the owner, let the owner answer. A node subscribes
ebs.provider.v1.owner.<volumeID>.<verb> as a plain subscription when it mounts
a volume and drops it on unmount, mirroring the per-volume ebs.config
subscription. Clients try that subject first and fall back to the queue group
only on ErrNoResponders, which means nothing is mounted and opening an engine
is correct. A timeout deliberately does not fall back, since the owner may
still be running the operation.
Applies to snapshot create and copy, expand and describe, the four verbs that
reuse a mounted engine. Delete keeps queue dispatch.
* fix: rebuild viperblockd mounted-volume registry after a restart
Config.MountedVolumes is in-memory and rebuilt from nothing on start, so a
viperblockd restart while a VM keeps running leaves nbdkit serving a volume
the daemon has forgotten. Owner routing then infers from ErrNoResponders that
nothing is mounted and falls back to the queue group, which opens a second
viperblock engine, with a live chunk uploader, on another node. The same empty
registry also defeats handlePublishVolume's double-mount guard, so a republish
could start a second nbdkit.
Discover surviving nbdkit processes from /proc at startup and rebuild the
registry from them, following orphan_scan.go's precedent: match comm exactly,
parse the argv for volume=<id> and --unix/-p, and adopt a PID only when
nbdkit-vol-<volume>.pid names that same PID. Nothing is started, signalled or
killed; anything that fails corroboration is logged and skipped. Recovery runs
before the first subscription so no handler can race it.
Extract mountVolume's engine construction into constructMountedVB so mount and
recovery cannot drift, and release the WAL syncer viperblock.New starts when
construction fails after it.
* fix: corroborate recovered nbdkit processes from argv, not a pidfile
nbdkit is launched with -f (required for the Go plugin) and never writes the
--pidfile it is given, so no such file exists on any node; the daemon's own
/tmp is a PrivateTmp namespace besides. Corroborating against it rejected
every genuine mount, making recovery a no-op in production.
Corroborate against what the kernel actually has: the process must serve our
plugin, name this daemon's base dir, and still hold its socket. Adopt a volume
once, so two live nbdkits for it leave one registry entry and a warning rather
than two conflicting ones.
* feat: add a second EBS provider backed by qcow2 and qemu-nbd
The decoupling was justified by swappability that nothing demonstrated: the
conformance suite ran against MemoryProvider and against a NATSProvider wired
to a MemoryProvider, so both arms were the same implementation behind
different transports. A fake is not a second implementation.
Add three pieces. natsserve serves the ebs.provider.v1.* contract from any
EBSProvider, depending on nothing but the interface, and replaces the
test-only server twin so conformance now exercises production code. The
conformance suite moves to its own package and is exported so implementations
outside ebsprovider can run it, with no assertion changed. qemunbdd is the
second implementation: one qcow2 file per volume, qemu-nbd for publish, real
snapshot copies instead of chunk references, and every exec behind a runner
seam so its tests need no qemu.
Owner-subject routing is deliberately not served neutrally: it needs a
per-node mounted-volume registry the interface does not model, so it stays in
viperblockd and a neutral server's clients correctly fall back to the queue
group.
* test: run the conformance suite against the qcow2 provider
The suite passes unmodified against an implementation sharing no code with
viperblock, both in-process and through natsserve over the real wire
contract, which is the swappability claim the provider boundary was built to
support.
Two real defects it caught. ExpandVolume and GetVolume inspected the image
before consulting the publication map, so qemu-img could not open a volume
qemu-nbd held the write lock on and the refusal surfaced as an internal
error instead of ErrVolumeInUse; qemu-img info now runs with --force-share,
so describing a volume no longer depends on whether it is attached. Publish
also failed with qemu-nbd's raw exit status when the socket path exceeded the
108-byte AF_UNIX limit, and now reports that reason itself.
* feat: make the EBS provider selectable by config
Add the qemunbd selector alongside viperblockd, a spx service command that
serves the qcow2 provider over natsserve, and service registration. Config
validation accepts the new value everywhere viperblockd is accepted, while
embedded stays rejected by name.
The daemon logged a hard-coded viperblockd constant as the active provider
rather than the resolved one, in configureEBSProvider and in both of
probeEBSProvider's lines; all three now report what is actually configured.
Both daemons answer the same subjects, so exactly one may run on a cluster.
* test: add a live lifecycle check for the qcow2 provider
Drives create, publish, write, snapshot, restore and read-back entirely
through ebsprovider.NewNATSProvider against a real qemunbdd daemon on a
cluster node, so the client under test is the same shape the control plane
uses and never touches the provider type. Skipped unless SPINIFEX_QEMUNBD_LIVE
is set, so it stays out of preflight.
* fix: publish NBD URIs in the form standard clients accept
PublishedVolume.NBDURI carried nbd:unix:/path, which is QEMU's legacy
pre-URI filename syntax rather than an NBD URI. libnbd rejects it outright
with "URI must begin with 'nbd://'", so no NBD client but QEMU could ever
consume what the EBS provider boundary hands back.
Emit nbd+unix:///?socket=/path instead, the form libnbd, qemu and nbdkit
all accept. ParseNBDURI now takes both that and the TCP form, and still
accepts the legacy nbd:unix: syntax for state written by older daemons, so
a conformant third-party provider's URI is no longer rejected on the way
in either.
Verified against qemu 10.0.11: qemu-img and a real -drive option string
both open the URI form, and qemu reports both forms as nbd+unix://?socket=,
so it already treats the URI as canonical.
* test: validate published exports with libnbd's client tools
Every other assertion about the EBS provider goes through our own Go
client, so nothing has ever checked whether the export we publish is usable
by an NBD client that knows nothing about this codebase. That is the layer
neither the conformance suite nor a CSI-shaped contract can reach.
RunNBDClientSuite drives nbdinfo and nbdcopy against a published volume:
the URI connects, the export advertises its size and block-size
constraints, data written externally reads back byte for byte, a ReadOnly
publish sets the read-only transmission flag, and UnpublishVolume tears the
export down. It skips when libnbd-bin is absent rather than passing
vacuously.
Wired to qemunbdd, where all five failed before the URI fix. The publish
assertion in the main suite now requires a dialable URI instead of merely
a non-empty string, which is what let the wrong form through.
* test: run the external NBD client suite against a live viperblockd
The suite so far only ever saw qemu-nbd. Pointing it at viperblockd's nbdkit
export is the comparison worth having: one external client, two providers it
cannot tell apart, and every difference is a candidate leak in the boundary.
RunNBDClientSuiteWithConfig takes the node name and volume prefix a live
cluster needs, and publishForNBD now unpublishes and deletes what it created
so a run against a real cluster leaves nothing behind. Subtests dial a
normalised URI so a provider emitting the legacy syntax still gets its export
tested rather than every subtest failing on the same string; whether the URI
itself is standard is now its own subtest.
Adds a reconnect check, since publication holds the volume rather than any one
connection. It retries the real operation because a cheaper liveness probe is
itself a connection and re-triggers whatever refused it.
* feat: make the conformance suite capability-driven
The suite asserted a fixed Capabilities literal, so it could only ever pass
against a provider advertising exactly that set. Every implementation had to
match the suite rather than the suite testing the contract. RunSuite now reads
GetCapabilities from the provider under test and gates optional behaviour on
what it advertises, and the fixed literal is demoted to ReferenceCapabilities,
a convenience for callers constructing a MemoryProvider.
Add ReadOnlyPublish to Capabilities. A provider that leaves it false must
refuse a ReadOnly publish rather than hand back a writable export, which is
what viperblockd already does; it now has a way to say so up front.
Add ErrUnsupportedCapability and the unsupported_capability error code. A
request for optional behaviour a provider does not advertise is well formed,
so classifying it as invalid_argument told the caller to fix a request that
was never wrong. viperblockd's ReadOnly refusal is reclassified accordingly.
Declared capabilities now match observed behaviour: qemunbdd advertises
SparseExtentReporting (qemu-nbd exports base:allocation, confirmed with
nbdinfo) and ReadOnlyPublish (PublishVolume passes -r). viperblockd advertises
neither: the nbdkit Go binding exposes no extents callback, and the mount path
never passes -r.
The external NBD client suite gains a sparse-extent subtest that checks the
export's advertised contexts against the capability, so SparseExtentReporting
is falsifiable from outside rather than taken on trust, and its read-only
subtest now branches on the capability instead of assuming refusal.
Cover the new branches with a MemoryProvider advertising no optional
capabilities and one advertising OnlineExpansion, the branch no provider we
ship exercises.
* refactor: rename SourceVolumeID to SourceSnapshotVolumeID
CreateVolumeRequest.SourceVolumeID meant three different things. viperblockd
required it whenever SourceSnapshotID was set and used it to resolve the
snapshot's chunk references, so there it named the snapshot's origin volume.
MemoryProvider ignored it entirely. qemunbdd implemented it as an independent
volume-to-volume clone, which is what the name and doc comment implied. The
conformance suite could not catch the divergence because the reference
provider had no branch for the field at all.
The contract already has a field meaning "the volume this snapshot came from":
Snapshot.SourceVolumeID, which every provider populates. So the request field
is the same fact travelling back the other way, and its name was the problem.
Rename it to SourceSnapshotVolumeID, document it as the source snapshot's
origin volume, and state that it is required with SourceSnapshotID and
meaningless without it.
All three providers now enforce one rule: a source snapshot must be
accompanied by its origin volume, or the request is invalid_argument.
MemoryProvider additionally rejects an origin that does not match the
snapshot's own SourceVolumeID, so a wrong name fails instead of resolving
blocks against a volume the snapshot never came from. qemunbdd loses its
volume-to-volume clone branch, which implemented behaviour the contract no
longer offers and no caller ever requested.
The suite pins both the requirement and a full create-snapshot-restore round
trip, so the field is exercised end to end rather than only on the error path.
* feat: make the conformance suite runnable against a live provider
The suite had only ever run against MemoryProvider and qemunbdd, both of which
are discarded with the test binary. viperblockd, the provider production
actually uses, was the one implementation the suite never judged, because the
suite assumed each subtest got fresh state: hard-coded volume IDs collide with
an earlier run's leftovers on a store that keeps them, and a hard-coded node ID
addresses a node that does not exist.
SuiteConfig adds NamePrefix, NodeID and OtherNodeID. A tracking wrapper records
every volume, snapshot and publication the suite creates and tears them down in
dependency order, logging rather than failing so a teardown problem cannot mask
the assertion a subtest was making. Two new tests cover this by running the
whole suite twice against one shared provider instance, which passes only
because cleanup works, and once per prefix to cover concurrent runs sharing a
daemon.
TestLive_ViperblockdConformance wires this to a real cluster behind
SPINIFEX_VIPERBLOCK_LIVE, alongside the existing external NBD client suite.
* fix(viperblockd): close the four contract gaps the live conformance run found
Unpublishing a volume this node has no mount for now succeeds instead of
returning not_found, matching the legacy unmount path: nothing mounted here is
already the state the request asks for.
CreateSnapshot refuses a snapshot ID that already exists against a different
source volume with already_exists, rather than overwriting it.
DeleteVolume refuses a published volume with volume_in_use instead of tearing
the mount down, on this node or on whichever node answers the volume's owner
subject. The export and the guest writing through it survive the metadata
deletion, so the old behaviour surfaced later as corruption.
CreateVolume reads the source snapshot before writing the volume, returning
not_found when it is absent and invalid_argument when it was taken from a
different volume than the request names. Previously both produced a blank
volume and a successful response.
CopySnapshot classifies viperblock's foreign-volume refusal as invalid_argument
now that it carries a sentinel.
* feat(ebsprovider): pin the source snapshot's origin volume in the contract
The suite only required source_snapshot_volume_id to be present, so a caller
naming a volume the snapshot never came from was accepted everywhere. The
snapshot records its own origin, so this is checkable: creating from a snapshot
with a mismatched source volume is now invalid_argument for every provider,
enforced by qemunbdd against its snapshot metadata and by viperblockd against
the snapshot's persisted SourceVolumeName.
* test(ebsprovider): run the contract suite against viperblockd in CI
CI installed neither qemu nor nbdkit, so every conformance test that needs a
real implementation skipped silently and MemoryProvider was the only provider
the suite judged. CI now installs both, builds viperblock's nbdkit plugin, and
sets SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses them fails
instead of skipping.
TestViperblockdConformance stands the provider handlers up over an embedded
NATS and a real predastore and runs the same suite against them, which is what
caught CopySnapshot accepting a foreign volume's snapshot on unencrypted
volumes.
* feat(viperblockd): honour read-only publish and advertise block size constraints
ReadOnlyPublish was advertised false and read_only=true was rejected,
so the capability was true only by omission. nbdkit -r now starts the
export when the request asks for it, which sets the NBD read-only
transmission flag and makes the plugin refuse writes. Access mode is
fixed when nbdkit starts, so a republish asking for the other mode is
refused with VolumeInUse rather than handed the running export.
Block size constraints were absent entirely: the nbdkit Go binding has
no block-size callback, so the plugin cannot advertise them. The
blocksize-policy filter can, without a binding callback. Minimum 1
because viperblock's WriteAt handles unaligned requests with a
read-modify-write, preferred 4096 to match its block size, maximum 32M
to match what qemu-nbd advertises.
* test(conformance): judge sparse reporting by the allocation map, not the context
The subtest asserted that base:allocation was offered. nbdkit offers
that context whether or not the plugin can compute extents, answering
'all allocated' when it cannot, so the assertion passed for a server
with no extent support at all — it could not fail. What separates the
two is the answer: a volume with nothing written to it is entirely
holes to a server that really tracks allocation.
Read the map with nbdinfo --map --json and compare holes against the
advertised capability. qemunbdd reports holes and advertises true;
viperblockd reports none and advertises false.
* test(fixtures): keep SSL_CERT_FILE set for the fixture's whole lifetime
The predastore fixture is one daemon per test binary, but its cert was
exported with t.Setenv, which reverts when the first test that called
Start finishes. In-process clients did not notice because crypto/x509
caches the pool behind a sync.Once, loaded while the variable was still
set. Child processes did: nbdkit spawned by a later test inherited an
environment with no SSL_CERT_FILE and could not verify the daemon, so
the whole NBD client suite failed when the package ran as a whole while
passing when run alone.
Also install libnbd-bin in CI, so the NBD client suite runs there
instead of failing on missing tools.
* ci: install nbdkit-plugin-dev so the viperblock plugin can build
* test(volume): give the snapshot fixture an owner
CreateVolume now fails closed on a snapshot whose owner it cannot match,
including an unset owner_id. The fixture predates that and wrote none,
so the provider test failed with InvalidSnapshot.NotFound before ever
reaching the provider.
* test(integration): give the volume harness a real provider
The control plane no longer builds volumes itself, so CreateVolume in
the integration harness failed with "no EBS provider configured": the
harness wired a VolumeServiceImpl with nothing behind it. It now serves
viperblockd's provider subjects against the same predastore fixture, so
the blocks are still written by real viperblock to a real daemon.
Three persisted-state assertions moved with it. TenantID, VolumeType
and AvailabilityZone are control-plane facts, held in the ebsmetadata
document rather than viperblock's own state, so the storage oracle now
asserts their absence and DescribeVolumes -- whose answer is
account-scoped -- is what checks them.
* refactor: route image import through the EBS provider and drop the legacy metadata path
The control plane could still only run against viperblock: the AMI import and
the ochre weights import both linked the storage library directly, and the
ebsmetadata store kept a read fallback onto viperblock's config.json.
Import now goes through the provider contract with no new verb: CreateVolume,
PublishVolume, a bulk copy over the published NBD export with qemu-img, then
UnpublishVolume and CreateSnapshot. Bulk data rides NBD rather than NATS,
whose request size the contract caps far below a disk image.
The migration scaffolding goes with it. ebsmetadata documents are now the only
thing the control plane reads: no legacy prefix scan, no config.json fallback,
no backfill on daemon start. config.json belongs to the provider, which is the
single writer of it.
ListAMIs keeps its tolerant/strict split. Skipping an undecodable document is
right for DescribeImages, where one corrupt document must not hide the fleet,
and wrong for a name-uniqueness check, which cannot rule out a name it failed
to read.
spinifex and cmd no longer import viperblock outside services/viperblockd and
testutil/vbscan.
* feat: add a volume enumeration verb so a lost document cannot strand blocks
Making ebsmetadata documents the sole index of volumes left one failure with
no recovery: a volume whose document is gone disappears from DescribeVolumes,
from the leak reaper (which enumerates documents), and from the provider,
whose contract had no verb that asks what it holds.
ListVolumes closes that. It is capability-gated on VolumeEnumeration and
paginated, following CSI's optional LIST_VOLUMES rather than inventing a
shape. Refs are deliberately not Volumes: enumeration answers what exists,
and a provider must not open every volume's engine to say so.
Each implementation reads what actually exists rather than what it remembers.
qemunbdd walks its volumes directory, not its in-memory map, which does not
survive a restart; viperblockd walks the bucket's top-level prefixes. A
volume the provider has forgotten is precisely what a caller is looking for.
The consumer is a read-only orphan report: provider volumes minus documented
ones, exposed as spx admin volumes orphans. It never deletes. An EFI store is
excused only when its base volume is documented, so a stranded pair is still
reported. Listing documents stays strict, because reporting a volume as
orphaned on the strength of a document that failed to decode would point an
operator at live data.
* fix: stop the orphan scan accusing storage of holding things it does not
A live env19 run reported three orphans against a healthy cluster: an AMI's
backing volume, a snapshot, and the cluster key store. All three were wrong,
and a report that cries wolf on a clean cluster is worse than no report.
Enumeration treated every top-level bucket prefix as a volume. Snapshots sit
beside volumes at that level and have a config of their own, so presence of a
config cannot separate them; the snap- name is what does, which is already how
the AMI removal path reads them back. keys/ and spinifex/ are not volumes at
all.
An AMI's blocks are a genuine provider volume, so enumeration keeps reporting
them. What was missing is on the reading side: an AMI is recorded as an image
document, not a volume one, so the scan now treats a documented image as
known. Listing images is strict for the same reason listing volumes is —
unreadable and absent are different answers, and only one means orphaned.
* fix: let an orphaned image's blocks be reclaimed
The orphan report tells an operator that reclaiming a stranded volume is their
decision, and images remove --force is the tool it points them at. On env19
that tool failed on exactly those volumes: removal drops the ebsmetadata
document first, and predastore answers NoSuchKey for a document that is
already gone, which is the definition of an orphan.
Deleting the document stays first and stays fatal. That ordering is the
barrier that stops a launch landing on blocks about to be deleted, and an AMI
still advertised after its blocks are gone is the worse failure. Only an
already-absent document is now tolerated: the barrier is satisfied, and the
blocks are the sole thing left to reclaim.
S3ObjectStore.DeleteObject normalises a missing key the way GetObject already
does, so this is recognised rather than matched on error text.
The obvious unit test passed without the fix, because the in-memory store is
silently tolerant on delete and real S3 answers 204 — neither behaves like the
backend this runs against. The test now drives a store that reports the miss,
and fails without the fix.
* feat: add per-verb OTel spans to the EBS provider contract
The ten provider verbs cross a single NATS seam and nothing measured it.
The only spans on the storage path were the legacy ebs.mount/ebs.unmount
pair, so control-plane storage latency could not be attributed to a verb
or compared across provider implementations.
NATSProvider.request is the one client-side funnel every verb passes
through, including both legs of an owner-subject fallback, so a single
span there covers the whole contract and measures what the control plane
experiences regardless of which provider answers.
Span names come from a closed verb set rather than the subject. Owner
and snapshot-create subjects embed a volume ID and the mount/unmount
subjects embed a node ID; naming spans after them would give every
volume a span name of its own. The IDs move to attributes and an
unrecognised subject collapses to a single unknown verb.
Trace context rides the message headers, so the serving provider joins
the caller's trace instead of rooting one of its own. The carrier is a
copy of the one in spinifex/utils rather than an import: that package
pulls in the aws-sdk and the cluster key store, which the wire contract
must not depend on.
Responses carry no shared accessor for their error field, so both sides
re-read just that field from the encoded reply, and only when the span
is recording. Without it a failed verb looks like a successful round
trip.
viperblockd serves the contract with its own handlers rather than
natsserve, so its handlers now take the per-message context and its
publish/unpublish pair drops the subject-named consumer span it opened
for itself.
* feat: add null and hostile providers, and a per-verb bench suite
The provider boundary made it possible to put something other than
viperblock behind the control plane, and nothing yet did.
nullprovider is the floor. It is not a new type: MemoryProvider already
answers every verb from a map and natsserve already serves any
EBSProvider, so the package names that composition and states what it
measures. On loopback with no storage behind it a round trip costs
around 195us against 74ns in process, which is the seam's own price and
is not attributable to any storage engine.
conformance.RunBenchSuite prices one round trip per verb and drives
every implementation through the same calls in the same order, so two
providers' numbers are comparable. It measures control operations only:
guest I/O never crosses these verbs.
hostile wraps any provider in one that misbehaves. Faults derive from
the seed and a per-target sequence rather than a global counter, so
concurrent callers on different volumes cannot reorder each other's
faults and a failure found deep into a soak run replays exactly. Error
and error-after-work are separate faults on purpose: the second reports
failure after the work has already happened, which is how an orphan is
made, and is what makes an enumeration scan the right oracle for a soak.
Enumeration itself is never faulted, since an oracle that can be told
what to say cannot falsify anything.
The soak asserts that every volume the provider still holds is
attributable to an injected fault. A leftover with nothing behind it is
the control plane losing track of storage on its own.
Two bugs were found by falsifying the new tests rather than by running
them. The FNV-derived draw clustered badly enough that two fault bands
were never reachable in 1200 draws, so the draw now goes through a
SplitMix64 finalizer. And the first version of the lie test was
worthless: MemoryProvider returns copies, so a lie could not corrupt it
whether or not the injector copied, and the test now drives a provider
that hands out pointers into its own state.
* test: bench the provider contract against qemunbdd and a live viperblockd
RunBenchSuite was only ever pointed at the null provider, so the numbers
priced the seam and nothing else. Point it at the two real implementations.
BenchConfig gains a VolumeBytes knob: 1 GiB per iteration is fine against a
provider that stores nothing and wrong against one that does, so qemunbdd and
viperblockd are both measured at 64 MiB.
liveProvider splits into a testing.TB flavour. Benchmarks need the same
connection and the same node and cannot be handed a *testing.T.
* fix(viperblockd): carry the trace context into every viperblock engine call
Both S3 clients only emit a span when the request already carries one, so
calling the no-context Init, LoadState, SaveState, LoadBlockState and Close
left the whole engine open untraceable. Every provider verb traced as one flat
server span, and the 342ms describe measured on env19 could not be attributed
to anything.
viperblock already exposes the Ctx variants and Backend already declares
InitCtx, so this is threading, not new instrumentation. openVolumeVB,
openLoadedVolumeVB, sealVolumeVB, describeVolumeEngine, snapshotVolumeEngine,
copySnapshotOnVolume, seedVolume and applyConfigUpdate now take a context, and
mount/unmount keep the context their own span derives instead of dropping it.
completeCreateSnapshot gets WithoutCancel: the background half of
accept-then-publish has to outlive the request it was accepted by and still be
attributable to it.
The per-volume ebs.config handler opens a consumer span like the other legacy
subjects, so a config update is no longer the one path with no trace at all.
* perf: describe a volume without opening an engine on it
describeVolumeEngine fell through to openVolumeVB for any volume not mounted
on the node serving the request, which constructed a full viperblock engine
to read one integer. On env19 that decomposed as a 22 ms reachability list, a
3 ms state read and a 16 ms PutObject of the volume's own config.json — the
write coming from the SeqNum bootstrap a read-write open performs.
Use viperblock's read-only state load instead. A describe is now the 3 ms
read: no background goroutines, no reachability probe, no SeqNum window
claimed, and no write issued by a worker that does not own the volume.
Falsified: restoring the openVolumeVB call fails
TestDescribeVolume_ReadsWithoutOpeningTheVolume on both the write and the
list, with the fake backend answering each so neither assertion hides behind
the other.
* perf: share one HTTP client across every viperblock backend
Four sites built the same S3Config by hand, each leaving HTTPClient unset, so
every engine open and state read got its own connection pool and paid its own
TLS handshake. That was invisible while Backend.Init issued a whole-bucket
list — the list paid for the connection and the state read rode it — and it
became the dominant cost once describe was reduced to a single request.
Route all four through cfg.volumeS3Config, which builds the client once.
Falsified: dropping the HTTPClient field takes
TestDescribeVolume_ReusesOneConnection from one connection to three.
* perf: wait for nbdkit to be ready instead of waiting a fixed second
mountVolume spawned nbdkit and then ran a confirm loop until a one-second
deadline passed. The loop only broke early when the process exited, so the
fast-failure path worked and the success path always cost the full second.
That second was ~1015 ms of a ~1040 ms publish, and it emitted no spans, so
the timings never contradicted the assumption that it was process spawn.
Wait for the endpoint to accept a connection instead. That is also the
question the caller needs answered, since it is about to hand the URI to
something that will connect to it — where 'still alive after a second' only
correlates with it. A nbdkit that exits during startup still fails the mount
immediately, and one that never comes up now fails on a deadline rather than
being assumed healthy.
Falsified: reintroducing a fixed sleep in pollUntil fails both the readiness
timing assertion and the fast-failure one.
* test: measure the data path, not just the control verbs
The conformance benchmarks price create, publish and describe. Guest
I/O never crosses those verbs, so none of those numbers say anything
about what a guest experiences. This drives the published export with
qemu-img bench, which sits on the same block layer virtio-blk does.
It asserts only that the I/O completes and logs a table: what a given
store is capable of is not a contract, and pinning a number here would
only produce a flaky test.
Read profiles stay inside the region the write profiles covered, or
they would be timing how fast the export answers with zeroes, and one
write profile flushes, because an unflushed write measures how fast the
WAL accepts data rather than how fast the store takes it.
Profiles are spaced apart because the plugin tears an engine down per
disconnect and removes the volume's local directory as it goes, so a
connection opened while the previous one is closing fails its
handshake.
* test: run data-path profiles back to back
The spacing was a workaround for a volume lock that did not hold across
close. With that fixed, consecutive profiles are worth more than the
workaround was: each is its own NBD connection, so a run that completes
has also shown that an arriving connection waits out the previous one's
close rather than racing it.
* fix: refuse a delete that cannot establish whether a volume is published
volumeMountedElsewhere returned false on every failure path, including a
request timeout, so a busy or partitioned owner was indistinguishable
from no owner at all and the delete proceeded. Deleting a published
volume leaves the nbdkit export and the guest writing through it alive
over storage that is going away.
Report the three outcomes separately. NATS distinguishes no-responders
from no-answer, so only the former means unmounted; an unanswered probe
is refused and the caller may retry once the owner is reachable.
* feat: exclude a second viperblock engine with a JetStream volume lease
Nothing in the stack stopped two viperblock engines opening one volume from
different nodes. For an encrypted volume that is not just a consistency risk:
the AES-GCM nonce is fully consumed by SeqNum, VolumeUUID and a domain byte,
and the SeqNum window claim is a read-then-write with no CAS, so two engines
opening concurrently issue overlapping nonces under the same key.
Every engine open now takes a lease first, keyed by volume in a JetStream KV
bucket. A lease is a KV create, which JetStream serialises, so a second
claimant loses instead of both winning. The revision that won is the
generation: monotonic across the bucket, so a later opener always carries a
higher one than the writer it displaced. Entries carry a TTL and a renewal
loop, so a node that dies mid-mount does not wedge its volumes forever.
Losing a lease is logged, not enforced. Until the object store can refuse a
stale writer's PUT, failing live guest I/O on lease loss would add a failure
mode without stopping the zombie it is aimed at.
A daemon with no lease store refuses to open rather than opening blind, so
nbdkit recovery binds the store before it adopts surviving exports. Repeat
acquisitions on one node share a lease: the unmount seal opens a detached
engine while the mount being torn down still holds it, and cross-node
exclusion is what the lease is for. A restart reclaims this node's own stale
entries, since the exports that outlived the daemon are still running.
* refactor: consume masterkey from bluebottle in the test this branch added
* feat(ebsprovider): classify refusals a caller may retry
Every ErrorCode said what went wrong and none said whether the same
request could succeed later, so a caller could not tell a permanent
refusal from a transient one. Transient conditions shipped as internal,
which tells a caller to give up when it should retry.
Adds ErrUnavailable / ErrorCodeUnavailable and ErrorCode.Retryable(),
which callers branch on instead of on the code itself. DeleteVolume's
"could not establish whether the volume is published" refusal moves off
internal onto it: the owner that did not answer may answer later.
TestErrorCodeRetryable names every code, so adding one without deciding
its retry meaning fails rather than silently defaulting to permanent.
TestClassifyError pins the sentinel-to-wire-code mapping, which had no
coverage at all; a sentinel missing from classifyError falls through to
internal and loses both the reason and the retry signal.
* fix(ebsprovider): drop CopyOnWriteClone, which no request can ask for
Four providers advertised the capability and nothing consumed it: there
is no clone verb on the interface. CreateVolume's SourceSnapshotID
restores from a snapshot and CopySnapshot duplicates snapshot metadata;
neither is a volume clone, and the field documenting SourceSnapshotID
says so explicitly. CSI expresses clone as a volume_content_source with
a volume source, which this contract does not implement.
Advertising it told a caller a feature was available and then gave them
no way to use it. Removed rather than backed by a method: adding a clone
verb is a feature, not a contract fix, and the capability can come back
alongside it. The Capabilities doc comment now states the rule, and
qemunbdd's stale comment about sparse extents is corrected while there.
* feat(ebsprovider): add ListSnapshots so snapshots are reconcilable
The contract could create, copy and delete snapshots but never say which
ones it held. That left the control-plane metadata document as the only
index of them, so losing one stranded the data it pointed at with no way
to find it again. This is the same exposure ListVolumes closed for
volumes, and it stayed invisible because ListSnapshots is optional in
CSI too.
ListSnapshots pages exactly as ListVolumes does, gated on a new
SnapshotEnumeration capability. SnapshotRef is deliberately not a
Snapshot: enumeration answers what exists, and reading a snapshot's size
or state can cost far more than listing it.
viperblockd walks the bucket's snap- prefixes, sharing the top-level
walk with ListVolumes. It leaves SourceVolumeID empty because the link
lives inside the snapshot's own encrypted state and reading it would
mean a fetch per snapshot listed. qemunbdd walks its snapshots
directory rather than its in-memory map, so a snapshot that outlived the
process still appears; its SourceVolumeID is empty in that case for the
same reason CreateSnapshot refuses to reuse the ID.
Five conformance subtests cover it, and they run green against
MemoryProvider, qemunbdd and viperblockd over a real predastore.
* feat(ebsprovider): state the exclusion guarantee in the contract
A caller could not tell which providers prevent two writers from holding
one volume, at what scope, or for how long. The guarantee existed in
viperblockd and was invisible from the outside, which is why the
conformance suite had no concurrency dimension at all.
Capabilities.Exclusion states it, and deliberately separates three
questions the contract used to run together: how far exclusion reaches
(none, node, cluster), whether a dead owner's claim ever expires, and
whether a writer that lost its claim is actually stopped. That last one
matters most and is the least flattering: exclusion refuses the second
opener, fencing stops the first. viperblockd advertises cluster scope
with a 45s TTL and FencesLostClaim false, because a partitioned node
whose lease lapsed keeps writing until the object store can refuse it.
Four new conformance subtests. Two required a second viperblockd node in
the harness: cluster exclusion cannot be exercised by one node, because
the second publish goes to a subject nobody serves and times out rather
than being refused. Both nodes share the NATS the leases live in and the
bucket the volumes live in, so a refusal comes from the lease.
Standing that second node up immediately found a real gap, in a test
that had been silently skipping for want of one. A publish refused by
another node's lease came back as internal rather than volume_in_use, so
a caller was told something had broken when the answer was that someone
else held the volume. Mapped in handlePublishVolume.
* build: pin viperblock to the release carrying the provider's dependencies
The branch has been building against a local replace pointing at a
viperblock worktree, so its committed pin never actually built: it named
a version without CopySnapshotMeta, ErrSnapshotVolumeMismatch,
NewStateReader or s3.NewHTTPClient, all of which this branch calls.
Points at the merged viperblock dev instead and drops the replace.
GOWORK=off now builds clean, which is the deploy path ansible uses.
* fix: let viperblockd's delete sweep tolerate an object another sweep removed
deleteObjectPrefix claimed to mirror handlers/ec2/volume's deleteS3Prefix and
give DeleteVolume and DeleteSnapshot idempotent-when-absent behaviour, but it
returned on any DeleteObject error. Two overlapping deletes for one volume left
the loser reporting an internal failure once the winner had removed a chunk
between the list and the delete.
The control-plane copy that carried this fix is gone with the embedded engine
path, so the provider path reintroduced the bug it had just fixed.
* fix: stop CreateVolume leaving local state on whichever node served it
A state read prefers this node's local copy over the backend, and provider RPCs
are not owner-routed, so the node that happened to serve a create kept a copy
that answered describe as if the volume still existed after another node
deleted it. The two-node conformance harness caught it as DeleteVolume/success
failing about one run in three.
The seeded path already removed its local files after closing. The unseeded
path saved state and left them, which was the only difference between the two.
* fix: guard the mount path itself against a second export of one volume
handlePublishVolume refused a volume this node already exported, but
mountVolume did not, and the legacy ebs.<node>.mount subject is the route
production takes for instance launch and hot-attach. A retried attach racing a
fresh one started two real nbdkit against one volume, with no refusal and no
warning: the volume lease only refuses a different node, and a same-node
re-acquire finds the volume already held and increments its refcount.
That also made the advertised exclusion inaccurate. viperblockd claims
cluster-scoped exclusion on the grounds that a second opener is refused, while
the conformance subtests reach it only through PublishVolume, so they passed
with the legacy route unguarded. Moving the check into mountVolume gives both
entry points the same behaviour and makes the claim true for the path
production uses.
* fix: tolerate a missing legacy AMI config on deregister
DeregisterImage deletes the AMI document, then the pre-provider
{imageID}/config.json. Predastore answers a delete of an absent key with
NoSuchKey, and that was treated as fatal, so every AMI created since the
metadata store landed failed to deregister after its document was already
gone. A retrying caller then saw InvalidAMIID.NotFound for an image it had
just been told it could not delete.
The regression test uses a store that answers a missing-key delete the way
Predastore does; MemoryObjectStore silently succeeds, which is why the unit
suite stayed green.
* fix: reap ENIs left behind by an abandoned instance launch
A launch creates the instance's ENI and then attaches it. When the daemon
dies in between, the record survives with no instance and nothing ever looks
at it again. The security group dependency check counts an ENI that lists the
group, so the group cannot be deleted for as long as the record exists.
The in-process launch path already rolls back on error; a process that dies
cannot. This adds a cluster-wide sweep over records that are launch-created,
never attached, and older than the launch RPC timeout by a wide margin. The
description prefix is now a shared constant because it is the only thing
separating an abandoned launch from a standalone ENI a caller asked for.
This commit is contained in:
Josh Siviour
@@ -76,6 +76,20 @@ jobs:
|
||||
- name: Setup Go Workspace
|
||||
run: go work init ./spinifex ./viperblock ./predastore ./northstar ./bluebottle
|
||||
|
||||
# The EBS provider contract suite needs a real implementation on each
|
||||
# side of the boundary: qemu-img/qemu-nbd/qemu-io back qemunbdd, which
|
||||
# shares no code with viperblock, and nbdkit plus viperblock's plugin
|
||||
# back viperblockd, the provider production actually uses. Without them
|
||||
# the suite runs against MemoryProvider alone. nbdkit-plugin-dev carries
|
||||
# the headers and nbdkit.pc that cgo needs to build that plugin, and
|
||||
# libnbd-bin supplies the third-party NBD client (nbdinfo/nbdcopy) that
|
||||
# judges the published export by something outside this codebase.
|
||||
- name: Install conformance tools
|
||||
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends qemu-utils nbdkit nbdkit-plugin-dev libnbd-bin
|
||||
|
||||
- name: Build viperblock nbdkit plugin
|
||||
run: make -C viperblock go_build_nbd
|
||||
|
||||
- name: Build Spinifex
|
||||
run: make -C spinifex build
|
||||
|
||||
@@ -83,6 +97,9 @@ jobs:
|
||||
run: make -C spinifex manifest-check manifest-lint
|
||||
|
||||
- name: Run Tests with Coverage
|
||||
env:
|
||||
SPINIFEX_REQUIRE_CONFORMANCE_TOOLS: "1"
|
||||
VIPERBLOCK_NBDKIT_PLUGIN: ${{ github.workspace }}/viperblock/lib/nbdkit-viperblock-plugin.so
|
||||
run: make -C spinifex test-cover
|
||||
|
||||
- name: Run Action Tests
|
||||
|
||||
@@ -69,6 +69,18 @@ linters:
|
||||
- pkg: math/rand$
|
||||
desc: Use math/rand/v2 instead, see https://go.dev/blog/randv2
|
||||
|
||||
# The EC2 control plane reaches block storage through ebsprovider, not
|
||||
# by importing the engine. viperblock.New starts a chunk uploader, so
|
||||
# every construction is a writer against a volume viperblockd serves.
|
||||
# Nothing under handlers/ may import viperblock, in any form, tests
|
||||
# included. A types-only import counts: one became an engine import.
|
||||
ec2-control-plane:
|
||||
files:
|
||||
- '**/spinifex/handlers/**'
|
||||
deny:
|
||||
- pkg: github.com/mulgadc/viperblock
|
||||
desc: Use the ebsprovider.EBSProvider interface; the control plane must not construct or name viperblock types.
|
||||
|
||||
embeddedstructfieldcheck:
|
||||
forbid-mutex: true
|
||||
|
||||
|
||||
+122
-108
@@ -30,6 +30,8 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/iam"
|
||||
"github.com/mulgadc/spinifex/spinifex/admin"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/formation"
|
||||
"github.com/mulgadc/spinifex/spinifex/gpu"
|
||||
handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
|
||||
@@ -38,9 +40,6 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/network/host"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/mulgadc/viperblock/viperblock/backends/s3"
|
||||
"github.com/mulgadc/viperblock/viperblock/v_utils"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/pterm/pterm"
|
||||
"github.com/spf13/cobra"
|
||||
@@ -198,6 +197,25 @@ takes effect immediately. Prompts for confirmation unless --yes is passed.`,
|
||||
Run: runimagesPromoteCmd,
|
||||
}
|
||||
|
||||
var volumesCmd = &cobra.Command{
|
||||
Use: "volumes",
|
||||
Short: "Inspect block storage volumes",
|
||||
Long: `Inspect the volumes the storage provider holds, independently of the EC2 API.`,
|
||||
}
|
||||
|
||||
var volumesOrphansCmd = &cobra.Command{
|
||||
Use: "orphans",
|
||||
Short: "Report volumes the provider holds with no control-plane record",
|
||||
Long: `List volumes the storage provider holds that the control plane has no record
|
||||
of. Their blocks are consuming space but they have no API handle, so they
|
||||
cannot be described or deleted through the EC2 API.
|
||||
|
||||
This command only reports. It never deletes: the evidence that an orphan is
|
||||
still wanted is the record that went missing, so removal is an operator
|
||||
decision made per volume.`,
|
||||
Run: runVolumesOrphansCmd,
|
||||
}
|
||||
|
||||
var accountCmd = &cobra.Command{
|
||||
Use: "account",
|
||||
Short: "Manage Spinifex accounts",
|
||||
@@ -296,6 +314,9 @@ func init() {
|
||||
imagesCmd.AddCommand(imagesRemoveCmd)
|
||||
imagesCmd.AddCommand(imagesPromoteCmd)
|
||||
|
||||
adminCmd.AddCommand(volumesCmd)
|
||||
volumesCmd.AddCommand(volumesOrphansCmd)
|
||||
|
||||
adminCmd.AddCommand(accountCmd)
|
||||
accountCmd.AddCommand(accountCreateCmd)
|
||||
accountCmd.AddCommand(accountListCmd)
|
||||
@@ -403,6 +424,10 @@ func init() {
|
||||
|
||||
const bytesPerGiB = 1024 * 1024 * 1024
|
||||
|
||||
// imageImportTimeout bounds each provider call the import makes. Generous
|
||||
// because the snapshot at the end runs over the whole imported volume.
|
||||
const imageImportTimeout = 30 * time.Minute
|
||||
|
||||
// amiVolumeSizeGiB returns the smallest whole GiB that still holds sizeBytes.
|
||||
//
|
||||
// Rounding up is load-bearing. The image is copied into a root volume of
|
||||
@@ -430,16 +455,10 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
|
||||
var imageStat os.FileInfo
|
||||
var err error
|
||||
|
||||
cfgFile, _ := cmd.Flags().GetString("config")
|
||||
forceCmd, _ := cmd.Flags().GetBool("force")
|
||||
skipVerify, _ := cmd.Flags().GetBool("skip-verify")
|
||||
ostmpDir, _ := cmd.Flags().GetString("tmp-dir")
|
||||
|
||||
// Use default config path
|
||||
if cfgFile == "" {
|
||||
cfgFile = DefaultConfigFile()
|
||||
}
|
||||
|
||||
//configDir, _ := cmd.Flags().GetString("config-dir")
|
||||
baseDir, _ := cmd.Flags().GetString("spinifex-dir")
|
||||
|
||||
@@ -610,57 +629,46 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Create the specified manifest file to describe the image/AMI
|
||||
manifest := viperblock.VolumeConfig{}
|
||||
|
||||
// Calculate the size
|
||||
|
||||
manifest.AMIMetadata.Name = fmt.Sprintf("ami-%s-%s-%s", image.Distro, image.Version, image.Arch)
|
||||
if amiNameOverride != "" {
|
||||
manifest.AMIMetadata.Name = amiNameOverride
|
||||
}
|
||||
// Describe the image/AMI as the control-plane document it will become.
|
||||
volumeId := utils.GenerateResourceID("ami")
|
||||
manifest.AMIMetadata.ImageID = volumeId
|
||||
amiName := fmt.Sprintf("ami-%s-%s-%s", image.Distro, image.Version, image.Arch)
|
||||
if amiNameOverride != "" {
|
||||
amiName = amiNameOverride
|
||||
}
|
||||
|
||||
manifest.AMIMetadata.Description = fmt.Sprintf("%s cloud image prepared for Spinifex", manifest.AMIMetadata.Name)
|
||||
manifest.AMIMetadata.Architecture = image.Arch
|
||||
manifest.AMIMetadata.PlatformDetails = image.Platform
|
||||
manifest.AMIMetadata.CreationDate = time.Now()
|
||||
manifest.AMIMetadata.RootDeviceType = "ebs"
|
||||
manifest.AMIMetadata.Virtualization = "hvm"
|
||||
manifest.AMIMetadata.ImageOwnerAlias = "system"
|
||||
manifest.AMIMetadata.VolumeSizeGiB = amiVolumeSizeGiB(imageStat.Size())
|
||||
manifest.AMIMetadata.BootMode = image.BootMode
|
||||
manifest.AMIMetadata.Distro = image.Distro
|
||||
manifest.AMIMetadata.DistroFamily = utils.DistroFamily(image.Distro)
|
||||
ami := ebsmetadata.AMI{
|
||||
ImageID: volumeId,
|
||||
Name: amiName,
|
||||
Description: fmt.Sprintf("%s cloud image prepared for Spinifex", amiName),
|
||||
Architecture: image.Arch,
|
||||
PlatformDetails: image.Platform,
|
||||
CreationDate: time.Now(),
|
||||
RootDeviceType: "ebs",
|
||||
Virtualization: "hvm",
|
||||
ImageOwnerAlias: "system",
|
||||
VolumeSizeGiB: amiVolumeSizeGiB(imageStat.Size()),
|
||||
SnapshotID: admin.SnapPrefix(volumeId),
|
||||
BootMode: image.BootMode,
|
||||
Distro: image.Distro,
|
||||
DistroFamily: utils.DistroFamily(image.Distro),
|
||||
}
|
||||
|
||||
// Copy catalog-provided tags (e.g. spinifex:managed-by for system AMIs)
|
||||
// onto the imported AMI so the UI can filter them out.
|
||||
if len(image.Tags) > 0 {
|
||||
manifest.AMIMetadata.Tags = make(map[string]string, len(image.Tags))
|
||||
maps.Copy(manifest.AMIMetadata.Tags, image.Tags)
|
||||
ami.Tags = make(map[string]string, len(image.Tags))
|
||||
maps.Copy(ami.Tags, image.Tags)
|
||||
}
|
||||
|
||||
// Volume Data
|
||||
manifest.VolumeMetadata.VolumeID = volumeId // TODO: Confirm if unique, e.g vol-, if ami- used
|
||||
manifest.VolumeMetadata.VolumeName = manifest.AMIMetadata.Name
|
||||
manifest.VolumeMetadata.TenantID = "system"
|
||||
manifest.VolumeMetadata.SizeGiB = manifest.AMIMetadata.VolumeSizeGiB
|
||||
manifest.VolumeMetadata.State = "available"
|
||||
manifest.VolumeMetadata.AvailabilityZone = "" // TODO: Confirm
|
||||
manifest.VolumeMetadata.CreatedAt = time.Now()
|
||||
manifest.VolumeMetadata.VolumeType = "gp3"
|
||||
manifest.VolumeMetadata.IOPS = 1000
|
||||
|
||||
// Write the manifest to disk
|
||||
// Save as JSON
|
||||
jsonData, err := json.Marshal(manifest)
|
||||
jsonData, err := json.Marshal(ami)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Could not marshal manifest: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
manifestFilename := fmt.Sprintf("%s/%s.json", imagePath, manifest.AMIMetadata.Name)
|
||||
manifestFilename := fmt.Sprintf("%s/%s.json", imagePath, ami.Name)
|
||||
// Write to file
|
||||
err = os.WriteFile(manifestFilename, jsonData, 0600)
|
||||
if err != nil {
|
||||
@@ -668,51 +676,15 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Upload the image to S3 (predastore)
|
||||
|
||||
appConfig, err := config.LoadConfig(cfgFile)
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
appConfig, nc, err := loadConfigAndConnect()
|
||||
if err != nil {
|
||||
fmt.Println("Error loading config file:", err)
|
||||
return
|
||||
}
|
||||
|
||||
s3Config := s3.S3Config{
|
||||
VolumeName: volumeId,
|
||||
VolumeSize: utils.SafeInt64ToUint64(imageStat.Size()),
|
||||
Bucket: appConfig.Nodes[appConfig.Node].Predastore.Bucket,
|
||||
Region: appConfig.Nodes[appConfig.Node].Predastore.Region,
|
||||
AccessKey: appConfig.Nodes[appConfig.Node].Predastore.AccessKey,
|
||||
SecretKey: appConfig.Nodes[appConfig.Node].Predastore.SecretKey,
|
||||
Host: appConfig.Nodes[appConfig.Node].Predastore.Host,
|
||||
}
|
||||
|
||||
mkey, err := utils.LoadViperblockMasterKey(appConfig.Nodes[appConfig.Node].Viperblock.EncryptionKeyFile)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Could not load viperblock encryption key: %v\n", err)
|
||||
fmt.Fprintf(os.Stderr, "Could not connect to the cluster: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Import is an interactive command whose output is a progress bar, so the
|
||||
// volume's routine mount chatter is noise here: a fresh import has no prior
|
||||
// state, making the "no state found" / 404 lines expected rather than
|
||||
// notable. Scoping the logger to this VB (New copies it onto the backend
|
||||
// too) keeps it off the process-wide default. Errors still surface, both
|
||||
// through this logger and as the returned error the caller prints.
|
||||
vbConfig := viperblock.VB{
|
||||
VolumeName: volumeId,
|
||||
VolumeSize: utils.SafeInt64ToUint64(imageStat.Size()),
|
||||
BaseDir: tmpDir,
|
||||
Cache: viperblock.Cache{
|
||||
Config: viperblock.CacheConfig{
|
||||
Size: 0,
|
||||
},
|
||||
},
|
||||
VolumeConfig: manifest,
|
||||
MasterKey: mkey,
|
||||
EncryptionEnabled: mkey != nil,
|
||||
Logger: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelError})),
|
||||
}
|
||||
defer nc.Close()
|
||||
node := appConfig.Nodes[appConfig.Node]
|
||||
|
||||
// Bake the deployment CA into the image trust store so a stock cloud image
|
||||
// trusts the gateway from first boot. Best-effort: never fails the import.
|
||||
@@ -721,32 +693,31 @@ func runimagesImportCmd(cmd *cobra.Command, args []string) {
|
||||
// command, which would collapse the path to a relative config/ca.pem.
|
||||
bakeCACertIntoImage(extractedImagePath, filepath.Join(baseDir, "config", "ca.pem"))
|
||||
|
||||
// Render the flush bar here rather than inside viperblock, which stays a
|
||||
// pure storage library. The bar is built lazily on the first update so it is
|
||||
// never drawn if the import fails before any bytes flush; viperblock
|
||||
// throttles the callback to ≤101 invocations, so the human-readable title
|
||||
// renders without the per-block render-frequency regression.
|
||||
var flushBar *pterm.ProgressbarPrinter
|
||||
var flushUpdate func(current uint64)
|
||||
progress := func(current, total uint64) {
|
||||
if flushBar == nil {
|
||||
flushBar, flushUpdate = utils.NewByteProgressBar("Flushing image to storage", total)
|
||||
}
|
||||
flushUpdate(current)
|
||||
}
|
||||
|
||||
err = v_utils.ImportDiskImage(&s3Config, &vbConfig, extractedImagePath, progress)
|
||||
|
||||
if flushBar != nil {
|
||||
_, _ = flushBar.Stop()
|
||||
}
|
||||
|
||||
fmt.Println("Writing image to storage ...")
|
||||
provider := ebsprovider.NewNATSProvider(nc, imageImportTimeout)
|
||||
err = admin.ImportImage(context.Background(), provider, admin.ImportOpts{
|
||||
VolumeID: volumeId,
|
||||
NodeID: appConfig.Node,
|
||||
SizeBytes: utils.SafeUint64ToInt64(ami.VolumeSizeGiB * bytesPerGiB),
|
||||
AvailabilityZone: node.AZ,
|
||||
SourcePath: extractedImagePath,
|
||||
Snapshot: true,
|
||||
Progress: os.Stdout,
|
||||
})
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Could not import image to predastore: %v\n", err)
|
||||
fmt.Fprintf(os.Stderr, "Could not import image: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
defer os.RemoveAll(tmpDir)
|
||||
// The document is what DescribeImages enumerates, so it is written last:
|
||||
// until it exists the AMI is not launchable, and an import that failed
|
||||
// partway leaves no half-registered image behind.
|
||||
ami.State = "available"
|
||||
metaStore := objectstore.NewS3ObjectStoreFromConfig(node.Predastore.Host, node.Predastore.Region, node.Predastore.AccessKey, node.Predastore.SecretKey)
|
||||
if err := ebsmetadata.NewStore(metaStore, node.Predastore.Bucket).PutAMI(context.Background(), ami); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Imported %s but could not register it: %v\n", volumeId, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Printf("✅ Image import complete. Image-ID (AMI): %s\n", volumeId)
|
||||
}
|
||||
@@ -809,6 +780,49 @@ func bakeCACertIntoImage(imagePath, caCertPath string) {
|
||||
}
|
||||
}
|
||||
|
||||
func runVolumesOrphansCmd(_ *cobra.Command, _ []string) {
|
||||
appConfig, nc, err := loadConfigAndConnect()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Could not connect to the cluster: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer nc.Close()
|
||||
|
||||
node := appConfig.Nodes[appConfig.Node]
|
||||
store := objectstore.NewS3ObjectStoreFromConfig(
|
||||
node.Predastore.Host,
|
||||
node.Predastore.Region,
|
||||
node.Predastore.AccessKey,
|
||||
node.Predastore.SecretKey,
|
||||
)
|
||||
|
||||
orphans, err := admin.FindOrphanVolumes(
|
||||
context.Background(),
|
||||
ebsprovider.NewNATSProvider(nc, imageImportTimeout),
|
||||
ebsmetadata.NewStore(store, node.Predastore.Bucket),
|
||||
)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "Orphan scan failed:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if len(orphans) == 0 {
|
||||
fmt.Println("No orphaned volumes: every volume the provider holds has a control-plane record.")
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Printf("%d orphaned volume(s) — held by the provider, unknown to the control plane:\n\n", len(orphans))
|
||||
for _, orphan := range orphans {
|
||||
fmt.Printf(" %s\n", orphan.VolumeID)
|
||||
fmt.Printf(" handle: %s\n", orphan.Handle)
|
||||
if orphan.Derived {
|
||||
fmt.Println(" note: derived volume; its base volume is also unknown")
|
||||
}
|
||||
}
|
||||
fmt.Println()
|
||||
fmt.Println("Nothing was deleted. Each of these holds data that cannot be reached")
|
||||
fmt.Println("through the EC2 API; removing one is a per-volume operator decision.")
|
||||
}
|
||||
|
||||
func runimagesRemoveCmd(cmd *cobra.Command, args []string) {
|
||||
imageID, _ := cmd.Flags().GetString("image-id")
|
||||
force, _ := cmd.Flags().GetBool("force")
|
||||
|
||||
+27
-137
@@ -4,7 +4,6 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path"
|
||||
@@ -15,16 +14,13 @@ import (
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/bluebottle/pkg/masterkey"
|
||||
"github.com/mulgadc/spinifex/spinifex/admin"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
gateway_bedrock "github.com/mulgadc/spinifex/spinifex/gateway/bedrock"
|
||||
handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
vbs3 "github.com/mulgadc/viperblock/viperblock/backends/s3"
|
||||
"github.com/mulgadc/viperblock/viperblock/v_utils"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"github.com/pterm/pterm"
|
||||
"github.com/spf13/cobra"
|
||||
@@ -307,63 +303,6 @@ func buildWeightsImage(srcDir, imagePath string, contentBytes int64) error {
|
||||
return mkfsExt4Runner(srcDir, imagePath)
|
||||
}
|
||||
|
||||
// snapshotImportedWeightsVolume snapshots a viperblock volume that
|
||||
// v_utils.ImportDiskImage just wrote and closed -- never attached, never
|
||||
// nbdkit-served, so there is no live checkpoint to load. This mirrors the
|
||||
// offline snapshot sequence handlers/ec2/image/service_impl.go's
|
||||
// snapshotStoppedVolume uses for a stopped instance's root volume: reopen
|
||||
// read-only, load the numbered checkpoint Close() wrote, then CreateSnapshot.
|
||||
//
|
||||
// az names the zone recorded on the snapshot for DescribeSnapshots; it is not
|
||||
// used to resolve the clone.
|
||||
func snapshotImportedWeightsVolume(s3Config *vbs3.S3Config, volumeID string, volumeSize uint64, walDir, az string, mkey *masterkey.Key) (string, error) {
|
||||
vbConfig := viperblock.VB{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: volumeSize,
|
||||
BaseDir: walDir,
|
||||
Cache: viperblock.Cache{Config: viperblock.CacheConfig{Size: 0}},
|
||||
MasterKey: mkey,
|
||||
EncryptionEnabled: mkey != nil,
|
||||
Logger: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelError})),
|
||||
}
|
||||
|
||||
vb, err := viperblock.New(&vbConfig, "s3", *s3Config)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("new viperblock: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
vb.StopChunkUploader()
|
||||
vb.StopWALSyncer()
|
||||
}()
|
||||
|
||||
if err := vb.Backend.Init(); err != nil {
|
||||
return "", fmt.Errorf("backend init: %w", err)
|
||||
}
|
||||
if err := vb.LoadState(); err != nil {
|
||||
return "", fmt.Errorf("load state: %w", err)
|
||||
}
|
||||
if err := vb.LoadBlockState(); err != nil {
|
||||
return "", fmt.Errorf("load block state: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
if err := vb.RemoveLocalFiles(); err != nil {
|
||||
slog.Warn("snapshotImportedWeightsVolume: failed to remove local files", "volumeId", volumeID, "err", err)
|
||||
}
|
||||
}()
|
||||
|
||||
snapshotID := admin.SnapPrefix(volumeID)
|
||||
if _, err := vb.CreateSnapshot(snapshotID); err != nil {
|
||||
return "", fmt.Errorf("create snapshot: %w", err)
|
||||
}
|
||||
|
||||
store := objectstore.NewS3ObjectStoreFromConfig(s3Config.Host, s3Config.Region, s3Config.AccessKey, s3Config.SecretKey)
|
||||
if err := registerWeightsSnapshot(store, s3Config.Bucket, snapshotID, volumeID, volumeSize, az, mkey != nil); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return snapshotID, nil
|
||||
}
|
||||
|
||||
// registerWeightsSnapshot writes the EC2 control plane's half of the snapshot
|
||||
// prefix. CreateSnapshot above writes only viperblock's half -- the block
|
||||
// checkpoint and config.json -- but CreateVolume resolves a SnapshotId through
|
||||
@@ -485,12 +424,11 @@ func runStageWeights(ctx context.Context, store objectstore.ObjectStore, weights
|
||||
}
|
||||
|
||||
// materializeWeightsVolume is the real weightsMaterializer: it packages
|
||||
// downloadDir into a filesystem image, imports it into a new viperblock
|
||||
// volume backed by node's predastore, and snapshots it. This is the
|
||||
// expensive, environment-dependent half of stage -- it shells out to
|
||||
// mkfs.ext4 and talks to a live S3-shaped backend -- so it is exercised live
|
||||
// (see the plan's Testing section) rather than in unit tests.
|
||||
func materializeWeightsVolume(node config.Config, downloadDir string, contentBytes int64, mkey *masterkey.Key) (string, error) {
|
||||
// downloadDir into a filesystem image, imports it into a new volume through
|
||||
// the EBS provider, and snapshots it. This is the expensive,
|
||||
// environment-dependent half of stage -- it shells out to mkfs.ext4 and needs
|
||||
// a live provider -- so it is exercised live rather than in unit tests.
|
||||
func materializeWeightsVolume(ctx context.Context, provider ebsprovider.EBSProvider, nodeID string, node config.Config, downloadDir string, contentBytes int64) (string, error) {
|
||||
tmpDir := filepath.Dir(downloadDir)
|
||||
volumeId := utils.GenerateResourceID("vol")
|
||||
imagePath := filepath.Join(tmpDir, volumeId+".img")
|
||||
@@ -505,74 +443,32 @@ func materializeWeightsVolume(node config.Config, downloadDir string, contentByt
|
||||
return "", fmt.Errorf("could not stat image: %w", err)
|
||||
}
|
||||
|
||||
// Round the volume up to a whole GiB rather than using the raw image size.
|
||||
// viperblock rejects a tail write whose block overruns VolumeSize, so an
|
||||
// unaligned size fails at the last block. This also keeps the byte size and
|
||||
// the manifest's SizeGiB describing the same volume.
|
||||
// Round the volume up to a whole GiB rather than using the raw image size,
|
||||
// so the byte size and the recorded SizeGiB describe the same volume.
|
||||
sizeGiB := amiVolumeSizeGiB(imageStat.Size())
|
||||
volumeBytes := sizeGiB * bytesPerGiB
|
||||
|
||||
s3Config := vbs3.S3Config{
|
||||
VolumeName: volumeId,
|
||||
VolumeSize: volumeBytes,
|
||||
Bucket: node.Predastore.Bucket,
|
||||
Region: node.Predastore.Region,
|
||||
AccessKey: node.Predastore.AccessKey,
|
||||
SecretKey: node.Predastore.SecretKey,
|
||||
Host: node.Predastore.Host,
|
||||
}
|
||||
|
||||
// AMIMetadata is deliberately left zero-valued: a weights volume is not
|
||||
// bootable and must never be registered as a launchable AMI. Leaving it
|
||||
// unset also means ImportDiskImage does not attempt its own automatic
|
||||
// snapshot -- that happens explicitly below, after the volume is closed.
|
||||
manifest := viperblock.VolumeConfig{}
|
||||
manifest.VolumeMetadata.VolumeID = volumeId
|
||||
manifest.VolumeMetadata.VolumeName = volumeId
|
||||
manifest.VolumeMetadata.TenantID = "system"
|
||||
manifest.VolumeMetadata.SizeGiB = sizeGiB
|
||||
manifest.VolumeMetadata.State = "available"
|
||||
manifest.VolumeMetadata.AvailabilityZone = node.Predastore.Region
|
||||
manifest.VolumeMetadata.CreatedAt = time.Now()
|
||||
manifest.VolumeMetadata.VolumeType = "gp3"
|
||||
manifest.VolumeMetadata.IOPS = 1000
|
||||
|
||||
vbConfig := viperblock.VB{
|
||||
VolumeName: volumeId,
|
||||
VolumeSize: volumeBytes,
|
||||
BaseDir: tmpDir,
|
||||
Cache: viperblock.Cache{Config: viperblock.CacheConfig{Size: 0}},
|
||||
VolumeConfig: manifest,
|
||||
MasterKey: mkey,
|
||||
EncryptionEnabled: mkey != nil,
|
||||
Logger: slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelError})),
|
||||
}
|
||||
|
||||
var flushBar *pterm.ProgressbarPrinter
|
||||
var flushUpdate func(current uint64)
|
||||
progress := func(current, total uint64) {
|
||||
if flushBar == nil {
|
||||
flushBar, flushUpdate = utils.NewByteProgressBar("Flushing weights to storage", total)
|
||||
}
|
||||
flushUpdate(current)
|
||||
}
|
||||
|
||||
// Name the volume in the failure path: a part-written import leaves blocks
|
||||
// in predastore, and without the ID an operator cannot find them to reclaim.
|
||||
if err := v_utils.ImportDiskImage(&s3Config, &vbConfig, imagePath, progress); err != nil {
|
||||
if flushBar != nil {
|
||||
_, _ = flushBar.Stop()
|
||||
}
|
||||
fmt.Println("Writing weights to storage ...")
|
||||
if err := admin.ImportImage(ctx, provider, admin.ImportOpts{
|
||||
VolumeID: volumeId,
|
||||
NodeID: nodeID,
|
||||
SizeBytes: utils.SafeUint64ToInt64(volumeBytes),
|
||||
AvailabilityZone: node.AZ,
|
||||
SourcePath: imagePath,
|
||||
Snapshot: true,
|
||||
Progress: os.Stdout,
|
||||
}); err != nil {
|
||||
return "", fmt.Errorf("could not import weights volume %s: %w", volumeId, err)
|
||||
}
|
||||
if flushBar != nil {
|
||||
_, _ = flushBar.Stop()
|
||||
}
|
||||
|
||||
fmt.Println("Snapshotting weights volume ...")
|
||||
snapshotID, err := snapshotImportedWeightsVolume(&s3Config, volumeId, volumeBytes, tmpDir, node.AZ, mkey)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("could not snapshot weights volume: %w", err)
|
||||
// The provider wrote viperblock's half of the snapshot prefix; this writes
|
||||
// the EC2 control plane's, without which a launch cannot resolve it.
|
||||
snapshotID := admin.SnapPrefix(volumeId)
|
||||
store := objectstore.NewS3ObjectStoreFromConfig(node.Predastore.Host, node.Predastore.Region, node.Predastore.AccessKey, node.Predastore.SecretKey)
|
||||
if err := registerWeightsSnapshot(store, node.Predastore.Bucket, snapshotID, volumeId, volumeBytes, node.AZ, node.Viperblock.EncryptionKeyFile != ""); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return snapshotID, nil
|
||||
}
|
||||
@@ -629,15 +525,9 @@ func runOchreWeightsStage(cmd *cobra.Command, _ []string) {
|
||||
weightsStore := gateway_bedrock.NewWeightsStore(js, len(appConfig.Nodes))
|
||||
store := objectstore.NewS3ObjectStoreFromConfig(node.Predastore.Host, node.Predastore.Region, node.Predastore.AccessKey, node.Predastore.SecretKey)
|
||||
|
||||
mkey, err := utils.LoadViperblockMasterKey(node.Viperblock.EncryptionKeyFile)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Could not load viperblock encryption key: %v\n", err)
|
||||
ochreExit(1)
|
||||
return
|
||||
}
|
||||
|
||||
materialize := func(_ context.Context, downloadDir string, contentBytes int64) (string, error) {
|
||||
return materializeWeightsVolume(node, downloadDir, contentBytes, mkey)
|
||||
provider := ebsprovider.NewNATSProvider(nc, imageImportTimeout)
|
||||
materialize := func(ctx context.Context, downloadDir string, contentBytes int64) (string, error) {
|
||||
return materializeWeightsVolume(ctx, provider, appConfig.Node, node, downloadDir, contentBytes)
|
||||
}
|
||||
checkSnapshotLive := newWeightsSnapshotChecker(store, node.Predastore.Bucket)
|
||||
|
||||
|
||||
+124
-1
@@ -18,6 +18,7 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/services/nats"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/northstar"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/predastore"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/qemunbdd"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/qmpcollector"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/viperblockd"
|
||||
@@ -373,7 +374,9 @@ var viperblockStartCmd = &cobra.Command{
|
||||
encryptionKeyFile = envKey
|
||||
}
|
||||
|
||||
defer initTelemetry("viperblockd", false)()
|
||||
debug := viper.GetBool("viperblock-debug")
|
||||
|
||||
defer initTelemetry("viperblockd", debug)()
|
||||
|
||||
service, err := service.New("viperblock", &viperblockd.Config{
|
||||
NatsHost: nodeConfig.NATS.Host,
|
||||
@@ -390,6 +393,7 @@ var viperblockStartCmd = &cobra.Command{
|
||||
ShardWAL: shardWAL,
|
||||
GCEnabled: gcEnabled,
|
||||
EncryptionKeyFile: encryptionKeyFile,
|
||||
Debug: debug,
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
@@ -439,6 +443,111 @@ var viperblockStatusCmd = &cobra.Command{
|
||||
},
|
||||
}
|
||||
|
||||
var qemunbdCmd = &cobra.Command{
|
||||
Use: "qemunbd",
|
||||
Short: "Manage the qemunbd service",
|
||||
}
|
||||
|
||||
// Repeat for qemunbd. It answers the same ebs.provider.v1.* subjects as
|
||||
// viperblock, so the two must never run against the same NATS cluster
|
||||
// together.
|
||||
var qemunbdStartCmd = &cobra.Command{
|
||||
Use: "start",
|
||||
Short: "Start the qemunbd service",
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
fmt.Println("Starting qemunbd service...")
|
||||
|
||||
cfgFile := viper.GetString("config")
|
||||
|
||||
if cfgFile == "" {
|
||||
fmt.Println("Config file is not set")
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Println("Loading config from:", cfgFile)
|
||||
|
||||
clusterConfig, err := config.LoadConfig(cfgFile)
|
||||
if err != nil {
|
||||
fmt.Println("Error loading config file:", err)
|
||||
return
|
||||
}
|
||||
nodeConfig := clusterConfig.Nodes[clusterConfig.Node]
|
||||
|
||||
natsHost := viper.GetString("nats-host")
|
||||
|
||||
if natsHost != "" {
|
||||
fmt.Println("Overwriting natsHost:", natsHost)
|
||||
nodeConfig.NATS.Host = natsHost
|
||||
}
|
||||
|
||||
baseDir := viper.GetString("base-dir")
|
||||
if baseDir != "" {
|
||||
fmt.Println("Overwriting base-dir:", baseDir)
|
||||
nodeConfig.Predastore.BaseDir = baseDir
|
||||
}
|
||||
|
||||
// Apply changes back to cluster config
|
||||
clusterConfig.Nodes[clusterConfig.Node] = nodeConfig
|
||||
|
||||
debug := viper.GetBool("qemunbd-debug")
|
||||
|
||||
defer initTelemetry("qemunbdd", debug)()
|
||||
|
||||
service, err := service.New("qemunbd", &qemunbdd.Config{
|
||||
NatsHost: nodeConfig.NATS.Host,
|
||||
NatsToken: nodeConfig.NATS.ACL.Token,
|
||||
NatsCACert: nodeConfig.NATS.CACert,
|
||||
BaseDir: nodeConfig.Predastore.BaseDir,
|
||||
NodeName: clusterConfig.Node,
|
||||
Debug: debug,
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
fmt.Println("Error starting qemunbd service:", err)
|
||||
return
|
||||
}
|
||||
|
||||
_, err = service.Start()
|
||||
|
||||
if err != nil {
|
||||
fmt.Println("Error starting qemunbd service:", err)
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Println("qemunbd service started")
|
||||
},
|
||||
}
|
||||
|
||||
var qemunbdStopCmd = &cobra.Command{
|
||||
Use: "stop",
|
||||
Short: "Stop the qemunbd service",
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
fmt.Println("Stopping qemunbd service...")
|
||||
|
||||
service, err := service.New("qemunbd", &qemunbdd.Config{})
|
||||
|
||||
if err != nil {
|
||||
fmt.Println("Error stopping qemunbd service:", err)
|
||||
return
|
||||
}
|
||||
|
||||
if err = service.Stop(); err != nil {
|
||||
fmt.Println("Error stopping qemunbd service:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Println("qemunbd service stopped")
|
||||
},
|
||||
}
|
||||
|
||||
var qemunbdStatusCmd = &cobra.Command{
|
||||
Use: "status",
|
||||
Short: "Get status of the qemunbd service",
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
fmt.Println("qemunbd service status: ...")
|
||||
},
|
||||
}
|
||||
|
||||
// Repeat for nats.
|
||||
var natsStartCmd = &cobra.Command{
|
||||
Use: "start",
|
||||
@@ -1300,10 +1409,24 @@ func init() {
|
||||
viper.BindEnv("viperblock-encryption-key-file", "SPINIFEX_VIPERBLOCK_ENCRYPTION_KEY_FILE")
|
||||
viper.BindPFlag("viperblock-encryption-key-file", viperblockCmd.PersistentFlags().Lookup("encryption-key-file"))
|
||||
|
||||
viperblockCmd.PersistentFlags().Bool("debug", false, "Viperblock (EBS) debug logging")
|
||||
viper.BindEnv("viperblock-debug", "SPINIFEX_VIPERBLOCK_DEBUG")
|
||||
viper.BindPFlag("viperblock-debug", viperblockCmd.PersistentFlags().Lookup("debug"))
|
||||
|
||||
viperblockCmd.AddCommand(viperblockStartCmd)
|
||||
viperblockCmd.AddCommand(viperblockStopCmd)
|
||||
viperblockCmd.AddCommand(viperblockStatusCmd)
|
||||
|
||||
serviceCmd.AddCommand(qemunbdCmd)
|
||||
|
||||
qemunbdCmd.PersistentFlags().Bool("debug", false, "qemunbd (EBS) debug logging")
|
||||
viper.BindEnv("qemunbd-debug", "SPINIFEX_QEMUNBD_DEBUG")
|
||||
viper.BindPFlag("qemunbd-debug", qemunbdCmd.PersistentFlags().Lookup("debug"))
|
||||
|
||||
qemunbdCmd.AddCommand(qemunbdStartCmd)
|
||||
qemunbdCmd.AddCommand(qemunbdStopCmd)
|
||||
qemunbdCmd.AddCommand(qemunbdStatusCmd)
|
||||
|
||||
// Nats
|
||||
serviceCmd.AddCommand(natsCmd)
|
||||
|
||||
|
||||
@@ -115,6 +115,9 @@ gc_enabled = true
|
||||
encryption_key_file = "{{.EncryptionKeyFile}}"
|
||||
{{- end}}
|
||||
|
||||
[nodes.{{.Node}}.ebs]
|
||||
provider = "viperblockd"
|
||||
|
||||
[nodes.{{.Node}}.vpcd]
|
||||
ovn_nb_addr = "{{.OVNNBAddr}}"
|
||||
ovn_sb_addr = "{{.OVNSBAddr}}"
|
||||
|
||||
@@ -24,7 +24,7 @@ require (
|
||||
github.com/mulgadc/bluebottle v1.16.1-0.20260812012137-683e88e465e3
|
||||
github.com/mulgadc/northstar v1.16.1-0.20260812012604-5d7a622ad059
|
||||
github.com/mulgadc/predastore v1.16.1-0.20260812012531-12c90bd2e14a
|
||||
github.com/mulgadc/viperblock v1.16.1-0.20260812012547-bb1225320f0c
|
||||
github.com/mulgadc/viperblock v1.16.1-0.20260812033921-fc825c93eff6
|
||||
github.com/nats-io/nats-server/v2 v2.14.4
|
||||
github.com/nats-io/nats.go v1.52.0
|
||||
github.com/opencontainers/runtime-spec v1.3.0
|
||||
|
||||
@@ -1327,6 +1327,8 @@ github.com/mulgadc/predastore v1.16.1-0.20260812012531-12c90bd2e14a h1:XSvL4FucG
|
||||
github.com/mulgadc/predastore v1.16.1-0.20260812012531-12c90bd2e14a/go.mod h1:S5vcni7R3VYi3nccelMagKbWdX58WtbKf0I1R/7MmNI=
|
||||
github.com/mulgadc/viperblock v1.16.1-0.20260812012547-bb1225320f0c h1:5PCU4696Gelc+4X4K+E+EzNVsyoLLWEiF/TcsIaAGi4=
|
||||
github.com/mulgadc/viperblock v1.16.1-0.20260812012547-bb1225320f0c/go.mod h1:8NT5u+udY1CsoKz/JAPY2SSMcQ98Omi+42r8BkoA4FY=
|
||||
github.com/mulgadc/viperblock v1.16.1-0.20260812033921-fc825c93eff6 h1:QfdcaqnrDDzsO4neJNX7BdL8vzyga/WPn4A/04QZcVw=
|
||||
github.com/mulgadc/viperblock v1.16.1-0.20260812033921-fc825c93eff6/go.mod h1:8NT5u+udY1CsoKz/JAPY2SSMcQ98Omi+42r8BkoA4FY=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
||||
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
package admin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
)
|
||||
|
||||
// ImportOpts describes one raw disk image to import into a new volume.
|
||||
type ImportOpts struct {
|
||||
// VolumeID names the volume to create. For an AMI import this is the
|
||||
// image ID, so the snapshot lands at SnapPrefix(imageID).
|
||||
VolumeID string
|
||||
// NodeID is the node whose provider serves the NBD export written to. It
|
||||
// must be the node this process runs on: the export is local to it.
|
||||
NodeID string
|
||||
// SizeBytes is the volume's capacity, which the caller rounds to a whole
|
||||
// GiB. It must be at least the source image's size.
|
||||
SizeBytes int64
|
||||
// AvailabilityZone is recorded on the volume.
|
||||
AvailabilityZone string
|
||||
// SourcePath is the raw disk image on local disk.
|
||||
SourcePath string
|
||||
// Snapshot creates SnapPrefix(VolumeID) once the write completes, so
|
||||
// launches can clone the image instead of copying it block by block.
|
||||
Snapshot bool
|
||||
// Progress receives the importer's own output (a percentage counter).
|
||||
// Nil discards it.
|
||||
Progress io.Writer
|
||||
}
|
||||
|
||||
// ImportImage writes a local raw disk image into a new provider volume over
|
||||
// NBD, then optionally snapshots it. Everything here goes through the provider
|
||||
// contract, so an import works against any conforming provider rather than
|
||||
// only the one whose storage library this process happens to link.
|
||||
//
|
||||
// A failed import deliberately leaves the volume behind: its blocks are
|
||||
// already in the object store, and deleting it would take the ID an operator
|
||||
// needs to reclaim them with it.
|
||||
func ImportImage(ctx context.Context, provider ebsprovider.EBSProvider, opts ImportOpts) error {
|
||||
if opts.VolumeID == "" || opts.NodeID == "" || opts.SourcePath == "" {
|
||||
return errors.New("import: volume ID, node ID and source path are all required")
|
||||
}
|
||||
if opts.SizeBytes <= 0 {
|
||||
return fmt.Errorf("import: invalid volume size %d", opts.SizeBytes)
|
||||
}
|
||||
|
||||
if _, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: opts.VolumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: opts.SizeBytes},
|
||||
AvailabilityZone: opts.AvailabilityZone,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("import: create volume %s: %w", opts.VolumeID, err)
|
||||
}
|
||||
|
||||
published, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: opts.VolumeID,
|
||||
NodeID: opts.NodeID,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("import: publish volume %s: %w", opts.VolumeID, err)
|
||||
}
|
||||
|
||||
writeErr := writeImage(ctx, opts.SourcePath, published.NBDURI, opts.Progress)
|
||||
|
||||
// Unpublish regardless: a volume left published cannot be snapshotted or
|
||||
// attached, and on the failure path it would strand the export too.
|
||||
if err := provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: opts.VolumeID,
|
||||
NodeID: opts.NodeID,
|
||||
}); err != nil {
|
||||
if writeErr != nil {
|
||||
return fmt.Errorf("import: write image: %w (unpublish also failed: %w)", writeErr, err)
|
||||
}
|
||||
return fmt.Errorf("import: unpublish volume %s: %w", opts.VolumeID, err)
|
||||
}
|
||||
if writeErr != nil {
|
||||
return fmt.Errorf("import: write image to %s: %w", opts.VolumeID, writeErr)
|
||||
}
|
||||
|
||||
if opts.Snapshot {
|
||||
if _, err := provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
SnapshotID: SnapPrefix(opts.VolumeID),
|
||||
VolumeID: opts.VolumeID,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("import: snapshot volume %s: %w", opts.VolumeID, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeImage is the bulk copy step, indirected so a test can drive the verb
|
||||
// sequence around it without a live NBD export to write to.
|
||||
var writeImage = writeImageToNBD
|
||||
|
||||
// writeImageToNBD copies a raw image onto a published export with qemu-img,
|
||||
// which speaks NBD as a target and is already a runtime dependency. Passing
|
||||
// the export as the target rather than a file is what keeps the bulk data off
|
||||
// NATS, whose request size the provider contract caps far below an image.
|
||||
func writeImageToNBD(ctx context.Context, sourcePath, nbdURI string, progress io.Writer) error {
|
||||
target, err := qemuNBDTarget(nbdURI)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// -n keeps qemu-img from trying to create a target that already exists,
|
||||
// and --target-is-zero lets it skip the source's holes: the volume was
|
||||
// created moments ago and has never been written.
|
||||
cmd := exec.CommandContext(ctx, "qemu-img", "convert",
|
||||
"-f", "raw", "-O", "raw", "-n", "--target-is-zero", "-p",
|
||||
sourcePath, target)
|
||||
if progress != nil {
|
||||
cmd.Stdout = progress
|
||||
}
|
||||
var stderr []byte
|
||||
stderrPipe, err := cmd.StderrPipe()
|
||||
if err != nil {
|
||||
return fmt.Errorf("qemu-img stderr: %w", err)
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return fmt.Errorf("start qemu-img: %w", err)
|
||||
}
|
||||
stderr, _ = io.ReadAll(stderrPipe)
|
||||
if err := cmd.Wait(); err != nil {
|
||||
return fmt.Errorf("qemu-img convert: %w: %s", err, stderr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// qemuNBDTarget renders a provider NBD URI in the form qemu-img accepts. The
|
||||
// contract's unix form is not qemu's, so it is rebuilt rather than passed on.
|
||||
func qemuNBDTarget(nbdURI string) (string, error) {
|
||||
serverType, socketPath, host, port, err := utils.ParseNBDURI(nbdURI)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("parse NBD URI %q: %w", nbdURI, err)
|
||||
}
|
||||
switch serverType {
|
||||
case "unix":
|
||||
if _, err := os.Stat(socketPath); err != nil {
|
||||
return "", fmt.Errorf("NBD socket %s: %w", socketPath, err)
|
||||
}
|
||||
return "nbd+unix:///?socket=" + socketPath, nil
|
||||
case "inet":
|
||||
return "nbd://" + net.JoinHostPort(host, strconv.Itoa(port)), nil
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported NBD server type %q in %q", serverType, nbdURI)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
package admin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// recordingProvider wraps MemoryProvider to record the verb sequence and the
|
||||
// arguments the import passes, so a test can assert the order and the IDs
|
||||
// rather than only the end state.
|
||||
type recordingProvider struct {
|
||||
*ebsprovider.MemoryProvider
|
||||
|
||||
calls []string
|
||||
publishes []ebsprovider.PublishVolumeRequest
|
||||
snapshots []ebsprovider.CreateSnapshotRequest
|
||||
publishURI string
|
||||
}
|
||||
|
||||
func newRecordingProvider() *recordingProvider {
|
||||
return &recordingProvider{MemoryProvider: ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{CrashConsistentSnapshot: true}), publishURI: "nbd:unix:/tmp/does-not-matter.sock"}
|
||||
}
|
||||
|
||||
func (p *recordingProvider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
p.calls = append(p.calls, "create")
|
||||
return p.MemoryProvider.CreateVolume(ctx, req)
|
||||
}
|
||||
|
||||
func (p *recordingProvider) PublishVolume(ctx context.Context, req ebsprovider.PublishVolumeRequest) (*ebsprovider.PublishedVolume, error) {
|
||||
p.calls = append(p.calls, "publish")
|
||||
p.publishes = append(p.publishes, req)
|
||||
if _, err := p.MemoryProvider.PublishVolume(ctx, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ebsprovider.PublishedVolume{VolumeID: req.VolumeID, NodeID: req.NodeID, NBDURI: p.publishURI}, nil
|
||||
}
|
||||
|
||||
func (p *recordingProvider) UnpublishVolume(ctx context.Context, req ebsprovider.UnpublishVolumeRequest) error {
|
||||
p.calls = append(p.calls, "unpublish")
|
||||
return p.MemoryProvider.UnpublishVolume(ctx, req)
|
||||
}
|
||||
|
||||
func (p *recordingProvider) CreateSnapshot(ctx context.Context, req ebsprovider.CreateSnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
p.calls = append(p.calls, "snapshot")
|
||||
p.snapshots = append(p.snapshots, req)
|
||||
return p.MemoryProvider.CreateSnapshot(ctx, req)
|
||||
}
|
||||
|
||||
// stubWriteImage swaps the bulk copy for one that records what it was handed
|
||||
// and returns err, restoring the real writer when the test ends.
|
||||
func stubWriteImage(t *testing.T, err error, seen *string) {
|
||||
t.Helper()
|
||||
original := writeImage
|
||||
writeImage = func(_ context.Context, _, nbdURI string, _ io.Writer) error {
|
||||
if seen != nil {
|
||||
*seen = nbdURI
|
||||
}
|
||||
return err
|
||||
}
|
||||
t.Cleanup(func() { writeImage = original })
|
||||
}
|
||||
|
||||
// TestImportImage_VerbSequenceAndSnapshotID is the assertion the old
|
||||
// config.json read-back stood for: the snapshot the import creates must be the
|
||||
// one the AMI document will name, and both derive from SnapPrefix. A snapshot
|
||||
// under any other ID leaves the registered AMI pointing at nothing.
|
||||
func TestImportImage_VerbSequenceAndSnapshotID(t *testing.T) {
|
||||
var wroteTo string
|
||||
stubWriteImage(t, nil, &wroteTo)
|
||||
provider := newRecordingProvider()
|
||||
|
||||
err := ImportImage(context.Background(), provider, ImportOpts{
|
||||
VolumeID: "ami-import001",
|
||||
NodeID: "node-a",
|
||||
SizeBytes: 8 << 30,
|
||||
AvailabilityZone: "ap-southeast-2a",
|
||||
SourcePath: "/dev/null",
|
||||
Snapshot: true,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, []string{"create", "publish", "unpublish", "snapshot"}, provider.calls,
|
||||
"the volume must be unpublished before it is snapshotted")
|
||||
require.Len(t, provider.snapshots, 1)
|
||||
assert.Equal(t, SnapPrefix("ami-import001"), provider.snapshots[0].SnapshotID)
|
||||
assert.Equal(t, "ami-import001", provider.snapshots[0].VolumeID)
|
||||
require.Len(t, provider.publishes, 1)
|
||||
assert.Equal(t, "node-a", provider.publishes[0].NodeID)
|
||||
assert.Equal(t, provider.publishURI, wroteTo, "the copy must target the URI the provider published")
|
||||
}
|
||||
|
||||
// TestImportImage_WriteFailureStillUnpublishes covers the failure path: a
|
||||
// volume left published can be neither attached nor snapshotted, so the
|
||||
// export must be released even when the copy that needed it failed.
|
||||
func TestImportImage_WriteFailureStillUnpublishes(t *testing.T) {
|
||||
stubWriteImage(t, errors.New("copy exploded"), nil)
|
||||
provider := newRecordingProvider()
|
||||
|
||||
err := ImportImage(context.Background(), provider, ImportOpts{
|
||||
VolumeID: "ami-import002",
|
||||
NodeID: "node-a",
|
||||
SizeBytes: 1 << 30,
|
||||
SourcePath: "/dev/null",
|
||||
Snapshot: true,
|
||||
})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "copy exploded")
|
||||
|
||||
assert.Equal(t, []string{"create", "publish", "unpublish"}, provider.calls,
|
||||
"a failed copy must not be snapshotted, and must still be unpublished")
|
||||
}
|
||||
|
||||
// TestImportImage_NoSnapshotWhenNotAsked covers the weights-style import that
|
||||
// wants the volume but not an AMI snapshot.
|
||||
func TestImportImage_NoSnapshotWhenNotAsked(t *testing.T) {
|
||||
stubWriteImage(t, nil, nil)
|
||||
provider := newRecordingProvider()
|
||||
|
||||
require.NoError(t, ImportImage(context.Background(), provider, ImportOpts{
|
||||
VolumeID: "vol-import003",
|
||||
NodeID: "node-a",
|
||||
SizeBytes: 1 << 30,
|
||||
SourcePath: "/dev/null",
|
||||
}))
|
||||
assert.Equal(t, []string{"create", "publish", "unpublish"}, provider.calls)
|
||||
assert.Empty(t, provider.snapshots)
|
||||
}
|
||||
|
||||
// TestImportImage_RejectsIncompleteOpts keeps a caller from creating a volume
|
||||
// it then cannot address, which would strand blocks under a name nobody holds.
|
||||
func TestImportImage_RejectsIncompleteOpts(t *testing.T) {
|
||||
stubWriteImage(t, nil, nil)
|
||||
|
||||
for name, opts := range map[string]ImportOpts{
|
||||
"no volume ID": {NodeID: "node-a", SizeBytes: 1, SourcePath: "/dev/null"},
|
||||
"no node ID": {VolumeID: "vol-a", SizeBytes: 1, SourcePath: "/dev/null"},
|
||||
"no source": {VolumeID: "vol-a", NodeID: "node-a", SizeBytes: 1},
|
||||
"no size": {VolumeID: "vol-a", NodeID: "node-a", SourcePath: "/dev/null"},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
provider := newRecordingProvider()
|
||||
require.Error(t, ImportImage(context.Background(), provider, opts))
|
||||
assert.Empty(t, provider.calls, "nothing may be created before the options are checked")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestQemuNBDTarget_RewritesUnixForm locks the translation: the contract's
|
||||
// unix URI is not a form qemu accepts, so passing it through unchanged would
|
||||
// fail at the copy rather than here.
|
||||
func TestQemuNBDTarget_RewritesUnixForm(t *testing.T) {
|
||||
socket := t.TempDir() + "/export.sock"
|
||||
require.NoError(t, os.WriteFile(socket, nil, 0o600))
|
||||
|
||||
got, err := qemuNBDTarget("nbd:unix:" + socket)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "nbd+unix:///?socket="+socket, got)
|
||||
|
||||
got, err = qemuNBDTarget("nbd://10.0.0.5:10809")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "nbd://10.0.0.5:10809", got)
|
||||
|
||||
_, err = qemuNBDTarget("nbd:unix:" + t.TempDir() + "/absent.sock")
|
||||
require.Error(t, err, "a socket that is not there must fail before qemu-img is run")
|
||||
}
|
||||
@@ -1,21 +1,16 @@
|
||||
package admin
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
handlers_ec2_image "github.com/mulgadc/spinifex/spinifex/handlers/ec2/image"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
)
|
||||
|
||||
// SystemOwnerAlias is the fixed owner alias written to AMI config on promotion.
|
||||
@@ -49,13 +44,13 @@ func PromoteSystemImage(store objectstore.ObjectStore, bucket string, opts Promo
|
||||
return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
|
||||
}
|
||||
|
||||
meta, err := readAMIConfig(store, bucket, opts.ImageID)
|
||||
meta, err := readAMI(store, bucket, opts.ImageID)
|
||||
switch {
|
||||
case err == nil:
|
||||
// ok
|
||||
case objectstore.IsNoSuchKeyError(err):
|
||||
return nil, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
case errors.Is(err, handlers_ec2_image.ErrCorruptAMIConfig):
|
||||
case errors.Is(err, ebsmetadata.ErrCorruptDocument):
|
||||
return nil, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
default:
|
||||
slog.Error("PromoteSystemImage: read AMI config", "imageId", opts.ImageID, "err", err)
|
||||
@@ -69,8 +64,8 @@ func PromoteSystemImage(store objectstore.ObjectStore, bucket string, opts Promo
|
||||
prev := meta.ImageOwnerAlias
|
||||
meta.ImageOwnerAlias = SystemOwnerAlias
|
||||
|
||||
if err := writeAMIConfig(store, bucket, opts.ImageID, meta); err != nil {
|
||||
slog.Error("PromoteSystemImage: write AMI config", "imageId", opts.ImageID, "err", err)
|
||||
if err := ebsmetadata.NewStore(store, bucket).PutAMI(context.Background(), meta); err != nil {
|
||||
slog.Error("PromoteSystemImage: write AMI document", "imageId", opts.ImageID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
@@ -78,39 +73,18 @@ func PromoteSystemImage(store objectstore.ObjectStore, bucket string, opts Promo
|
||||
return &PromoteImageResult{PreviousOwner: prev}, nil
|
||||
}
|
||||
|
||||
// writeAMIConfig persists updated AMIMetadata to {imageID}/config.json,
|
||||
// preserving the VBState wrapper used by GetAMIConfig.
|
||||
func writeAMIConfig(store objectstore.ObjectStore, bucket, imageID string, meta viperblock.AMIMetadata) error {
|
||||
state := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: meta,
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(state)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String(bucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Body: bytes.NewReader(data),
|
||||
ContentType: aws.String("application/json"),
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// GetAMIMetadata reads and returns the AMIMetadata for the given image ID.
|
||||
// Returns ErrorInvalidAMIIDNotFound for missing or corrupt configs.
|
||||
func GetAMIMetadata(store objectstore.ObjectStore, bucket, imageID string) (viperblock.AMIMetadata, error) {
|
||||
meta, err := readAMIConfig(store, bucket, imageID)
|
||||
// GetAMIMetadata reads and returns the control-plane document for the given
|
||||
// image ID. Returns ErrorInvalidAMIIDNotFound for missing or corrupt documents.
|
||||
func GetAMIMetadata(store objectstore.ObjectStore, bucket, imageID string) (ebsmetadata.AMI, error) {
|
||||
meta, err := readAMI(store, bucket, imageID)
|
||||
switch {
|
||||
case err == nil:
|
||||
return meta, nil
|
||||
case objectstore.IsNoSuchKeyError(err):
|
||||
return viperblock.AMIMetadata{}, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
case errors.Is(err, handlers_ec2_image.ErrCorruptAMIConfig):
|
||||
return viperblock.AMIMetadata{}, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
return ebsmetadata.AMI{}, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
case errors.Is(err, ebsmetadata.ErrCorruptDocument):
|
||||
return ebsmetadata.AMI{}, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
default:
|
||||
return viperblock.AMIMetadata{}, errors.New(awserrors.ErrorServerInternal)
|
||||
return ebsmetadata.AMI{}, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,11 +1,8 @@
|
||||
package admin
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -21,8 +18,8 @@ func TestPromoteSystemImage_HappyPath(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, testRemoveAccountID, result.PreviousOwner)
|
||||
|
||||
// Verify the persisted config now carries the system alias.
|
||||
meta, err := readAMIConfig(store, testRemoveBucket, id)
|
||||
// Verify the persisted document now carries the system alias.
|
||||
meta, err := readAMI(store, testRemoveBucket, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, SystemOwnerAlias, meta.ImageOwnerAlias)
|
||||
// Other fields must be preserved.
|
||||
@@ -71,14 +68,9 @@ func TestPromoteSystemImage_InvalidPrefix_Malformed(t *testing.T) {
|
||||
func TestPromoteSystemImage_CorruptConfig_NotFound(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
const id = "ami-corrupt-promote"
|
||||
_, err := store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(id + "/config.json"),
|
||||
Body: bytes.NewReader([]byte("{not valid json")),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putCorruptAMI(t, store, id)
|
||||
|
||||
_, err = PromoteSystemImage(store, testRemoveBucket, PromoteImageOpts{ImageID: id})
|
||||
_, err := PromoteSystemImage(store, testRemoveBucket, PromoteImageOpts{ImageID: id})
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
|
||||
}
|
||||
@@ -105,14 +97,9 @@ func TestGetAMIMetadata_MissingConfig_NotFound(t *testing.T) {
|
||||
func TestGetAMIMetadata_CorruptConfig_NotFound(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
const id = "ami-corrupt-meta"
|
||||
_, err := store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(id + "/config.json"),
|
||||
Body: bytes.NewReader([]byte("not json {")),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putCorruptAMI(t, store, id)
|
||||
|
||||
_, err = GetAMIMetadata(store, testRemoveBucket, id)
|
||||
_, err := GetAMIMetadata(store, testRemoveBucket, id)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
|
||||
}
|
||||
|
||||
+42
-108
@@ -2,10 +2,8 @@ package admin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -13,11 +11,10 @@ import (
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
handlers_ec2_image "github.com/mulgadc/spinifex/spinifex/handlers/ec2/image"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
)
|
||||
|
||||
// RemoveImageOpts configures RemoveSystemImage.
|
||||
@@ -59,8 +56,8 @@ type RemovePreview struct {
|
||||
Name string
|
||||
Owner string
|
||||
Created time.Time
|
||||
ConfigPresent bool // false when ami-<id>/config.json is missing
|
||||
ConfigCorrupt bool // true when config.json exists but is undecodable
|
||||
ConfigPresent bool // false when the AMI's ebsmetadata document is missing
|
||||
ConfigCorrupt bool // true when the document exists but is undecodable
|
||||
IsSystemOwned bool // ImageOwnerAlias is set and not an account ID
|
||||
|
||||
AMIObjectCount int
|
||||
@@ -87,7 +84,7 @@ func PreviewRemoveSystemImage(store objectstore.ObjectStore, bucket, imageID str
|
||||
|
||||
preview := &RemovePreview{ImageID: imageID}
|
||||
|
||||
meta, configErr := readAMIConfig(store, bucket, imageID)
|
||||
meta, configErr := readAMI(store, bucket, imageID)
|
||||
switch {
|
||||
case configErr == nil:
|
||||
preview.ConfigPresent = true
|
||||
@@ -97,7 +94,7 @@ func PreviewRemoveSystemImage(store objectstore.ObjectStore, bucket, imageID str
|
||||
preview.IsSystemOwned = meta.ImageOwnerAlias != "" && !utils.IsAccountID(meta.ImageOwnerAlias)
|
||||
case objectstore.IsNoSuchKeyError(configErr):
|
||||
// Config absent — salvage candidate. Leave ConfigPresent=false.
|
||||
case errors.Is(configErr, handlers_ec2_image.ErrCorruptAMIConfig):
|
||||
case errors.Is(configErr, ebsmetadata.ErrCorruptDocument):
|
||||
preview.ConfigCorrupt = true
|
||||
default:
|
||||
return nil, fmt.Errorf("preview: read AMI config: %w", configErr)
|
||||
@@ -170,43 +167,32 @@ func FindAMIDependents(store objectstore.ObjectStore, bucket, imageID string) (D
|
||||
volSnapRefs[s] = true
|
||||
}
|
||||
|
||||
// Pass 2: volumes (vol-*/config.json) whose SnapshotID matches.
|
||||
for _, p := range prefixes {
|
||||
if !strings.HasPrefix(p, "vol-") {
|
||||
continue
|
||||
}
|
||||
volID := strings.TrimSuffix(p, "/")
|
||||
cfg, err := readVolumeConfig(store, bucket, volID)
|
||||
if err != nil {
|
||||
if objectstore.IsNoSuchKeyError(err) || errors.Is(err, errCorruptVolumeConfig) {
|
||||
continue
|
||||
}
|
||||
return Dependents{}, fmt.Errorf("read volume %s: %w", volID, err)
|
||||
}
|
||||
if volSnapRefs[cfg.VolumeMetadata.SnapshotID] {
|
||||
deps.Volumes = append(deps.Volumes, volID)
|
||||
// Pass 2: volumes whose SnapshotID matches. The ebsmetadata documents are
|
||||
// the whole fleet — a volume with blocks but no document does not exist as
|
||||
// far as the control plane is concerned — and they carry SnapshotID
|
||||
// already, so no second read per volume is needed.
|
||||
volDocs, err := ebsmetadata.NewStore(store, bucket).ListVolumes(context.Background())
|
||||
if err != nil {
|
||||
return Dependents{}, fmt.Errorf("list ebsmetadata volumes: %w", err)
|
||||
}
|
||||
for _, v := range volDocs {
|
||||
if volSnapRefs[v.SnapshotID] {
|
||||
deps.Volumes = append(deps.Volumes, v.VolumeID)
|
||||
}
|
||||
}
|
||||
|
||||
// Pass 3: AMIs (ami-*/config.json) whose SnapshotID is a derived snap.
|
||||
// Skip the target AMI itself.
|
||||
for _, p := range prefixes {
|
||||
if !strings.HasPrefix(p, "ami-") {
|
||||
// Pass 3: AMIs whose SnapshotID is a derived snap, skipping the target AMI
|
||||
// itself.
|
||||
docAMIs, err := ebsmetadata.NewStore(store, bucket).ListAMIs(context.Background())
|
||||
if err != nil {
|
||||
return Dependents{}, fmt.Errorf("list ebsmetadata AMIs: %w", err)
|
||||
}
|
||||
for _, a := range docAMIs {
|
||||
if a.ImageID == imageID {
|
||||
continue
|
||||
}
|
||||
otherAMI := strings.TrimSuffix(p, "/")
|
||||
if otherAMI == imageID {
|
||||
continue
|
||||
}
|
||||
meta, err := readAMIConfig(store, bucket, otherAMI)
|
||||
if err != nil {
|
||||
if objectstore.IsNoSuchKeyError(err) || errors.Is(err, handlers_ec2_image.ErrCorruptAMIConfig) {
|
||||
continue
|
||||
}
|
||||
return Dependents{}, fmt.Errorf("read AMI %s: %w", otherAMI, err)
|
||||
}
|
||||
if derived[meta.SnapshotID] {
|
||||
deps.AMIs = append(deps.AMIs, otherAMI)
|
||||
if derived[a.SnapshotID] {
|
||||
deps.AMIs = append(deps.AMIs, a.ImageID)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -221,9 +207,9 @@ func RemoveSystemImage(store objectstore.ObjectStore, bucket string, opts Remove
|
||||
return nil, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
|
||||
}
|
||||
|
||||
meta, configErr := readAMIConfig(store, bucket, opts.ImageID)
|
||||
meta, configErr := readAMI(store, bucket, opts.ImageID)
|
||||
configMissing := objectstore.IsNoSuchKeyError(configErr)
|
||||
configCorrupt := errors.Is(configErr, handlers_ec2_image.ErrCorruptAMIConfig)
|
||||
configCorrupt := errors.Is(configErr, ebsmetadata.ErrCorruptDocument)
|
||||
switch {
|
||||
case configErr == nil:
|
||||
// fine
|
||||
@@ -256,18 +242,18 @@ func RemoveSystemImage(store objectstore.ObjectStore, bucket string, opts Remove
|
||||
|
||||
result := &RemoveImageResult{}
|
||||
|
||||
// Step 1: drop config.json first — the barrier that hides the AMI from
|
||||
// DescribeImages so no new launches can land on the blocks we're deleting.
|
||||
if configErr == nil || (opts.Force && !configMissing) {
|
||||
n, b, err := deletePrefix(store, bucket, opts.ImageID+"/config.json")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("delete config: %w", err)
|
||||
}
|
||||
result.ObjectsDeleted += n
|
||||
result.BytesDeleted += b
|
||||
// Step 1: drop the ebsmetadata document first — the barrier that hides the
|
||||
// AMI from DescribeImages, so no new launch can land on blocks step 2 is
|
||||
// about to delete. Fatal, unlike the blocks: an AMI still advertised after
|
||||
// its blocks are gone is worse than one whose blocks outlive it.
|
||||
// An already-absent document is the barrier being satisfied, not a failure:
|
||||
// this is exactly the state of an orphaned image, whose blocks are the only
|
||||
// thing left to reclaim and the whole reason --force exists.
|
||||
if err := ebsmetadata.NewStore(store, bucket).DeleteAMI(context.Background(), opts.ImageID); err != nil && !objectstore.IsNoSuchKeyError(err) {
|
||||
return nil, fmt.Errorf("delete ebsmetadata document: %w", err)
|
||||
}
|
||||
|
||||
// Step 2: drop the rest of ami-<id>/ (chunks, WAL, checkpoints).
|
||||
// Step 2: drop ami-<id>/ (chunks, WAL, checkpoints).
|
||||
n, b, err := deletePrefix(store, bucket, opts.ImageID+"/")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("delete ami prefix: %w", err)
|
||||
@@ -304,62 +290,10 @@ func (e *DependentError) Error() string {
|
||||
e.ImageID, len(e.Dependents.Volumes), len(e.Dependents.Snapshots), len(e.Dependents.AMIs))
|
||||
}
|
||||
|
||||
// readAMIConfig reads ami-<id>/config.json and returns the AMIMetadata.
|
||||
// Mirrors ImageServiceImpl.GetAMIConfig but operates package-locally so the
|
||||
// admin tooling doesn't require an ImageServiceImpl (which carries NATS).
|
||||
func readAMIConfig(store objectstore.ObjectStore, bucket, imageID string) (viperblock.AMIMetadata, error) {
|
||||
key := imageID + "/config.json"
|
||||
res, err := store.GetObject(context.Background(), &s3.GetObjectInput{
|
||||
Bucket: aws.String(bucket),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
if err != nil {
|
||||
return viperblock.AMIMetadata{}, err
|
||||
}
|
||||
defer res.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(res.Body)
|
||||
if err != nil {
|
||||
return viperblock.AMIMetadata{}, err
|
||||
}
|
||||
|
||||
var state viperblock.VBState
|
||||
if err := json.Unmarshal(viperblock.StateBody(body), &state); err != nil {
|
||||
return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %w", handlers_ec2_image.ErrCorruptAMIConfig, key, err)
|
||||
}
|
||||
return state.VolumeConfig.AMIMetadata, nil
|
||||
}
|
||||
|
||||
// errCorruptVolumeConfig distinguishes an unparse-able config (walk continues)
|
||||
// from a transport error (walk fails closed to prevent deleting live blocks).
|
||||
var errCorruptVolumeConfig = errors.New("corrupt volume config")
|
||||
|
||||
// readVolumeConfig reads vol-<id>/config.json into VolumeConfig.
|
||||
type volumeConfigWrapper struct {
|
||||
VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
|
||||
}
|
||||
|
||||
func readVolumeConfig(store objectstore.ObjectStore, bucket, volumeID string) (*viperblock.VolumeConfig, error) {
|
||||
key := volumeID + "/config.json"
|
||||
res, err := store.GetObject(context.Background(), &s3.GetObjectInput{
|
||||
Bucket: aws.String(bucket),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer res.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(res.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var w volumeConfigWrapper
|
||||
if err := json.Unmarshal(viperblock.StateBody(body), &w); err != nil {
|
||||
return nil, fmt.Errorf("%w: %s: %w", errCorruptVolumeConfig, key, err)
|
||||
}
|
||||
return &w.VolumeConfig, nil
|
||||
// readAMI resolves an AMI's control-plane document. It exists so the admin
|
||||
// tooling can read an AMI without an ImageServiceImpl, which carries NATS.
|
||||
func readAMI(store objectstore.ObjectStore, bucket, imageID string) (ebsmetadata.AMI, error) {
|
||||
return ebsmetadata.NewStore(store, bucket).GetAMI(context.Background(), imageID)
|
||||
}
|
||||
|
||||
// listCommonPrefixes returns the top-level "directory" prefixes in the bucket
|
||||
|
||||
@@ -2,15 +2,15 @@ package admin
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
@@ -20,27 +20,30 @@ const (
|
||||
testRemoveAccountID = "000000000001"
|
||||
)
|
||||
|
||||
// putAMI writes an ami-<id>/config.json with the given owner. Owner "" produces
|
||||
// a corrupt config (empty ImageOwnerAlias is treated as corrupt by readers).
|
||||
// putAMI registers an AMI the only way the control plane knows one: as an
|
||||
// ebsmetadata document. It writes nothing under ami-<id>/, so an object count
|
||||
// taken over that prefix sees blocks alone.
|
||||
func putAMI(t *testing.T, store *objectstore.MemoryObjectStore, imageID, name, owner, snapshotID string) {
|
||||
t.Helper()
|
||||
state := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: viperblock.AMIMetadata{
|
||||
ImageID: imageID,
|
||||
Name: name,
|
||||
ImageOwnerAlias: owner,
|
||||
SnapshotID: snapshotID,
|
||||
VolumeSizeGiB: 8,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(state)
|
||||
require.NoError(t, ebsmetadata.NewStore(store, testRemoveBucket).PutAMI(t.Context(), ebsmetadata.AMI{
|
||||
ImageID: imageID,
|
||||
Name: name,
|
||||
ImageOwnerAlias: owner,
|
||||
SnapshotID: snapshotID,
|
||||
VolumeSizeGiB: 8,
|
||||
}))
|
||||
}
|
||||
|
||||
// putCorruptAMI writes an undecodable document at the AMI's key, the state the
|
||||
// salvage path exists for: present, so not not-found, but unreadable.
|
||||
func putCorruptAMI(t *testing.T, store *objectstore.MemoryObjectStore, imageID string) {
|
||||
t.Helper()
|
||||
key, err := ebsmetadata.AMIKey(imageID)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Body: bytes.NewReader(data),
|
||||
Key: aws.String(key),
|
||||
Body: bytes.NewReader([]byte("{not valid json")),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
@@ -86,26 +89,14 @@ func putSnapMetadata(t *testing.T, store *objectstore.MemoryObjectStore, snapID,
|
||||
require.NoError(t, handlers_ec2_snapshot.WriteSnapshotConfig(store, testRemoveBucket, snapID, cfg))
|
||||
}
|
||||
|
||||
// putVolume writes vol-<id>/config.json with the given SnapshotID reference.
|
||||
// putVolume registers a volume document carrying the given SnapshotID.
|
||||
func putVolume(t *testing.T, store *objectstore.MemoryObjectStore, volID, snapshotID string) {
|
||||
t.Helper()
|
||||
wrapper := volumeConfigWrapper{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: volID,
|
||||
SnapshotID: snapshotID,
|
||||
SizeGiB: 8,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(wrapper)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(volID + "/config.json"),
|
||||
Body: bytes.NewReader(data),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, ebsmetadata.NewStore(store, testRemoveBucket).PutVolume(t.Context(), ebsmetadata.Volume{
|
||||
VolumeID: volID,
|
||||
SnapshotID: snapshotID,
|
||||
CapacityGiB: 8,
|
||||
}))
|
||||
}
|
||||
|
||||
func TestRemoveSystemImage_HappyPath(t *testing.T) {
|
||||
@@ -120,8 +111,8 @@ func TestRemoveSystemImage_HappyPath(t *testing.T) {
|
||||
assert.True(t, preview.ConfigPresent)
|
||||
assert.True(t, preview.IsSystemOwned)
|
||||
assert.Equal(t, "system", preview.Owner)
|
||||
// config.json + 3 chunks = 4 ami-prefix objects.
|
||||
assert.Equal(t, 4, preview.AMIObjectCount)
|
||||
// Only the 3 chunks: the document lives outside the ami-<id>/ prefix.
|
||||
assert.Equal(t, 3, preview.AMIObjectCount)
|
||||
assert.Equal(t, 2, preview.SnapObjectCount)
|
||||
assert.True(t, preview.Dependents.Empty())
|
||||
|
||||
@@ -132,6 +123,39 @@ func TestRemoveSystemImage_HappyPath(t *testing.T) {
|
||||
assert.Equal(t, 0, store.Count())
|
||||
}
|
||||
|
||||
// strictDeleteStore reports a missing key on delete, the way predastore does.
|
||||
// The in-memory store is silently tolerant and AWS S3 answers 204, so neither
|
||||
// reproduces the backend this actually runs against.
|
||||
type strictDeleteStore struct {
|
||||
*objectstore.MemoryObjectStore
|
||||
}
|
||||
|
||||
var _ objectstore.ObjectStore = (*strictDeleteStore)(nil)
|
||||
|
||||
func (s *strictDeleteStore) DeleteObject(ctx context.Context, input *awss3.DeleteObjectInput) (*awss3.DeleteObjectOutput, error) {
|
||||
key := aws.StringValue(input.Key)
|
||||
if _, err := s.HeadObject(ctx, &awss3.HeadObjectInput{Bucket: input.Bucket, Key: input.Key}); err != nil {
|
||||
return nil, &objectstore.NoSuchKeyError{Key: key}
|
||||
}
|
||||
return s.MemoryObjectStore.DeleteObject(ctx, input)
|
||||
}
|
||||
|
||||
// TestRemoveSystemImage_OrphanedBlocksAreReclaimable covers the state the
|
||||
// orphan report points operators at: blocks in the object store with no
|
||||
// document. A live env19 run failed here on NoSuchKey, so the one documented
|
||||
// way to reclaim an orphan was broken by the very thing that defines one.
|
||||
func TestRemoveSystemImage_OrphanedBlocksAreReclaimable(t *testing.T) {
|
||||
memory := objectstore.NewMemoryObjectStore()
|
||||
store := &strictDeleteStore{MemoryObjectStore: memory}
|
||||
const id = "ami-orphaned0001"
|
||||
putAMIBlocks(t, memory, id, 3, 128)
|
||||
|
||||
res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
|
||||
require.NoError(t, err, "an image whose document is already gone must still have its blocks reclaimable")
|
||||
assert.Equal(t, 3, res.ObjectsDeleted)
|
||||
assert.Equal(t, 0, memory.Count(), "the stranded blocks must actually be released")
|
||||
}
|
||||
|
||||
func TestRemoveSystemImage_AccountOwned_Refused(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
const id = "ami-acct-001"
|
||||
@@ -173,14 +197,9 @@ func TestRemoveSystemImage_MissingConfig_NotFound(t *testing.T) {
|
||||
func TestRemoveSystemImage_CorruptConfig_NotFound(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
const id = "ami-corrupt"
|
||||
_, err := store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(id + "/config.json"),
|
||||
Body: bytes.NewReader([]byte("{not valid json")),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putCorruptAMI(t, store, id)
|
||||
|
||||
_, err = RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
|
||||
_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
|
||||
}
|
||||
@@ -245,16 +264,11 @@ func TestRemoveSystemImage_Force_OverridesDependents(t *testing.T) {
|
||||
res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
|
||||
require.NoError(t, err)
|
||||
assert.Positive(t, res.ObjectsDeleted)
|
||||
// vol-orphan/config.json remains; the AMI is gone.
|
||||
_, err = store.GetObject(t.Context(), &awss3.GetObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(id + "/config.json"),
|
||||
})
|
||||
// The dependent volume's document remains; the AMI's is gone.
|
||||
metadata := ebsmetadata.NewStore(store, testRemoveBucket)
|
||||
_, err = metadata.GetAMI(t.Context(), id)
|
||||
require.Error(t, err)
|
||||
_, err = store.GetObject(t.Context(), &awss3.GetObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String("vol-orphan/config.json"),
|
||||
})
|
||||
_, err = metadata.GetVolume(t.Context(), "vol-orphan")
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
@@ -280,23 +294,19 @@ func TestRemoveSystemImage_Salvage_MissingConfig_ForceCleans(t *testing.T) {
|
||||
func TestRemoveSystemImage_Salvage_CorruptConfig_ForceCleans(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
const id = "ami-salvage-2"
|
||||
_, err := store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(id + "/config.json"),
|
||||
Body: bytes.NewReader([]byte("garbage")),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putCorruptAMI(t, store, id)
|
||||
putAMIBlocks(t, store, id, 1, 8)
|
||||
|
||||
// Without --force.
|
||||
_, err = RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
|
||||
_, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id})
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
|
||||
|
||||
// With --force: corrupt config + blocks all go.
|
||||
// With --force: the corrupt document and the blocks all go. Only the
|
||||
// block is counted; the document is deleted by key, not by prefix walk.
|
||||
res, err := RemoveSystemImage(store, testRemoveBucket, RemoveImageOpts{ImageID: id, Force: true})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 2, res.ObjectsDeleted)
|
||||
assert.Equal(t, 1, res.ObjectsDeleted)
|
||||
assert.Equal(t, 0, store.Count())
|
||||
}
|
||||
|
||||
@@ -344,12 +354,7 @@ func TestPreviewRemoveSystemImage_Salvage(t *testing.T) {
|
||||
func TestPreviewRemoveSystemImage_CorruptConfig(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
const id = "ami-corrupt-preview"
|
||||
_, err := store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testRemoveBucket),
|
||||
Key: aws.String(id + "/config.json"),
|
||||
Body: bytes.NewReader([]byte("{nope")),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putCorruptAMI(t, store, id)
|
||||
|
||||
preview, err := PreviewRemoveSystemImage(store, testRemoveBucket, id)
|
||||
require.NoError(t, err)
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
package admin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
)
|
||||
|
||||
// efiVolumeSuffix marks a volume derived from another one. An EFI variable
|
||||
// store is created alongside its root volume and deliberately gets no
|
||||
// control-plane document, so it is only an orphan when its base is one too.
|
||||
const efiVolumeSuffix = "-efi"
|
||||
|
||||
// OrphanVolume is a volume the provider holds that the control plane has no
|
||||
// document for. Its blocks are consuming space with no API handle, so it can
|
||||
// be neither described nor deleted through EC2.
|
||||
type OrphanVolume struct {
|
||||
VolumeID string
|
||||
Handle string
|
||||
|
||||
// Derived marks a volume named after another one, such as an EFI variable
|
||||
// store. It is reported only when its base volume is also undocumented,
|
||||
// which means the pair was stranded together.
|
||||
Derived bool
|
||||
}
|
||||
|
||||
// FindOrphanVolumes reports what storage holds that the control plane cannot
|
||||
// see. It reads both sides independently: the provider for what exists, the
|
||||
// metadata store for what is known, and returns the difference.
|
||||
//
|
||||
// It never deletes. An undocumented volume is the case where deletion is most
|
||||
// tempting and most dangerous, because the evidence that it is still wanted is
|
||||
// exactly the document that went missing.
|
||||
func FindOrphanVolumes(ctx context.Context, provider ebsprovider.EBSProvider, metadata *ebsmetadata.Store) ([]OrphanVolume, error) {
|
||||
if provider == nil || metadata == nil {
|
||||
return nil, fmt.Errorf("orphan scan: provider and metadata store are both required")
|
||||
}
|
||||
|
||||
capabilities, err := provider.GetCapabilities(ctx, ebsprovider.GetCapabilitiesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("orphan scan: read provider capabilities: %w", err)
|
||||
}
|
||||
if !capabilities.Capabilities.VolumeEnumeration {
|
||||
return nil, fmt.Errorf("orphan scan: %w: provider does not enumerate volumes", ebsprovider.ErrUnsupportedCapability)
|
||||
}
|
||||
|
||||
// Strict listing on purpose: a document this cannot decode must abort the
|
||||
// scan, never be treated as absent. Reporting a volume as orphaned because
|
||||
// its document was unreadable would point an operator at live data.
|
||||
documented, err := metadata.ListVolumes(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("orphan scan: list control-plane volumes: %w", err)
|
||||
}
|
||||
known := make(map[string]bool, len(documented))
|
||||
for _, volume := range documented {
|
||||
known[volume.VolumeID] = true
|
||||
}
|
||||
|
||||
// An AMI's blocks live in a provider volume named after the image, and it
|
||||
// is recorded as an AMI document rather than a volume one. Without this
|
||||
// every registered image reads as an orphan.
|
||||
images, err := metadata.ListAMIsStrict(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("orphan scan: list control-plane images: %w", err)
|
||||
}
|
||||
for _, image := range images {
|
||||
known[image.ImageID] = true
|
||||
}
|
||||
|
||||
held, err := listProviderVolumes(ctx, provider)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var orphans []OrphanVolume
|
||||
for _, ref := range held {
|
||||
if known[ref.ID] {
|
||||
continue
|
||||
}
|
||||
base, derived := strings.CutSuffix(ref.ID, efiVolumeSuffix)
|
||||
if derived && known[base] {
|
||||
continue
|
||||
}
|
||||
orphans = append(orphans, OrphanVolume{VolumeID: ref.ID, Handle: ref.Handle, Derived: derived})
|
||||
}
|
||||
sort.Slice(orphans, func(i, j int) bool { return orphans[i].VolumeID < orphans[j].VolumeID })
|
||||
return orphans, nil
|
||||
}
|
||||
|
||||
// listProviderVolumes drains every page the provider offers. A partial walk
|
||||
// would under-report orphans, which is the one direction this must not fail
|
||||
// in: a missed orphan is a volume nobody ever goes looking for again.
|
||||
func listProviderVolumes(ctx context.Context, provider ebsprovider.EBSProvider) ([]ebsprovider.VolumeRef, error) {
|
||||
var refs []ebsprovider.VolumeRef
|
||||
var token string
|
||||
for {
|
||||
resp, err := provider.ListVolumes(ctx, ebsprovider.ListVolumesRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
StartingToken: token,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("orphan scan: list provider volumes: %w", err)
|
||||
}
|
||||
refs = append(refs, resp.Volumes...)
|
||||
if resp.NextToken == "" {
|
||||
return refs, nil
|
||||
}
|
||||
if resp.NextToken == token {
|
||||
return nil, fmt.Errorf("orphan scan: provider returned a non-advancing page token %q", token)
|
||||
}
|
||||
token = resp.NextToken
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,190 @@
|
||||
package admin
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const orphanTestBucket = "test-bucket"
|
||||
|
||||
func newOrphanScanFixture(t *testing.T) (*ebsprovider.MemoryProvider, *ebsmetadata.Store) {
|
||||
t.Helper()
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeEnumeration: true, VolumeSeeding: true})
|
||||
return provider, ebsmetadata.NewStore(objectstore.NewMemoryObjectStore(), orphanTestBucket)
|
||||
}
|
||||
|
||||
// createProviderVolume gives the provider blocks without telling the control
|
||||
// plane, which is what a volume looks like after its document is lost.
|
||||
func createProviderVolume(t *testing.T, provider *ebsprovider.MemoryProvider, volumeID string) {
|
||||
t.Helper()
|
||||
_, err := provider.CreateVolume(t.Context(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func documentVolume(t *testing.T, metadata *ebsmetadata.Store, volumeID string) {
|
||||
t.Helper()
|
||||
require.NoError(t, metadata.PutVolume(t.Context(), ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: "123456789012", CapacityGiB: 1, State: "available",
|
||||
}))
|
||||
}
|
||||
|
||||
// putRawVolumeDocument writes bytes straight to a volume's document key,
|
||||
// so a test can plant one the store cannot decode.
|
||||
func putRawVolumeDocument(t *testing.T, store objectstore.ObjectStore, volumeID string, body []byte) {
|
||||
t.Helper()
|
||||
key, err := ebsmetadata.VolumeKey(volumeID)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(t.Context(), &s3.PutObjectInput{
|
||||
Bucket: aws.String(orphanTestBucket),
|
||||
Key: aws.String(key),
|
||||
Body: bytes.NewReader(body),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// putRawAMIDocument writes bytes straight to an AMI's document key.
|
||||
func putRawAMIDocument(t *testing.T, store objectstore.ObjectStore, imageID string, body []byte) {
|
||||
t.Helper()
|
||||
key, err := ebsmetadata.AMIKey(imageID)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(t.Context(), &s3.PutObjectInput{
|
||||
Bucket: aws.String(orphanTestBucket),
|
||||
Key: aws.String(key),
|
||||
Body: bytes.NewReader(body),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// TestFindOrphanVolumes_FindsAVolumeWhoseDocumentIsGone is the reason the
|
||||
// enumeration verb exists. Deleting the document is exactly the failure the
|
||||
// consolidated boundary made unrecoverable: the volume vanishes from
|
||||
// DescribeVolumes and from the leak reaper, both of which read documents.
|
||||
func TestFindOrphanVolumes_FindsAVolumeWhoseDocumentIsGone(t *testing.T) {
|
||||
provider, metadata := newOrphanScanFixture(t)
|
||||
createProviderVolume(t, provider, "vol-documented000")
|
||||
documentVolume(t, metadata, "vol-documented000")
|
||||
createProviderVolume(t, provider, "vol-stranded00000")
|
||||
|
||||
orphans, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, orphans, 1, "the scan must report exactly the volume with no document")
|
||||
assert.Equal(t, "vol-stranded00000", orphans[0].VolumeID,
|
||||
"a volume the provider holds with no control-plane document is unreachable through EC2 and must be surfaced")
|
||||
assert.False(t, orphans[0].Derived)
|
||||
}
|
||||
|
||||
// TestFindOrphanVolumes_NeverDeletes locks ADR-0005 §3 for this tool: an
|
||||
// orphan carries data, and the evidence it is still wanted is the very
|
||||
// document that is missing. The scan may only report.
|
||||
func TestFindOrphanVolumes_NeverDeletes(t *testing.T) {
|
||||
provider, metadata := newOrphanScanFixture(t)
|
||||
createProviderVolume(t, provider, "vol-stranded00000")
|
||||
|
||||
orphans, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, orphans, 1)
|
||||
|
||||
got, err := provider.GetVolume(t.Context(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-stranded00000",
|
||||
})
|
||||
require.NoErrorf(t, err, "ADR-0005 §3: the orphan scan must never delete the volume it reports")
|
||||
assert.Equal(t, "vol-stranded00000", got.ID)
|
||||
}
|
||||
|
||||
// An EFI variable store is created without a document by design, so it is only
|
||||
// an orphan when the root volume it belongs to is also undocumented.
|
||||
func TestFindOrphanVolumes_DerivedVolumeExcusedByItsBase(t *testing.T) {
|
||||
provider, metadata := newOrphanScanFixture(t)
|
||||
createProviderVolume(t, provider, "vol-live00000000")
|
||||
createProviderVolume(t, provider, "vol-live00000000-efi")
|
||||
documentVolume(t, metadata, "vol-live00000000")
|
||||
|
||||
orphans, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, orphans, "an EFI store whose root volume is documented is expected, not orphaned")
|
||||
}
|
||||
|
||||
func TestFindOrphanVolumes_DerivedVolumeReportedWhenItsBaseIsGone(t *testing.T) {
|
||||
provider, metadata := newOrphanScanFixture(t)
|
||||
createProviderVolume(t, provider, "vol-gone00000000")
|
||||
createProviderVolume(t, provider, "vol-gone00000000-efi")
|
||||
|
||||
orphans, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, orphans, 2, "a stranded pair must both be reported; the EFI store's lifecycle owner is gone too")
|
||||
assert.Equal(t, "vol-gone00000000", orphans[0].VolumeID)
|
||||
assert.Equal(t, "vol-gone00000000-efi", orphans[1].VolumeID)
|
||||
assert.True(t, orphans[1].Derived)
|
||||
}
|
||||
|
||||
// An AMI's blocks live in a provider volume named after the image, recorded as
|
||||
// an AMI document rather than a volume one. A live env19 run reported every
|
||||
// registered image as an orphan before this was handled.
|
||||
func TestFindOrphanVolumes_AMIBackedVolumeIsNotAnOrphan(t *testing.T) {
|
||||
provider, metadata := newOrphanScanFixture(t)
|
||||
createProviderVolume(t, provider, "ami-dd7f063caded1ff82")
|
||||
require.NoError(t, metadata.PutAMI(t.Context(), ebsmetadata.AMI{
|
||||
ImageID: "ami-dd7f063caded1ff82", Name: "ubuntu", State: "available",
|
||||
}))
|
||||
|
||||
orphans, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, orphans, "a registered AMI's backing volume is documented as an image, not as an orphan")
|
||||
}
|
||||
|
||||
// An undecodable AMI document must abort the scan for the same reason an
|
||||
// undecodable volume document does: absence and unreadability are different
|
||||
// answers, and only one of them means orphaned.
|
||||
func TestFindOrphanVolumes_UndecodableAMIDocumentAbortsTheScan(t *testing.T) {
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeEnumeration: true})
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
metadata := ebsmetadata.NewStore(store, orphanTestBucket)
|
||||
createProviderVolume(t, provider, "ami-corrupted000")
|
||||
putRawAMIDocument(t, store, "ami-corrupted000", []byte("{not json"))
|
||||
|
||||
_, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
// An unreadable document must abort the scan, never read as absent: reporting
|
||||
// a volume as orphaned because its document failed to decode would point an
|
||||
// operator at live data.
|
||||
func TestFindOrphanVolumes_UndecodableDocumentAbortsTheScan(t *testing.T) {
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeEnumeration: true})
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
metadata := ebsmetadata.NewStore(store, orphanTestBucket)
|
||||
createProviderVolume(t, provider, "vol-corruptdoc00")
|
||||
putRawVolumeDocument(t, store, "vol-corruptdoc00", []byte("{not json"))
|
||||
|
||||
_, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.Error(t, err, "an undecodable document must fail the scan, not be treated as a missing one")
|
||||
}
|
||||
|
||||
func TestFindOrphanVolumes_RequiresEnumerationCapability(t *testing.T) {
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
metadata := ebsmetadata.NewStore(objectstore.NewMemoryObjectStore(), orphanTestBucket)
|
||||
|
||||
_, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedCapability)
|
||||
}
|
||||
|
||||
func TestFindOrphanVolumes_EmptyProviderReportsNothing(t *testing.T) {
|
||||
provider, metadata := newOrphanScanFixture(t)
|
||||
orphans, err := FindOrphanVolumes(t.Context(), provider, metadata)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, orphans)
|
||||
}
|
||||
@@ -100,6 +100,7 @@ type Config struct {
|
||||
NATS NATSConfig `json:"NATS" mapstructure:"nats"`
|
||||
Predastore PredastoreConfig `json:"Predastore" mapstructure:"predastore"`
|
||||
Viperblock ViperblockConfig `json:"Viperblock" mapstructure:"viperblock"`
|
||||
EBS EBSConfig `json:"EBS" mapstructure:"ebs"`
|
||||
AWSGW AWSGWConfig `json:"AWSGW" mapstructure:"awsgw"`
|
||||
VPCD VPCDConfig `json:"VPCD" mapstructure:"vpcd"`
|
||||
Northstar NorthstarConfig `json:"Northstar" mapstructure:"northstar"`
|
||||
@@ -136,6 +137,35 @@ type ViperblockConfig struct {
|
||||
GCEnabled *bool `json:"GCEnabled" mapstructure:"gc_enabled"`
|
||||
}
|
||||
|
||||
// EBS provider selectors. Viperblockd routes EBS calls to the viperblockd
|
||||
// daemon and qemunbd to the qemunbdd daemon, both over the versioned
|
||||
// ebs.provider.v1.* NATS contract. Embedded named the removed in-process
|
||||
// engine and survives solely so a config that still selects it can be
|
||||
// rejected by name.
|
||||
const (
|
||||
EBSProviderEmbedded = "embedded"
|
||||
EBSProviderViperblockd = "viperblockd"
|
||||
EBSProviderQEMUNBD = "qemunbd"
|
||||
)
|
||||
|
||||
// EBSConfig selects which provider backs EBS. Not nested under
|
||||
// ViperblockConfig: it names the provider boundary, not one provider's
|
||||
// settings, so a second provider never needs a rename.
|
||||
type EBSConfig struct {
|
||||
// Provider is "viperblockd" or "qemunbd" and may be left unset. Volumes
|
||||
// are persisted in ebsmetadata.
|
||||
Provider string `json:"Provider" mapstructure:"provider"`
|
||||
}
|
||||
|
||||
// ResolvedProvider normalizes an empty Provider to EBSProviderViperblockd,
|
||||
// the default provider.
|
||||
func (c EBSConfig) ResolvedProvider() string {
|
||||
if c.Provider == "" {
|
||||
return EBSProviderViperblockd
|
||||
}
|
||||
return c.Provider
|
||||
}
|
||||
|
||||
// VPCDConfig holds the VPC daemon (vpcd) configuration.
|
||||
type VPCDConfig struct {
|
||||
OVNNBAddr string `json:"OVNNBAddr" mapstructure:"ovn_nb_addr"` // OVN Northbound DB address; comma-separated list for a RAFT cluster (e.g., "tcp:127.0.0.1:6641" or "tcp:ip1:6641,tcp:ip2:6641,tcp:ip3:6641")
|
||||
@@ -414,10 +444,20 @@ func validateClusterConfig(cc *ClusterConfig) error {
|
||||
if viper.IsSet("network.external_dhcp") {
|
||||
return fmt.Errorf("config: [network] external_dhcp is no longer supported; remove the key (static WAN-pool allocation only)")
|
||||
}
|
||||
for nodeName := range cc.Nodes {
|
||||
for nodeName, nodeCfg := range cc.Nodes {
|
||||
if viper.IsSet("nodes." + nodeName + ".vpcd.dhcp_bind_bridge") {
|
||||
return fmt.Errorf("config: [nodes.%s.vpcd] dhcp_bind_bridge is no longer supported; remove the key (vpcd no longer runs a DHCP client)", nodeName)
|
||||
}
|
||||
switch nodeCfg.EBS.Provider {
|
||||
case "", EBSProviderViperblockd, EBSProviderQEMUNBD:
|
||||
case EBSProviderEmbedded:
|
||||
// Refuse rather than silently upgrade: the embedded engine is gone,
|
||||
// and starting anyway would migrate this node's volumes to
|
||||
// ebsmetadata one-way without the operator ever asking for it.
|
||||
return fmt.Errorf("config: [nodes.%s.ebs] provider=%q has been removed; set provider = %q or remove the key (this is a one-way switch: volumes move to ebsmetadata)", nodeName, EBSProviderEmbedded, EBSProviderViperblockd)
|
||||
default:
|
||||
return fmt.Errorf("config: [nodes.%s.ebs] provider=%q unsupported; use %q or %q, or remove the key", nodeName, nodeCfg.EBS.Provider, EBSProviderViperblockd, EBSProviderQEMUNBD)
|
||||
}
|
||||
}
|
||||
|
||||
if len(cc.Network.NATExemptCIDRs) > 0 && cc.Network.ExternalMode != "nat" {
|
||||
|
||||
@@ -944,3 +944,95 @@ ovn_nb_addr = "tcp:127.0.0.1:6641"
|
||||
assert.Contains(t, err.Error(), `dhcp_mac is only valid with source="dhcp"`)
|
||||
})
|
||||
}
|
||||
|
||||
func TestEBSConfig_ResolvedProvider(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
in string
|
||||
want string
|
||||
}{
|
||||
{"empty defaults to viperblockd", "", EBSProviderViperblockd},
|
||||
{"embedded passes through unresolved", EBSProviderEmbedded, EBSProviderEmbedded},
|
||||
{"viperblockd passes through", EBSProviderViperblockd, EBSProviderViperblockd},
|
||||
{"qemunbd passes through", EBSProviderQEMUNBD, EBSProviderQEMUNBD},
|
||||
{"unknown value passes through unresolved", "bogus", "bogus"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
cfg := EBSConfig{Provider: tc.in}
|
||||
assert.Equal(t, tc.want, cfg.ResolvedProvider())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadConfig_EBSProvider_Valid(t *testing.T) {
|
||||
base := `
|
||||
node = "n1"
|
||||
|
||||
[nodes.n1]
|
||||
region = "us-east-1"
|
||||
%s
|
||||
`
|
||||
cases := []struct {
|
||||
name string
|
||||
section string
|
||||
wantResolved string
|
||||
}{
|
||||
{"unset defaults to viperblockd", "", EBSProviderViperblockd},
|
||||
{"viperblockd explicit", "[nodes.n1.ebs]\nprovider = \"viperblockd\"\n", EBSProviderViperblockd},
|
||||
{"qemunbd explicit", "[nodes.n1.ebs]\nprovider = \"qemunbd\"\n", EBSProviderQEMUNBD},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
resetViper(t)
|
||||
path := filepath.Join(t.TempDir(), "spinifex.toml")
|
||||
require.NoError(t, os.WriteFile(path, fmt.Appendf(nil, base, tc.section), 0600))
|
||||
|
||||
cfg, err := LoadConfig(path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, tc.wantResolved, cfg.Nodes["n1"].EBS.ResolvedProvider())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A deployed spinifex.toml that still names the removed embedded provider must
|
||||
// fail loudly at load: starting anyway would migrate the node's volumes to
|
||||
// ebsmetadata one-way without the operator having asked for it.
|
||||
func TestLoadConfig_EBSProvider_EmbeddedRejected(t *testing.T) {
|
||||
resetViper(t)
|
||||
path := filepath.Join(t.TempDir(), "spinifex.toml")
|
||||
toml := `
|
||||
node = "n1"
|
||||
|
||||
[nodes.n1]
|
||||
region = "us-east-1"
|
||||
|
||||
[nodes.n1.ebs]
|
||||
provider = "embedded"
|
||||
`
|
||||
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
|
||||
|
||||
_, err := LoadConfig(path)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), `provider="embedded" has been removed`)
|
||||
assert.Contains(t, err.Error(), `viperblockd`)
|
||||
}
|
||||
|
||||
func TestLoadConfig_EBSProvider_UnknownRejected(t *testing.T) {
|
||||
resetViper(t)
|
||||
path := filepath.Join(t.TempDir(), "spinifex.toml")
|
||||
toml := `
|
||||
node = "n1"
|
||||
|
||||
[nodes.n1]
|
||||
region = "us-east-1"
|
||||
|
||||
[nodes.n1.ebs]
|
||||
provider = "bogus"
|
||||
`
|
||||
require.NoError(t, os.WriteFile(path, []byte(toml), 0600))
|
||||
|
||||
_, err := LoadConfig(path)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), `provider="bogus"`)
|
||||
}
|
||||
|
||||
+66
-13
@@ -32,6 +32,7 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/admin"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/gpu"
|
||||
handlers_acm "github.com/mulgadc/spinifex/spinifex/handlers/acm"
|
||||
handlers_bedrock "github.com/mulgadc/spinifex/spinifex/handlers/bedrock"
|
||||
@@ -124,19 +125,21 @@ var _ handlers_ec2_igw.GatePublisher = (*handlers_ec2_routetable.RouteTableServi
|
||||
|
||||
// Daemon represents the main daemon service.
|
||||
type Daemon struct {
|
||||
node string
|
||||
clusterConfig *config.ClusterConfig
|
||||
config *config.Config
|
||||
natsConn *nats.Conn
|
||||
resourceMgr *ResourceManager
|
||||
instanceService *handlers_ec2_instance.InstanceServiceImpl
|
||||
dnsWriter *handlers_dns.Writer
|
||||
dnsReconciler *handlers_dns.Reconciler
|
||||
dnsBaseDomain string
|
||||
dnsInternalDomain string
|
||||
keyService *handlers_ec2_key.KeyServiceImpl
|
||||
imageService *handlers_ec2_image.ImageServiceImpl
|
||||
volumeService *handlers_ec2_volume.VolumeServiceImpl
|
||||
node string
|
||||
clusterConfig *config.ClusterConfig
|
||||
config *config.Config
|
||||
natsConn *nats.Conn
|
||||
resourceMgr *ResourceManager
|
||||
instanceService *handlers_ec2_instance.InstanceServiceImpl
|
||||
dnsWriter *handlers_dns.Writer
|
||||
dnsReconciler *handlers_dns.Reconciler
|
||||
dnsBaseDomain string
|
||||
dnsInternalDomain string
|
||||
keyService *handlers_ec2_key.KeyServiceImpl
|
||||
imageService *handlers_ec2_image.ImageServiceImpl
|
||||
volumeService *handlers_ec2_volume.VolumeServiceImpl
|
||||
// ebsProvider is the sole EBS backend, set once during startup.
|
||||
ebsProvider ebsprovider.EBSProvider
|
||||
accountService *handlers_ec2_account.AccountSettingsServiceImpl
|
||||
snapshotService *handlers_ec2_snapshot.SnapshotServiceImpl
|
||||
tagsService *handlers_ec2_tags.TagsServiceImpl
|
||||
@@ -1435,6 +1438,9 @@ func (d *Daemon) startCluster() error {
|
||||
d.snapshotService = snap.svc
|
||||
|
||||
d.volumeService = handlers_ec2_volume.NewVolumeServiceImpl(d.config, d.natsConn, snap.kv)
|
||||
if err := d.configureEBSProvider(); err != nil {
|
||||
return fmt.Errorf("configure EBS provider: %w", err)
|
||||
}
|
||||
d.tagsService = handlers_ec2_tags.NewTagsServiceImpl(d.config)
|
||||
|
||||
d.eigwService, err = initServiceWithRetry("EIGW service", func() (*handlers_ec2_eigw.EgressOnlyIGWServiceImpl, error) {
|
||||
@@ -1887,6 +1893,9 @@ func (d *Daemon) startCluster() error {
|
||||
if eniRec := d.newENIReconciler(); eniRec != nil {
|
||||
reapers = append(reapers, eniRec)
|
||||
}
|
||||
if eniOrphan := d.newENIOrphanReaper(); eniOrphan != nil {
|
||||
reapers = append(reapers, eniOrphan)
|
||||
}
|
||||
if gpuRec := d.newGPUPoolReconciler(); gpuRec != nil {
|
||||
reapers = append(reapers, gpuRec)
|
||||
}
|
||||
@@ -1920,6 +1929,50 @@ func (d *Daemon) startCluster() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// ebsProviderRequestTimeout bounds each ebs.provider.v1.* NATS request/reply.
|
||||
// Passed explicitly rather than relying on NATSProvider's own 30s default so
|
||||
// the daemon's behavior doesn't silently drift if that default ever changes.
|
||||
const ebsProviderRequestTimeout = 30 * time.Second
|
||||
|
||||
// ebsProviderProbeTimeout bounds the startup reachability check. Short because
|
||||
// a missing responder must be reported promptly, and the check is advisory.
|
||||
const ebsProviderProbeTimeout = 2 * time.Second
|
||||
|
||||
// configureEBSProvider wires a single NATSProvider into every EBS-adjacent
|
||||
// service. The control plane never constructs a provider implementation
|
||||
// itself, so this always runs regardless of which daemon answers on the wire.
|
||||
func (d *Daemon) configureEBSProvider() error {
|
||||
provider := ebsprovider.NewNATSProvider(d.natsConn, ebsProviderRequestTimeout)
|
||||
d.ebsProvider = provider
|
||||
d.instanceService.SetEBSProvider(provider)
|
||||
d.imageService.SetEBSProvider(provider)
|
||||
d.snapshotService.SetEBSProvider(provider)
|
||||
d.volumeService.SetEBSProvider(provider)
|
||||
slog.Info("EBS provider path active", "provider", d.config.EBS.ResolvedProvider())
|
||||
d.probeEBSProvider(provider)
|
||||
return nil
|
||||
}
|
||||
|
||||
// probeEBSProvider reports whether anything is serving the provider contract.
|
||||
// Advisory rather than fatal: this daemon and viperblockd start independently,
|
||||
// so an unanswered probe here can simply mean viperblockd has not come up yet.
|
||||
func (d *Daemon) probeEBSProvider(provider ebsprovider.EBSProvider) {
|
||||
// Own base context rather than the daemon's: the probe is bounded well
|
||||
// below any shutdown deadline, and startCluster runs before d.ctx is set
|
||||
// on some construction paths.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), ebsProviderProbeTimeout)
|
||||
defer cancel()
|
||||
|
||||
capabilities, err := provider.GetCapabilities(ctx, ebsprovider.GetCapabilitiesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
if err != nil {
|
||||
slog.Warn("EBS provider did not answer the capability probe; EBS calls will fail until it does",
|
||||
"provider", d.config.EBS.ResolvedProvider(), "err", err)
|
||||
return
|
||||
}
|
||||
slog.Info("EBS provider reachable", "provider", d.config.EBS.ResolvedProvider(),
|
||||
"capabilities", capabilities.Capabilities)
|
||||
}
|
||||
|
||||
// clusterSweepLease gates the GC's cluster-wide reapers so exactly one node runs
|
||||
// them. The RDS reconciler's lease is that election — cluster-singular, held
|
||||
// continuously, and re-evaluated on its own ticker — so being its holder is the
|
||||
|
||||
@@ -27,6 +27,13 @@ func (d *Daemon) handleSpinifexPromoteImage(msg *nats.Msg) {
|
||||
handleNATSRequest(promoteImage)(msg)
|
||||
}
|
||||
|
||||
// stoppedInstanceOwnerElector is implemented by state stores that can
|
||||
// atomically mutate a stopped instance record. Checked via assertion so
|
||||
// vm.StateStore itself need not widen for this one call site.
|
||||
type stoppedInstanceOwnerElector interface {
|
||||
UpdateStoppedInstance(id string, mutate func(*vm.VM)) (*vm.VM, error)
|
||||
}
|
||||
|
||||
// handleEC2CreateImage is a stateful handler that extracts instance context
|
||||
// (root volume ID, source AMI, running state) before delegating to the image service.
|
||||
func (d *Daemon) handleEC2CreateImage(msg *nats.Msg) {
|
||||
@@ -88,6 +95,40 @@ func (d *Daemon) handleEC2CreateImage(msg *nats.Msg) {
|
||||
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
// The KV is cluster-shared, so every node's lookup above succeeds;
|
||||
// only LastNode picks a single owner. Non-owners decline like the
|
||||
// miss above — Gather only short-circuits on success, so this can't
|
||||
// preempt the owner's reply, it just stops a duplicate AMI/snapshot.
|
||||
if stopped.LastNode == "" {
|
||||
// Pre-LastNode record from an older build. Elect an owner via
|
||||
// CAS so exactly one node wins, instead of every node
|
||||
// proceeding (the leak) or every node declining forever.
|
||||
elector, canElect := d.stateStore.(stoppedInstanceOwnerElector)
|
||||
if !canElect {
|
||||
slog.Error("CreateImage: state store cannot elect owner for legacy stopped instance", "instanceId", instanceID)
|
||||
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
|
||||
return
|
||||
}
|
||||
elected, uerr := elector.UpdateStoppedInstance(instanceID, func(v *vm.VM) {
|
||||
if v.LastNode == "" {
|
||||
v.LastNode = d.node
|
||||
}
|
||||
})
|
||||
if uerr != nil || elected == nil || elected.LastNode == "" {
|
||||
slog.Warn("CreateImage: failed to elect owner for legacy stopped instance", "instanceId", instanceID, "err", uerr)
|
||||
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
|
||||
return
|
||||
}
|
||||
stopped.LastNode = elected.LastNode
|
||||
}
|
||||
if stopped.LastNode != d.node {
|
||||
slog.Info("CreateImage: declining, instance owned by another node",
|
||||
"instanceId", instanceID, "lastNode", stopped.LastNode)
|
||||
respondWithError(msg, awserrors.ErrorInvalidInstanceIDNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
instance = stopped
|
||||
status = stopped.Status
|
||||
if stopped.Instance != nil {
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
package daemon
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// Tests in this file cover handleEC2CreateImage's stopped-instance LastNode
|
||||
// gate. The image service always fails downstream here, so "not NotFound"
|
||||
// proves a node passed the gate, and NotFound proves it declined first.
|
||||
|
||||
// seedStoppedInstance writes a stopped VM directly to the daemon's shared KV
|
||||
// so handleEC2CreateImage's vmMgr lookup misses and falls through to the
|
||||
// LastNode-gated KV branch.
|
||||
func seedStoppedInstance(t *testing.T, daemon *Daemon, id, lastNode string) {
|
||||
t.Helper()
|
||||
stoppedVM := &vm.VM{
|
||||
ID: id,
|
||||
Status: vm.StateStopped,
|
||||
AccountID: testAccountID,
|
||||
LastNode: lastNode,
|
||||
Instance: &ec2.Instance{
|
||||
InstanceId: aws.String(id),
|
||||
ImageId: aws.String("ami-source-stopped"),
|
||||
BlockDeviceMappings: []*ec2.InstanceBlockDeviceMapping{
|
||||
{
|
||||
Ebs: &ec2.EbsInstanceBlockDevice{VolumeId: aws.String("vol-" + id)},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
err := daemon.jsManager.WriteStoppedInstance(id, stoppedVM)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = daemon.jsManager.DeleteStoppedInstance(id) })
|
||||
}
|
||||
|
||||
// TestHandleEC2CreateImage_StoppedInstance_NonOwnerDeclines checks that a
|
||||
// node which is not the stopped instance's LastNode declines exactly like
|
||||
// an unknown instance ID, instead of proceeding into the image service.
|
||||
func TestHandleEC2CreateImage_StoppedInstance_NonOwnerDeclines(t *testing.T) {
|
||||
natsURL := sharedJSNATSURL
|
||||
daemon := createFullTestDaemonWithJetStream(t, natsURL)
|
||||
// createTestDaemon fixes the cluster config's own node at "node-1".
|
||||
seedStoppedInstance(t, daemon, "i-stopped-nonowner", "node-2")
|
||||
|
||||
sub, err := daemon.natsConn.Subscribe("ec2.CreateImage", daemon.handleEC2CreateImage)
|
||||
require.NoError(t, err)
|
||||
defer sub.Unsubscribe()
|
||||
|
||||
input := &ec2.CreateImageInput{
|
||||
InstanceId: aws.String("i-stopped-nonowner"),
|
||||
Name: aws.String("efi-verify-stopped-image"),
|
||||
}
|
||||
reqData, _ := json.Marshal(input)
|
||||
reply, err := natsRequest(daemon.natsConn, "ec2.CreateImage", reqData, 5*time.Second)
|
||||
require.NoError(t, err)
|
||||
|
||||
var errResp map[string]any
|
||||
require.NoError(t, json.Unmarshal(reply.Data, &errResp))
|
||||
assert.Equal(t, awserrors.ErrorInvalidInstanceIDNotFound, errResp["Code"],
|
||||
"node-1 is not LastNode (node-2) — it must decline instead of creating a duplicate AMI/snapshot")
|
||||
}
|
||||
|
||||
// TestHandleEC2CreateImage_StoppedInstance_OwnerProceeds checks that the
|
||||
// node recorded as LastNode is let through the ownership gate into the
|
||||
// image service, rather than declining with NotFound.
|
||||
func TestHandleEC2CreateImage_StoppedInstance_OwnerProceeds(t *testing.T) {
|
||||
natsURL := sharedJSNATSURL
|
||||
daemon := createFullTestDaemonWithJetStream(t, natsURL)
|
||||
seedStoppedInstance(t, daemon, "i-stopped-owner", daemon.node)
|
||||
|
||||
sub, err := daemon.natsConn.Subscribe("ec2.CreateImage", daemon.handleEC2CreateImage)
|
||||
require.NoError(t, err)
|
||||
defer sub.Unsubscribe()
|
||||
|
||||
input := &ec2.CreateImageInput{
|
||||
InstanceId: aws.String("i-stopped-owner"),
|
||||
Name: aws.String("efi-verify-stopped-image"),
|
||||
}
|
||||
reqData, _ := json.Marshal(input)
|
||||
reply, err := natsRequest(daemon.natsConn, "ec2.CreateImage", reqData, 5*time.Second)
|
||||
require.NoError(t, err)
|
||||
|
||||
var errResp map[string]any
|
||||
require.NoError(t, json.Unmarshal(reply.Data, &errResp))
|
||||
assert.NotEqual(t, awserrors.ErrorInvalidInstanceIDNotFound, errResp["Code"],
|
||||
"node-1 is LastNode — it must be allowed to proceed into the image service")
|
||||
}
|
||||
|
||||
// TestHandleEC2CreateImage_RunningInstance_LastNodeIgnored checks the
|
||||
// running-instance path stays governed solely by vmMgr's local-map filter —
|
||||
// a stale VM.LastNode left over from a prior stop must not cause a decline.
|
||||
func TestHandleEC2CreateImage_RunningInstance_LastNodeIgnored(t *testing.T) {
|
||||
natsURL := sharedNATSURL
|
||||
daemon := createFullTestDaemon(t, natsURL)
|
||||
|
||||
daemon.vmMgr.Insert(&vm.VM{
|
||||
ID: "i-running-stale-lastnode",
|
||||
Status: vm.StateRunning,
|
||||
AccountID: testAccountID,
|
||||
LastNode: "node-99", // stale value from a previous stop; must be ignored
|
||||
Instance: &ec2.Instance{
|
||||
InstanceId: aws.String("i-running-stale-lastnode"),
|
||||
ImageId: aws.String("ami-source"),
|
||||
BlockDeviceMappings: []*ec2.InstanceBlockDeviceMapping{
|
||||
{
|
||||
Ebs: &ec2.EbsInstanceBlockDevice{VolumeId: aws.String("vol-running123")},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
sub, err := daemon.natsConn.Subscribe("ec2.CreateImage", daemon.handleEC2CreateImage)
|
||||
require.NoError(t, err)
|
||||
defer sub.Unsubscribe()
|
||||
|
||||
input := &ec2.CreateImageInput{
|
||||
InstanceId: aws.String("i-running-stale-lastnode"),
|
||||
Name: aws.String("my-image"),
|
||||
}
|
||||
reqData, _ := json.Marshal(input)
|
||||
reply, err := natsRequest(daemon.natsConn, "ec2.CreateImage", reqData, 5*time.Second)
|
||||
require.NoError(t, err)
|
||||
|
||||
var errResp map[string]any
|
||||
require.NoError(t, json.Unmarshal(reply.Data, &errResp))
|
||||
assert.NotEqual(t, awserrors.ErrorInvalidInstanceIDNotFound, errResp["Code"],
|
||||
"running instance found locally in vmMgr must proceed regardless of a stale VM.LastNode")
|
||||
}
|
||||
|
||||
// TestHandleEC2CreateImage_StoppedInstance_EmptyLastNodeElectsSelf checks a
|
||||
// pre-LastNode legacy record: the requesting node self-elects via CAS
|
||||
// instead of proceeding unconditionally or declining forever.
|
||||
func TestHandleEC2CreateImage_StoppedInstance_EmptyLastNodeElectsSelf(t *testing.T) {
|
||||
natsURL := sharedJSNATSURL
|
||||
daemon := createFullTestDaemonWithJetStream(t, natsURL)
|
||||
seedStoppedInstance(t, daemon, "i-stopped-legacy", "") // no LastNode, as an older build would leave it
|
||||
|
||||
sub, err := daemon.natsConn.Subscribe("ec2.CreateImage", daemon.handleEC2CreateImage)
|
||||
require.NoError(t, err)
|
||||
defer sub.Unsubscribe()
|
||||
|
||||
input := &ec2.CreateImageInput{
|
||||
InstanceId: aws.String("i-stopped-legacy"),
|
||||
Name: aws.String("efi-verify-stopped-image"),
|
||||
}
|
||||
reqData, _ := json.Marshal(input)
|
||||
reply, err := natsRequest(daemon.natsConn, "ec2.CreateImage", reqData, 5*time.Second)
|
||||
require.NoError(t, err)
|
||||
|
||||
var errResp map[string]any
|
||||
require.NoError(t, json.Unmarshal(reply.Data, &errResp))
|
||||
assert.NotEqual(t, awserrors.ErrorInvalidInstanceIDNotFound, errResp["Code"],
|
||||
"the requesting node must self-elect and proceed rather than decline forever")
|
||||
|
||||
updated, err := daemon.jsManager.LoadStoppedInstance("i-stopped-legacy")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, updated)
|
||||
assert.Equal(t, daemon.node, updated.LastNode,
|
||||
"self-election must persist LastNode in the KV so later requests see a resolved owner")
|
||||
}
|
||||
@@ -18,6 +18,7 @@ import (
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
handlers_ec2_account "github.com/mulgadc/spinifex/spinifex/handlers/ec2/account"
|
||||
handlers_ec2_eigw "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eigw"
|
||||
handlers_ec2_eip "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eip"
|
||||
@@ -35,9 +36,9 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/qmp"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil/ebsfake"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
@@ -69,6 +70,7 @@ func createFullTestDaemonWithStore(t *testing.T, natsURL string) (*Daemon, *obje
|
||||
daemon.volumeService = handlers_ec2_volume.NewVolumeServiceImplWithStore(cfg, memStore, daemon.natsConn)
|
||||
daemon.snapshotService = handlers_ec2_snapshot.NewSnapshotServiceImplWithStore(cfg, memStore, daemon.natsConn)
|
||||
daemon.tagsService = handlers_ec2_tags.NewTagsServiceImplWithStore(cfg, memStore)
|
||||
wireTestEBSProvider(daemon, memStore)
|
||||
initAccountServiceForTest(t, daemon)
|
||||
|
||||
// Wire RunInstances deps now that image/key services exist. vpcService and
|
||||
@@ -79,6 +81,25 @@ func createFullTestDaemonWithStore(t *testing.T, natsURL string) (*Daemon, *obje
|
||||
return daemon, memStore
|
||||
}
|
||||
|
||||
// wireTestEBSProvider mirrors configureEBSProvider for the test daemon: one
|
||||
// in-memory provider behind every EBS-adjacent service.
|
||||
func wireTestEBSProvider(daemon *Daemon, store objectstore.ObjectStore) {
|
||||
provider := ebsfake.New(store, daemon.config.Predastore.Bucket)
|
||||
daemon.ebsProvider = provider
|
||||
daemon.instanceService.SetEBSProvider(provider)
|
||||
daemon.imageService.SetEBSProvider(provider)
|
||||
daemon.snapshotService.SetEBSProvider(provider)
|
||||
daemon.volumeService.SetEBSProvider(provider)
|
||||
}
|
||||
|
||||
// seedVolumeDocument registers a volume the only way the control plane knows
|
||||
// one: as an ebsmetadata document. Tests that need a volume to exist seed it
|
||||
// here rather than writing storage state the control plane never reads.
|
||||
func seedVolumeDocument(t *testing.T, store objectstore.ObjectStore, volume ebsmetadata.Volume) {
|
||||
t.Helper()
|
||||
require.NoError(t, ebsmetadata.NewStore(store, "test-bucket").PutVolume(t.Context(), volume))
|
||||
}
|
||||
|
||||
// createFullTestDaemon creates a test daemon with ALL services initialized (including
|
||||
// key, image, snapshot, tags, eigw, account) using in-memory object stores.
|
||||
func createFullTestDaemon(t *testing.T, natsURL string) *Daemon {
|
||||
@@ -1470,24 +1491,12 @@ func TestAttachVolume_ZoneMismatch(t *testing.T) {
|
||||
daemon.vmMgr.Insert(instance)
|
||||
|
||||
// Create a volume in a different AZ
|
||||
wrapper := struct {
|
||||
VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
|
||||
}{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: volumeID,
|
||||
SizeGiB: 10,
|
||||
State: "available",
|
||||
AvailabilityZone: "us-west-2a",
|
||||
TenantID: testAccountID,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, _ := json.Marshal(wrapper)
|
||||
store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
CapacityGiB: 10,
|
||||
State: "available",
|
||||
AvailabilityZone: "us-west-2a",
|
||||
TenantID: testAccountID,
|
||||
})
|
||||
|
||||
// Subscribe handler
|
||||
@@ -2671,23 +2680,11 @@ func TestHandleEC2ModifyVolume_Success(t *testing.T) {
|
||||
|
||||
// Seed a volume in the store
|
||||
volumeID := "vol-modify-success"
|
||||
wrapper := struct {
|
||||
VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
|
||||
}{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: volumeID,
|
||||
SizeGiB: 10,
|
||||
State: "available",
|
||||
VolumeType: "gp3",
|
||||
},
|
||||
},
|
||||
}
|
||||
data, _ := json.Marshal(wrapper)
|
||||
store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
CapacityGiB: 10,
|
||||
State: "available",
|
||||
VolumeType: "gp3",
|
||||
})
|
||||
|
||||
// Subscribe a dummy ebs.sync handler so the NATS Request doesn't time out
|
||||
@@ -2810,22 +2807,10 @@ func TestHandleEC2CreateImage_RunningInstanceReachesService(t *testing.T) {
|
||||
sourceImageID := "ami-source-001"
|
||||
|
||||
// Seed a root volume config
|
||||
wrapper := struct {
|
||||
VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
|
||||
}{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: rootVolumeID,
|
||||
SizeGiB: 8,
|
||||
State: "in-use",
|
||||
},
|
||||
},
|
||||
}
|
||||
volData, _ := json.Marshal(wrapper)
|
||||
store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(rootVolumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(volData)),
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: rootVolumeID,
|
||||
CapacityGiB: 8,
|
||||
State: "in-use",
|
||||
})
|
||||
|
||||
daemon.vmMgr.Insert(&vm.VM{
|
||||
@@ -3011,23 +2996,11 @@ func TestAttachVolume_VolumeInUse(t *testing.T) {
|
||||
daemon.vmMgr.Insert(instance)
|
||||
|
||||
// Seed a volume that is already in-use
|
||||
wrapper := struct {
|
||||
VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
|
||||
}{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: volumeID,
|
||||
SizeGiB: 10,
|
||||
State: "in-use",
|
||||
TenantID: testAccountID,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, _ := json.Marshal(wrapper)
|
||||
store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
CapacityGiB: 10,
|
||||
State: "in-use",
|
||||
TenantID: testAccountID,
|
||||
})
|
||||
|
||||
sub, err := daemon.natsConn.Subscribe(
|
||||
|
||||
@@ -42,31 +42,31 @@ func (d *Daemon) handleAttachVolume(ctx context.Context, msg *nats.Msg, command
|
||||
return
|
||||
}
|
||||
|
||||
volCfg, err := d.volumeService.GetVolumeConfig(volumeID)
|
||||
volMeta, err := d.volumeService.GetVolumeMetadata(volumeID)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "AttachVolume: failed to get volume config", "volumeId", volumeID, "err", err)
|
||||
slog.ErrorContext(ctx, "AttachVolume: failed to get volume metadata", "volumeId", volumeID, "err", err)
|
||||
respondWithError(msg, awserrors.ErrorInvalidVolumeNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
callerAccountID := utils.AccountIDFromMsg(msg)
|
||||
if !volumeVisibleTo(volCfg.VolumeMetadata.TenantID, callerAccountID) {
|
||||
if !volumeVisibleTo(volMeta.TenantID, callerAccountID) {
|
||||
slog.WarnContext(ctx, "AttachVolume: account does not own volume",
|
||||
"volumeId", volumeID,
|
||||
"callerAccount", callerAccountID,
|
||||
"ownerAccount", volCfg.VolumeMetadata.TenantID)
|
||||
"ownerAccount", volMeta.TenantID)
|
||||
respondWithError(msg, awserrors.ErrorInvalidVolumeNotFound)
|
||||
return
|
||||
}
|
||||
|
||||
if volCfg.VolumeMetadata.State != "available" {
|
||||
if volCfg.VolumeMetadata.AttachedInstance == command.ID {
|
||||
if volMeta.State != "available" {
|
||||
if volMeta.AttachedInstance == command.ID {
|
||||
if command.AttachVolumeData.Device != "" &&
|
||||
command.AttachVolumeData.Device != volCfg.VolumeMetadata.DeviceName {
|
||||
command.AttachVolumeData.Device != volMeta.DeviceName {
|
||||
slog.ErrorContext(ctx, "AttachVolume: requested device conflicts with existing attachment",
|
||||
"volumeId", volumeID, "instanceId", command.ID,
|
||||
"requestedDevice", command.AttachVolumeData.Device,
|
||||
"attachedDevice", volCfg.VolumeMetadata.DeviceName)
|
||||
"attachedDevice", volMeta.DeviceName)
|
||||
// AWS returns VolumeInUse (not InvalidParameterValue) when a
|
||||
// re-attach targets a device other than the one already in use.
|
||||
respondWithError(msg, awserrors.ErrorVolumeInUse)
|
||||
@@ -77,22 +77,22 @@ func (d *Daemon) handleAttachVolume(ctx context.Context, msg *nats.Msg, command
|
||||
// ControllerPublishVolume retry after a slow first attach).
|
||||
// Treat as an idempotent success instead of VolumeInUse.
|
||||
slog.InfoContext(ctx, "AttachVolume: volume already attached to requesting instance, returning idempotent success",
|
||||
"volumeId", volumeID, "instanceId", command.ID, "device", volCfg.VolumeMetadata.DeviceName)
|
||||
d.respondWithVolumeAttachment(msg, volumeID, command.ID, volCfg.VolumeMetadata.DeviceName, "attached")
|
||||
"volumeId", volumeID, "instanceId", command.ID, "device", volMeta.DeviceName)
|
||||
d.respondWithVolumeAttachment(msg, volumeID, command.ID, volMeta.DeviceName, "attached")
|
||||
return
|
||||
}
|
||||
|
||||
slog.ErrorContext(ctx, "AttachVolume: volume not available",
|
||||
"volumeId", volumeID, "state", volCfg.VolumeMetadata.State)
|
||||
"volumeId", volumeID, "state", volMeta.State)
|
||||
respondWithError(msg, awserrors.ErrorVolumeInUse)
|
||||
return
|
||||
}
|
||||
|
||||
if volCfg.VolumeMetadata.AvailabilityZone != "" && d.config.AZ != "" &&
|
||||
volCfg.VolumeMetadata.AvailabilityZone != d.config.AZ {
|
||||
if volMeta.AvailabilityZone != "" && d.config.AZ != "" &&
|
||||
volMeta.AvailabilityZone != d.config.AZ {
|
||||
slog.ErrorContext(ctx, "AttachVolume: volume and instance are in different AZs",
|
||||
"volumeId", volumeID,
|
||||
"volumeAZ", volCfg.VolumeMetadata.AvailabilityZone,
|
||||
"volumeAZ", volMeta.AvailabilityZone,
|
||||
"instanceAZ", d.config.AZ)
|
||||
respondWithError(msg, awserrors.ErrorInvalidVolumeZoneMismatch)
|
||||
return
|
||||
|
||||
@@ -8,18 +8,17 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/qmp"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -103,24 +102,12 @@ func TestAttachDetachErrorCode(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// seedVolumeConfig writes a minimal VolumeConfig to config.json, matching
|
||||
// the seeding pattern used by the handleAttachVolume tests in
|
||||
// daemon_handlers_test.go.
|
||||
func seedVolumeConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string, meta viperblock.VolumeMetadata) {
|
||||
// seedVolumeConfig makes a volume exist on both sides at once: the provider
|
||||
// holds the blocks, the ebsmetadata document is what the control plane reads.
|
||||
func seedVolumeConfig(t *testing.T, daemon *Daemon, store *objectstore.MemoryObjectStore, volume ebsmetadata.Volume) {
|
||||
t.Helper()
|
||||
wrapper := struct {
|
||||
VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
|
||||
}{
|
||||
VolumeConfig: viperblock.VolumeConfig{VolumeMetadata: meta},
|
||||
}
|
||||
data, err := json.Marshal(wrapper)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
seedProviderVolume(t, daemon, volume.VolumeID, utils.SafeUint64ToInt64(volume.CapacityGiB))
|
||||
seedVolumeDocument(t, store, volume)
|
||||
}
|
||||
|
||||
// TestAttachVolume_IdempotentSameInstance verifies that re-attaching a
|
||||
@@ -156,9 +143,9 @@ func TestAttachVolume_IdempotentSameInstance(t *testing.T) {
|
||||
}
|
||||
daemon.vmMgr.Insert(instance)
|
||||
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "in-use",
|
||||
TenantID: testAccountID,
|
||||
AttachedInstance: instanceID,
|
||||
@@ -224,9 +211,9 @@ func TestAttachVolume_InUseDifferentInstance(t *testing.T) {
|
||||
}
|
||||
daemon.vmMgr.Insert(instance)
|
||||
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "in-use",
|
||||
TenantID: testAccountID,
|
||||
AttachedInstance: otherInstanceID,
|
||||
@@ -281,9 +268,9 @@ func TestAttachVolume_IdempotentSameInstance_DeviceMismatch(t *testing.T) {
|
||||
}
|
||||
daemon.vmMgr.Insert(instance)
|
||||
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "in-use",
|
||||
TenantID: testAccountID,
|
||||
AttachedInstance: instanceID,
|
||||
@@ -569,3 +556,17 @@ func TestDrainVolume_DispatchedGoroutineDoesNotLeak(t *testing.T) {
|
||||
|
||||
goleak.VerifyNone(t, ignoreExisting)
|
||||
}
|
||||
|
||||
// seedProviderVolume allocates volumeID in the daemon's EBS provider so
|
||||
// operations that reach the provider (expand, delete) find it there.
|
||||
func seedProviderVolume(t *testing.T, daemon *Daemon, volumeID string, sizeGiB int64) {
|
||||
t.Helper()
|
||||
if daemon == nil || daemon.ebsProvider == nil || volumeID == "" || sizeGiB <= 0 {
|
||||
return
|
||||
}
|
||||
_, err := daemon.ebsProvider.CreateVolume(t.Context(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: sizeGiB * 1024 * 1024 * 1024},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
@@ -28,9 +28,9 @@ import (
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/gpu"
|
||||
handlers_ec2_account "github.com/mulgadc/spinifex/spinifex/handlers/ec2/account"
|
||||
handlers_ec2_eigw "github.com/mulgadc/spinifex/spinifex/handlers/ec2/eigw"
|
||||
@@ -52,7 +52,6 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -171,18 +170,15 @@ func getTestInstanceType(t *testing.T) string {
|
||||
return "t3.micro" // Default fallback
|
||||
}
|
||||
|
||||
// seedTestAMI creates a minimal AMI config in the memory store so that
|
||||
// seedTestAMI creates a minimal AMI document in the memory store so that
|
||||
// handleEC2RunInstances AMI validation passes.
|
||||
func seedTestAMI(t *testing.T, store *objectstore.MemoryObjectStore, bucket, imageID string) {
|
||||
t.Helper()
|
||||
amiConfig := `{"VolumeConfig":{"AMIMetadata":{"ImageID":"` + imageID + `","Name":"test"}}}`
|
||||
_, err := store.PutObject(t.Context(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(bucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Body: strings.NewReader(amiConfig),
|
||||
ContentType: aws.String("application/json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, ebsmetadata.NewStore(store, bucket).PutAMI(t.Context(), ebsmetadata.AMI{
|
||||
ImageID: imageID,
|
||||
Name: "test",
|
||||
State: "available",
|
||||
}))
|
||||
}
|
||||
|
||||
// TestResourceManager tests resource manager functionality.
|
||||
@@ -1573,12 +1569,13 @@ func TestInstanceCleanerAdapter_DeleteVolumes_BootVolumeDeletedAfterAttachmentCl
|
||||
})
|
||||
require.NoError(t, err)
|
||||
daemon.volumeService = handlers_ec2_volume.NewVolumeServiceImplWithStore(daemon.config, store, daemon.natsConn, snapKV)
|
||||
daemon.volumeService.SetEBSProvider(daemon.ebsProvider)
|
||||
|
||||
volumeID := "vol-root-attached"
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testAccountID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "available",
|
||||
AttachedInstance: "i-test-boot-delete", // stale: never cleared by Stop/shutdownAndUnmount's Boot carve-out
|
||||
})
|
||||
@@ -1597,7 +1594,7 @@ func TestInstanceCleanerAdapter_DeleteVolumes_BootVolumeDeletedAfterAttachmentCl
|
||||
err = cleaner.DeleteVolumes(instance)
|
||||
require.NoError(t, err, "the still-attached boot volume must be deleted, not rejected as VolumeInUse")
|
||||
|
||||
_, err = daemon.volumeService.GetVolumeConfig(volumeID)
|
||||
_, err = daemon.volumeService.GetVolumeMetadata(volumeID)
|
||||
require.Error(t, err, "the volume must actually be deleted")
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorInvalidVolumeNotFound)
|
||||
}
|
||||
@@ -1613,10 +1610,10 @@ func TestInstanceCleanerAdapter_DeleteVolumes_NonDoTBootVolumeDetachedNotDeleted
|
||||
daemon, store := createFullTestDaemonWithStore(t, sharedNATSURL)
|
||||
|
||||
volumeID := "vol-root-nondot-attached"
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testAccountID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "available",
|
||||
AttachedInstance: "i-test-boot-detach", // stale: never cleared by Unmount's Boot carve-out
|
||||
})
|
||||
@@ -1635,10 +1632,10 @@ func TestInstanceCleanerAdapter_DeleteVolumes_NonDoTBootVolumeDetachedNotDeleted
|
||||
err := cleaner.DeleteVolumes(instance)
|
||||
require.NoError(t, err)
|
||||
|
||||
cfg, err := daemon.volumeService.GetVolumeConfig(volumeID)
|
||||
meta, err := daemon.volumeService.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err, "a DeleteOnTermination=false volume must survive terminate")
|
||||
assert.Equal(t, "available", cfg.VolumeMetadata.State)
|
||||
assert.Empty(t, cfg.VolumeMetadata.AttachedInstance, "terminate must still detach it")
|
||||
assert.Equal(t, "available", meta.State)
|
||||
assert.Empty(t, meta.AttachedInstance, "terminate must still detach it")
|
||||
}
|
||||
|
||||
// TestInstanceCleanerAdapter_DeleteVolumes_NonBootNonDoTVolumeDetachedNotDeleted
|
||||
@@ -1653,10 +1650,10 @@ func TestInstanceCleanerAdapter_DeleteVolumes_NonBootNonDoTVolumeDetachedNotDele
|
||||
daemon, store := createFullTestDaemonWithStore(t, sharedNATSURL)
|
||||
|
||||
volumeID := "vol-data-nondot-attached"
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testAccountID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "in-use",
|
||||
AttachedInstance: "i-test-data-detach", // stale: Unmount's seal failed or never ran
|
||||
})
|
||||
@@ -1675,10 +1672,10 @@ func TestInstanceCleanerAdapter_DeleteVolumes_NonBootNonDoTVolumeDetachedNotDele
|
||||
err := cleaner.DeleteVolumes(instance)
|
||||
require.NoError(t, err)
|
||||
|
||||
cfg, err := daemon.volumeService.GetVolumeConfig(volumeID)
|
||||
meta, err := daemon.volumeService.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err, "a DeleteOnTermination=false volume must survive terminate")
|
||||
assert.Equal(t, "available", cfg.VolumeMetadata.State)
|
||||
assert.Empty(t, cfg.VolumeMetadata.AttachedInstance, "terminate must still detach it")
|
||||
assert.Equal(t, "available", meta.State)
|
||||
assert.Empty(t, meta.AttachedInstance, "terminate must still detach it")
|
||||
}
|
||||
|
||||
// TestTerminatedTeardownReaper_SelfHealsFailedVolumeTeardown proves the
|
||||
@@ -1707,13 +1704,14 @@ func TestTerminatedTeardownReaper_SelfHealsFailedVolumeTeardown(t *testing.T) {
|
||||
})
|
||||
require.NoError(t, err)
|
||||
daemon.volumeService = handlers_ec2_volume.NewVolumeServiceImplWithStore(daemon.config, store, daemon.natsConn, snapKV)
|
||||
daemon.volumeService.SetEBSProvider(daemon.ebsProvider)
|
||||
|
||||
volumeID := "vol-root-self-heal"
|
||||
instanceID := "i-self-heal"
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testAccountID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "available",
|
||||
AttachedInstance: instanceID, // stale, left by the earlier failed terminate
|
||||
})
|
||||
@@ -1744,7 +1742,7 @@ func TestTerminatedTeardownReaper_SelfHealsFailedVolumeTeardown(t *testing.T) {
|
||||
_, err = reaper.Sweep(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = daemon.volumeService.GetVolumeConfig(volumeID)
|
||||
_, err = daemon.volumeService.GetVolumeMetadata(volumeID)
|
||||
require.Error(t, err, "the retried delete must actually remove the volume")
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorInvalidVolumeNotFound)
|
||||
|
||||
@@ -1771,10 +1769,10 @@ func TestTerminatedTeardownReaper_SelfHealsFailedVolumeDetach(t *testing.T) {
|
||||
|
||||
volumeID := "vol-data-self-heal"
|
||||
instanceID := "i-self-heal-detach"
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testAccountID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "in-use",
|
||||
AttachedInstance: instanceID, // stale, left by the earlier failed terminate
|
||||
})
|
||||
@@ -1805,10 +1803,10 @@ func TestTerminatedTeardownReaper_SelfHealsFailedVolumeDetach(t *testing.T) {
|
||||
_, err := reaper.Sweep(context.Background())
|
||||
require.NoError(t, err)
|
||||
|
||||
cfg, err := daemon.volumeService.GetVolumeConfig(volumeID)
|
||||
meta, err := daemon.volumeService.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err, "the volume must survive: DeleteOnTermination=false")
|
||||
assert.Equal(t, "available", cfg.VolumeMetadata.State)
|
||||
assert.Empty(t, cfg.VolumeMetadata.AttachedInstance, "the retry must clear the stale attachment")
|
||||
assert.Equal(t, "available", meta.State)
|
||||
assert.Empty(t, meta.AttachedInstance, "the retry must clear the stale attachment")
|
||||
|
||||
remaining, err := fakeStore.ListTerminatedInstances()
|
||||
require.NoError(t, err)
|
||||
@@ -1832,10 +1830,10 @@ func TestStuckTerminateReaper_DetachesNonDoTVolumeWithoutUnmount(t *testing.T) {
|
||||
|
||||
const instanceID = "i-wedged-terminate"
|
||||
volumeID := "vol-data-wedged"
|
||||
seedVolumeConfig(t, store, volumeID, viperblock.VolumeMetadata{
|
||||
seedVolumeConfig(t, daemon, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testAccountID,
|
||||
SizeGiB: 10,
|
||||
CapacityGiB: 10,
|
||||
State: "in-use",
|
||||
AttachedInstance: instanceID, // never cleared: Unmount never ran on this path
|
||||
})
|
||||
@@ -1863,11 +1861,11 @@ func TestStuckTerminateReaper_DetachesNonDoTVolumeWithoutUnmount(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, reaped, "the wedged terminate must be force-completed")
|
||||
|
||||
cfg, err := daemon.volumeService.GetVolumeConfig(volumeID)
|
||||
meta, err := daemon.volumeService.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "available", cfg.VolumeMetadata.State,
|
||||
assert.Equal(t, "available", meta.State,
|
||||
"the force-complete path must still detach a non-Boot DeleteOnTermination=false volume")
|
||||
assert.Empty(t, cfg.VolumeMetadata.AttachedInstance)
|
||||
assert.Empty(t, meta.AttachedInstance)
|
||||
}
|
||||
|
||||
// TestHandleEC2Events_AttachVolume tests the attach-volume handler in handleEC2Events.
|
||||
@@ -4501,3 +4499,61 @@ func TestAssertNoClusterServicesInitialised_PerField(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// --- configureEBSProvider ---
|
||||
|
||||
// newEBSProviderTestDaemon builds a daemon with real (constructor-built, not
|
||||
// zero-valued) EBS services, so configureEBSProvider has something to wire the
|
||||
// provider into rather than a set of zero values that would accept anything.
|
||||
func newEBSProviderTestDaemon(t *testing.T, provider string) *Daemon {
|
||||
t.Helper()
|
||||
nc, err := nats.Connect(sharedJSNATSURL)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(nc.Close)
|
||||
|
||||
jsManager, err := NewJetStreamManager(nc, 1)
|
||||
require.NoError(t, err)
|
||||
|
||||
ctx := context.Background()
|
||||
cfg := &config.Config{
|
||||
EBS: config.EBSConfig{Provider: provider},
|
||||
Predastore: config.PredastoreConfig{Bucket: "test-bucket"},
|
||||
}
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
|
||||
return &Daemon{
|
||||
ctx: ctx,
|
||||
config: cfg,
|
||||
natsConn: nc,
|
||||
jsManager: jsManager,
|
||||
instanceService: &handlers_ec2_instance.InstanceServiceImpl{},
|
||||
imageService: handlers_ec2_image.NewImageServiceImplWithStore(store, cfg.Predastore.Bucket),
|
||||
snapshotService: handlers_ec2_snapshot.NewSnapshotServiceImplWithStore(cfg, store, nc),
|
||||
volumeService: handlers_ec2_volume.NewVolumeServiceImplWithStore(cfg, store, nc),
|
||||
}
|
||||
}
|
||||
|
||||
// An unset provider must still wire viperblockd everywhere: there is no
|
||||
// second provider to fall back to, so leaving it nil would silently strand
|
||||
// every EBS call on a node whose spinifex.toml predates the key.
|
||||
func TestConfigureEBSProvider_UnsetInjectsProviderEverywhere(t *testing.T) {
|
||||
d := newEBSProviderTestDaemon(t, "")
|
||||
require.NoError(t, d.configureEBSProvider())
|
||||
|
||||
require.NotNil(t, d.ebsProvider)
|
||||
assert.Same(t, d.ebsProvider, d.instanceService.EBSProvider())
|
||||
assert.Same(t, d.ebsProvider, d.imageService.EBSProvider())
|
||||
assert.Same(t, d.ebsProvider, d.snapshotService.EBSProvider())
|
||||
assert.Same(t, d.ebsProvider, d.volumeService.EBSProvider())
|
||||
}
|
||||
|
||||
func TestConfigureEBSProvider_ViperblockdInjectsSameInstanceEverywhere(t *testing.T) {
|
||||
d := newEBSProviderTestDaemon(t, config.EBSProviderViperblockd)
|
||||
require.NoError(t, d.configureEBSProvider())
|
||||
|
||||
require.NotNil(t, d.ebsProvider, "daemon must retain the provider it constructed")
|
||||
assert.Same(t, d.ebsProvider, d.instanceService.EBSProvider())
|
||||
assert.Same(t, d.ebsProvider, d.imageService.EBSProvider())
|
||||
assert.Same(t, d.ebsProvider, d.snapshotService.EBSProvider())
|
||||
assert.Same(t, d.ebsProvider, d.volumeService.EBSProvider())
|
||||
}
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package daemon
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
)
|
||||
|
||||
// eniOrphanMinAge is how long an unattached instance ENI must sit before the
|
||||
// sweep will delete it. The launch RPC gives a node five minutes to answer, so
|
||||
// this leaves a wide margin past the point where a launch could still attach.
|
||||
const eniOrphanMinAge = 15 * time.Minute
|
||||
|
||||
// eniOrphanReaper deletes ENI records left behind by a launch that created the
|
||||
// interface and then died before attaching it. The record outlives the launch,
|
||||
// and because the security group dependency check counts an ENI that lists the
|
||||
// group, the SG can never be deleted afterwards.
|
||||
//
|
||||
// Cluster-wide rather than node-local: the record has no instance, so no node
|
||||
// owns it, and nothing local would ever look at it again.
|
||||
type eniOrphanReaper struct {
|
||||
vpc *handlers_ec2_vpc.VPCServiceImpl
|
||||
minAge time.Duration
|
||||
}
|
||||
|
||||
var _ vm.Reaper = (*eniOrphanReaper)(nil)
|
||||
|
||||
// newENIOrphanReaper builds the sweep over the daemon's VPC service, or nil
|
||||
// when there is no control plane on this node to sweep with.
|
||||
func (d *Daemon) newENIOrphanReaper() *eniOrphanReaper {
|
||||
if d.vpcService == nil {
|
||||
return nil
|
||||
}
|
||||
return &eniOrphanReaper{vpc: d.vpcService, minAge: eniOrphanMinAge}
|
||||
}
|
||||
|
||||
func (r *eniOrphanReaper) Class() string { return "eni-orphan" }
|
||||
func (r *eniOrphanReaper) Scope() vm.ReaperScope { return vm.ScopeClusterWide }
|
||||
|
||||
// Sweep deletes every abandoned instance ENI. A per-record failure is logged
|
||||
// and skipped so one bad record cannot stall the rest of the sweep.
|
||||
func (r *eniOrphanReaper) Sweep(ctx context.Context) (int, error) {
|
||||
orphans, err := r.vpc.ListAbandonedInstanceENIs(ctx, r.minAge)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
reaped := 0
|
||||
for _, orphan := range orphans {
|
||||
if ctx.Err() != nil {
|
||||
return reaped, ctx.Err()
|
||||
}
|
||||
eniID := orphan.Record.NetworkInterfaceId
|
||||
slog.InfoContext(ctx, "eni-orphan: deleting ENI left by an abandoned launch",
|
||||
"eniId", eniID,
|
||||
"accountId", orphan.AccountID,
|
||||
"description", orphan.Record.Description,
|
||||
"securityGroups", orphan.Record.SecurityGroupIds,
|
||||
"createdAt", orphan.Record.CreatedAt)
|
||||
|
||||
_, err := r.vpc.DeleteNetworkInterface(ctx, &ec2.DeleteNetworkInterfaceInput{
|
||||
NetworkInterfaceId: aws.String(eniID),
|
||||
}, orphan.AccountID)
|
||||
// Another sweep may have deleted it between the list and here, which
|
||||
// is the outcome this asked for either way.
|
||||
if err != nil && err.Error() != awserrors.ErrorInvalidNetworkInterfaceIDNotFound {
|
||||
slog.WarnContext(ctx, "eni-orphan: delete failed",
|
||||
"eniId", eniID, "accountId", orphan.AccountID, "err", err)
|
||||
continue
|
||||
}
|
||||
reaped++
|
||||
}
|
||||
return reaped, nil
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package daemon
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
handlers_ec2_vpc "github.com/mulgadc/spinifex/spinifex/handlers/ec2/vpc"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const reaperTestAccountID = "123456789012"
|
||||
|
||||
// TestENIOrphanReaperScopeIsClusterWide pins the scope. The record it reaps has
|
||||
// no instance, so no node owns it; running the sweep node-locally would have
|
||||
// every node racing to delete the same records.
|
||||
func TestENIOrphanReaperScopeIsClusterWide(t *testing.T) {
|
||||
r := &eniOrphanReaper{}
|
||||
assert.Equal(t, vm.ScopeClusterWide, r.Scope())
|
||||
assert.Equal(t, "eni-orphan", r.Class())
|
||||
}
|
||||
|
||||
func TestENIOrphanReaperSweep(t *testing.T) {
|
||||
_, nc, _ := testutil.StartTestJetStream(t)
|
||||
vpcSvc, err := handlers_ec2_vpc.NewVPCServiceImplWithNATS(t.Context(), nil, nc)
|
||||
require.NoError(t, err)
|
||||
testutil.StubVpcdSGResponder(t, nc)
|
||||
|
||||
ctx := context.Background()
|
||||
vpcOut, err := vpcSvc.CreateVpc(ctx, &ec2.CreateVpcInput{
|
||||
CidrBlock: aws.String("10.0.0.0/16"),
|
||||
}, reaperTestAccountID)
|
||||
require.NoError(t, err)
|
||||
subnetOut, err := vpcSvc.CreateSubnet(ctx, &ec2.CreateSubnetInput{
|
||||
VpcId: vpcOut.Vpc.VpcId,
|
||||
CidrBlock: aws.String("10.0.1.0/24"),
|
||||
}, reaperTestAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
eniOut, err := vpcSvc.CreateNetworkInterface(ctx, &ec2.CreateNetworkInterfaceInput{
|
||||
SubnetId: subnetOut.Subnet.SubnetId,
|
||||
Description: aws.String(handlers_ec2_vpc.AutoENIDescriptionPrefix + "i-abandoned"),
|
||||
}, reaperTestAccountID)
|
||||
require.NoError(t, err)
|
||||
eniID := *eniOut.NetworkInterface.NetworkInterfaceId
|
||||
require.NoError(t, vpcSvc.UpdateENI(reaperTestAccountID, eniID, func(rec *handlers_ec2_vpc.ENIRecord) {
|
||||
rec.CreatedAt = time.Now().Add(-time.Hour)
|
||||
}))
|
||||
|
||||
r := &eniOrphanReaper{vpc: vpcSvc, minAge: 15 * time.Minute}
|
||||
|
||||
reaped, err := r.Sweep(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, reaped)
|
||||
|
||||
_, err = vpcSvc.GetENIRecord(reaperTestAccountID, eniID)
|
||||
require.Error(t, err, "the record must be gone, or it keeps blocking its security group")
|
||||
|
||||
// A second pass has nothing left and must not report work or fail.
|
||||
reaped, err = r.Sweep(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Zero(t, reaped)
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
// Package ebsmetadata owns Spinifex's control-plane view of EBS resources.
|
||||
// These documents deliberately do not mirror viperblock's VBState schema.
|
||||
package ebsmetadata
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const SchemaVersion uint16 = 1
|
||||
|
||||
// ErrCorruptDocument wraps decode and schema-version failures so callers can
|
||||
// tell a document that exists but cannot be read from one that is absent. The
|
||||
// two deserve different answers: salvage versus not-found.
|
||||
var ErrCorruptDocument = errors.New("corrupt EBS metadata document")
|
||||
|
||||
// Volume is the control-plane record used for EC2 Describe/Modify/attachment
|
||||
// operations. ProviderHandle is opaque and is never decoded by the API layer.
|
||||
type Volume struct {
|
||||
SchemaVersion uint16 `json:"schema_version"`
|
||||
VolumeID string `json:"volume_id"`
|
||||
VolumeName string `json:"volume_name,omitempty"`
|
||||
TenantID string `json:"tenant_id"`
|
||||
CapacityGiB uint64 `json:"capacity_gib"`
|
||||
State string `json:"state"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
AttachedAt time.Time `json:"attached_at,omitzero"`
|
||||
AvailabilityZone string `json:"availability_zone"`
|
||||
AttachedInstance string `json:"attached_instance,omitempty"`
|
||||
DeviceName string `json:"device_name,omitempty"`
|
||||
VolumeType string `json:"volume_type"`
|
||||
IOPS int `json:"iops"`
|
||||
Throughput int `json:"throughput,omitempty"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
SnapshotID string `json:"snapshot_id,omitempty"`
|
||||
DeleteOnTermination bool `json:"delete_on_termination,omitempty"`
|
||||
Encrypted bool `json:"encrypted,omitempty"`
|
||||
ProviderHandle string `json:"provider_handle,omitempty"`
|
||||
Modification *VolumeModification `json:"modification,omitempty"`
|
||||
}
|
||||
|
||||
// VolumeModification is the control-plane record of a completed or in-flight
|
||||
// ModifyVolume operation, read back by DescribeVolumesModifications. It owns
|
||||
// its own fields rather than reusing viperblock.VolumeModification: this
|
||||
// package must stay free of viperblock types. VolumeID is deliberately not
|
||||
// duplicated here — callers have it from the owning Volume.
|
||||
type VolumeModification struct {
|
||||
ModificationState string `json:"modification_state"`
|
||||
Progress int64 `json:"progress"`
|
||||
StatusMessage string `json:"status_message,omitempty"`
|
||||
OriginalSize int64 `json:"original_size"`
|
||||
OriginalIOPS int64 `json:"original_iops"`
|
||||
OriginalVolumeType string `json:"original_volume_type"`
|
||||
TargetSize int64 `json:"target_size"`
|
||||
TargetIOPS int64 `json:"target_iops"`
|
||||
TargetVolumeType string `json:"target_volume_type"`
|
||||
StartTime time.Time `json:"start_time"`
|
||||
EndTime time.Time `json:"end_time,omitzero"`
|
||||
}
|
||||
|
||||
// AMI is the control-plane record used for EC2 image operations.
|
||||
type AMI struct {
|
||||
SchemaVersion uint16 `json:"schema_version"`
|
||||
ImageID string `json:"image_id"`
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Architecture string `json:"architecture"`
|
||||
PlatformDetails string `json:"platform_details"`
|
||||
CreationDate time.Time `json:"creation_date"`
|
||||
RootDeviceType string `json:"root_device_type"`
|
||||
Virtualization string `json:"virtualization"`
|
||||
ImageOwnerAlias string `json:"image_owner_alias"`
|
||||
VolumeSizeGiB uint64 `json:"volume_size_gib"`
|
||||
SnapshotID string `json:"snapshot_id,omitempty"`
|
||||
BootMode string `json:"boot_mode,omitempty"`
|
||||
Distro string `json:"distro,omitempty"`
|
||||
DistroFamily string `json:"distro_family,omitempty"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
State string `json:"state,omitempty"`
|
||||
}
|
||||
|
||||
func VolumeKey(volumeID string) (string, error) { return key("volumes", volumeID) }
|
||||
func AMIKey(imageID string) (string, error) { return key("amis", imageID) }
|
||||
|
||||
func key(kind, id string) (string, error) {
|
||||
if id == "" || id == "." || id == ".." || strings.ContainsAny(id, "/\\") {
|
||||
return "", fmt.Errorf("invalid EBS metadata ID %q", id)
|
||||
}
|
||||
return "spinifex/ebsmetadata/v1/" + kind + "/" + id + ".json", nil
|
||||
}
|
||||
|
||||
func MarshalVolume(volume Volume) ([]byte, error) {
|
||||
volume.SchemaVersion = SchemaVersion
|
||||
return json.Marshal(volume)
|
||||
}
|
||||
|
||||
func UnmarshalVolume(data []byte) (Volume, error) {
|
||||
var volume Volume
|
||||
if err := json.Unmarshal(data, &volume); err != nil {
|
||||
return Volume{}, fmt.Errorf("%w: %w", ErrCorruptDocument, err)
|
||||
}
|
||||
if volume.SchemaVersion != SchemaVersion {
|
||||
return Volume{}, fmt.Errorf("%w: unsupported volume metadata schema version %d", ErrCorruptDocument, volume.SchemaVersion)
|
||||
}
|
||||
return volume, nil
|
||||
}
|
||||
|
||||
func MarshalAMI(ami AMI) ([]byte, error) {
|
||||
ami.SchemaVersion = SchemaVersion
|
||||
return json.Marshal(ami)
|
||||
}
|
||||
|
||||
func UnmarshalAMI(data []byte) (AMI, error) {
|
||||
var ami AMI
|
||||
if err := json.Unmarshal(data, &ami); err != nil {
|
||||
return AMI{}, fmt.Errorf("%w: %w", ErrCorruptDocument, err)
|
||||
}
|
||||
if ami.SchemaVersion != SchemaVersion {
|
||||
return AMI{}, fmt.Errorf("%w: unsupported AMI metadata schema version %d", ErrCorruptDocument, ami.SchemaVersion)
|
||||
}
|
||||
return ami, nil
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
package ebsmetadata
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestVolumeMetadataRoundTripOwnsItsSchema(t *testing.T) {
|
||||
want := Volume{
|
||||
VolumeID: "vol-1", TenantID: "acct-1", CapacityGiB: 20,
|
||||
State: "available", CreatedAt: time.Date(2026, 8, 5, 0, 0, 0, 0, time.UTC),
|
||||
ProviderHandle: "opaque-provider-handle", Tags: map[string]string{"env": "test"},
|
||||
}
|
||||
data, err := MarshalVolume(want)
|
||||
require.NoError(t, err)
|
||||
got, err := UnmarshalVolume(data)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, SchemaVersion, got.SchemaVersion)
|
||||
assert.Equal(t, want.VolumeID, got.VolumeID)
|
||||
assert.Equal(t, want.ProviderHandle, got.ProviderHandle)
|
||||
assert.Equal(t, want.Tags, got.Tags)
|
||||
}
|
||||
|
||||
// TestVolumeRoundTripsEncryptedAndModification covers the two fields added
|
||||
// to close the schema gaps behind the provider-branch consumer bugs: a
|
||||
// volume's encryption bit and its persisted modification record must both
|
||||
// survive a marshal/unmarshal round-trip.
|
||||
func TestVolumeRoundTripsEncryptedAndModification(t *testing.T) {
|
||||
want := Volume{
|
||||
VolumeID: "vol-1", TenantID: "acct-1", CapacityGiB: 20,
|
||||
State: "available", Encrypted: true,
|
||||
Modification: &VolumeModification{
|
||||
ModificationState: "completed", Progress: 100,
|
||||
OriginalSize: 8, OriginalIOPS: 3000, OriginalVolumeType: "gp3",
|
||||
TargetSize: 16, TargetIOPS: 3000, TargetVolumeType: "gp3",
|
||||
StartTime: time.Date(2026, 8, 5, 0, 0, 0, 0, time.UTC),
|
||||
EndTime: time.Date(2026, 8, 5, 0, 0, 1, 0, time.UTC),
|
||||
},
|
||||
}
|
||||
data, err := MarshalVolume(want)
|
||||
require.NoError(t, err)
|
||||
got, err := UnmarshalVolume(data)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, got.Encrypted)
|
||||
require.NotNil(t, got.Modification)
|
||||
assert.Equal(t, *want.Modification, *got.Modification)
|
||||
}
|
||||
|
||||
func TestMetadataRejectsUnknownSchemaVersion(t *testing.T) {
|
||||
_, err := UnmarshalAMI([]byte(`{"schema_version":99,"image_id":"ami-1"}`))
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "unsupported")
|
||||
}
|
||||
|
||||
func TestMetadataKeysRejectPathTraversal(t *testing.T) {
|
||||
for _, id := range []string{"", "..", "../escape", "a/b", "a\\b"} {
|
||||
_, err := VolumeKey(id)
|
||||
require.Error(t, err, "ID %q must not escape metadata prefix", id)
|
||||
}
|
||||
key, err := AMIKey("ami-1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "spinifex/ebsmetadata/v1/amis/ami-1.json", key)
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
package ebsmetadata
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
)
|
||||
|
||||
// Store persists Spinifex-owned metadata documents in the existing object
|
||||
// store. It intentionally knows nothing about provider state or VBState.
|
||||
type Store struct {
|
||||
objects objectstore.ObjectStore
|
||||
bucket string
|
||||
}
|
||||
|
||||
func NewStore(objects objectstore.ObjectStore, bucket string) *Store {
|
||||
return &Store{objects: objects, bucket: bucket}
|
||||
}
|
||||
|
||||
func (s *Store) PutVolume(ctx context.Context, volume Volume) error {
|
||||
key, err := VolumeKey(volume.VolumeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data, err := MarshalVolume(volume)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal volume metadata: %w", err)
|
||||
}
|
||||
return s.put(ctx, key, data)
|
||||
}
|
||||
|
||||
// GetVolume returns the volume's ebsmetadata document. A volume with no
|
||||
// document does not exist as far as the control plane is concerned.
|
||||
func (s *Store) GetVolume(ctx context.Context, volumeID string) (Volume, error) {
|
||||
key, err := VolumeKey(volumeID)
|
||||
if err != nil {
|
||||
return Volume{}, err
|
||||
}
|
||||
data, err := s.get(ctx, key)
|
||||
if err != nil {
|
||||
return Volume{}, err
|
||||
}
|
||||
return UnmarshalVolume(data)
|
||||
}
|
||||
|
||||
func (s *Store) DeleteVolume(ctx context.Context, volumeID string) error {
|
||||
key, err := VolumeKey(volumeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = s.objects.DeleteObject(ctx, &s3.DeleteObjectInput{Bucket: aws.String(s.bucket), Key: aws.String(key)})
|
||||
return err
|
||||
}
|
||||
|
||||
// ListVolumes returns every ebsmetadata volume document.
|
||||
func (s *Store) ListVolumes(ctx context.Context) ([]Volume, error) {
|
||||
if s == nil || s.objects == nil {
|
||||
return nil, errors.New("metadata store is not configured")
|
||||
}
|
||||
return s.listVolumeDocuments(ctx)
|
||||
}
|
||||
|
||||
func (s *Store) listVolumeDocuments(ctx context.Context) ([]Volume, error) {
|
||||
result, err := s.objects.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(s.bucket), Prefix: aws.String("spinifex/ebsmetadata/v1/volumes/"),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
volumes := make([]Volume, 0, len(result.Contents))
|
||||
for _, object := range result.Contents {
|
||||
if object.Key == nil {
|
||||
continue
|
||||
}
|
||||
data, err := s.get(ctx, *object.Key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
volume, err := UnmarshalVolume(data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
volumes = append(volumes, volume)
|
||||
}
|
||||
return volumes, nil
|
||||
}
|
||||
|
||||
// ListAMIs returns every AMI document, skipping any it cannot decode. One
|
||||
// corrupt document must not make every image in the cluster invisible, so the
|
||||
// tolerance is deliberate — the skip is logged rather than silent.
|
||||
func (s *Store) ListAMIs(ctx context.Context) ([]AMI, error) {
|
||||
if s == nil || s.objects == nil {
|
||||
return nil, errors.New("metadata store is not configured")
|
||||
}
|
||||
return s.listAMIDocuments(ctx, true)
|
||||
}
|
||||
|
||||
// ListAMIsStrict is ListAMIs without the tolerance. It is for callers whose
|
||||
// answer would be wrong rather than merely partial: a name-uniqueness check
|
||||
// cannot rule out a name held by an AMI it failed to read.
|
||||
func (s *Store) ListAMIsStrict(ctx context.Context) ([]AMI, error) {
|
||||
if s == nil || s.objects == nil {
|
||||
return nil, errors.New("metadata store is not configured")
|
||||
}
|
||||
return s.listAMIDocuments(ctx, false)
|
||||
}
|
||||
|
||||
func (s *Store) listAMIDocuments(ctx context.Context, skipCorrupt bool) ([]AMI, error) {
|
||||
result, err := s.objects.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(s.bucket), Prefix: aws.String("spinifex/ebsmetadata/v1/amis/"),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
amis := make([]AMI, 0, len(result.Contents))
|
||||
for _, object := range result.Contents {
|
||||
if object.Key == nil {
|
||||
continue
|
||||
}
|
||||
data, err := s.get(ctx, *object.Key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ami, err := UnmarshalAMI(data)
|
||||
if err != nil {
|
||||
if skipCorrupt && errors.Is(err, ErrCorruptDocument) {
|
||||
slog.WarnContext(ctx, "skipping undecodable AMI document", "key", *object.Key, "err", err)
|
||||
continue
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
amis = append(amis, ami)
|
||||
}
|
||||
return amis, nil
|
||||
}
|
||||
|
||||
func (s *Store) PutAMI(ctx context.Context, ami AMI) error {
|
||||
key, err := AMIKey(ami.ImageID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data, err := MarshalAMI(ami)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal AMI metadata: %w", err)
|
||||
}
|
||||
return s.put(ctx, key, data)
|
||||
}
|
||||
|
||||
// GetAMI returns the AMI's ebsmetadata document. An AMI with no document does
|
||||
// not exist as far as the control plane is concerned.
|
||||
func (s *Store) GetAMI(ctx context.Context, imageID string) (AMI, error) {
|
||||
key, err := AMIKey(imageID)
|
||||
if err != nil {
|
||||
return AMI{}, err
|
||||
}
|
||||
data, err := s.get(ctx, key)
|
||||
if err != nil {
|
||||
return AMI{}, err
|
||||
}
|
||||
return UnmarshalAMI(data)
|
||||
}
|
||||
|
||||
func (s *Store) DeleteAMI(ctx context.Context, imageID string) error {
|
||||
key, err := AMIKey(imageID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = s.objects.DeleteObject(ctx, &s3.DeleteObjectInput{Bucket: aws.String(s.bucket), Key: aws.String(key)})
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) put(ctx context.Context, key string, data []byte) error {
|
||||
if s == nil || s.objects == nil {
|
||||
return errors.New("metadata store is not configured")
|
||||
}
|
||||
_, err := s.objects.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(s.bucket), Key: aws.String(key), Body: bytes.NewReader(data),
|
||||
ContentType: aws.String("application/json"),
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) get(ctx context.Context, key string) ([]byte, error) {
|
||||
if s == nil || s.objects == nil {
|
||||
return nil, errors.New("metadata store is not configured")
|
||||
}
|
||||
result, err := s.objects.GetObject(ctx, &s3.GetObjectInput{Bucket: aws.String(s.bucket), Key: aws.String(key)})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer result.Body.Close()
|
||||
return io.ReadAll(result.Body)
|
||||
}
|
||||
@@ -0,0 +1,225 @@
|
||||
package ebsmetadata
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestStoreRoundTripsVolumeAndAMI(t *testing.T) {
|
||||
objects := objectstore.NewMemoryObjectStore()
|
||||
store := NewStore(objects, "control-plane")
|
||||
|
||||
volume := Volume{VolumeID: "vol-1", TenantID: "acct-1", CapacityGiB: 10, State: "available", ProviderHandle: "opaque"}
|
||||
require.NoError(t, store.PutVolume(context.Background(), volume))
|
||||
gotVolume, err := store.GetVolume(context.Background(), volume.VolumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, volume.VolumeID, gotVolume.VolumeID)
|
||||
assert.Equal(t, volume.ProviderHandle, gotVolume.ProviderHandle)
|
||||
assert.Equal(t, SchemaVersion, gotVolume.SchemaVersion)
|
||||
|
||||
ami := AMI{ImageID: "ami-1", Name: "test", SnapshotID: "snap-1"}
|
||||
require.NoError(t, store.PutAMI(context.Background(), ami))
|
||||
gotAMI, err := store.GetAMI(context.Background(), ami.ImageID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, ami.ImageID, gotAMI.ImageID)
|
||||
assert.Equal(t, ami.SnapshotID, gotAMI.SnapshotID)
|
||||
assert.Equal(t, SchemaVersion, gotAMI.SchemaVersion)
|
||||
|
||||
require.NoError(t, store.DeleteVolume(context.Background(), volume.VolumeID))
|
||||
require.NoError(t, store.DeleteAMI(context.Background(), ami.ImageID))
|
||||
}
|
||||
|
||||
func TestListVolumes_Empty(t *testing.T) {
|
||||
store := NewStore(objectstore.NewMemoryObjectStore(), "control-plane")
|
||||
volumes, err := store.ListVolumes(context.Background())
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, volumes)
|
||||
}
|
||||
|
||||
func TestListVolumes_ReturnsAllStoredVolumes(t *testing.T) {
|
||||
store := NewStore(objectstore.NewMemoryObjectStore(), "control-plane")
|
||||
ctx := context.Background()
|
||||
|
||||
want := []Volume{
|
||||
{VolumeID: "vol-1", TenantID: "acct-1", CapacityGiB: 8, State: "available"},
|
||||
{VolumeID: "vol-2", TenantID: "acct-2", CapacityGiB: 16, State: "in-use"},
|
||||
{VolumeID: "vol-3", TenantID: "acct-1", CapacityGiB: 32, State: "available"},
|
||||
}
|
||||
for _, v := range want {
|
||||
require.NoError(t, store.PutVolume(ctx, v))
|
||||
}
|
||||
|
||||
got, err := store.ListVolumes(ctx)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, len(want))
|
||||
|
||||
gotIDs := make(map[string]uint64, len(got))
|
||||
for _, v := range got {
|
||||
gotIDs[v.VolumeID] = v.CapacityGiB
|
||||
}
|
||||
for _, v := range want {
|
||||
assert.Equal(t, v.CapacityGiB, gotIDs[v.VolumeID])
|
||||
}
|
||||
}
|
||||
|
||||
// TestListVolumes_CorruptObjectReturnsError covers the UnmarshalVolume error
|
||||
// path: an object under the volumes prefix that isn't a valid Volume record
|
||||
// must fail the whole listing rather than being silently skipped.
|
||||
func TestListVolumes_CorruptObjectReturnsError(t *testing.T) {
|
||||
objects := objectstore.NewMemoryObjectStore()
|
||||
store := NewStore(objects, "control-plane")
|
||||
ctx := context.Background()
|
||||
|
||||
require.NoError(t, store.PutVolume(ctx, Volume{VolumeID: "vol-good", CapacityGiB: 1}))
|
||||
|
||||
key, err := VolumeKey("vol-corrupt")
|
||||
require.NoError(t, err)
|
||||
_, err = objects.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String("control-plane"), Key: aws.String(key), Body: bytes.NewReader([]byte("not json")),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.ListVolumes(ctx)
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
// TestListVolumes_NotConfigured covers the nil-store guard shared by every
|
||||
// Store method: a zero-value *Store (no ObjectStore wired up) must error
|
||||
// instead of panicking on a nil dereference.
|
||||
func TestListVolumes_NotConfigured(t *testing.T) {
|
||||
var store *Store
|
||||
_, err := store.ListVolumes(context.Background())
|
||||
require.Error(t, err)
|
||||
|
||||
empty := &Store{}
|
||||
_, err = empty.ListVolumes(context.Background())
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
// TestListAMIs_Empty mirrors TestListVolumes_Empty: an AMI-less store lists
|
||||
// as an empty slice, not an error.
|
||||
func TestListAMIs_Empty(t *testing.T) {
|
||||
store := NewStore(objectstore.NewMemoryObjectStore(), "control-plane")
|
||||
amis, err := store.ListAMIs(context.Background())
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, amis)
|
||||
}
|
||||
|
||||
// TestListAMIs_ReturnsAllStoredAMIs mirrors TestListVolumes_ReturnsAllStoredVolumes.
|
||||
func TestListAMIs_ReturnsAllStoredAMIs(t *testing.T) {
|
||||
store := NewStore(objectstore.NewMemoryObjectStore(), "control-plane")
|
||||
ctx := context.Background()
|
||||
|
||||
want := []AMI{
|
||||
{ImageID: "ami-1", Name: "one", VolumeSizeGiB: 8},
|
||||
{ImageID: "ami-2", Name: "two", VolumeSizeGiB: 16},
|
||||
}
|
||||
for _, a := range want {
|
||||
require.NoError(t, store.PutAMI(ctx, a))
|
||||
}
|
||||
|
||||
got, err := store.ListAMIs(ctx)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, len(want))
|
||||
|
||||
gotSizes := make(map[string]uint64, len(got))
|
||||
for _, a := range got {
|
||||
gotSizes[a.ImageID] = a.VolumeSizeGiB
|
||||
}
|
||||
for _, a := range want {
|
||||
assert.Equal(t, a.VolumeSizeGiB, gotSizes[a.ImageID])
|
||||
}
|
||||
}
|
||||
|
||||
// TestListAMIs_NotConfigured mirrors TestListVolumes_NotConfigured.
|
||||
func TestListAMIs_NotConfigured(t *testing.T) {
|
||||
var store *Store
|
||||
_, err := store.ListAMIs(context.Background())
|
||||
require.Error(t, err)
|
||||
|
||||
empty := &Store{}
|
||||
_, err = empty.ListAMIs(context.Background())
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
// --- Legacy fallback tests ---
|
||||
// TestGetVolume_MissingDocumentIsNotFound locks the only answer a volume with
|
||||
// no document gets: it does not exist as far as the control plane is
|
||||
// concerned, reported as the object store's not-found rather than a zero value.
|
||||
func TestGetVolume_MissingDocumentIsNotFound(t *testing.T) {
|
||||
store := NewStore(objectstore.NewMemoryObjectStore(), "control-plane")
|
||||
_, err := store.GetVolume(context.Background(), "vol-missing")
|
||||
require.Error(t, err)
|
||||
assert.True(t, objectstore.IsNoSuchKeyError(err))
|
||||
}
|
||||
|
||||
// TestGetAMI_MissingDocumentIsNotFound mirrors TestGetVolume_MissingDocumentIsNotFound.
|
||||
func TestGetAMI_MissingDocumentIsNotFound(t *testing.T) {
|
||||
store := NewStore(objectstore.NewMemoryObjectStore(), "control-plane")
|
||||
_, err := store.GetAMI(context.Background(), "ami-missing")
|
||||
require.Error(t, err)
|
||||
assert.True(t, objectstore.IsNoSuchKeyError(err))
|
||||
}
|
||||
|
||||
// TestGet_CorruptDocumentIsDistinguishable is what lets the admin tooling tell
|
||||
// salvage from not-found: an undecodable document must not read as absent, or
|
||||
// a --force removal would refuse the one case it exists for.
|
||||
func TestGet_CorruptDocumentIsDistinguishable(t *testing.T) {
|
||||
objects := objectstore.NewMemoryObjectStore()
|
||||
store := NewStore(objects, "control-plane")
|
||||
ctx := context.Background()
|
||||
|
||||
volKey, err := VolumeKey("vol-corrupt")
|
||||
require.NoError(t, err)
|
||||
writeRaw(t, objects, volKey, []byte("{not json"))
|
||||
amiKey, err := AMIKey("ami-corrupt")
|
||||
require.NoError(t, err)
|
||||
writeRaw(t, objects, amiKey, []byte("{not json"))
|
||||
|
||||
_, err = store.GetVolume(ctx, "vol-corrupt")
|
||||
require.ErrorIs(t, err, ErrCorruptDocument)
|
||||
assert.False(t, objectstore.IsNoSuchKeyError(err), "corrupt must not read as absent")
|
||||
|
||||
_, err = store.GetAMI(ctx, "ami-corrupt")
|
||||
require.ErrorIs(t, err, ErrCorruptDocument)
|
||||
assert.False(t, objectstore.IsNoSuchKeyError(err), "corrupt must not read as absent")
|
||||
}
|
||||
|
||||
// writeRaw puts bytes at key without going through Marshal, so a test can
|
||||
// store a document the store cannot decode.
|
||||
func writeRaw(t *testing.T, objects objectstore.ObjectStore, key string, data []byte) {
|
||||
t.Helper()
|
||||
_, err := objects.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("control-plane"), Key: aws.String(key), Body: bytes.NewReader(data),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// TestListAMIs_SkipsCorruptButStrictDoesNot pins the blast radius of one bad
|
||||
// document: DescribeImages keeps working and loses only the unreadable image,
|
||||
// while a caller that cannot answer partially still gets the error.
|
||||
func TestListAMIs_SkipsCorruptButStrictDoesNot(t *testing.T) {
|
||||
objects := objectstore.NewMemoryObjectStore()
|
||||
store := NewStore(objects, "control-plane")
|
||||
ctx := context.Background()
|
||||
|
||||
require.NoError(t, store.PutAMI(ctx, AMI{ImageID: "ami-good", Name: "readable"}))
|
||||
badKey, err := AMIKey("ami-bad")
|
||||
require.NoError(t, err)
|
||||
writeRaw(t, objects, badKey, []byte("{not json"))
|
||||
|
||||
got, err := store.ListAMIs(ctx)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 1, "the readable AMI must survive its neighbour being corrupt")
|
||||
assert.Equal(t, "ami-good", got[0].ImageID)
|
||||
|
||||
_, err = store.ListAMIsStrict(ctx)
|
||||
require.ErrorIs(t, err, ErrCorruptDocument)
|
||||
}
|
||||
@@ -0,0 +1,185 @@
|
||||
package conformance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
)
|
||||
|
||||
// defaultBenchVolumeBytes is the size a benchmarked volume is created at when
|
||||
// BenchConfig does not say. A provider that allocates lazily costs the same at
|
||||
// any size, and one that does not should be measured at a size its operator
|
||||
// recognises rather than at whatever this file picked.
|
||||
const defaultBenchVolumeBytes int64 = 1 << 30
|
||||
|
||||
// BenchConfig tunes a benchmark run for the provider it is pointed at.
|
||||
type BenchConfig struct {
|
||||
SuiteConfig
|
||||
|
||||
// VolumeBytes sizes every volume the suite creates. Against a live store
|
||||
// this wants to be small enough that a run finishes and large enough that
|
||||
// the provider does real work.
|
||||
VolumeBytes int64
|
||||
}
|
||||
|
||||
func (c BenchConfig) volumeBytes() int64 {
|
||||
if c.VolumeBytes <= 0 {
|
||||
return defaultBenchVolumeBytes
|
||||
}
|
||||
return c.VolumeBytes
|
||||
}
|
||||
|
||||
// benchProvider is what a benchmark measures. Unlike the conformance suite it
|
||||
// is built once for the whole run: constructing a provider per iteration would
|
||||
// price the constructor, which is not a verb.
|
||||
type benchProvider struct {
|
||||
provider ebsprovider.EBSProvider
|
||||
cfg BenchConfig
|
||||
}
|
||||
|
||||
func (p benchProvider) createVolume(b *testing.B, volumeID string) {
|
||||
b.Helper()
|
||||
_, err := p.provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: p.cfg.volumeBytes()},
|
||||
})
|
||||
if err != nil {
|
||||
b.Fatalf("create %s: %v", volumeID, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (p benchProvider) deleteVolume(b *testing.B, volumeID string) {
|
||||
b.Helper()
|
||||
if err := p.provider.DeleteVolume(context.Background(), ebsprovider.DeleteVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
}); err != nil {
|
||||
b.Fatalf("delete %s: %v", volumeID, err)
|
||||
}
|
||||
}
|
||||
|
||||
// RunBenchSuite prices one round trip per verb. Every implementation is driven
|
||||
// through the same calls in the same order, so two providers' numbers are
|
||||
// comparable and the only difference is what answers.
|
||||
//
|
||||
// It measures control operations. Guest I/O never crosses these verbs, so no
|
||||
// number here says anything about data-path throughput.
|
||||
func RunBenchSuite(b *testing.B, provider ebsprovider.EBSProvider, cfg BenchConfig) {
|
||||
p := benchProvider{provider: provider, cfg: cfg}
|
||||
capabilities := benchCapabilities(b, provider)
|
||||
|
||||
b.Run("GetCapabilities", func(b *testing.B) {
|
||||
for b.Loop() {
|
||||
if _, err := provider.GetCapabilities(context.Background(), ebsprovider.GetCapabilitiesRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Create and delete are measured as a pair rather than separately: a
|
||||
// benchmark that only created would leave b.N volumes behind on a live
|
||||
// provider, and one that only deleted would have nothing to delete.
|
||||
b.Run("CreateDeleteVolume", func(b *testing.B) {
|
||||
i := 0
|
||||
for b.Loop() {
|
||||
volumeID := cfg.id(fmt.Sprintf("vol-bench%08d", i))
|
||||
p.createVolume(b, volumeID)
|
||||
p.deleteVolume(b, volumeID)
|
||||
i++
|
||||
}
|
||||
})
|
||||
|
||||
b.Run("GetVolume", func(b *testing.B) {
|
||||
volumeID := cfg.id("vol-benchdescribe")
|
||||
p.createVolume(b, volumeID)
|
||||
b.Cleanup(func() { p.deleteVolume(b, volumeID) })
|
||||
b.ResetTimer()
|
||||
for b.Loop() {
|
||||
if _, err := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
if capabilities.VolumeEnumeration {
|
||||
b.Run("ListVolumes", func(b *testing.B) {
|
||||
for b.Loop() {
|
||||
if _, err := provider.ListVolumes(context.Background(), ebsprovider.ListVolumesRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Publish and unpublish are the attach path, and the pair is what an
|
||||
// instance launch actually costs. Measuring publish alone would leave the
|
||||
// volume attached and the next iteration would fail on it.
|
||||
b.Run("PublishUnpublishVolume", func(b *testing.B) {
|
||||
volumeID := cfg.id("vol-benchpublish0")
|
||||
p.createVolume(b, volumeID)
|
||||
b.Cleanup(func() { p.deleteVolume(b, volumeID) })
|
||||
b.ResetTimer()
|
||||
for b.Loop() {
|
||||
if _, err := provider.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
NodeID: cfg.node(),
|
||||
}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
if err := provider.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
NodeID: cfg.node(),
|
||||
}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
b.Run("CreateDeleteSnapshot", func(b *testing.B) {
|
||||
volumeID := cfg.id("vol-benchsnapshot")
|
||||
p.createVolume(b, volumeID)
|
||||
b.Cleanup(func() { p.deleteVolume(b, volumeID) })
|
||||
b.ResetTimer()
|
||||
i := 0
|
||||
for b.Loop() {
|
||||
snapshotID := cfg.id(fmt.Sprintf("snap-bench%08d", i))
|
||||
if _, err := provider.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
SnapshotID: snapshotID,
|
||||
}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
if err := provider.DeleteSnapshot(context.Background(), ebsprovider.DeleteSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
SnapshotID: snapshotID,
|
||||
}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
i++
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func benchCapabilities(b *testing.B, provider ebsprovider.EBSProvider) ebsprovider.Capabilities {
|
||||
b.Helper()
|
||||
resp, err := provider.GetCapabilities(context.Background(), ebsprovider.GetCapabilitiesRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
})
|
||||
if err != nil {
|
||||
b.Fatalf("get capabilities: %v", err)
|
||||
}
|
||||
return resp.Capabilities
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package conformance_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/conformance"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/natsserve"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/nullprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
)
|
||||
|
||||
// BenchmarkNullProvider_InProcess prices the contract with no transport and no
|
||||
// storage. Anything a real provider costs above this is its own.
|
||||
func BenchmarkNullProvider_InProcess(b *testing.B) {
|
||||
conformance.RunBenchSuite(b, nullprovider.New(), conformance.BenchConfig{})
|
||||
}
|
||||
|
||||
// BenchmarkNullProvider_OverNATS adds the transport and nothing else, so the
|
||||
// gap between it and the in-process run is what the seam costs: encode,
|
||||
// publish, queue dispatch, decode.
|
||||
func BenchmarkNullProvider_OverNATS(b *testing.B) {
|
||||
_, conn := testutil.StartTestNATS(b)
|
||||
client, stop, err := nullprovider.Serve(b.Context(), conn, natsserve.Options{NoQueueGroup: true})
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
b.Cleanup(stop)
|
||||
conformance.RunBenchSuite(b, client, conformance.BenchConfig{})
|
||||
}
|
||||
@@ -0,0 +1,933 @@
|
||||
// Package conformance exercises the ebsprovider.EBSProvider contract
|
||||
// itself, not any one implementation's internals, so it can run unmodified
|
||||
// against every implementation of the interface; no assertion here has changed from the original suite it was moved out of.
|
||||
package conformance
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// ReferenceCapabilities is what a full-featured provider advertises. It is
|
||||
// only a convenience for callers constructing a MemoryProvider; the suite
|
||||
// itself never assumes any implementation reports this set.
|
||||
var ReferenceCapabilities = ebsprovider.Capabilities{
|
||||
OnlineExpansion: false,
|
||||
SparseExtentReporting: true,
|
||||
CrashConsistentSnapshot: true,
|
||||
VolumeSeeding: true,
|
||||
ReadOnlyPublish: true,
|
||||
VolumeEnumeration: true,
|
||||
SnapshotEnumeration: true,
|
||||
// MemoryProvider keeps publication in one process's map, so node is the
|
||||
// honest answer. A reference set that claimed cluster would make the
|
||||
// suite's concurrency subtests pass against something that cannot hold.
|
||||
Exclusion: ebsprovider.ExclusionSemantics{Scope: ebsprovider.ExclusionScopeNode},
|
||||
}
|
||||
|
||||
// capabilitiesOf reads what a provider advertises, so the suite can branch on
|
||||
// it the way the contract tells callers to.
|
||||
func capabilitiesOf(t *testing.T, provider ebsprovider.EBSProvider) ebsprovider.Capabilities {
|
||||
t.Helper()
|
||||
resp, err := provider.GetCapabilities(context.Background(), ebsprovider.GetCapabilitiesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, resp)
|
||||
return resp.Capabilities
|
||||
}
|
||||
|
||||
// maxListPages bounds a drain so a provider whose token fails to advance
|
||||
// fails the test instead of hanging it.
|
||||
const maxListPages = 100
|
||||
|
||||
// drainVolumeIDs walks every page of ListVolumes and returns the IDs in the
|
||||
// order they arrived, duplicates included: detecting a token that replays or
|
||||
// overshoots a boundary is the point, so this must not deduplicate.
|
||||
func drainVolumeIDs(t *testing.T, provider ebsprovider.EBSProvider, maxResults int32) []string {
|
||||
t.Helper()
|
||||
var ids []string
|
||||
var token string
|
||||
for pages := 0; ; pages++ {
|
||||
require.Lessf(t, pages, maxListPages, "ListVolumes did not terminate within %d pages; the next token is not advancing", maxListPages)
|
||||
resp, err := provider.ListVolumes(context.Background(), ebsprovider.ListVolumesRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
MaxResults: maxResults,
|
||||
StartingToken: token,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, resp)
|
||||
for _, ref := range resp.Volumes {
|
||||
ids = append(ids, ref.ID)
|
||||
}
|
||||
if resp.NextToken == "" {
|
||||
return ids
|
||||
}
|
||||
token = resp.NextToken
|
||||
}
|
||||
}
|
||||
|
||||
// drainSnapshotIDs is drainVolumeIDs for snapshots, and keeps duplicates for
|
||||
// the same reason: a token that replays a boundary must be visible.
|
||||
func drainSnapshotIDs(t *testing.T, provider ebsprovider.EBSProvider, maxResults int32) []string {
|
||||
t.Helper()
|
||||
var ids []string
|
||||
var token string
|
||||
for pages := 0; ; pages++ {
|
||||
require.Lessf(t, pages, maxListPages, "ListSnapshots did not terminate within %d pages; the next token is not advancing", maxListPages)
|
||||
resp, err := provider.ListSnapshots(context.Background(), ebsprovider.ListSnapshotsRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
MaxResults: maxResults,
|
||||
StartingToken: token,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, resp)
|
||||
for _, ref := range resp.Snapshots {
|
||||
ids = append(ids, ref.ID)
|
||||
}
|
||||
if resp.NextToken == "" {
|
||||
return ids
|
||||
}
|
||||
token = resp.NextToken
|
||||
}
|
||||
}
|
||||
|
||||
// SuiteConfig tunes a run for the provider it is pointed at.
|
||||
type SuiteConfig struct {
|
||||
// NamePrefix is inserted into every volume and snapshot ID the suite
|
||||
// creates, after the vol-/snap- token. A provider that keeps state across
|
||||
// runs needs it: without it a second run collides with the first run's
|
||||
// volumes and fails on already_exists.
|
||||
NamePrefix string
|
||||
|
||||
// NodeID is the node PublishVolume targets. A live provider answers only
|
||||
// for the node it runs on, so the in-process default is wrong there.
|
||||
NodeID string
|
||||
|
||||
// OtherNodeID is a second node, used to prove a published volume cannot
|
||||
// also be published elsewhere. Empty skips that subtest, for a caller with
|
||||
// only one node to offer.
|
||||
OtherNodeID string
|
||||
}
|
||||
|
||||
func (c SuiteConfig) node() string {
|
||||
if c.NodeID == "" {
|
||||
return "node-1"
|
||||
}
|
||||
return c.NodeID
|
||||
}
|
||||
|
||||
func (c SuiteConfig) otherNode() string {
|
||||
if c.NodeID == "" && c.OtherNodeID == "" {
|
||||
return "node-2"
|
||||
}
|
||||
return c.OtherNodeID
|
||||
}
|
||||
|
||||
// id builds a suite-owned identifier, keeping the vol-/snap- prefix the rest of
|
||||
// the system reads as a type token.
|
||||
func (c SuiteConfig) id(base string) string {
|
||||
if c.NamePrefix == "" {
|
||||
return base
|
||||
}
|
||||
if kind, rest, ok := strings.Cut(base, "-"); ok {
|
||||
return kind + "-" + c.NamePrefix + rest
|
||||
}
|
||||
return c.NamePrefix + base
|
||||
}
|
||||
|
||||
// RunSuite runs the full EBSProvider conformance suite against a provider
|
||||
// newProvider constructs fresh for each subtest.
|
||||
func RunSuite(t *testing.T, newProvider func(t *testing.T) ebsprovider.EBSProvider) {
|
||||
RunSuiteWithConfig(t, newProvider, SuiteConfig{})
|
||||
}
|
||||
|
||||
// RunSuiteWithConfig is RunSuite with control over the identifiers it uses, so
|
||||
// the same suite can run against a live provider whose state outlives the test.
|
||||
// Optional behaviour is gated on what the provider advertises, never on a set
|
||||
// the suite picked.
|
||||
func RunSuiteWithConfig(t *testing.T, newRawProvider func(t *testing.T) ebsprovider.EBSProvider, cfg SuiteConfig) {
|
||||
// Everything the suite creates is tracked and torn down, so a live
|
||||
// provider is left as the run found it.
|
||||
newProvider := func(t *testing.T) ebsprovider.EBSProvider {
|
||||
return newTrackingProvider(t, newRawProvider(t))
|
||||
}
|
||||
|
||||
capabilities := capabilitiesOf(t, newProvider(t))
|
||||
|
||||
t.Run("GetCapabilities", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
resp, err := provider.GetCapabilities(context.Background(), ebsprovider.GetCapabilitiesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, resp)
|
||||
assert.Equal(t, ebsprovider.SchemaVersion, resp.SchemaVersion, "response must carry the negotiated schema version")
|
||||
|
||||
// Capabilities describe the implementation, so they must not vary
|
||||
// between calls: a caller branches on them once and caches.
|
||||
again, err := provider.GetCapabilities(context.Background(), ebsprovider.GetCapabilitiesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, resp.Capabilities, again.Capabilities, "GetCapabilities must be stable across calls")
|
||||
})
|
||||
|
||||
t.Run("CreateVolume", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
vol, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-create-ok"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, vol)
|
||||
assert.Equal(t, cfg.id("vol-create-ok"), vol.ID)
|
||||
assert.Equal(t, int64(1<<30), vol.CapacityBytes)
|
||||
assert.Equal(t, ebsprovider.VolumeStateAvailable, vol.State)
|
||||
assert.NotEmpty(t, vol.Handle)
|
||||
})
|
||||
|
||||
t.Run("already_exists on conflicting recreate", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-conflict"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-conflict"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 2 << 30}})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrAlreadyExists)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument on empty volume id", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
t.Run("not_found on absent source snapshot", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-from-missing-snap"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SourceSnapshotID: cfg.id("snap-missing"), SourceSnapshotVolumeID: cfg.id("vol-missing-origin"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
})
|
||||
|
||||
// SourceSnapshotVolumeID names the volume the snapshot came from, which
|
||||
// a provider may need to resolve the snapshot's blocks. Omitting it is a
|
||||
// malformed request, not a request for behaviour the provider lacks.
|
||||
t.Run("invalid_argument when a source snapshot has no source volume", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-snapsrc-noorigin"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}, SourceSnapshotID: cfg.id("snap-any"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
t.Run("volume is created from a snapshot", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-snapsrc-origin"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
snap, err := provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-snapsrc"), VolumeID: cfg.id("vol-snapsrc-origin"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, cfg.id("vol-snapsrc-origin"), snap.SourceVolumeID, "a snapshot must report the volume it was taken from")
|
||||
|
||||
restored, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-snapsrc-restored"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SourceSnapshotID: snap.ID, SourceSnapshotVolumeID: snap.SourceVolumeID,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, ebsprovider.VolumeStateAvailable, restored.State)
|
||||
})
|
||||
|
||||
// The snapshot records its own origin, so a request naming a different
|
||||
// one is checkable and wrong. Accepting it resolves the snapshot's
|
||||
// blocks against a volume they were never written to.
|
||||
t.Run("invalid_argument on a source volume the snapshot did not come from", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-wrongorigin-src"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
snap, err := provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-wrongorigin"), VolumeID: cfg.id("vol-wrongorigin-src"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-wrongorigin-restored"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SourceSnapshotID: snap.ID, SourceSnapshotVolumeID: cfg.id("vol-wrongorigin-other"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
t.Run("unsupported_version", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
VolumeID: cfg.id("vol-noversion"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedVersion)
|
||||
})
|
||||
|
||||
t.Run("seeded volume is created", func(t *testing.T) {
|
||||
if !capabilities.VolumeSeeding {
|
||||
t.Skip("provider does not advertise VolumeSeeding")
|
||||
}
|
||||
provider := newProvider(t)
|
||||
vol, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-seeded"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4096},
|
||||
SeedData: bytes.Repeat([]byte{0xAB}, 4096),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, vol)
|
||||
assert.Equal(t, int64(4096), vol.CapacityBytes)
|
||||
})
|
||||
|
||||
// A provider that cannot seed must say so, not accept the seed and
|
||||
// drop it: the caller has no other way to learn the volume is blank.
|
||||
t.Run("unsupported_capability on seed when seeding is not advertised", func(t *testing.T) {
|
||||
if capabilities.VolumeSeeding {
|
||||
t.Skip("provider advertises VolumeSeeding")
|
||||
}
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-seed-unsupported"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4096},
|
||||
SeedData: bytes.Repeat([]byte{0xAB}, 4096),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedCapability)
|
||||
})
|
||||
|
||||
// A seed above MaxSeedBytes must fail as invalid_argument rather than
|
||||
// reaching the transport, where NATS would refuse the oversized publish
|
||||
// with an error that says nothing about firmware.
|
||||
t.Run("invalid_argument on oversized seed", func(t *testing.T) {
|
||||
if !capabilities.VolumeSeeding {
|
||||
t.Skip("provider does not advertise VolumeSeeding")
|
||||
}
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-seed-toobig"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SeedData: make([]byte, ebsprovider.MaxSeedBytes+1),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument when seed exceeds capacity", func(t *testing.T) {
|
||||
if !capabilities.VolumeSeeding {
|
||||
t.Skip("provider does not advertise VolumeSeeding")
|
||||
}
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-seed-overcap"),
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 512},
|
||||
SeedData: make([]byte, 4096),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("GetVolume", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
created, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-get-ok"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
got, err := provider.GetVolume(ctx, ebsprovider.GetVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-get-ok")})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, created.Handle, got.Handle)
|
||||
assert.Equal(t, created.CapacityBytes, got.CapacityBytes)
|
||||
})
|
||||
|
||||
t.Run("not_found on absent volume", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-never-existed")})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument on empty volume id", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("ListVolumes", func(t *testing.T) {
|
||||
capabilities := capabilitiesOf(t, newProvider(t))
|
||||
|
||||
t.Run("unsupported_capability when enumeration is not advertised", func(t *testing.T) {
|
||||
if capabilities.VolumeEnumeration {
|
||||
t.Skip("provider advertises volume enumeration")
|
||||
}
|
||||
_, err := newProvider(t).ListVolumes(context.Background(), ebsprovider.ListVolumesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedCapability)
|
||||
})
|
||||
|
||||
if !capabilities.VolumeEnumeration {
|
||||
return
|
||||
}
|
||||
|
||||
t.Run("a created volume is enumerated and a deleted one is not", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
volumeID := cfg.id("vol-list-visible")
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, drainVolumeIDs(t, provider, 0), volumeID,
|
||||
"enumeration must report a volume the provider holds; it is the only way to find one whose control-plane document is lost")
|
||||
|
||||
require.NoError(t, provider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID}))
|
||||
assert.NotContains(t, drainVolumeIDs(t, provider, 0), volumeID,
|
||||
"a deleted volume must stop being enumerated, or the report it feeds accuses storage of holding blocks it released")
|
||||
})
|
||||
|
||||
t.Run("pagination returns every volume exactly once", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
want := []string{cfg.id("vol-list-page-a"), cfg.id("vol-list-page-b"), cfg.id("vol-list-page-c")}
|
||||
for _, volumeID := range want {
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// One volume per page forces every token boundary to be exercised.
|
||||
paged := drainVolumeIDs(t, provider, 1)
|
||||
seen := make(map[string]int, len(paged))
|
||||
for _, id := range paged {
|
||||
seen[id]++
|
||||
}
|
||||
for id, count := range seen {
|
||||
assert.Equalf(t, 1, count, "volume %s appeared %d times across pages; a duplicate means the token replays a boundary", id, count)
|
||||
}
|
||||
for _, volumeID := range want {
|
||||
assert.Containsf(t, paged, volumeID, "volume %s was skipped across pages; a gap means the token overshoots", volumeID)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("max results above the cap is clamped, not refused", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
resp, err := provider.ListVolumes(context.Background(), ebsprovider.ListVolumesRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
MaxResults: ebsprovider.MaxListResults * 10,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, resp)
|
||||
assert.LessOrEqual(t, len(resp.Volumes), int(ebsprovider.MaxListResults),
|
||||
"a page must never exceed MaxListResults; the reply has to fit one NATS message")
|
||||
})
|
||||
|
||||
t.Run("unsupported_version", func(t *testing.T) {
|
||||
_, err := newProvider(t).ListVolumes(context.Background(), ebsprovider.ListVolumesRequest{Versioned: ebsprovider.Versioned{SchemaVersion: ebsprovider.SchemaVersion + 1}})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedVersion)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("ExpandVolume", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-expand-ok"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
expanded, err := provider.ExpandVolume(ctx, ebsprovider.ExpandVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-expand-ok"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 2 << 30}})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(2<<30), expanded.CapacityBytes)
|
||||
})
|
||||
|
||||
t.Run("not_found on absent volume", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.ExpandVolume(context.Background(), ebsprovider.ExpandVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-never-existed"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument on shrink", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-shrink"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 2 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.ExpandVolume(ctx, ebsprovider.ExpandVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-shrink"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
// Expanding a published volume is the one operation OnlineExpansion
|
||||
// describes, so both answers are contractual: succeed, or refuse with
|
||||
// volume_in_use. Silently doing nothing is neither.
|
||||
t.Run("expanding a published volume matches OnlineExpansion", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-expand-inuse"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-expand-inuse"), NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
|
||||
expanded, err := provider.ExpandVolume(ctx, ebsprovider.ExpandVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-expand-inuse"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 2 << 30}})
|
||||
if !capabilities.OnlineExpansion {
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse)
|
||||
return
|
||||
}
|
||||
require.NoError(t, err, "provider advertises OnlineExpansion, so expanding a published volume must succeed")
|
||||
require.NotNil(t, expanded)
|
||||
assert.Equal(t, int64(2<<30), expanded.CapacityBytes)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("DeleteVolume", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-delete-ok"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, provider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-delete-ok")}))
|
||||
_, err = provider.GetVolume(ctx, ebsprovider.GetVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-delete-ok")})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
})
|
||||
|
||||
// CSI's controller service treats DeleteVolume on an absent volume as
|
||||
// success (idempotent-when-absent), not not_found.
|
||||
t.Run("absent target is idempotent", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
require.NoError(t, provider.DeleteVolume(context.Background(), ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-never-existed")}))
|
||||
})
|
||||
|
||||
t.Run("volume_in_use when published", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-delete-inuse"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-delete-inuse"), NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
err = provider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-delete-inuse")})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("CreateSnapshot", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-snap-src"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
snap, err := provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-ok"), VolumeID: cfg.id("vol-snap-src")})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, snap)
|
||||
assert.Equal(t, cfg.id("snap-ok"), snap.ID)
|
||||
assert.Equal(t, cfg.id("vol-snap-src"), snap.SourceVolumeID)
|
||||
assert.Equal(t, ebsprovider.SnapshotStateCompleted, snap.State)
|
||||
})
|
||||
|
||||
t.Run("not_found on absent source volume", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-orphan"), VolumeID: cfg.id("vol-never-existed")})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
})
|
||||
|
||||
t.Run("already_exists on conflicting source volume", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-snap-a"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-snap-b"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-conflict"), VolumeID: cfg.id("vol-snap-a")})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-conflict"), VolumeID: cfg.id("vol-snap-b")})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrAlreadyExists)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument on missing ids", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("DeleteSnapshot", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-delsnap-src"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-delete-ok"), VolumeID: cfg.id("vol-delsnap-src")})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-delete-ok")}))
|
||||
})
|
||||
|
||||
// memory.go implements delete-of-absent-snapshot as a no-op success,
|
||||
// matching CSI's idempotency rule the same way DeleteVolume does.
|
||||
t.Run("absent target is idempotent", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
require.NoError(t, provider.DeleteSnapshot(context.Background(), ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-never-existed")}))
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("CopySnapshot", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-copysnap-src"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-copysnap-src"), VolumeID: cfg.id("vol-copysnap-src")})
|
||||
require.NoError(t, err)
|
||||
|
||||
copied, err := provider.CopySnapshot(ctx, ebsprovider.CopySnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SourceSnapshotID: cfg.id("snap-copysnap-src"), DestinationSnapshotID: cfg.id("snap-copysnap-dst"), VolumeID: cfg.id("vol-copysnap-src"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, copied)
|
||||
assert.Equal(t, cfg.id("snap-copysnap-dst"), copied.ID)
|
||||
assert.Equal(t, cfg.id("vol-copysnap-src"), copied.SourceVolumeID)
|
||||
assert.Equal(t, ebsprovider.SnapshotStateCompleted, copied.State)
|
||||
assert.NotEmpty(t, copied.Handle)
|
||||
|
||||
// The destination is a real, independently addressable snapshot.
|
||||
require.NoError(t, provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-copysnap-src")}))
|
||||
require.NoError(t, provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-copysnap-dst")}))
|
||||
})
|
||||
|
||||
t.Run("not_found on absent source snapshot", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-copysnap-nosrc"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CopySnapshot(ctx, ebsprovider.CopySnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SourceSnapshotID: cfg.id("snap-copysnap-missing"), DestinationSnapshotID: cfg.id("snap-copysnap-missing-dst"), VolumeID: cfg.id("vol-copysnap-nosrc"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
})
|
||||
|
||||
t.Run("already_exists on conflicting destination", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-copysnap-conflict"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-copysnap-a"), VolumeID: cfg.id("vol-copysnap-conflict")})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-copysnap-b"), VolumeID: cfg.id("vol-copysnap-conflict")})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CopySnapshot(ctx, ebsprovider.CopySnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SourceSnapshotID: cfg.id("snap-copysnap-a"), DestinationSnapshotID: cfg.id("snap-copysnap-b"), VolumeID: cfg.id("vol-copysnap-conflict"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrAlreadyExists)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument on missing ids", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CopySnapshot(context.Background(), ebsprovider.CopySnapshotRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument when source and destination match", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-copysnap-same"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-copysnap-same"), VolumeID: cfg.id("vol-copysnap-same")})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CopySnapshot(ctx, ebsprovider.CopySnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SourceSnapshotID: cfg.id("snap-copysnap-same"), DestinationSnapshotID: cfg.id("snap-copysnap-same"), VolumeID: cfg.id("vol-copysnap-same"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument when volume id does not own the source snapshot", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-copysnap-owner"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-copysnap-foreign"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: cfg.id("snap-copysnap-owned"), VolumeID: cfg.id("vol-copysnap-owner")})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CopySnapshot(ctx, ebsprovider.CopySnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SourceSnapshotID: cfg.id("snap-copysnap-owned"), DestinationSnapshotID: cfg.id("snap-copysnap-owned-dst"), VolumeID: cfg.id("vol-copysnap-foreign"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
|
||||
t.Run("unsupported_version", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.CopySnapshot(context.Background(), ebsprovider.CopySnapshotRequest{
|
||||
SourceSnapshotID: cfg.id("snap-a"), DestinationSnapshotID: cfg.id("snap-b"), VolumeID: cfg.id("vol-a"),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedVersion)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("ListSnapshots", func(t *testing.T) {
|
||||
capabilities := capabilitiesOf(t, newProvider(t))
|
||||
|
||||
t.Run("unsupported_capability when enumeration is not advertised", func(t *testing.T) {
|
||||
if capabilities.SnapshotEnumeration {
|
||||
t.Skip("provider advertises snapshot enumeration")
|
||||
}
|
||||
_, err := newProvider(t).ListSnapshots(context.Background(), ebsprovider.ListSnapshotsRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedCapability)
|
||||
})
|
||||
|
||||
if !capabilities.SnapshotEnumeration {
|
||||
return
|
||||
}
|
||||
|
||||
t.Run("a created snapshot is enumerated and a deleted one is not", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
volumeID := cfg.id("vol-snaplist-src")
|
||||
snapshotID := cfg.id("snap-list-visible")
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, VolumeID: volumeID})
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, drainSnapshotIDs(t, provider, 0), snapshotID,
|
||||
"enumeration must report a snapshot the provider holds; it is the only way to find one whose control-plane document is lost")
|
||||
|
||||
require.NoError(t, provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID}))
|
||||
assert.NotContains(t, drainSnapshotIDs(t, provider, 0), snapshotID,
|
||||
"a deleted snapshot must stop being enumerated, or the report it feeds accuses storage of holding data it released")
|
||||
})
|
||||
|
||||
t.Run("pagination returns every snapshot exactly once", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
volumeID := cfg.id("vol-snaplist-page-src")
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
want := []string{cfg.id("snap-list-page-a"), cfg.id("snap-list-page-b"), cfg.id("snap-list-page-c")}
|
||||
for _, snapshotID := range want {
|
||||
_, err := provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, VolumeID: volumeID})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// One snapshot per page forces every token boundary to be exercised.
|
||||
paged := drainSnapshotIDs(t, provider, 1)
|
||||
seen := make(map[string]int, len(paged))
|
||||
for _, id := range paged {
|
||||
seen[id]++
|
||||
}
|
||||
for id, count := range seen {
|
||||
assert.Equalf(t, 1, count, "snapshot %s appeared %d times across pages; a duplicate means the token replays a boundary", id, count)
|
||||
}
|
||||
for _, snapshotID := range want {
|
||||
assert.Containsf(t, paged, snapshotID, "snapshot %s was skipped across pages; a gap means the token overshoots", snapshotID)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("max results above the cap is clamped, not refused", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
resp, err := provider.ListSnapshots(context.Background(), ebsprovider.ListSnapshotsRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
MaxResults: ebsprovider.MaxListResults * 10,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, resp)
|
||||
assert.LessOrEqual(t, len(resp.Snapshots), int(ebsprovider.MaxListResults),
|
||||
"a page must never exceed MaxListResults; the reply has to fit one NATS message")
|
||||
})
|
||||
|
||||
t.Run("unsupported_version", func(t *testing.T) {
|
||||
_, err := newProvider(t).ListSnapshots(context.Background(), ebsprovider.ListSnapshotsRequest{Versioned: ebsprovider.Versioned{SchemaVersion: ebsprovider.SchemaVersion + 1}})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedVersion)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("PublishVolume", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-ok"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
pub, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-ok"), NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, pub)
|
||||
assert.Equal(t, cfg.id("vol-pub-ok"), pub.VolumeID)
|
||||
assert.Equal(t, cfg.node(), pub.NodeID)
|
||||
assertNBDURI(t, pub.NBDURI)
|
||||
})
|
||||
|
||||
t.Run("idempotent republish to the same node", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-idem"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
first, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-idem"), NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
second, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-idem"), NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, first, second)
|
||||
})
|
||||
|
||||
t.Run("volume_in_use when published to a different node", func(t *testing.T) {
|
||||
if cfg.otherNode() == "" {
|
||||
t.Skip("no second node configured")
|
||||
}
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-conflict"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-conflict"), NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-pub-conflict"), NodeID: cfg.otherNode()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse)
|
||||
})
|
||||
|
||||
t.Run("not_found on absent volume", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-never-existed"), NodeID: cfg.node()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
})
|
||||
|
||||
t.Run("invalid_argument on missing ids", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
_, err := provider.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("UnpublishVolume", func(t *testing.T) {
|
||||
t.Run("success", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-unpub-ok"), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-unpub-ok"), NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-unpub-ok"), NodeID: cfg.node()}))
|
||||
})
|
||||
|
||||
t.Run("absent target is idempotent", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
require.NoError(t, provider.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.id("vol-never-existed"), NodeID: cfg.node()}))
|
||||
})
|
||||
|
||||
t.Run("invalid_argument on missing ids", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
err := provider.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
})
|
||||
})
|
||||
|
||||
t.Run("Exclusion", func(t *testing.T) {
|
||||
exclusion := capabilities.Exclusion
|
||||
|
||||
// An advertised guarantee that contradicts itself is worse than a
|
||||
// missing one: a caller reads it and plans around something the
|
||||
// provider never claimed to do.
|
||||
t.Run("advertised semantics are self-consistent", func(t *testing.T) {
|
||||
switch exclusion.Scope {
|
||||
case ebsprovider.ExclusionScopeNone, ebsprovider.ExclusionScopeNode, ebsprovider.ExclusionScopeCluster:
|
||||
default:
|
||||
t.Fatalf("unknown exclusion scope %q; a caller cannot branch on a scope it does not recognise", exclusion.Scope)
|
||||
}
|
||||
if exclusion.Scope == ebsprovider.ExclusionScopeNone {
|
||||
assert.Zero(t, exclusion.ClaimTTLSeconds, "a provider that excludes nothing holds no claim, so a TTL on it means nothing")
|
||||
assert.False(t, exclusion.FencesLostClaim, "there is no claim to lose, so nothing can be fenced on losing it")
|
||||
}
|
||||
assert.GreaterOrEqual(t, exclusion.ClaimTTLSeconds, 0, "a negative TTL is not a duration")
|
||||
if exclusion.FencesLostClaim {
|
||||
assert.NotEqual(t, ebsprovider.ExclusionScopeNone, exclusion.Scope,
|
||||
"fencing a lost claim requires there to be a claim")
|
||||
}
|
||||
assert.Equal(t, exclusion.Scope == ebsprovider.ExclusionScopeCluster, exclusion.SingleWriter(),
|
||||
"SingleWriter must agree with Scope, or callers branching on it get a different answer from callers reading the scope")
|
||||
})
|
||||
|
||||
if exclusion.Scope == ebsprovider.ExclusionScopeNone {
|
||||
return
|
||||
}
|
||||
|
||||
t.Run("a second publish elsewhere is refused while the first holds", func(t *testing.T) {
|
||||
if cfg.otherNode() == "" {
|
||||
t.Skip("no second node configured")
|
||||
}
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
volumeID := cfg.id("vol-excl-second")
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: cfg.otherNode()})
|
||||
require.ErrorIsf(t, err, ebsprovider.ErrVolumeInUse,
|
||||
"scope %q promises a second writer is refused; letting this through is the two-writer case the guarantee exists to prevent", exclusion.Scope)
|
||||
|
||||
// Releasing the claim must make the volume publishable again, or
|
||||
// the guarantee is indistinguishable from a permanent lock.
|
||||
require.NoError(t, provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: cfg.node()}))
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: cfg.otherNode()})
|
||||
require.NoError(t, err, "a released claim must be reclaimable, or an unpublish strands the volume")
|
||||
})
|
||||
|
||||
// Concurrent publishes settle to one winner. Both succeeding is the
|
||||
// corruption case; both failing means the claim was lost by neither
|
||||
// and the volume is stranded.
|
||||
t.Run("racing publishes from two nodes produce exactly one winner", func(t *testing.T) {
|
||||
if cfg.otherNode() == "" {
|
||||
t.Skip("no second node configured")
|
||||
}
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
volumeID := cfg.id("vol-excl-race")
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
|
||||
nodes := []string{cfg.node(), cfg.otherNode()}
|
||||
errs := make([]error, len(nodes))
|
||||
start := make(chan struct{})
|
||||
var wg sync.WaitGroup
|
||||
for i, node := range nodes {
|
||||
wg.Go(func() {
|
||||
<-start
|
||||
_, errs[i] = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: node,
|
||||
})
|
||||
})
|
||||
}
|
||||
close(start)
|
||||
wg.Wait()
|
||||
|
||||
var winners int
|
||||
for i, err := range errs {
|
||||
if err == nil {
|
||||
winners++
|
||||
continue
|
||||
}
|
||||
assert.ErrorIsf(t, err, ebsprovider.ErrVolumeInUse,
|
||||
"the loser of a publish race must be told the volume is in use, not %v; node %s cannot retry sensibly otherwise", err, nodes[i])
|
||||
}
|
||||
assert.Equalf(t, 1, winners, "%d of %d racing publishes succeeded; two winners is the two-writer case, zero strands the volume", winners, len(nodes))
|
||||
})
|
||||
|
||||
// A delete arriving while a volume is published must lose to the
|
||||
// publication, not race it. Deleting under a live writer destroys
|
||||
// blocks the writer still believes it owns.
|
||||
t.Run("delete racing a live publication is refused", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
ctx := context.Background()
|
||||
volumeID := cfg.id("vol-excl-delrace")
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: cfg.node()})
|
||||
require.NoError(t, err)
|
||||
|
||||
err = provider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse,
|
||||
"deleting a published volume must be refused; the export is still serving reads and writes against it")
|
||||
|
||||
require.NoError(t, provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: cfg.node()}))
|
||||
require.NoError(t, provider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID}),
|
||||
"once unpublished the delete must go through, or a volume can never be removed after being used")
|
||||
})
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package conformance_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/conformance"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/natsserve"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMemoryProviderConformance(t *testing.T) {
|
||||
conformance.RunSuite(t, func(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
return ebsprovider.NewMemoryProvider(conformance.ReferenceCapabilities)
|
||||
})
|
||||
}
|
||||
|
||||
// TestMemoryProviderMinimalCapabilitiesConformance is what makes the suite
|
||||
// capability-driven rather than capability-asserting: the same suite, a
|
||||
// provider advertising none of the optional behaviour, and it still passes.
|
||||
func TestMemoryProviderMinimalCapabilitiesConformance(t *testing.T) {
|
||||
conformance.RunSuite(t, func(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
return ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
})
|
||||
}
|
||||
|
||||
// TestMemoryProviderOnlineExpansionConformance covers the branch no provider
|
||||
// we ship exercises: expanding a published volume must succeed when the
|
||||
// provider advertises that it can.
|
||||
func TestMemoryProviderOnlineExpansionConformance(t *testing.T) {
|
||||
conformance.RunSuite(t, func(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
capabilities := conformance.ReferenceCapabilities
|
||||
capabilities.OnlineExpansion = true
|
||||
return ebsprovider.NewMemoryProvider(capabilities)
|
||||
})
|
||||
}
|
||||
|
||||
// TestMemoryProviderSharedInstanceConformance runs the suite twice against one
|
||||
// provider instance, standing in for a live daemon whose volumes outlive the
|
||||
// test binary. It passes only if the suite removes everything it creates: with
|
||||
// no cleanup the second run collides with the first on already_exists.
|
||||
func TestMemoryProviderSharedInstanceConformance(t *testing.T) {
|
||||
shared := ebsprovider.NewMemoryProvider(conformance.ReferenceCapabilities)
|
||||
newProvider := func(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
return shared
|
||||
}
|
||||
|
||||
for _, run := range []string{"first run", "second run"} {
|
||||
t.Run(run, func(t *testing.T) {
|
||||
conformance.RunSuiteWithConfig(t, newProvider, conformance.SuiteConfig{})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestMemoryProviderNamePrefixConformance covers the other half of running
|
||||
// against a live provider: distinct prefixes let concurrent runs share one
|
||||
// daemon without naming the same volumes.
|
||||
func TestMemoryProviderNamePrefixConformance(t *testing.T) {
|
||||
shared := ebsprovider.NewMemoryProvider(conformance.ReferenceCapabilities)
|
||||
newProvider := func(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
return shared
|
||||
}
|
||||
|
||||
for _, prefix := range []string{"alpha-", "beta-"} {
|
||||
t.Run(prefix, func(t *testing.T) {
|
||||
conformance.RunSuiteWithConfig(t, newProvider, conformance.SuiteConfig{NamePrefix: prefix})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestNATSProviderConformance runs the suite against a NATSProvider backed
|
||||
// by natsserve.Serve, the production-shaped neutral server, rather than a
|
||||
// test-only twin: this exercises the transport this package's implementors actually run in production, not a hand-written stand-in for it.
|
||||
func TestNATSProviderConformance(t *testing.T) {
|
||||
conformance.RunSuite(t, func(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
stop, err := natsserve.Serve(t.Context(), conn, ebsprovider.NewMemoryProvider(conformance.ReferenceCapabilities), natsserve.Options{})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(stop)
|
||||
return ebsprovider.NewNATSProvider(conn, 5*time.Second)
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
package conformance
|
||||
|
||||
// The rest of this package prices the control verbs: create, publish,
|
||||
// describe. Guest I/O never crosses those, so none of those numbers say
|
||||
// anything about what a guest actually experiences. This file drives the
|
||||
// published export with qemu-img bench, which is the same block layer a
|
||||
// guest's virtio-blk sits on.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// dataPathVolumeBytes is large enough that a strided profile can skip far
|
||||
// enough to defeat readahead, and small enough to stay quick over a real
|
||||
// store.
|
||||
const dataPathVolumeBytes int64 = 256 << 20
|
||||
|
||||
// DataPathProfile is one qemu-img bench run: a request size, a queue depth
|
||||
// and how far apart consecutive requests land.
|
||||
type DataPathProfile struct {
|
||||
Name string
|
||||
|
||||
// BufferSize is the size of each request.
|
||||
BufferSize int64
|
||||
|
||||
// StepSize is the offset increment between requests. Zero means
|
||||
// sequential, i.e. one buffer.
|
||||
StepSize int64
|
||||
|
||||
Count int
|
||||
Depth int
|
||||
Write bool
|
||||
|
||||
// FlushInterval, when set, flushes every N write requests. Without it a
|
||||
// write profile measures how fast the WAL accepts data, not how fast the
|
||||
// store takes it: chunk upload is asynchronous.
|
||||
FlushInterval int
|
||||
}
|
||||
|
||||
func (p DataPathProfile) step() int64 {
|
||||
if p.StepSize == 0 {
|
||||
return p.BufferSize
|
||||
}
|
||||
return p.StepSize
|
||||
}
|
||||
|
||||
// bytes is how much payload the profile moves, which is not the span it
|
||||
// covers when it strides.
|
||||
func (p DataPathProfile) bytes() int64 {
|
||||
return p.BufferSize * int64(p.Count)
|
||||
}
|
||||
|
||||
// DataPathResult is one profile's timing as qemu-img reported it. The
|
||||
// harness's own wall clock is deliberately not used: it would fold in
|
||||
// process startup and the NBD handshake, which a running guest pays once
|
||||
// rather than per request.
|
||||
type DataPathResult struct {
|
||||
Profile DataPathProfile
|
||||
Elapsed time.Duration
|
||||
}
|
||||
|
||||
// ThroughputMiBs is the payload rate.
|
||||
func (r DataPathResult) ThroughputMiBs() float64 {
|
||||
if r.Elapsed <= 0 {
|
||||
return 0
|
||||
}
|
||||
return float64(r.Profile.bytes()) / (1 << 20) / r.Elapsed.Seconds()
|
||||
}
|
||||
|
||||
// PerRequest is elapsed divided by request count. At depth > 1 this is
|
||||
// throughput expressed per request, not the latency of one request.
|
||||
func (r DataPathResult) PerRequest() time.Duration {
|
||||
if r.Profile.Count == 0 {
|
||||
return 0
|
||||
}
|
||||
return r.Elapsed / time.Duration(r.Profile.Count)
|
||||
}
|
||||
|
||||
// writtenSpan is how much of the volume the leading write profile covers.
|
||||
// Every read profile stays inside it: a read past the written region is
|
||||
// answered as a hole, which measures nothing.
|
||||
const writtenSpan int64 = 64 << 20
|
||||
|
||||
// DefaultDataPathProfiles writes before it reads, so the reads have
|
||||
// something to find. A read profile that ran first would be measuring how
|
||||
// fast the export can answer with zeroes.
|
||||
func DefaultDataPathProfiles() []DataPathProfile {
|
||||
return []DataPathProfile{
|
||||
{Name: "seq-write-64k", BufferSize: 64 << 10, Count: 1024, Depth: 16, Write: true},
|
||||
{Name: "seq-write-4k", BufferSize: 4 << 10, Count: 4096, Depth: 16, Write: true},
|
||||
{Name: "seq-write-64k-flush", BufferSize: 64 << 10, Count: 256, Depth: 16, Write: true, FlushInterval: 16},
|
||||
{Name: "seq-read-64k", BufferSize: 64 << 10, Count: 1024, Depth: 16},
|
||||
{Name: "seq-read-4k", BufferSize: 4 << 10, Count: 4096, Depth: 16},
|
||||
// 256 requests one 256K step apart covers exactly the written span.
|
||||
{Name: "strided-read-4k", BufferSize: 4 << 10, StepSize: writtenSpan / 256, Count: 256, Depth: 16},
|
||||
{Name: "seq-read-64k-qd1", BufferSize: 64 << 10, Count: 256, Depth: 1},
|
||||
}
|
||||
}
|
||||
|
||||
// RequireDataPathTools skips when qemu-img is absent rather than reporting a
|
||||
// data path nothing measured.
|
||||
func RequireDataPathTools(t *testing.T) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("qemu-img"); err != nil {
|
||||
t.Skip("qemu-img not installed (qemu-utils)")
|
||||
}
|
||||
}
|
||||
|
||||
// benchElapsed pulls the timing out of qemu-img bench's own report.
|
||||
var benchElapsed = regexp.MustCompile(`Run completed in ([0-9.]+) seconds`)
|
||||
|
||||
// runDataPathProfile executes one profile against uri.
|
||||
func runDataPathProfile(t *testing.T, uri string, p DataPathProfile) DataPathResult {
|
||||
t.Helper()
|
||||
|
||||
args := []string{
|
||||
"bench", "-f", "raw",
|
||||
"-c", strconv.Itoa(p.Count),
|
||||
"-s", strconv.FormatInt(p.BufferSize, 10),
|
||||
"-S", strconv.FormatInt(p.step(), 10),
|
||||
"-d", strconv.Itoa(p.Depth),
|
||||
}
|
||||
if p.Write {
|
||||
args = append(args, "-w")
|
||||
if p.FlushInterval > 0 {
|
||||
args = append(args, "--flush-interval", strconv.Itoa(p.FlushInterval))
|
||||
}
|
||||
}
|
||||
args = append(args, uri)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
out, err := exec.CommandContext(ctx, "qemu-img", args...).CombinedOutput()
|
||||
require.NoErrorf(t, err, "qemu-img %v: %s", args, out)
|
||||
|
||||
match := benchElapsed.FindSubmatch(out)
|
||||
require.NotNilf(t, match, "qemu-img bench reported no elapsed time: %s", out)
|
||||
seconds, err := strconv.ParseFloat(string(match[1]), 64)
|
||||
require.NoError(t, err)
|
||||
|
||||
return DataPathResult{Profile: p, Elapsed: time.Duration(seconds * float64(time.Second))}
|
||||
}
|
||||
|
||||
// RunDataPathSuite publishes one volume and drives every profile against it,
|
||||
// logging a table. It asserts only that each run completes and moves a
|
||||
// non-zero amount of data: what a given store is capable of is not a
|
||||
// contract, so pinning a number here would only produce a flaky test.
|
||||
func RunDataPathSuite(t *testing.T, newProvider func(t *testing.T) ebsprovider.EBSProvider, cfg NBDClientConfig) {
|
||||
RequireNBDTools(t)
|
||||
RequireDataPathTools(t)
|
||||
|
||||
cfg = cfg.withDefaults()
|
||||
if cfg.VolumeBytes < dataPathVolumeBytes {
|
||||
cfg.VolumeBytes = dataPathVolumeBytes
|
||||
}
|
||||
|
||||
provider := newProvider(t)
|
||||
volumeID := cfg.VolumePrefix + "datapath"
|
||||
|
||||
pub := publishForNBD(t, provider, cfg, volumeID, false)
|
||||
assertNBDURI(t, pub.NBDURI)
|
||||
uri := dialURI(t, pub.NBDURI)
|
||||
|
||||
export := nbdInfo(t, uri)
|
||||
require.Equalf(t, cfg.VolumeBytes, export.Size, "export size does not match the volume that was created")
|
||||
|
||||
results := make([]DataPathResult, 0, len(DefaultDataPathProfiles()))
|
||||
// Profiles run back to back deliberately. Each is its own NBD connection,
|
||||
// so a run that gets this far has also shown that an arriving connection
|
||||
// waits out the previous one's close rather than racing it.
|
||||
for _, profile := range DefaultDataPathProfiles() {
|
||||
result := runDataPathProfile(t, uri, profile)
|
||||
require.Positivef(t, result.Elapsed, "%s completed in no measurable time", profile.Name)
|
||||
results = append(results, result)
|
||||
}
|
||||
|
||||
t.Log(formatDataPathResults(results))
|
||||
}
|
||||
|
||||
// formatDataPathResults renders the table that is the point of the run.
|
||||
func formatDataPathResults(results []DataPathResult) string {
|
||||
report := fmt.Sprintf("\n%-18s %8s %6s %6s %10s %12s %12s\n",
|
||||
"profile", "req", "size", "depth", "elapsed", "MiB/s", "per-req")
|
||||
for _, r := range results {
|
||||
report += fmt.Sprintf("%-18s %8d %5dK %6d %10s %12.1f %12s\n",
|
||||
r.Profile.Name, r.Profile.Count, r.Profile.BufferSize>>10, r.Profile.Depth,
|
||||
r.Elapsed.Round(time.Millisecond), r.ThroughputMiBs(), r.PerRequest().Round(time.Microsecond))
|
||||
}
|
||||
return report
|
||||
}
|
||||
@@ -0,0 +1,380 @@
|
||||
package conformance
|
||||
|
||||
// Everything else in this package asserts the contract through our own Go
|
||||
// client. This file asserts the published export through libnbd's tools,
|
||||
// which were written with no knowledge of this codebase.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// defaultNBDVolumeBytes is small enough for a full nbdcopy round trip to be
|
||||
// quick and large enough to span many allocation units.
|
||||
const defaultNBDVolumeBytes int64 = 64 << 20
|
||||
|
||||
const defaultNBDNodeID = "node-a"
|
||||
|
||||
const defaultNBDVolumePrefix = "vol-nbd"
|
||||
|
||||
// reconnectTimeout bounds how long an export may refuse a reconnecting
|
||||
// client while it finishes tearing the previous connection down.
|
||||
const reconnectTimeout = 2 * time.Minute
|
||||
|
||||
// NBDClientConfig adapts the suite to where it runs. An in-process provider
|
||||
// needs none of it; a live cluster needs its own node name, and unique
|
||||
// volume IDs so a rerun does not collide with a previous run's leftovers.
|
||||
type NBDClientConfig struct {
|
||||
NodeID string
|
||||
VolumePrefix string
|
||||
VolumeBytes int64
|
||||
}
|
||||
|
||||
func (c NBDClientConfig) withDefaults() NBDClientConfig {
|
||||
if c.NodeID == "" {
|
||||
c.NodeID = defaultNBDNodeID
|
||||
}
|
||||
if c.VolumePrefix == "" {
|
||||
c.VolumePrefix = defaultNBDVolumePrefix
|
||||
}
|
||||
if c.VolumeBytes == 0 {
|
||||
c.VolumeBytes = defaultNBDVolumeBytes
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// nbdExport is the subset of `nbdinfo --json` this suite asserts on.
|
||||
type nbdExport struct {
|
||||
Name string `json:"export-name"`
|
||||
Size int64 `json:"export-size"`
|
||||
ReadOnly bool `json:"is_read_only"`
|
||||
BlockSizeMinimum int64 `json:"block_size_minimum"`
|
||||
BlockSizePreferred int64 `json:"block_size_preferred"`
|
||||
CanFlush bool `json:"can_flush"`
|
||||
Contexts []string `json:"contexts"`
|
||||
}
|
||||
|
||||
type nbdInfoOutput struct {
|
||||
Exports []nbdExport `json:"exports"`
|
||||
}
|
||||
|
||||
// nbdExtent is one entry of `nbdinfo --map --json`. Description is libnbd's
|
||||
// rendering of the allocation flags, e.g. "data" or "hole,zero".
|
||||
type nbdExtent struct {
|
||||
Offset int64 `json:"offset"`
|
||||
Length int64 `json:"length"`
|
||||
Description string `json:"description"`
|
||||
}
|
||||
|
||||
// assertNBDURI requires a published URI to be one an NBD client can dial:
|
||||
// nbd+unix:///?socket=/path or nbd://host:port. QEMU's legacy nbd:unix:
|
||||
// filename syntax is not a URI and no NBD client but QEMU accepts it.
|
||||
func assertNBDURI(t *testing.T, nbdURI string) {
|
||||
t.Helper()
|
||||
require.NotEmpty(t, nbdURI)
|
||||
assert.Truef(t, strings.HasPrefix(nbdURI, "nbd+unix://") || strings.HasPrefix(nbdURI, "nbd://"),
|
||||
"published NBD URI %q is not an NBD URI", nbdURI)
|
||||
}
|
||||
|
||||
// dialURI normalises whatever form a provider published into one libnbd will
|
||||
// dial, so a provider emitting the legacy syntax still gets its export
|
||||
// tested rather than every subtest failing on the same string.
|
||||
func dialURI(t *testing.T, nbdURI string) string {
|
||||
t.Helper()
|
||||
serverType, socketPath, host, port, err := utils.ParseNBDURI(nbdURI)
|
||||
require.NoErrorf(t, err, "parse published NBD URI %q", nbdURI)
|
||||
|
||||
if serverType == "unix" {
|
||||
return utils.FormatNBDSocketURI(socketPath)
|
||||
}
|
||||
return utils.FormatNBDTCPURI(host, port)
|
||||
}
|
||||
|
||||
// RequireNBDTools skips when libnbd's client tools are absent, so a host
|
||||
// that cannot run an external NBD client does not report a false pass.
|
||||
func RequireNBDTools(t *testing.T) {
|
||||
t.Helper()
|
||||
for _, bin := range []string{"nbdinfo", "nbdcopy"} {
|
||||
if _, err := exec.LookPath(bin); err != nil {
|
||||
// CI sets this so a runner image that loses libnbd fails rather
|
||||
// than dropping the only external witness to the export.
|
||||
if os.Getenv("SPINIFEX_REQUIRE_CONFORMANCE_TOOLS") != "" {
|
||||
t.Fatalf("%s not installed (libnbd-bin), but SPINIFEX_REQUIRE_CONFORMANCE_TOOLS is set", bin)
|
||||
}
|
||||
t.Skipf("%s not installed (libnbd-bin)", bin)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// nbdInfo runs nbdinfo against uri and returns its single export.
|
||||
func nbdInfo(t *testing.T, uri string) nbdExport {
|
||||
t.Helper()
|
||||
out, err := exec.CommandContext(context.Background(), "nbdinfo", "--json", uri).CombinedOutput()
|
||||
require.NoErrorf(t, err, "nbdinfo %s: %s", uri, out)
|
||||
t.Logf("nbdinfo %s:\n%s", uri, out)
|
||||
|
||||
var parsed nbdInfoOutput
|
||||
require.NoError(t, json.Unmarshal(out, &parsed))
|
||||
require.Lenf(t, parsed.Exports, 1, "expected exactly one export from %s", uri)
|
||||
return parsed.Exports[0]
|
||||
}
|
||||
|
||||
// nbdMap runs nbdinfo --map against uri and returns the allocation map the
|
||||
// export reports.
|
||||
func nbdMap(t *testing.T, uri string) []nbdExtent {
|
||||
t.Helper()
|
||||
out, err := exec.CommandContext(context.Background(), "nbdinfo", "--map", "--json", uri).CombinedOutput()
|
||||
require.NoErrorf(t, err, "nbdinfo --map %s: %s", uri, out)
|
||||
t.Logf("nbdinfo --map %s:\n%s", uri, out)
|
||||
|
||||
var extents []nbdExtent
|
||||
require.NoError(t, json.Unmarshal(out, &extents))
|
||||
return extents
|
||||
}
|
||||
|
||||
// nbdInfoRetry retries nbdinfo until the export accepts it, reporting how
|
||||
// many attempts and how long it took. Each attempt is a real connection, so
|
||||
// polling with a cheaper probe would only re-trigger whatever refused it.
|
||||
func nbdInfoRetry(t *testing.T, uri string) (nbdExport, int, time.Duration) {
|
||||
t.Helper()
|
||||
start := time.Now()
|
||||
deadline := start.Add(reconnectTimeout)
|
||||
|
||||
var lastOut []byte
|
||||
for attempt := 1; ; attempt++ {
|
||||
out, err := exec.CommandContext(context.Background(), "nbdinfo", "--json", uri).CombinedOutput()
|
||||
if err == nil {
|
||||
var parsed nbdInfoOutput
|
||||
require.NoError(t, json.Unmarshal(out, &parsed))
|
||||
require.Lenf(t, parsed.Exports, 1, "expected exactly one export from %s", uri)
|
||||
return parsed.Exports[0], attempt, time.Since(start)
|
||||
}
|
||||
lastOut = out
|
||||
if time.Now().After(deadline) {
|
||||
require.FailNowf(t, "export never accepted a reconnection",
|
||||
"after %s and %d attempts: %s", reconnectTimeout, attempt, lastOut)
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// nbdReachable reports whether nbdinfo can still connect, for asserting that
|
||||
// an export has gone away.
|
||||
func nbdReachable(uri string) bool {
|
||||
return exec.CommandContext(context.Background(), "nbdinfo", "--size", uri).Run() == nil
|
||||
}
|
||||
|
||||
// publishForNBD creates and publishes a volume, registering cleanup so a run
|
||||
// against a real cluster does not leave volumes behind.
|
||||
func publishForNBD(t *testing.T, provider ebsprovider.EBSProvider, cfg NBDClientConfig, volumeID string, readOnly bool) *ebsprovider.PublishedVolume {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: cfg.VolumeBytes},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { cleanupVolume(t, provider, cfg, volumeID) })
|
||||
|
||||
pub, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
NodeID: cfg.NodeID,
|
||||
ReadOnly: readOnly,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, pub)
|
||||
return pub
|
||||
}
|
||||
|
||||
// cleanupVolume unpublishes and deletes best-effort: cleanup runs after the
|
||||
// assertions and must never turn a passing test red.
|
||||
func cleanupVolume(t *testing.T, provider ebsprovider.EBSProvider, cfg NBDClientConfig, volumeID string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
if err := provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: cfg.NodeID,
|
||||
}); err != nil {
|
||||
t.Logf("cleanup: unpublish %s: %v", volumeID, err)
|
||||
}
|
||||
if err := provider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
}); err != nil {
|
||||
t.Logf("cleanup: delete %s: %v", volumeID, err)
|
||||
}
|
||||
}
|
||||
|
||||
// RunNBDClientSuite drives a provider's published export with libnbd's
|
||||
// client tools. newProvider must return a provider that serves real NBD;
|
||||
// the in-memory reference implementation does not qualify.
|
||||
func RunNBDClientSuite(t *testing.T, newProvider func(t *testing.T) ebsprovider.EBSProvider) {
|
||||
RunNBDClientSuiteWithConfig(t, newProvider, NBDClientConfig{})
|
||||
}
|
||||
|
||||
// RunNBDClientSuiteWithConfig is RunNBDClientSuite against a named node with
|
||||
// caller-chosen volume IDs, for driving a real daemon over a live cluster.
|
||||
func RunNBDClientSuiteWithConfig(t *testing.T, newProvider func(t *testing.T) ebsprovider.EBSProvider, cfg NBDClientConfig) {
|
||||
RequireNBDTools(t)
|
||||
cfg = cfg.withDefaults()
|
||||
|
||||
t.Run("published URI is a standard NBD URI", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
pub := publishForNBD(t, provider, cfg, cfg.VolumePrefix+"-uri", false)
|
||||
assertNBDURI(t, pub.NBDURI)
|
||||
})
|
||||
|
||||
t.Run("export is connectable by a standard NBD client", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
pub := publishForNBD(t, provider, cfg, cfg.VolumePrefix+"-connect", false)
|
||||
export := nbdInfo(t, dialURI(t, pub.NBDURI))
|
||||
assert.Equal(t, cfg.VolumeBytes, export.Size)
|
||||
})
|
||||
|
||||
t.Run("export advertises usable block size constraints", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
pub := publishForNBD(t, provider, cfg, cfg.VolumePrefix+"-blocksize", false)
|
||||
export := nbdInfo(t, dialURI(t, pub.NBDURI))
|
||||
|
||||
require.Positivef(t, export.BlockSizeMinimum,
|
||||
"export advertises no minimum block size, so a client cannot know how to align its requests")
|
||||
assert.GreaterOrEqual(t, export.BlockSizePreferred, export.BlockSizeMinimum)
|
||||
assert.Zero(t, cfg.VolumeBytes%export.BlockSizeMinimum,
|
||||
"export size must be a whole number of minimum blocks")
|
||||
})
|
||||
|
||||
// A guest that reboots, or any client that drops, reconnects to the same
|
||||
// export. Publication is what holds the volume, not any one connection.
|
||||
t.Run("export accepts a client reconnecting after the first disconnects", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
pub := publishForNBD(t, provider, cfg, cfg.VolumePrefix+"-reconnect", false)
|
||||
uri := dialURI(t, pub.NBDURI)
|
||||
|
||||
first := nbdInfo(t, uri)
|
||||
|
||||
second, attempts, elapsed := nbdInfoRetry(t, uri)
|
||||
t.Logf("reconnect accepted on attempt %d after %s", attempts, elapsed.Round(10*time.Millisecond))
|
||||
assert.Equalf(t, 1, attempts,
|
||||
"reconnect was refused for %s after the first client disconnected; a client that does not retry sees a hard failure", elapsed.Round(10*time.Millisecond))
|
||||
assert.Equal(t, first.Size, second.Size)
|
||||
})
|
||||
|
||||
t.Run("data written by an external client reads back byte for byte", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
pub := publishForNBD(t, provider, cfg, cfg.VolumePrefix+"-roundtrip", false)
|
||||
|
||||
dir := t.TempDir()
|
||||
in := filepath.Join(dir, "in.bin")
|
||||
out := filepath.Join(dir, "out.bin")
|
||||
want := patternBytes(int(cfg.VolumeBytes))
|
||||
require.NoError(t, os.WriteFile(in, want, 0o600))
|
||||
|
||||
uri := dialURI(t, pub.NBDURI)
|
||||
runNBDCopy(t, in, uri)
|
||||
runNBDCopy(t, uri, out)
|
||||
|
||||
got, err := os.ReadFile(out)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, len(want))
|
||||
assert.True(t, bytes.Equal(got, want), "data read back over NBD differs from what was written")
|
||||
})
|
||||
|
||||
// SparseExtentReporting is only observable at the export. base:allocation
|
||||
// being offered proves nothing on its own — nbdkit offers it whether or not
|
||||
// the plugin can compute extents, answering "all allocated" when it cannot.
|
||||
// What separates the two is the answer: an untouched volume is entirely
|
||||
// holes to a server that really tracks allocation.
|
||||
t.Run("sparse extent reporting matches the advertised capability", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
capabilities := capabilitiesOf(t, provider)
|
||||
pub := publishForNBD(t, provider, cfg, cfg.VolumePrefix+"-extents", false)
|
||||
|
||||
holes := false
|
||||
for _, extent := range nbdMap(t, dialURI(t, pub.NBDURI)) {
|
||||
if strings.Contains(extent.Description, "hole") {
|
||||
holes = true
|
||||
break
|
||||
}
|
||||
}
|
||||
assert.Equalf(t, capabilities.SparseExtentReporting, holes,
|
||||
"SparseExtentReporting=%v but a volume with nothing written to it reports holes=%v",
|
||||
capabilities.SparseExtentReporting, holes)
|
||||
})
|
||||
|
||||
t.Run("read-only publish matches the advertised capability", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
capabilities := capabilitiesOf(t, provider)
|
||||
|
||||
if !capabilities.ReadOnlyPublish {
|
||||
// Refusing is the contract for a provider that cannot honour it;
|
||||
// handing back a writable export would be the real failure.
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: cfg.VolumePrefix + "-readonly",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: cfg.VolumeBytes},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { cleanupVolume(t, provider, cfg, cfg.VolumePrefix+"-readonly") })
|
||||
|
||||
_, err = provider.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: cfg.VolumePrefix + "-readonly",
|
||||
NodeID: cfg.NodeID, ReadOnly: true,
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedCapability)
|
||||
return
|
||||
}
|
||||
|
||||
pub := publishForNBD(t, provider, cfg, cfg.VolumePrefix+"-readonly", true)
|
||||
export := nbdInfo(t, dialURI(t, pub.NBDURI))
|
||||
assert.True(t, export.ReadOnly, "a volume published ReadOnly must set the NBD read-only transmission flag")
|
||||
})
|
||||
|
||||
t.Run("unpublish tears the export down", func(t *testing.T) {
|
||||
provider := newProvider(t)
|
||||
volumeID := cfg.VolumePrefix + "-unpublish"
|
||||
pub := publishForNBD(t, provider, cfg, volumeID, false)
|
||||
uri := dialURI(t, pub.NBDURI)
|
||||
require.True(t, nbdReachable(uri), "export should be reachable before unpublish")
|
||||
|
||||
require.NoError(t, provider.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
NodeID: cfg.NodeID,
|
||||
}))
|
||||
|
||||
assert.Eventually(t, func() bool { return !nbdReachable(uri) }, 30*time.Second, 250*time.Millisecond,
|
||||
"export still answers after UnpublishVolume")
|
||||
})
|
||||
}
|
||||
|
||||
// runNBDCopy copies between a local file and an NBD URI in either direction.
|
||||
func runNBDCopy(t *testing.T, src, dst string) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "nbdcopy", src, dst).CombinedOutput()
|
||||
require.NoErrorf(t, err, "nbdcopy %s %s: %s", src, dst, out)
|
||||
}
|
||||
|
||||
// patternBytes builds a position-dependent pattern, so a copy that lands at
|
||||
// the wrong offset fails as loudly as one that loses data.
|
||||
func patternBytes(n int) []byte {
|
||||
buf := make([]byte, n)
|
||||
for i := range buf {
|
||||
buf[i] = byte(i*31 + i/4096)
|
||||
}
|
||||
return buf
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package conformance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
)
|
||||
|
||||
// trackingProvider delegates every call to the provider under test and records
|
||||
// what it created, so the suite can remove it afterwards. An in-process
|
||||
// provider is discarded with the test, but a live one keeps its volumes in S3
|
||||
// and would fail the next run on already_exists.
|
||||
type trackingProvider struct {
|
||||
ebsprovider.EBSProvider
|
||||
|
||||
t *testing.T
|
||||
|
||||
volumes map[string]string
|
||||
snapshots map[string]string
|
||||
published map[string]string
|
||||
}
|
||||
|
||||
var _ ebsprovider.EBSProvider = (*trackingProvider)(nil)
|
||||
|
||||
func newTrackingProvider(t *testing.T, inner ebsprovider.EBSProvider) *trackingProvider {
|
||||
t.Helper()
|
||||
p := &trackingProvider{
|
||||
EBSProvider: inner,
|
||||
t: t,
|
||||
volumes: make(map[string]string),
|
||||
snapshots: make(map[string]string),
|
||||
published: make(map[string]string),
|
||||
}
|
||||
t.Cleanup(p.cleanup)
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *trackingProvider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
vol, err := p.EBSProvider.CreateVolume(ctx, req)
|
||||
if err == nil && vol != nil {
|
||||
p.volumes[vol.ID] = vol.Handle
|
||||
}
|
||||
return vol, err
|
||||
}
|
||||
|
||||
func (p *trackingProvider) DeleteVolume(ctx context.Context, req ebsprovider.DeleteVolumeRequest) error {
|
||||
err := p.EBSProvider.DeleteVolume(ctx, req)
|
||||
if err == nil {
|
||||
delete(p.volumes, req.VolumeID)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (p *trackingProvider) CreateSnapshot(ctx context.Context, req ebsprovider.CreateSnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
snap, err := p.EBSProvider.CreateSnapshot(ctx, req)
|
||||
if err == nil && snap != nil {
|
||||
p.snapshots[snap.ID] = snap.Handle
|
||||
}
|
||||
return snap, err
|
||||
}
|
||||
|
||||
func (p *trackingProvider) CopySnapshot(ctx context.Context, req ebsprovider.CopySnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
snap, err := p.EBSProvider.CopySnapshot(ctx, req)
|
||||
if err == nil && snap != nil {
|
||||
p.snapshots[snap.ID] = snap.Handle
|
||||
}
|
||||
return snap, err
|
||||
}
|
||||
|
||||
func (p *trackingProvider) DeleteSnapshot(ctx context.Context, req ebsprovider.DeleteSnapshotRequest) error {
|
||||
err := p.EBSProvider.DeleteSnapshot(ctx, req)
|
||||
if err == nil {
|
||||
delete(p.snapshots, req.SnapshotID)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (p *trackingProvider) PublishVolume(ctx context.Context, req ebsprovider.PublishVolumeRequest) (*ebsprovider.PublishedVolume, error) {
|
||||
pub, err := p.EBSProvider.PublishVolume(ctx, req)
|
||||
if err == nil {
|
||||
p.published[req.VolumeID] = req.NodeID
|
||||
}
|
||||
return pub, err
|
||||
}
|
||||
|
||||
func (p *trackingProvider) UnpublishVolume(ctx context.Context, req ebsprovider.UnpublishVolumeRequest) error {
|
||||
err := p.EBSProvider.UnpublishVolume(ctx, req)
|
||||
if err == nil {
|
||||
delete(p.published, req.VolumeID)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// cleanup removes what the subtest left behind, in dependency order: a
|
||||
// published volume cannot be deleted, and a provider may hold a volume for its
|
||||
// snapshots. Failures are logged rather than failed on, so a teardown problem
|
||||
// never masks the assertion the subtest was actually making.
|
||||
func (p *trackingProvider) cleanup() {
|
||||
ctx := context.Background()
|
||||
|
||||
for volumeID, nodeID := range p.published {
|
||||
if err := p.EBSProvider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, Handle: p.volumes[volumeID], NodeID: nodeID,
|
||||
}); err != nil {
|
||||
p.t.Logf("conformance cleanup: unpublish %s: %v", volumeID, err)
|
||||
}
|
||||
}
|
||||
|
||||
for snapshotID, handle := range p.snapshots {
|
||||
if err := p.EBSProvider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, Handle: handle,
|
||||
}); err != nil {
|
||||
p.t.Logf("conformance cleanup: delete snapshot %s: %v", snapshotID, err)
|
||||
}
|
||||
}
|
||||
|
||||
for volumeID, handle := range p.volumes {
|
||||
if err := p.EBSProvider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, Handle: handle,
|
||||
}); err != nil {
|
||||
p.t.Logf("conformance cleanup: delete volume %s: %v", volumeID, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrAlreadyExists = errors.New("resource already exists with different parameters")
|
||||
ErrInvalidArgument = errors.New("invalid argument")
|
||||
ErrNotFound = errors.New("resource not found")
|
||||
ErrUnsupportedVersion = errors.New("unsupported EBS provider schema version")
|
||||
ErrVolumeInUse = errors.New("volume is in use")
|
||||
|
||||
// ErrUnsupportedCapability is returned when a request asks for optional
|
||||
// behaviour this provider does not advertise in Capabilities. It is
|
||||
// distinct from ErrInvalidArgument: the request is well formed.
|
||||
ErrUnsupportedCapability = errors.New("provider does not support the requested capability")
|
||||
|
||||
// ErrUnavailable is returned when the provider could not answer now and
|
||||
// the same request may succeed later: a peer it had to consult was
|
||||
// unreachable, or state it had to establish could not be established.
|
||||
//
|
||||
// It is the only code that says anything about retrying. Every other one
|
||||
// classifies what went wrong, which leaves a caller unable to tell a
|
||||
// permanent refusal from a transient one; returning ErrInternal for a
|
||||
// transient condition tells a caller to give up when it should retry.
|
||||
ErrUnavailable = errors.New("provider is temporarily unable to answer")
|
||||
)
|
||||
|
||||
type ErrorCode string
|
||||
|
||||
const (
|
||||
ErrorCodeAlreadyExists ErrorCode = "already_exists"
|
||||
ErrorCodeInvalidArgument ErrorCode = "invalid_argument"
|
||||
ErrorCodeNotFound ErrorCode = "not_found"
|
||||
ErrorCodeUnsupportedVersion ErrorCode = "unsupported_version"
|
||||
ErrorCodeVolumeInUse ErrorCode = "volume_in_use"
|
||||
ErrorCodeUnsupportedCap ErrorCode = "unsupported_capability"
|
||||
ErrorCodeUnavailable ErrorCode = "unavailable"
|
||||
ErrorCodeInternal ErrorCode = "internal"
|
||||
)
|
||||
|
||||
// Retryable reports whether repeating the identical request could succeed.
|
||||
// Callers branch on this rather than on the code, so adding a transient code
|
||||
// later does not need every call site to be found again.
|
||||
func (c ErrorCode) Retryable() bool {
|
||||
return c == ErrorCodeUnavailable
|
||||
}
|
||||
|
||||
// ProviderError is the stable error representation carried over NATS.
|
||||
type ProviderError struct {
|
||||
Code ErrorCode `json:"code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func (e *ProviderError) Error() string {
|
||||
if e == nil {
|
||||
return ""
|
||||
}
|
||||
return e.Message
|
||||
}
|
||||
|
||||
func (e *ProviderError) Unwrap() error {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
switch e.Code {
|
||||
case ErrorCodeAlreadyExists:
|
||||
return ErrAlreadyExists
|
||||
case ErrorCodeInvalidArgument:
|
||||
return ErrInvalidArgument
|
||||
case ErrorCodeNotFound:
|
||||
return ErrNotFound
|
||||
case ErrorCodeUnsupportedVersion:
|
||||
return ErrUnsupportedVersion
|
||||
case ErrorCodeVolumeInUse:
|
||||
return ErrVolumeInUse
|
||||
case ErrorCodeUnsupportedCap:
|
||||
return ErrUnsupportedCapability
|
||||
case ErrorCodeUnavailable:
|
||||
return ErrUnavailable
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func checkVersion(version uint16) error {
|
||||
if version != SchemaVersion {
|
||||
return fmt.Errorf("%w: got %d, want %d", ErrUnsupportedVersion, version, SchemaVersion)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// TestProviderErrorUnwrapTable covers every ErrorCode's mapping back to its
|
||||
// sentinel error, plus the two codes with no sentinel (internal, unknown)
|
||||
// and the nil-receiver methods a caller can hit via a typed-nil *ProviderError.
|
||||
func TestProviderErrorUnwrapTable(t *testing.T) {
|
||||
tests := []struct {
|
||||
code ErrorCode
|
||||
want error
|
||||
}{
|
||||
{ErrorCodeAlreadyExists, ErrAlreadyExists},
|
||||
{ErrorCodeInvalidArgument, ErrInvalidArgument},
|
||||
{ErrorCodeNotFound, ErrNotFound},
|
||||
{ErrorCodeUnsupportedVersion, ErrUnsupportedVersion},
|
||||
{ErrorCodeVolumeInUse, ErrVolumeInUse},
|
||||
{ErrorCodeUnsupportedCap, ErrUnsupportedCapability},
|
||||
{ErrorCodeUnavailable, ErrUnavailable},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(string(tt.code), func(t *testing.T) {
|
||||
err := &ProviderError{Code: tt.code, Message: "msg: " + string(tt.code)}
|
||||
assert.ErrorIs(t, err, tt.want)
|
||||
assert.Equal(t, "msg: "+string(tt.code), err.Error())
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("internal code has no sentinel", func(t *testing.T) {
|
||||
err := &ProviderError{Code: ErrorCodeInternal, Message: "boom"}
|
||||
assert.NoError(t, err.Unwrap())
|
||||
})
|
||||
|
||||
t.Run("unknown code has no sentinel", func(t *testing.T) {
|
||||
err := &ProviderError{Code: ErrorCode("bogus"), Message: "boom"}
|
||||
assert.NoError(t, err.Unwrap())
|
||||
})
|
||||
|
||||
t.Run("nil receiver", func(t *testing.T) {
|
||||
var err *ProviderError
|
||||
assert.NoError(t, err.Unwrap())
|
||||
assert.Empty(t, err.Error())
|
||||
})
|
||||
}
|
||||
|
||||
// TestErrorCodeRetryable pins which refusals a caller may repeat. Every code
|
||||
// here is named, so adding one without deciding its retry meaning fails rather
|
||||
// than defaulting to permanent and stranding whatever the caller was doing.
|
||||
func TestErrorCodeRetryable(t *testing.T) {
|
||||
retryable := []ErrorCode{ErrorCodeUnavailable}
|
||||
permanent := []ErrorCode{
|
||||
ErrorCodeAlreadyExists,
|
||||
ErrorCodeInvalidArgument,
|
||||
ErrorCodeNotFound,
|
||||
ErrorCodeUnsupportedVersion,
|
||||
ErrorCodeVolumeInUse,
|
||||
ErrorCodeUnsupportedCap,
|
||||
ErrorCodeInternal,
|
||||
}
|
||||
|
||||
for _, code := range retryable {
|
||||
assert.Truef(t, code.Retryable(), "%s must read as retryable, or a caller gives up on a request that would have succeeded", code)
|
||||
}
|
||||
for _, code := range permanent {
|
||||
assert.Falsef(t, code.Retryable(), "%s must read as permanent, or a caller retries a request that can never succeed", code)
|
||||
}
|
||||
assert.False(t, ErrorCode("bogus").Retryable(), "an unrecognised code must not invite a retry loop")
|
||||
}
|
||||
|
||||
func TestCheckVersion(t *testing.T) {
|
||||
assert.NoError(t, checkVersion(SchemaVersion))
|
||||
assert.ErrorIs(t, checkVersion(SchemaVersion+1), ErrUnsupportedVersion)
|
||||
assert.ErrorIs(t, checkVersion(0), ErrUnsupportedVersion)
|
||||
}
|
||||
@@ -0,0 +1,263 @@
|
||||
// Package hostile wraps any EBSProvider in one that misbehaves on purpose.
|
||||
// It exists because the control plane's handling of a provider that fails,
|
||||
// stalls, or answers wrongly is the part nothing tests, and the part that
|
||||
// bites in production.
|
||||
//
|
||||
// Faults are derived from a seed, not drawn from a global source, so a
|
||||
// failure found at hour six of a soak run replays exactly in a unit test. A
|
||||
// fault injector whose output cannot be reproduced only produces bug reports
|
||||
// nobody can act on.
|
||||
package hostile
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"fmt"
|
||||
"hash/fnv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
|
||||
// Fault is what was injected into one call.
|
||||
type Fault string
|
||||
|
||||
const (
|
||||
// FaultNone is the call passing through untouched.
|
||||
FaultNone Fault = "none"
|
||||
|
||||
// FaultError fails the call without reaching the inner provider. The
|
||||
// caller is told no, and nothing happened.
|
||||
FaultError Fault = "error"
|
||||
|
||||
// FaultErrorAfterWork fails the call after the inner provider has already
|
||||
// done it. This is the interesting half of the pair: the control plane is
|
||||
// told the volume was not created, and the blocks exist anyway. It is how
|
||||
// orphans are made, so it is the fault an orphan scan is the oracle for.
|
||||
FaultErrorAfterWork Fault = "error_after_work"
|
||||
|
||||
// FaultLatency delays the call. Configure MaxLatency past the caller's
|
||||
// request timeout to reach the case where the caller gives up and the
|
||||
// provider is still working.
|
||||
FaultLatency Fault = "latency"
|
||||
|
||||
// FaultUnavailable answers as though nothing is listening, which is the
|
||||
// condition owner-first routing treats as safe to retry elsewhere.
|
||||
FaultUnavailable Fault = "unavailable"
|
||||
|
||||
// FaultLie succeeds and returns a wrong answer: a capacity, state or
|
||||
// handle that does not match what the provider holds. Error paths are
|
||||
// written deliberately; a plausible wrong answer is what corrupts
|
||||
// control-plane state quietly.
|
||||
FaultLie Fault = "lie"
|
||||
)
|
||||
|
||||
// Config sets how often each fault fires. Rates are independent probabilities
|
||||
// applied in the order the fields are declared, and a call takes the first
|
||||
// fault that fires, so the sum need not be 1. All-zero is a pass-through
|
||||
// provider, which is the right default: a caller must ask to be hurt.
|
||||
type Config struct {
|
||||
// Seed fixes the fault sequence. Two runs with the same seed over the
|
||||
// same calls inject the same faults.
|
||||
Seed uint64
|
||||
|
||||
ErrorRate float64
|
||||
ErrorAfterWorkRate float64
|
||||
LatencyRate float64
|
||||
UnavailableRate float64
|
||||
LieRate float64
|
||||
|
||||
// MaxLatency bounds FaultLatency. Zero disables the delay while leaving
|
||||
// the fault recorded, which is useful for a fast test of the wiring.
|
||||
MaxLatency time.Duration
|
||||
}
|
||||
|
||||
// Injection records one fault, with everything needed to reproduce it: the
|
||||
// seed is in the provider, and Verb, Target and Sequence are the rest of the
|
||||
// key the fault was derived from.
|
||||
type Injection struct {
|
||||
Verb string
|
||||
Target string
|
||||
Sequence uint64
|
||||
Fault Fault
|
||||
Detail string
|
||||
}
|
||||
|
||||
func (i Injection) String() string {
|
||||
return fmt.Sprintf("%s %s #%d: %s (%s)", i.Verb, i.Target, i.Sequence, i.Fault, i.Detail)
|
||||
}
|
||||
|
||||
// injectedErrors is the closed set FaultError and FaultErrorAfterWork draw
|
||||
// from. Every sentinel the contract defines appears, so a soak exercises each
|
||||
// branch the control plane has for them.
|
||||
var injectedErrors = []error{
|
||||
ebsprovider.ErrAlreadyExists,
|
||||
ebsprovider.ErrInvalidArgument,
|
||||
ebsprovider.ErrNotFound,
|
||||
ebsprovider.ErrVolumeInUse,
|
||||
ebsprovider.ErrUnsupportedCapability,
|
||||
}
|
||||
|
||||
// Provider decorates an EBSProvider with faults.
|
||||
type Provider struct {
|
||||
inner ebsprovider.EBSProvider
|
||||
config Config
|
||||
|
||||
mu sync.Mutex
|
||||
sequences map[string]uint64
|
||||
injections []Injection
|
||||
}
|
||||
|
||||
var _ ebsprovider.EBSProvider = (*Provider)(nil)
|
||||
|
||||
// New wraps inner. A zero Config injects nothing.
|
||||
func New(inner ebsprovider.EBSProvider, config Config) *Provider {
|
||||
return &Provider{inner: inner, config: config, sequences: make(map[string]uint64)}
|
||||
}
|
||||
|
||||
// Injections returns what has been injected so far, in order.
|
||||
func (p *Provider) Injections() []Injection {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
return append([]Injection(nil), p.injections...)
|
||||
}
|
||||
|
||||
// draw decides this call's fault. The key is (verb, target, per-key sequence)
|
||||
// rather than a global counter, so concurrent calls on different volumes
|
||||
// cannot reorder each other's faults; only repeated calls on the same target
|
||||
// advance, and those are ordered by definition.
|
||||
func (p *Provider) draw(verb, target string) Injection {
|
||||
return p.drawFault(verb, target, true)
|
||||
}
|
||||
|
||||
// drawTruthful is draw for a verb that must never return a wrong answer. A
|
||||
// drawn lie degrades to no fault, and the log says so, so the injection
|
||||
// record never claims a lie that was not told.
|
||||
func (p *Provider) drawTruthful(verb, target string) Injection {
|
||||
return p.drawFault(verb, target, false)
|
||||
}
|
||||
|
||||
func (p *Provider) drawFault(verb, target string, allowLie bool) Injection {
|
||||
p.mu.Lock()
|
||||
key := verb + "\x00" + target
|
||||
sequence := p.sequences[key]
|
||||
p.sequences[key] = sequence + 1
|
||||
p.mu.Unlock()
|
||||
|
||||
injection := Injection{Verb: verb, Target: target, Sequence: sequence, Fault: FaultNone}
|
||||
value, selector := p.hash(key, sequence)
|
||||
|
||||
switch {
|
||||
case value < p.config.ErrorRate:
|
||||
injection.Fault = FaultError
|
||||
case value < p.config.ErrorRate+p.config.ErrorAfterWorkRate:
|
||||
injection.Fault = FaultErrorAfterWork
|
||||
case value < p.config.ErrorRate+p.config.ErrorAfterWorkRate+p.config.LatencyRate:
|
||||
injection.Fault = FaultLatency
|
||||
case value < p.config.ErrorRate+p.config.ErrorAfterWorkRate+p.config.LatencyRate+p.config.UnavailableRate:
|
||||
injection.Fault = FaultUnavailable
|
||||
case value < p.config.ErrorRate+p.config.ErrorAfterWorkRate+p.config.LatencyRate+p.config.UnavailableRate+p.config.LieRate:
|
||||
if !allowLie {
|
||||
return injection
|
||||
}
|
||||
injection.Fault = FaultLie
|
||||
default:
|
||||
return injection
|
||||
}
|
||||
|
||||
injection.Detail = p.detail(injection.Fault, selector)
|
||||
p.mu.Lock()
|
||||
p.injections = append(p.injections, injection)
|
||||
p.mu.Unlock()
|
||||
return injection
|
||||
}
|
||||
|
||||
// mix is the SplitMix64 finalizer. FNV alone is not good enough here: its
|
||||
// output over keys that share long prefixes clusters badly enough to leave
|
||||
// whole fault bands undrawable, which silently disables a fault class.
|
||||
func mix(x uint64) uint64 {
|
||||
x += 0x9e3779b97f4a7c15
|
||||
x = (x ^ (x >> 30)) * 0xbf58476d1ce4e5b9
|
||||
x = (x ^ (x >> 27)) * 0x94d049bb133111eb
|
||||
return x ^ (x >> 31)
|
||||
}
|
||||
|
||||
// hash turns the seed and key into a uniform draw plus an independent
|
||||
// selector, so which fault fires and which flavour it takes do not correlate.
|
||||
func (p *Provider) hash(key string, sequence uint64) (value float64, selector uint64) {
|
||||
digest := fnv.New64a()
|
||||
_, _ = digest.Write([]byte(key))
|
||||
sum := mix(digest.Sum64() ^ mix(p.config.Seed+sequence))
|
||||
return float64(sum>>11) / float64(1<<53), mix(sum)
|
||||
}
|
||||
|
||||
func (p *Provider) detail(fault Fault, selector uint64) string {
|
||||
switch fault {
|
||||
case FaultError, FaultErrorAfterWork:
|
||||
return injectedErrors[selector%uint64(len(injectedErrors))].Error()
|
||||
case FaultLatency:
|
||||
return p.latency(selector).String()
|
||||
case FaultUnavailable:
|
||||
return nats.ErrNoResponders.Error()
|
||||
case FaultLie:
|
||||
return lieFlavours[selector%uint64(len(lieFlavours))]
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Provider) latency(selector uint64) time.Duration {
|
||||
if p.config.MaxLatency <= 0 {
|
||||
return 0
|
||||
}
|
||||
// Scaling a fraction of MaxLatency keeps the bound obvious and avoids
|
||||
// reducing an unsigned selector into a signed Duration.
|
||||
fraction := float64(selector>>11) / float64(1<<53)
|
||||
return time.Duration(fraction * float64(p.config.MaxLatency))
|
||||
}
|
||||
|
||||
func (p *Provider) injectedError(selector uint64) error {
|
||||
return fmt.Errorf("hostile: %w", injectedErrors[selector%uint64(len(injectedErrors))])
|
||||
}
|
||||
|
||||
// apply runs the part of a fault that happens before the inner provider. It
|
||||
// reports whether the caller should stop, and with what.
|
||||
func (p *Provider) apply(ctx context.Context, injection Injection) (stop bool, err error) {
|
||||
_, selector := p.hash(injection.Verb+"\x00"+injection.Target, injection.Sequence)
|
||||
switch injection.Fault {
|
||||
case FaultError:
|
||||
return true, p.injectedError(selector)
|
||||
case FaultUnavailable:
|
||||
return true, fmt.Errorf("hostile: %w", nats.ErrNoResponders)
|
||||
case FaultLatency:
|
||||
return false, sleep(ctx, p.latency(selector))
|
||||
default:
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
// after runs the part of a fault that happens once the inner provider has
|
||||
// already done the work.
|
||||
func (p *Provider) after(injection Injection, err error) error {
|
||||
if err != nil || injection.Fault != FaultErrorAfterWork {
|
||||
return err
|
||||
}
|
||||
_, selector := p.hash(injection.Verb+"\x00"+injection.Target, injection.Sequence)
|
||||
return p.injectedError(selector)
|
||||
}
|
||||
|
||||
func sleep(ctx context.Context, d time.Duration) error {
|
||||
if d <= 0 {
|
||||
return nil
|
||||
}
|
||||
timer := time.NewTimer(d)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case <-timer.C:
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
package hostile_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/conformance"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/hostile"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/nullprovider"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const volumeBytes = 1 << 30
|
||||
|
||||
// countingProvider records how many times the inner provider was actually
|
||||
// reached, which is the only way to tell FaultError (nothing happened) from
|
||||
// FaultErrorAfterWork (it happened and was reported as failed).
|
||||
type countingProvider struct {
|
||||
ebsprovider.EBSProvider
|
||||
|
||||
creates int
|
||||
deletes int
|
||||
}
|
||||
|
||||
var _ ebsprovider.EBSProvider = (*countingProvider)(nil)
|
||||
|
||||
func (c *countingProvider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
c.creates++
|
||||
return c.EBSProvider.CreateVolume(ctx, req)
|
||||
}
|
||||
|
||||
func (c *countingProvider) DeleteVolume(ctx context.Context, req ebsprovider.DeleteVolumeRequest) error {
|
||||
c.deletes++
|
||||
return c.EBSProvider.DeleteVolume(ctx, req)
|
||||
}
|
||||
|
||||
func createVolume(ctx context.Context, provider ebsprovider.EBSProvider, volumeID string) (*ebsprovider.Volume, error) {
|
||||
return provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: volumeBytes},
|
||||
})
|
||||
}
|
||||
|
||||
// drive runs a fixed sequence of calls, so two runs differ only in the seed.
|
||||
func drive(t *testing.T, provider ebsprovider.EBSProvider, count int) {
|
||||
t.Helper()
|
||||
for i := range count {
|
||||
volumeID := fmt.Sprintf("vol-drive%011d", i)
|
||||
_, _ = createVolume(t.Context(), provider, volumeID)
|
||||
_, _ = provider.GetVolume(t.Context(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
})
|
||||
_ = provider.DeleteVolume(t.Context(), ebsprovider.DeleteVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func hostileConfig(seed uint64) hostile.Config {
|
||||
return hostile.Config{
|
||||
Seed: seed,
|
||||
ErrorRate: 0.15,
|
||||
ErrorAfterWorkRate: 0.15,
|
||||
LatencyRate: 0.10,
|
||||
UnavailableRate: 0.10,
|
||||
LieRate: 0.15,
|
||||
}
|
||||
}
|
||||
|
||||
// TestZeroConfigInjectsNothing is the default that matters: wrapping a
|
||||
// provider must not hurt it until a caller asks.
|
||||
func TestZeroConfigInjectsNothing(t *testing.T) {
|
||||
provider := hostile.New(nullprovider.New(), hostile.Config{})
|
||||
drive(t, provider, 50)
|
||||
assert.Empty(t, provider.Injections())
|
||||
|
||||
volume, err := createVolume(t.Context(), provider, "vol-zeroconfig00001")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(volumeBytes), volume.CapacityBytes)
|
||||
}
|
||||
|
||||
// TestPassThroughIsConformant holds the decorator to the same contract it
|
||||
// decorates. Every difference a run shows must come from a fault that was
|
||||
// asked for, never from the wrapping.
|
||||
func TestPassThroughIsConformant(t *testing.T) {
|
||||
conformance.RunSuite(t, func(t *testing.T) ebsprovider.EBSProvider {
|
||||
return hostile.New(ebsprovider.NewMemoryProvider(conformance.ReferenceCapabilities), hostile.Config{})
|
||||
})
|
||||
}
|
||||
|
||||
// TestSameSeedReplaysTheSameFaults is the whole reason this is seeded. A soak
|
||||
// failure at hour six has to be reproducible in a test, and it can only be if
|
||||
// the fault sequence is a function of the seed and the calls.
|
||||
func TestSameSeedReplaysTheSameFaults(t *testing.T) {
|
||||
first := hostile.New(nullprovider.New(), hostileConfig(42))
|
||||
second := hostile.New(nullprovider.New(), hostileConfig(42))
|
||||
drive(t, first, 40)
|
||||
drive(t, second, 40)
|
||||
|
||||
require.NotEmpty(t, first.Injections(), "the run injected nothing, so it proves nothing")
|
||||
assert.Equal(t, first.Injections(), second.Injections())
|
||||
}
|
||||
|
||||
func TestDifferentSeedsDiverge(t *testing.T) {
|
||||
first := hostile.New(nullprovider.New(), hostileConfig(42))
|
||||
second := hostile.New(nullprovider.New(), hostileConfig(43))
|
||||
drive(t, first, 40)
|
||||
drive(t, second, 40)
|
||||
|
||||
assert.NotEqual(t, first.Injections(), second.Injections())
|
||||
}
|
||||
|
||||
// TestEveryFaultClassIsReachable guards against a threshold arithmetic slip
|
||||
// that would silently disable a class: a fault nobody can draw is a fault
|
||||
// nobody tests against.
|
||||
func TestEveryFaultClassIsReachable(t *testing.T) {
|
||||
provider := hostile.New(nullprovider.New(), hostileConfig(7))
|
||||
drive(t, provider, 400)
|
||||
|
||||
seen := map[hostile.Fault]bool{}
|
||||
for _, injection := range provider.Injections() {
|
||||
seen[injection.Fault] = true
|
||||
assert.NotEmpty(t, injection.Detail, "%s has no detail to reproduce it from", injection)
|
||||
}
|
||||
for _, fault := range []hostile.Fault{
|
||||
hostile.FaultError,
|
||||
hostile.FaultErrorAfterWork,
|
||||
hostile.FaultLatency,
|
||||
hostile.FaultUnavailable,
|
||||
hostile.FaultLie,
|
||||
} {
|
||||
assert.True(t, seen[fault], "%s was never drawn", fault)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaultErrorNeverReachesTheProvider and its ErrorAfterWork counterpart pin
|
||||
// the distinction the two faults exist for.
|
||||
func TestFaultErrorNeverReachesTheProvider(t *testing.T) {
|
||||
counting := &countingProvider{EBSProvider: nullprovider.New()}
|
||||
provider := hostile.New(counting, hostile.Config{Seed: 1, ErrorRate: 1})
|
||||
|
||||
_, err := createVolume(t.Context(), provider, "vol-neverreaches001")
|
||||
require.Error(t, err)
|
||||
assert.Zero(t, counting.creates, "the provider was reached for a call that was supposed to fail first")
|
||||
}
|
||||
|
||||
// TestFaultErrorAfterWorkLeavesTheVolumeBehind is the orphan-making case, and
|
||||
// the reason an enumeration-based scan is the right oracle for a soak: the
|
||||
// caller was told the create failed, and the volume exists.
|
||||
func TestFaultErrorAfterWorkLeavesTheVolumeBehind(t *testing.T) {
|
||||
inner := nullprovider.New()
|
||||
counting := &countingProvider{EBSProvider: inner}
|
||||
provider := hostile.New(counting, hostile.Config{Seed: 1, ErrorAfterWorkRate: 1})
|
||||
|
||||
const volumeID = "vol-leftbehind00001"
|
||||
_, err := createVolume(t.Context(), provider, volumeID)
|
||||
require.Error(t, err, "the caller must be told this failed")
|
||||
assert.Equal(t, 1, counting.creates, "the work must actually have happened")
|
||||
|
||||
// The oracle reads the provider directly. Asking through the injector
|
||||
// would be asking the thing under test whether it lied.
|
||||
resp, err := inner.ListVolumes(t.Context(), ebsprovider.ListVolumesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
var ids []string
|
||||
for _, ref := range resp.Volumes {
|
||||
ids = append(ids, ref.ID)
|
||||
}
|
||||
assert.Contains(t, ids, volumeID, "the provider holds a volume the caller believes was never created")
|
||||
}
|
||||
|
||||
// TestUnavailableIsRetryable matters because owner-first routing treats
|
||||
// no-responders as safe to retry elsewhere and a timeout as not.
|
||||
func TestUnavailableIsRetryable(t *testing.T) {
|
||||
provider := hostile.New(nullprovider.New(), hostile.Config{Seed: 1, UnavailableRate: 1})
|
||||
_, err := createVolume(t.Context(), provider, "vol-unavailable0001")
|
||||
require.ErrorIs(t, err, nats.ErrNoResponders)
|
||||
}
|
||||
|
||||
// aliasingProvider hands back a pointer into its own state, the way a
|
||||
// provider that forgot to copy would. MemoryProvider returns copies, so a
|
||||
// lie told through it cannot corrupt anything and proves nothing about
|
||||
// whether the injector copies.
|
||||
type aliasingProvider struct {
|
||||
ebsprovider.EBSProvider
|
||||
|
||||
volume *ebsprovider.Volume
|
||||
}
|
||||
|
||||
var _ ebsprovider.EBSProvider = (*aliasingProvider)(nil)
|
||||
|
||||
func (a *aliasingProvider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
volume, err := a.EBSProvider.CreateVolume(ctx, req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a.volume = volume
|
||||
return a.volume, nil
|
||||
}
|
||||
|
||||
// TestLiesDoNotCorruptTheProvider keeps the fault a fault. If a wrong answer
|
||||
// also changed what the provider held, the oracle would be measuring the
|
||||
// injector's bugs instead of the control plane's.
|
||||
func TestLiesDoNotCorruptTheProvider(t *testing.T) {
|
||||
aliasing := &aliasingProvider{EBSProvider: nullprovider.New()}
|
||||
provider := hostile.New(aliasing, hostile.Config{Seed: 3, LieRate: 1})
|
||||
|
||||
const volumeID = "vol-lieaboutit00001"
|
||||
lied, err := createVolume(t.Context(), provider, volumeID)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NotNil(t, aliasing.volume)
|
||||
assert.Equal(t, int64(volumeBytes), aliasing.volume.CapacityBytes, "the lie was written into the provider's own object")
|
||||
assert.NotEqual(t, *aliasing.volume, *lied, "a lie that matches the truth is not a lie")
|
||||
}
|
||||
|
||||
// TestListVolumesNeverLies protects the oracle. A soak checks its invariants
|
||||
// with enumeration, so an enumeration that can be made to agree with a broken
|
||||
// control plane cannot falsify anything.
|
||||
func TestListVolumesNeverLies(t *testing.T) {
|
||||
provider := hostile.New(nullprovider.New(), hostile.Config{Seed: 5, LieRate: 1})
|
||||
drive(t, provider, 30)
|
||||
for range 30 {
|
||||
_, err := provider.ListVolumes(t.Context(), ebsprovider.ListVolumesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
for _, injection := range provider.Injections() {
|
||||
assert.NotEqual(t, "volume.list", injection.Verb, "enumeration was faulted: %s", injection)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLatencyRespectsCancellation stops a long delay from outliving the caller
|
||||
// that gave up on it, which is what would make a soak run leak goroutines.
|
||||
func TestLatencyRespectsCancellation(t *testing.T) {
|
||||
provider := hostile.New(nullprovider.New(), hostile.Config{
|
||||
Seed: 1, LatencyRate: 1, MaxLatency: time.Hour,
|
||||
})
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
started := time.Now()
|
||||
_, err := createVolume(ctx, provider, "vol-cancelme0000001")
|
||||
require.ErrorIs(t, err, context.DeadlineExceeded)
|
||||
assert.Less(t, time.Since(started), 5*time.Second)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package hostile_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/hostile"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/nullprovider"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// soakDuration keeps the default short enough for CI while letting a real
|
||||
// soak run for hours from the same code. A fault injector only earns its
|
||||
// keep over a long run, but a long run must not be the only way to use it.
|
||||
func soakDuration(t *testing.T) time.Duration {
|
||||
t.Helper()
|
||||
raw := os.Getenv("EBS_SOAK_DURATION")
|
||||
if raw == "" {
|
||||
return 2 * time.Second
|
||||
}
|
||||
parsed, err := time.ParseDuration(raw)
|
||||
require.NoError(t, err, "EBS_SOAK_DURATION")
|
||||
return parsed
|
||||
}
|
||||
|
||||
// belief is what the caller was told about a volume, which is all a control
|
||||
// plane ever has. The provider's own answer is the truth it is checked
|
||||
// against.
|
||||
type belief struct {
|
||||
createdOK bool
|
||||
deletedOK bool
|
||||
}
|
||||
|
||||
// soakResult is what one run produced, so two runs can be compared.
|
||||
type soakResult struct {
|
||||
beliefs map[string]belief
|
||||
injections []hostile.Injection
|
||||
held []string
|
||||
}
|
||||
|
||||
func runSoak(t *testing.T, seed uint64, workers int, duration time.Duration) soakResult {
|
||||
t.Helper()
|
||||
inner := nullprovider.New()
|
||||
provider := hostile.New(inner, hostile.Config{
|
||||
Seed: seed,
|
||||
ErrorRate: 0.10,
|
||||
ErrorAfterWorkRate: 0.10,
|
||||
LatencyRate: 0.05,
|
||||
UnavailableRate: 0.05,
|
||||
LieRate: 0.10,
|
||||
MaxLatency: 2 * time.Millisecond,
|
||||
})
|
||||
|
||||
var mu sync.Mutex
|
||||
beliefs := map[string]belief{}
|
||||
deadline := time.Now().Add(duration)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for worker := range workers {
|
||||
wg.Go(func() {
|
||||
for i := 0; time.Now().Before(deadline); i++ {
|
||||
// Each worker owns its own ID space, so faults drawn per
|
||||
// volume never depend on how the workers interleave.
|
||||
volumeID := fmt.Sprintf("vol-soak%04d%07d", worker, i)
|
||||
state := soakOneVolume(t.Context(), provider, volumeID)
|
||||
mu.Lock()
|
||||
beliefs[volumeID] = state
|
||||
mu.Unlock()
|
||||
}
|
||||
})
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
return soakResult{
|
||||
beliefs: beliefs,
|
||||
injections: provider.Injections(),
|
||||
held: drainHeldVolumes(t, inner),
|
||||
}
|
||||
}
|
||||
|
||||
// soakOneVolume drives one volume through its whole life without retrying.
|
||||
// Retries would hide leaks, and what is being looked for here is what the
|
||||
// control plane loses track of when a provider misbehaves once.
|
||||
func soakOneVolume(ctx context.Context, provider ebsprovider.EBSProvider, volumeID string) belief {
|
||||
var state belief
|
||||
if _, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: volumeBytes},
|
||||
}); err != nil {
|
||||
return state
|
||||
}
|
||||
state.createdOK = true
|
||||
|
||||
_, _ = provider.GetVolume(ctx, ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
})
|
||||
if _, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: "node-1",
|
||||
}); err == nil {
|
||||
_ = provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: "node-1",
|
||||
})
|
||||
}
|
||||
|
||||
if err := provider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
}); err == nil {
|
||||
state.deletedOK = true
|
||||
}
|
||||
return state
|
||||
}
|
||||
|
||||
func drainHeldVolumes(t *testing.T, provider ebsprovider.EBSProvider) []string {
|
||||
t.Helper()
|
||||
var ids []string
|
||||
var token string
|
||||
for range 1000 {
|
||||
resp, err := provider.ListVolumes(context.Background(), ebsprovider.ListVolumesRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), StartingToken: token,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
for _, ref := range resp.Volumes {
|
||||
ids = append(ids, ref.ID)
|
||||
}
|
||||
if resp.NextToken == "" {
|
||||
sort.Strings(ids)
|
||||
return ids
|
||||
}
|
||||
token = resp.NextToken
|
||||
}
|
||||
t.Fatal("ListVolumes did not terminate")
|
||||
return nil
|
||||
}
|
||||
|
||||
// explainedBy reports whether an injection recorded for volumeID accounts for
|
||||
// the provider still holding it: work that was done and then reported failed.
|
||||
func explainedBy(injections []hostile.Injection, volumeID string) bool {
|
||||
for _, injection := range injections {
|
||||
if injection.Target != volumeID {
|
||||
continue
|
||||
}
|
||||
switch injection.Fault {
|
||||
case hostile.FaultErrorAfterWork, hostile.FaultLatency, hostile.FaultLie:
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// TestSoak_EveryLeakIsAttributable is the oracle. Injecting faults proves
|
||||
// nothing on its own; what proves something is that every volume the provider
|
||||
// still holds can be traced to a fault that was injected. A leftover volume
|
||||
// with no injection behind it is the control plane losing track of storage on
|
||||
// its own, which is the bug worth finding.
|
||||
func TestSoak_EveryLeakIsAttributable(t *testing.T) {
|
||||
result := runSoak(t, 20260811, 8, soakDuration(t))
|
||||
require.NotEmpty(t, result.injections, "the run injected nothing, so it proves nothing")
|
||||
require.NotEmpty(t, result.beliefs)
|
||||
|
||||
held := map[string]bool{}
|
||||
for _, id := range result.held {
|
||||
held[id] = true
|
||||
}
|
||||
|
||||
var unexplained []string
|
||||
for volumeID, state := range result.beliefs {
|
||||
switch {
|
||||
case state.deletedOK && held[volumeID]:
|
||||
unexplained = append(unexplained, volumeID+" (delete reported success, still held)")
|
||||
case !state.createdOK && held[volumeID] && !explainedBy(result.injections, volumeID):
|
||||
unexplained = append(unexplained, volumeID+" (create reported failure, held anyway)")
|
||||
case state.createdOK && !state.deletedOK && !held[volumeID] && !explainedBy(result.injections, volumeID):
|
||||
unexplained = append(unexplained, volumeID+" (created and not deleted, gone anyway)")
|
||||
}
|
||||
}
|
||||
sort.Strings(unexplained)
|
||||
assert.Empty(t, unexplained, "volumes the injection log cannot account for")
|
||||
}
|
||||
|
||||
// TestSoak_ConcurrentRunsWithTheSameSeedMatch is what makes a long run worth
|
||||
// running: reproducing hour six needs the faults to be a function of the seed
|
||||
// and the calls, not of how the workers happened to interleave.
|
||||
func TestSoak_ConcurrentRunsWithTheSameSeedMatch(t *testing.T) {
|
||||
const seed = 99
|
||||
const duration = 500 * time.Millisecond
|
||||
|
||||
first := runSoak(t, seed, 4, duration)
|
||||
second := runSoak(t, seed, 4, duration)
|
||||
|
||||
// The runs cover different volumes, since each stops on a deadline. Only
|
||||
// the volumes both reached can be compared, and for those the faults must
|
||||
// be identical despite the interleaving being different.
|
||||
shared := 0
|
||||
firstByVolume := indexInjections(first.injections)
|
||||
secondByVolume := indexInjections(second.injections)
|
||||
for volumeID, injections := range firstByVolume {
|
||||
other, ok := secondByVolume[volumeID]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
shared++
|
||||
assert.Equal(t, injections, other, "volume %s drew different faults across runs with the same seed", volumeID)
|
||||
}
|
||||
require.Positive(t, shared, "the runs shared no volumes, so nothing was compared")
|
||||
}
|
||||
|
||||
func indexInjections(injections []hostile.Injection) map[string][]hostile.Injection {
|
||||
byVolume := map[string][]hostile.Injection{}
|
||||
for _, injection := range injections {
|
||||
byVolume[injection.Target] = append(byVolume[injection.Target], injection)
|
||||
}
|
||||
for _, list := range byVolume {
|
||||
sort.Slice(list, func(i, j int) bool {
|
||||
if list[i].Verb != list[j].Verb {
|
||||
return list[i].Verb < list[j].Verb
|
||||
}
|
||||
return list[i].Sequence < list[j].Sequence
|
||||
})
|
||||
}
|
||||
return byVolume
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
package hostile
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
)
|
||||
|
||||
// Verb names match the span names the contract emits, so an injection log and
|
||||
// a trace can be read side by side.
|
||||
const (
|
||||
verbCapabilities = "capabilities"
|
||||
verbVolumeCreate = "volume.create"
|
||||
verbVolumeDescribe = "volume.describe"
|
||||
verbVolumeList = "volume.list"
|
||||
verbVolumeExpand = "volume.expand"
|
||||
verbVolumeDelete = "volume.delete"
|
||||
verbVolumePublish = "volume.publish"
|
||||
verbVolumeUnpublish = "volume.unpublish"
|
||||
verbSnapshotCreate = "snapshot.create"
|
||||
verbSnapshotDelete = "snapshot.delete"
|
||||
verbSnapshotCopy = "snapshot.copy"
|
||||
verbSnapshotList = "snapshot.list"
|
||||
)
|
||||
|
||||
// lieFlavours are the ways a successful answer can be wrong. Each one is a
|
||||
// field the control plane reads and acts on.
|
||||
var lieFlavours = []string{"capacity", "handle", "state"}
|
||||
|
||||
// corruptVolume returns a copy of volume with one field wrong. It copies
|
||||
// rather than mutating: the inner provider's own object must stay correct, or
|
||||
// the fault becomes a real bug in the fake and nothing downstream means
|
||||
// anything.
|
||||
func corruptVolume(volume *ebsprovider.Volume, flavour string) *ebsprovider.Volume {
|
||||
if volume == nil {
|
||||
return nil
|
||||
}
|
||||
corrupted := *volume
|
||||
switch flavour {
|
||||
case "capacity":
|
||||
corrupted.CapacityBytes *= 2
|
||||
case "handle":
|
||||
corrupted.Handle = "hostile://wrong-handle"
|
||||
case "state":
|
||||
if corrupted.State == ebsprovider.VolumeStateAvailable {
|
||||
corrupted.State = ebsprovider.VolumeStateInUse
|
||||
} else {
|
||||
corrupted.State = ebsprovider.VolumeStateAvailable
|
||||
}
|
||||
}
|
||||
return &corrupted
|
||||
}
|
||||
|
||||
func corruptSnapshot(snapshot *ebsprovider.Snapshot, flavour string) *ebsprovider.Snapshot {
|
||||
if snapshot == nil {
|
||||
return nil
|
||||
}
|
||||
corrupted := *snapshot
|
||||
switch flavour {
|
||||
case "capacity":
|
||||
corrupted.SizeBytes *= 2
|
||||
case "handle":
|
||||
corrupted.SourceVolumeID = "vol-hostilewrongsrc"
|
||||
case "state":
|
||||
corrupted.State = ebsprovider.SnapshotStatePending
|
||||
}
|
||||
return &corrupted
|
||||
}
|
||||
|
||||
func corruptPublished(published *ebsprovider.PublishedVolume, flavour string) *ebsprovider.PublishedVolume {
|
||||
if published == nil {
|
||||
return nil
|
||||
}
|
||||
corrupted := *published
|
||||
switch flavour {
|
||||
case "handle", "capacity":
|
||||
corrupted.NBDURI = "nbd+unix:///?socket=/hostile/wrong.sock"
|
||||
case "state":
|
||||
corrupted.NodeID = "hostile-wrong-node"
|
||||
}
|
||||
return &corrupted
|
||||
}
|
||||
|
||||
func (p *Provider) GetCapabilities(ctx context.Context, req ebsprovider.GetCapabilitiesRequest) (*ebsprovider.GetCapabilitiesResponse, error) {
|
||||
injection := p.draw(verbCapabilities, "")
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := p.inner.GetCapabilities(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func (p *Provider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
injection := p.draw(verbVolumeCreate, req.VolumeID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
volume, err := p.inner.CreateVolume(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if injection.Fault == FaultLie {
|
||||
return corruptVolume(volume, injection.Detail), nil
|
||||
}
|
||||
return volume, nil
|
||||
}
|
||||
|
||||
func (p *Provider) GetVolume(ctx context.Context, req ebsprovider.GetVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
injection := p.draw(verbVolumeDescribe, req.VolumeID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
volume, err := p.inner.GetVolume(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if injection.Fault == FaultLie {
|
||||
return corruptVolume(volume, injection.Detail), nil
|
||||
}
|
||||
return volume, nil
|
||||
}
|
||||
|
||||
// ListVolumes is never made to lie. Enumeration is the oracle a soak run
|
||||
// checks its invariants with, and an instrument that reports what it was told
|
||||
// to report cannot falsify anything.
|
||||
func (p *Provider) ListVolumes(ctx context.Context, req ebsprovider.ListVolumesRequest) (*ebsprovider.ListVolumesResponse, error) {
|
||||
injection := p.drawTruthful(verbVolumeList, "")
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := p.inner.ListVolumes(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func (p *Provider) ExpandVolume(ctx context.Context, req ebsprovider.ExpandVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
injection := p.draw(verbVolumeExpand, req.VolumeID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
volume, err := p.inner.ExpandVolume(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if injection.Fault == FaultLie {
|
||||
return corruptVolume(volume, injection.Detail), nil
|
||||
}
|
||||
return volume, nil
|
||||
}
|
||||
|
||||
func (p *Provider) DeleteVolume(ctx context.Context, req ebsprovider.DeleteVolumeRequest) error {
|
||||
injection := p.draw(verbVolumeDelete, req.VolumeID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return err
|
||||
}
|
||||
return p.after(injection, p.inner.DeleteVolume(ctx, req))
|
||||
}
|
||||
|
||||
func (p *Provider) CreateSnapshot(ctx context.Context, req ebsprovider.CreateSnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
injection := p.draw(verbSnapshotCreate, req.SnapshotID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
snapshot, err := p.inner.CreateSnapshot(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if injection.Fault == FaultLie {
|
||||
return corruptSnapshot(snapshot, injection.Detail), nil
|
||||
}
|
||||
return snapshot, nil
|
||||
}
|
||||
|
||||
func (p *Provider) DeleteSnapshot(ctx context.Context, req ebsprovider.DeleteSnapshotRequest) error {
|
||||
injection := p.draw(verbSnapshotDelete, req.SnapshotID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return err
|
||||
}
|
||||
return p.after(injection, p.inner.DeleteSnapshot(ctx, req))
|
||||
}
|
||||
|
||||
func (p *Provider) CopySnapshot(ctx context.Context, req ebsprovider.CopySnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
injection := p.draw(verbSnapshotCopy, req.DestinationSnapshotID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
snapshot, err := p.inner.CopySnapshot(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if injection.Fault == FaultLie {
|
||||
return corruptSnapshot(snapshot, injection.Detail), nil
|
||||
}
|
||||
return snapshot, nil
|
||||
}
|
||||
|
||||
// ListSnapshots is never made to lie, for the same reason ListVolumes is not:
|
||||
// it is an oracle, and an instrument that reports what it was told to report
|
||||
// cannot falsify anything.
|
||||
func (p *Provider) ListSnapshots(ctx context.Context, req ebsprovider.ListSnapshotsRequest) (*ebsprovider.ListSnapshotsResponse, error) {
|
||||
injection := p.drawTruthful(verbSnapshotList, "")
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := p.inner.ListSnapshots(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func (p *Provider) PublishVolume(ctx context.Context, req ebsprovider.PublishVolumeRequest) (*ebsprovider.PublishedVolume, error) {
|
||||
injection := p.draw(verbVolumePublish, req.VolumeID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return nil, err
|
||||
}
|
||||
published, err := p.inner.PublishVolume(ctx, req)
|
||||
if err := p.after(injection, err); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if injection.Fault == FaultLie {
|
||||
return corruptPublished(published, injection.Detail), nil
|
||||
}
|
||||
return published, nil
|
||||
}
|
||||
|
||||
func (p *Provider) UnpublishVolume(ctx context.Context, req ebsprovider.UnpublishVolumeRequest) error {
|
||||
injection := p.draw(verbVolumeUnpublish, req.VolumeID)
|
||||
if stop, err := p.apply(ctx, injection); stop || err != nil {
|
||||
return err
|
||||
}
|
||||
return p.after(injection, p.inner.UnpublishVolume(ctx, req))
|
||||
}
|
||||
@@ -0,0 +1,443 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// MemoryProvider is a deterministic, concurrency-safe Provider used by
|
||||
// control-plane tests. It implements the same idempotency rules expected of a
|
||||
// real provider rather than acting as a programmable mock.
|
||||
type MemoryProvider struct {
|
||||
mu sync.RWMutex
|
||||
capabilities Capabilities
|
||||
volumes map[string]*memoryVolume
|
||||
snapshots map[string]*Snapshot
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
type memoryVolume struct {
|
||||
volume Volume
|
||||
sourceSnapshotID string
|
||||
parameters []byte
|
||||
seed []byte
|
||||
published *PublishedVolume
|
||||
}
|
||||
|
||||
var _ EBSProvider = (*MemoryProvider)(nil)
|
||||
|
||||
func NewMemoryProvider(capabilities Capabilities) *MemoryProvider {
|
||||
// Publication lives in this process's map, so node is what this provider
|
||||
// does whatever the caller passed. An unset scope is filled rather than
|
||||
// left empty: the contract has no valid empty scope, and a caller reading
|
||||
// one would have to guess.
|
||||
if capabilities.Exclusion.Scope == "" {
|
||||
capabilities.Exclusion.Scope = ExclusionScopeNode
|
||||
}
|
||||
return &MemoryProvider{
|
||||
capabilities: capabilities,
|
||||
volumes: make(map[string]*memoryVolume),
|
||||
snapshots: make(map[string]*Snapshot),
|
||||
now: time.Now,
|
||||
}
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) GetCapabilities(_ context.Context, req GetCapabilitiesRequest) (*GetCapabilitiesResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &GetCapabilitiesResponse{Versioned: NewVersioned(), Capabilities: m.capabilities}, nil
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) CreateVolume(_ context.Context, req CreateVolumeRequest) (*Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" || req.CapacityRange.RequiredBytes <= 0 ||
|
||||
(req.CapacityRange.LimitBytes > 0 && req.CapacityRange.RequiredBytes > req.CapacityRange.LimitBytes) {
|
||||
return nil, fmt.Errorf("%w: invalid volume ID or capacity range", ErrInvalidArgument)
|
||||
}
|
||||
if req.SourceSnapshotID != "" && req.SourceSnapshotVolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: source snapshot volume ID is required with a source snapshot", ErrInvalidArgument)
|
||||
}
|
||||
if err := ValidateSeedData(req.SeedData); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(req.SeedData) > 0 && !m.capabilities.VolumeSeeding {
|
||||
return nil, fmt.Errorf("%w: volume seeding", ErrUnsupportedCapability)
|
||||
}
|
||||
if int64(len(req.SeedData)) > req.CapacityRange.RequiredBytes {
|
||||
return nil, fmt.Errorf("%w: seed data is larger than the requested capacity", ErrInvalidArgument)
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
// SeedData is deliberately absent from the mismatch check below: a seed
|
||||
// applies only to a volume this call creates, so a repeated create returns
|
||||
// the existing volume rather than overwriting bytes the guest has since set.
|
||||
if existing := m.volumes[req.VolumeID]; existing != nil {
|
||||
if existing.volume.CapacityBytes != req.CapacityRange.RequiredBytes ||
|
||||
existing.volume.AvailabilityZone != req.AvailabilityZone ||
|
||||
existing.sourceSnapshotID != req.SourceSnapshotID ||
|
||||
!bytes.Equal(existing.parameters, req.Parameters) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ErrAlreadyExists, req.VolumeID)
|
||||
}
|
||||
return cloneVolume(&existing.volume), nil
|
||||
}
|
||||
if req.SourceSnapshotID != "" {
|
||||
snapshot := m.snapshots[req.SourceSnapshotID]
|
||||
if snapshot == nil {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ErrNotFound, req.SourceSnapshotID)
|
||||
}
|
||||
if req.CapacityRange.RequiredBytes < snapshot.SizeBytes {
|
||||
return nil, fmt.Errorf("%w: requested capacity is smaller than snapshot %s", ErrInvalidArgument, req.SourceSnapshotID)
|
||||
}
|
||||
// The caller names the snapshot's origin volume, so a wrong name is a
|
||||
// caller bug the reference provider refuses rather than resolving
|
||||
// blocks against a volume the snapshot never came from.
|
||||
if req.SourceSnapshotVolumeID != snapshot.SourceVolumeID {
|
||||
return nil, fmt.Errorf("%w: snapshot %s was taken from volume %s, not %s",
|
||||
ErrInvalidArgument, req.SourceSnapshotID, snapshot.SourceVolumeID, req.SourceSnapshotVolumeID)
|
||||
}
|
||||
}
|
||||
|
||||
volume := Volume{
|
||||
ID: req.VolumeID,
|
||||
CapacityBytes: req.CapacityRange.RequiredBytes,
|
||||
State: VolumeStateAvailable,
|
||||
Handle: "memory://volume/" + req.VolumeID,
|
||||
AvailabilityZone: req.AvailabilityZone,
|
||||
}
|
||||
m.volumes[req.VolumeID] = &memoryVolume{
|
||||
volume: volume,
|
||||
sourceSnapshotID: req.SourceSnapshotID,
|
||||
parameters: bytes.Clone(req.Parameters),
|
||||
seed: bytes.Clone(req.SeedData),
|
||||
}
|
||||
return cloneVolume(&volume), nil
|
||||
}
|
||||
|
||||
// SeedData returns the bytes a CreateVolume call seeded the volume with, so a
|
||||
// control-plane test can assert what it asked the provider to write.
|
||||
func (m *MemoryProvider) SeedData(volumeID string) ([]byte, bool) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
volume := m.volumes[volumeID]
|
||||
if volume == nil {
|
||||
return nil, false
|
||||
}
|
||||
return bytes.Clone(volume.seed), true
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) GetVolume(_ context.Context, req GetVolumeRequest) (*Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: volume ID is required", ErrInvalidArgument)
|
||||
}
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
volume := m.volumes[req.VolumeID]
|
||||
if volume == nil || (req.Handle != "" && req.Handle != volume.volume.Handle) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ErrNotFound, req.VolumeID)
|
||||
}
|
||||
return cloneVolume(&volume.volume), nil
|
||||
}
|
||||
|
||||
// ListVolumes pages through the volumes this provider holds, ordered by ID so
|
||||
// a token stays meaningful across calls. The token is the ID to resume after,
|
||||
// which keeps paging correct when volumes are created or deleted mid-walk.
|
||||
func (m *MemoryProvider) ListVolumes(_ context.Context, req ListVolumesRequest) (*ListVolumesResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !m.capabilities.VolumeEnumeration {
|
||||
return nil, fmt.Errorf("%w: volume enumeration", ErrUnsupportedCapability)
|
||||
}
|
||||
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
ids := make([]string, 0, len(m.volumes))
|
||||
for id := range m.volumes {
|
||||
if id > req.StartingToken {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
sort.Strings(ids)
|
||||
|
||||
pageSize := int(req.PageSize())
|
||||
response := &ListVolumesResponse{Versioned: NewVersioned()}
|
||||
if len(ids) > pageSize {
|
||||
response.NextToken = ids[pageSize-1]
|
||||
ids = ids[:pageSize]
|
||||
}
|
||||
for _, id := range ids {
|
||||
response.Volumes = append(response.Volumes, VolumeRef{ID: id, Handle: m.volumes[id].volume.Handle})
|
||||
}
|
||||
return response, nil
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) ExpandVolume(_ context.Context, req ExpandVolumeRequest) (*Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" || req.CapacityRange.RequiredBytes <= 0 ||
|
||||
(req.CapacityRange.LimitBytes > 0 && req.CapacityRange.RequiredBytes > req.CapacityRange.LimitBytes) {
|
||||
return nil, fmt.Errorf("%w: invalid volume ID or capacity range", ErrInvalidArgument)
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
volume := m.volumes[req.VolumeID]
|
||||
if volume == nil || (req.Handle != "" && req.Handle != volume.volume.Handle) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ErrNotFound, req.VolumeID)
|
||||
}
|
||||
if req.CapacityRange.RequiredBytes < volume.volume.CapacityBytes {
|
||||
return nil, fmt.Errorf("%w: volume expansion is grow-only", ErrInvalidArgument)
|
||||
}
|
||||
if volume.published != nil && !m.capabilities.OnlineExpansion && req.CapacityRange.RequiredBytes > volume.volume.CapacityBytes {
|
||||
return nil, fmt.Errorf("%w: provider does not support online expansion", ErrVolumeInUse)
|
||||
}
|
||||
volume.volume.CapacityBytes = req.CapacityRange.RequiredBytes
|
||||
return cloneVolume(&volume.volume), nil
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) DeleteVolume(_ context.Context, req DeleteVolumeRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.VolumeID == "" {
|
||||
return fmt.Errorf("%w: volume ID is required", ErrInvalidArgument)
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
volume := m.volumes[req.VolumeID]
|
||||
if volume == nil {
|
||||
return nil
|
||||
}
|
||||
if req.Handle != "" && req.Handle != volume.volume.Handle {
|
||||
return nil
|
||||
}
|
||||
if volume.published != nil {
|
||||
return fmt.Errorf("%w: volume %s", ErrVolumeInUse, req.VolumeID)
|
||||
}
|
||||
delete(m.volumes, req.VolumeID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) CreateSnapshot(_ context.Context, req CreateSnapshotRequest) (*Snapshot, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.SnapshotID == "" || req.VolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: snapshot and volume IDs are required", ErrInvalidArgument)
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if existing := m.snapshots[req.SnapshotID]; existing != nil {
|
||||
if existing.SourceVolumeID != req.VolumeID {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ErrAlreadyExists, req.SnapshotID)
|
||||
}
|
||||
return cloneSnapshot(existing), nil
|
||||
}
|
||||
volume := m.volumes[req.VolumeID]
|
||||
if volume == nil || (req.VolumeHandle != "" && req.VolumeHandle != volume.volume.Handle) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ErrNotFound, req.VolumeID)
|
||||
}
|
||||
snapshot := &Snapshot{
|
||||
ID: req.SnapshotID,
|
||||
SourceVolumeID: req.VolumeID,
|
||||
SizeBytes: volume.volume.CapacityBytes,
|
||||
CreatedAt: m.now().UTC(),
|
||||
State: SnapshotStateCompleted,
|
||||
Handle: "memory://snapshot/" + req.SnapshotID,
|
||||
}
|
||||
m.snapshots[req.SnapshotID] = snapshot
|
||||
return cloneSnapshot(snapshot), nil
|
||||
}
|
||||
|
||||
// Snapshot reports what the provider holds under snapshotID, so a
|
||||
// control-plane test can assert a snapshot really reached the provider rather
|
||||
// than existing only as a control-plane document.
|
||||
func (m *MemoryProvider) Snapshot(snapshotID string) (*Snapshot, bool) {
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
snapshot := m.snapshots[snapshotID]
|
||||
if snapshot == nil {
|
||||
return nil, false
|
||||
}
|
||||
return cloneSnapshot(snapshot), true
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) DeleteSnapshot(_ context.Context, req DeleteSnapshotRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.SnapshotID == "" {
|
||||
return fmt.Errorf("%w: snapshot ID is required", ErrInvalidArgument)
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
snapshot := m.snapshots[req.SnapshotID]
|
||||
if snapshot == nil || (req.Handle != "" && req.Handle != snapshot.Handle) {
|
||||
return nil
|
||||
}
|
||||
delete(m.snapshots, req.SnapshotID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// CopySnapshot duplicates SourceSnapshotID under DestinationSnapshotID.
|
||||
// Mirrors viperblock.CopySnapshotMeta's contract: it refuses an existing
|
||||
// destination, an empty or equal ID pair, and a snapshot that does not
|
||||
// belong to VolumeID (the caller's claimed source volume).
|
||||
func (m *MemoryProvider) CopySnapshot(_ context.Context, req CopySnapshotRequest) (*Snapshot, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.SourceSnapshotID == "" || req.DestinationSnapshotID == "" || req.VolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: source snapshot, destination snapshot, and volume IDs are required", ErrInvalidArgument)
|
||||
}
|
||||
if req.SourceSnapshotID == req.DestinationSnapshotID {
|
||||
return nil, fmt.Errorf("%w: destination snapshot must differ from the source", ErrInvalidArgument)
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
src := m.snapshots[req.SourceSnapshotID]
|
||||
if src == nil {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ErrNotFound, req.SourceSnapshotID)
|
||||
}
|
||||
if src.SourceVolumeID != req.VolumeID {
|
||||
return nil, fmt.Errorf("%w: snapshot %s belongs to volume %s, not %s", ErrInvalidArgument, req.SourceSnapshotID, src.SourceVolumeID, req.VolumeID)
|
||||
}
|
||||
if existing := m.snapshots[req.DestinationSnapshotID]; existing != nil {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ErrAlreadyExists, req.DestinationSnapshotID)
|
||||
}
|
||||
|
||||
dst := &Snapshot{
|
||||
ID: req.DestinationSnapshotID,
|
||||
SourceVolumeID: src.SourceVolumeID,
|
||||
SizeBytes: src.SizeBytes,
|
||||
CreatedAt: m.now().UTC(),
|
||||
State: SnapshotStateCompleted,
|
||||
Handle: "memory://snapshot/" + req.DestinationSnapshotID,
|
||||
}
|
||||
m.snapshots[req.DestinationSnapshotID] = dst
|
||||
return cloneSnapshot(dst), nil
|
||||
}
|
||||
|
||||
// ListSnapshots pages through the snapshots this provider holds, ordered by ID
|
||||
// so a token stays meaningful across calls. The token is the ID to resume
|
||||
// after, which keeps paging correct when snapshots come and go mid-walk.
|
||||
func (m *MemoryProvider) ListSnapshots(_ context.Context, req ListSnapshotsRequest) (*ListSnapshotsResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !m.capabilities.SnapshotEnumeration {
|
||||
return nil, fmt.Errorf("%w: snapshot enumeration", ErrUnsupportedCapability)
|
||||
}
|
||||
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
ids := make([]string, 0, len(m.snapshots))
|
||||
for id := range m.snapshots {
|
||||
if id > req.StartingToken {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
sort.Strings(ids)
|
||||
|
||||
pageSize := int(req.PageSize())
|
||||
response := &ListSnapshotsResponse{Versioned: NewVersioned()}
|
||||
if len(ids) > pageSize {
|
||||
response.NextToken = ids[pageSize-1]
|
||||
ids = ids[:pageSize]
|
||||
}
|
||||
for _, id := range ids {
|
||||
snapshot := m.snapshots[id]
|
||||
response.Snapshots = append(response.Snapshots, SnapshotRef{
|
||||
ID: id,
|
||||
SourceVolumeID: snapshot.SourceVolumeID,
|
||||
Handle: snapshot.Handle,
|
||||
})
|
||||
}
|
||||
return response, nil
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) PublishVolume(_ context.Context, req PublishVolumeRequest) (*PublishedVolume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" || req.NodeID == "" {
|
||||
return nil, fmt.Errorf("%w: volume and node IDs are required", ErrInvalidArgument)
|
||||
}
|
||||
if req.ReadOnly && !m.capabilities.ReadOnlyPublish {
|
||||
return nil, fmt.Errorf("%w: read-only publication", ErrUnsupportedCapability)
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
volume := m.volumes[req.VolumeID]
|
||||
if volume == nil || (req.Handle != "" && req.Handle != volume.volume.Handle) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ErrNotFound, req.VolumeID)
|
||||
}
|
||||
if volume.published != nil {
|
||||
if volume.published.NodeID != req.NodeID {
|
||||
return nil, fmt.Errorf("%w: volume %s is published to %s", ErrVolumeInUse, req.VolumeID, volume.published.NodeID)
|
||||
}
|
||||
return clonePublished(volume.published), nil
|
||||
}
|
||||
volume.published = &PublishedVolume{
|
||||
VolumeID: req.VolumeID,
|
||||
NodeID: req.NodeID,
|
||||
NBDURI: "nbd+unix:///?socket=/memory/" + req.VolumeID + ".sock",
|
||||
}
|
||||
volume.volume.State = VolumeStateInUse
|
||||
return clonePublished(volume.published), nil
|
||||
}
|
||||
|
||||
func (m *MemoryProvider) UnpublishVolume(_ context.Context, req UnpublishVolumeRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.VolumeID == "" || req.NodeID == "" {
|
||||
return fmt.Errorf("%w: volume and node IDs are required", ErrInvalidArgument)
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
volume := m.volumes[req.VolumeID]
|
||||
if volume == nil || volume.published == nil || volume.published.NodeID != req.NodeID {
|
||||
return nil
|
||||
}
|
||||
volume.published = nil
|
||||
volume.volume.State = VolumeStateAvailable
|
||||
return nil
|
||||
}
|
||||
|
||||
func cloneVolume(volume *Volume) *Volume {
|
||||
if volume == nil {
|
||||
return nil
|
||||
}
|
||||
clone := *volume
|
||||
clone.ProviderData = bytes.Clone(volume.ProviderData)
|
||||
return &clone
|
||||
}
|
||||
|
||||
func cloneSnapshot(snapshot *Snapshot) *Snapshot {
|
||||
if snapshot == nil {
|
||||
return nil
|
||||
}
|
||||
clone := *snapshot
|
||||
return &clone
|
||||
}
|
||||
|
||||
func clonePublished(published *PublishedVolume) *PublishedVolume {
|
||||
if published == nil {
|
||||
return nil
|
||||
}
|
||||
clone := *published
|
||||
return &clone
|
||||
}
|
||||
@@ -0,0 +1,274 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMemoryProviderLifecycleAndIdempotency(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{CrashConsistentSnapshot: true})
|
||||
createdAt := time.Date(2026, 8, 5, 6, 0, 0, 0, time.UTC)
|
||||
provider.now = func() time.Time { return createdAt }
|
||||
ctx := context.Background()
|
||||
|
||||
create := CreateVolumeRequest{
|
||||
Versioned: NewVersioned(),
|
||||
VolumeID: "vol-1",
|
||||
CapacityRange: CapacityRange{RequiredBytes: 8 << 30},
|
||||
AvailabilityZone: "ap-southeast-2a",
|
||||
}
|
||||
volume, err := provider.CreateVolume(ctx, create)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "memory://volume/vol-1", volume.Handle)
|
||||
assert.Equal(t, VolumeStateAvailable, volume.State)
|
||||
|
||||
repeated, err := provider.CreateVolume(ctx, create)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, volume, repeated, "same-name/same-parameters create must be idempotent")
|
||||
|
||||
changed := create
|
||||
changed.CapacityRange.RequiredBytes++
|
||||
_, err = provider.CreateVolume(ctx, changed)
|
||||
require.ErrorIs(t, err, ErrAlreadyExists)
|
||||
|
||||
published, err := provider.PublishVolume(ctx, PublishVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle, NodeID: "node-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "nbd+unix:///?socket=/memory/vol-1.sock", published.NBDURI)
|
||||
|
||||
repeatedPublish, err := provider.PublishVolume(ctx, PublishVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle, NodeID: "node-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, published, repeatedPublish)
|
||||
|
||||
_, err = provider.PublishVolume(ctx, PublishVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle, NodeID: "node-2",
|
||||
})
|
||||
require.ErrorIs(t, err, ErrVolumeInUse)
|
||||
require.ErrorIs(t, provider.DeleteVolume(ctx, DeleteVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle,
|
||||
}), ErrVolumeInUse)
|
||||
|
||||
require.NoError(t, provider.UnpublishVolume(ctx, UnpublishVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle, NodeID: "node-1",
|
||||
}))
|
||||
|
||||
expanded, err := provider.ExpandVolume(ctx, ExpandVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle,
|
||||
CapacityRange: CapacityRange{RequiredBytes: 16 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(16<<30), expanded.CapacityBytes)
|
||||
|
||||
snapshot, err := provider.CreateSnapshot(ctx, CreateSnapshotRequest{
|
||||
Versioned: NewVersioned(), SnapshotID: "snap-1", VolumeID: volume.ID, VolumeHandle: volume.Handle,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, createdAt, snapshot.CreatedAt)
|
||||
assert.Equal(t, SnapshotStateCompleted, snapshot.State)
|
||||
assert.Equal(t, expanded.CapacityBytes, snapshot.SizeBytes)
|
||||
|
||||
repeatedSnapshot, err := provider.CreateSnapshot(ctx, CreateSnapshotRequest{
|
||||
Versioned: NewVersioned(), SnapshotID: "snap-1", VolumeID: volume.ID, VolumeHandle: volume.Handle,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, snapshot, repeatedSnapshot)
|
||||
|
||||
require.NoError(t, provider.DeleteVolume(ctx, DeleteVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle,
|
||||
}))
|
||||
require.NoError(t, provider.DeleteVolume(ctx, DeleteVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: volume.ID, Handle: volume.Handle,
|
||||
}), "delete of an absent volume must be idempotent")
|
||||
|
||||
require.NoError(t, provider.DeleteSnapshot(ctx, DeleteSnapshotRequest{
|
||||
Versioned: NewVersioned(), SnapshotID: snapshot.ID, Handle: snapshot.Handle,
|
||||
}))
|
||||
require.NoError(t, provider.DeleteSnapshot(ctx, DeleteSnapshotRequest{
|
||||
Versioned: NewVersioned(), SnapshotID: snapshot.ID, Handle: snapshot.Handle,
|
||||
}), "delete of an absent snapshot must be idempotent")
|
||||
}
|
||||
|
||||
func TestMemoryProviderRequiresExplicitSchemaVersion(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
_, err := provider.CreateVolume(context.Background(), CreateVolumeRequest{
|
||||
VolumeID: "vol-unversioned", CapacityRange: CapacityRange{RequiredBytes: 1},
|
||||
})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
}
|
||||
|
||||
func TestProviderErrorUnwrap(t *testing.T) {
|
||||
err := &ProviderError{Code: ErrorCodeNotFound, Message: "volume disappeared"}
|
||||
assert.ErrorIs(t, err, ErrNotFound)
|
||||
assert.Equal(t, "volume disappeared", err.Error())
|
||||
}
|
||||
|
||||
// TestMemoryProviderChecksVersionOnEveryMethod covers the checkVersion guard
|
||||
// on every one of the nine EBSProvider methods directly on MemoryProvider,
|
||||
// independent of NATSProvider's own client-side pre-check.
|
||||
func TestMemoryProviderChecksVersionOnEveryMethod(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
ctx := context.Background()
|
||||
|
||||
_, err := provider.GetCapabilities(ctx, GetCapabilitiesRequest{})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
_, err = provider.CreateVolume(ctx, CreateVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
_, err = provider.GetVolume(ctx, GetVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
_, err = provider.ExpandVolume(ctx, ExpandVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
err = provider.DeleteVolume(ctx, DeleteVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
_, err = provider.CreateSnapshot(ctx, CreateSnapshotRequest{SnapshotID: "snap-1", VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
err = provider.DeleteSnapshot(ctx, DeleteSnapshotRequest{SnapshotID: "snap-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
_, err = provider.PublishVolume(ctx, PublishVolumeRequest{VolumeID: "vol-1", NodeID: "node-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
err = provider.UnpublishVolume(ctx, UnpublishVolumeRequest{VolumeID: "vol-1", NodeID: "node-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
}
|
||||
|
||||
// TestMemoryProviderCreateVolume_SourceSnapshotTooSmall covers CreateVolume's
|
||||
// snapshot-capacity guard: a volume created from a snapshot must be at least
|
||||
// as large as the snapshot it is cloned from.
|
||||
func TestMemoryProviderCreateVolume_SourceSnapshotTooSmall(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
ctx := context.Background()
|
||||
|
||||
_, err := provider.CreateVolume(ctx, CreateVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-src", CapacityRange: CapacityRange{RequiredBytes: 4 << 30}})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, CreateSnapshotRequest{Versioned: NewVersioned(), SnapshotID: "snap-src", VolumeID: "vol-src"})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = provider.CreateVolume(ctx, CreateVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: "vol-too-small", CapacityRange: CapacityRange{RequiredBytes: 1 << 30}, SourceSnapshotID: "snap-src",
|
||||
})
|
||||
require.ErrorIs(t, err, ErrInvalidArgument)
|
||||
}
|
||||
|
||||
// TestMemoryProviderExpandVolume_InvalidArgument covers ExpandVolume's own
|
||||
// argument validation (empty volume ID, non-positive capacity), separate
|
||||
// from the not_found and grow-only cases covered elsewhere.
|
||||
func TestMemoryProviderExpandVolume_InvalidArgument(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
_, err := provider.ExpandVolume(context.Background(), ExpandVolumeRequest{Versioned: NewVersioned()})
|
||||
require.ErrorIs(t, err, ErrInvalidArgument)
|
||||
}
|
||||
|
||||
// TestMemoryProviderExpandVolume_HandleMismatchIsNotFound covers the handle
|
||||
// mismatch arm of ExpandVolume's not_found check: a stale/wrong handle must
|
||||
// be treated the same as a missing volume, not silently ignored.
|
||||
func TestMemoryProviderExpandVolume_HandleMismatchIsNotFound(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, CreateVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-handle", CapacityRange: CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = provider.ExpandVolume(ctx, ExpandVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: "vol-handle", Handle: "memory://volume/stale", CapacityRange: CapacityRange{RequiredBytes: 2 << 30},
|
||||
})
|
||||
require.ErrorIs(t, err, ErrNotFound)
|
||||
}
|
||||
|
||||
// TestMemoryProviderDeleteVolume_InvalidArgumentAndHandleMismatch covers
|
||||
// DeleteVolume's empty-ID rejection and its handle-mismatch no-op: a caller
|
||||
// deleting by a stale handle must not delete the current volume out from
|
||||
// under a newer handle.
|
||||
func TestMemoryProviderDeleteVolume_InvalidArgumentAndHandleMismatch(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
ctx := context.Background()
|
||||
|
||||
err := provider.DeleteVolume(ctx, DeleteVolumeRequest{Versioned: NewVersioned()})
|
||||
require.ErrorIs(t, err, ErrInvalidArgument)
|
||||
|
||||
_, createErr := provider.CreateVolume(ctx, CreateVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-stale-handle", CapacityRange: CapacityRange{RequiredBytes: 1 << 30}})
|
||||
require.NoError(t, createErr)
|
||||
require.NoError(t, provider.DeleteVolume(ctx, DeleteVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-stale-handle", Handle: "memory://volume/wrong"}))
|
||||
|
||||
// The volume must still exist: the mismatched-handle delete above must
|
||||
// have been a no-op, not a real deletion.
|
||||
got, err := provider.GetVolume(ctx, GetVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-stale-handle"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "vol-stale-handle", got.ID)
|
||||
}
|
||||
|
||||
// TestMemoryProviderDeleteSnapshot_InvalidArgument covers DeleteSnapshot's
|
||||
// empty-ID rejection, the one argument-validation arm no other test reaches.
|
||||
func TestMemoryProviderDeleteSnapshot_InvalidArgument(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
err := provider.DeleteSnapshot(context.Background(), DeleteSnapshotRequest{Versioned: NewVersioned()})
|
||||
require.ErrorIs(t, err, ErrInvalidArgument)
|
||||
}
|
||||
|
||||
// TestCloneHelpersHandleNil covers the nil-input arm of each clone helper:
|
||||
// none of MemoryProvider's current call sites pass a nil pointer, so this
|
||||
// arm is otherwise unreachable dead code from the method-level tests alone.
|
||||
func TestCloneHelpersHandleNil(t *testing.T) {
|
||||
assert.Nil(t, cloneVolume(nil))
|
||||
assert.Nil(t, cloneSnapshot(nil))
|
||||
assert.Nil(t, clonePublished(nil))
|
||||
}
|
||||
|
||||
// A provider that does not advertise VolumeSeeding must refuse a seed outright
|
||||
// rather than accepting the request and silently dropping the bytes, which
|
||||
// would leave a guest booting an EFI volume full of zeroes.
|
||||
func TestMemoryProviderSeedRequiresCapability(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{})
|
||||
_, err := provider.CreateVolume(context.Background(), CreateVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: "vol-seed-nocap",
|
||||
CapacityRange: CapacityRange{RequiredBytes: 4096},
|
||||
SeedData: bytes.Repeat([]byte{0x01}, 16),
|
||||
})
|
||||
require.ErrorIs(t, err, ErrUnsupportedCapability)
|
||||
}
|
||||
|
||||
func TestMemoryProviderSeedIsStoredOnce(t *testing.T) {
|
||||
provider := NewMemoryProvider(Capabilities{VolumeSeeding: true})
|
||||
ctx := context.Background()
|
||||
seed := bytes.Repeat([]byte{0xAB}, 512)
|
||||
|
||||
create := CreateVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: "vol-seed-once",
|
||||
CapacityRange: CapacityRange{RequiredBytes: 4096}, SeedData: seed,
|
||||
}
|
||||
_, err := provider.CreateVolume(ctx, create)
|
||||
require.NoError(t, err)
|
||||
|
||||
stored, ok := provider.SeedData("vol-seed-once")
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, seed, stored)
|
||||
|
||||
// A relaunch reissues the same create. Reseeding here would overwrite an
|
||||
// EFI variable store the guest has since written its own BootOrder into.
|
||||
reseed := create
|
||||
reseed.SeedData = bytes.Repeat([]byte{0xCD}, 512)
|
||||
_, err = provider.CreateVolume(ctx, reseed)
|
||||
require.NoError(t, err, "a repeat create must stay idempotent regardless of seed")
|
||||
|
||||
stored, ok = provider.SeedData("vol-seed-once")
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, seed, stored, "an existing volume must never be reseeded")
|
||||
}
|
||||
|
||||
func TestValidateSeedData(t *testing.T) {
|
||||
require.NoError(t, ValidateSeedData(nil))
|
||||
require.NoError(t, ValidateSeedData(make([]byte, MaxSeedBytes)))
|
||||
require.ErrorIs(t, ValidateSeedData(make([]byte, MaxSeedBytes+1)), ErrInvalidArgument)
|
||||
}
|
||||
@@ -0,0 +1,455 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
|
||||
// subjectPrefix is common to every subject in the contract, including the
|
||||
// per-node mount/unmount subjects that have no constant of their own.
|
||||
const subjectPrefix = "ebs.provider.v1."
|
||||
|
||||
const (
|
||||
CapabilitiesSubject = "ebs.provider.v1.capabilities"
|
||||
CreateVolumeSubject = "ebs.provider.v1.volume.create"
|
||||
GetVolumeSubject = "ebs.provider.v1.volume.describe"
|
||||
ListVolumesSubject = "ebs.provider.v1.volume.list"
|
||||
ExpandVolumeSubject = "ebs.provider.v1.volume.expand"
|
||||
DeleteVolumeSubject = "ebs.provider.v1.volume.delete"
|
||||
DeleteSnapshotSubject = "ebs.provider.v1.snapshot.delete"
|
||||
CopySnapshotSubject = "ebs.provider.v1.snapshot.copy"
|
||||
ListSnapshotsSubject = "ebs.provider.v1.snapshot.list"
|
||||
|
||||
// SnapshotCreateSubjectPrefix is the wildcard prefix servers subscribe to
|
||||
// (SnapshotCreateSubjectPrefix + "*") to catch every per-volume create
|
||||
// subject SnapshotSubject builds.
|
||||
SnapshotCreateSubjectPrefix = "ebs.provider.v1.snapshot.create."
|
||||
)
|
||||
|
||||
const defaultRequestTimeout = 30 * time.Second
|
||||
|
||||
// SnapshotSubject addresses a create by source volume so the owning node can
|
||||
// serve it. The create verb is its own token: a bare volume ID here would
|
||||
// collide with DeleteSnapshotSubject and make the wildcard unsubscribable.
|
||||
func SnapshotSubject(volumeID string) (string, error) {
|
||||
if err := validateSubjectToken(volumeID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return SnapshotCreateSubjectPrefix + volumeID, nil
|
||||
}
|
||||
|
||||
func SnapshotCompletionSubject(snapshotID string) (string, error) {
|
||||
if err := validateSubjectToken(snapshotID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "ebs.provider.v1.snapshot.response." + snapshotID, nil
|
||||
}
|
||||
|
||||
func PublishSubject(nodeID string) (string, error) {
|
||||
if err := validateSubjectToken(nodeID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return subjectPrefix + nodeID + ".mount", nil
|
||||
}
|
||||
|
||||
func UnpublishSubject(nodeID string) (string, error) {
|
||||
if err := validateSubjectToken(nodeID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return subjectPrefix + nodeID + ".unmount", nil
|
||||
}
|
||||
|
||||
// ownerSubjectPrefix is common to every owner-routed subject OwnerSubject
|
||||
// builds: ebs.provider.v1.owner.{volumeID}.{verb}.
|
||||
const ownerSubjectPrefix = subjectPrefix + "owner."
|
||||
|
||||
// Owner verbs are a closed set: only these four operations have an
|
||||
// owner-routed variant, so callers use the named wrapper functions below
|
||||
// instead of a string literal at the call site.
|
||||
const (
|
||||
verbSnapshotCreate = "snapshot.create"
|
||||
verbSnapshotCopy = "snapshot.copy"
|
||||
verbVolumeExpand = "volume.expand"
|
||||
verbVolumeDescribe = "volume.describe"
|
||||
)
|
||||
|
||||
// OwnerSubject addresses volumeID's mounting node directly, bypassing the
|
||||
// spinifex-workers queue group that would otherwise deliver the request to
|
||||
// a random node. verb must be one of the unexported verb constants above.
|
||||
func OwnerSubject(volumeID, verb string) (string, error) {
|
||||
if err := validateSubjectToken(volumeID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return ownerSubjectPrefix + volumeID + "." + verb, nil
|
||||
}
|
||||
|
||||
func SnapshotCreateOwnerSubject(volumeID string) (string, error) {
|
||||
return OwnerSubject(volumeID, verbSnapshotCreate)
|
||||
}
|
||||
|
||||
func SnapshotCopyOwnerSubject(volumeID string) (string, error) {
|
||||
return OwnerSubject(volumeID, verbSnapshotCopy)
|
||||
}
|
||||
|
||||
func ExpandVolumeOwnerSubject(volumeID string) (string, error) {
|
||||
return OwnerSubject(volumeID, verbVolumeExpand)
|
||||
}
|
||||
|
||||
func GetVolumeOwnerSubject(volumeID string) (string, error) {
|
||||
return OwnerSubject(volumeID, verbVolumeDescribe)
|
||||
}
|
||||
|
||||
// ParseOwnerSubject recovers the volumeID and verb OwnerSubject encoded into
|
||||
// subject. volumeID cannot itself contain '.' (validateSubjectToken enforces
|
||||
// this), so the first '.' after ownerSubjectPrefix is always the
|
||||
// volumeID/verb boundary, even though every verb also contains one.
|
||||
func ParseOwnerSubject(subject string) (volumeID, verb string, ok bool) {
|
||||
rest, found := strings.CutPrefix(subject, ownerSubjectPrefix)
|
||||
if !found {
|
||||
return "", "", false
|
||||
}
|
||||
volumeID, verb, found = strings.Cut(rest, ".")
|
||||
if !found || volumeID == "" || verb == "" {
|
||||
return "", "", false
|
||||
}
|
||||
return volumeID, verb, true
|
||||
}
|
||||
|
||||
func validateSubjectToken(value string) error {
|
||||
if value == "" || strings.ContainsAny(value, ".*>") {
|
||||
return fmt.Errorf("%w: invalid NATS subject token %q", ErrInvalidArgument, value)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NATSProvider drives a provider daemon through the versioned ebs.* contract.
|
||||
type NATSProvider struct {
|
||||
conn *nats.Conn
|
||||
requestTimeout time.Duration
|
||||
}
|
||||
|
||||
var _ EBSProvider = (*NATSProvider)(nil)
|
||||
|
||||
func NewNATSProvider(conn *nats.Conn, requestTimeout time.Duration) *NATSProvider {
|
||||
if requestTimeout <= 0 {
|
||||
requestTimeout = defaultRequestTimeout
|
||||
}
|
||||
return &NATSProvider{conn: conn, requestTimeout: requestTimeout}
|
||||
}
|
||||
|
||||
func (p *NATSProvider) GetCapabilities(ctx context.Context, req GetCapabilitiesRequest) (*GetCapabilitiesResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response GetCapabilitiesResponse
|
||||
if err := p.request(ctx, CapabilitiesSubject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &response, nil
|
||||
}
|
||||
|
||||
func (p *NATSProvider) CreateVolume(ctx context.Context, req CreateVolumeRequest) (*Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := ValidateSeedData(req.SeedData); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response CreateVolumeResponse
|
||||
if err := p.request(ctx, CreateVolumeSubject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Volume == nil {
|
||||
return nil, fmt.Errorf("ebs.create returned no volume")
|
||||
}
|
||||
return response.Volume, nil
|
||||
}
|
||||
|
||||
func (p *NATSProvider) GetVolume(ctx context.Context, req GetVolumeRequest) (*Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ownerSubject, err := GetVolumeOwnerSubject(req.VolumeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response GetVolumeResponse
|
||||
if err := p.requestOwnerFirst(ctx, ownerSubject, GetVolumeSubject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Volume == nil {
|
||||
return nil, fmt.Errorf("ebs.describe returned no volume")
|
||||
}
|
||||
return response.Volume, nil
|
||||
}
|
||||
|
||||
// ListVolumes asks the provider what it holds. It is not owner-routed: the
|
||||
// question is about the whole provider, not one volume, so it goes to the
|
||||
// queue group rather than to any single volume's mounting node.
|
||||
func (p *NATSProvider) ListVolumes(ctx context.Context, req ListVolumesRequest) (*ListVolumesResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response ListVolumesResponse
|
||||
if err := p.request(ctx, ListVolumesSubject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &response, nil
|
||||
}
|
||||
|
||||
func (p *NATSProvider) ExpandVolume(ctx context.Context, req ExpandVolumeRequest) (*Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ownerSubject, err := ExpandVolumeOwnerSubject(req.VolumeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response ExpandVolumeResponse
|
||||
if err := p.requestOwnerFirst(ctx, ownerSubject, ExpandVolumeSubject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Volume == nil {
|
||||
return nil, fmt.Errorf("ebs.expand returned no volume")
|
||||
}
|
||||
return response.Volume, nil
|
||||
}
|
||||
|
||||
func (p *NATSProvider) DeleteVolume(ctx context.Context, req DeleteVolumeRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
var response DeleteVolumeResponse
|
||||
if err := p.request(ctx, DeleteVolumeSubject, req, &response); err != nil {
|
||||
return err
|
||||
}
|
||||
return responseError(response.SchemaVersion, response.Error)
|
||||
}
|
||||
|
||||
func (p *NATSProvider) CreateSnapshot(ctx context.Context, req CreateSnapshotRequest) (*Snapshot, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
requestSubject, err := SnapshotSubject(req.VolumeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ownerSubject, err := SnapshotCreateOwnerSubject(req.VolumeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
completionSubject, err := SnapshotCompletionSubject(req.SnapshotID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if p.conn == nil || !p.conn.IsConnected() {
|
||||
return nil, nats.ErrConnectionClosed
|
||||
}
|
||||
|
||||
completionSub, err := p.conn.SubscribeSync(completionSubject)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("subscribe to snapshot completion: %w", err)
|
||||
}
|
||||
defer completionSub.Unsubscribe()
|
||||
if err := p.conn.Flush(); err != nil {
|
||||
return nil, fmt.Errorf("flush snapshot completion subscription: %w", err)
|
||||
}
|
||||
|
||||
var accepted CreateSnapshotResponse
|
||||
if err := p.requestOwnerFirst(ctx, ownerSubject, requestSubject, req, &accepted); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(accepted.SchemaVersion, accepted.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if accepted.Snapshot != nil && accepted.Snapshot.State != SnapshotStatePending {
|
||||
return accepted.Snapshot, nil
|
||||
}
|
||||
if accepted.OperationID == "" {
|
||||
return nil, fmt.Errorf("%s returned no operation ID", requestSubject)
|
||||
}
|
||||
if accepted.CompletionSubject != "" && accepted.CompletionSubject != completionSubject {
|
||||
return nil, fmt.Errorf("snapshot completion subject mismatch: got %q, want %q", accepted.CompletionSubject, completionSubject)
|
||||
}
|
||||
|
||||
waitCtx, waitSpan := startCompletionSpan(ctx, verbSnapshotCreate, completionSubject)
|
||||
defer waitSpan.End()
|
||||
msg, err := completionSub.NextMsgWithContext(waitCtx)
|
||||
if err != nil {
|
||||
RecordSpanError(waitSpan, err)
|
||||
return nil, fmt.Errorf("wait for snapshot %s completion: %w", req.SnapshotID, err)
|
||||
}
|
||||
RecordResponseError(waitSpan, msg.Data)
|
||||
var completed CreateSnapshotResponse
|
||||
if err := json.Unmarshal(msg.Data, &completed); err != nil {
|
||||
return nil, fmt.Errorf("decode snapshot completion: %w", err)
|
||||
}
|
||||
if err := responseError(completed.SchemaVersion, completed.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if completed.OperationID != accepted.OperationID {
|
||||
return nil, fmt.Errorf("snapshot operation mismatch: got %q, want %q", completed.OperationID, accepted.OperationID)
|
||||
}
|
||||
if completed.Snapshot == nil || completed.Snapshot.State != SnapshotStateCompleted {
|
||||
return nil, fmt.Errorf("snapshot %s completion returned no completed snapshot", req.SnapshotID)
|
||||
}
|
||||
return completed.Snapshot, nil
|
||||
}
|
||||
|
||||
func (p *NATSProvider) DeleteSnapshot(ctx context.Context, req DeleteSnapshotRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
var response DeleteSnapshotResponse
|
||||
if err := p.request(ctx, DeleteSnapshotSubject, req, &response); err != nil {
|
||||
return err
|
||||
}
|
||||
return responseError(response.SchemaVersion, response.Error)
|
||||
}
|
||||
|
||||
// CopySnapshot is a plain synchronous request/reply, unlike CreateSnapshot's
|
||||
// accept-then-publish pattern: duplicating a snapshot's metadata is a couple
|
||||
// of small object writes with no flush/upload step slow enough to warrant a
|
||||
// completion subject.
|
||||
func (p *NATSProvider) CopySnapshot(ctx context.Context, req CopySnapshotRequest) (*Snapshot, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ownerSubject, err := SnapshotCopyOwnerSubject(req.VolumeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response CopySnapshotResponse
|
||||
if err := p.requestOwnerFirst(ctx, ownerSubject, CopySnapshotSubject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Snapshot == nil {
|
||||
return nil, fmt.Errorf("ebs.snapshot.copy returned no snapshot")
|
||||
}
|
||||
return response.Snapshot, nil
|
||||
}
|
||||
|
||||
// ListSnapshots asks the provider what it holds. Like ListVolumes it is not
|
||||
// owner-routed: the question is about the whole provider, not one snapshot.
|
||||
func (p *NATSProvider) ListSnapshots(ctx context.Context, req ListSnapshotsRequest) (*ListSnapshotsResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response ListSnapshotsResponse
|
||||
if err := p.request(ctx, ListSnapshotsSubject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &response, nil
|
||||
}
|
||||
|
||||
func (p *NATSProvider) PublishVolume(ctx context.Context, req PublishVolumeRequest) (*PublishedVolume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
subject, err := PublishSubject(req.NodeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var response PublishVolumeResponse
|
||||
if err := p.request(ctx, subject, req, &response); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := responseError(response.SchemaVersion, response.Error); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if response.Published == nil {
|
||||
return nil, fmt.Errorf("%s returned no published volume", subject)
|
||||
}
|
||||
return response.Published, nil
|
||||
}
|
||||
|
||||
func (p *NATSProvider) UnpublishVolume(ctx context.Context, req UnpublishVolumeRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
subject, err := UnpublishSubject(req.NodeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var response UnpublishVolumeResponse
|
||||
if err := p.request(ctx, subject, req, &response); err != nil {
|
||||
return err
|
||||
}
|
||||
return responseError(response.SchemaVersion, response.Error)
|
||||
}
|
||||
|
||||
func (p *NATSProvider) request(ctx context.Context, subject string, input, output any) error {
|
||||
if p.conn == nil || !p.conn.IsConnected() {
|
||||
return nats.ErrConnectionClosed
|
||||
}
|
||||
payload, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encode %s request: %w", subject, err)
|
||||
}
|
||||
|
||||
request := &nats.Msg{Subject: subject, Data: payload, Header: nats.Header{}}
|
||||
ctx, span := startClientSpan(ctx, subject, request.Header)
|
||||
defer span.End()
|
||||
|
||||
requestCtx, cancel := context.WithTimeout(ctx, p.requestTimeout)
|
||||
defer cancel()
|
||||
msg, err := p.conn.RequestMsgWithContext(requestCtx, request)
|
||||
if err != nil {
|
||||
RecordSpanError(span, err)
|
||||
return fmt.Errorf("request %s: %w", subject, err)
|
||||
}
|
||||
if err := json.Unmarshal(msg.Data, output); err != nil {
|
||||
RecordSpanError(span, err)
|
||||
return fmt.Errorf("decode %s response: %w", subject, err)
|
||||
}
|
||||
RecordResponseError(span, msg.Data)
|
||||
return nil
|
||||
}
|
||||
|
||||
// requestOwnerFirst tries ownerSubject, falling back to queueSubject only on
|
||||
// nats.ErrNoResponders, which means nothing is mounted. A timeout must never
|
||||
// fall back: a still-running owner would have the operation run twice.
|
||||
func (p *NATSProvider) requestOwnerFirst(ctx context.Context, ownerSubject, queueSubject string, input, output any) error {
|
||||
err := p.request(ctx, ownerSubject, input, output)
|
||||
if err == nil || !errors.Is(err, nats.ErrNoResponders) {
|
||||
return err
|
||||
}
|
||||
return p.request(ctx, queueSubject, input, output)
|
||||
}
|
||||
|
||||
func responseError(version uint16, providerErr *ProviderError) error {
|
||||
if err := checkVersion(version); err != nil {
|
||||
return err
|
||||
}
|
||||
if providerErr != nil {
|
||||
return providerErr
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,383 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestNATSProviderCreateVolumeUsesVersionedContract(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
sub, err := conn.Subscribe(CreateVolumeSubject, func(msg *nats.Msg) {
|
||||
var request CreateVolumeRequest
|
||||
require.NoError(t, json.Unmarshal(msg.Data, &request))
|
||||
assert.Equal(t, SchemaVersion, request.SchemaVersion)
|
||||
assert.Equal(t, "vol-1", request.VolumeID)
|
||||
response, marshalErr := json.Marshal(CreateVolumeResponse{
|
||||
Versioned: NewVersioned(),
|
||||
Volume: &Volume{ID: request.VolumeID, CapacityBytes: request.CapacityRange.RequiredBytes, Handle: "vb://vol-1"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(response))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
volume, err := provider.CreateVolume(t.Context(), CreateVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: "vol-1", CapacityRange: CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "vb://vol-1", volume.Handle)
|
||||
}
|
||||
|
||||
func TestNATSProviderRejectsResponseVersionSkew(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
sub, err := conn.Subscribe(GetVolumeSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(GetVolumeResponse{
|
||||
Versioned: Versioned{SchemaVersion: SchemaVersion + 1},
|
||||
Volume: &Volume{ID: "vol-1"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
_, err = provider.GetVolume(t.Context(), GetVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
}
|
||||
|
||||
func TestNATSProviderReturnsTypedProviderError(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
sub, err := conn.Subscribe(DeleteVolumeSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(DeleteVolumeResponse{
|
||||
Versioned: NewVersioned(),
|
||||
Error: &ProviderError{Code: ErrorCodeVolumeInUse, Message: "volume is mounted"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
err = provider.DeleteVolume(t.Context(), DeleteVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrVolumeInUse)
|
||||
assert.Equal(t, "volume is mounted", err.Error())
|
||||
}
|
||||
|
||||
func TestNATSProviderSnapshotWaitsForAsyncCompletion(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
subject, err := SnapshotSubject("vol-1")
|
||||
require.NoError(t, err)
|
||||
completionSubject, err := SnapshotCompletionSubject("snap-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
sub, err := conn.Subscribe(subject, func(msg *nats.Msg) {
|
||||
accepted, marshalErr := json.Marshal(CreateSnapshotResponse{
|
||||
Versioned: NewVersioned(), OperationID: "op-snap-1", CompletionSubject: completionSubject,
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(accepted))
|
||||
|
||||
completed, marshalErr := json.Marshal(CreateSnapshotResponse{
|
||||
Versioned: NewVersioned(), OperationID: "op-snap-1",
|
||||
Snapshot: &Snapshot{ID: "snap-1", SourceVolumeID: "vol-1", State: SnapshotStateCompleted},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, conn.Publish(completionSubject, completed))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
snapshot, err := provider.CreateSnapshot(t.Context(), CreateSnapshotRequest{
|
||||
Versioned: NewVersioned(), SnapshotID: "snap-1", VolumeID: "vol-1",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "snap-1", snapshot.ID)
|
||||
}
|
||||
|
||||
func TestNATSProviderRejectsUnsafeSubjectTokens(t *testing.T) {
|
||||
provider := NewNATSProvider(nil, time.Second)
|
||||
_, err := provider.PublishVolume(t.Context(), PublishVolumeRequest{
|
||||
Versioned: NewVersioned(), VolumeID: "vol-1", NodeID: "node.*",
|
||||
})
|
||||
require.ErrorIs(t, err, ErrInvalidArgument)
|
||||
assert.NotErrorIs(t, err, nats.ErrConnectionClosed, "subject validation must happen before transport use")
|
||||
}
|
||||
|
||||
// TestSubjectBuildersRejectUnsafeTokens tables every NATS subject-token
|
||||
// validation across the four functions that embed a caller-supplied ID in a
|
||||
// subject: empty, ".", "..", and any token containing a NATS wildcard
|
||||
// character must be rejected before a subject string is ever built.
|
||||
func TestSubjectBuildersRejectUnsafeTokens(t *testing.T) {
|
||||
badTokens := []string{"", ".", "..", "a.b", "a*b", "a>b"}
|
||||
|
||||
builders := []struct {
|
||||
name string
|
||||
build func(string) (string, error)
|
||||
}{
|
||||
{"SnapshotSubject", SnapshotSubject},
|
||||
{"SnapshotCompletionSubject", SnapshotCompletionSubject},
|
||||
{"PublishSubject", PublishSubject},
|
||||
{"UnpublishSubject", UnpublishSubject},
|
||||
}
|
||||
|
||||
for _, b := range builders {
|
||||
t.Run(b.name, func(t *testing.T) {
|
||||
for _, token := range badTokens {
|
||||
t.Run(fmt.Sprintf("%q", token), func(t *testing.T) {
|
||||
_, err := b.build(token)
|
||||
require.ErrorIs(t, err, ErrInvalidArgument)
|
||||
})
|
||||
}
|
||||
t.Run("valid token", func(t *testing.T) {
|
||||
subject, err := b.build("node-1")
|
||||
require.NoError(t, err)
|
||||
assert.NotEmpty(t, subject)
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestNATSProviderChecksVersionBeforeTransport confirms every EBSProvider
|
||||
// method on NATSProvider rejects an unsupported SchemaVersion via
|
||||
// checkVersion before it ever touches the connection. A nil *nats.Conn would
|
||||
// panic or return nats.ErrConnectionClosed if any method reached transport,
|
||||
// so seeing ErrUnsupportedVersion here proves the ordering.
|
||||
func TestNATSProviderChecksVersionBeforeTransport(t *testing.T) {
|
||||
provider := NewNATSProvider(nil, time.Second)
|
||||
ctx := t.Context()
|
||||
|
||||
t.Run("GetCapabilities", func(t *testing.T) {
|
||||
_, err := provider.GetCapabilities(ctx, GetCapabilitiesRequest{})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("CreateVolume", func(t *testing.T) {
|
||||
_, err := provider.CreateVolume(ctx, CreateVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("GetVolume", func(t *testing.T) {
|
||||
_, err := provider.GetVolume(ctx, GetVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("ExpandVolume", func(t *testing.T) {
|
||||
_, err := provider.ExpandVolume(ctx, ExpandVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("DeleteVolume", func(t *testing.T) {
|
||||
err := provider.DeleteVolume(ctx, DeleteVolumeRequest{VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("CreateSnapshot", func(t *testing.T) {
|
||||
_, err := provider.CreateSnapshot(ctx, CreateSnapshotRequest{SnapshotID: "snap-1", VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("DeleteSnapshot", func(t *testing.T) {
|
||||
err := provider.DeleteSnapshot(ctx, DeleteSnapshotRequest{SnapshotID: "snap-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("PublishVolume", func(t *testing.T) {
|
||||
_, err := provider.PublishVolume(ctx, PublishVolumeRequest{VolumeID: "vol-1", NodeID: "node-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
t.Run("UnpublishVolume", func(t *testing.T) {
|
||||
err := provider.UnpublishVolume(ctx, UnpublishVolumeRequest{VolumeID: "vol-1", NodeID: "node-1"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
})
|
||||
}
|
||||
|
||||
// TestNewNATSProviderDefaultsTimeout covers the requestTimeout<=0 branch:
|
||||
// callers passing a zero or negative timeout get defaultRequestTimeout
|
||||
// rather than a provider that never times out.
|
||||
func TestNewNATSProviderDefaultsTimeout(t *testing.T) {
|
||||
assert.Equal(t, defaultRequestTimeout, NewNATSProvider(nil, 0).requestTimeout)
|
||||
assert.Equal(t, defaultRequestTimeout, NewNATSProvider(nil, -time.Second).requestTimeout)
|
||||
assert.Equal(t, 7*time.Second, NewNATSProvider(nil, 7*time.Second).requestTimeout)
|
||||
}
|
||||
|
||||
// TestNATSProviderGetCapabilities_TransportError covers the p.request error
|
||||
// branch (no responder on the subject) for a method that otherwise never
|
||||
// exercises it elsewhere in this suite.
|
||||
func TestNATSProviderGetCapabilities_TransportError(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
provider := NewNATSProvider(conn, 200*time.Millisecond)
|
||||
_, err := provider.GetCapabilities(t.Context(), GetCapabilitiesRequest{Versioned: NewVersioned()})
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
// TestNATSProviderGetCapabilities_ProviderError covers responseError's
|
||||
// populated-Error branch for GetCapabilities specifically.
|
||||
func TestNATSProviderGetCapabilities_ProviderError(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
sub, err := conn.Subscribe(CapabilitiesSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(GetCapabilitiesResponse{
|
||||
Versioned: NewVersioned(), Error: &ProviderError{Code: ErrorCodeInternal, Message: "capabilities unavailable"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
_, err = provider.GetCapabilities(t.Context(), GetCapabilitiesRequest{Versioned: NewVersioned()})
|
||||
require.EqualError(t, err, "capabilities unavailable")
|
||||
}
|
||||
|
||||
// TestNATSProviderRequest covers p.request's three error branches directly:
|
||||
// an unmarshalable input, a subject with no responder, and a responder that
|
||||
// answers with invalid JSON.
|
||||
func TestNATSProviderRequest(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
provider := NewNATSProvider(conn, 200*time.Millisecond)
|
||||
|
||||
t.Run("marshal error", func(t *testing.T) {
|
||||
var out GetVolumeResponse
|
||||
err := provider.request(t.Context(), "ebs.provider.v1.test.marshal", make(chan int), &out)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "encode")
|
||||
})
|
||||
|
||||
t.Run("no responder", func(t *testing.T) {
|
||||
var out GetVolumeResponse
|
||||
err := provider.request(t.Context(), "ebs.provider.v1.test.noresponder", GetVolumeRequest{Versioned: NewVersioned()}, &out)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "request")
|
||||
})
|
||||
|
||||
t.Run("decode error", func(t *testing.T) {
|
||||
const subject = "ebs.provider.v1.test.baddecode"
|
||||
sub, err := conn.Subscribe(subject, func(msg *nats.Msg) {
|
||||
require.NoError(t, msg.Respond([]byte("{not valid json")))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
var out GetVolumeResponse
|
||||
err = provider.request(t.Context(), subject, GetVolumeRequest{Versioned: NewVersioned()}, &out)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "decode")
|
||||
})
|
||||
}
|
||||
|
||||
// TestNATSProviderOwnerFirst_RoutesToOwnerSubject proves the owner subject
|
||||
// is tried first: only a responder on GetVolumeOwnerSubject exists (no
|
||||
// GetVolumeSubject responder at all), so a successful reply proves the
|
||||
// request never needed the queue-group subject.
|
||||
func TestNATSProviderOwnerFirst_RoutesToOwnerSubject(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
|
||||
ownerSubject, err := GetVolumeOwnerSubject("vol-ownerroute001")
|
||||
require.NoError(t, err)
|
||||
ownerSub, err := conn.Subscribe(ownerSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(GetVolumeResponse{
|
||||
Versioned: NewVersioned(), Volume: &Volume{ID: "vol-ownerroute001", Handle: "owner-handle"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = ownerSub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
volume, err := provider.GetVolume(t.Context(), GetVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-ownerroute001"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "owner-handle", volume.Handle, "the owner subject's own responder must have answered")
|
||||
}
|
||||
|
||||
// TestNATSProviderOwnerFirst_NoRespondersFallsBackToQueueSubject proves the
|
||||
// one condition allowed to trigger a fallback: no responder at all on the
|
||||
// owner subject. Only GetVolumeSubject (the queue-group subject) has a
|
||||
// responder here, so success proves the fallback fired.
|
||||
func TestNATSProviderOwnerFirst_NoRespondersFallsBackToQueueSubject(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
|
||||
queueSub, err := conn.Subscribe(GetVolumeSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(GetVolumeResponse{
|
||||
Versioned: NewVersioned(), Volume: &Volume{ID: "vol-ownerfallback1", Handle: "queue-handle"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = queueSub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
volume, err := provider.GetVolume(t.Context(), GetVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-ownerfallback1"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "queue-handle", volume.Handle, "no owner responder must fall back to the queue-group subject")
|
||||
}
|
||||
|
||||
// TestNATSProviderOwnerFirst_NonNoRespondersDoesNotFallBack is the dangerous
|
||||
// case requestOwnerFirst exists to avoid: an owner subject that has a
|
||||
// responder but never replies produces a timeout, not nats.ErrNoResponders,
|
||||
// and must NOT fall back to the queue subject — a queue fallback here could
|
||||
// run the same operation (e.g. a snapshot) a second time while the owner is
|
||||
// still mid-flight. A responsive queue-subject responder is present so a
|
||||
// wrongful fallback would succeed silently instead of failing loudly.
|
||||
func TestNATSProviderOwnerFirst_NonNoRespondersDoesNotFallBack(t *testing.T) {
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
|
||||
ownerSubject, err := GetVolumeOwnerSubject("vol-nofallback001")
|
||||
require.NoError(t, err)
|
||||
ownerSub, err := conn.Subscribe(ownerSubject, func(msg *nats.Msg) {
|
||||
// Deliberately never responds, forcing the client to time out.
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = ownerSub.Unsubscribe() })
|
||||
|
||||
var queueHits atomic.Int32
|
||||
queueSub, err := conn.Subscribe(GetVolumeSubject, func(msg *nats.Msg) {
|
||||
queueHits.Add(1)
|
||||
payload, marshalErr := json.Marshal(GetVolumeResponse{
|
||||
Versioned: NewVersioned(), Volume: &Volume{ID: "vol-nofallback001", Handle: "queue-handle"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = queueSub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, 200*time.Millisecond)
|
||||
_, err = provider.GetVolume(t.Context(), GetVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-nofallback001"})
|
||||
require.Error(t, err, "the owner responder never replies, so this must time out")
|
||||
require.NotErrorIs(t, err, nats.ErrNoResponders, "a timeout must never be mistaken for no responders")
|
||||
assert.Equal(t, int32(0), queueHits.Load(), "a timeout must never fall back to the queue-group subject")
|
||||
}
|
||||
|
||||
// TestResponseErrorRejectsZeroSchemaVersion covers responseError directly: a
|
||||
// reply carrying the zero SchemaVersion (e.g. a server that forgot to set
|
||||
// Versioned on an error path) must be rejected before its Error field is
|
||||
// ever inspected, even when that field is nil.
|
||||
func TestResponseErrorRejectsZeroSchemaVersion(t *testing.T) {
|
||||
err := responseError(0, nil)
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion)
|
||||
|
||||
err = responseError(0, &ProviderError{Code: ErrorCodeNotFound, Message: "ignored"})
|
||||
require.ErrorIs(t, err, ErrUnsupportedVersion, "the version guard must win over a populated error field")
|
||||
|
||||
require.NoError(t, responseError(SchemaVersion, nil))
|
||||
|
||||
err = responseError(SchemaVersion, &ProviderError{Code: ErrorCodeVolumeInUse, Message: "in use"})
|
||||
require.ErrorIs(t, err, ErrVolumeInUse)
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package natsserve
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// TestClassifyError pins every sentinel to the code it crosses the wire as. A
|
||||
// sentinel missing from classifyError falls through to internal, which tells a
|
||||
// caller a retryable refusal is permanent and loses the reason with it.
|
||||
func TestClassifyError(t *testing.T) {
|
||||
tests := []struct {
|
||||
err error
|
||||
want ebsprovider.ErrorCode
|
||||
}{
|
||||
{ebsprovider.ErrAlreadyExists, ebsprovider.ErrorCodeAlreadyExists},
|
||||
{ebsprovider.ErrInvalidArgument, ebsprovider.ErrorCodeInvalidArgument},
|
||||
{ebsprovider.ErrNotFound, ebsprovider.ErrorCodeNotFound},
|
||||
{ebsprovider.ErrUnsupportedVersion, ebsprovider.ErrorCodeUnsupportedVersion},
|
||||
{ebsprovider.ErrVolumeInUse, ebsprovider.ErrorCodeVolumeInUse},
|
||||
{ebsprovider.ErrUnsupportedCapability, ebsprovider.ErrorCodeUnsupportedCap},
|
||||
{ebsprovider.ErrUnavailable, ebsprovider.ErrorCodeUnavailable},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(string(tt.want), func(t *testing.T) {
|
||||
wrapped := fmt.Errorf("vol-1: %w", tt.err)
|
||||
got := classifyError(wrapped)
|
||||
assert.Equal(t, tt.want, got.Code)
|
||||
assert.Equal(t, wrapped.Error(), got.Message, "the wrapping context must survive classification")
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("unrecognised error is internal", func(t *testing.T) {
|
||||
got := classifyError(fmt.Errorf("something unexpected"))
|
||||
assert.Equal(t, ebsprovider.ErrorCodeInternal, got.Code)
|
||||
assert.False(t, got.Code.Retryable(), "an error we cannot classify must not invite a retry loop")
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,525 @@
|
||||
// Package natsserve is the provider-neutral NATS server for the
|
||||
// ebs.provider.v1.* wire contract defined by spinifex/ebsprovider. It
|
||||
// depends on nothing but the ebsprovider.EBSProvider interface, so any implementation of that interface can be served over NATS by calling Serve.
|
||||
package natsserve
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
// DefaultQueueGroup is the NATS queue group Serve subscribes the
|
||||
// multi-node-shared subjects (every subject except the per-node
|
||||
// publish/unpublish pair) under by default, matching viperblockd's production subscriptions.
|
||||
const DefaultQueueGroup = "spinifex-workers"
|
||||
|
||||
// Options configures Serve's subscription behavior.
|
||||
type Options struct {
|
||||
// QueueGroup names the queue group the shared subjects (Capabilities,
|
||||
// CreateVolume, GetVolume, ExpandVolume, DeleteVolume, DeleteSnapshot,
|
||||
// CopySnapshot, and the snapshot-create wildcard) subscribe under. Empty defaults to DefaultQueueGroup; set NoQueueGroup to opt out.
|
||||
QueueGroup string
|
||||
|
||||
// NoQueueGroup subscribes the shared subjects plainly instead of under
|
||||
// a queue group, e.g. for a single-instance server where load balancing
|
||||
// across workers does not apply.
|
||||
NoQueueGroup bool
|
||||
|
||||
// NodeID scopes PublishVolume/UnpublishVolume to this node's own
|
||||
// subjects (ebsprovider.PublishSubject / UnpublishSubject), matching a
|
||||
// production daemon that can only mount volumes on its own node. Left empty (the default), Serve instead subscribes the wildcard "ebs.provider.v1.*.mount"/"unmount" subjects, answering requests for any node.
|
||||
NodeID string
|
||||
}
|
||||
|
||||
// msgHandler takes the per-message context so the server span opened for the
|
||||
// message reaches the work the handler does, rather than only timing it.
|
||||
type msgHandler func(ctx context.Context, msg *nats.Msg)
|
||||
|
||||
// traced opens a server span for each message, joining the caller's trace
|
||||
// from its headers, and ends it when the handler returns.
|
||||
func traced(base context.Context, handler msgHandler) nats.MsgHandler {
|
||||
return func(msg *nats.Msg) {
|
||||
ctx, span := ebsprovider.StartServerSpan(base, msg)
|
||||
defer span.End()
|
||||
handler(ctx, msg)
|
||||
}
|
||||
}
|
||||
|
||||
// Serve subscribes every subject spinifex/ebsprovider/nats.go defines,
|
||||
// except owner-routed subjects (see the comment in the function body), and
|
||||
// delegates each to provider. The returned stop function unsubscribes everything Serve registered; it is safe to call more than once.
|
||||
func Serve(ctx context.Context, nc *nats.Conn, provider ebsprovider.EBSProvider, opts Options) (stop func(), err error) {
|
||||
// Owner-subject routing (ebsprovider.OwnerSubject and its per-verb
|
||||
// wrappers) is not served: it routes a request to the node holding a
|
||||
// volume's live engine via a per-node mounted-volume registry the EBSProvider interface does not model, so it stays viperblockd-specific.
|
||||
queueGroup := opts.QueueGroup
|
||||
if queueGroup == "" {
|
||||
queueGroup = DefaultQueueGroup
|
||||
}
|
||||
if opts.NoQueueGroup {
|
||||
queueGroup = ""
|
||||
}
|
||||
|
||||
var subs []*nats.Subscription
|
||||
cleanup := func() {
|
||||
for _, sub := range subs {
|
||||
_ = sub.Unsubscribe()
|
||||
}
|
||||
}
|
||||
|
||||
subscribeShared := func(subject string, handler msgHandler) error {
|
||||
var sub *nats.Subscription
|
||||
var subErr error
|
||||
if queueGroup != "" {
|
||||
sub, subErr = nc.QueueSubscribe(subject, queueGroup, traced(ctx, handler))
|
||||
} else {
|
||||
sub, subErr = nc.Subscribe(subject, traced(ctx, handler))
|
||||
}
|
||||
if subErr != nil {
|
||||
return fmt.Errorf("subscribe to %s: %w", subject, subErr)
|
||||
}
|
||||
subs = append(subs, sub)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Publish/Unpublish are never queue-grouped: each subject already
|
||||
// addresses exactly one node (or, in the wildcard case, is answered by
|
||||
// the one backing provider standing in for every node), so load balancing across a group has nothing to add.
|
||||
subscribePlain := func(subject string, handler msgHandler) error {
|
||||
sub, subErr := nc.Subscribe(subject, traced(ctx, handler))
|
||||
if subErr != nil {
|
||||
return fmt.Errorf("subscribe to %s: %w", subject, subErr)
|
||||
}
|
||||
subs = append(subs, sub)
|
||||
return nil
|
||||
}
|
||||
|
||||
steps := []struct {
|
||||
subject string
|
||||
handler msgHandler
|
||||
}{
|
||||
{ebsprovider.CapabilitiesSubject, handleCapabilities(provider)},
|
||||
{ebsprovider.CreateVolumeSubject, handleCreateVolume(provider)},
|
||||
{ebsprovider.GetVolumeSubject, handleGetVolume(provider)},
|
||||
{ebsprovider.ListVolumesSubject, handleListVolumes(provider)},
|
||||
{ebsprovider.ExpandVolumeSubject, handleExpandVolume(provider)},
|
||||
{ebsprovider.DeleteVolumeSubject, handleDeleteVolume(provider)},
|
||||
{ebsprovider.DeleteSnapshotSubject, handleDeleteSnapshot(provider)},
|
||||
{ebsprovider.CopySnapshotSubject, handleCopySnapshot(provider)},
|
||||
{ebsprovider.ListSnapshotsSubject, handleListSnapshots(provider)},
|
||||
{ebsprovider.SnapshotCreateSubjectPrefix + "*", handleCreateSnapshot(nc, provider)},
|
||||
}
|
||||
for _, step := range steps {
|
||||
if err := subscribeShared(step.subject, step.handler); err != nil {
|
||||
cleanup()
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
if opts.NodeID == "" {
|
||||
if err := subscribePlain("ebs.provider.v1.*.mount", handlePublishVolume(provider)); err != nil {
|
||||
cleanup()
|
||||
return nil, err
|
||||
}
|
||||
if err := subscribePlain("ebs.provider.v1.*.unmount", handleUnpublishVolume(provider)); err != nil {
|
||||
cleanup()
|
||||
return nil, err
|
||||
}
|
||||
} else {
|
||||
publishSubject, err := ebsprovider.PublishSubject(opts.NodeID)
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return nil, fmt.Errorf("build publish subject: %w", err)
|
||||
}
|
||||
if err := subscribePlain(publishSubject, handlePublishVolume(provider)); err != nil {
|
||||
cleanup()
|
||||
return nil, err
|
||||
}
|
||||
unpublishSubject, err := ebsprovider.UnpublishSubject(opts.NodeID)
|
||||
if err != nil {
|
||||
cleanup()
|
||||
return nil, fmt.Errorf("build unpublish subject: %w", err)
|
||||
}
|
||||
if err := subscribePlain(unpublishSubject, handleUnpublishVolume(provider)); err != nil {
|
||||
cleanup()
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
if err := nc.Flush(); err != nil {
|
||||
cleanup()
|
||||
return nil, fmt.Errorf("flush subscriptions: %w", err)
|
||||
}
|
||||
|
||||
var once sync.Once
|
||||
return func() { once.Do(cleanup) }, nil
|
||||
}
|
||||
|
||||
// respond marks the message's span with whatever error the reply carries, so
|
||||
// a verb that failed shows as an error rather than a successful round trip.
|
||||
func respond(ctx context.Context, msg *nats.Msg, v any) {
|
||||
data, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
slog.Error("ebs.provider: failed to marshal response", "subject", msg.Subject, "err", err)
|
||||
return
|
||||
}
|
||||
ebsprovider.RecordResponseError(trace.SpanFromContext(ctx), data)
|
||||
if err := msg.Respond(data); err != nil {
|
||||
slog.Error("ebs.provider: failed to send response", "subject", msg.Subject, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// badRequestError maps a JSON decode failure to the wire error shape,
|
||||
// mirroring viperblockd's badRequestError: a malformed request body is the
|
||||
// caller's mistake, not a server-side failure.
|
||||
func badRequestError(err error) *ebsprovider.ProviderError {
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeInvalidArgument, Message: fmt.Sprintf("bad request: %v", err)}
|
||||
}
|
||||
|
||||
func versionError(got uint16) *ebsprovider.ProviderError {
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeUnsupportedVersion, Message: fmt.Sprintf("unsupported schema version %d, want %d", got, ebsprovider.SchemaVersion)}
|
||||
}
|
||||
|
||||
func invalidArgumentError(format string, args ...any) *ebsprovider.ProviderError {
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeInvalidArgument, Message: fmt.Sprintf(format, args...)}
|
||||
}
|
||||
|
||||
func internalError(format string, args ...any) *ebsprovider.ProviderError {
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeInternal, Message: fmt.Sprintf(format, args...)}
|
||||
}
|
||||
|
||||
// classifyError maps a provider error to the wire ProviderError shape,
|
||||
// using only the sentinel errors and ErrorCode values errors.go exports:
|
||||
// it has no forward (error -> ProviderError) mapper of its own to call.
|
||||
func classifyError(err error) *ebsprovider.ProviderError {
|
||||
switch {
|
||||
case errors.Is(err, ebsprovider.ErrAlreadyExists):
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeAlreadyExists, Message: err.Error()}
|
||||
case errors.Is(err, ebsprovider.ErrInvalidArgument):
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeInvalidArgument, Message: err.Error()}
|
||||
case errors.Is(err, ebsprovider.ErrNotFound):
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeNotFound, Message: err.Error()}
|
||||
case errors.Is(err, ebsprovider.ErrUnsupportedVersion):
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeUnsupportedVersion, Message: err.Error()}
|
||||
case errors.Is(err, ebsprovider.ErrVolumeInUse):
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeVolumeInUse, Message: err.Error()}
|
||||
case errors.Is(err, ebsprovider.ErrUnsupportedCapability):
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeUnsupportedCap, Message: err.Error()}
|
||||
case errors.Is(err, ebsprovider.ErrUnavailable):
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeUnavailable, Message: err.Error()}
|
||||
default:
|
||||
return &ebsprovider.ProviderError{Code: ebsprovider.ErrorCodeInternal, Message: err.Error()}
|
||||
}
|
||||
}
|
||||
|
||||
func handleCapabilities(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.GetCapabilitiesRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.GetCapabilitiesResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.GetCapabilitiesResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
resp, err := provider.GetCapabilities(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.GetCapabilitiesResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, *resp)
|
||||
}
|
||||
}
|
||||
|
||||
func handleCreateVolume(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.CreateVolumeRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.CreateVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.CreateVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
vol, err := provider.CreateVolume(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.CreateVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.CreateVolumeResponse{Versioned: ebsprovider.NewVersioned(), Volume: vol})
|
||||
}
|
||||
}
|
||||
|
||||
func handleGetVolume(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.GetVolumeRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.GetVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.GetVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
vol, err := provider.GetVolume(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.GetVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.GetVolumeResponse{Versioned: ebsprovider.NewVersioned(), Volume: vol})
|
||||
}
|
||||
}
|
||||
|
||||
func handleListVolumes(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.ListVolumesRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.ListVolumesResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.ListVolumesResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
resp, err := provider.ListVolumes(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.ListVolumesResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, *resp)
|
||||
}
|
||||
}
|
||||
|
||||
func handleListSnapshots(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.ListSnapshotsRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.ListSnapshotsResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.ListSnapshotsResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
resp, err := provider.ListSnapshots(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.ListSnapshotsResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, *resp)
|
||||
}
|
||||
}
|
||||
|
||||
func handleExpandVolume(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.ExpandVolumeRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.ExpandVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.ExpandVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
vol, err := provider.ExpandVolume(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.ExpandVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.ExpandVolumeResponse{Versioned: ebsprovider.NewVersioned(), Volume: vol})
|
||||
}
|
||||
}
|
||||
|
||||
func handleDeleteVolume(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.DeleteVolumeRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.DeleteVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.DeleteVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
if err := provider.DeleteVolume(ctx, req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.DeleteVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.DeleteVolumeResponse{Versioned: ebsprovider.NewVersioned()})
|
||||
}
|
||||
}
|
||||
|
||||
func handleDeleteSnapshot(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.DeleteSnapshotRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.DeleteSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.DeleteSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
if err := provider.DeleteSnapshot(ctx, req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.DeleteSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.DeleteSnapshotResponse{Versioned: ebsprovider.NewVersioned()})
|
||||
}
|
||||
}
|
||||
|
||||
func handleCopySnapshot(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.CopySnapshotRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.CopySnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.CopySnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
snap, err := provider.CopySnapshot(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.CopySnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.CopySnapshotResponse{Versioned: ebsprovider.NewVersioned(), Snapshot: snap})
|
||||
}
|
||||
}
|
||||
|
||||
// handleCreateSnapshot serves the SnapshotCreateSubjectPrefix wildcard by
|
||||
// replying immediately with a pending Snapshot plus an OperationID, then
|
||||
// running the blocking EBSProvider.CreateSnapshot call in the background and publishing its result: a transport-level wrapping, so no async form is needed in the interface itself.
|
||||
func handleCreateSnapshot(nc *nats.Conn, provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.CreateSnapshotRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.CreateSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.CreateSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
|
||||
// The wildcard subject carries the source volume ID as its own
|
||||
// token (SnapshotCreateSubjectPrefix + volumeID); a mismatch with
|
||||
// the request body means the two disagree about which volume this create is for.
|
||||
if subjectVolumeID, ok := strings.CutPrefix(msg.Subject, ebsprovider.SnapshotCreateSubjectPrefix); ok && subjectVolumeID != req.VolumeID {
|
||||
respond(ctx, msg, ebsprovider.CreateSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: invalidArgumentError("volume id %q in subject does not match request volume id %q", subjectVolumeID, req.VolumeID)})
|
||||
return
|
||||
}
|
||||
|
||||
completionSubject, err := ebsprovider.SnapshotCompletionSubject(req.SnapshotID)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.CreateSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
operationID, err := newOperationID()
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.CreateSnapshotResponse{Versioned: ebsprovider.NewVersioned(), Error: internalError("generate operation id: %v", err)})
|
||||
return
|
||||
}
|
||||
|
||||
respond(ctx, msg, ebsprovider.CreateSnapshotResponse{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
OperationID: operationID,
|
||||
CompletionSubject: completionSubject,
|
||||
Snapshot: &ebsprovider.Snapshot{
|
||||
ID: req.SnapshotID,
|
||||
SourceVolumeID: req.VolumeID,
|
||||
State: ebsprovider.SnapshotStatePending,
|
||||
},
|
||||
})
|
||||
|
||||
go completeCreateSnapshot(ctx, nc, provider, req, operationID, completionSubject)
|
||||
}
|
||||
}
|
||||
|
||||
func completeCreateSnapshot(ctx context.Context, nc *nats.Conn, provider ebsprovider.EBSProvider, req ebsprovider.CreateSnapshotRequest, operationID, completionSubject string) {
|
||||
completion := ebsprovider.CreateSnapshotResponse{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
OperationID: operationID,
|
||||
CompletionSubject: completionSubject,
|
||||
}
|
||||
|
||||
snapshot, err := provider.CreateSnapshot(ctx, req)
|
||||
if err != nil {
|
||||
completion.Error = classifyError(err)
|
||||
} else {
|
||||
completion.Snapshot = snapshot
|
||||
}
|
||||
|
||||
data, err := json.Marshal(completion)
|
||||
if err != nil {
|
||||
slog.Error("ebs.provider.snapshot.create: failed to marshal completion", "volume", req.VolumeID, "snapshot", req.SnapshotID, "err", err)
|
||||
return
|
||||
}
|
||||
if err := nc.Publish(completionSubject, data); err != nil {
|
||||
slog.Error("ebs.provider.snapshot.create: failed to publish completion", "subject", completionSubject, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// newOperationID mints a random operation ID without depending on anything
|
||||
// beyond the standard library, keeping natsserve's only non-stdlib
|
||||
// dependencies ebsprovider and nats.go.
|
||||
func newOperationID() (string, error) {
|
||||
b := make([]byte, 9)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", fmt.Errorf("read random bytes: %w", err)
|
||||
}
|
||||
return "op-" + hex.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
func handlePublishVolume(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.PublishVolumeRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.PublishVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.PublishVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
pub, err := provider.PublishVolume(ctx, req)
|
||||
if err != nil {
|
||||
respond(ctx, msg, ebsprovider.PublishVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.PublishVolumeResponse{Versioned: ebsprovider.NewVersioned(), Published: pub})
|
||||
}
|
||||
}
|
||||
|
||||
func handleUnpublishVolume(provider ebsprovider.EBSProvider) msgHandler {
|
||||
return func(ctx context.Context, msg *nats.Msg) {
|
||||
var req ebsprovider.UnpublishVolumeRequest
|
||||
if err := json.Unmarshal(msg.Data, &req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.UnpublishVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: badRequestError(err)})
|
||||
return
|
||||
}
|
||||
if req.SchemaVersion != ebsprovider.SchemaVersion {
|
||||
respond(ctx, msg, ebsprovider.UnpublishVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: versionError(req.SchemaVersion)})
|
||||
return
|
||||
}
|
||||
if err := provider.UnpublishVolume(ctx, req); err != nil {
|
||||
respond(ctx, msg, ebsprovider.UnpublishVolumeResponse{Versioned: ebsprovider.NewVersioned(), Error: classifyError(err)})
|
||||
return
|
||||
}
|
||||
respond(ctx, msg, ebsprovider.UnpublishVolumeResponse{Versioned: ebsprovider.NewVersioned()})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
package natsserve_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/natsserve"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
||||
"go.opentelemetry.io/otel/sdk/trace/tracetest"
|
||||
)
|
||||
|
||||
func spansByKind(t *testing.T, recorder *tracetest.SpanRecorder) (client, server sdktrace.ReadOnlySpan) {
|
||||
t.Helper()
|
||||
client, server = testutil.SpansByKind(t, recorder)
|
||||
require.NotNil(t, client, "no client span recorded")
|
||||
require.NotNil(t, server, "no server span recorded")
|
||||
return client, server
|
||||
}
|
||||
|
||||
func serveMemoryProvider(t *testing.T) *ebsprovider.NATSProvider {
|
||||
t.Helper()
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeEnumeration: true})
|
||||
stop, err := natsserve.Serve(t.Context(), conn, provider, natsserve.Options{NoQueueGroup: true})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(stop)
|
||||
return ebsprovider.NewNATSProvider(conn, 5*time.Second)
|
||||
}
|
||||
|
||||
// TestServerSpan_JoinsTheCallersTrace is the property that makes provider
|
||||
// latency attributable: without it the two halves of one operation are two
|
||||
// unrelated traces, and time spent in the provider cannot be told from time
|
||||
// spent on the wire.
|
||||
func TestServerSpan_JoinsTheCallersTrace(t *testing.T) {
|
||||
recorder := testutil.RecordSpans(t)
|
||||
client := serveMemoryProvider(t)
|
||||
|
||||
_, err := client.ListVolumes(t.Context(), ebsprovider.ListVolumesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
|
||||
clientSpan, serverSpan := spansByKind(t, recorder)
|
||||
assert.Equal(t, "ebs.volume.list", clientSpan.Name())
|
||||
assert.Equal(t, "ebs.volume.list", serverSpan.Name())
|
||||
assert.Equal(t, clientSpan.SpanContext().TraceID(), serverSpan.SpanContext().TraceID())
|
||||
assert.Equal(t, clientSpan.SpanContext().SpanID(), serverSpan.Parent().SpanID())
|
||||
}
|
||||
|
||||
// TestServerSpan_RecordsTheErrorItReturns keeps a failed verb from looking
|
||||
// like a successful round trip on the serving side.
|
||||
func TestServerSpan_RecordsTheErrorItReturns(t *testing.T) {
|
||||
recorder := testutil.RecordSpans(t)
|
||||
client := serveMemoryProvider(t)
|
||||
|
||||
_, err := client.GetVolume(t.Context(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-absent",
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
|
||||
_, serverSpan := spansByKind(t, recorder)
|
||||
assert.Equal(t, codes.Error, serverSpan.Status().Code)
|
||||
}
|
||||
|
||||
// TestServerSpan_SurvivesAnUntracedCaller covers a caller that never installed
|
||||
// a propagator: the provider must still open its own span rather than drop the
|
||||
// message or inherit a broken parent.
|
||||
func TestServerSpan_SurvivesAnUntracedCaller(t *testing.T) {
|
||||
recorder := testutil.RecordSpans(t)
|
||||
client := serveMemoryProvider(t)
|
||||
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator())
|
||||
|
||||
_, err := client.ListVolumes(t.Context(), ebsprovider.ListVolumesRequest{Versioned: ebsprovider.NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, serverSpan := spansByKind(t, recorder)
|
||||
assert.Equal(t, "ebs.volume.list", serverSpan.Name())
|
||||
assert.False(t, serverSpan.Parent().IsValid(), "an untraced caller must not leave a dangling parent")
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
// Package nullprovider serves the EBSProvider contract without any storage
|
||||
// behind it. It exists to establish the floor: a measurement taken against it
|
||||
// is the cost of the contract, the transport and the control plane's own
|
||||
// bookkeeping, with no engine, no object store and no disk in the number.
|
||||
//
|
||||
// It is not a type of its own. ebsprovider.MemoryProvider already answers
|
||||
// every verb from a map, and natsserve already serves any EBSProvider, so a
|
||||
// null provider is those two composed; this package only names that
|
||||
// composition and states what it does and does not measure.
|
||||
package nullprovider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/natsserve"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
|
||||
// Capabilities is what a null provider advertises. Everything the contract
|
||||
// makes optional is on: the point is to price the verbs, and a capability
|
||||
// turned off is a verb that returns early instead of being measured.
|
||||
var Capabilities = ebsprovider.Capabilities{
|
||||
OnlineExpansion: true,
|
||||
SparseExtentReporting: true,
|
||||
CrashConsistentSnapshot: true,
|
||||
VolumeSeeding: true,
|
||||
ReadOnlyPublish: true,
|
||||
VolumeEnumeration: true,
|
||||
SnapshotEnumeration: true,
|
||||
Exclusion: ebsprovider.ExclusionSemantics{Scope: ebsprovider.ExclusionScopeNode},
|
||||
}
|
||||
|
||||
// New returns an in-process null provider. Calls against it skip NATS
|
||||
// entirely, so it prices the contract's own validation and bookkeeping and
|
||||
// nothing else.
|
||||
func New() *ebsprovider.MemoryProvider {
|
||||
return ebsprovider.NewMemoryProvider(Capabilities)
|
||||
}
|
||||
|
||||
// Serve stands a null provider up on nc and returns a client that reaches it
|
||||
// the way the control plane reaches any provider: encode, publish, queue
|
||||
// dispatch, decode. The difference between this and New is the transport, and
|
||||
// the difference between this and a real provider is the storage.
|
||||
//
|
||||
// PublishVolume answers with a URI nothing serves, so this bounds control
|
||||
// operations only. It cannot stand in for a data-path comparison; qemunbdd is
|
||||
// the control for that.
|
||||
func Serve(ctx context.Context, nc *nats.Conn, opts natsserve.Options) (client *ebsprovider.NATSProvider, stop func(), err error) {
|
||||
stop, err = natsserve.Serve(ctx, nc, New(), opts)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("serve null provider: %w", err)
|
||||
}
|
||||
return ebsprovider.NewNATSProvider(nc, 0), stop, nil
|
||||
}
|
||||
@@ -0,0 +1,437 @@
|
||||
// Package ebsprovider defines Spinifex's provider-neutral block-storage
|
||||
// contract. Provider implementations must not expose their engine's Go types
|
||||
// through this package; provider-specific settings and handles stay opaque.
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// SchemaVersion is the only wire-contract version understood by this build.
|
||||
// Every request and response carries it so version skew fails at the boundary
|
||||
// instead of silently decoding a changed payload.
|
||||
const SchemaVersion uint16 = 1
|
||||
|
||||
// Versioned is embedded in every request and response sent over NATS.
|
||||
type Versioned struct {
|
||||
SchemaVersion uint16 `json:"schema_version"`
|
||||
}
|
||||
|
||||
func NewVersioned() Versioned { return Versioned{SchemaVersion: SchemaVersion} }
|
||||
|
||||
// EBSProvider is the block-storage controller contract consumed by Spinifex.
|
||||
// Its operation and idempotency shape follows the CSI controller service, but
|
||||
// the transport remains NATS request/reply rather than CSI's gRPC transport.
|
||||
type EBSProvider interface {
|
||||
GetCapabilities(context.Context, GetCapabilitiesRequest) (*GetCapabilitiesResponse, error)
|
||||
CreateVolume(context.Context, CreateVolumeRequest) (*Volume, error)
|
||||
GetVolume(context.Context, GetVolumeRequest) (*Volume, error)
|
||||
ListVolumes(context.Context, ListVolumesRequest) (*ListVolumesResponse, error)
|
||||
ExpandVolume(context.Context, ExpandVolumeRequest) (*Volume, error)
|
||||
DeleteVolume(context.Context, DeleteVolumeRequest) error
|
||||
CreateSnapshot(context.Context, CreateSnapshotRequest) (*Snapshot, error)
|
||||
DeleteSnapshot(context.Context, DeleteSnapshotRequest) error
|
||||
CopySnapshot(context.Context, CopySnapshotRequest) (*Snapshot, error)
|
||||
ListSnapshots(context.Context, ListSnapshotsRequest) (*ListSnapshotsResponse, error)
|
||||
PublishVolume(context.Context, PublishVolumeRequest) (*PublishedVolume, error)
|
||||
UnpublishVolume(context.Context, UnpublishVolumeRequest) error
|
||||
}
|
||||
|
||||
// Provider is retained as a concise alias for implementation code. New
|
||||
// consumers should name the dependency EBSProvider at composition boundaries.
|
||||
type Provider = EBSProvider
|
||||
|
||||
// Capabilities advertises optional provider behavior. Callers must branch on
|
||||
// these values instead of assuming all implementations behave like viperblock.
|
||||
//
|
||||
// Every field must name behaviour some request in this interface can ask for.
|
||||
// A capability no verb reaches tells a caller a feature is available and then
|
||||
// gives them no way to use it.
|
||||
type Capabilities struct {
|
||||
OnlineExpansion bool `json:"online_expansion"`
|
||||
SparseExtentReporting bool `json:"sparse_extent_reporting"`
|
||||
CrashConsistentSnapshot bool `json:"crash_consistent_snapshot"`
|
||||
VolumeSeeding bool `json:"volume_seeding"`
|
||||
|
||||
// ReadOnlyPublish advertises that PublishVolume honours ReadOnly by
|
||||
// exporting the volume read-only. A provider leaving this false must
|
||||
// refuse a ReadOnly request, never hand back a writable export instead.
|
||||
ReadOnlyPublish bool `json:"read_only_publish"`
|
||||
|
||||
// VolumeEnumeration advertises that ListVolumes answers with what the
|
||||
// provider actually holds. Without it the control plane's only index of
|
||||
// volumes is its own metadata, so a lost document strands the blocks.
|
||||
VolumeEnumeration bool `json:"volume_enumeration"`
|
||||
|
||||
// SnapshotEnumeration advertises that ListSnapshots answers with what the
|
||||
// provider actually holds. It is the same exposure VolumeEnumeration
|
||||
// closes, for snapshots, and is advertised separately because a provider
|
||||
// can enumerate one without the other.
|
||||
SnapshotEnumeration bool `json:"snapshot_enumeration"`
|
||||
|
||||
// OwnerRouting advertises that CreateSnapshot, CopySnapshot, ExpandVolume
|
||||
// and GetVolume are answered directly by a volume's mounting node over
|
||||
// its OwnerSubject when one exists, instead of always fanning out to the
|
||||
// spinifex-workers queue group.
|
||||
OwnerRouting bool `json:"owner_routing"`
|
||||
|
||||
// Exclusion states how strongly this provider prevents two writers from
|
||||
// holding one volume at once. It is not optional behaviour a caller may
|
||||
// ignore: a caller that assumes cluster-wide exclusion from a provider
|
||||
// that only excludes within a node will corrupt the volume.
|
||||
Exclusion ExclusionSemantics `json:"exclusion"`
|
||||
}
|
||||
|
||||
// ExclusionScope names how far a provider's single-writer guarantee reaches.
|
||||
type ExclusionScope string
|
||||
|
||||
const (
|
||||
// ExclusionScopeNone means nothing prevents two writers. Two opens of one
|
||||
// volume both succeed and both write.
|
||||
ExclusionScopeNone ExclusionScope = "none"
|
||||
|
||||
// ExclusionScopeNode means a second writer is refused on the same node and
|
||||
// nowhere else. A different node opening the same backing store is not
|
||||
// seen, so this is a guarantee about a process, not about a volume.
|
||||
ExclusionScopeNode ExclusionScope = "node"
|
||||
|
||||
// ExclusionScopeCluster means a second writer is refused wherever it runs,
|
||||
// because the claim is held somewhere both nodes must consult.
|
||||
ExclusionScopeCluster ExclusionScope = "cluster"
|
||||
)
|
||||
|
||||
// ExclusionSemantics is the single-writer guarantee, stated so a caller can
|
||||
// tell what it actually gets. Three questions have distinct answers and the
|
||||
// contract used to conflate them: how far exclusion reaches, whether a dead
|
||||
// owner's claim ever clears, and whether a writer that lost its claim is
|
||||
// actually stopped.
|
||||
type ExclusionSemantics struct {
|
||||
Scope ExclusionScope `json:"scope"`
|
||||
|
||||
// ClaimTTLSeconds is how long a claim outlives the writer holding it.
|
||||
// Zero means it never expires: a node that dies holding one strands the
|
||||
// volume until an operator clears it. Meaningless when Scope is none.
|
||||
ClaimTTLSeconds int `json:"claim_ttl_seconds,omitempty"`
|
||||
|
||||
// FencesLostClaim reports whether a writer whose claim lapsed is stopped
|
||||
// from writing, rather than merely finding out. False is the dangerous
|
||||
// case and the reason this is not a bool on its own: exclusion refuses
|
||||
// the *second* opener, fencing stops the *first* one. Without fencing, a
|
||||
// partitioned node whose claim expired keeps writing, and the claim
|
||||
// expiring is precisely what lets a second writer in beside it.
|
||||
FencesLostClaim bool `json:"fences_lost_claim"`
|
||||
}
|
||||
|
||||
// SingleWriter reports whether opening a volume twice anywhere is refused.
|
||||
// Callers branch on this rather than comparing Scope, so adding a scope later
|
||||
// does not need every call site found again.
|
||||
func (e ExclusionSemantics) SingleWriter() bool {
|
||||
return e.Scope == ExclusionScopeCluster
|
||||
}
|
||||
|
||||
type GetCapabilitiesRequest struct{ Versioned }
|
||||
|
||||
type GetCapabilitiesResponse struct {
|
||||
Versioned
|
||||
|
||||
Capabilities Capabilities `json:"capabilities"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// CapacityRange mirrors CSI's capacity-range semantics. RequiredBytes is the
|
||||
// minimum acceptable capacity; LimitBytes, when non-zero, is the maximum.
|
||||
type CapacityRange struct {
|
||||
RequiredBytes int64 `json:"required_bytes"`
|
||||
LimitBytes int64 `json:"limit_bytes,omitempty"`
|
||||
}
|
||||
|
||||
type VolumeState string
|
||||
|
||||
const (
|
||||
VolumeStateAvailable VolumeState = "available"
|
||||
VolumeStateInUse VolumeState = "in-use"
|
||||
)
|
||||
|
||||
// Volume contains only facts the control plane can rely on across providers.
|
||||
// Handle and ProviderData are opaque and must be passed back uninterpreted.
|
||||
type Volume struct {
|
||||
ID string `json:"id"`
|
||||
CapacityBytes int64 `json:"capacity_bytes"`
|
||||
State VolumeState `json:"state"`
|
||||
Handle string `json:"handle"`
|
||||
AvailabilityZone string `json:"availability_zone,omitempty"`
|
||||
ProviderData json.RawMessage `json:"provider_data,omitempty"`
|
||||
}
|
||||
|
||||
// MaxSeedBytes bounds CreateVolumeRequest.SeedData. JSON encodes a []byte as
|
||||
// base64, inflating it by 4/3, so this leaves the encoded request comfortably
|
||||
// inside the 1MB NATS max_payload the cluster runs with.
|
||||
const MaxSeedBytes = 640 * 1024
|
||||
|
||||
type CreateVolumeRequest struct {
|
||||
Versioned
|
||||
|
||||
VolumeID string `json:"volume_id"`
|
||||
CapacityRange CapacityRange `json:"capacity_range"`
|
||||
AvailabilityZone string `json:"availability_zone,omitempty"`
|
||||
SourceSnapshotID string `json:"source_snapshot_id,omitempty"`
|
||||
Parameters json.RawMessage `json:"parameters,omitempty"`
|
||||
|
||||
// SourceSnapshotVolumeID names the volume SourceSnapshotID was taken from,
|
||||
// matching that snapshot's Snapshot.SourceVolumeID. It is required with
|
||||
// SourceSnapshotID and meaningless without it: this is not a clone source.
|
||||
SourceSnapshotVolumeID string `json:"source_snapshot_volume_id,omitempty"`
|
||||
|
||||
// SeedData is written at offset 0 of a newly created volume. It exists so
|
||||
// the caller can supply host-local bytes, such as a firmware VARS template
|
||||
// whose layout must match the launching node, without shipping a path.
|
||||
SeedData []byte `json:"seed_data,omitempty"`
|
||||
}
|
||||
|
||||
// ValidateSeedData rejects a seed the NATS transport cannot carry, so an
|
||||
// oversized firmware template fails with an actionable error at the caller
|
||||
// rather than as a truncated or refused publish.
|
||||
func ValidateSeedData(seed []byte) error {
|
||||
if len(seed) > MaxSeedBytes {
|
||||
return fmt.Errorf("%w: seed data is %d bytes, limit is %d", ErrInvalidArgument, len(seed), MaxSeedBytes)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type CreateVolumeResponse struct {
|
||||
Versioned
|
||||
|
||||
Volume *Volume `json:"volume,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type GetVolumeRequest struct {
|
||||
Versioned
|
||||
|
||||
VolumeID string `json:"volume_id"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
}
|
||||
|
||||
type GetVolumeResponse struct {
|
||||
Versioned
|
||||
|
||||
Volume *Volume `json:"volume,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// VolumeRef identifies a volume the provider holds. It is deliberately not a
|
||||
// Volume: enumeration answers "what exists", and a provider must not have to
|
||||
// open every volume's engine to answer that.
|
||||
type VolumeRef struct {
|
||||
ID string `json:"id"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
}
|
||||
|
||||
// MaxListResults bounds one page of ListVolumes or ListSnapshots. The reply
|
||||
// rides a NATS message, so the page must fit the cluster's max_payload; a
|
||||
// VolumeRef or SnapshotRef encodes to well under 1KB, leaving headroom here.
|
||||
const MaxListResults int32 = 1000
|
||||
|
||||
// ListVolumesRequest pages through the volumes a provider holds. A MaxResults
|
||||
// above MaxListResults is clamped rather than refused: a caller asking for
|
||||
// more than fits should get a page and a token, not an error.
|
||||
type ListVolumesRequest struct {
|
||||
Versioned
|
||||
|
||||
MaxResults int32 `json:"max_results,omitempty"`
|
||||
StartingToken string `json:"starting_token,omitempty"`
|
||||
}
|
||||
|
||||
// ListVolumesResponse carries one page. An empty NextToken means the last
|
||||
// page; an empty Volumes list with no error means the provider holds none.
|
||||
type ListVolumesResponse struct {
|
||||
Versioned
|
||||
|
||||
Volumes []VolumeRef `json:"volumes,omitempty"`
|
||||
NextToken string `json:"next_token,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// PageSize is the number of refs a provider should return for this request.
|
||||
func (r ListVolumesRequest) PageSize() int32 {
|
||||
if r.MaxResults <= 0 || r.MaxResults > MaxListResults {
|
||||
return MaxListResults
|
||||
}
|
||||
return r.MaxResults
|
||||
}
|
||||
|
||||
type ExpandVolumeRequest struct {
|
||||
Versioned
|
||||
|
||||
VolumeID string `json:"volume_id"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
CapacityRange CapacityRange `json:"capacity_range"`
|
||||
}
|
||||
|
||||
type ExpandVolumeResponse struct {
|
||||
Versioned
|
||||
|
||||
Volume *Volume `json:"volume,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type DeleteVolumeRequest struct {
|
||||
Versioned
|
||||
|
||||
VolumeID string `json:"volume_id"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
}
|
||||
|
||||
type DeleteVolumeResponse struct {
|
||||
Versioned
|
||||
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type SnapshotState string
|
||||
|
||||
const (
|
||||
SnapshotStatePending SnapshotState = "pending"
|
||||
SnapshotStateCompleted SnapshotState = "completed"
|
||||
SnapshotStateError SnapshotState = "error"
|
||||
)
|
||||
|
||||
type Snapshot struct {
|
||||
ID string `json:"id"`
|
||||
SourceVolumeID string `json:"source_volume_id"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
State SnapshotState `json:"state"`
|
||||
Handle string `json:"handle"`
|
||||
}
|
||||
|
||||
type CreateSnapshotRequest struct {
|
||||
Versioned
|
||||
|
||||
SnapshotID string `json:"snapshot_id"`
|
||||
VolumeID string `json:"volume_id"`
|
||||
VolumeHandle string `json:"volume_handle,omitempty"`
|
||||
}
|
||||
|
||||
// CreateSnapshotResponse is both the immediate accepted response and the
|
||||
// completion event. Pending responses carry a completion subject; completed
|
||||
// responses additionally carry the provider-neutral Snapshot.
|
||||
type CreateSnapshotResponse struct {
|
||||
Versioned
|
||||
|
||||
OperationID string `json:"operation_id"`
|
||||
CompletionSubject string `json:"completion_subject,omitempty"`
|
||||
Snapshot *Snapshot `json:"snapshot,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type DeleteSnapshotRequest struct {
|
||||
Versioned
|
||||
|
||||
SnapshotID string `json:"snapshot_id"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
}
|
||||
|
||||
type DeleteSnapshotResponse struct {
|
||||
Versioned
|
||||
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// CopySnapshotRequest duplicates SourceSnapshotID under DestinationSnapshotID
|
||||
// as a second, independently addressable snapshot over the same frozen data.
|
||||
// VolumeID is the snapshot's own source volume, not a new source: the caller
|
||||
// already knows it (it is the same VolumeID the original CreateSnapshotRequest
|
||||
// carried), so the provider is not asked to resolve snapshot ownership itself.
|
||||
type CopySnapshotRequest struct {
|
||||
Versioned
|
||||
|
||||
SourceSnapshotID string `json:"source_snapshot_id"`
|
||||
DestinationSnapshotID string `json:"destination_snapshot_id"`
|
||||
VolumeID string `json:"volume_id"`
|
||||
}
|
||||
|
||||
type CopySnapshotResponse struct {
|
||||
Versioned
|
||||
|
||||
Snapshot *Snapshot `json:"snapshot,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// SnapshotRef identifies a snapshot the provider holds. Like VolumeRef it is
|
||||
// deliberately not a Snapshot: enumeration answers "what exists", and reading
|
||||
// a snapshot's size or state can cost far more than listing it.
|
||||
//
|
||||
// SourceVolumeID is carried because reconciliation needs it: a snapshot found
|
||||
// with no metadata document is only actionable once its volume is known.
|
||||
type SnapshotRef struct {
|
||||
ID string `json:"id"`
|
||||
SourceVolumeID string `json:"source_volume_id,omitempty"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
}
|
||||
|
||||
// ListSnapshotsRequest pages through the snapshots a provider holds. It is
|
||||
// paged and clamped exactly as ListVolumesRequest is.
|
||||
type ListSnapshotsRequest struct {
|
||||
Versioned
|
||||
|
||||
MaxResults int32 `json:"max_results,omitempty"`
|
||||
StartingToken string `json:"starting_token,omitempty"`
|
||||
}
|
||||
|
||||
// ListSnapshotsResponse carries one page. An empty NextToken means the last
|
||||
// page; an empty Snapshots list with no error means the provider holds none.
|
||||
type ListSnapshotsResponse struct {
|
||||
Versioned
|
||||
|
||||
Snapshots []SnapshotRef `json:"snapshots,omitempty"`
|
||||
NextToken string `json:"next_token,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// PageSize is the number of refs a provider should return for this request.
|
||||
func (r ListSnapshotsRequest) PageSize() int32 {
|
||||
if r.MaxResults <= 0 || r.MaxResults > MaxListResults {
|
||||
return MaxListResults
|
||||
}
|
||||
return r.MaxResults
|
||||
}
|
||||
|
||||
type PublishVolumeRequest struct {
|
||||
Versioned
|
||||
|
||||
VolumeID string `json:"volume_id"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
NodeID string `json:"node_id"`
|
||||
ReadOnly bool `json:"read_only,omitempty"`
|
||||
}
|
||||
|
||||
type PublishedVolume struct {
|
||||
VolumeID string `json:"volume_id"`
|
||||
NodeID string `json:"node_id"`
|
||||
NBDURI string `json:"nbd_uri"`
|
||||
}
|
||||
|
||||
type PublishVolumeResponse struct {
|
||||
Versioned
|
||||
|
||||
Published *PublishedVolume `json:"published,omitempty"`
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type UnpublishVolumeRequest struct {
|
||||
Versioned
|
||||
|
||||
VolumeID string `json:"volume_id"`
|
||||
Handle string `json:"handle,omitempty"`
|
||||
NodeID string `json:"node_id"`
|
||||
}
|
||||
|
||||
type UnpublishVolumeResponse struct {
|
||||
Versioned
|
||||
|
||||
Error *ProviderError `json:"error,omitempty"`
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
const tracerName = "github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
|
||||
// spanNamePrefix puts provider spans in the same namespace as the ebs.mount
|
||||
// and ebs.unmount spans the legacy path emits.
|
||||
const spanNamePrefix = "ebs."
|
||||
|
||||
const (
|
||||
attrSubject = "ebs.provider.subject"
|
||||
attrVolumeID = "ebs.provider.volume_id"
|
||||
attrNodeID = "ebs.provider.node_id"
|
||||
attrOwnerRouted = "ebs.provider.owner_routed"
|
||||
attrErrorCode = "ebs.provider.error_code"
|
||||
)
|
||||
|
||||
// VerbUnknown is the verb reported for a subject that is not part of the
|
||||
// contract. Unrecognised subjects collapse to it rather than becoming span
|
||||
// names of their own, which is what keeps the name set bounded.
|
||||
const VerbUnknown = "unknown"
|
||||
|
||||
const (
|
||||
verbCapabilities = "capabilities"
|
||||
verbVolumeCreate = "volume.create"
|
||||
verbVolumeList = "volume.list"
|
||||
verbVolumeDelete = "volume.delete"
|
||||
verbVolumePublish = "volume.publish"
|
||||
verbVolumeUnpublish = "volume.unpublish"
|
||||
verbSnapshotDelete = "snapshot.delete"
|
||||
verbSnapshotList = "snapshot.list"
|
||||
)
|
||||
|
||||
// subjectVerbs maps every fixed subject to its verb. The dynamic subject
|
||||
// families (owner-routed, snapshot create, per-node mount) are handled by
|
||||
// subjectTarget, which strips the ID token before naming the verb.
|
||||
var subjectVerbs = map[string]string{
|
||||
CapabilitiesSubject: verbCapabilities,
|
||||
CreateVolumeSubject: verbVolumeCreate,
|
||||
GetVolumeSubject: verbVolumeDescribe,
|
||||
ListVolumesSubject: verbVolumeList,
|
||||
ExpandVolumeSubject: verbVolumeExpand,
|
||||
DeleteVolumeSubject: verbVolumeDelete,
|
||||
DeleteSnapshotSubject: verbSnapshotDelete,
|
||||
CopySnapshotSubject: verbSnapshotCopy,
|
||||
ListSnapshotsSubject: verbSnapshotList,
|
||||
}
|
||||
|
||||
// ownerVerbs is the closed set OwnerSubject can build. Parsed subjects are
|
||||
// checked against it so a malformed subject cannot invent a verb.
|
||||
var ownerVerbs = map[string]bool{
|
||||
verbSnapshotCreate: true,
|
||||
verbSnapshotCopy: true,
|
||||
verbVolumeExpand: true,
|
||||
verbVolumeDescribe: true,
|
||||
}
|
||||
|
||||
// subjectTarget describes what a subject addresses. The verb is drawn from a
|
||||
// closed set and is safe to use as a span name; volumeID and nodeID are
|
||||
// unbounded and belong in attributes only.
|
||||
type subjectTarget struct {
|
||||
verb string
|
||||
volumeID string
|
||||
nodeID string
|
||||
ownerRouted bool
|
||||
}
|
||||
|
||||
// SubjectVerb reports the operation subject names, with any embedded volume
|
||||
// or node ID removed. A subject outside the contract reports VerbUnknown.
|
||||
func SubjectVerb(subject string) string {
|
||||
return parseSubject(subject).verb
|
||||
}
|
||||
|
||||
func parseSubject(subject string) subjectTarget {
|
||||
if verb, ok := subjectVerbs[subject]; ok {
|
||||
return subjectTarget{verb: verb}
|
||||
}
|
||||
if volumeID, verb, ok := ParseOwnerSubject(subject); ok && ownerVerbs[verb] {
|
||||
return subjectTarget{verb: verb, volumeID: volumeID, ownerRouted: true}
|
||||
}
|
||||
if volumeID, ok := strings.CutPrefix(subject, SnapshotCreateSubjectPrefix); ok && validateSubjectToken(volumeID) == nil {
|
||||
return subjectTarget{verb: verbSnapshotCreate, volumeID: volumeID}
|
||||
}
|
||||
rest, ok := strings.CutPrefix(subject, subjectPrefix)
|
||||
if !ok {
|
||||
return subjectTarget{verb: VerbUnknown}
|
||||
}
|
||||
nodeID, suffix, ok := strings.Cut(rest, ".")
|
||||
if !ok || validateSubjectToken(nodeID) != nil {
|
||||
return subjectTarget{verb: VerbUnknown}
|
||||
}
|
||||
switch suffix {
|
||||
case "mount":
|
||||
return subjectTarget{verb: verbVolumePublish, nodeID: nodeID}
|
||||
case "unmount":
|
||||
return subjectTarget{verb: verbVolumeUnpublish, nodeID: nodeID}
|
||||
default:
|
||||
return subjectTarget{verb: VerbUnknown}
|
||||
}
|
||||
}
|
||||
|
||||
func (t subjectTarget) attributes(subject string) []attribute.KeyValue {
|
||||
attrs := []attribute.KeyValue{
|
||||
attribute.String("messaging.system", "nats"),
|
||||
attribute.String(attrSubject, subject),
|
||||
}
|
||||
if t.volumeID != "" {
|
||||
attrs = append(attrs, attribute.String(attrVolumeID, t.volumeID))
|
||||
}
|
||||
if t.nodeID != "" {
|
||||
attrs = append(attrs, attribute.String(attrNodeID, t.nodeID))
|
||||
}
|
||||
if t.ownerRouted {
|
||||
attrs = append(attrs, attribute.Bool(attrOwnerRouted, true))
|
||||
}
|
||||
return attrs
|
||||
}
|
||||
|
||||
var _ propagation.TextMapCarrier = (*headerCarrier)(nil)
|
||||
|
||||
// headerCarrier adapts nats.Header to the OTel carrier. It is a copy of the
|
||||
// one in spinifex/utils rather than an import: this package is the wire
|
||||
// contract and must not depend on the control plane's helper tree.
|
||||
type headerCarrier nats.Header
|
||||
|
||||
func (c headerCarrier) Get(key string) string { return nats.Header(c).Get(key) }
|
||||
func (c headerCarrier) Set(key, value string) { nats.Header(c).Set(key, value) }
|
||||
func (c headerCarrier) Keys() []string {
|
||||
keys := make([]string, 0, len(c))
|
||||
for key := range c {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
// startClientSpan opens a span for an outbound request and injects the trace
|
||||
// context into hdr so the serving provider joins the same trace.
|
||||
func startClientSpan(ctx context.Context, subject string, hdr nats.Header) (context.Context, trace.Span) {
|
||||
target := parseSubject(subject)
|
||||
ctx, span := otel.Tracer(tracerName).Start(ctx, spanNamePrefix+target.verb,
|
||||
trace.WithSpanKind(trace.SpanKindClient),
|
||||
trace.WithAttributes(target.attributes(subject)...))
|
||||
otel.GetTextMapPropagator().Inject(ctx, headerCarrier(hdr))
|
||||
return ctx, span
|
||||
}
|
||||
|
||||
// startCompletionSpan covers the wait for an asynchronous completion publish.
|
||||
// A snapshot spends nearly all its time here, and the request span for the
|
||||
// accept leg ends long before it.
|
||||
func startCompletionSpan(ctx context.Context, verb, completionSubject string) (context.Context, trace.Span) {
|
||||
return otel.Tracer(tracerName).Start(ctx, spanNamePrefix+verb+".wait",
|
||||
trace.WithSpanKind(trace.SpanKindConsumer),
|
||||
trace.WithAttributes(
|
||||
attribute.String("messaging.system", "nats"),
|
||||
attribute.String(attrSubject, completionSubject),
|
||||
))
|
||||
}
|
||||
|
||||
// StartServerSpan joins the trace carried in msg's headers and opens a span
|
||||
// for the verb its subject names. Callers must End the returned span and
|
||||
// should pass the returned context into the work the handler does.
|
||||
func StartServerSpan(ctx context.Context, msg *nats.Msg) (context.Context, trace.Span) {
|
||||
target := parseSubject(msg.Subject)
|
||||
if msg.Header != nil {
|
||||
ctx = otel.GetTextMapPropagator().Extract(ctx, headerCarrier(msg.Header))
|
||||
}
|
||||
return otel.Tracer(tracerName).Start(ctx, spanNamePrefix+target.verb,
|
||||
trace.WithSpanKind(trace.SpanKindServer),
|
||||
trace.WithAttributes(target.attributes(msg.Subject)...))
|
||||
}
|
||||
|
||||
// RecordSpanError marks span failed for a transport or handler error.
|
||||
func RecordSpanError(span trace.Span, err error) {
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
span.RecordError(err)
|
||||
span.SetStatus(codes.Error, err.Error())
|
||||
}
|
||||
|
||||
// RecordProviderError tags span with the wire error code a handler is about
|
||||
// to return, so a failed operation is visible as an error in the trace and
|
||||
// not merely as a successful round trip.
|
||||
func RecordProviderError(span trace.Span, providerErr *ProviderError) {
|
||||
if providerErr == nil {
|
||||
return
|
||||
}
|
||||
span.SetAttributes(attribute.String(attrErrorCode, string(providerErr.Code)))
|
||||
span.SetStatus(codes.Error, providerErr.Message)
|
||||
}
|
||||
|
||||
// RecordResponseError tags span with the error an encoded response carries.
|
||||
// The response types share no accessor for it, so this re-reads the one field
|
||||
// rather than making every verb implement one; it runs only when recording.
|
||||
func RecordResponseError(span trace.Span, data []byte) {
|
||||
if !span.IsRecording() {
|
||||
return
|
||||
}
|
||||
var envelope struct {
|
||||
Error *ProviderError `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &envelope); err != nil {
|
||||
return
|
||||
}
|
||||
RecordProviderError(span, envelope.Error)
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
package ebsprovider
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
)
|
||||
|
||||
func TestSubjectVerb_StripsIDsFromEveryDynamicSubject(t *testing.T) {
|
||||
const volumeID = "vol-0123456789abcdef0"
|
||||
const nodeID = "casuarina"
|
||||
|
||||
ownerDescribe, err := GetVolumeOwnerSubject(volumeID)
|
||||
require.NoError(t, err)
|
||||
ownerExpand, err := ExpandVolumeOwnerSubject(volumeID)
|
||||
require.NoError(t, err)
|
||||
ownerSnapshotCreate, err := SnapshotCreateOwnerSubject(volumeID)
|
||||
require.NoError(t, err)
|
||||
ownerSnapshotCopy, err := SnapshotCopyOwnerSubject(volumeID)
|
||||
require.NoError(t, err)
|
||||
snapshotCreate, err := SnapshotSubject(volumeID)
|
||||
require.NoError(t, err)
|
||||
publish, err := PublishSubject(nodeID)
|
||||
require.NoError(t, err)
|
||||
unpublish, err := UnpublishSubject(nodeID)
|
||||
require.NoError(t, err)
|
||||
|
||||
cases := map[string]string{
|
||||
CapabilitiesSubject: verbCapabilities,
|
||||
CreateVolumeSubject: verbVolumeCreate,
|
||||
GetVolumeSubject: verbVolumeDescribe,
|
||||
ListVolumesSubject: verbVolumeList,
|
||||
ExpandVolumeSubject: verbVolumeExpand,
|
||||
DeleteVolumeSubject: verbVolumeDelete,
|
||||
DeleteSnapshotSubject: verbSnapshotDelete,
|
||||
CopySnapshotSubject: verbSnapshotCopy,
|
||||
ownerDescribe: verbVolumeDescribe,
|
||||
ownerExpand: verbVolumeExpand,
|
||||
ownerSnapshotCreate: verbSnapshotCreate,
|
||||
ownerSnapshotCopy: verbSnapshotCopy,
|
||||
snapshotCreate: verbSnapshotCreate,
|
||||
publish: verbVolumePublish,
|
||||
unpublish: verbVolumeUnpublish,
|
||||
}
|
||||
|
||||
for subject, want := range cases {
|
||||
verb := SubjectVerb(subject)
|
||||
assert.Equal(t, want, verb, "subject %q", subject)
|
||||
assert.NotContains(t, verb, volumeID, "verb for %q leaks the volume ID", subject)
|
||||
assert.NotContains(t, verb, nodeID, "verb for %q leaks the node ID", subject)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubjectVerb_UnrecognisedSubjectCollapsesToUnknown(t *testing.T) {
|
||||
for _, subject := range []string{
|
||||
"",
|
||||
"ebs.mount",
|
||||
"ebs.provider.v1.volume.invent",
|
||||
"ebs.provider.v1.owner.vol-1.invent",
|
||||
"ebs.provider.v1.node.rename",
|
||||
"ebs.provider.v2.volume.create",
|
||||
} {
|
||||
assert.Equal(t, VerbUnknown, SubjectVerb(subject), "subject %q", subject)
|
||||
}
|
||||
}
|
||||
|
||||
// TestClientSpan_NamesTheVerbNotTheSubject pins the property that makes these
|
||||
// spans aggregatable: a per-volume subject must not become a span name of its
|
||||
// own, so the volume ID belongs in an attribute.
|
||||
func TestClientSpan_NamesTheVerbNotTheSubject(t *testing.T) {
|
||||
recorder := testutil.RecordSpans(t)
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
|
||||
const volumeID = "vol-0123456789abcdef0"
|
||||
ownerSubject, err := GetVolumeOwnerSubject(volumeID)
|
||||
require.NoError(t, err)
|
||||
sub, err := conn.Subscribe(ownerSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(GetVolumeResponse{Versioned: NewVersioned(), Volume: &Volume{ID: volumeID}})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
_, err = provider.GetVolume(t.Context(), GetVolumeRequest{Versioned: NewVersioned(), VolumeID: volumeID})
|
||||
require.NoError(t, err)
|
||||
|
||||
spans := recorder.Ended()
|
||||
require.Len(t, spans, 1)
|
||||
assert.Equal(t, "ebs.volume.describe", spans[0].Name())
|
||||
assert.NotContains(t, spans[0].Name(), volumeID)
|
||||
assert.Equal(t, volumeID, testutil.SpanAttribute(t, spans[0], attrVolumeID).AsString())
|
||||
assert.Equal(t, ownerSubject, testutil.SpanAttribute(t, spans[0], attrSubject).AsString())
|
||||
assert.True(t, testutil.SpanAttribute(t, spans[0], attrOwnerRouted).AsBool())
|
||||
}
|
||||
|
||||
// TestClientSpan_PutsTraceparentOnTheWire is the half of propagation this
|
||||
// package owns. Without it the serving provider roots a trace of its own and
|
||||
// the two halves of an operation can never be joined.
|
||||
func TestClientSpan_PutsTraceparentOnTheWire(t *testing.T) {
|
||||
recorder := testutil.RecordSpans(t)
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
|
||||
received := make(chan string, 1)
|
||||
sub, err := conn.Subscribe(ListVolumesSubject, func(msg *nats.Msg) {
|
||||
received <- msg.Header.Get("traceparent")
|
||||
payload, marshalErr := json.Marshal(ListVolumesResponse{Versioned: NewVersioned()})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
_, err = provider.ListVolumes(t.Context(), ListVolumesRequest{Versioned: NewVersioned()})
|
||||
require.NoError(t, err)
|
||||
|
||||
spans := recorder.Ended()
|
||||
require.Len(t, spans, 1)
|
||||
select {
|
||||
case traceparent := <-received:
|
||||
require.NotEmpty(t, traceparent, "no traceparent header reached the provider")
|
||||
assert.Contains(t, traceparent, spans[0].SpanContext().TraceID().String())
|
||||
assert.Contains(t, traceparent, spans[0].SpanContext().SpanID().String())
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("provider never received the request")
|
||||
}
|
||||
}
|
||||
|
||||
// TestClientSpan_RecordsTheWireErrorCode covers the case a transport-only span
|
||||
// would report as success: the round trip completed, but the operation failed.
|
||||
func TestClientSpan_RecordsTheWireErrorCode(t *testing.T) {
|
||||
recorder := testutil.RecordSpans(t)
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
|
||||
sub, err := conn.Subscribe(DeleteVolumeSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(DeleteVolumeResponse{
|
||||
Versioned: NewVersioned(),
|
||||
Error: &ProviderError{Code: ErrorCodeVolumeInUse, Message: "volume vol-1 is published"},
|
||||
})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
err = provider.DeleteVolume(t.Context(), DeleteVolumeRequest{Versioned: NewVersioned(), VolumeID: "vol-1"})
|
||||
require.ErrorIs(t, err, ErrVolumeInUse)
|
||||
|
||||
spans := recorder.Ended()
|
||||
require.Len(t, spans, 1)
|
||||
assert.Equal(t, codes.Error, spans[0].Status().Code)
|
||||
assert.Equal(t, string(ErrorCodeVolumeInUse), testutil.SpanAttribute(t, spans[0], attrErrorCode).AsString())
|
||||
}
|
||||
|
||||
// TestClientSpan_OwnerFallbackRecordsBothLegs keeps the wasted owner attempt
|
||||
// visible. A fallback that showed only the leg that answered would hide the
|
||||
// cost of asking a node that holds nothing.
|
||||
func TestClientSpan_OwnerFallbackRecordsBothLegs(t *testing.T) {
|
||||
recorder := testutil.RecordSpans(t)
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
|
||||
const volumeID = "vol-0123456789abcdef0"
|
||||
sub, err := conn.Subscribe(GetVolumeSubject, func(msg *nats.Msg) {
|
||||
payload, marshalErr := json.Marshal(GetVolumeResponse{Versioned: NewVersioned(), Volume: &Volume{ID: volumeID}})
|
||||
require.NoError(t, marshalErr)
|
||||
require.NoError(t, msg.Respond(payload))
|
||||
})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = sub.Unsubscribe() })
|
||||
require.NoError(t, conn.Flush())
|
||||
|
||||
provider := NewNATSProvider(conn, time.Second)
|
||||
_, err = provider.GetVolume(t.Context(), GetVolumeRequest{Versioned: NewVersioned(), VolumeID: volumeID})
|
||||
require.NoError(t, err)
|
||||
|
||||
spans := recorder.Ended()
|
||||
require.Len(t, spans, 2, "owner attempt and queue-group fallback should each be a span")
|
||||
assert.Equal(t, "ebs.volume.describe", spans[0].Name())
|
||||
assert.True(t, testutil.SpanAttribute(t, spans[0], attrOwnerRouted).AsBool())
|
||||
assert.Equal(t, codes.Error, spans[0].Status().Code, "the unanswered owner attempt is a failure")
|
||||
|
||||
assert.Equal(t, "ebs.volume.describe", spans[1].Name())
|
||||
assert.Equal(t, GetVolumeSubject, testutil.SpanAttribute(t, spans[1], attrSubject).AsString())
|
||||
assert.Equal(t, codes.Unset, spans[1].Status().Code)
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package handlers_ec2_image
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// putSourceVolumeAMI registers an AMI document plus, when volumeID is set, the
|
||||
// snapshot metadata recording which volume the snapshot was taken from.
|
||||
func putSourceVolumeAMI(t *testing.T, svc *ImageServiceImpl, store *objectstore.MemoryObjectStore, ami ebsmetadata.AMI, volumeID string) {
|
||||
t.Helper()
|
||||
require.NoError(t, svc.MetadataStore().PutAMI(context.Background(), ami))
|
||||
if volumeID == "" {
|
||||
return
|
||||
}
|
||||
require.NoError(t, handlers_ec2_snapshot.WriteSnapshotConfig(store, testBucket, ami.SnapshotID,
|
||||
&handlers_ec2_snapshot.SnapshotConfig{SnapshotID: ami.SnapshotID, VolumeID: volumeID}))
|
||||
}
|
||||
|
||||
// TestGetAMISourceVolumeID_ReadsSnapshotMetadata locks the normal case: the
|
||||
// source volume comes from the snapshot's own metadata.json.
|
||||
func TestGetAMISourceVolumeID_ReadsSnapshotMetadata(t *testing.T) {
|
||||
svc, store := setupProviderImageService(t)
|
||||
putSourceVolumeAMI(t, svc, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-src01", SnapshotID: "snap-src01", ImageOwnerAlias: testAccountID,
|
||||
}, "vol-origin")
|
||||
|
||||
got, err := svc.GetAMISourceVolumeID(context.Background(), "ami-src01")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "vol-origin", got)
|
||||
}
|
||||
|
||||
// TestGetAMISourceVolumeID_BundledSystemAMI locks the fallback for bundled
|
||||
// system AMIs, whose snapshot is named after the AMI and carries no metadata.json.
|
||||
func TestGetAMISourceVolumeID_BundledSystemAMI(t *testing.T) {
|
||||
svc, store := setupProviderImageService(t)
|
||||
putSourceVolumeAMI(t, svc, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-sys01", SnapshotID: "snap-ami-sys01", ImageOwnerAlias: "system",
|
||||
}, "")
|
||||
|
||||
got, err := svc.GetAMISourceVolumeID(context.Background(), "ami-sys01")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ami-sys01", got, "the bundled AMI's snapshot reads chunks from a volume named after the AMI")
|
||||
}
|
||||
|
||||
// TestGetAMISourceVolumeID_AccountAMIMissingSnapshotMetadata locks that the
|
||||
// bundled fallback does not apply to account-owned AMIs.
|
||||
func TestGetAMISourceVolumeID_AccountAMIMissingSnapshotMetadata(t *testing.T) {
|
||||
svc, store := setupProviderImageService(t)
|
||||
putSourceVolumeAMI(t, svc, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-acct01", SnapshotID: "snap-acct01", ImageOwnerAlias: testAccountID,
|
||||
}, "")
|
||||
|
||||
_, err := svc.GetAMISourceVolumeID(context.Background(), "ami-acct01")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidSnapshotNotFound, err.Error())
|
||||
}
|
||||
|
||||
// TestGetAMISourceVolumeID_AMIWithoutSnapshot locks that an AMI carrying no
|
||||
// snapshot reference is reported as not found rather than resolving to "".
|
||||
func TestGetAMISourceVolumeID_AMIWithoutSnapshot(t *testing.T) {
|
||||
svc, store := setupProviderImageService(t)
|
||||
putSourceVolumeAMI(t, svc, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-nosnap", ImageOwnerAlias: testAccountID,
|
||||
}, "")
|
||||
|
||||
_, err := svc.GetAMISourceVolumeID(context.Background(), "ami-nosnap")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
|
||||
}
|
||||
|
||||
// TestGetAMISourceVolumeID_UnknownAMI locks the missing-AMI mapping.
|
||||
func TestGetAMISourceVolumeID_UnknownAMI(t *testing.T) {
|
||||
svc, _ := setupProviderImageService(t)
|
||||
|
||||
_, err := svc.GetAMISourceVolumeID(context.Background(), "ami-nothere")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
|
||||
}
|
||||
|
||||
// TestGetAMISourceVolumeID_EmptySnapshotSourceVolume locks that corrupt
|
||||
// snapshot metadata naming no source volume fails instead of returning "".
|
||||
func TestGetAMISourceVolumeID_EmptySnapshotSourceVolume(t *testing.T) {
|
||||
svc, store := setupProviderImageService(t)
|
||||
require.NoError(t, svc.MetadataStore().PutAMI(context.Background(), ebsmetadata.AMI{
|
||||
ImageID: "ami-empty", SnapshotID: "snap-empty", ImageOwnerAlias: testAccountID,
|
||||
}))
|
||||
require.NoError(t, handlers_ec2_snapshot.WriteSnapshotConfig(store, testBucket, "snap-empty",
|
||||
&handlers_ec2_snapshot.SnapshotConfig{SnapshotID: "snap-empty"}))
|
||||
|
||||
_, err := svc.GetAMISourceVolumeID(context.Background(), "ami-empty")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidSnapshotNotFound, err.Error())
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package handlers_ec2_image
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// strictDeleteStore answers a delete of an absent key with NoSuchKey, which is
|
||||
// what Predastore does. MemoryObjectStore silently succeeds instead, so a
|
||||
// handler that treats a missing key as a failure passes every test and still
|
||||
// breaks in production.
|
||||
type strictDeleteStore struct {
|
||||
*objectstore.MemoryObjectStore
|
||||
}
|
||||
|
||||
func (s strictDeleteStore) DeleteObject(ctx context.Context, input *awss3.DeleteObjectInput) (*awss3.DeleteObjectOutput, error) {
|
||||
if _, err := s.HeadObject(ctx, &awss3.HeadObjectInput{Bucket: input.Bucket, Key: input.Key}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.MemoryObjectStore.DeleteObject(ctx, input)
|
||||
}
|
||||
|
||||
// TestDeregisterImage_NoLegacyConfig covers every AMI created since the
|
||||
// metadata store landed: they have no {imageID}/config.json, so deleting one
|
||||
// answers NoSuchKey. Treating that as a failure returned InternalError after
|
||||
// the AMI document was already gone, and a retrying caller then saw
|
||||
// InvalidAMIID.NotFound for an image it had just been told it could not delete.
|
||||
func TestDeregisterImage_NoLegacyConfig(t *testing.T) {
|
||||
mem := objectstore.NewMemoryObjectStore()
|
||||
svc := NewImageServiceImplWithStore(strictDeleteStore{MemoryObjectStore: mem}, testBucket)
|
||||
|
||||
amiID := "ami-nolegacy001"
|
||||
createTestAMIConfigWithOwner(t, mem, amiID, "no-legacy-config", testAccountID)
|
||||
|
||||
out, err := svc.DeregisterImage(context.Background(), &ec2.DeregisterImageInput{ImageId: aws.String(amiID)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, out)
|
||||
|
||||
_, getErr := mem.GetObject(context.Background(), &awss3.GetObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(amiDocumentKey(t, amiID)),
|
||||
})
|
||||
require.Error(t, getErr)
|
||||
assert.True(t, objectstore.IsNoSuchKeyError(getErr), "the AMI document must still be deleted")
|
||||
}
|
||||
@@ -42,7 +42,8 @@ func setupDrainImageService(t *testing.T) (*ImageServiceImpl, *objectstore.Memor
|
||||
Bucket: testBucket,
|
||||
},
|
||||
}
|
||||
return NewImageServiceImplWithConfig(cfg, store, nc), store, nc
|
||||
svc := NewImageServiceImplWithConfig(cfg, store, nc)
|
||||
return svc, store, nc
|
||||
}
|
||||
|
||||
func seedImageVolumeAttachment(t *testing.T, store *objectstore.MemoryObjectStore, volumeID, state, instanceID string) {
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
package handlers_ec2_image
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const providerSnapshotAZ = "ap-southeast-2a"
|
||||
|
||||
// setupProviderSnapshotImageService builds an image service whose only
|
||||
// storage dependency for the snapshot paths is the provider. Predastore
|
||||
// points at a closed port so any accidental fall-through to the embedded
|
||||
// engine fails loudly (connection refused) instead of silently passing.
|
||||
func setupProviderSnapshotImageService(t *testing.T) (*ImageServiceImpl, *objectstore.MemoryObjectStore, ebsprovider.EBSProvider) {
|
||||
t.Helper()
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
cfg := &config.Config{
|
||||
WalDir: t.TempDir(),
|
||||
AZ: providerSnapshotAZ,
|
||||
Predastore: config.PredastoreConfig{
|
||||
Host: "127.0.0.1:1",
|
||||
Bucket: testBucket,
|
||||
Region: "us-east-1",
|
||||
},
|
||||
}
|
||||
svc := NewImageServiceImplWithConfig(cfg, store, nil)
|
||||
svc.SetEBSProvider(provider)
|
||||
return svc, store, provider
|
||||
}
|
||||
|
||||
// seedProviderVolume creates a provider-backed volume plus its ebsmetadata
|
||||
// document, wiring the document's ProviderHandle to the volume's real handle
|
||||
// so the fixture cannot drift from what the provider actually issued.
|
||||
func seedProviderVolume(t *testing.T, svc *ImageServiceImpl, provider ebsprovider.EBSProvider, volumeID string, sizeGiB uint64) ebsmetadata.Volume {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
sizeBytes := int64(sizeGiB) * 1024 * 1024 * 1024
|
||||
created, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: sizeBytes},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
doc := ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testAccountID,
|
||||
CapacityGiB: sizeGiB,
|
||||
State: "available",
|
||||
CreatedAt: time.Now(),
|
||||
AvailabilityZone: providerSnapshotAZ,
|
||||
VolumeType: "gp3",
|
||||
ProviderHandle: created.Handle,
|
||||
}
|
||||
require.NoError(t, svc.MetadataStore().PutVolume(ctx, doc))
|
||||
return doc
|
||||
}
|
||||
|
||||
// TestGetVolumeMetadata_Provider_ResolvesProviderCreatedVolume is the
|
||||
// regression guard for the whole change: a provider-created volume has no
|
||||
// {volumeID}/config.json, so reading the legacy key made every CreateImage
|
||||
// from such a volume fail with NoSuchKey, the same bug AttachVolume had
|
||||
// before its volume-config read was fixed.
|
||||
func TestGetVolumeMetadata_Provider_ResolvesProviderCreatedVolume(t *testing.T) {
|
||||
svc, _, provider := setupProviderSnapshotImageService(t)
|
||||
doc := seedProviderVolume(t, svc, provider, "vol-provider001", 10)
|
||||
|
||||
meta, err := svc.getVolumeMetadata(context.Background(), doc.VolumeID)
|
||||
require.NoError(t, err, "a provider-created volume with no config.json must resolve")
|
||||
assert.Equal(t, doc.VolumeID, meta.VolumeID)
|
||||
assert.Equal(t, testAccountID, meta.TenantID)
|
||||
assert.Equal(t, uint64(10), meta.CapacityGiB)
|
||||
assert.Equal(t, "available", meta.State)
|
||||
assert.Equal(t, providerSnapshotAZ, meta.AvailabilityZone)
|
||||
}
|
||||
|
||||
// TestGetVolumeMetadata_Provider_UnknownVolumeIsNotFound locks that a missing
|
||||
// ebsmetadata document maps to InvalidVolume.NotFound, not a generic error.
|
||||
func TestGetVolumeMetadata_Provider_UnknownVolumeIsNotFound(t *testing.T) {
|
||||
svc, _, _ := setupProviderSnapshotImageService(t)
|
||||
|
||||
_, err := svc.getVolumeMetadata(context.Background(), "vol-does-not-exist")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidVolumeNotFound, err.Error())
|
||||
}
|
||||
|
||||
// TestSnapshotStoppedVolume_Provider_CreatesThroughProvider locks that a
|
||||
// stopped-volume snapshot under a provider is created via provider.CreateSnapshot
|
||||
// and never falls through to building an embedded engine — the Predastore host
|
||||
// is a closed port, so any fallback would fail with connection refused.
|
||||
func TestSnapshotStoppedVolume_Provider_CreatesThroughProvider(t *testing.T) {
|
||||
svc, _, provider := setupProviderSnapshotImageService(t)
|
||||
doc := seedProviderVolume(t, svc, provider, "vol-stopped001", 8)
|
||||
|
||||
err := svc.snapshotStoppedVolume(doc.VolumeID, "snap-stopped001")
|
||||
require.NoError(t, err, "the provider path must not fall through to the embedded engine against the closed Predastore port")
|
||||
|
||||
// CreateSnapshot is idempotent on (SnapshotID, VolumeID): re-issuing it
|
||||
// returns the existing snapshot rather than creating a second one, which
|
||||
// confirms the snapshot genuinely exists in the provider (sourced from
|
||||
// the right volume) instead of the call having silently no-opped.
|
||||
snap, err := provider.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: "snap-stopped001", VolumeID: doc.VolumeID,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, doc.VolumeID, snap.SourceVolumeID)
|
||||
}
|
||||
|
||||
// TestSnapshotRunningVolume_Provider_PassesProviderHandleAsVolumeHandle locks
|
||||
// that the ebsmetadata document's ProviderHandle is threaded through as
|
||||
// CreateSnapshotRequest.VolumeHandle. A wrong handle must be rejected by the
|
||||
// provider (MemoryProvider treats an empty VolumeHandle as "skip the check",
|
||||
// so only a mismatched non-empty value proves the real value is forwarded).
|
||||
func TestSnapshotRunningVolume_Provider_PassesProviderHandleAsVolumeHandle(t *testing.T) {
|
||||
svc, _, provider := setupProviderSnapshotImageService(t)
|
||||
doc := seedProviderVolume(t, svc, provider, "vol-running001", 8)
|
||||
|
||||
err := svc.snapshotRunningVolume(doc.VolumeID, "snap-running001", testAccountID)
|
||||
require.NoError(t, err, "the correct provider handle must let CreateSnapshot succeed")
|
||||
|
||||
doc.ProviderHandle = "wrong-handle"
|
||||
require.NoError(t, svc.MetadataStore().PutVolume(context.Background(), doc))
|
||||
|
||||
err = svc.snapshotRunningVolume(doc.VolumeID, "snap-running002", testAccountID)
|
||||
require.Error(t, err, "a mismatched provider handle must be rejected, proving the value came from the document")
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
}
|
||||
|
||||
// TestCreateImageFromInstance_Provider_UsesEbsMetadataVolumeSize is the
|
||||
// end-to-end case: under a provider, CreateImageFromInstance's AMI.VolumeSizeGiB
|
||||
// must come from the ebsmetadata document, not a legacy config.json (which a
|
||||
// provider-created volume never has).
|
||||
func TestCreateImageFromInstance_Provider_UsesEbsMetadataVolumeSize(t *testing.T) {
|
||||
svc, _, provider := setupProviderSnapshotImageService(t)
|
||||
doc := seedProviderVolume(t, svc, provider, "vol-e2e001", 20)
|
||||
|
||||
out, err := svc.CreateImageFromInstance(CreateImageParams{
|
||||
Input: &ec2.CreateImageInput{
|
||||
InstanceId: aws.String("i-e2e001"),
|
||||
Name: aws.String("provider-ami"),
|
||||
},
|
||||
RootVolumeID: doc.VolumeID,
|
||||
IsRunning: false,
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, out.ImageId)
|
||||
|
||||
ami, err := svc.MetadataStore().GetAMI(context.Background(), aws.StringValue(out.ImageId))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, uint64(20), ami.VolumeSizeGiB, "the AMI's volume size must come from the ebsmetadata document")
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package handlers_ec2_image
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// setupProviderImageService creates an image service with the EBS provider
|
||||
// injected, so GetAMIConfig/putAMIConfig/DescribeImages take the provider
|
||||
// branch instead of the legacy embedded path.
|
||||
func setupProviderImageService(t *testing.T) (*ImageServiceImpl, *objectstore.MemoryObjectStore) {
|
||||
svc, store := setupTestImageService(t)
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
return svc, store
|
||||
}
|
||||
|
||||
// TestGetAMIConfig_Provider_ReturnsDocument locks that under the provider,
|
||||
// GetAMIConfig reads the ebsmetadata document rather than legacy config.json.
|
||||
func TestGetAMIConfig_Provider_ReturnsDocument(t *testing.T) {
|
||||
svc, _ := setupProviderImageService(t)
|
||||
|
||||
ami := ebsmetadata.AMI{
|
||||
ImageID: "ami-doc001", Name: "provider-ami", Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX", Virtualization: "hvm", RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 12, ImageOwnerAlias: testAccountID,
|
||||
}
|
||||
require.NoError(t, svc.MetadataStore().PutAMI(context.Background(), ami))
|
||||
|
||||
got, err := svc.GetAMIConfig(context.Background(), "ami-doc001")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, ami.Name, got.Name)
|
||||
assert.Equal(t, ami.VolumeSizeGiB, got.VolumeSizeGiB)
|
||||
assert.Equal(t, ami.ImageOwnerAlias, got.ImageOwnerAlias)
|
||||
}
|
||||
|
||||
// TestGetAMIConfig_Embedded_ReturnsConvertedLegacyValue locks that the
|
||||
// embedded path still reads legacy config.json, converted into the same
|
||||
// ebsmetadata.AMI shape the provider path returns.
|
||||
func TestGetAMIConfig_Embedded_ReturnsConvertedLegacyValue(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
createTestAMIConfigWithName(t, store, "ami-legacy001", "legacy-ami")
|
||||
|
||||
got, err := svc.GetAMIConfig(context.Background(), "ami-legacy001")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "legacy-ami", got.Name)
|
||||
assert.Equal(t, testAccountID, got.ImageOwnerAlias)
|
||||
assert.Equal(t, uint64(8), got.VolumeSizeGiB)
|
||||
}
|
||||
|
||||
// TestPutAMIConfig_Provider_WritesDocumentNotConfigJSON locks the same
|
||||
// single-source-of-truth contract as
|
||||
// TestUpdateVolumeState_WritesStateJSONNotConfig in the volume package: under
|
||||
// the provider, AMI metadata lives only in the ebsmetadata document, never in
|
||||
// {imageID}/config.json.
|
||||
func TestPutAMIConfig_Provider_WritesDocumentNotConfigJSON(t *testing.T) {
|
||||
svc, store := setupProviderImageService(t)
|
||||
|
||||
meta := ebsmetadata.AMI{
|
||||
ImageID: "ami-nocfg001", Name: "provider-write", Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX", Virtualization: "hvm", RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 10, ImageOwnerAlias: testAccountID,
|
||||
}
|
||||
require.NoError(t, svc.putAMIConfig(context.Background(), meta.ImageID, meta))
|
||||
|
||||
_, err := store.GetObject(context.Background(), &awss3.GetObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(meta.ImageID + "/config.json"),
|
||||
})
|
||||
require.Error(t, err)
|
||||
assert.True(t, objectstore.IsNoSuchKeyError(err), "putAMIConfig must not write config.json under the provider")
|
||||
|
||||
got, err := svc.GetAMIConfig(context.Background(), meta.ImageID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, meta.Name, got.Name)
|
||||
}
|
||||
|
||||
// TestDescribeImages_Provider_ListsViaListAMIs_DocumentOnlyAMI locks that
|
||||
// DescribeImages under the provider enumerates via Store.ListAMIs and
|
||||
// surfaces an AMI that exists only as an ebsmetadata document.
|
||||
func TestDescribeImages_Provider_ListsViaListAMIs_DocumentOnlyAMI(t *testing.T) {
|
||||
svc, _ := setupProviderImageService(t)
|
||||
|
||||
ami := ebsmetadata.AMI{
|
||||
ImageID: "ami-listonly001", Name: "list-only", Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX", Virtualization: "hvm", RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8, ImageOwnerAlias: testAccountID, CreationDate: time.Now(), State: "available",
|
||||
}
|
||||
require.NoError(t, svc.MetadataStore().PutAMI(context.Background(), ami))
|
||||
|
||||
out, err := svc.DescribeImages(context.Background(), &ec2.DescribeImagesInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, out.Images, 1, "an AMI that exists only as an ebsmetadata document must be visible under the provider")
|
||||
assert.Equal(t, "ami-listonly001", aws.StringValue(out.Images[0].ImageId))
|
||||
assert.Equal(t, "list-only", aws.StringValue(out.Images[0].Name))
|
||||
}
|
||||
@@ -1,12 +1,9 @@
|
||||
package handlers_ec2_image
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"strconv"
|
||||
@@ -18,12 +15,12 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/filterutil"
|
||||
handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
vbs3 "github.com/mulgadc/viperblock/viperblock/backends/s3"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
|
||||
@@ -45,6 +42,26 @@ type ImageServiceImpl struct {
|
||||
store objectstore.ObjectStore
|
||||
bucketName string
|
||||
natsConn *nats.Conn
|
||||
metadata *ebsmetadata.Store
|
||||
provider ebsprovider.EBSProvider
|
||||
}
|
||||
|
||||
// SetEBSProvider injects the provider boundary used by the control plane.
|
||||
// Legacy AMI/snapshot paths remain in place until their metadata migration is complete.
|
||||
func (s *ImageServiceImpl) SetEBSProvider(provider ebsprovider.EBSProvider) {
|
||||
s.provider = provider
|
||||
}
|
||||
|
||||
// EBSProvider returns the injected provider boundary, or nil on the legacy
|
||||
// embedded-engine path. Primarily for composition-root tests to observe wiring.
|
||||
func (s *ImageServiceImpl) EBSProvider() ebsprovider.EBSProvider {
|
||||
return s.provider
|
||||
}
|
||||
|
||||
// MetadataStore returns the control-plane metadata store. Primarily for
|
||||
// composition-root tests to observe wiring.
|
||||
func (s *ImageServiceImpl) MetadataStore() *ebsmetadata.Store {
|
||||
return s.metadata
|
||||
}
|
||||
|
||||
// NewImageServiceImpl creates a new daemon-side image service. natsConn is used
|
||||
@@ -63,6 +80,7 @@ func NewImageServiceImpl(cfg *config.Config, natsConn *nats.Conn) *ImageServiceI
|
||||
store: store,
|
||||
bucketName: cfg.Predastore.Bucket,
|
||||
natsConn: natsConn,
|
||||
metadata: ebsmetadata.NewStore(store, cfg.Predastore.Bucket),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -71,6 +89,7 @@ func NewImageServiceImplWithStore(store objectstore.ObjectStore, bucketName stri
|
||||
return &ImageServiceImpl{
|
||||
store: store,
|
||||
bucketName: bucketName,
|
||||
metadata: ebsmetadata.NewStore(store, bucketName),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -83,6 +102,7 @@ func NewImageServiceImplWithConfig(cfg *config.Config, store objectstore.ObjectS
|
||||
store: store,
|
||||
bucketName: cfg.Predastore.Bucket,
|
||||
natsConn: natsConn,
|
||||
metadata: ebsmetadata.NewStore(store, cfg.Predastore.Bucket),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -114,108 +134,26 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
return nil, errors.New(awserrors.ErrorInvalidParameterValue)
|
||||
}
|
||||
|
||||
// List all prefixes in the bucket (AMIs are stored as ami-<id>/ directories)
|
||||
result, err := s.store.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Delimiter: aws.String("/"),
|
||||
})
|
||||
return s.describeImages(ctx, input, accountID, parsedFilters)
|
||||
}
|
||||
|
||||
// describeImages enumerates AMIs via Store.ListAMIs, then filters and renders
|
||||
// each document into the AWS SDK image shape.
|
||||
func (s *ImageServiceImpl) describeImages(ctx context.Context, input *ec2.DescribeImagesInput, accountID string, parsedFilters map[string][]string) (*ec2.DescribeImagesOutput, error) {
|
||||
amis, err := s.metadata.ListAMIs(ctx)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to list S3 objects", "err", err)
|
||||
slog.ErrorContext(ctx, "DescribeImages: failed to list AMIs", "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Extract image-id filter values for early prefix skipping to avoid
|
||||
// unnecessary S3 GetObject calls on non-matching AMIs.
|
||||
var imageIDFilterValues []string
|
||||
if parsedFilters != nil {
|
||||
imageIDFilterValues = parsedFilters["image-id"]
|
||||
}
|
||||
|
||||
var images []*ec2.Image
|
||||
encryptedAtRest := s.clusterEncryptionEnabled()
|
||||
var images []*ec2.Image
|
||||
|
||||
// Counting prefixes seen against those dropped for an unreadable config.json
|
||||
// makes a mismatch with len(images) visible at INFO, so an empty result is
|
||||
// distinguishable from one where every config failed to load.
|
||||
var prefixesExamined, unreadableConfigs int
|
||||
|
||||
// Iterate over CommonPrefixes to find ami-* directories
|
||||
for _, prefix := range result.CommonPrefixes {
|
||||
if prefix.Prefix == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
prefixStr := *prefix.Prefix
|
||||
slog.InfoContext(ctx, "Processing S3 prefix", "prefix", prefixStr)
|
||||
|
||||
// Only check prefixes that match pattern: ami-<id>/
|
||||
if !strings.HasPrefix(prefixStr, "ami-") {
|
||||
continue
|
||||
}
|
||||
prefixesExamined++
|
||||
|
||||
// Early skip: if image-id filter is set, check the prefix (ami-xxx/)
|
||||
// against filter values before fetching config from S3.
|
||||
if len(imageIDFilterValues) > 0 {
|
||||
amiID := strings.TrimSuffix(prefixStr, "/")
|
||||
if !filterutil.MatchesAny(imageIDFilterValues, amiID) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
// Construct path to config.json for this AMI directory
|
||||
configKey := prefixStr + "config.json"
|
||||
|
||||
// Get the config.json file
|
||||
getResult, err := s.store.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(configKey),
|
||||
})
|
||||
if err != nil {
|
||||
if objectstore.IsNoSuchKeyError(err) {
|
||||
// No config.json means this is a legitimate non-AMI directory,
|
||||
// not a fault, so it stays at DEBUG and isn't counted below.
|
||||
slog.DebugContext(ctx, "Config file not found", "key", configKey)
|
||||
} else {
|
||||
unreadableConfigs++
|
||||
slog.WarnContext(ctx, "Failed to get config file", "key", configKey, "err", err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(getResult.Body)
|
||||
// A close failure alone doesn't drop the prefix (read below may still
|
||||
// succeed), so it's only warned about, not counted as unreadable.
|
||||
if closeErr := getResult.Body.Close(); closeErr != nil {
|
||||
slog.WarnContext(ctx, "Failed to close config body", "key", configKey, "err", closeErr)
|
||||
}
|
||||
if err != nil {
|
||||
unreadableConfigs++
|
||||
slog.WarnContext(ctx, "Failed to read config body", "key", configKey, "err", err)
|
||||
continue
|
||||
}
|
||||
|
||||
// Parse the viperblock config which contains VolumeConfig with AMIMetadata.
|
||||
// StateBody unwraps the at-rest encryption envelope (the metadata ships
|
||||
// authenticated-but-plaintext); it is a no-op for unencrypted volumes.
|
||||
var vbConfig struct {
|
||||
VolumeConfig viperblock.VolumeConfig `json:"VolumeConfig"`
|
||||
}
|
||||
if err := json.Unmarshal(viperblock.StateBody(body), &vbConfig); err != nil {
|
||||
unreadableConfigs++
|
||||
slog.WarnContext(ctx, "Failed to unmarshal config", "key", configKey, "err", err)
|
||||
continue
|
||||
}
|
||||
|
||||
amiMeta := vbConfig.VolumeConfig.AMIMetadata
|
||||
|
||||
// Skip if this is not an AMI (ImageID is empty)
|
||||
for _, amiMeta := range amis {
|
||||
if amiMeta.ImageID == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// Filter by ImageId if specified
|
||||
if len(input.ImageIds) > 0 {
|
||||
found := false
|
||||
for _, filterID := range input.ImageIds {
|
||||
@@ -229,8 +167,6 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
}
|
||||
}
|
||||
|
||||
// AMIs with a non-account-ID owner (e.g. "spinifex") are system AMIs
|
||||
// visible to all; account-ID owners are private.
|
||||
amiOwner := amiMeta.ImageOwnerAlias
|
||||
isSystemAMI := amiOwner != "" && !utils.IsAccountID(amiOwner)
|
||||
|
||||
@@ -238,7 +174,6 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
continue
|
||||
}
|
||||
|
||||
// Filter by Owner if specified
|
||||
if len(input.Owners) > 0 {
|
||||
found := false
|
||||
for _, owner := range input.Owners {
|
||||
@@ -247,14 +182,10 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
}
|
||||
switch *owner {
|
||||
case "self":
|
||||
// "self" matches only AMIs owned by the caller
|
||||
if amiOwner == accountID {
|
||||
found = true
|
||||
}
|
||||
default:
|
||||
// Match by explicit account ID. System AMIs are stored
|
||||
// with a non-account owner (e.g. "spinifex") but report
|
||||
// GlobalAccountID in the response, so match against both.
|
||||
if amiOwner == *owner {
|
||||
found = true
|
||||
} else if isSystemAMI && *owner == utils.GlobalAccountID {
|
||||
@@ -270,15 +201,11 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
}
|
||||
}
|
||||
|
||||
// Resolve the OwnerId for the response. System AMIs use global account.
|
||||
ownerID := amiOwner
|
||||
if isSystemAMI {
|
||||
ownerID = utils.GlobalAccountID
|
||||
}
|
||||
|
||||
// Build EC2 Image from AMIMetadata. Empty BootMode passes through as
|
||||
// empty so legacy AMIs registered before this field existed don't get
|
||||
// a synthesized value.
|
||||
image := &ec2.Image{
|
||||
ImageId: aws.String(amiMeta.ImageID),
|
||||
Name: aws.String(amiMeta.Name),
|
||||
@@ -294,7 +221,7 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
Public: aws.Bool(false),
|
||||
State: aws.String(amiImageState(amiMeta.State)),
|
||||
ImageType: aws.String("machine"),
|
||||
Hypervisor: aws.String("xen"), // Default hypervisor
|
||||
Hypervisor: aws.String("xen"),
|
||||
BootMode: aws.String(amiMeta.BootMode),
|
||||
}
|
||||
|
||||
@@ -305,7 +232,6 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
|
||||
image.Tags = utils.MapToEC2Tags(amiMeta.Tags)
|
||||
|
||||
// Apply filters against the fully-built image
|
||||
if len(parsedFilters) > 0 && !imageMatchesFilters(image, parsedFilters, amiMeta.Tags) {
|
||||
continue
|
||||
}
|
||||
@@ -313,7 +239,6 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
images = append(images, image)
|
||||
}
|
||||
|
||||
// If specific ImageIds were requested, verify all were found
|
||||
if len(input.ImageIds) > 0 {
|
||||
foundIDs := make(map[string]bool, len(images))
|
||||
for _, img := range images {
|
||||
@@ -328,12 +253,8 @@ func (s *ImageServiceImpl) DescribeImages(ctx context.Context, input *ec2.Descri
|
||||
}
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "DescribeImages completed", "count", len(images),
|
||||
"prefixesExamined", prefixesExamined, "unreadableConfigs", unreadableConfigs)
|
||||
|
||||
return &ec2.DescribeImagesOutput{
|
||||
Images: images,
|
||||
}, nil
|
||||
slog.InfoContext(ctx, "DescribeImages completed", "count", len(images))
|
||||
return &ec2.DescribeImagesOutput{Images: images}, nil
|
||||
}
|
||||
|
||||
// amiImageState maps AMIMetadata.State to the ec2.Image state string. An
|
||||
@@ -452,15 +373,15 @@ func (s *ImageServiceImpl) CreateImageFromInstance(params CreateImageParams, acc
|
||||
}
|
||||
|
||||
// Step 2: Read source volume config for size
|
||||
volumeConfig, err := s.getVolumeConfig(context.Background(), params.RootVolumeID)
|
||||
volMeta, err := s.getVolumeMetadata(context.Background(), params.RootVolumeID)
|
||||
if err != nil {
|
||||
slog.Error("CreateImageFromInstance: failed to read volume config", "volumeId", params.RootVolumeID, "err", err)
|
||||
slog.Error("CreateImageFromInstance: failed to read volume metadata", "volumeId", params.RootVolumeID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
volumeSizeGiB := volumeConfig.VolumeMetadata.SizeGiB
|
||||
volumeSizeGiB := volMeta.CapacityGiB
|
||||
|
||||
// Step 3: Read source AMI config for architecture, platform, etc.
|
||||
sourceAMI := viperblock.AMIMetadata{
|
||||
sourceAMI := ebsmetadata.AMI{
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
@@ -481,7 +402,7 @@ func (s *ImageServiceImpl) CreateImageFromInstance(params CreateImageParams, acc
|
||||
}
|
||||
|
||||
// Step 5: Build and store AMI config
|
||||
meta := viperblock.AMIMetadata{
|
||||
meta := ebsmetadata.AMI{
|
||||
ImageID: amiID,
|
||||
Name: name,
|
||||
Description: aws.StringValue(input.Description),
|
||||
@@ -522,283 +443,135 @@ func (s *ImageServiceImpl) snapshotRunningVolume(volumeID, snapshotID, accountID
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
volConfig, err := s.getVolumeConfig(context.Background(), volumeID)
|
||||
volMeta, err := s.getVolumeMetadata(context.Background(), volumeID)
|
||||
if err != nil {
|
||||
slog.Error("snapshotRunningVolume: failed to read volume config", "volumeId", volumeID, "err", err)
|
||||
slog.Error("snapshotRunningVolume: failed to read volume metadata", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
volumeSize := volConfig.VolumeMetadata.SizeGiB * 1024 * 1024 * 1024
|
||||
|
||||
// Flush the writes still buffered by whichever node serves the volume, so
|
||||
// the live checkpoint below is current rather than an arbitrarily stale one
|
||||
// the snapshot below is current rather than an arbitrarily stale one
|
||||
// predating e.g. a guest-triggered GPT rewrite (growpart on first boot). An
|
||||
// attached volume that cannot be drained fails here rather than silently
|
||||
// producing an image from a stale checkpoint.
|
||||
if err := handlers_ec2_snapshot.DrainVolume(context.Background(), s.config, s.store, s.natsConn, volumeID, volConfig.VolumeMetadata, accountID); err != nil {
|
||||
if err := handlers_ec2_snapshot.DrainVolume(context.Background(), s.config, s.store, s.natsConn, volumeID, volMeta.State, volMeta.AttachedInstance, accountID); err != nil {
|
||||
slog.Error("snapshotRunningVolume: drain failed", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
cfg := vbs3.S3Config{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: volumeSize,
|
||||
Bucket: s.config.Predastore.Bucket,
|
||||
Region: s.config.Predastore.Region,
|
||||
AccessKey: s.config.Predastore.AccessKey,
|
||||
SecretKey: s.config.Predastore.SecretKey,
|
||||
Host: s.config.Predastore.Host,
|
||||
}
|
||||
|
||||
mkey, err := utils.LoadViperblockMasterKey(s.config.Viperblock.EncryptionKeyFile)
|
||||
if err != nil {
|
||||
slog.Error("snapshotRunningVolume: failed to load encryption key", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
vbconfig := viperblock.VB{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: volumeSize,
|
||||
BaseDir: s.config.WalDir,
|
||||
Cache: viperblock.Cache{Config: viperblock.CacheConfig{Size: 0}},
|
||||
MasterKey: mkey,
|
||||
EncryptionEnabled: mkey != nil,
|
||||
}
|
||||
|
||||
vb, err := viperblock.New(&vbconfig, "s3", cfg)
|
||||
if err != nil {
|
||||
slog.Error("snapshotRunningVolume: failed to create viperblock instance", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
defer func() {
|
||||
vb.StopChunkUploader()
|
||||
vb.StopWALSyncer()
|
||||
}()
|
||||
|
||||
if err := vb.Backend.Init(); err != nil {
|
||||
slog.Error("snapshotRunningVolume: failed to init backend", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := vb.LoadState(); err != nil {
|
||||
slog.Error("snapshotRunningVolume: failed to load state", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := vb.LoadLiveCheckpoint(); err != nil {
|
||||
slog.Error("snapshotRunningVolume: failed to load live checkpoint", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if _, err := vb.CreateSnapshot(snapshotID); err != nil {
|
||||
slog.Error("snapshotRunningVolume: CreateSnapshot failed", "volumeId", volumeID, "snapshotId", snapshotID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
slog.Info("snapshotRunningVolume: snapshot created", "volumeId", volumeID, "snapshotId", snapshotID)
|
||||
return nil
|
||||
return s.createSnapshot(context.Background(), volMeta, snapshotID, "snapshotRunningVolume")
|
||||
}
|
||||
|
||||
// snapshotStoppedVolume creates a snapshot offline by loading viperblock state from S3.
|
||||
// snapshotStoppedVolume creates a snapshot of a detached volume. No drain: the
|
||||
// volume is stopped/detached, so nothing is buffered on a serving node.
|
||||
// viperblockd's handleCreateSnapshot prefers a live mounted engine over opening
|
||||
// a second one, so it handles the mounted case itself.
|
||||
func (s *ImageServiceImpl) snapshotStoppedVolume(volumeID, snapshotID string) error {
|
||||
if s.config == nil {
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Read volume config to get the size (required by viperblock.New)
|
||||
volConfig, err := s.getVolumeConfig(context.Background(), volumeID)
|
||||
volMeta, err := s.getVolumeMetadata(context.Background(), volumeID)
|
||||
if err != nil {
|
||||
slog.Error("snapshotStoppedVolume: failed to read volume config", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
volumeSize := volConfig.VolumeMetadata.SizeGiB * 1024 * 1024 * 1024
|
||||
|
||||
cfg := vbs3.S3Config{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: volumeSize,
|
||||
Bucket: s.config.Predastore.Bucket,
|
||||
Region: s.config.Predastore.Region,
|
||||
AccessKey: s.config.Predastore.AccessKey,
|
||||
SecretKey: s.config.Predastore.SecretKey,
|
||||
Host: s.config.Predastore.Host,
|
||||
}
|
||||
|
||||
mkey, err := utils.LoadViperblockMasterKey(s.config.Viperblock.EncryptionKeyFile)
|
||||
if err != nil {
|
||||
slog.Error("snapshotStoppedVolume: failed to load encryption key", "volumeId", volumeID, "err", err)
|
||||
slog.Error("snapshotStoppedVolume: failed to read volume metadata", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
vbconfig := viperblock.VB{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: volumeSize,
|
||||
BaseDir: s.config.WalDir,
|
||||
Cache: viperblock.Cache{Config: viperblock.CacheConfig{Size: 0}},
|
||||
MasterKey: mkey,
|
||||
EncryptionEnabled: mkey != nil,
|
||||
}
|
||||
return s.createSnapshot(context.Background(), volMeta, snapshotID, "snapshotStoppedVolume")
|
||||
}
|
||||
|
||||
vb, err := viperblock.New(&vbconfig, "s3", cfg)
|
||||
if err != nil {
|
||||
slog.Error("snapshotStoppedVolume: failed to create viperblock instance", "volumeId", volumeID, "err", err)
|
||||
// createSnapshot delegates snapshot creation for volMeta to the EBS provider.
|
||||
// op names the calling path so a failure is attributable in the log.
|
||||
func (s *ImageServiceImpl) createSnapshot(ctx context.Context, volMeta ebsmetadata.Volume, snapshotID, op string) error {
|
||||
if s.provider == nil {
|
||||
slog.ErrorContext(ctx, "no EBS provider configured", "op", op)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
defer func() {
|
||||
vb.StopChunkUploader()
|
||||
vb.StopWALSyncer()
|
||||
}()
|
||||
|
||||
if err := vb.Backend.Init(); err != nil {
|
||||
slog.Error("snapshotStoppedVolume: failed to init backend", "volumeId", volumeID, "err", err)
|
||||
created, err := s.provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, VolumeID: volMeta.VolumeID, VolumeHandle: volMeta.ProviderHandle,
|
||||
})
|
||||
if err != nil || created == nil {
|
||||
slog.ErrorContext(ctx, "provider CreateSnapshot failed", "op", op, "volumeId", volMeta.VolumeID, "snapshotId", snapshotID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := vb.LoadState(); err != nil {
|
||||
slog.Error("snapshotStoppedVolume: failed to load state", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := vb.LoadBlockState(); err != nil {
|
||||
slog.Error("snapshotStoppedVolume: failed to load block state", "volumeId", volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Clean up local WAL files regardless of snapshot success or failure
|
||||
defer func() {
|
||||
if err := vb.RemoveLocalFiles(); err != nil {
|
||||
slog.Error("snapshotStoppedVolume: failed to remove local files", "volumeId", volumeID, "err", err)
|
||||
}
|
||||
}()
|
||||
|
||||
if _, err := vb.CreateSnapshot(snapshotID); err != nil {
|
||||
slog.Error("snapshotStoppedVolume: CreateSnapshot failed", "volumeId", volumeID, "snapshotId", snapshotID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
slog.Info("snapshotStoppedVolume: snapshot created", "volumeId", volumeID, "snapshotId", snapshotID)
|
||||
slog.InfoContext(ctx, "snapshot created", "op", op, "volumeId", volMeta.VolumeID, "snapshotId", snapshotID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// getVolumeConfig reads a volume's VBState config from S3.
|
||||
func (s *ImageServiceImpl) getVolumeConfig(ctx context.Context, volumeID string) (*viperblock.VolumeConfig, error) {
|
||||
configKey := fmt.Sprintf("%s/config.json", volumeID)
|
||||
result, err := s.store.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(configKey),
|
||||
})
|
||||
// getVolumeMetadata reads a volume's control-plane document, remapping a
|
||||
// missing document to InvalidVolume.NotFound.
|
||||
func (s *ImageServiceImpl) getVolumeMetadata(ctx context.Context, volumeID string) (ebsmetadata.Volume, error) {
|
||||
meta, err := s.metadata.GetVolume(ctx, volumeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
if objectstore.IsNoSuchKeyError(err) {
|
||||
return ebsmetadata.Volume{}, errors.New(awserrors.ErrorInvalidVolumeNotFound)
|
||||
}
|
||||
return ebsmetadata.Volume{}, fmt.Errorf("get volume metadata: %w", err)
|
||||
}
|
||||
defer result.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(result.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var vbState viperblock.VBState
|
||||
if err := json.Unmarshal(viperblock.StateBody(body), &vbState); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &vbState.VolumeConfig, nil
|
||||
return meta, nil
|
||||
}
|
||||
|
||||
// amiNameExists checks if any existing AMI already uses the given name.
|
||||
// NoSuchKey is skipped (concurrent deregister race); any other read/decode
|
||||
// error is surfaced so we don't silently allow a duplicate.
|
||||
// amiNameExists checks if any existing AMI already uses the given name. It
|
||||
// lists strictly: an AMI this cannot read is a name it cannot rule out, and
|
||||
// answering false would let a duplicate through and mask the corruption.
|
||||
func (s *ImageServiceImpl) amiNameExists(ctx context.Context, name string) (bool, error) {
|
||||
listResult, err := s.store.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Prefix: aws.String("ami-"),
|
||||
Delimiter: aws.String("/"),
|
||||
})
|
||||
amis, err := s.metadata.ListAMIsStrict(ctx)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("amiNameExists: failed to list AMIs: %w", err)
|
||||
}
|
||||
|
||||
for _, prefix := range listResult.CommonPrefixes {
|
||||
if prefix.Prefix == nil {
|
||||
continue
|
||||
}
|
||||
configKey := *prefix.Prefix + "config.json"
|
||||
result, err := s.store.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(configKey),
|
||||
})
|
||||
if err != nil {
|
||||
if objectstore.IsNoSuchKeyError(err) {
|
||||
continue
|
||||
}
|
||||
return false, fmt.Errorf("amiNameExists: failed to read %s: %w", configKey, err)
|
||||
}
|
||||
|
||||
body, readErr := io.ReadAll(result.Body)
|
||||
_ = result.Body.Close()
|
||||
if readErr != nil {
|
||||
return false, fmt.Errorf("amiNameExists: failed to read %s: %w", configKey, readErr)
|
||||
}
|
||||
var vbState viperblock.VBState
|
||||
if decodeErr := json.Unmarshal(viperblock.StateBody(body), &vbState); decodeErr != nil {
|
||||
return false, fmt.Errorf("amiNameExists: failed to decode %s: %w", configKey, decodeErr)
|
||||
}
|
||||
|
||||
if vbState.VolumeConfig.AMIMetadata.Name == name {
|
||||
for _, ami := range amis {
|
||||
if ami.Name == name {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// ErrCorruptAMIConfig wraps JSON decode failures on AMI config.json so callers
|
||||
// can distinguish a truly-missing AMI from one whose config exists but can't
|
||||
// be parsed.
|
||||
var ErrCorruptAMIConfig = errors.New("corrupt AMI config")
|
||||
|
||||
// GetAMIConfig returns NoSuchKeyError if the AMI is missing, or an error
|
||||
// wrapping ErrCorruptAMIConfig on decode failure.
|
||||
func (s *ImageServiceImpl) GetAMIConfig(ctx context.Context, imageID string) (viperblock.AMIMetadata, error) {
|
||||
configKey := fmt.Sprintf("%s/config.json", imageID)
|
||||
result, err := s.store.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(configKey),
|
||||
})
|
||||
if err != nil {
|
||||
return viperblock.AMIMetadata{}, err
|
||||
}
|
||||
defer result.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(result.Body)
|
||||
if err != nil {
|
||||
return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %w", ErrCorruptAMIConfig, configKey, err)
|
||||
}
|
||||
var vbState viperblock.VBState
|
||||
if err := json.Unmarshal(viperblock.StateBody(body), &vbState); err != nil {
|
||||
return viperblock.AMIMetadata{}, fmt.Errorf("%w: %s: %w", ErrCorruptAMIConfig, configKey, err)
|
||||
}
|
||||
return vbState.VolumeConfig.AMIMetadata, nil
|
||||
// GetAMIConfig reads an AMI's control-plane document. Errors come from the
|
||||
// metadata store.
|
||||
func (s *ImageServiceImpl) GetAMIConfig(ctx context.Context, imageID string) (ebsmetadata.AMI, error) {
|
||||
return s.metadata.GetAMI(ctx, imageID)
|
||||
}
|
||||
|
||||
// putAMIConfig writes AMI metadata to {imageID}/config.json, preserving the
|
||||
// VBState wrapper used by GetAMIConfig.
|
||||
func (s *ImageServiceImpl) putAMIConfig(ctx context.Context, imageID string, meta viperblock.AMIMetadata) error {
|
||||
state := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: meta,
|
||||
},
|
||||
}
|
||||
|
||||
data, err := json.Marshal(state)
|
||||
// GetAMISourceVolumeID returns the volume whose blocks imageID's snapshot
|
||||
// references, read from the snapshot's metadata.json. Bundled system AMIs have
|
||||
// no standalone snapshot metadata; their snapshot is named after the AMI.
|
||||
func (s *ImageServiceImpl) GetAMISourceVolumeID(ctx context.Context, imageID string) (string, error) {
|
||||
meta, err := s.GetAMIConfig(ctx, imageID)
|
||||
if err != nil {
|
||||
return err
|
||||
if objectstore.IsNoSuchKeyError(err) || errors.Is(err, ebsmetadata.ErrCorruptDocument) {
|
||||
return "", errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
}
|
||||
slog.ErrorContext(ctx, "GetAMISourceVolumeID: failed to read AMI config", "imageId", imageID, "err", err)
|
||||
return "", errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if meta.SnapshotID == "" {
|
||||
return "", errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
}
|
||||
|
||||
configKey := fmt.Sprintf("%s/config.json", imageID)
|
||||
_, err = s.store.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(configKey),
|
||||
Body: bytes.NewReader(data),
|
||||
ContentType: aws.String("application/json"),
|
||||
})
|
||||
return err
|
||||
snapCfg, err := handlers_ec2_snapshot.ReadSnapshotConfig(s.store, s.bucketName, meta.SnapshotID)
|
||||
if err != nil {
|
||||
if objectstore.IsNoSuchKeyError(err) && meta.ImageOwnerAlias != "" && !utils.IsAccountID(meta.ImageOwnerAlias) {
|
||||
return imageID, nil
|
||||
}
|
||||
if objectstore.IsNoSuchKeyError(err) || errors.Is(err, handlers_ec2_snapshot.ErrCorruptSnapshotMetadata) {
|
||||
return "", errors.New(awserrors.ErrorInvalidSnapshotNotFound)
|
||||
}
|
||||
slog.ErrorContext(ctx, "GetAMISourceVolumeID: failed to read snapshot metadata",
|
||||
"imageId", imageID, "snapshotId", meta.SnapshotID, "err", err)
|
||||
return "", errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if snapCfg.VolumeID == "" {
|
||||
slog.ErrorContext(ctx, "GetAMISourceVolumeID: snapshot metadata has no source volume",
|
||||
"imageId", imageID, "snapshotId", meta.SnapshotID)
|
||||
return "", errors.New(awserrors.ErrorInvalidSnapshotNotFound)
|
||||
}
|
||||
return snapCfg.VolumeID, nil
|
||||
}
|
||||
|
||||
// putAMIConfig writes AMI metadata to the ebsmetadata document store.
|
||||
func (s *ImageServiceImpl) putAMIConfig(ctx context.Context, _ string, meta ebsmetadata.AMI) error {
|
||||
return s.metadata.PutAMI(ctx, meta)
|
||||
}
|
||||
|
||||
// ApplyRecordTags mirrors CreateTags into the owning AMI config so
|
||||
@@ -867,7 +640,7 @@ func (s *ImageServiceImpl) updateAMITags(ctx context.Context, imageID, accountID
|
||||
|
||||
// checkAMIOwnership rejects cross-account and system-AMI mutations. Empty
|
||||
// owner is ServerInternal (corrupt config) rather than a misleading 403.
|
||||
func (s *ImageServiceImpl) checkAMIOwnership(meta viperblock.AMIMetadata, accountID string) error {
|
||||
func (s *ImageServiceImpl) checkAMIOwnership(meta ebsmetadata.AMI, accountID string) error {
|
||||
owner := meta.ImageOwnerAlias
|
||||
if owner == "" {
|
||||
slog.Error("checkAMIOwnership: AMI config has empty ImageOwnerAlias", "imageId", meta.ImageID)
|
||||
@@ -881,7 +654,7 @@ func (s *ImageServiceImpl) checkAMIOwnership(meta viperblock.AMIMetadata, accoun
|
||||
|
||||
// callerCanReadAMI: empty owner is invisible (corrupt); non-account owner is
|
||||
// a system AMI visible to all; account owner is private to that account.
|
||||
func callerCanReadAMI(meta viperblock.AMIMetadata, accountID string) bool {
|
||||
func callerCanReadAMI(meta ebsmetadata.AMI, accountID string) bool {
|
||||
owner := meta.ImageOwnerAlias
|
||||
if owner == "" {
|
||||
return false
|
||||
@@ -895,22 +668,22 @@ func callerCanReadAMI(meta viperblock.AMIMetadata, accountID string) bool {
|
||||
// loadAMIForMutation fetches the AMI and enforces ownership. Cross-account
|
||||
// callers see UnauthorizedOperation, not NotFound (they already know the ID).
|
||||
// No CAS: concurrent writers last-write-wins on the full struct.
|
||||
func (s *ImageServiceImpl) loadAMIForMutation(ctx context.Context, imageID, accountID string) (viperblock.AMIMetadata, error) {
|
||||
func (s *ImageServiceImpl) loadAMIForMutation(ctx context.Context, imageID, accountID string) (ebsmetadata.AMI, error) {
|
||||
if !strings.HasPrefix(imageID, "ami-") {
|
||||
return viperblock.AMIMetadata{}, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
|
||||
return ebsmetadata.AMI{}, errors.New(awserrors.ErrorInvalidAMIIDMalformed)
|
||||
}
|
||||
|
||||
meta, err := s.GetAMIConfig(ctx, imageID)
|
||||
if err != nil {
|
||||
if objectstore.IsNoSuchKeyError(err) {
|
||||
return viperblock.AMIMetadata{}, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
return ebsmetadata.AMI{}, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
}
|
||||
slog.Error("loadAMIForMutation: failed to read AMI config", "imageId", imageID, "err", err)
|
||||
return viperblock.AMIMetadata{}, errors.New(awserrors.ErrorServerInternal)
|
||||
return ebsmetadata.AMI{}, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := s.checkAMIOwnership(meta, accountID); err != nil {
|
||||
return viperblock.AMIMetadata{}, err
|
||||
return ebsmetadata.AMI{}, err
|
||||
}
|
||||
return meta, nil
|
||||
}
|
||||
@@ -945,7 +718,7 @@ func (s *ImageServiceImpl) CopyImage(ctx context.Context, input *ec2.CopyImageIn
|
||||
if err != nil {
|
||||
// Corrupt source is treated as NotFound so callers can't tell which
|
||||
// half of the AMI/snapshot pair is broken.
|
||||
if objectstore.IsNoSuchKeyError(err) || errors.Is(err, ErrCorruptAMIConfig) {
|
||||
if objectstore.IsNoSuchKeyError(err) || errors.Is(err, ebsmetadata.ErrCorruptDocument) {
|
||||
return nil, errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
}
|
||||
slog.ErrorContext(ctx, "CopyImage: failed to read source AMI config", "sourceImageId", sourceImageID, "err", err)
|
||||
@@ -1012,7 +785,7 @@ func (s *ImageServiceImpl) CopyImage(ctx context.Context, input *ec2.CopyImageIn
|
||||
|
||||
tags := mergeCopyImageTags(srcMeta.Tags, input.TagSpecifications, aws.BoolValue(input.CopyImageTags))
|
||||
|
||||
meta := viperblock.AMIMetadata{
|
||||
meta := ebsmetadata.AMI{
|
||||
ImageID: newImageID,
|
||||
Name: name,
|
||||
Description: description,
|
||||
@@ -1129,7 +902,7 @@ func (s *ImageServiceImpl) clusterEncryptionEnabled() bool {
|
||||
// synthesizeRootBlockDeviceMapping returns /dev/sda1 with size+snapshot from AMIMetadata,
|
||||
// or nil for non-EBS AMIs. encrypted reflects the cluster-level encryption posture
|
||||
// (master key configured); AMI metadata carries no per-image encryption flag.
|
||||
func synthesizeRootBlockDeviceMapping(meta viperblock.AMIMetadata, encrypted bool) []*ec2.BlockDeviceMapping {
|
||||
func synthesizeRootBlockDeviceMapping(meta ebsmetadata.AMI, encrypted bool) []*ec2.BlockDeviceMapping {
|
||||
if meta.RootDeviceType != "ebs" {
|
||||
return nil
|
||||
}
|
||||
@@ -1218,7 +991,7 @@ func (s *ImageServiceImpl) RegisterImage(ctx context.Context, input *ec2.Registe
|
||||
tags := utils.ExtractTags(input.TagSpecifications, "image")
|
||||
|
||||
amiID := utils.GenerateResourceID("ami")
|
||||
meta := viperblock.AMIMetadata{
|
||||
meta := ebsmetadata.AMI{
|
||||
ImageID: amiID,
|
||||
Name: name,
|
||||
Description: description,
|
||||
@@ -1268,8 +1041,10 @@ func pickRootSnapshotBDM(mappings []*ec2.BlockDeviceMapping, rootDeviceName *str
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeregisterImage hard-deletes config.json. Backing snapshot is untouched, so
|
||||
// operators run delete-snapshot separately to reclaim block storage.
|
||||
// DeregisterImage hard-deletes the AMI document. The pre-provider
|
||||
// {imageID}/config.json goes too, or the metadata store's legacy read fallback
|
||||
// would resurrect the image. Backing snapshot is untouched, so operators run
|
||||
// delete-snapshot separately to reclaim block storage.
|
||||
func (s *ImageServiceImpl) DeregisterImage(ctx context.Context, input *ec2.DeregisterImageInput, accountID string) (*ec2.DeregisterImageOutput, error) {
|
||||
if input == nil || input.ImageId == nil || *input.ImageId == "" {
|
||||
return nil, errors.New(awserrors.ErrorMissingParameter)
|
||||
@@ -1281,12 +1056,17 @@ func (s *ImageServiceImpl) DeregisterImage(ctx context.Context, input *ec2.Dereg
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := s.metadata.DeleteAMI(ctx, imageID); err != nil {
|
||||
slog.ErrorContext(ctx, "DeregisterImage: failed to delete AMI document", "imageId", imageID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
configKey := fmt.Sprintf("%s/config.json", imageID)
|
||||
if _, err := s.store.DeleteObject(ctx, &s3.DeleteObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(configKey),
|
||||
}); err != nil {
|
||||
slog.ErrorContext(ctx, "DeregisterImage: failed to delete AMI config", "imageId", imageID, "err", err)
|
||||
}); err != nil && !objectstore.IsNoSuchKeyError(err) {
|
||||
slog.ErrorContext(ctx, "DeregisterImage: failed to delete legacy AMI config", "imageId", imageID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@ package handlers_ec2_image
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
@@ -17,9 +16,9 @@ import (
|
||||
awss3 "github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
handlers_ec2_snapshot "github.com/mulgadc/spinifex/spinifex/handlers/ec2/snapshot"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
@@ -27,117 +26,96 @@ import (
|
||||
const testBucket = "test-bucket"
|
||||
const testAccountID = "000000000001"
|
||||
|
||||
// setupTestImageService creates an image service with in-memory storage for testing.
|
||||
// setupTestImageService creates an image service with in-memory storage for
|
||||
// testing.
|
||||
func setupTestImageService(t *testing.T) (*ImageServiceImpl, *objectstore.MemoryObjectStore) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := NewImageServiceImplWithStore(store, testBucket)
|
||||
return svc, store
|
||||
}
|
||||
|
||||
// createTestVolumeConfig creates a test volume config in the mock store.
|
||||
func createTestVolumeConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string, sizeGiB int) {
|
||||
volumeState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
SizeGiB: uint64(sizeGiB),
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(volumeState)
|
||||
require.NoError(t, err)
|
||||
// putAMIDocument writes an AMI's control-plane document, which is what
|
||||
// DescribeImages enumerates.
|
||||
func putAMIDocument(t *testing.T, store *objectstore.MemoryObjectStore, ami ebsmetadata.AMI) {
|
||||
t.Helper()
|
||||
require.NoError(t, ebsmetadata.NewStore(store, testBucket).PutAMI(context.Background(), ami))
|
||||
}
|
||||
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
// putRawAMIDocument writes AMI document bytes verbatim, for a corrupt document
|
||||
// a typed seed cannot express.
|
||||
func putRawAMIDocument(t *testing.T, store *objectstore.MemoryObjectStore, imageID, body string) {
|
||||
t.Helper()
|
||||
_, err := store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(amiDocumentKey(t, imageID)),
|
||||
Body: strings.NewReader(body),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// amiDocumentKey is where an AMI's control-plane document lives.
|
||||
func amiDocumentKey(t *testing.T, imageID string) string {
|
||||
t.Helper()
|
||||
key, err := ebsmetadata.AMIKey(imageID)
|
||||
require.NoError(t, err)
|
||||
return key
|
||||
}
|
||||
|
||||
// putVolumeDocument writes a volume's control-plane document.
|
||||
func putVolumeDocument(t *testing.T, store *objectstore.MemoryObjectStore, volume ebsmetadata.Volume) {
|
||||
t.Helper()
|
||||
require.NoError(t, ebsmetadata.NewStore(store, testBucket).PutVolume(context.Background(), volume))
|
||||
}
|
||||
|
||||
// createTestVolumeConfig creates a test volume config in the mock store.
|
||||
func createTestVolumeConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string, sizeGiB int) {
|
||||
putVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
CapacityGiB: uint64(sizeGiB),
|
||||
})
|
||||
}
|
||||
|
||||
// createTestAMIConfig creates a test AMI config in the mock store.
|
||||
func createTestAMIConfig(t *testing.T, store *objectstore.MemoryObjectStore, imageID string) {
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: viperblock.AMIMetadata{
|
||||
ImageID: imageID,
|
||||
Name: "test-ami",
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
putAMIDocument(t, store, ebsmetadata.AMI{
|
||||
ImageID: imageID,
|
||||
Name: "test-ami",
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// createTestAMIConfigWithName creates a test AMI config with a specified name.
|
||||
// Owner defaults to testAccountID so the AMI is visible to the default caller —
|
||||
// an empty ImageOwnerAlias would be filtered out as a corrupt config.
|
||||
func createTestAMIConfigWithName(t *testing.T, store *objectstore.MemoryObjectStore, imageID, name string) {
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: viperblock.AMIMetadata{
|
||||
ImageID: imageID,
|
||||
Name: name,
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
ImageOwnerAlias: testAccountID,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
putAMIDocument(t, store, ebsmetadata.AMI{
|
||||
ImageID: imageID,
|
||||
Name: name,
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
ImageOwnerAlias: testAccountID,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// createTestAMIConfigWithOwner creates a test AMI config with a specified name and owner.
|
||||
func createTestAMIConfigWithOwner(t *testing.T, store *objectstore.MemoryObjectStore, imageID, name, owner string) {
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: viperblock.AMIMetadata{
|
||||
ImageID: imageID,
|
||||
Name: name,
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
ImageOwnerAlias: owner,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
putAMIDocument(t, store, ebsmetadata.AMI{
|
||||
ImageID: imageID,
|
||||
Name: name,
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
ImageOwnerAlias: owner,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func TestCreateImageFromInstance_NilInput(t *testing.T) {
|
||||
@@ -211,7 +189,7 @@ func TestDescribeImages_AfterCreate(t *testing.T) {
|
||||
func TestDescribeImages_BootModeProjection(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-uefi001",
|
||||
Name: "uefi-image",
|
||||
Architecture: "x86_64",
|
||||
@@ -222,7 +200,7 @@ func TestDescribeImages_BootModeProjection(t *testing.T) {
|
||||
ImageOwnerAlias: testAccountID,
|
||||
BootMode: "uefi",
|
||||
})
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-legacy001",
|
||||
Name: "legacy-image",
|
||||
Architecture: "x86_64",
|
||||
@@ -269,7 +247,7 @@ func TestDescribeImages_StateProjection(t *testing.T) {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-state001",
|
||||
Name: "state-image",
|
||||
Architecture: "x86_64",
|
||||
@@ -291,21 +269,20 @@ func TestDescribeImages_StateProjection(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetVolumeConfig(t *testing.T) {
|
||||
func TestGetVolumeMetadata(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestVolumeConfig(t, store, "vol-abc123", 20)
|
||||
|
||||
cfg, err := svc.getVolumeConfig(context.Background(), "vol-abc123")
|
||||
meta, err := svc.getVolumeMetadata(context.Background(), "vol-abc123")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, cfg)
|
||||
assert.Equal(t, uint64(20), cfg.VolumeMetadata.SizeGiB)
|
||||
assert.Equal(t, uint64(20), meta.CapacityGiB)
|
||||
}
|
||||
|
||||
func TestGetVolumeConfig_NotFound(t *testing.T) {
|
||||
func TestGetVolumeMetadata_NotFound(t *testing.T) {
|
||||
svc, _ := setupTestImageService(t)
|
||||
|
||||
_, err := svc.getVolumeConfig(context.Background(), "vol-nonexistent")
|
||||
_, err := svc.getVolumeMetadata(context.Background(), "vol-nonexistent")
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
@@ -455,7 +432,7 @@ func TestCreateImageFromInstance_UniqueNameAllowed(t *testing.T) {
|
||||
func TestPutAMIConfig_RoundTrip(t *testing.T) {
|
||||
svc, _ := setupTestImageService(t)
|
||||
|
||||
meta := viperblock.AMIMetadata{
|
||||
meta := ebsmetadata.AMI{
|
||||
ImageID: "ami-roundtrip01",
|
||||
Name: "round-trip",
|
||||
Description: "hello",
|
||||
@@ -499,7 +476,7 @@ func TestCheckAMIOwnership(t *testing.T) {
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := svc.checkAMIOwnership(viperblock.AMIMetadata{ImageOwnerAlias: tt.owner}, caller)
|
||||
err := svc.checkAMIOwnership(ebsmetadata.AMI{ImageOwnerAlias: tt.owner}, caller)
|
||||
if tt.wantErr == "" {
|
||||
assert.NoError(t, err)
|
||||
} else {
|
||||
@@ -520,10 +497,10 @@ func TestDeregisterImage_HappyPath(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.NotNil(t, out)
|
||||
|
||||
// AMI config gone from S3
|
||||
// AMI document gone from S3
|
||||
_, getErr := store.GetObject(context.Background(), &awss3.GetObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(amiID + "/config.json"),
|
||||
Key: aws.String(amiDocumentKey(t, amiID)),
|
||||
})
|
||||
require.Error(t, getErr)
|
||||
assert.True(t, objectstore.IsNoSuchKeyError(getErr))
|
||||
@@ -573,7 +550,7 @@ func TestDeregisterImage_CrossAccount(t *testing.T) {
|
||||
// Confirm AMI still present after rejected mutation.
|
||||
_, getErr := store.GetObject(context.Background(), &awss3.GetObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(amiID + "/config.json"),
|
||||
Key: aws.String(amiDocumentKey(t, amiID)),
|
||||
})
|
||||
require.NoError(t, getErr)
|
||||
}
|
||||
@@ -594,33 +571,21 @@ func TestDeregisterImage_DoesNotTouchSnapshot(t *testing.T) {
|
||||
snapID := "snap-keep001"
|
||||
|
||||
// AMI pointing at a snapshot
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: viperblock.AMIMetadata{
|
||||
ImageID: amiID,
|
||||
Name: "with-snapshot",
|
||||
SnapshotID: snapID,
|
||||
ImageOwnerAlias: testAccountID,
|
||||
Architecture: "x86_64",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(amiID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
putAMIDocument(t, store, ebsmetadata.AMI{
|
||||
ImageID: amiID,
|
||||
Name: "with-snapshot",
|
||||
SnapshotID: snapID,
|
||||
ImageOwnerAlias: testAccountID,
|
||||
Architecture: "x86_64",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Backing snapshot metadata
|
||||
require.NoError(t, svc.putSnapshotMetadata(context.Background(), snapID, "vol-keep", 8, testAccountID))
|
||||
|
||||
_, err = svc.DeregisterImage(context.Background(), &ec2.DeregisterImageInput{ImageId: aws.String(amiID)}, testAccountID)
|
||||
_, err := svc.DeregisterImage(context.Background(), &ec2.DeregisterImageInput{ImageId: aws.String(amiID)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Snapshot metadata still present
|
||||
@@ -723,13 +688,8 @@ func TestDescribeImages_NonAMIPrefixIgnored(t *testing.T) {
|
||||
func TestDescribeImages_InvalidConfigJSON(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
// Store invalid JSON as an AMI config
|
||||
_, err := store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("ami-bad123/config.json"),
|
||||
Body: strings.NewReader("not valid json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
// Store invalid JSON as an AMI document
|
||||
putRawAMIDocument(t, store, "ami-bad123", "not valid json")
|
||||
|
||||
// Should skip the invalid AMI without error
|
||||
result, err := svc.DescribeImages(context.Background(), &ec2.DescribeImagesInput{}, testAccountID)
|
||||
@@ -737,38 +697,30 @@ func TestDescribeImages_InvalidConfigJSON(t *testing.T) {
|
||||
assert.Empty(t, result.Images)
|
||||
}
|
||||
|
||||
// TestDescribeImages_ConfigFaultCounting covers a config.json that
|
||||
// exists but can't be fetched/read/parsed must be promoted to WARN and counted
|
||||
// as unreadable so it stays visible at INFO, while a genuinely absent
|
||||
// config.json (a non-AMI directory) stays silent at DEBUG and uncounted, and a
|
||||
// healthy prefix is unaffected either way.
|
||||
// TestDescribeImages_ConfigFaultCounting covers a config.json that exists but
|
||||
// can't be fetched/read/parsed being promoted to WARN so it stays visible at
|
||||
// the default level, while a genuinely absent config.json (a non-AMI
|
||||
// directory) stays silent, and a healthy prefix is unaffected either way.
|
||||
func TestDescribeImages_ConfigFaultCounting(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
setup func(t *testing.T, store *objectstore.MemoryObjectStore)
|
||||
wantImageCount int
|
||||
wantUnreadable int
|
||||
wantWarnLogged bool
|
||||
}{
|
||||
{
|
||||
name: "malformed config.json is counted as unreadable and does not vanish silently",
|
||||
name: "malformed document is counted as unreadable and does not vanish silently",
|
||||
setup: func(t *testing.T, store *objectstore.MemoryObjectStore) {
|
||||
_, err := store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("ami-malformed1/config.json"),
|
||||
Body: strings.NewReader("{not valid json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putRawAMIDocument(t, store, "ami-malformed1", "{not valid json")
|
||||
},
|
||||
wantImageCount: 0,
|
||||
wantUnreadable: 1,
|
||||
wantWarnLogged: true,
|
||||
},
|
||||
{
|
||||
name: "missing config.json (NoSuchKey) is not counted as a fault",
|
||||
name: "an object that is not a document is not counted as a fault",
|
||||
setup: func(t *testing.T, store *objectstore.MemoryObjectStore) {
|
||||
// A placeholder object under the prefix makes ListObjectsV2 surface
|
||||
// "ami-nokey1/" as a CommonPrefix with no config.json underneath.
|
||||
// An object outside the document prefix is not an AMI at all, so
|
||||
// it must neither appear nor register as a fault.
|
||||
_, err := store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("ami-nokey1/placeholder.txt"),
|
||||
@@ -777,7 +729,6 @@ func TestDescribeImages_ConfigFaultCounting(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
},
|
||||
wantImageCount: 0,
|
||||
wantUnreadable: 0,
|
||||
wantWarnLogged: false,
|
||||
},
|
||||
{
|
||||
@@ -786,7 +737,6 @@ func TestDescribeImages_ConfigFaultCounting(t *testing.T) {
|
||||
createTestAMIConfigWithOwner(t, store, "ami-healthy1", "healthy-ami", testAccountID)
|
||||
},
|
||||
wantImageCount: 1,
|
||||
wantUnreadable: 0,
|
||||
wantWarnLogged: false,
|
||||
},
|
||||
}
|
||||
@@ -807,8 +757,6 @@ func TestDescribeImages_ConfigFaultCounting(t *testing.T) {
|
||||
assert.Len(t, out.Images, tt.wantImageCount)
|
||||
|
||||
logs := buf.String()
|
||||
assert.Contains(t, logs, "prefixesExamined=1", "the single ami-* prefix must be counted as examined")
|
||||
assert.Contains(t, logs, fmt.Sprintf("unreadableConfigs=%d", tt.wantUnreadable))
|
||||
|
||||
if tt.wantWarnLogged {
|
||||
assert.Contains(t, logs, "level=WARN",
|
||||
@@ -824,24 +772,10 @@ func TestDescribeImages_ConfigFaultCounting(t *testing.T) {
|
||||
func TestDescribeImages_EmptyImageIDSkipped(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
// AMI config with empty ImageID
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: viperblock.AMIMetadata{
|
||||
ImageID: "",
|
||||
Name: "empty-id-ami",
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("ami-emptyid/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
// A document whose image_id is empty names no image, so listing it would
|
||||
// hand out an AMI nothing can be launched from.
|
||||
putRawAMIDocument(t, store, "ami-emptyid",
|
||||
`{"schema_version":1,"image_id":"","name":"empty-id-ami"}`)
|
||||
|
||||
result, err := svc.DescribeImages(context.Background(), &ec2.DescribeImagesInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
@@ -851,30 +785,17 @@ func TestDescribeImages_EmptyImageIDSkipped(t *testing.T) {
|
||||
func TestDescribeImages_WithTags(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: viperblock.AMIMetadata{
|
||||
ImageID: "ami-tagged123",
|
||||
Name: "tagged-ami",
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
ImageOwnerAlias: testAccountID,
|
||||
Tags: map[string]string{"Environment": "test", "Name": "my-ami"},
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("ami-tagged123/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
putAMIDocument(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-tagged123",
|
||||
Name: "tagged-ami",
|
||||
Architecture: "x86_64",
|
||||
PlatformDetails: "Linux/UNIX",
|
||||
Virtualization: "hvm",
|
||||
RootDeviceType: "ebs",
|
||||
VolumeSizeGiB: 8,
|
||||
ImageOwnerAlias: testAccountID,
|
||||
Tags: map[string]string{"Environment": "test", "Name": "my-ami"},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
result, err := svc.DescribeImages(context.Background(), &ec2.DescribeImagesInput{
|
||||
ImageIds: []*string{aws.String("ami-tagged123")},
|
||||
@@ -904,28 +825,25 @@ func TestDescribeImages_OwnerFilterNilEntry(t *testing.T) {
|
||||
func TestGetAMIConfig_InvalidJSON(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
_, err := store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("ami-badjson/config.json"),
|
||||
Body: strings.NewReader("not json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putRawAMIDocument(t, store, "ami-badjson", "not json")
|
||||
|
||||
_, err = svc.GetAMIConfig(context.Background(), "ami-badjson")
|
||||
_, err := svc.GetAMIConfig(context.Background(), "ami-badjson")
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
func TestGetVolumeConfig_InvalidJSON(t *testing.T) {
|
||||
func TestGetVolumeMetadata_InvalidJSON(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
_, err := store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
key, err := ebsmetadata.VolumeKey("vol-badjson")
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("vol-badjson/config.json"),
|
||||
Key: aws.String(key),
|
||||
Body: strings.NewReader("{invalid"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.getVolumeConfig(context.Background(), "vol-badjson")
|
||||
_, err = svc.getVolumeMetadata(context.Background(), "vol-badjson")
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
@@ -960,17 +878,12 @@ func TestAmiNameExists_NotFound(t *testing.T) {
|
||||
func TestAmiNameExists_InvalidJSON(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
_, err := store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String("ami-bad/config.json"),
|
||||
Body: strings.NewReader("not json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putRawAMIDocument(t, store, "ami-bad", "not json")
|
||||
|
||||
// A corrupt AMI config is a real store-side problem. Surface it rather
|
||||
// A corrupt AMI document is a real store-side problem. Surface it rather
|
||||
// than silently under-counting names (which would let a caller write a
|
||||
// duplicate and mask the corruption).
|
||||
_, err = svc.amiNameExists(context.Background(), "any-name")
|
||||
_, err := svc.amiNameExists(context.Background(), "any-name")
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
@@ -979,25 +892,12 @@ func TestAmiNameExists_InvalidJSON(t *testing.T) {
|
||||
// createTestAMIConfigFull creates an AMI with all metadata fields for filter testing.
|
||||
// Owner defaults to testAccountID when the caller leaves ImageOwnerAlias empty —
|
||||
// an empty owner would be filtered out as corrupt.
|
||||
func createTestAMIConfigFull(t *testing.T, store *objectstore.MemoryObjectStore, meta viperblock.AMIMetadata) {
|
||||
func createTestAMIConfigFull(t *testing.T, store *objectstore.MemoryObjectStore, meta ebsmetadata.AMI) {
|
||||
t.Helper()
|
||||
if meta.ImageOwnerAlias == "" {
|
||||
meta.ImageOwnerAlias = testAccountID
|
||||
}
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
AMIMetadata: meta,
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(meta.ImageID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putAMIDocument(t, store, meta)
|
||||
}
|
||||
|
||||
// TestDescribeImages_FilterBy verifies single-attribute filters; each subtest builds
|
||||
@@ -1049,11 +949,11 @@ func TestDescribeImages_FilterBy(t *testing.T) {
|
||||
{
|
||||
name: "architecture",
|
||||
setup: func(t *testing.T, store *objectstore.MemoryObjectStore) {
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-x86", Name: "x86-img", Architecture: "x86_64",
|
||||
RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-arm", Name: "arm-img", Architecture: "arm64",
|
||||
RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
@@ -1066,12 +966,12 @@ func TestDescribeImages_FilterBy(t *testing.T) {
|
||||
{
|
||||
name: "tag",
|
||||
setup: func(t *testing.T, store *objectstore.MemoryObjectStore) {
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-tagged", Name: "tagged-img", Architecture: "x86_64",
|
||||
RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
Tags: map[string]string{"Environment": "prod"},
|
||||
})
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-untagged", Name: "untagged-img", Architecture: "x86_64",
|
||||
RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
@@ -1104,11 +1004,11 @@ func TestDescribeImages_FilterBy(t *testing.T) {
|
||||
{
|
||||
name: "virtualization type",
|
||||
setup: func(t *testing.T, store *objectstore.MemoryObjectStore) {
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-hvm", Name: "hvm-img", Architecture: "x86_64",
|
||||
Virtualization: "hvm", RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-pv", Name: "pv-img", Architecture: "x86_64",
|
||||
Virtualization: "paravirtual", RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
@@ -1121,11 +1021,11 @@ func TestDescribeImages_FilterBy(t *testing.T) {
|
||||
{
|
||||
name: "root device type",
|
||||
setup: func(t *testing.T, store *objectstore.MemoryObjectStore) {
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-ebs", Name: "ebs-img", Architecture: "x86_64",
|
||||
Virtualization: "hvm", RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-is", Name: "is-img", Architecture: "x86_64",
|
||||
Virtualization: "hvm", RootDeviceType: "instance-store", VolumeSizeGiB: 8,
|
||||
})
|
||||
@@ -1174,11 +1074,11 @@ func TestDescribeImages_FilterMultipleValues_OR(t *testing.T) {
|
||||
|
||||
func TestDescribeImages_FilterMultipleNames_AND(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-match", Name: "debian-13", Architecture: "x86_64",
|
||||
RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
createTestAMIConfigFull(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigFull(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-nomatch", Name: "debian-13", Architecture: "arm64",
|
||||
RootDeviceType: "ebs", VolumeSizeGiB: 8,
|
||||
})
|
||||
@@ -1602,13 +1502,13 @@ func TestRegisterImage_OwnerSetToCaller(t *testing.T) {
|
||||
|
||||
// --- CopyImage tests ---
|
||||
|
||||
// readAMIConfigBytes returns the raw bytes of {imageID}/config.json from the
|
||||
// store so tests can prove the source was not mutated by a copy/modify/etc.
|
||||
// readAMIConfigBytes returns the raw bytes of an AMI's document from the store
|
||||
// so tests can prove the source was not mutated by a copy/modify/etc.
|
||||
func readAMIConfigBytes(t *testing.T, store *objectstore.MemoryObjectStore, imageID string) []byte {
|
||||
t.Helper()
|
||||
result, err := store.GetObject(context.Background(), &awss3.GetObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Key: aws.String(amiDocumentKey(t, imageID)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
defer result.Body.Close()
|
||||
@@ -1634,7 +1534,7 @@ func readSnapshotConfigBytes(t *testing.T, store *objectstore.MemoryObjectStore,
|
||||
// putTestAMIConfigWithSnapshot seeds an AMI config that carries a real
|
||||
// SnapshotID — distinct from createTestAMIConfigWithOwner which sets no
|
||||
// snapshot and would be treated as orphaned by CopyImage.
|
||||
func putTestAMIConfigWithSnapshot(t *testing.T, store *objectstore.MemoryObjectStore, imageID, name, owner, snapshotID string, meta viperblock.AMIMetadata) {
|
||||
func putTestAMIConfigWithSnapshot(t *testing.T, store *objectstore.MemoryObjectStore, imageID, name, owner, snapshotID string, meta ebsmetadata.AMI) {
|
||||
t.Helper()
|
||||
meta.ImageID = imageID
|
||||
meta.Name = name
|
||||
@@ -1655,18 +1555,7 @@ func putTestAMIConfigWithSnapshot(t *testing.T, store *objectstore.MemoryObjectS
|
||||
if meta.VolumeSizeGiB == 0 {
|
||||
meta.VolumeSizeGiB = 8
|
||||
}
|
||||
amiState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{AMIMetadata: meta},
|
||||
}
|
||||
data, err := json.Marshal(amiState)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(imageID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putAMIDocument(t, store, meta)
|
||||
}
|
||||
|
||||
// seedCopyableAMI writes a matching (snapshot, AMI) pair so CopyImage can complete
|
||||
@@ -1691,7 +1580,7 @@ func seedCopyableAMI(t *testing.T, store *objectstore.MemoryObjectStore, imageID
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
putTestAMIConfigWithSnapshot(t, store, imageID, name, owner, snapshotID, viperblock.AMIMetadata{
|
||||
putTestAMIConfigWithSnapshot(t, store, imageID, name, owner, snapshotID, ebsmetadata.AMI{
|
||||
VolumeSizeGiB: uint64(sizeGiB),
|
||||
Description: "source desc",
|
||||
})
|
||||
@@ -1758,7 +1647,7 @@ func TestCopyImage_InheritsSourceFields(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
// Source AMI with non-default fields that must propagate.
|
||||
putTestAMIConfigWithSnapshot(t, store, "ami-arm001", "arm-source", testAccountID, "snap-arm001", viperblock.AMIMetadata{
|
||||
putTestAMIConfigWithSnapshot(t, store, "ami-arm001", "arm-source", testAccountID, "snap-arm001", ebsmetadata.AMI{
|
||||
Architecture: "arm64",
|
||||
PlatformDetails: "Linux/UNIX (arm64)",
|
||||
Virtualization: "hvm",
|
||||
@@ -1836,7 +1725,7 @@ func TestCopyImage_BundledSystemAMINoStandaloneSnap(t *testing.T) {
|
||||
// Seed only the AMI config, no snap-xxx/metadata.json object — matches the
|
||||
// on-disk layout produced by `spx admin images import`.
|
||||
putTestAMIConfigWithSnapshot(t, store, "ami-bundled01", "alpine-bundled",
|
||||
"system", "snap-ami-bundled01", viperblock.AMIMetadata{
|
||||
"system", "snap-ami-bundled01", ebsmetadata.AMI{
|
||||
VolumeSizeGiB: 8,
|
||||
Description: "bundled source",
|
||||
})
|
||||
@@ -1894,7 +1783,7 @@ func TestCopyImage_OrphanedSource_MissingSnapshot(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
// AMI config points at a snapshot that doesn't exist on S3.
|
||||
putTestAMIConfigWithSnapshot(t, store, "ami-orphan", "orphan", testAccountID, "snap-ghost", viperblock.AMIMetadata{})
|
||||
putTestAMIConfigWithSnapshot(t, store, "ami-orphan", "orphan", testAccountID, "snap-ghost", ebsmetadata.AMI{})
|
||||
|
||||
_, err := svc.CopyImage(context.Background(), validCopyImageServiceInput("ami-orphan", "orphan-copy"), testAccountID)
|
||||
require.Error(t, err)
|
||||
@@ -2070,27 +1959,15 @@ func TestCopyImage_MissingRequiredFields(t *testing.T) {
|
||||
|
||||
// --- Image attribute tests ---
|
||||
|
||||
func createTestAMIConfigRich(t *testing.T, store *objectstore.MemoryObjectStore, meta viperblock.AMIMetadata) {
|
||||
func createTestAMIConfigRich(t *testing.T, store *objectstore.MemoryObjectStore, meta ebsmetadata.AMI) {
|
||||
t.Helper()
|
||||
state := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{AMIMetadata: meta},
|
||||
}
|
||||
data, err := json.Marshal(state)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.PutObject(context.Background(), &awss3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket),
|
||||
Key: aws.String(meta.ImageID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putAMIDocument(t, store, meta)
|
||||
}
|
||||
|
||||
func TestDescribeImageAttribute_Description(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-desc01",
|
||||
Name: "desc-ami",
|
||||
Description: "the stored description",
|
||||
@@ -2119,7 +1996,7 @@ func TestDescribeImageAttribute_Description(t *testing.T) {
|
||||
func TestDescribeImageAttribute_BlockDeviceMapping(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-bdm01",
|
||||
Name: "bdm-ami",
|
||||
SnapshotID: "snap-bdm01",
|
||||
@@ -2183,7 +2060,7 @@ func TestDescribeImageAttribute_NotFound(t *testing.T) {
|
||||
func TestDescribeImageAttribute_CrossAccountHidesExistence(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-cross01",
|
||||
Name: "cross-ami",
|
||||
Description: "secret",
|
||||
@@ -2202,7 +2079,7 @@ func TestDescribeImageAttribute_CrossAccountHidesExistence(t *testing.T) {
|
||||
func TestDescribeImageAttribute_SystemAMIReadable(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-sys01",
|
||||
Name: "system-ami",
|
||||
Description: "baked-in",
|
||||
@@ -2240,7 +2117,7 @@ func TestDescribeImageAttribute_UnsupportedAttribute(t *testing.T) {
|
||||
func TestModifyImageAttribute_Description(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-mod01",
|
||||
Name: "mod-ami",
|
||||
Description: "old",
|
||||
@@ -2273,7 +2150,7 @@ func TestModifyImageAttribute_Description(t *testing.T) {
|
||||
func TestModifyImageAttribute_DescriptionEmptyValueClears(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-modclr01",
|
||||
Name: "modclr-ami",
|
||||
Description: "will-be-cleared",
|
||||
@@ -2296,7 +2173,7 @@ func TestModifyImageAttribute_DescriptionEmptyValueClears(t *testing.T) {
|
||||
func TestModifyImageAttribute_CrossAccount(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-modx01",
|
||||
Name: "modx-ami",
|
||||
Description: "dont-touch",
|
||||
@@ -2321,7 +2198,7 @@ func TestModifyImageAttribute_CrossAccount(t *testing.T) {
|
||||
func TestModifyImageAttribute_SystemAMIImmutable(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-modsys01",
|
||||
Name: "sys-ami",
|
||||
Description: "baked-in",
|
||||
@@ -2385,7 +2262,7 @@ func TestModifyImageAttribute_MissingParameters(t *testing.T) {
|
||||
func TestResetImageAttribute_Description(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-reset01",
|
||||
Name: "reset-ami",
|
||||
Description: "will-be-cleared",
|
||||
@@ -2416,7 +2293,7 @@ func TestResetImageAttribute_Description(t *testing.T) {
|
||||
func TestResetImageAttribute_CrossAccount(t *testing.T) {
|
||||
svc, store := setupTestImageService(t)
|
||||
|
||||
createTestAMIConfigRich(t, store, viperblock.AMIMetadata{
|
||||
createTestAMIConfigRich(t, store, ebsmetadata.AMI{
|
||||
ImageID: "ami-rstx01",
|
||||
Name: "rstx",
|
||||
Description: "dont-touch",
|
||||
|
||||
@@ -8,9 +8,9 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
@@ -380,7 +380,7 @@ func TestPrepareRunInstances_PlatformTokenDefault(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc", PlatformDetails: tc.platformDetails},
|
||||
}},
|
||||
resourceMgr: &fakeResourceCapacityProvider{
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
package handlers_ec2_instance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
// TestPrepareEFIVolume_BackendInitFailure_NoGoroutineLeak exercises the first
|
||||
// of prepareEFIVolume's six error returns (Backend.Init failure) and asserts
|
||||
// the chunk uploader and WAL syncer goroutines started by newViperblock do
|
||||
// not survive the error return.
|
||||
func TestPrepareEFIVolume_BackendInitFailure_NoGoroutineLeak(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
// Swap in fake firmware files so the test doesn't depend on OVMF being
|
||||
// installed on the host; FirmwarePathCandidates is exported for exactly
|
||||
// this purpose.
|
||||
fwDir := t.TempDir()
|
||||
codePath := filepath.Join(fwDir, "CODE.fd")
|
||||
varsPath := filepath.Join(fwDir, "VARS.fd")
|
||||
require.NoError(t, os.WriteFile(codePath, []byte("fake-code"), 0o600))
|
||||
require.NoError(t, os.WriteFile(varsPath, []byte("fake-vars-template"), 0o600))
|
||||
|
||||
origCandidates := vm.FirmwarePathCandidates
|
||||
//nolint:reassign // test-only override of vm's exported var, restored via t.Cleanup; same pattern vm/firmware_test.go uses within its own package.
|
||||
vm.FirmwarePathCandidates = map[string][]vm.FirmwareCandidate{
|
||||
"x86_64": {{Code: codePath, VarsTemplate: varsPath}},
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
//nolint:reassign // restoring the original value swapped above
|
||||
vm.FirmwarePathCandidates = origCandidates
|
||||
})
|
||||
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{
|
||||
WalDir: t.TempDir(),
|
||||
Predastore: config.PredastoreConfig{
|
||||
// Host/creds are unset on purpose: Backend.Init's ListObjectsV2
|
||||
// call fails fast on empty static credentials, no network needed.
|
||||
Host: "127.0.0.1:1",
|
||||
Bucket: "test-bucket",
|
||||
Region: "us-east-1",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
instance := &vm.VM{}
|
||||
err := svc.prepareEFIVolume(context.Background(), "test-image", viperblock.VolumeConfig{}, instance, "x86_64")
|
||||
require.Error(t, err, "Backend.Init must fail against an unreachable predastore host")
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
package handlers_ec2_instance
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// varsTemplateBytes stands in for a firmware VARS template. The real one is
|
||||
// 540,672 bytes, but nothing here depends on the size beyond it being exact.
|
||||
func varsTemplateBytes() []byte { return bytes.Repeat([]byte{0xAB}, 4096) }
|
||||
|
||||
func providerEFIService(t *testing.T, provider ebsprovider.EBSProvider) *InstanceServiceImpl {
|
||||
t.Helper()
|
||||
return providerRootVolumeService(t, provider, rootVolumeAMILoader(), objectstore.NewMemoryObjectStore())
|
||||
}
|
||||
|
||||
// TestCreateEFIVolume_Provider_SeedsFromTemplate is the core of the EFI
|
||||
// decouple: the variable store is allocated through the provider with the
|
||||
// template as seed bytes, so no control-plane engine is built for it.
|
||||
func TestCreateEFIVolume_Provider_SeedsFromTemplate(t *testing.T) {
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeSeeding: true})
|
||||
svc := providerEFIService(t, provider)
|
||||
template := varsTemplateBytes()
|
||||
|
||||
require.NoError(t, svc.createEFIVolumeViaProvider(context.Background(), "vol-efiseed001-efi", template))
|
||||
|
||||
volume, err := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-efiseed001-efi",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(len(template)), volume.CapacityBytes,
|
||||
"the store must be sized byte-exactly to the VARS template")
|
||||
|
||||
seed, ok := provider.SeedData("vol-efiseed001-efi")
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, template, seed, "the template must reach the provider as seed bytes")
|
||||
}
|
||||
|
||||
// A provider that rounds capacity to a GiB boundary produces a VARS region
|
||||
// pflash refuses, which would surface as an unbootable guest rather than a
|
||||
// failed launch. The mismatch has to fail the launch instead.
|
||||
func TestCreateEFIVolume_Provider_CapacityMismatchFails(t *testing.T) {
|
||||
provider := &roundingProvider{
|
||||
EBSProvider: ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeSeeding: true}),
|
||||
}
|
||||
svc := providerEFIService(t, provider)
|
||||
|
||||
err := svc.createEFIVolumeViaProvider(context.Background(), "vol-efiround01-efi", varsTemplateBytes())
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
}
|
||||
|
||||
// A relaunch reissues the same create. Reseeding would overwrite the BootOrder
|
||||
// the guest wrote on its first boot, so the second call must be a no-op.
|
||||
func TestCreateEFIVolume_Provider_RepeatDoesNotReseed(t *testing.T) {
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeSeeding: true})
|
||||
svc := providerEFIService(t, provider)
|
||||
ctx := context.Background()
|
||||
template := varsTemplateBytes()
|
||||
|
||||
require.NoError(t, svc.createEFIVolumeViaProvider(ctx, "vol-efirepeat1-efi", template))
|
||||
require.NoError(t, svc.createEFIVolumeViaProvider(ctx, "vol-efirepeat1-efi", template))
|
||||
|
||||
seed, ok := provider.SeedData("vol-efirepeat1-efi")
|
||||
require.True(t, ok)
|
||||
assert.Equal(t, template, seed)
|
||||
}
|
||||
|
||||
// A seed above the wire bound must fail the launch with an error naming the
|
||||
// limit, not reach NATS and fail as an oversized publish.
|
||||
func TestCreateEFIVolume_Provider_OversizedTemplateFails(t *testing.T) {
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeSeeding: true})
|
||||
svc := providerEFIService(t, provider)
|
||||
|
||||
err := svc.createEFIVolumeViaProvider(context.Background(), "vol-efitoobig1-efi", make([]byte, ebsprovider.MaxSeedBytes+1))
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
}
|
||||
|
||||
// TestPrepareEFIVolume_Provider_RegistersEBSRequest covers the whole function
|
||||
// against this host's real firmware, including the name the VM launches with.
|
||||
func TestPrepareEFIVolume_Provider_RegistersEBSRequest(t *testing.T) {
|
||||
if _, _, _, err := vm.FirmwarePaths("x86_64"); err != nil {
|
||||
t.Skipf("no UEFI firmware on this host: %v", err)
|
||||
}
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{VolumeSeeding: true})
|
||||
svc := providerEFIService(t, provider)
|
||||
instance := &vm.VM{}
|
||||
|
||||
require.NoError(t, svc.prepareEFIVolume(context.Background(), "vol-efiprepare001", instance, "x86_64"))
|
||||
|
||||
instance.EBSRequests.Mu.Lock()
|
||||
defer instance.EBSRequests.Mu.Unlock()
|
||||
require.Len(t, instance.EBSRequests.Requests, 1)
|
||||
assert.Equal(t, "vol-efiprepare001-efi", instance.EBSRequests.Requests[0].Name)
|
||||
assert.True(t, instance.EBSRequests.Requests[0].EFI)
|
||||
assert.False(t, instance.EBSRequests.Requests[0].Boot)
|
||||
|
||||
seed, ok := provider.SeedData("vol-efiprepare001-efi")
|
||||
require.True(t, ok)
|
||||
assert.NotEmpty(t, seed, "the store must be seeded from this host's VARS template")
|
||||
}
|
||||
|
||||
// roundingProvider reports a GiB-rounded capacity the way a provider with no
|
||||
// sub-GiB volume granularity would.
|
||||
type roundingProvider struct {
|
||||
ebsprovider.EBSProvider
|
||||
}
|
||||
|
||||
func (p *roundingProvider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
volume, err := p.EBSProvider.CreateVolume(ctx, req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
volume.CapacityBytes = 1 << 30
|
||||
return volume, nil
|
||||
}
|
||||
@@ -0,0 +1,307 @@
|
||||
package handlers_ec2_instance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
spxtypes "github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.uber.org/goleak"
|
||||
)
|
||||
|
||||
const (
|
||||
testRootVolumeBytes = 4 * 1024 * 1024 * 1024
|
||||
testRootBucket = "test-bucket"
|
||||
testRootAZ = "ap-southeast-2a"
|
||||
testRootAccount = "000000000001"
|
||||
)
|
||||
|
||||
// providerRootVolumeService builds an instance service whose only storage
|
||||
// dependency is the provider. Predastore points at a closed port so any
|
||||
// fallback to the embedded engine fails loudly instead of silently passing.
|
||||
func providerRootVolumeService(t *testing.T, provider ebsprovider.EBSProvider, loader AMIMetaLoader, store objectstore.ObjectStore) *InstanceServiceImpl {
|
||||
t.Helper()
|
||||
return &InstanceServiceImpl{
|
||||
config: &config.Config{
|
||||
WalDir: t.TempDir(),
|
||||
AZ: testRootAZ,
|
||||
Predastore: config.PredastoreConfig{
|
||||
Host: "127.0.0.1:1",
|
||||
Bucket: testRootBucket,
|
||||
Region: "us-east-1",
|
||||
},
|
||||
},
|
||||
objectStore: store,
|
||||
metadata: ebsmetadata.NewStore(store, testRootBucket),
|
||||
amiLoader: loader,
|
||||
ebsProvider: provider,
|
||||
}
|
||||
}
|
||||
|
||||
// seedProviderSnapshot registers a source volume and a snapshot of it, which is
|
||||
// what an AMI's snapshot looks like to the provider.
|
||||
func seedProviderSnapshot(t *testing.T, provider ebsprovider.EBSProvider, sourceVolumeID, snapshotID string) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: sourceVolumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: testRootVolumeBytes},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, VolumeID: sourceVolumeID,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// rootVolumeAMILoader returns a loader for one AMI backed by snap-source on
|
||||
// vol-origin, the shape every provider root-volume test needs.
|
||||
func rootVolumeAMILoader() *fakeAMILoader {
|
||||
return &fakeAMILoader{
|
||||
byID: map[string]ebsmetadata.AMI{"ami-1": {ImageID: "ami-1", SnapshotID: "snap-source"}},
|
||||
sourceByID: map[string]string{"ami-1": "vol-origin"},
|
||||
}
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_CreatesFromAMISnapshot locks that an injected
|
||||
// provider owns root-volume creation: the volume is restored from the AMI's
|
||||
// snapshot rather than copied block-by-block by the control plane.
|
||||
func TestPrepareRootVolume_Provider_CreatesFromAMISnapshot(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{CrashConsistentSnapshot: true})
|
||||
seedProviderSnapshot(t, provider, "vol-origin", "snap-source")
|
||||
svc := providerRootVolumeService(t, provider, rootVolumeAMILoader(), objectstore.NewMemoryObjectStore())
|
||||
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
input := &ec2.RunInstancesInput{ImageId: aws.String("ami-1")}
|
||||
err := svc.prepareRootVolume(context.Background(), input, "vol-root", testRootVolumeBytes, 0,
|
||||
instance, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
created, err := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-root",
|
||||
})
|
||||
require.NoError(t, err, "the root volume must exist in the provider")
|
||||
assert.Equal(t, int64(testRootVolumeBytes), created.CapacityBytes)
|
||||
|
||||
require.Len(t, instance.EBSRequests.Requests, 1)
|
||||
assert.Equal(t, "vol-root", instance.EBSRequests.Requests[0].Name)
|
||||
assert.True(t, instance.EBSRequests.Requests[0].Boot, "the root volume must be marked bootable")
|
||||
assert.False(t, instance.EBSRequests.Requests[0].DeleteOnTermination)
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_WritesMetadataDocument locks the control-plane
|
||||
// record of the root volume. Without it the volume has neither a document nor a
|
||||
// legacy config.json, so DescribeVolumes cannot see it at all.
|
||||
func TestPrepareRootVolume_Provider_WritesMetadataDocument(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
seedProviderSnapshot(t, provider, "vol-origin", "snap-source")
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := providerRootVolumeService(t, provider, rootVolumeAMILoader(), store)
|
||||
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
err := svc.prepareRootVolume(context.Background(), &ec2.RunInstancesInput{ImageId: aws.String("ami-1")},
|
||||
"vol-root", testRootVolumeBytes, 0, instance, true)
|
||||
require.NoError(t, err)
|
||||
|
||||
doc, err := ebsmetadata.NewStore(store, testRootBucket).GetVolume(context.Background(), "vol-root")
|
||||
require.NoError(t, err, "the root volume must have an ebsmetadata document")
|
||||
|
||||
assert.Equal(t, "vol-root", doc.VolumeID)
|
||||
assert.Equal(t, testRootAccount, doc.TenantID)
|
||||
assert.Equal(t, uint64(4), doc.CapacityGiB)
|
||||
assert.Equal(t, string(ebsprovider.VolumeStateAvailable), doc.State,
|
||||
"attach flips the volume to in-use later; creation must not pre-empt it")
|
||||
assert.False(t, doc.CreatedAt.IsZero())
|
||||
assert.Equal(t, testRootAZ, doc.AvailabilityZone)
|
||||
assert.Equal(t, spxtypes.VolumeTypeGP3, doc.VolumeType)
|
||||
assert.Equal(t, spxtypes.DefaultGP3IOPS, doc.IOPS)
|
||||
assert.Equal(t, spxtypes.DefaultGP3Throughput, doc.Throughput)
|
||||
assert.Equal(t, "snap-source", doc.SnapshotID)
|
||||
assert.True(t, doc.DeleteOnTermination)
|
||||
assert.False(t, doc.Encrypted, "no encryption key is configured in this service")
|
||||
assert.NotEmpty(t, doc.ProviderHandle, "the opaque provider handle must be recorded")
|
||||
assert.Empty(t, doc.AttachedInstance, "creation must not claim an attachment")
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_HonoursRequestedIOPS locks that a launch
|
||||
// naming Iops in its block device mapping keeps that value.
|
||||
func TestPrepareRootVolume_Provider_HonoursRequestedIOPS(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
seedProviderSnapshot(t, provider, "vol-origin", "snap-source")
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := providerRootVolumeService(t, provider, rootVolumeAMILoader(), store)
|
||||
|
||||
err := svc.prepareRootVolume(context.Background(), &ec2.RunInstancesInput{ImageId: aws.String("ami-1")},
|
||||
"vol-root", testRootVolumeBytes, 5000, &vm.VM{AccountID: testRootAccount}, true)
|
||||
require.NoError(t, err)
|
||||
|
||||
doc, err := ebsmetadata.NewStore(store, testRootBucket).GetVolume(context.Background(), "vol-root")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 5000, doc.IOPS)
|
||||
}
|
||||
|
||||
// failingPutObjectStore fails writes under failPrefix, so a test can break the
|
||||
// metadata write without disturbing any other object the launch touches.
|
||||
type failingPutObjectStore struct {
|
||||
objectstore.ObjectStore
|
||||
|
||||
failPrefix string
|
||||
}
|
||||
|
||||
func (s *failingPutObjectStore) PutObject(ctx context.Context, input *s3.PutObjectInput) (*s3.PutObjectOutput, error) {
|
||||
if strings.HasPrefix(aws.StringValue(input.Key), s.failPrefix) {
|
||||
return nil, errors.New("simulated metadata write failure")
|
||||
}
|
||||
return s.ObjectStore.PutObject(ctx, input)
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_RollbackOnMetadataWriteFailure locks that a
|
||||
// failed document write deletes the volume the provider just allocated, so a
|
||||
// failed launch cannot strand an untracked volume.
|
||||
func TestPrepareRootVolume_Provider_RollbackOnMetadataWriteFailure(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
seedProviderSnapshot(t, provider, "vol-origin", "snap-source")
|
||||
store := &failingPutObjectStore{
|
||||
ObjectStore: objectstore.NewMemoryObjectStore(),
|
||||
failPrefix: "spinifex/ebsmetadata/v1/volumes/",
|
||||
}
|
||||
svc := providerRootVolumeService(t, provider, rootVolumeAMILoader(), store)
|
||||
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
err := svc.prepareRootVolume(context.Background(), &ec2.RunInstancesInput{ImageId: aws.String("ami-1")},
|
||||
"vol-root", testRootVolumeBytes, 0, instance, true)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Empty(t, instance.EBSRequests.Requests, "a failed launch must not claim a boot volume")
|
||||
|
||||
_, getErr := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-root",
|
||||
})
|
||||
require.ErrorIs(t, getErr, ebsprovider.ErrNotFound, "rollback must delete the just-created provider volume")
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_RepeatIsIdempotent locks that re-running the
|
||||
// same launch does not re-clone or error: the provider returns the existing
|
||||
// volume when the requested capacity matches.
|
||||
func TestPrepareRootVolume_Provider_RepeatIsIdempotent(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
seedProviderSnapshot(t, provider, "vol-origin", "snap-source")
|
||||
svc := providerRootVolumeService(t, provider, rootVolumeAMILoader(), objectstore.NewMemoryObjectStore())
|
||||
input := &ec2.RunInstancesInput{ImageId: aws.String("ami-1")}
|
||||
|
||||
for range 2 {
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
err := svc.prepareRootVolume(context.Background(), input, "vol-root", testRootVolumeBytes, 0,
|
||||
instance, true)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, instance.EBSRequests.Requests, 1)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_AMIWithoutSnapshotFails locks that an AMI with
|
||||
// no snapshot still fails, and allocates nothing.
|
||||
func TestPrepareRootVolume_Provider_AMIWithoutSnapshotFails(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc := providerRootVolumeService(t, provider, &fakeAMILoader{
|
||||
byID: map[string]ebsmetadata.AMI{"ami-1": {ImageID: "ami-1"}},
|
||||
}, objectstore.NewMemoryObjectStore())
|
||||
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
err := svc.prepareRootVolume(context.Background(), &ec2.RunInstancesInput{ImageId: aws.String("ami-1")},
|
||||
"vol-root", testRootVolumeBytes, 0, instance, true)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Empty(t, instance.EBSRequests.Requests, "a failed launch must not claim a boot volume")
|
||||
|
||||
_, getErr := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-root",
|
||||
})
|
||||
require.ErrorIs(t, getErr, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_UnknownAMIFails locks the AMI lookup failure
|
||||
// mapping the legacy clone path already used.
|
||||
func TestPrepareRootVolume_Provider_UnknownAMIFails(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
svc := providerRootVolumeService(t, ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}),
|
||||
&fakeAMILoader{err: errors.New("missing")}, objectstore.NewMemoryObjectStore())
|
||||
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
err := svc.prepareRootVolume(context.Background(), &ec2.RunInstancesInput{ImageId: aws.String("ami-gone")},
|
||||
"vol-root", testRootVolumeBytes, 0, instance, true)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidAMIIDNotFound, err.Error())
|
||||
assert.Empty(t, instance.EBSRequests.Requests)
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_SourceVolumeUnresolvableFails locks that an
|
||||
// unresolvable snapshot source aborts rather than sending an incomplete
|
||||
// CreateVolume the provider would reject.
|
||||
func TestPrepareRootVolume_Provider_SourceVolumeUnresolvableFails(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
seedProviderSnapshot(t, provider, "vol-origin", "snap-source")
|
||||
svc := providerRootVolumeService(t, provider, &fakeAMILoader{
|
||||
byID: map[string]ebsmetadata.AMI{"ami-1": {ImageID: "ami-1", SnapshotID: "snap-source"}},
|
||||
sourceErr: errors.New("snapshot metadata missing"),
|
||||
}, objectstore.NewMemoryObjectStore())
|
||||
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
err := svc.prepareRootVolume(context.Background(), &ec2.RunInstancesInput{ImageId: aws.String("ami-1")},
|
||||
"vol-root", testRootVolumeBytes, 0, instance, true)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Empty(t, instance.EBSRequests.Requests)
|
||||
|
||||
_, getErr := provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-root",
|
||||
})
|
||||
require.ErrorIs(t, getErr, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
// TestPrepareRootVolume_Provider_CreateFailureIsFatal locks that a provider
|
||||
// error aborts the launch rather than falling back to the embedded engine.
|
||||
func TestPrepareRootVolume_Provider_CreateFailureIsFatal(t *testing.T) {
|
||||
defer goleak.VerifyNone(t)
|
||||
|
||||
// No snapshot seeded, so the provider rejects the clone request.
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc := providerRootVolumeService(t, provider, &fakeAMILoader{
|
||||
byID: map[string]ebsmetadata.AMI{"ami-1": {ImageID: "ami-1", SnapshotID: "snap-missing"}},
|
||||
sourceByID: map[string]string{"ami-1": "vol-origin"},
|
||||
}, objectstore.NewMemoryObjectStore())
|
||||
|
||||
instance := &vm.VM{AccountID: testRootAccount}
|
||||
err := svc.prepareRootVolume(context.Background(), &ec2.RunInstancesInput{ImageId: aws.String("ami-1")},
|
||||
"vol-root", testRootVolumeBytes, 0, instance, true)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Empty(t, instance.EBSRequests.Requests)
|
||||
}
|
||||
@@ -4,9 +4,9 @@ import (
|
||||
"context"
|
||||
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/gpu"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
)
|
||||
|
||||
// InstanceService defines the interface for EC2 instance operations business logic.
|
||||
@@ -137,9 +137,15 @@ type PublicIPReleaser interface {
|
||||
}
|
||||
|
||||
// AMIMetaLoader resolves an AMI ID to its metadata for ownership/validation
|
||||
// during RunInstances. Implemented by handlers/ec2/image.ImageServiceImpl.
|
||||
// during RunInstances. ImageServiceImpl converts on its embedded path, so
|
||||
// this boundary names no viperblock type whichever EBS provider is selected.
|
||||
type AMIMetaLoader interface {
|
||||
GetAMIConfig(ctx context.Context, imageID string) (viperblock.AMIMetadata, error)
|
||||
GetAMIConfig(ctx context.Context, imageID string) (ebsmetadata.AMI, error)
|
||||
|
||||
// GetAMISourceVolumeID returns the volume whose blocks the AMI's snapshot
|
||||
// references. An EBS provider needs it alongside the snapshot ID to resolve
|
||||
// the clone's base data.
|
||||
GetAMISourceVolumeID(ctx context.Context, imageID string) (string, error)
|
||||
}
|
||||
|
||||
// KeyPairValidator checks that a named key pair exists for an account during
|
||||
|
||||
@@ -18,6 +18,8 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/filterutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/gpu"
|
||||
handlers_dns "github.com/mulgadc/spinifex/spinifex/handlers/dns"
|
||||
@@ -28,13 +30,14 @@ import (
|
||||
spxtypes "github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/types"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/mulgadc/viperblock/viperblock/backends/s3"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// bytesPerGiB converts the byte sizes the launch path works in to the GiB
|
||||
// units ebsmetadata records.
|
||||
const bytesPerGiB = 1024 * 1024 * 1024
|
||||
|
||||
// VolumeInfo holds volume information returned from GenerateVolumes
|
||||
// for populating BlockDeviceMappings in the EC2 API response.
|
||||
type VolumeInfo struct {
|
||||
@@ -135,6 +138,8 @@ type InstanceServiceImpl struct {
|
||||
tagWriter InstanceTagWriter
|
||||
gpuClaimer GPUClaimer
|
||||
amiLoader AMIMetaLoader
|
||||
ebsProvider ebsprovider.EBSProvider
|
||||
metadata *ebsmetadata.Store
|
||||
keyValidator KeyPairValidator
|
||||
eniCreator ENICreator
|
||||
ipAllocator PublicIPAllocator
|
||||
@@ -142,6 +147,19 @@ type InstanceServiceImpl struct {
|
||||
dnsInternalDomain string
|
||||
}
|
||||
|
||||
// SetEBSProvider injects the provider boundary used for instance-created
|
||||
// volumes. Once set, root volumes are allocated through it instead of a
|
||||
// control-plane storage engine.
|
||||
func (s *InstanceServiceImpl) SetEBSProvider(provider ebsprovider.EBSProvider) {
|
||||
s.ebsProvider = provider
|
||||
}
|
||||
|
||||
// EBSProvider returns the injected provider boundary, or nil on the legacy
|
||||
// embedded-engine path. Primarily for composition-root tests to observe wiring.
|
||||
func (s *InstanceServiceImpl) EBSProvider() ebsprovider.EBSProvider {
|
||||
return s.ebsProvider
|
||||
}
|
||||
|
||||
// NewInstanceServiceImpl creates a new instance service implementation for daemon use.
|
||||
func NewInstanceServiceImpl(
|
||||
cfg *config.Config,
|
||||
@@ -152,6 +170,12 @@ func NewInstanceServiceImpl(
|
||||
resourceMgr InstanceTypeAllocator,
|
||||
stoppedStore StoppedInstanceStore,
|
||||
) *InstanceServiceImpl {
|
||||
// cfg is nil-tolerated by the DNS resolvers below, so keep the metadata
|
||||
// store's bucket lookup equally tolerant.
|
||||
bucket := ""
|
||||
if cfg != nil {
|
||||
bucket = cfg.Predastore.Bucket
|
||||
}
|
||||
return &InstanceServiceImpl{
|
||||
config: cfg,
|
||||
instanceTypes: instanceTypes,
|
||||
@@ -160,6 +184,7 @@ func NewInstanceServiceImpl(
|
||||
vmMgr: vmMgr,
|
||||
resourceMgr: resourceMgr,
|
||||
stoppedStore: stoppedStore,
|
||||
metadata: ebsmetadata.NewStore(store, bucket),
|
||||
dnsBaseDomain: handlers_dns.ResolveBaseDomain(cfg),
|
||||
dnsInternalDomain: handlers_dns.ResolveInternalDomain(cfg),
|
||||
}
|
||||
@@ -621,7 +646,7 @@ func (s *InstanceServiceImpl) PrepareRunInstances(ctx context.Context, input *ec
|
||||
if input.SubnetId != nil && *input.SubnetId != "" && s.eniCreator != nil {
|
||||
eniOut, eniErr := s.eniCreator.CreateNetworkInterface(ctx, &ec2.CreateNetworkInterfaceInput{
|
||||
SubnetId: input.SubnetId,
|
||||
Description: aws.String("Primary network interface for " + instance.ID),
|
||||
Description: aws.String(handlers_ec2_vpc.AutoENIDescriptionPrefix + instance.ID),
|
||||
Groups: input.SecurityGroupIds,
|
||||
}, accountID)
|
||||
if eniErr != nil {
|
||||
@@ -1392,27 +1417,13 @@ func (s *InstanceServiceImpl) GenerateVolumes(ctx context.Context, input *ec2.Ru
|
||||
// Capture attach time for the root volume
|
||||
attachTime := time.Now()
|
||||
|
||||
volumeConfig := viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: p.imageId,
|
||||
SizeGiB: utils.SafeIntToUint64(p.size / 1024 / 1024 / 1024),
|
||||
CreatedAt: attachTime,
|
||||
DeviceName: p.deviceName,
|
||||
VolumeType: p.volumeType,
|
||||
IOPS: p.iops,
|
||||
SnapshotID: p.snapshotId,
|
||||
DeleteOnTermination: p.deleteOnTermination,
|
||||
TenantID: instance.AccountID,
|
||||
},
|
||||
}
|
||||
|
||||
size := p.size
|
||||
imageId := p.imageId
|
||||
deviceName := p.deviceName
|
||||
deleteOnTermination := p.deleteOnTermination
|
||||
|
||||
// Step 1: Create or validate root volume
|
||||
err := s.prepareRootVolume(ctx, input, imageId, size, volumeConfig, instance, deleteOnTermination)
|
||||
err := s.prepareRootVolume(ctx, input, imageId, size, p.iops, instance, deleteOnTermination)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -1421,7 +1432,7 @@ func (s *InstanceServiceImpl) GenerateVolumes(ctx context.Context, input *ec2.Ru
|
||||
// guests must not allocate an orphan VARS volume).
|
||||
if instance.BootMode == "uefi" || instance.BootMode == "uefi-preferred" {
|
||||
arch := instanceArchitecture(s.instanceTypes[*input.InstanceType])
|
||||
err = s.prepareEFIVolume(ctx, imageId, volumeConfig, instance, arch)
|
||||
err = s.prepareEFIVolume(ctx, imageId, instance, arch)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -1440,133 +1451,137 @@ func (s *InstanceServiceImpl) GenerateVolumes(ctx context.Context, input *ec2.Ru
|
||||
return volumeInfos, nil
|
||||
}
|
||||
|
||||
// newViperblock creates a viperblock instance with the service's S3/Predastore credentials.
|
||||
func (s *InstanceServiceImpl) newViperblock(volumeName string, size int, volumeConfig viperblock.VolumeConfig) (*viperblock.VB, error) {
|
||||
cfg := s3.S3Config{
|
||||
VolumeName: volumeName,
|
||||
VolumeSize: utils.SafeIntToUint64(size),
|
||||
Bucket: s.config.Predastore.Bucket,
|
||||
Region: s.config.Predastore.Region,
|
||||
AccessKey: s.config.Predastore.AccessKey,
|
||||
SecretKey: s.config.Predastore.SecretKey,
|
||||
Host: s.config.Predastore.Host,
|
||||
// prepareRootVolume allocates the root volume through the EBS provider and
|
||||
// records the boot volume the launcher must attach.
|
||||
func (s *InstanceServiceImpl) prepareRootVolume(ctx context.Context, input *ec2.RunInstancesInput, imageId string, size, iops int, instance *vm.VM, deleteOnTermination bool) error {
|
||||
if s.ebsProvider == nil {
|
||||
slog.ErrorContext(ctx, "no EBS provider configured", "op", "prepareRootVolume")
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
mkey, err := utils.LoadViperblockMasterKey(s.config.Viperblock.EncryptionKeyFile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
if err := s.createRootVolumeViaProvider(ctx, rootVolumeSpec{
|
||||
amiID: aws.StringValue(input.ImageId),
|
||||
volumeID: imageId,
|
||||
accountID: instance.AccountID,
|
||||
sizeBytes: size,
|
||||
iops: iops,
|
||||
deleteOnTermination: deleteOnTermination,
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
vbconfig := viperblock.VB{
|
||||
VolumeName: volumeName,
|
||||
VolumeSize: utils.SafeIntToUint64(size),
|
||||
BaseDir: s.config.WalDir,
|
||||
Cache: viperblock.Cache{Config: viperblock.CacheConfig{Size: 0}},
|
||||
VolumeConfig: volumeConfig,
|
||||
MasterKey: mkey,
|
||||
EncryptionEnabled: mkey != nil,
|
||||
}
|
||||
|
||||
vb, err := viperblock.New(&vbconfig, "s3", cfg)
|
||||
return vb, err
|
||||
appendRootEBSRequest(instance, imageId, deleteOnTermination)
|
||||
return nil
|
||||
}
|
||||
|
||||
// prepareRootVolume handles creation/cloning of the root volume.
|
||||
func (s *InstanceServiceImpl) prepareRootVolume(ctx context.Context, input *ec2.RunInstancesInput, imageId string, size int, volumeConfig viperblock.VolumeConfig, instance *vm.VM, deleteOnTermination bool) error {
|
||||
vb, err := s.newViperblock(imageId, size, volumeConfig)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to connect to Viperblock store", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
defer func() {
|
||||
vb.StopChunkUploader()
|
||||
vb.StopWALSyncer()
|
||||
}()
|
||||
|
||||
// Initialize the backend
|
||||
err = vb.Backend.Init()
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to initialize backend", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Only os.ErrNotExist means "clone from AMI"; any other error must abort.
|
||||
// Treating a tamper/mismatch as missing would overwrite live volume state.
|
||||
_, err = vb.LoadStateRequest("")
|
||||
if err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
slog.ErrorContext(ctx, "Failed to load root volume state from backend",
|
||||
"imageId", imageId, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if err != nil {
|
||||
slog.InfoContext(ctx, "Volume does not yet exist, creating from AMI ...")
|
||||
if err = s.cloneAMIToVolume(ctx, input, vb); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Append root volume to instance
|
||||
// appendRootEBSRequest records the boot volume the launcher must attach.
|
||||
func appendRootEBSRequest(instance *vm.VM, volumeID string, deleteOnTermination bool) {
|
||||
instance.EBSRequests.Mu.Lock()
|
||||
instance.EBSRequests.Requests = append(instance.EBSRequests.Requests, spxtypes.EBSRequest{
|
||||
Name: imageId,
|
||||
Name: volumeID,
|
||||
Boot: true,
|
||||
DeleteOnTermination: deleteOnTermination,
|
||||
})
|
||||
instance.EBSRequests.Mu.Unlock()
|
||||
}
|
||||
|
||||
// rootVolumeSpec describes the boot volume a launch needs, in provider-neutral
|
||||
// terms taken from the launch request.
|
||||
type rootVolumeSpec struct {
|
||||
amiID string
|
||||
volumeID string
|
||||
accountID string
|
||||
sizeBytes int
|
||||
iops int
|
||||
deleteOnTermination bool
|
||||
}
|
||||
|
||||
// createRootVolumeViaProvider allocates the root volume through the injected
|
||||
// provider, cloning the AMI's snapshot, then records it in ebsmetadata. No
|
||||
// control-plane engine is built, so a launch never becomes a second writer.
|
||||
func (s *InstanceServiceImpl) createRootVolumeViaProvider(ctx context.Context, spec rootVolumeSpec) error {
|
||||
amiConfig, err := s.amiLoader.GetAMIConfig(ctx, spec.amiID)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Could not load AMI config", "imageId", spec.amiID, "err", err)
|
||||
return errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
}
|
||||
if amiConfig.SnapshotID == "" {
|
||||
slog.ErrorContext(ctx, "AMI has no snapshot ID, cannot perform zero-copy clone", "imageId", spec.amiID)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// The provider resolves a clone's base blocks against the snapshot's source
|
||||
// volume, so the wire contract requires both IDs together.
|
||||
sourceVolumeID, err := s.amiLoader.GetAMISourceVolumeID(ctx, spec.amiID)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Could not resolve AMI snapshot source volume",
|
||||
"imageId", spec.amiID, "snapshotId", amiConfig.SnapshotID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// The control plane cannot see how a provider encrypts its volumes, but this
|
||||
// shared config knob is the same one the legacy path derives Encrypted from.
|
||||
mkey, err := utils.LoadViperblockMasterKey(s.config.Viperblock.EncryptionKeyFile)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Could not load encryption key for root volume", "volumeId", spec.volumeID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "Creating root volume from AMI snapshot via EBS provider",
|
||||
"imageId", spec.amiID, "volumeId", spec.volumeID, "snapshotId", amiConfig.SnapshotID, "sourceVolumeId", sourceVolumeID)
|
||||
|
||||
created, err := s.ebsProvider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: spec.volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: int64(spec.sizeBytes)},
|
||||
AvailabilityZone: s.config.AZ,
|
||||
SourceSnapshotID: amiConfig.SnapshotID,
|
||||
SourceSnapshotVolumeID: sourceVolumeID,
|
||||
})
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Provider root volume creation failed",
|
||||
"volumeId", spec.volumeID, "snapshotId", amiConfig.SnapshotID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if created == nil {
|
||||
slog.ErrorContext(ctx, "Provider returned no volume for the root volume", "volumeId", spec.volumeID)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// DescribeVolumes enumerates ebsmetadata documents only, so a root volume
|
||||
// without one is invisible and its attach-time state write fails.
|
||||
if err := s.metadata.PutVolume(ctx, ebsmetadata.Volume{
|
||||
VolumeID: spec.volumeID, TenantID: spec.accountID,
|
||||
CapacityGiB: utils.SafeIntToUint64(spec.sizeBytes / bytesPerGiB),
|
||||
State: string(ebsprovider.VolumeStateAvailable),
|
||||
CreatedAt: time.Now(), AvailabilityZone: s.config.AZ,
|
||||
VolumeType: spxtypes.VolumeTypeGP3, IOPS: rootVolumeIOPS(spec.iops),
|
||||
Throughput: spxtypes.DefaultGP3Throughput, SnapshotID: amiConfig.SnapshotID,
|
||||
DeleteOnTermination: spec.deleteOnTermination, Encrypted: mkey != nil,
|
||||
ProviderHandle: created.Handle,
|
||||
}); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to persist root volume metadata", "volumeId", spec.volumeID, "err", err)
|
||||
if delErr := s.ebsProvider.DeleteVolume(ctx, ebsprovider.DeleteVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: spec.volumeID, Handle: created.Handle,
|
||||
}); delErr != nil {
|
||||
slog.ErrorContext(ctx, "Failed to roll back untracked root volume", "volumeId", spec.volumeID, "err", delErr)
|
||||
}
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// cloneAMIToVolume creates a new volume from an AMI using snapshot-based
|
||||
// zero-copy cloning. The destination volume points at the AMI's frozen block
|
||||
// map and reads on-demand from the AMI's chunks (copy-on-write).
|
||||
func (s *InstanceServiceImpl) cloneAMIToVolume(ctx context.Context, input *ec2.RunInstancesInput, destVb *viperblock.VB) error {
|
||||
amiConfig, err := s.amiLoader.GetAMIConfig(ctx, *input.ImageId)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Could not load AMI config", "imageId", *input.ImageId, "err", err)
|
||||
return errors.New(awserrors.ErrorInvalidAMIIDNotFound)
|
||||
// rootVolumeIOPS falls back to the gp3 baseline when the launch request named
|
||||
// no Iops, so DescribeVolumes never reports a root volume as having zero.
|
||||
func rootVolumeIOPS(requested int) int {
|
||||
if requested <= 0 {
|
||||
return spxtypes.DefaultGP3IOPS
|
||||
}
|
||||
|
||||
snapshotID := amiConfig.SnapshotID
|
||||
if snapshotID == "" {
|
||||
slog.ErrorContext(ctx, "AMI has no snapshot ID, cannot perform zero-copy clone", "imageId", *input.ImageId)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "Cloning AMI via snapshot", "imageId", *input.ImageId, "snapshotID", snapshotID)
|
||||
|
||||
// Set up destination volume from the snapshot (zero-copy)
|
||||
err = destVb.OpenFromSnapshot(snapshotID)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to open from snapshot", "snapshotID", snapshotID, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Persist the snapshot relationship to the backend
|
||||
err = destVb.SaveState()
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to save state", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
err = destVb.SaveBlockState()
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to save block state", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
err = destVb.RemoveLocalFiles()
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Failed to remove local files", "err", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
return requested
|
||||
}
|
||||
|
||||
// prepareEFIVolume creates the per-VM EFI variable store, sized exactly to the
|
||||
// firmware VARS template (pflash requires byte-exact size) and seeded from it.
|
||||
// arch is "x86_64" | "arm64".
|
||||
func (s *InstanceServiceImpl) prepareEFIVolume(ctx context.Context, imageId string, volumeConfig viperblock.VolumeConfig, instance *vm.VM, arch string) error {
|
||||
func (s *InstanceServiceImpl) prepareEFIVolume(ctx context.Context, volumeID string, instance *vm.VM, arch string) error {
|
||||
codePath, varsTemplate, varsSize, err := vm.FirmwarePaths(arch)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "UEFI firmware not installed on this host", "arch", arch, "err", err)
|
||||
@@ -1583,80 +1598,18 @@ func (s *InstanceServiceImpl) prepareEFIVolume(ctx context.Context, imageId stri
|
||||
}
|
||||
slog.InfoContext(ctx, "Preparing EFI variable store", "arch", arch, "firmwarePath", codePath, "varsTemplate", varsTemplate, "size", varsSize)
|
||||
|
||||
efiVolumeName := fmt.Sprintf("%s-efi", imageId)
|
||||
efiVolumeConfig := volumeConfig
|
||||
efiVolumeConfig.VolumeMetadata.VolumeID = efiVolumeName
|
||||
// Zero SizeGiB prevents viperblock from rounding the EFI volume size
|
||||
// up to GiB boundaries; pflash rejects any size beyond the VARS region.
|
||||
efiVolumeConfig.VolumeMetadata.SizeGiB = 0
|
||||
|
||||
efiVb, err := s.newViperblock(efiVolumeName, int(varsSize), efiVolumeConfig)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Could not create EFI viperblock", "err", err)
|
||||
efiVolumeName := fmt.Sprintf("%s-efi", volumeID)
|
||||
if s.ebsProvider == nil {
|
||||
slog.ErrorContext(ctx, "no EBS provider configured", "op", "prepareEFIVolume")
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
// newViperblock starts the chunk uploader and WAL syncer goroutines
|
||||
// unconditionally, so every return path below must release them, not
|
||||
// just the happy path Close() at the bottom of this function.
|
||||
defer func() {
|
||||
efiVb.StopChunkUploader()
|
||||
efiVb.StopWALSyncer()
|
||||
}()
|
||||
|
||||
slog.DebugContext(ctx, "Initializing EFI Viperblock store backend")
|
||||
if err := efiVb.Backend.Init(); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to initialize EFI Viperblock store backend", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Only os.ErrNotExist means "seed from template"; other errors must abort.
|
||||
// Treating a transient failure as missing would clobber guest-set BootOrder.
|
||||
_, loadErr := efiVb.LoadStateRequest("")
|
||||
if loadErr != nil && !errors.Is(loadErr, os.ErrNotExist) {
|
||||
slog.ErrorContext(ctx, "Failed to load EFI volume state from backend", "name", efiVolumeName, "err", loadErr)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if loadErr != nil {
|
||||
slog.InfoContext(ctx, "EFI volume does not yet exist, seeding from firmware VARS template", "name", efiVolumeName)
|
||||
|
||||
var walErr error
|
||||
if efiVb.UseShardedWAL {
|
||||
walErr = efiVb.OpenShardedWAL()
|
||||
} else {
|
||||
walErr = efiVb.OpenWAL(&efiVb.WAL, fmt.Sprintf("%s/%s", efiVb.WAL.BaseDir, types.GetFilePath(types.FileTypeWALChunk, efiVb.WAL.WallNum.Load(), efiVb.GetVolume())))
|
||||
}
|
||||
if walErr != nil {
|
||||
slog.ErrorContext(ctx, "Failed to load WAL", "err", walErr)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := efiVb.OpenWAL(&efiVb.BlockToObjectWAL, fmt.Sprintf("%s/%s", efiVb.WAL.BaseDir, types.GetFilePath(types.FileTypeWALBlock, efiVb.BlockToObjectWAL.WallNum.Load(), efiVb.GetVolume()))); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to load block WAL", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := efiVb.WriteAt(0, template); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to seed EFI volume with VARS template", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if err := efiVb.Flush(); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to flush EFI volume", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
}
|
||||
|
||||
// Close is the durability boundary; a partial VARS write causes pflash to refuse launch.
|
||||
if err := efiVb.Close(); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to close EFI Viperblock store", "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if err := efiVb.RemoveLocalFiles(); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to remove local files", "err", err)
|
||||
if err := s.createEFIVolumeViaProvider(ctx, efiVolumeName, template); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
instance.EBSRequests.Mu.Lock()
|
||||
instance.EBSRequests.Requests = append(instance.EBSRequests.Requests, spxtypes.EBSRequest{
|
||||
Name: efiVb.VolumeName,
|
||||
Name: efiVolumeName,
|
||||
Boot: false,
|
||||
EFI: true,
|
||||
})
|
||||
@@ -1665,6 +1618,41 @@ func (s *InstanceServiceImpl) prepareEFIVolume(ctx context.Context, imageId stri
|
||||
return nil
|
||||
}
|
||||
|
||||
// createEFIVolumeViaProvider allocates the EFI variable store through the
|
||||
// provider, seeding it with this node's firmware VARS template. The bytes
|
||||
// travel with the request so the region matches the firmware that will run.
|
||||
func (s *InstanceServiceImpl) createEFIVolumeViaProvider(ctx context.Context, efiVolumeName string, template []byte) error {
|
||||
slog.InfoContext(ctx, "Creating EFI variable store via EBS provider", "name", efiVolumeName, "seedBytes", len(template))
|
||||
|
||||
// The store is internal to the launch, never attachable and filtered out of
|
||||
// DescribeVolumes, so it gets no ebsmetadata document the way a root volume
|
||||
// does. Its lifecycle is the root volume's DeleteVolume.
|
||||
created, err := s.ebsProvider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: efiVolumeName,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: int64(len(template))},
|
||||
AvailabilityZone: s.config.AZ,
|
||||
SeedData: template,
|
||||
})
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Provider EFI volume creation failed", "name", efiVolumeName, "err", err)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if created == nil {
|
||||
slog.ErrorContext(ctx, "Provider returned no volume for the EFI variable store", "name", efiVolumeName)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// pflash rejects a VARS region that is not byte-exact, so a provider that
|
||||
// rounded the capacity up to a GiB boundary yields a guest that cannot boot.
|
||||
if created.CapacityBytes != int64(len(template)) {
|
||||
slog.ErrorContext(ctx, "Provider EFI volume capacity does not match the VARS template",
|
||||
"name", efiVolumeName, "wantBytes", len(template), "gotBytes", created.CapacityBytes)
|
||||
return errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// instanceArchitecture returns the AWS architecture string from the instance type.
|
||||
// Returns "" on nil/malformed spec; the caller's firmware probe surfaces the error.
|
||||
func instanceArchitecture(it *ec2.InstanceTypeInfo) string {
|
||||
|
||||
@@ -14,12 +14,12 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/gpu"
|
||||
"github.com/mulgadc/spinifex/spinifex/tags"
|
||||
spxtypes "github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
@@ -275,7 +275,7 @@ func TestRunInstance_UniqueIDs(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestFloorVolumeSizeToAMI(t *testing.T) {
|
||||
loader := &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
loader := &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-rocky": {VolumeSizeGiB: 10},
|
||||
"ami-debian": {VolumeSizeGiB: 3},
|
||||
"ami-no-size": {VolumeSizeGiB: 0},
|
||||
@@ -2492,7 +2492,7 @@ func (raceVolumeMounter) UnmountOne(spxtypes.EBSRequest) error { return nil }
|
||||
// TestStartStoppedInstance_ConcurrentClaimRace is the regression test for
|
||||
// the double-start bug this claim closes: two nodes (or a forwarded call
|
||||
// racing a local fallback) could both observe the same stopped instance as
|
||||
// claimable and both launch it onto the same viperblock volume. It fires two
|
||||
// claimable and both launch it onto the same EBS volume. It fires two
|
||||
// concurrent StartStoppedInstance calls at the same stopped instance id and
|
||||
// asserts the atomic claim lets exactly one of them proceed past it —
|
||||
// reaching resourceMgr.Allocate / vmMgr.Insert / vmMgr.Run — while the other
|
||||
@@ -2576,18 +2576,35 @@ func TestStartStoppedInstance_ConcurrentClaimRace(t *testing.T) {
|
||||
// --- PrepareRunInstances / ec2.cmd dispatch tests ---------------------------
|
||||
|
||||
type fakeAMILoader struct {
|
||||
byID map[string]viperblock.AMIMetadata
|
||||
err error
|
||||
byID map[string]ebsmetadata.AMI
|
||||
err error
|
||||
sourceByID map[string]string
|
||||
sourceErr error
|
||||
}
|
||||
|
||||
func (f *fakeAMILoader) GetAMIConfig(_ context.Context, id string) (viperblock.AMIMetadata, error) {
|
||||
func (f *fakeAMILoader) GetAMIConfig(_ context.Context, id string) (ebsmetadata.AMI, error) {
|
||||
if f.err != nil {
|
||||
return viperblock.AMIMetadata{}, f.err
|
||||
return ebsmetadata.AMI{}, f.err
|
||||
}
|
||||
if meta, ok := f.byID[id]; ok {
|
||||
return meta, nil
|
||||
}
|
||||
return viperblock.AMIMetadata{}, errors.New("not found")
|
||||
return ebsmetadata.AMI{}, errors.New("not found")
|
||||
}
|
||||
|
||||
// GetAMISourceVolumeID defaults to the bundled system AMI convention, where the
|
||||
// snapshot's source volume is named after the AMI itself.
|
||||
func (f *fakeAMILoader) GetAMISourceVolumeID(_ context.Context, id string) (string, error) {
|
||||
if f.sourceErr != nil {
|
||||
return "", f.sourceErr
|
||||
}
|
||||
if volumeID, ok := f.sourceByID[id]; ok {
|
||||
return volumeID, nil
|
||||
}
|
||||
if _, ok := f.byID[id]; !ok {
|
||||
return "", errors.New("not found")
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
type fakeKeyValidator struct {
|
||||
@@ -2665,7 +2682,7 @@ func TestPrepareRunInstances_AMINotOwnedByCaller(t *testing.T) {
|
||||
types, _ := defaultPrepareInstanceTypes()
|
||||
svc := &InstanceServiceImpl{
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-other": {ImageOwnerAlias: "999988887777"},
|
||||
}},
|
||||
}
|
||||
@@ -2681,7 +2698,7 @@ func TestPrepareRunInstances_KeyPairNotFound(t *testing.T) {
|
||||
types, _ := defaultPrepareInstanceTypes()
|
||||
svc := &InstanceServiceImpl{
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
keyValidator: &fakeKeyValidator{err: errors.New("no key")},
|
||||
@@ -2706,7 +2723,7 @@ func TestPrepareRunInstances_InsufficientCapacity(t *testing.T) {
|
||||
_ = it
|
||||
svc := &InstanceServiceImpl{
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2730,7 +2747,7 @@ func TestPrepareRunInstances_HappyPathNoENI(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2764,7 +2781,7 @@ func TestPrepareRunInstances_PersistsIamInstanceProfile(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2798,7 +2815,7 @@ func TestPrepareRunInstances_ConsumesReservation(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2829,7 +2846,7 @@ func TestPrepareRunInstances_ReservationCapsNoSpill(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2856,7 +2873,7 @@ func TestPrepareRunInstances_ReservationExceeded(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2886,7 +2903,7 @@ func TestPrepareRunInstances_ReservationRollbackNoGeneralPoolLeak(t *testing.T)
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{Region: "us-east-1", AZ: "us-east-1a"},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2918,7 +2935,7 @@ func TestPrepareRunInstances_AmiLaunchIndexContiguous(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -2994,7 +3011,7 @@ func TestPrepareRunInstances_BootModePropagated(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc", BootMode: tc.amiBootMode},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -3189,7 +3206,7 @@ func prepareSvcWithENI(t *testing.T, eni *fakeENICreator, ipam *fakeIPAllocator)
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{Region: "us-east-1", AZ: "us-east-1a"},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -3571,7 +3588,7 @@ func TestPrepareRunInstances_PlacementGroup(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
@@ -3598,7 +3615,7 @@ func TestPrepareRunInstances_AllocateFailsMidLoop(t *testing.T) {
|
||||
svc := &InstanceServiceImpl{
|
||||
config: &config.Config{},
|
||||
instanceTypes: types,
|
||||
amiLoader: &fakeAMILoader{byID: map[string]viperblock.AMIMetadata{
|
||||
amiLoader: &fakeAMILoader{byID: map[string]ebsmetadata.AMI{
|
||||
"ami-1": {ImageOwnerAlias: "acc"},
|
||||
}},
|
||||
resourceMgr: prov,
|
||||
|
||||
@@ -12,12 +12,13 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ec2/volumestate"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -124,7 +125,7 @@ func TestDrainVolume_AttachedRoutesToHostingNode(t *testing.T) {
|
||||
seedVolumeAttachment(t, store, "vol-attached", "in-use", drainInstanceID)
|
||||
got := drainResponder(t, nc, drainInstanceID, drainedAck(t, "vol-attached"))
|
||||
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-attached", viperblock.VolumeMetadata{}, testAccountID))
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-attached", "", "", testAccountID))
|
||||
|
||||
command := awaitDrainCommand(t, got)
|
||||
assert.True(t, command.Attributes.DrainVolume)
|
||||
@@ -141,7 +142,7 @@ func TestDrainVolume_AttachedToStoppedInstanceTakesStoppedPath(t *testing.T) {
|
||||
svc, store, _ := setupDrainService(t)
|
||||
seedVolumeAttachment(t, store, "vol-no-host", "in-use", drainInstanceID)
|
||||
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-no-host", viperblock.VolumeMetadata{}, testAccountID))
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-no-host", "", "", testAccountID))
|
||||
}
|
||||
|
||||
// A host that still holds the instance but reports it not running has nothing
|
||||
@@ -156,7 +157,7 @@ func TestDrainVolume_NotRunningAckTakesStoppedPath(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
got := drainResponder(t, nc, drainInstanceID, ack)
|
||||
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-not-running", viperblock.VolumeMetadata{}, testAccountID))
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-not-running", "", "", testAccountID))
|
||||
assert.True(t, awaitDrainCommand(t, got).Attributes.DrainVolume)
|
||||
}
|
||||
|
||||
@@ -168,7 +169,7 @@ func TestDrainVolume_AttachedWithFailedAckFails(t *testing.T) {
|
||||
drainResponder(t, nc, drainInstanceID,
|
||||
[]byte(`{"Code":"`+awserrors.ErrorServerInternal+`","Message":"drain failed"}`))
|
||||
|
||||
err := svc.drainVolume(context.Background(), "vol-drain-err", viperblock.VolumeMetadata{}, testAccountID)
|
||||
err := svc.drainVolume(context.Background(), "vol-drain-err", "", "", testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "drain failed")
|
||||
}
|
||||
@@ -179,7 +180,7 @@ func TestDrainVolume_AttachedWithUnexpectedAckFails(t *testing.T) {
|
||||
seedVolumeAttachment(t, store, "vol-odd-ack", "in-use", drainInstanceID)
|
||||
drainResponder(t, nc, drainInstanceID, []byte(`{}`))
|
||||
|
||||
err := svc.drainVolume(context.Background(), "vol-odd-ack", viperblock.VolumeMetadata{}, testAccountID)
|
||||
err := svc.drainVolume(context.Background(), "vol-odd-ack", "", "", testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "unexpected ack")
|
||||
}
|
||||
@@ -200,7 +201,7 @@ func TestDrainVolume_AvailableDoesNotDialLocalSocket(t *testing.T) {
|
||||
svc.natsConn = nil
|
||||
|
||||
requireReturnsWithin(t, 2*time.Second, func() error {
|
||||
return svc.drainVolume(context.Background(), "vol-available", viperblock.VolumeMetadata{}, testAccountID)
|
||||
return svc.drainVolume(context.Background(), "vol-available", "", "", testAccountID)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -211,7 +212,7 @@ func TestDrainVolume_InUseWithoutInstanceTakesStoppedPath(t *testing.T) {
|
||||
seedVolumeAttachment(t, store, "vol-no-instance", "in-use", "")
|
||||
svc.natsConn = nil
|
||||
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-no-instance", viperblock.VolumeMetadata{}, testAccountID))
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-no-instance", "", "", testAccountID))
|
||||
}
|
||||
|
||||
// When the volume is served by this node the socket is local, so the drain
|
||||
@@ -225,7 +226,7 @@ func TestDrainVolume_LocalSocketShortCircuits(t *testing.T) {
|
||||
// drainInstanceID, so a routed command would fail.
|
||||
svc.natsConn = nil
|
||||
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-local", viperblock.VolumeMetadata{}, testAccountID))
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-local", "", "", testAccountID))
|
||||
}
|
||||
|
||||
// A socket that answers without acking means this node does serve the volume
|
||||
@@ -237,7 +238,7 @@ func TestDrainVolume_LocalSocketErrorAckFailsWithoutRouting(t *testing.T) {
|
||||
testutil.StartDrainSocket(t, svc.config.DataDir, "vol-local-err", "ERR\n")
|
||||
got := drainResponder(t, nc, drainInstanceID, drainedAck(t, "vol-local-err"))
|
||||
|
||||
err := svc.drainVolume(context.Background(), "vol-local-err", viperblock.VolumeMetadata{}, testAccountID)
|
||||
err := svc.drainVolume(context.Background(), "vol-local-err", "", "", testAccountID)
|
||||
require.Error(t, err)
|
||||
|
||||
assert.Empty(t, got, "a local drain that failed must not be re-issued to the host node")
|
||||
@@ -251,8 +252,9 @@ func TestDrainVolume_StateJSONOverridesConfigJSON(t *testing.T) {
|
||||
seedVolumeAttachment(t, store, "vol-stale-config", "in-use", drainInstanceID)
|
||||
got := drainResponder(t, nc, drainInstanceID, drainedAck(t, "vol-stale-config"))
|
||||
|
||||
stale := viperblock.VolumeMetadata{State: "available"}
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-stale-config", stale, testAccountID))
|
||||
// "available" mirrors the stale State a live NBD plugin leaves in
|
||||
// config.json; drainVolume must not trust it (see the test's own doc above).
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-stale-config", "available", "", testAccountID))
|
||||
|
||||
assert.True(t, awaitDrainCommand(t, got).Attributes.DrainVolume)
|
||||
}
|
||||
@@ -263,8 +265,8 @@ func TestDrainVolume_FallsBackToConfigJSONWhenNoStateObject(t *testing.T) {
|
||||
svc, _, nc := setupDrainService(t)
|
||||
got := drainResponder(t, nc, drainInstanceID, drainedAck(t, "vol-legacy"))
|
||||
|
||||
legacy := viperblock.VolumeMetadata{State: "in-use", AttachedInstance: drainInstanceID}
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-legacy", legacy, testAccountID))
|
||||
// The state/instance embedded in a legacy config.json, with no state.json to overlay it.
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-legacy", "in-use", drainInstanceID, testAccountID))
|
||||
|
||||
assert.True(t, awaitDrainCommand(t, got).Attributes.DrainVolume)
|
||||
}
|
||||
@@ -277,40 +279,66 @@ func TestDrainVolume_MetadataOnlySkipsDrain(t *testing.T) {
|
||||
svc.config.Predastore.Host = ""
|
||||
svc.natsConn = nil
|
||||
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-meta-only", viperblock.VolumeMetadata{}, testAccountID))
|
||||
require.NoError(t, svc.drainVolume(context.Background(), "vol-meta-only", "", "", testAccountID))
|
||||
}
|
||||
|
||||
// CreateSnapshot drains before it reads the checkpoint. The snapshot itself
|
||||
// then fails against the rejecting Predastore stub; what is asserted here is
|
||||
// that the drain was routed to the volume's host first.
|
||||
func TestCreateSnapshot_DrainsAttachedVolume(t *testing.T) {
|
||||
svc, store, nc := setupDrainService(t)
|
||||
createTestVolume(t, store, "vol-snap-drain", 10)
|
||||
seedVolumeAttachment(t, store, "vol-snap-drain", "in-use", drainInstanceID)
|
||||
got := drainResponder(t, nc, drainInstanceID, drainedAck(t, "vol-snap-drain"))
|
||||
// putProviderVolume seeds both the control-plane document CreateSnapshot's
|
||||
// provider branch reads for attachment state, and the provider's own record,
|
||||
// mirroring what CreateVolume's provider branch leaves behind.
|
||||
func putProviderVolume(t *testing.T, svc *SnapshotServiceImpl, provider ebsprovider.EBSProvider, volumeID, state, instanceID string) {
|
||||
t.Helper()
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testAccountID, CapacityGiB: 4,
|
||||
State: state, AttachedInstance: instanceID, AvailabilityZone: "us-east-1a",
|
||||
ProviderHandle: "memory://volume/" + volumeID,
|
||||
}))
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4 * 1024 * 1024 * 1024},
|
||||
AvailabilityZone: "us-east-1a",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// The provider branch used to skip the drain entirely, silently snapshotting a
|
||||
// stale checkpoint of an attached, actively-written volume. It must drain the
|
||||
// same way the legacy branch does.
|
||||
func TestCreateSnapshot_Provider_DrainsAttachedVolume(t *testing.T) {
|
||||
svc, _, nc := setupDrainService(t)
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc.SetEBSProvider(provider)
|
||||
putProviderVolume(t, svc, provider, "vol-provider-drain", "in-use", drainInstanceID)
|
||||
|
||||
got := drainResponder(t, nc, drainInstanceID, drainedAck(t, "vol-provider-drain"))
|
||||
|
||||
_, err := svc.CreateSnapshot(context.Background(),
|
||||
&ec2.CreateSnapshotInput{VolumeId: aws.String("vol-snap-drain")}, testAccountID)
|
||||
require.Error(t, err)
|
||||
&ec2.CreateSnapshotInput{VolumeId: aws.String("vol-provider-drain")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
command := awaitDrainCommand(t, got)
|
||||
assert.True(t, command.Attributes.DrainVolume)
|
||||
assert.Equal(t, "vol-snap-drain", command.DrainVolumeData.VolumeID)
|
||||
assert.Equal(t, "vol-provider-drain", command.DrainVolumeData.VolumeID)
|
||||
}
|
||||
|
||||
// An attached volume whose host reports the drain failed fails the snapshot: a
|
||||
// retryable error is strictly better than a successful snapshot of stale data.
|
||||
func TestCreateSnapshot_UndrainableAttachedVolumeFails(t *testing.T) {
|
||||
svc, store, nc := setupDrainService(t)
|
||||
createTestVolume(t, store, "vol-snap-nodrain", 10)
|
||||
seedVolumeAttachment(t, store, "vol-snap-nodrain", "in-use", drainInstanceID)
|
||||
// A provider-branch snapshot whose drain fails must fail closed rather than
|
||||
// return a snapshot of stale data: the provider must never be asked to
|
||||
// snapshot, and no snapshot metadata may be left behind.
|
||||
func TestCreateSnapshot_Provider_UndrainableAttachedVolumeFails(t *testing.T) {
|
||||
svc, _, nc := setupDrainService(t)
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc.SetEBSProvider(provider)
|
||||
putProviderVolume(t, svc, provider, "vol-provider-nodrain", "in-use", drainInstanceID)
|
||||
|
||||
got := drainResponder(t, nc, drainInstanceID,
|
||||
[]byte(`{"Code":"`+awserrors.ErrorServerInternal+`","Message":"drain failed"}`))
|
||||
|
||||
_, err := svc.CreateSnapshot(context.Background(),
|
||||
&ec2.CreateSnapshotInput{VolumeId: aws.String("vol-snap-nodrain")}, testAccountID)
|
||||
&ec2.CreateSnapshotInput{VolumeId: aws.String("vol-provider-nodrain")}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
|
||||
assert.True(t, awaitDrainCommand(t, got).Attributes.DrainVolume)
|
||||
|
||||
out, err := svc.DescribeSnapshots(context.Background(), &ec2.DescribeSnapshotsInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, out.Snapshots, "a failed drain must not leave a snapshot behind")
|
||||
}
|
||||
|
||||
@@ -6,9 +6,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
@@ -21,6 +19,8 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/filterutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ec2/volumestate"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvutil"
|
||||
@@ -28,8 +28,6 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
vbs3 "github.com/mulgadc/viperblock/viperblock/backends/s3"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
@@ -50,6 +48,26 @@ type SnapshotServiceImpl struct {
|
||||
natsConn *nats.Conn
|
||||
snapKV jetstream.KeyValue
|
||||
mutex sync.RWMutex
|
||||
metadata *ebsmetadata.Store
|
||||
provider ebsprovider.EBSProvider
|
||||
}
|
||||
|
||||
// SetEBSProvider injects the provider boundary for snapshot operations.
|
||||
// Legacy snapshot behavior remains until snapshot metadata is migrated.
|
||||
func (s *SnapshotServiceImpl) SetEBSProvider(provider ebsprovider.EBSProvider) {
|
||||
s.provider = provider
|
||||
}
|
||||
|
||||
// EBSProvider returns the injected provider boundary, or nil on the legacy
|
||||
// embedded-engine path. Primarily for composition-root tests to observe wiring.
|
||||
func (s *SnapshotServiceImpl) EBSProvider() ebsprovider.EBSProvider {
|
||||
return s.provider
|
||||
}
|
||||
|
||||
// MetadataStore returns the control-plane metadata store. Primarily for
|
||||
// composition-root tests to observe wiring.
|
||||
func (s *SnapshotServiceImpl) MetadataStore() *ebsmetadata.Store {
|
||||
return s.metadata
|
||||
}
|
||||
|
||||
// SnapshotConfig represents snapshot metadata stored in S3.
|
||||
@@ -65,6 +83,7 @@ type SnapshotConfig struct {
|
||||
OwnerID string `json:"owner_id"`
|
||||
AvailabilityZone string `json:"availability_zone"`
|
||||
Tags map[string]string `json:"tags"`
|
||||
ProviderHandle string `json:"provider_handle,omitempty"`
|
||||
}
|
||||
|
||||
// NewSnapshotServiceImplWithNATS creates a snapshot service with JetStream KV for volume-snapshot tracking.
|
||||
@@ -96,6 +115,7 @@ func NewSnapshotServiceImplWithNATS(ctx context.Context, cfg *config.Config, nat
|
||||
store: store,
|
||||
natsConn: natsConn,
|
||||
snapKV: kv,
|
||||
metadata: ebsmetadata.NewStore(store, cfg.Predastore.Bucket),
|
||||
}, kv, nil
|
||||
}
|
||||
|
||||
@@ -106,6 +126,7 @@ func NewSnapshotServiceImplWithStore(cfg *config.Config, store objectstore.Objec
|
||||
config: cfg,
|
||||
store: store,
|
||||
natsConn: natsConn,
|
||||
metadata: ebsmetadata.NewStore(store, cfg.Predastore.Bucket),
|
||||
}
|
||||
if len(snapshotKV) > 0 {
|
||||
svc.snapKV = snapshotKV[0]
|
||||
@@ -206,79 +227,49 @@ func (s *SnapshotServiceImpl) CreateSnapshot(ctx context.Context, input *ec2.Cre
|
||||
slog.InfoContext(ctx, "CreateSnapshot request", "volumeId", volumeID)
|
||||
|
||||
snapshotID := utils.GenerateResourceID("snap")
|
||||
if s.provider == nil {
|
||||
slog.ErrorContext(ctx, "no EBS provider configured", "op", "CreateSnapshot")
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
volumeConfigKey := fmt.Sprintf("%s/config.json", volumeID)
|
||||
volumeResult, err := s.store.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: aws.String(s.config.Predastore.Bucket),
|
||||
Key: aws.String(volumeConfigKey),
|
||||
})
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot failed to get volume config", "volumeId", volumeID, "err", err)
|
||||
// A volume predating tenancy records no owner. Refusing it would strand
|
||||
// every pre-tenancy volume as unsnapshottable, so an absent TenantID stays
|
||||
// open to any caller; a recorded one must match.
|
||||
volume, err := s.metadata.GetVolume(ctx, volumeID)
|
||||
if err != nil || (accountID != "" && volume.TenantID != "" && volume.TenantID != accountID) {
|
||||
return nil, errors.New(awserrors.ErrorInvalidVolumeNotFound)
|
||||
}
|
||||
defer volumeResult.Body.Close()
|
||||
|
||||
volumeBody, err := io.ReadAll(volumeResult.Body)
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot failed to read volume config", "volumeId", volumeID, "err", err)
|
||||
if volume.CapacityGiB == 0 {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// config.json may be an at-rest encryption envelope; StateBody unwraps it to
|
||||
// the inner VBState. Decoding the raw envelope yields a zero state
|
||||
// (SizeGiB==0), which the size guard below would reject as a 500.
|
||||
var volumeState viperblock.VBState
|
||||
if err := json.Unmarshal(viperblock.StateBody(volumeBody), &volumeState); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot failed to decode volume config", "volumeId", volumeID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
volumeConfig := volumeState.VolumeConfig
|
||||
|
||||
// Verify the caller owns the source volume
|
||||
if accountID != "" && volumeConfig.VolumeMetadata.TenantID != "" && volumeConfig.VolumeMetadata.TenantID != accountID {
|
||||
slog.WarnContext(ctx, "CreateSnapshot: account does not own volume", "volumeId", volumeID, "accountID", accountID, "tenantID", volumeConfig.VolumeMetadata.TenantID)
|
||||
return nil, errors.New(awserrors.ErrorInvalidVolumeNotFound)
|
||||
}
|
||||
|
||||
if volumeConfig.VolumeMetadata.SizeGiB == 0 {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot: source volume has zero size in config", "volumeId", volumeID)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Flush the writes still buffered by whichever node serves the volume, so the
|
||||
// live checkpoint this snapshot is about to read is current. An attached
|
||||
// volume that cannot be drained fails here rather than silently producing a
|
||||
// snapshot of an older checkpoint.
|
||||
if err := s.drainVolume(ctx, volumeID, volumeConfig.VolumeMetadata, accountID); err != nil {
|
||||
// Flush the writes buffered by whichever node serves the volume, so the
|
||||
// provider reads a current checkpoint. An attached volume that cannot be
|
||||
// drained fails here rather than silently snapshotting stale data.
|
||||
if err := s.drainVolume(ctx, volumeID, volume.State, volume.AttachedInstance, accountID); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot: drain failed", "volumeId", volumeID, "snapshotId", snapshotID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Snapshot the viperblock volume by reading the live checkpoint from S3.
|
||||
// The live checkpoint is updated on every NBD Flush by the running nbdkit process.
|
||||
// If the volume is not mounted (stopped), LoadLiveCheckpoint falls back to the
|
||||
// numbered checkpoint written by Close.
|
||||
if err := s.snapshotVolume(volumeID, snapshotID, volumeConfig.VolumeMetadata.SizeGiB*1024*1024*1024); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot: viperblock snapshot failed", "volumeId", volumeID, "snapshotId", snapshotID, "err", err)
|
||||
created, err := s.provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, VolumeID: volumeID, VolumeHandle: volume.ProviderHandle,
|
||||
})
|
||||
if err != nil || created == nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
slog.InfoContext(ctx, "CreateSnapshot: viperblock snapshot created", "volumeId", volumeID, "snapshotId", snapshotID)
|
||||
|
||||
now := time.Now()
|
||||
|
||||
snapshotCfg := &SnapshotConfig{
|
||||
SnapshotID: snapshotID,
|
||||
VolumeID: volumeID,
|
||||
VolumeSize: utils.SafeUint64ToInt64(volumeConfig.VolumeMetadata.SizeGiB),
|
||||
State: "completed",
|
||||
Progress: "100%",
|
||||
StartTime: now,
|
||||
Encrypted: volumeState.EncryptionEnabled,
|
||||
OwnerID: accountID,
|
||||
AvailabilityZone: volumeConfig.VolumeMetadata.AvailabilityZone,
|
||||
Tags: utils.ExtractTags(input.TagSpecifications, "snapshot"),
|
||||
SnapshotID: snapshotID, VolumeID: volumeID,
|
||||
VolumeSize: utils.SafeUint64ToInt64(volume.CapacityGiB),
|
||||
State: string(created.State), Progress: "100%", StartTime: now,
|
||||
Encrypted: volume.Encrypted,
|
||||
OwnerID: accountID, AvailabilityZone: volume.AvailabilityZone,
|
||||
Tags: utils.ExtractTags(input.TagSpecifications, "snapshot"),
|
||||
ProviderHandle: created.Handle,
|
||||
}
|
||||
if snapshotCfg.State == "" {
|
||||
snapshotCfg.State = "completed"
|
||||
}
|
||||
|
||||
if input.Description != nil {
|
||||
snapshotCfg.Description = *input.Description
|
||||
}
|
||||
@@ -287,14 +278,13 @@ func (s *SnapshotServiceImpl) CreateSnapshot(ctx context.Context, input *ec2.Cre
|
||||
// This ensures we never have an untracked snapshot in S3.
|
||||
if err := s.addSnapshotRef(ctx, volumeID, snapshotID); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot failed to add snapshot ref to KV", "snapshotId", snapshotID, "volumeId", volumeID, "err", err)
|
||||
_ = s.provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, Handle: created.Handle})
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := s.putSnapshotConfig(snapshotID, snapshotCfg); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateSnapshot failed to write config", "snapshotId", snapshotID, "err", err)
|
||||
if cleanupErr := s.removeSnapshotRefForCleanup(ctx, volumeID, snapshotID); cleanupErr != nil {
|
||||
slog.Error("CreateSnapshot failed to roll back snapshot reference", "snapshotId", snapshotID, "volumeId", volumeID, "err", cleanupErr)
|
||||
}
|
||||
_ = s.removeSnapshotRefForCleanup(ctx, volumeID, snapshotID)
|
||||
_ = s.provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, Handle: created.Handle})
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
@@ -353,8 +343,8 @@ func DrainVolumeSocket(dataDir, volumeID string) error {
|
||||
|
||||
// drainVolume flushes the volume's in-flight writes to S3 before the snapshot
|
||||
// reads the live checkpoint from there.
|
||||
func (s *SnapshotServiceImpl) drainVolume(ctx context.Context, volumeID string, meta viperblock.VolumeMetadata, accountID string) error {
|
||||
return DrainVolume(ctx, s.config, s.store, s.natsConn, volumeID, meta, accountID)
|
||||
func (s *SnapshotServiceImpl) drainVolume(ctx context.Context, volumeID string, fallbackState, fallbackInstance string, accountID string) error {
|
||||
return DrainVolume(ctx, s.config, s.store, s.natsConn, volumeID, fallbackState, fallbackInstance, accountID)
|
||||
}
|
||||
|
||||
// DrainVolume flushes the volume's in-flight writes to S3 before a live
|
||||
@@ -368,14 +358,14 @@ func (s *SnapshotServiceImpl) drainVolume(ctx context.Context, volumeID string,
|
||||
// volume's attachment record, never from whether the socket happens to be
|
||||
// local: an attached volume is being written to right now, and reading its
|
||||
// checkpoint without draining silently captures an older one.
|
||||
func DrainVolume(ctx context.Context, cfg *config.Config, store objectstore.ObjectStore, natsConn *nats.Conn, volumeID string, meta viperblock.VolumeMetadata, accountID string) error {
|
||||
func DrainVolume(ctx context.Context, cfg *config.Config, store objectstore.ObjectStore, natsConn *nats.Conn, volumeID string, fallbackState, fallbackInstance string, accountID string) error {
|
||||
// A metadata-only snapshot never reads the live checkpoint, so there is
|
||||
// nothing for a drain to make current.
|
||||
if cfg == nil || cfg.Predastore.Host == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
state, instanceID, err := volumeAttachment(ctx, store, cfg.Predastore.Bucket, volumeID, meta)
|
||||
state, instanceID, err := volumeAttachment(ctx, store, cfg.Predastore.Bucket, volumeID, fallbackState, fallbackInstance)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -410,10 +400,11 @@ func DrainVolume(ctx context.Context, cfg *config.Config, store objectstore.Obje
|
||||
}
|
||||
|
||||
// volumeAttachment returns the volume's authoritative state and attached
|
||||
// instance. state.json is control-plane-owned and authoritative; the copy in
|
||||
// config.json (passed in as meta) is rewritten by the live NBD plugin from its
|
||||
// stale in-memory state and is only used for volumes predating the split.
|
||||
func volumeAttachment(ctx context.Context, store objectstore.ObjectStore, bucket, volumeID string, meta viperblock.VolumeMetadata) (state, instanceID string, err error) {
|
||||
// instance. state.json is control-plane-owned and authoritative; the fallback
|
||||
// pair (passed in from the caller's own volume record) is rewritten by the
|
||||
// live NBD plugin from its stale in-memory state and is only used for volumes
|
||||
// predating the split.
|
||||
func volumeAttachment(ctx context.Context, store objectstore.ObjectStore, bucket, volumeID string, fallbackState, fallbackInstance string) (state, instanceID string, err error) {
|
||||
rec, found, err := volumestate.Read(ctx, store, bucket, volumeID)
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("read volume state for %s: %w", volumeID, err)
|
||||
@@ -421,7 +412,7 @@ func volumeAttachment(ctx context.Context, store objectstore.ObjectStore, bucket
|
||||
if found {
|
||||
return rec.State, rec.AttachedInstance, nil
|
||||
}
|
||||
return meta.State, meta.AttachedInstance, nil
|
||||
return fallbackState, fallbackInstance, nil
|
||||
}
|
||||
|
||||
// drainOnHostNode issues the drain on the node hosting instanceID. Only that
|
||||
@@ -473,66 +464,6 @@ func drainOnHostNode(ctx context.Context, natsConn *nats.Conn, volumeID, instanc
|
||||
}
|
||||
}
|
||||
|
||||
// snapshotVolume opens a read-only viperblock instance, reads the live checkpoint from S3
|
||||
// (written by nbdkit on every NBD Flush, and by the drain the caller has already run),
|
||||
// and calls CreateSnapshot. Falls back to the numbered checkpoint from Close if no live
|
||||
// checkpoint exists (stopped volume path).
|
||||
// If Predastore is not configured the snapshot proceeds as metadata-only.
|
||||
func (s *SnapshotServiceImpl) snapshotVolume(volumeID, snapshotID string, volumeSize uint64) error {
|
||||
if s.config == nil || s.config.Predastore.Host == "" {
|
||||
slog.Warn("snapshotVolume: Predastore not configured, skipping viperblock snapshot (metadata-only)", "volumeId", volumeID)
|
||||
return nil
|
||||
}
|
||||
|
||||
cfg := vbs3.S3Config{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: volumeSize,
|
||||
Bucket: s.config.Predastore.Bucket,
|
||||
Region: s.config.Predastore.Region,
|
||||
AccessKey: s.config.Predastore.AccessKey,
|
||||
SecretKey: s.config.Predastore.SecretKey,
|
||||
Host: s.config.Predastore.Host,
|
||||
}
|
||||
|
||||
mkey, err := utils.LoadViperblockMasterKey(s.config.Viperblock.EncryptionKeyFile)
|
||||
if err != nil {
|
||||
return fmt.Errorf("load encryption key: %w", err)
|
||||
}
|
||||
|
||||
vbconfig := viperblock.VB{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: volumeSize,
|
||||
BaseDir: s.config.WalDir,
|
||||
Cache: viperblock.Cache{Config: viperblock.CacheConfig{Size: 0}},
|
||||
MasterKey: mkey,
|
||||
EncryptionEnabled: mkey != nil,
|
||||
}
|
||||
|
||||
vb, err := viperblock.New(&vbconfig, "s3", cfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("new viperblock: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
vb.StopChunkUploader()
|
||||
vb.StopWALSyncer()
|
||||
}()
|
||||
|
||||
if err := vb.Backend.Init(); err != nil {
|
||||
return fmt.Errorf("backend init: %w", err)
|
||||
}
|
||||
if err := vb.LoadState(); err != nil {
|
||||
return fmt.Errorf("load state: %w", err)
|
||||
}
|
||||
|
||||
if err := vb.LoadLiveCheckpoint(); err != nil {
|
||||
return fmt.Errorf("load live checkpoint: %w", err)
|
||||
}
|
||||
if _, err := vb.CreateSnapshot(snapshotID); err != nil {
|
||||
return fmt.Errorf("create snapshot: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// describeSnapshotsValidFilters defines the set of filter names accepted by DescribeSnapshots.
|
||||
var describeSnapshotsValidFilters = map[string]bool{
|
||||
"snapshot-id": true,
|
||||
@@ -686,49 +617,19 @@ func snapshotMatchesFilters(cfg *SnapshotConfig, filters map[string][]string) bo
|
||||
return filterutil.MatchesTags(filters, cfg.Tags)
|
||||
}
|
||||
|
||||
// snapshotInUseByVolumes checks if any volume was created from the given snapshot.
|
||||
// snapshotInUseByVolumes checks if any volume was created from the given
|
||||
// snapshot. A clone records its source in its ebsmetadata document; missing one
|
||||
// here would let a delete strip the chunks a live clone still reads.
|
||||
func (s *SnapshotServiceImpl) snapshotInUseByVolumes(ctx context.Context, snapshotID string) (bool, error) {
|
||||
listResult, err := s.store.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(s.config.Predastore.Bucket),
|
||||
Prefix: aws.String("vol-"),
|
||||
Delimiter: aws.String("/"),
|
||||
})
|
||||
volumes, err := s.metadata.ListVolumes(ctx)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("snapshotInUseByVolumes: failed to list volumes: %w", err)
|
||||
}
|
||||
|
||||
for _, prefix := range listResult.CommonPrefixes {
|
||||
if prefix.Prefix == nil {
|
||||
continue
|
||||
}
|
||||
volumeID := strings.TrimSuffix(*prefix.Prefix, "/")
|
||||
configKey := fmt.Sprintf("%s/config.json", volumeID)
|
||||
|
||||
result, err := s.store.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: aws.String(s.config.Predastore.Bucket),
|
||||
Key: aws.String(configKey),
|
||||
})
|
||||
if err != nil {
|
||||
continue // volume may not have a config yet
|
||||
}
|
||||
|
||||
scanBody, readErr := io.ReadAll(result.Body)
|
||||
_ = result.Body.Close()
|
||||
if readErr != nil {
|
||||
continue
|
||||
}
|
||||
// Unwrap the encryption envelope so encrypted volumes are scanned too;
|
||||
// a raw decode yields a zero state and silently drops their snapshots.
|
||||
var state viperblock.VBState
|
||||
if decodeErr := json.Unmarshal(viperblock.StateBody(scanBody), &state); decodeErr != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
if state.VolumeConfig.VolumeMetadata.SnapshotID == snapshotID {
|
||||
for _, volume := range volumes {
|
||||
if volume.SnapshotID == snapshotID {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
|
||||
return false, nil
|
||||
}
|
||||
|
||||
@@ -764,6 +665,13 @@ func (s *SnapshotServiceImpl) DeleteSnapshot(ctx context.Context, input *ec2.Del
|
||||
slog.InfoContext(ctx, "DeleteSnapshot blocked: snapshot in use by volume", "snapshotId", snapshotID)
|
||||
return nil, errors.New(awserrors.ErrorInvalidSnapshotInUse)
|
||||
}
|
||||
if s.provider == nil {
|
||||
slog.ErrorContext(ctx, "no EBS provider configured", "op", "DeleteSnapshot")
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if err := s.provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, Handle: cfg.ProviderHandle}); err != nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
listResult, err := s.store.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(s.config.Predastore.Bucket),
|
||||
@@ -798,8 +706,10 @@ func (s *SnapshotServiceImpl) DeleteSnapshot(ctx context.Context, input *ec2.Del
|
||||
return &ec2.DeleteSnapshotOutput{}, nil
|
||||
}
|
||||
|
||||
// CopySnapshot copies a snapshot (within same region for now).
|
||||
// The copied snapshot is owned by the caller's account.
|
||||
// CopySnapshot duplicates a snapshot under a new ID owned by the caller.
|
||||
// Served only through the provider: the copy needs its own snapshot metadata
|
||||
// on the backend, and on an encrypted volume that metadata is AEAD-sealed with
|
||||
// the snapshot ID in the AAD, so only the storage engine can re-seal it.
|
||||
func (s *SnapshotServiceImpl) CopySnapshot(ctx context.Context, input *ec2.CopySnapshotInput, accountID string) (*ec2.CopySnapshotOutput, error) {
|
||||
if input == nil || input.SourceSnapshotId == nil {
|
||||
return nil, errors.New(awserrors.ErrorInvalidParameterValue)
|
||||
@@ -815,45 +725,59 @@ func (s *SnapshotServiceImpl) CopySnapshot(ctx context.Context, input *ec2.CopyS
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Verify the caller owns the source snapshot
|
||||
if accountID != "" && sourceCfg.OwnerID != "" && sourceCfg.OwnerID != accountID {
|
||||
slog.WarnContext(ctx, "CopySnapshot: account does not own source snapshot", "snapshotId", sourceSnapshotID, "accountID", accountID, "ownerID", sourceCfg.OwnerID)
|
||||
return nil, errors.New(awserrors.ErrorUnauthorizedOperation)
|
||||
}
|
||||
|
||||
// Without a provider the only reachable path writes the control-plane
|
||||
// config alone. That copy described and created volumes fine, then failed
|
||||
// at attach with no backend metadata behind the new ID, so refuse instead.
|
||||
if s.provider == nil {
|
||||
slog.WarnContext(ctx, "CopySnapshot is not supported without an EBS provider", "sourceSnapshotId", sourceSnapshotID)
|
||||
return nil, errors.New(awserrors.ErrorUnsupportedOperation)
|
||||
}
|
||||
|
||||
newSnapshotID := utils.GenerateResourceID("snap")
|
||||
|
||||
newCfg := &SnapshotConfig{
|
||||
SnapshotID: newSnapshotID,
|
||||
VolumeID: sourceCfg.VolumeID,
|
||||
VolumeSize: sourceCfg.VolumeSize,
|
||||
State: "completed",
|
||||
Progress: "100%",
|
||||
StartTime: time.Now(),
|
||||
Description: sourceCfg.Description,
|
||||
Encrypted: sourceCfg.Encrypted,
|
||||
OwnerID: accountID,
|
||||
AvailabilityZone: sourceCfg.AvailabilityZone,
|
||||
Tags: make(map[string]string),
|
||||
}
|
||||
|
||||
if input.Description != nil {
|
||||
newCfg.Description = *input.Description
|
||||
}
|
||||
|
||||
maps.Copy(newCfg.Tags, sourceCfg.Tags)
|
||||
|
||||
// Track the volume→snapshot dependency in KV before persisting to S3.
|
||||
if err := s.addSnapshotRef(ctx, sourceCfg.VolumeID, newSnapshotID); err != nil {
|
||||
slog.ErrorContext(ctx, "CopySnapshot failed to add snapshot ref to KV", "snapshotId", newSnapshotID, "volumeId", sourceCfg.VolumeID, "err", err)
|
||||
copied, err := s.provider.CopySnapshot(ctx, ebsprovider.CopySnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
SourceSnapshotID: sourceSnapshotID,
|
||||
DestinationSnapshotID: newSnapshotID,
|
||||
VolumeID: sourceCfg.VolumeID,
|
||||
})
|
||||
if err != nil || copied == nil {
|
||||
slog.ErrorContext(ctx, "CopySnapshot: provider copy failed", "sourceSnapshotId", sourceSnapshotID, "newSnapshotId", newSnapshotID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
if err := s.putSnapshotConfig(newSnapshotID, newCfg); err != nil {
|
||||
slog.ErrorContext(ctx, "CopySnapshot failed to write config", "snapshotId", newSnapshotID, "err", err)
|
||||
if cleanupErr := s.removeSnapshotRefForCleanup(ctx, sourceCfg.VolumeID, newSnapshotID); cleanupErr != nil {
|
||||
slog.Error("CopySnapshot failed to roll back snapshot reference", "snapshotId", newSnapshotID, "volumeId", sourceCfg.VolumeID, "err", cleanupErr)
|
||||
}
|
||||
now := time.Now()
|
||||
snapshotCfg := &SnapshotConfig{
|
||||
SnapshotID: newSnapshotID, VolumeID: sourceCfg.VolumeID,
|
||||
VolumeSize: sourceCfg.VolumeSize,
|
||||
State: string(copied.State), Progress: "100%", StartTime: now,
|
||||
Encrypted: sourceCfg.Encrypted,
|
||||
OwnerID: accountID, AvailabilityZone: sourceCfg.AvailabilityZone,
|
||||
Tags: utils.ExtractTags(input.TagSpecifications, "snapshot"),
|
||||
ProviderHandle: copied.Handle,
|
||||
}
|
||||
if snapshotCfg.State == "" {
|
||||
snapshotCfg.State = "completed"
|
||||
}
|
||||
snapshotCfg.Description = sourceCfg.Description
|
||||
if input.Description != nil {
|
||||
snapshotCfg.Description = *input.Description
|
||||
}
|
||||
|
||||
// The copy pins the same chunks as its source, so it must join the volume's
|
||||
// snapshot list before it is visible: a copy missing from that list would
|
||||
// let the volume be deleted out from under it.
|
||||
if err := s.addSnapshotRef(ctx, sourceCfg.VolumeID, newSnapshotID); err != nil {
|
||||
_ = s.provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: newSnapshotID, Handle: copied.Handle})
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if err := s.putSnapshotConfig(newSnapshotID, snapshotCfg); err != nil {
|
||||
_ = s.removeSnapshotRefForCleanup(ctx, sourceCfg.VolumeID, newSnapshotID)
|
||||
_ = s.provider.DeleteSnapshot(ctx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: newSnapshotID, Handle: copied.Handle})
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
@@ -861,6 +785,7 @@ func (s *SnapshotServiceImpl) CopySnapshot(ctx context.Context, input *ec2.CopyS
|
||||
|
||||
return &ec2.CopySnapshotOutput{
|
||||
SnapshotId: aws.String(newSnapshotID),
|
||||
Tags: utils.MapToEC2Tags(snapshotCfg.Tags),
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -3,9 +3,7 @@ package handlers_ec2_snapshot
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -14,9 +12,10 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -25,7 +24,9 @@ import (
|
||||
const testAccountID = "111122223333"
|
||||
const otherAccountID = "444455556666"
|
||||
|
||||
// setupTestSnapshotService creates a snapshot service with in-memory storage for testing.
|
||||
// setupTestSnapshotService creates a snapshot service with in-memory storage,
|
||||
// an in-memory EBS provider, and the same legacy volume read fallback the
|
||||
// composition root wires, so pre-migration config.json fixtures resolve here.
|
||||
func setupTestSnapshotService(t *testing.T) (*SnapshotServiceImpl, *objectstore.MemoryObjectStore) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{
|
||||
@@ -36,28 +37,40 @@ func setupTestSnapshotService(t *testing.T) (*SnapshotServiceImpl, *objectstore.
|
||||
}
|
||||
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil)
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
return svc, store
|
||||
}
|
||||
|
||||
// createTestVolume creates a test volume in the mock store
|
||||
// The real S3 stores VBState (which wraps VolumeConfig), so we match that format.
|
||||
func createTestVolume(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string, sizeGiB int) {
|
||||
volumeState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
SizeGiB: uint64(sizeGiB),
|
||||
AvailabilityZone: "us-east-1a",
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(volumeState)
|
||||
require.NoError(t, err)
|
||||
// createTestVolume seeds a volume on both sides: the provider holds the blocks
|
||||
// a snapshot copies, the document is what the control plane resolves.
|
||||
func createTestVolume(t *testing.T, svc *SnapshotServiceImpl, store *objectstore.MemoryObjectStore, volumeID string, sizeGiB int) {
|
||||
t.Helper()
|
||||
seedProviderVolume(t, svc, volumeID, sizeGiB)
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
CapacityGiB: uint64(sizeGiB),
|
||||
AvailabilityZone: "us-east-1a",
|
||||
})
|
||||
}
|
||||
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
ContentType: aws.String("application/json"),
|
||||
// seedVolumeDocument writes the control-plane document for a volume.
|
||||
func seedVolumeDocument(t *testing.T, store objectstore.ObjectStore, volume ebsmetadata.Volume) {
|
||||
t.Helper()
|
||||
require.NoError(t, ebsmetadata.NewStore(store, "test-bucket").PutVolume(context.Background(), volume))
|
||||
}
|
||||
|
||||
// seedProviderVolume gives the service's provider a volume to snapshot. A
|
||||
// control-plane document alone is not a volume: the provider owns the blocks,
|
||||
// and CreateSnapshot fails against one it has never allocated.
|
||||
func seedProviderVolume(t *testing.T, svc *SnapshotServiceImpl, volumeID string, sizeGiB int) {
|
||||
t.Helper()
|
||||
if svc == nil || svc.provider == nil {
|
||||
return
|
||||
}
|
||||
_, err := svc.provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: int64(sizeGiB) * 1024 * 1024 * 1024},
|
||||
AvailabilityZone: "us-east-1a",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
@@ -68,7 +81,7 @@ func TestCreateSnapshot(t *testing.T) {
|
||||
|
||||
// Create a test volume first
|
||||
volumeID := "vol-test123"
|
||||
createTestVolume(t, store, volumeID, 100)
|
||||
createTestVolume(t, svc, store, volumeID, 100)
|
||||
|
||||
// Create snapshot
|
||||
result, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
@@ -111,71 +124,40 @@ func TestCreateSnapshot_MissingVolumeId(t *testing.T) {
|
||||
|
||||
// TestCreateSnapshot_VolumeZeroSize tests that creating a snapshot from a volume with zero SizeGiB fails.
|
||||
func TestCreateSnapshot_VolumeZeroSize(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
svc, _ := setupTestSnapshotService(t)
|
||||
|
||||
// Store a volume config with SizeGiB == 0
|
||||
volumeState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
SizeGiB: 0,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(volumeState)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: "vol-zerosize", TenantID: testAccountID, CapacityGiB: 0,
|
||||
State: "available", AvailabilityZone: "us-east-1a",
|
||||
}))
|
||||
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String("vol-zerosize/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
_, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-zerosize"),
|
||||
}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// createEncryptedTestVolume seeds config.json as an at-rest encryption envelope
|
||||
// ({payload, authtag}) wrapping a VBState, matching what an encrypted volume
|
||||
// persists. The snapshot handler must unwrap it via StateBody, not decode raw.
|
||||
func createEncryptedTestVolume(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string, sizeGiB int) {
|
||||
inner := viperblock.VBState{
|
||||
EncryptionEnabled: true,
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
SizeGiB: uint64(sizeGiB),
|
||||
AvailabilityZone: "us-east-1a",
|
||||
},
|
||||
},
|
||||
}
|
||||
payload, err := json.Marshal(inner)
|
||||
require.NoError(t, err)
|
||||
|
||||
envelope, err := json.Marshal(map[string]any{
|
||||
"payload": json.RawMessage(payload),
|
||||
"authtag": "deadbeefdeadbeefdeadbeefdeadbeef",
|
||||
// createEncryptedTestVolume seeds an encrypted volume. Encryption is a fact
|
||||
// the document records; the provider's own at-rest envelope is its business.
|
||||
func createEncryptedTestVolume(t *testing.T, svc *SnapshotServiceImpl, store *objectstore.MemoryObjectStore, volumeID string, sizeGiB int) {
|
||||
t.Helper()
|
||||
seedProviderVolume(t, svc, volumeID, sizeGiB)
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
CapacityGiB: uint64(sizeGiB),
|
||||
AvailabilityZone: "us-east-1a",
|
||||
Encrypted: true,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(envelope)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// TestCreateSnapshot_EncryptedVolumeEnvelope verifies an encrypted volume (whose
|
||||
// config.json is an envelope) snapshots successfully instead of 500ing on a
|
||||
// zero-decoded VBState.
|
||||
func TestCreateSnapshot_EncryptedVolumeEnvelope(t *testing.T) {
|
||||
// TestCreateSnapshot_EncryptedVolume verifies an encrypted volume snapshots
|
||||
// successfully and carries its Encrypted flag onto the snapshot.
|
||||
func TestCreateSnapshot_EncryptedVolume(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
volumeID := "vol-enc-snap"
|
||||
createEncryptedTestVolume(t, store, volumeID, 100)
|
||||
createEncryptedTestVolume(t, svc, store, volumeID, 100)
|
||||
|
||||
result, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String(volumeID),
|
||||
@@ -186,31 +168,15 @@ func TestCreateSnapshot_EncryptedVolumeEnvelope(t *testing.T) {
|
||||
assert.True(t, *result.Encrypted)
|
||||
}
|
||||
|
||||
// TestSnapshotInUseByVolumes_EncryptedVolume verifies the membership scan sees an
|
||||
// encrypted volume's source SnapshotID through the envelope (a raw decode would
|
||||
// miss it, letting an in-use snapshot be deleted).
|
||||
// TestSnapshotInUseByVolumes_EncryptedVolume verifies the membership scan sees
|
||||
// an encrypted clone's source SnapshotID, without which an in-use snapshot
|
||||
// could be deleted out from under the volume built on it.
|
||||
func TestSnapshotInUseByVolumes_EncryptedVolume(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
inner := viperblock.VBState{
|
||||
EncryptionEnabled: true,
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{SizeGiB: 50, SnapshotID: "snap-src"},
|
||||
},
|
||||
}
|
||||
payload, err := json.Marshal(inner)
|
||||
require.NoError(t, err)
|
||||
envelope, err := json.Marshal(map[string]any{
|
||||
"payload": json.RawMessage(payload),
|
||||
"authtag": "deadbeefdeadbeefdeadbeefdeadbeef",
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: "vol-fromsnap", CapacityGiB: 50, SnapshotID: "snap-src", Encrypted: true,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String("vol-fromsnap/config.json"),
|
||||
Body: strings.NewReader(string(envelope)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
inUse, err := svc.snapshotInUseByVolumes(context.Background(), "snap-src")
|
||||
require.NoError(t, err)
|
||||
@@ -233,8 +199,8 @@ func TestDescribeSnapshots(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
// Create test volumes
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, store, "vol-2", 100)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-2", 100)
|
||||
|
||||
// Create multiple snapshots
|
||||
snap1, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
@@ -290,7 +256,7 @@ func TestDescribeSnapshots_ByID(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
// Create test volume
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
|
||||
// Create multiple snapshots
|
||||
snap1, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
@@ -319,7 +285,7 @@ func TestDescribeSnapshots_ByID(t *testing.T) {
|
||||
// TestDescribeSnapshots_FilterNoResults).
|
||||
func TestDescribeSnapshots_NotFound(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
|
||||
snap1, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-1"),
|
||||
@@ -347,7 +313,7 @@ func TestDescribeSnapshots_Empty(t *testing.T) {
|
||||
func TestDescribeSnapshots_AccountScoping(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
|
||||
// Account A creates a snapshot
|
||||
snapA, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
@@ -379,7 +345,7 @@ func TestDeleteSnapshot(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
// Create test volume and snapshot
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-1"),
|
||||
}, testAccountID)
|
||||
@@ -416,7 +382,7 @@ func TestDeleteSnapshot(t *testing.T) {
|
||||
func TestDeleteSnapshot_WrongAccount(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-1"),
|
||||
}, testAccountID)
|
||||
@@ -442,29 +408,16 @@ func TestDeleteSnapshot_InUseByVolume(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
// Create a test volume and snapshot
|
||||
createTestVolume(t, store, "vol-source", 50)
|
||||
createTestVolume(t, svc, store, "vol-source", 50)
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-source"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Create a volume that references this snapshot (simulates CreateVolume from snapshot)
|
||||
volumeState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
SizeGiB: 50,
|
||||
SnapshotID: *snap.SnapshotId,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(volumeState)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String("vol-cloned/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: "vol-cloned", CapacityGiB: 50, SnapshotID: *snap.SnapshotId,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Attempt to delete the snapshot — should fail
|
||||
_, err = svc.DeleteSnapshot(context.Background(), &ec2.DeleteSnapshotInput{
|
||||
@@ -501,64 +454,41 @@ func TestDeleteSnapshot_MissingID(t *testing.T) {
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorInvalidParameterValue)
|
||||
}
|
||||
|
||||
// TestCopySnapshot tests copying a snapshot.
|
||||
func TestCopySnapshot(t *testing.T) {
|
||||
// Without a provider a copy would need snapshot metadata under the new ID that
|
||||
// the control plane cannot re-seal on an encrypted volume. Writing only the
|
||||
// control-plane config produced a snapshot that described fine and then failed
|
||||
// at attach, so this path refuses instead of producing one.
|
||||
func TestCopySnapshot_UnsupportedWithoutProvider(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
// Create test volume and snapshot
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-1"),
|
||||
Description: aws.String("Original snapshot"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Copy snapshot
|
||||
copyResult, err := svc.CopySnapshot(context.Background(), &ec2.CopySnapshotInput{
|
||||
SourceSnapshotId: snap.SnapshotId,
|
||||
Description: aws.String("Copied snapshot"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, copyResult)
|
||||
assert.True(t, strings.HasPrefix(*copyResult.SnapshotId, "snap-"))
|
||||
assert.NotEqual(t, *snap.SnapshotId, *copyResult.SnapshotId)
|
||||
|
||||
// Verify both snapshots exist
|
||||
result, err := svc.DescribeSnapshots(context.Background(), &ec2.DescribeSnapshotsInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, result.Snapshots, 2)
|
||||
}
|
||||
|
||||
// TestCopySnapshot_SetsCallerAsOwner tests that copied snapshot is owned by the caller.
|
||||
func TestCopySnapshot_SetsCallerAsOwner(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-1"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Copy snapshot as the same account
|
||||
copyResult, err := svc.CopySnapshot(context.Background(), &ec2.CopySnapshotInput{
|
||||
SourceSnapshotId: snap.SnapshotId,
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
// Drop the provider only for the copy: the source snapshot has to exist
|
||||
// for the refusal to be about the copy rather than about the source.
|
||||
svc.SetEBSProvider(nil)
|
||||
|
||||
// Verify copied snapshot owner is the caller
|
||||
result, err := svc.DescribeSnapshots(context.Background(), &ec2.DescribeSnapshotsInput{
|
||||
SnapshotIds: []*string{copyResult.SnapshotId},
|
||||
_, err = svc.CopySnapshot(context.Background(), &ec2.CopySnapshotInput{
|
||||
SourceSnapshotId: snap.SnapshotId,
|
||||
Description: aws.String("Copied snapshot"),
|
||||
}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorUnsupportedOperation, err.Error())
|
||||
|
||||
// The refusal must not leave a half-created snapshot behind.
|
||||
result, err := svc.DescribeSnapshots(context.Background(), &ec2.DescribeSnapshotsInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, result.Snapshots, 1)
|
||||
assert.Equal(t, testAccountID, *result.Snapshots[0].OwnerId)
|
||||
assert.Len(t, result.Snapshots, 1, "only the source snapshot should exist")
|
||||
}
|
||||
|
||||
// TestCopySnapshot_WrongAccount tests that account B cannot copy account A's snapshot.
|
||||
func TestCopySnapshot_WrongAccount(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
createTestVolume(t, svc, store, "vol-1", 50)
|
||||
_, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-1"),
|
||||
}, testAccountID)
|
||||
@@ -597,42 +527,6 @@ func TestCopySnapshot_MissingSourceID(t *testing.T) {
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorInvalidParameterValue)
|
||||
}
|
||||
|
||||
// TestCopySnapshot_PreservesTags tests that tags are copied.
|
||||
func TestCopySnapshot_PreservesTags(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
// Create test volume and snapshot with tags
|
||||
createTestVolume(t, store, "vol-1", 50)
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-1"),
|
||||
TagSpecifications: []*ec2.TagSpecification{
|
||||
{
|
||||
ResourceType: aws.String("snapshot"),
|
||||
Tags: []*ec2.Tag{
|
||||
{Key: aws.String("Environment"), Value: aws.String("test")},
|
||||
},
|
||||
},
|
||||
},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Copy snapshot
|
||||
copyResult, err := svc.CopySnapshot(context.Background(), &ec2.CopySnapshotInput{
|
||||
SourceSnapshotId: snap.SnapshotId,
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify copied snapshot has tags
|
||||
result, err := svc.DescribeSnapshots(context.Background(), &ec2.DescribeSnapshotsInput{
|
||||
SnapshotIds: []*string{copyResult.SnapshotId},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, result.Snapshots, 1)
|
||||
assert.Len(t, result.Snapshots[0].Tags, 1)
|
||||
assert.Equal(t, "Environment", *result.Snapshots[0].Tags[0].Key)
|
||||
assert.Equal(t, "test", *result.Snapshots[0].Tags[0].Value)
|
||||
}
|
||||
|
||||
// setupTestNATSKV creates a NATS JetStream test server and returns a KV bucket for testing.
|
||||
func setupTestNATSKV(t *testing.T) jetstream.KeyValue {
|
||||
t.Helper()
|
||||
@@ -747,9 +641,10 @@ func TestCreateSnapshot_WritesKVEntry(t *testing.T) {
|
||||
},
|
||||
}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil, kv)
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
|
||||
volumeID := "vol-kvtest"
|
||||
createTestVolume(t, store, volumeID, 10)
|
||||
createTestVolume(t, svc, store, volumeID, 10)
|
||||
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String(volumeID),
|
||||
@@ -779,9 +674,10 @@ func TestDeleteSnapshot_RemovesKVEntry(t *testing.T) {
|
||||
},
|
||||
}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil, kv)
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
|
||||
volumeID := "vol-kvdelete"
|
||||
createTestVolume(t, store, volumeID, 10)
|
||||
createTestVolume(t, svc, store, volumeID, 10)
|
||||
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String(volumeID),
|
||||
@@ -800,66 +696,22 @@ func TestDeleteSnapshot_RemovesKVEntry(t *testing.T) {
|
||||
assert.False(t, has)
|
||||
}
|
||||
|
||||
func TestCopySnapshot_AddsKVEntry(t *testing.T) {
|
||||
kv := setupTestNATSKV(t)
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{
|
||||
Predastore: config.PredastoreConfig{
|
||||
Bucket: "test-bucket",
|
||||
AccessKey: "test-owner-123",
|
||||
},
|
||||
}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil, kv)
|
||||
|
||||
volumeID := "vol-kvcopy"
|
||||
createTestVolume(t, store, volumeID, 10)
|
||||
|
||||
snap, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String(volumeID),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
copyResult, err := svc.CopySnapshot(context.Background(), &ec2.CopySnapshotInput{
|
||||
SourceSnapshotId: snap.SnapshotId,
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Both snapshot IDs should be in KV
|
||||
entry, err := kv.Get(t.Context(), volumeID)
|
||||
require.NoError(t, err)
|
||||
var snapshots []string
|
||||
require.NoError(t, json.Unmarshal(entry.Value(), &snapshots))
|
||||
assert.Contains(t, snapshots, *snap.SnapshotId)
|
||||
assert.Contains(t, snapshots, *copyResult.SnapshotId)
|
||||
assert.Len(t, snapshots, 2)
|
||||
}
|
||||
|
||||
// TestCreateSnapshot_CrossAccountVolumeRejected tests that snapshotting another account's volume is rejected.
|
||||
func TestCreateSnapshot_CrossAccountVolumeRejected(t *testing.T) {
|
||||
svc, store := setupTestSnapshotService(t)
|
||||
|
||||
// Create a volume owned by testAccountID (via TenantID)
|
||||
volumeID := "vol-owned-by-alpha"
|
||||
volumeState := viperblock.VBState{
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
SizeGiB: 100,
|
||||
TenantID: testAccountID,
|
||||
AvailabilityZone: "us-east-1a",
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(volumeState)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
seedProviderVolume(t, svc, volumeID, 100)
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
CapacityGiB: 100,
|
||||
TenantID: testAccountID,
|
||||
AvailabilityZone: "us-east-1a",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
// Another account tries to snapshot the volume — should fail
|
||||
_, err = svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
_, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String(volumeID),
|
||||
}, otherAccountID)
|
||||
require.Error(t, err)
|
||||
@@ -879,7 +731,7 @@ func TestCreateSnapshot_PrePhase4VolumeAllowed(t *testing.T) {
|
||||
|
||||
// Create a volume with no TenantID (pre-phase4)
|
||||
volumeID := "vol-legacy"
|
||||
createTestVolume(t, store, volumeID, 50)
|
||||
createTestVolume(t, svc, store, volumeID, 50)
|
||||
|
||||
// Any account can snapshot — backward compatibility
|
||||
result, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
@@ -894,7 +746,7 @@ func TestCreateSnapshot_PrePhase4VolumeAllowed(t *testing.T) {
|
||||
// createTestSnapshot creates a snapshot and returns its ID.
|
||||
func createTestSnapshot(t *testing.T, svc *SnapshotServiceImpl, store *objectstore.MemoryObjectStore, volumeID string, sizeGiB int, tags map[string]string) string {
|
||||
t.Helper()
|
||||
createTestVolume(t, store, volumeID, sizeGiB)
|
||||
createTestVolume(t, svc, store, volumeID, sizeGiB)
|
||||
|
||||
tagSpecs := []*ec2.TagSpecification{}
|
||||
if len(tags) > 0 {
|
||||
@@ -1106,94 +958,264 @@ func TestDescribeSnapshots_FilterNoFilters(t *testing.T) {
|
||||
assert.Len(t, out.Snapshots, 2)
|
||||
}
|
||||
|
||||
// TestCreateSnapshot_PredastoreInitFails exercises snapshotVolume past the nil-host guard.
|
||||
// The viperblock S3 backend fails at Init() when the test server returns 403, covering the
|
||||
// snapshotVolume body (S3Config build, LoadViperblockMasterKey, viperblock.New, Backend.Init)
|
||||
// and the CreateSnapshot error return path.
|
||||
func TestCreateSnapshot_PredastoreInitFails(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusForbidden)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
func TestCreateDeleteSnapshot_UsesInjectedProvider(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{
|
||||
Predastore: config.PredastoreConfig{
|
||||
Host: srv.URL,
|
||||
Region: "us-east-1",
|
||||
Bucket: "test-bucket",
|
||||
},
|
||||
}
|
||||
cfg := &config.Config{Predastore: config.PredastoreConfig{Bucket: "test-bucket"}}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil)
|
||||
createTestVolume(t, store, "vol-initfail", 10)
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc.SetEBSProvider(provider)
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: "vol-provider", TenantID: testAccountID, CapacityGiB: 4, State: "available", AvailabilityZone: "us-east-1a", ProviderHandle: "memory://volume/vol-provider",
|
||||
}))
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-provider", CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4 * 1024 * 1024 * 1024}, AvailabilityZone: "us-east-1a"})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-initfail"),
|
||||
snapshot, err := svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-provider")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, snapshot)
|
||||
cfgStored, err := svc.getSnapshotConfig(aws.StringValue(snapshot.SnapshotId))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "memory://snapshot/"+aws.StringValue(snapshot.SnapshotId), cfgStored.ProviderHandle)
|
||||
_, err = svc.DeleteSnapshot(context.Background(), &ec2.DeleteSnapshotInput{SnapshotId: snapshot.SnapshotId}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// A provider-managed clone records its source snapshot only in ebsmetadata, so
|
||||
// the legacy config.json scan cannot see it. Deleting the snapshot underneath
|
||||
// one would strip chunks the clone still reads.
|
||||
func TestDeleteSnapshot_Provider_BlockedByCloneRecordedInMetadata(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{Predastore: config.PredastoreConfig{Bucket: "test-bucket"}}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil)
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc.SetEBSProvider(provider)
|
||||
|
||||
require.NoError(t, svc.metadata.PutVolume(ctx, ebsmetadata.Volume{
|
||||
VolumeID: "vol-source", TenantID: testAccountID, CapacityGiB: 4,
|
||||
State: "available", AvailabilityZone: "us-east-1a", ProviderHandle: "memory://volume/vol-source",
|
||||
}))
|
||||
_, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-source",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4 * 1024 * 1024 * 1024}, AvailabilityZone: "us-east-1a",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
snapshot, err := svc.CreateSnapshot(ctx, &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-source")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
snapshotID := aws.StringValue(snapshot.SnapshotId)
|
||||
|
||||
// The clone exists only as a control-plane document, as CreateVolume's
|
||||
// provider branch writes it.
|
||||
require.NoError(t, svc.metadata.PutVolume(ctx, ebsmetadata.Volume{
|
||||
VolumeID: "vol-clone", TenantID: testAccountID, CapacityGiB: 4, State: "available",
|
||||
AvailabilityZone: "us-east-1a", SnapshotID: snapshotID, ProviderHandle: "memory://volume/vol-clone",
|
||||
}))
|
||||
|
||||
_, err = svc.DeleteSnapshot(ctx, &ec2.DeleteSnapshotInput{SnapshotId: aws.String(snapshotID)}, testAccountID)
|
||||
assert.EqualError(t, err, awserrors.ErrorInvalidSnapshotInUse)
|
||||
|
||||
// Once the clone is gone the snapshot is deletable again.
|
||||
require.NoError(t, svc.metadata.DeleteVolume(ctx, "vol-clone"))
|
||||
_, err = svc.DeleteSnapshot(ctx, &ec2.DeleteSnapshotInput{SnapshotId: aws.String(snapshotID)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// snapshotMetadataFailingObjectStore fails PutObject for any snapshot
|
||||
// metadata.json write and records the last such key, so a test can recover
|
||||
// the randomly generated snapshot ID CreateSnapshot attempted to persist.
|
||||
type snapshotMetadataFailingObjectStore struct {
|
||||
objectstore.ObjectStore
|
||||
|
||||
attemptedKey string
|
||||
}
|
||||
|
||||
func (s *snapshotMetadataFailingObjectStore) PutObject(ctx context.Context, input *s3.PutObjectInput) (*s3.PutObjectOutput, error) {
|
||||
key := aws.StringValue(input.Key)
|
||||
if strings.HasSuffix(key, "/metadata.json") {
|
||||
s.attemptedKey = key
|
||||
return nil, errors.New("simulated metadata write failure")
|
||||
}
|
||||
return s.ObjectStore.PutObject(ctx, input)
|
||||
}
|
||||
|
||||
// TestCreateSnapshot_Provider_RollbackOnMetadataWriteFailure covers
|
||||
// CreateSnapshot's rollback path: when persisting the snapshot config fails
|
||||
// after the provider has already created the snapshot, CreateSnapshot must
|
||||
// delete the just-created provider snapshot rather than orphan it.
|
||||
func TestCreateSnapshot_Provider_RollbackOnMetadataWriteFailure(t *testing.T) {
|
||||
store := &snapshotMetadataFailingObjectStore{ObjectStore: objectstore.NewMemoryObjectStore()}
|
||||
cfg := &config.Config{Predastore: config.PredastoreConfig{Bucket: "test-bucket"}}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil)
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc.SetEBSProvider(provider)
|
||||
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: "vol-rollback", TenantID: testAccountID, CapacityGiB: 4, State: "available",
|
||||
AvailabilityZone: "us-east-1a", ProviderHandle: "memory://volume/vol-rollback",
|
||||
}))
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-rollback",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4 * 1024 * 1024 * 1024}, AvailabilityZone: "us-east-1a",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.CreateSnapshot(context.Background(), &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-rollback")}, testAccountID)
|
||||
require.EqualError(t, err, awserrors.ErrorServerInternal)
|
||||
require.NotEmpty(t, store.attemptedKey, "the snapshot metadata write must have been attempted")
|
||||
|
||||
snapshotID := strings.TrimSuffix(store.attemptedKey, "/metadata.json")
|
||||
_, err = provider.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-rollback"})
|
||||
require.NoError(t, err, "the source volume must be unaffected by the snapshot rollback")
|
||||
|
||||
// Recreating the same snapshot ID against a different source volume only
|
||||
// succeeds if the original snapshot no longer exists (MemoryProvider
|
||||
// returns already_exists for a conflicting source volume on a live
|
||||
// snapshot ID), proving the rollback actually deleted it rather than
|
||||
// merely leaving it orphaned but reachable.
|
||||
_, err = provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-rollback-check", CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 * 1024 * 1024 * 1024},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, VolumeID: "vol-rollback-check",
|
||||
})
|
||||
require.NoError(t, err, "rollback must have deleted the just-created provider snapshot")
|
||||
}
|
||||
|
||||
// providerSnapshotService builds a snapshot service backed by the memory
|
||||
// provider with one volume registered, ready to snapshot.
|
||||
func providerSnapshotService(t *testing.T, volumeID string) (*SnapshotServiceImpl, *ebsprovider.MemoryProvider) {
|
||||
t.Helper()
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{Predastore: config.PredastoreConfig{Bucket: "test-bucket"}}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, store, nil)
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
svc.SetEBSProvider(provider)
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testAccountID, CapacityGiB: 4, State: "available",
|
||||
AvailabilityZone: "us-east-1a", ProviderHandle: "memory://volume/" + volumeID,
|
||||
}))
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4 * 1024 * 1024 * 1024},
|
||||
AvailabilityZone: "us-east-1a",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
return svc, provider
|
||||
}
|
||||
|
||||
// The copy must exist in the provider under its own ID, not just as a
|
||||
// control-plane document. That gap is what made the previous implementation
|
||||
// describe a snapshot that failed at attach.
|
||||
func TestCopySnapshot_Provider_CreatesBackendSnapshot(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, provider := providerSnapshotService(t, "vol-copysrc")
|
||||
|
||||
src, err := svc.CreateSnapshot(ctx, &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-copysrc")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
out, err := svc.CopySnapshot(ctx, &ec2.CopySnapshotInput{
|
||||
SourceSnapshotId: src.SnapshotId, Description: aws.String("copied"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, out.SnapshotId)
|
||||
newID := aws.StringValue(out.SnapshotId)
|
||||
assert.NotEqual(t, aws.StringValue(src.SnapshotId), newID)
|
||||
|
||||
got, ok := provider.Snapshot(newID)
|
||||
require.True(t, ok, "the copy must exist in the provider, not only in the control plane")
|
||||
assert.Equal(t, "vol-copysrc", got.SourceVolumeID)
|
||||
|
||||
stored, err := svc.getSnapshotConfig(newID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "vol-copysrc", stored.VolumeID)
|
||||
assert.Equal(t, "copied", stored.Description)
|
||||
assert.Equal(t, testAccountID, stored.OwnerID)
|
||||
}
|
||||
|
||||
// Both the copy and its source must describe. This service is built without
|
||||
// JetStream, so it covers the control-plane documents only — the snapshot-ref
|
||||
// rollback needs a KV-backed service and is not exercised here.
|
||||
func TestCopySnapshot_Provider_CopyAndSourceBothDescribe(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, _ := providerSnapshotService(t, "vol-copyref")
|
||||
|
||||
src, err := svc.CreateSnapshot(ctx, &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-copyref")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
out, err := svc.CopySnapshot(ctx, &ec2.CopySnapshotInput{SourceSnapshotId: src.SnapshotId}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
listed, err := svc.DescribeSnapshots(ctx, &ec2.DescribeSnapshotsInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
ids := make([]string, 0, len(listed.Snapshots))
|
||||
for _, s := range listed.Snapshots {
|
||||
ids = append(ids, aws.StringValue(s.SnapshotId))
|
||||
}
|
||||
assert.Contains(t, ids, aws.StringValue(out.SnapshotId))
|
||||
assert.Contains(t, ids, aws.StringValue(src.SnapshotId))
|
||||
}
|
||||
|
||||
// The copy inherits the source description when the caller supplies none,
|
||||
// rather than landing with an empty one.
|
||||
func TestCopySnapshot_Provider_InheritsSourceDescription(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, _ := providerSnapshotService(t, "vol-copydesc")
|
||||
|
||||
src, err := svc.CreateSnapshot(ctx, &ec2.CreateSnapshotInput{
|
||||
VolumeId: aws.String("vol-copydesc"), Description: aws.String("original text"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
out, err := svc.CopySnapshot(ctx, &ec2.CopySnapshotInput{SourceSnapshotId: src.SnapshotId}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
stored, err := svc.getSnapshotConfig(aws.StringValue(out.SnapshotId))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "original text", stored.Description)
|
||||
}
|
||||
|
||||
// A provider failure must leave nothing behind: no control-plane document and
|
||||
// no entry in the volume's snapshot list.
|
||||
func TestCopySnapshot_Provider_FailureLeavesNothingBehind(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, provider := providerSnapshotService(t, "vol-copyfail")
|
||||
|
||||
src, err := svc.CreateSnapshot(ctx, &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-copyfail")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
svc.SetEBSProvider(©FailingProvider{EBSProvider: provider})
|
||||
_, err = svc.CopySnapshot(ctx, &ec2.CopySnapshotInput{SourceSnapshotId: src.SnapshotId}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
|
||||
listed, err := svc.DescribeSnapshots(ctx, &ec2.DescribeSnapshotsInput{}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, listed.Snapshots, 1, "only the source snapshot should exist")
|
||||
}
|
||||
|
||||
// s3MockServer returns a test server that passes ListObjectsV2 (Backend.Init) but
|
||||
// returns 404 for every other request (GetObject, PutObject, etc.).
|
||||
func s3MockServer(t *testing.T) *httptest.Server {
|
||||
t.Helper()
|
||||
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if strings.Contains(r.URL.RawQuery, "list-type=2") {
|
||||
w.Header().Set("Content-Type", "application/xml")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
io.WriteString(w, `<?xml version="1.0" encoding="UTF-8"?>`+
|
||||
`<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">`+
|
||||
`<Name>test-bucket</Name><KeyCount>0</KeyCount>`+
|
||||
`<MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>`)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/xml")
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
io.WriteString(w, `<?xml version="1.0" encoding="UTF-8"?>`+
|
||||
`<Error><Code>NoSuchKey</Code>`+
|
||||
`<Message>The specified key does not exist.</Message></Error>`)
|
||||
}))
|
||||
}
|
||||
// Ownership is checked before the copy runs, so another tenant's snapshot is
|
||||
// refused rather than duplicated into the caller's account.
|
||||
func TestCopySnapshot_Provider_ForeignSnapshotRefused(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, provider := providerSnapshotService(t, "vol-copyown")
|
||||
|
||||
func TestSnapshotVolume_EncryptionKeyLoadError(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer srv.Close()
|
||||
src, err := svc.CreateSnapshot(ctx, &ec2.CreateSnapshotInput{VolumeId: aws.String("vol-copyown")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
cfg := &config.Config{
|
||||
Predastore: config.PredastoreConfig{
|
||||
Host: srv.URL,
|
||||
Region: "us-east-1",
|
||||
Bucket: "test-bucket",
|
||||
},
|
||||
Viperblock: config.ViperblockConfig{
|
||||
EncryptionKeyFile: "/nonexistent-snap-test-key.bin",
|
||||
},
|
||||
}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, objectstore.NewMemoryObjectStore(), nil)
|
||||
|
||||
err := svc.snapshotVolume("vol-enckey", "snap-enckey", 10*1024*1024*1024)
|
||||
_, err = svc.CopySnapshot(ctx, &ec2.CopySnapshotInput{SourceSnapshotId: src.SnapshotId}, "999988887777")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "load encryption key")
|
||||
assert.Equal(t, awserrors.ErrorUnauthorizedOperation, err.Error())
|
||||
|
||||
_, ok := provider.Snapshot(aws.StringValue(src.SnapshotId))
|
||||
require.True(t, ok, "the source must survive a refused copy")
|
||||
}
|
||||
|
||||
func TestSnapshotVolume_LoadStateFails(t *testing.T) {
|
||||
srv := s3MockServer(t)
|
||||
defer srv.Close()
|
||||
|
||||
cfg := &config.Config{
|
||||
Predastore: config.PredastoreConfig{
|
||||
Host: srv.URL,
|
||||
Region: "us-east-1",
|
||||
Bucket: "test-bucket",
|
||||
AccessKey: "test-key",
|
||||
SecretKey: "test-secret",
|
||||
},
|
||||
}
|
||||
svc := NewSnapshotServiceImplWithStore(cfg, objectstore.NewMemoryObjectStore(), nil)
|
||||
|
||||
err := svc.snapshotVolume("vol-lsf", "snap-lsf", 10*1024*1024*1024)
|
||||
require.Error(t, err)
|
||||
// copyFailingProvider fails only CopySnapshot, leaving every other operation
|
||||
// intact so the rollback path is what the test exercises.
|
||||
type copyFailingProvider struct {
|
||||
ebsprovider.EBSProvider
|
||||
}
|
||||
|
||||
func (p *copyFailingProvider) CopySnapshot(context.Context, ebsprovider.CopySnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
return nil, errors.New("provider copy failed")
|
||||
}
|
||||
|
||||
@@ -1,54 +1,29 @@
|
||||
package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ec2/volumestate"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// seedUnencryptedConfig writes a plain (EncryptionEnabled=false) full VBState.
|
||||
func seedUnencryptedConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string) {
|
||||
t.Helper()
|
||||
state := viperblock.VBState{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: 10 * 1024 * 1024 * 1024,
|
||||
BlockSize: 4096,
|
||||
SeqNum: 7,
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{VolumeID: volumeID, SizeGiB: 10, State: "available"},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(state)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// TestUpdateVolumeState_WritesStateJSONNotConfig locks the single-writer
|
||||
// TestUpdateVolumeState_WritesDocumentNotConfig locks the single-writer
|
||||
// contract: the live nbdkit VB owns config.json and rewrites it from its stale
|
||||
// in-memory State on every SaveState, so the control plane MUST persist
|
||||
// attachment state to the separate state.json object and leave config.json
|
||||
// byte-for-byte untouched. Writing State into config.json out-of-band is
|
||||
// silently clobbered under write load (the EKS-worker root-volume flip).
|
||||
func TestUpdateVolumeState_WritesStateJSONNotConfig(t *testing.T) {
|
||||
// attachment state to its own document and leave config.json byte-for-byte
|
||||
// untouched. Writing State into config.json out-of-band is silently clobbered
|
||||
// under write load (the EKS-worker root-volume flip).
|
||||
func TestUpdateVolumeState_WritesDocumentNotConfig(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
|
||||
volumeID := "vol-single-writer"
|
||||
seedUnencryptedConfig(t, store, volumeID)
|
||||
seedProviderConfig(t, store, volumeID)
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testVolAccountID, CapacityGiB: 10, State: "available",
|
||||
})
|
||||
before := getStoredConfig(t, store, volumeID)
|
||||
|
||||
require.NoError(t, svc.UpdateVolumeState(volumeID, "in-use", "i-abc", "/dev/nbd0"))
|
||||
@@ -57,23 +32,27 @@ func TestUpdateVolumeState_WritesStateJSONNotConfig(t *testing.T) {
|
||||
assert.Equal(t, string(before), string(after),
|
||||
"UpdateVolumeState must not write config.json; the live VB owns it")
|
||||
|
||||
rec := getStoredState(t, store, volumeID)
|
||||
assert.Equal(t, "in-use", rec.State)
|
||||
assert.Equal(t, "i-abc", rec.AttachedInstance)
|
||||
assert.Equal(t, "/dev/nbd0", rec.DeviceName)
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "in-use", meta.State)
|
||||
assert.Equal(t, "i-abc", meta.AttachedInstance)
|
||||
assert.Equal(t, "/dev/nbd0", meta.DeviceName)
|
||||
}
|
||||
|
||||
// TestUpdateVolumeState_EncryptedConfigUntouched locks the corruption-safety
|
||||
// half of the contract: config.json for an encrypted volume is a sealed VBState
|
||||
// whose AES-GCM nonce is derived from StateSeqNum. A second out-of-band writer
|
||||
// advances the nonce and reuses it (catastrophic for AES-GCM — the decrypted
|
||||
// garbage image layers). UpdateVolumeState MUST never touch the sealed object.
|
||||
// half of the contract: config.json for an encrypted volume is a sealed blob
|
||||
// whose AES-GCM nonce is derived from its sequence number. A second out-of-band
|
||||
// writer advances the nonce and reuses it (catastrophic for AES-GCM — the
|
||||
// decrypted garbage image layers). UpdateVolumeState MUST never touch it.
|
||||
func TestUpdateVolumeState_EncryptedConfigUntouched(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
|
||||
volumeID := "vol-enc-single-writer"
|
||||
seedEncryptedConfig(t, store, volumeID)
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testVolAccountID, CapacityGiB: 10, State: "available",
|
||||
})
|
||||
before := getStoredConfig(t, store, volumeID)
|
||||
|
||||
require.NoError(t, svc.UpdateVolumeState(volumeID, "in-use", "i-enc", "/dev/nbd0"))
|
||||
@@ -81,54 +60,48 @@ func TestUpdateVolumeState_EncryptedConfigUntouched(t *testing.T) {
|
||||
after := getStoredConfig(t, store, volumeID)
|
||||
assert.Equal(t, string(before), string(after),
|
||||
"UpdateVolumeState must not rewrite the sealed config.json (AES-GCM nonce reuse)")
|
||||
assert.Equal(t, "in-use", getStoredState(t, store, volumeID).State)
|
||||
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "in-use", meta.State)
|
||||
}
|
||||
|
||||
// TestGetVolumeConfig_OverlaysStateJSON locks the read side: the authoritative
|
||||
// attachment state is state.json, which MUST override the (stale) State the live
|
||||
// VB persisted into config.json.
|
||||
func TestGetVolumeConfig_OverlaysStateJSON(t *testing.T) {
|
||||
// TestGetVolumeMetadata_DocumentBeatsConfig locks the read side of the same
|
||||
// contract: config.json carries a State the provider wrote and the control
|
||||
// plane does not own, so a document that disagrees with it MUST win. Reading
|
||||
// State back out of config.json is what let a stale SaveState resurrect a
|
||||
// detached volume as attached.
|
||||
func TestGetVolumeMetadata_DocumentBeatsConfig(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
|
||||
volumeID := "vol-overlay"
|
||||
seedUnencryptedConfig(t, store, volumeID) // config.json State == "available"
|
||||
require.NoError(t, svc.UpdateVolumeState(volumeID, "in-use", "i-overlay", "/dev/nbd0"))
|
||||
|
||||
cfg, err := svc.GetVolumeConfig(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "in-use", cfg.VolumeMetadata.State,
|
||||
"state.json must override the stale State in config.json")
|
||||
assert.Equal(t, "i-overlay", cfg.VolumeMetadata.AttachedInstance)
|
||||
assert.Equal(t, "/dev/nbd0", cfg.VolumeMetadata.DeviceName)
|
||||
}
|
||||
|
||||
// TestGetVolumeConfig_FallsBackWhenNoStateJSON locks the migration path: a volume
|
||||
// predating the state.json split has no state.json, so the State embedded in
|
||||
// config.json is used unchanged.
|
||||
func TestGetVolumeConfig_FallsBackWhenNoStateJSON(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
|
||||
volumeID := "vol-legacy"
|
||||
seedUnencryptedConfig(t, store, volumeID) // State == "available", no state.json
|
||||
|
||||
cfg, err := svc.GetVolumeConfig(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "available", cfg.VolumeMetadata.State,
|
||||
"absent state.json must fall back to the State embedded in config.json")
|
||||
}
|
||||
|
||||
// getStoredState reads and decodes the raw state.json from the memory store.
|
||||
func getStoredState(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string) volumestate.Record {
|
||||
t.Helper()
|
||||
res, err := store.GetObject(context.Background(), &s3.GetObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/state.json"),
|
||||
volumeID := "vol-document-wins"
|
||||
seedProviderConfig(t, store, volumeID) // its embedded State says "available"
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testVolAccountID, CapacityGiB: 10,
|
||||
State: "in-use", AttachedInstance: "i-overlay", DeviceName: "/dev/nbd0",
|
||||
})
|
||||
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
defer res.Body.Close()
|
||||
var rec volumestate.Record
|
||||
require.NoError(t, json.NewDecoder(res.Body).Decode(&rec))
|
||||
return rec
|
||||
assert.Equal(t, "in-use", meta.State,
|
||||
"the ebsmetadata document must override any State in config.json")
|
||||
assert.Equal(t, "i-overlay", meta.AttachedInstance)
|
||||
assert.Equal(t, "/dev/nbd0", meta.DeviceName)
|
||||
}
|
||||
|
||||
// TestGetVolumeMetadata_ConfigAloneIsNotAVolume is the other half: provider
|
||||
// state without a document is not a volume the control plane knows. Falling
|
||||
// back to config.json would resurrect volumes whose document was deleted, so
|
||||
// an absent document must read as not-found.
|
||||
func TestGetVolumeMetadata_ConfigAloneIsNotAVolume(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
|
||||
volumeID := "vol-config-only"
|
||||
seedProviderConfig(t, store, volumeID)
|
||||
|
||||
_, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "InvalidVolume.NotFound")
|
||||
}
|
||||
|
||||
@@ -1,147 +0,0 @@
|
||||
package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// startTestNATS spins up an in-process NATS server and returns a connected client.
|
||||
func startTestNATS(t *testing.T) *nats.Conn {
|
||||
t.Helper()
|
||||
opts := &server.Options{
|
||||
Host: "127.0.0.1",
|
||||
Port: -1,
|
||||
JetStream: true,
|
||||
StoreDir: t.TempDir(),
|
||||
NoLog: true,
|
||||
NoSigs: true,
|
||||
}
|
||||
ns, err := server.NewServer(opts)
|
||||
require.NoError(t, err)
|
||||
go ns.Start()
|
||||
require.True(t, ns.ReadyForConnections(5*time.Second))
|
||||
t.Cleanup(func() { ns.Shutdown() })
|
||||
|
||||
nc, err := nats.Connect(ns.ClientURL())
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { nc.Close() })
|
||||
return nc
|
||||
}
|
||||
|
||||
// seedEncryptedConfig writes a sealed (EncryptionEnabled=true) VBState to the store.
|
||||
func seedEncryptedConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string) {
|
||||
t.Helper()
|
||||
state := viperblock.VBState{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: 10 * 1024 * 1024 * 1024,
|
||||
BlockSize: 4096,
|
||||
EncryptionEnabled: true,
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: volumeID,
|
||||
TenantID: testVolAccountID,
|
||||
SizeGiB: 10,
|
||||
State: "available",
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(state)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// getStoredConfig reads the raw config.json bytes from the memory store.
|
||||
func getStoredConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string) []byte {
|
||||
t.Helper()
|
||||
res, err := store.GetObject(context.Background(), &s3.GetObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
defer res.Body.Close()
|
||||
data, err := io.ReadAll(res.Body)
|
||||
require.NoError(t, err)
|
||||
return data
|
||||
}
|
||||
|
||||
// encConfig builds a VolumeConfig for an encrypted-volume putVolumeConfig call.
|
||||
func encConfig(volumeID string) *viperblock.VolumeConfig {
|
||||
return &viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{VolumeID: volumeID, SizeGiB: 10},
|
||||
}
|
||||
}
|
||||
|
||||
// TestPutVolumeConfig_EncryptedRoutesToQueueGroup verifies a config write for an
|
||||
// encrypted volume is delegated to the ebs.config keyholder queue group rather
|
||||
// than re-sealing the object out-of-band. The control plane lacks the master key,
|
||||
// so a direct PutObject would strip the AES-GCM tag and brick the volume. This is
|
||||
// the safe non-live writer path (markVolumeOrphaned / ModifyVolume); UpdateVolumeState
|
||||
// no longer routes here at all (it writes state.json).
|
||||
func TestPutVolumeConfig_EncryptedRoutesToQueueGroup(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
svc.natsConn = startTestNATS(t)
|
||||
|
||||
volumeID := "vol-enc-detached"
|
||||
seedEncryptedConfig(t, store, volumeID)
|
||||
|
||||
var hits atomic.Int32
|
||||
sub, err := svc.natsConn.QueueSubscribe("ebs.config", "spinifex-workers", func(msg *nats.Msg) {
|
||||
hits.Add(1)
|
||||
data, _ := json.Marshal(types.EBSConfigUpdateResponse{Volume: volumeID, Success: true})
|
||||
_ = msg.Respond(data)
|
||||
})
|
||||
require.NoError(t, err)
|
||||
defer sub.Unsubscribe()
|
||||
|
||||
require.NoError(t, svc.putVolumeConfig(context.Background(), volumeID, encConfig(volumeID)))
|
||||
assert.Equal(t, int32(1), hits.Load(), "encrypted config write must hit the ebs.config keyholder queue group")
|
||||
|
||||
// The sealed object must NOT have been rewritten out-of-band by the handler.
|
||||
raw := getStoredConfig(t, store, volumeID)
|
||||
var still viperblock.VBState
|
||||
require.NoError(t, json.Unmarshal(viperblock.StateBody(raw), &still))
|
||||
assert.True(t, still.EncryptionEnabled)
|
||||
}
|
||||
|
||||
// TestPutVolumeConfig_EncryptedKeyholderError surfaces a keyholder failure
|
||||
// instead of silently succeeding.
|
||||
func TestPutVolumeConfig_EncryptedKeyholderError(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
svc.natsConn = startTestNATS(t)
|
||||
|
||||
volumeID := "vol-enc-err"
|
||||
seedEncryptedConfig(t, store, volumeID)
|
||||
|
||||
sub, err := svc.natsConn.QueueSubscribe("ebs.config", "spinifex-workers", func(msg *nats.Msg) {
|
||||
data, _ := json.Marshal(types.EBSConfigUpdateResponse{Volume: volumeID, Error: "reseal failed"})
|
||||
_ = msg.Respond(data)
|
||||
})
|
||||
require.NoError(t, err)
|
||||
defer sub.Unsubscribe()
|
||||
|
||||
err = svc.putVolumeConfig(context.Background(), volumeID, encConfig(volumeID))
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "reseal failed")
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
)
|
||||
|
||||
// silenceLogs discards slog output for the duration of a benchmark's timed
|
||||
// loop, so per-call INFO logging (which scales with iteration count) doesn't
|
||||
// dominate the measured cost. Restores the previous default logger on return.
|
||||
func silenceLogs(b *testing.B) func() {
|
||||
b.Helper()
|
||||
prev := slog.Default()
|
||||
slog.SetDefault(slog.New(slog.DiscardHandler))
|
||||
return func() { slog.SetDefault(prev) }
|
||||
}
|
||||
|
||||
// seedProviderVolumes creates n volumes under accountID and returns the ID of
|
||||
// the last one created, so callers can benchmark a single-ID lookup against a
|
||||
// bucket holding n volume documents.
|
||||
func seedProviderVolumes(b *testing.B, svc *VolumeServiceImpl, n int, accountID string) string {
|
||||
b.Helper()
|
||||
ctx := context.Background()
|
||||
var lastID string
|
||||
for i := range n {
|
||||
vol, err := svc.CreateVolume(ctx, &ec2.CreateVolumeInput{
|
||||
Size: aws.Int64(8), AvailabilityZone: aws.String("ap-southeast-2a"),
|
||||
}, accountID)
|
||||
if err != nil {
|
||||
b.Fatalf("seed volume %d: %v", i, err)
|
||||
}
|
||||
lastID = aws.StringValue(vol.VolumeId)
|
||||
}
|
||||
return lastID
|
||||
}
|
||||
|
||||
// BenchmarkDescribeVolumes_Provider_SingleID measures the direct-GetVolume
|
||||
// fast path as the bucket grows. Before this change, a single-ID request
|
||||
// went through ListVolumes and filtered in memory, so cost scaled with the
|
||||
// number of volumes in the bucket (N document reads for 1 requested). The
|
||||
// fast path should stay flat across bucket sizes.
|
||||
func BenchmarkDescribeVolumes_Provider_SingleID(b *testing.B) {
|
||||
for _, n := range []int{1, 10, 100, 500} {
|
||||
b.Run(fmt.Sprintf("volumes=%d", n), func(b *testing.B) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
targetID := seedProviderVolumes(b, svc, n, "acct-bench")
|
||||
ctx := context.Background()
|
||||
input := &ec2.DescribeVolumesInput{VolumeIds: []*string{aws.String(targetID)}}
|
||||
|
||||
defer silenceLogs(b)()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if _, err := svc.DescribeVolumes(ctx, input, "acct-bench"); err != nil {
|
||||
b.Fatalf("DescribeVolumes: %v", err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkDescribeVolumes_Provider_ListAndFilter measures the pre-fix
|
||||
// approach directly (ListVolumes + in-memory filter for one ID), reusing the
|
||||
// same seeded buckets as BenchmarkDescribeVolumes_Provider_SingleID so the
|
||||
// two can be compared at matching bucket sizes to show how listing scales
|
||||
// with N while the fast path (above) does not.
|
||||
func BenchmarkDescribeVolumes_Provider_ListAndFilter(b *testing.B) {
|
||||
for _, n := range []int{1, 10, 100, 500} {
|
||||
b.Run(fmt.Sprintf("volumes=%d", n), func(b *testing.B) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
targetID := seedProviderVolumes(b, svc, n, "acct-bench")
|
||||
ctx := context.Background()
|
||||
|
||||
defer silenceLogs(b)()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
metadata, err := svc.metadata.ListVolumes(ctx)
|
||||
if err != nil {
|
||||
b.Fatalf("ListVolumes: %v", err)
|
||||
}
|
||||
var found *ec2.Volume
|
||||
for _, meta := range metadata {
|
||||
if meta.TenantID != "acct-bench" || meta.VolumeID != targetID {
|
||||
continue
|
||||
}
|
||||
found = metadataVolumeToEC2(meta)
|
||||
}
|
||||
if found == nil {
|
||||
b.Fatalf("volume %s not found", targetID)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// TestDescribeVolumes_Provider_FastPath_SingleID is the regression for the
|
||||
// +18ms single-volume DescribeVolumes latency: under the provider path, a
|
||||
// single requested ID must resolve via a direct GetVolume, not by listing
|
||||
// (and decoding) every volume document in the bucket.
|
||||
func TestDescribeVolumes_Provider_FastPath_SingleID(t *testing.T) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
ctx := context.Background()
|
||||
|
||||
vol, err := svc.CreateVolume(ctx, &ec2.CreateVolumeInput{Size: aws.Int64(8), AvailabilityZone: aws.String("ap-southeast-2a")}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
out, err := svc.DescribeVolumes(ctx, &ec2.DescribeVolumesInput{VolumeIds: []*string{vol.VolumeId}}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, out.Volumes, 1)
|
||||
assert.Equal(t, aws.StringValue(vol.VolumeId), aws.StringValue(out.Volumes[0].VolumeId))
|
||||
assert.Equal(t, int64(8), aws.Int64Value(out.Volumes[0].Size))
|
||||
}
|
||||
|
||||
// TestDescribeVolumes_Provider_FastPath_CrossTenantIsNotFound is the most
|
||||
// important test in this file: a directly-fetched volume belonging to a
|
||||
// different tenant must never be returned, and must report NotFound exactly
|
||||
// like an unknown ID — a regression here is a cross-tenant data leak.
|
||||
func TestDescribeVolumes_Provider_FastPath_CrossTenantIsNotFound(t *testing.T) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
ctx := context.Background()
|
||||
|
||||
otherTenantVol, err := svc.CreateVolume(ctx, &ec2.CreateVolumeInput{Size: aws.Int64(8), AvailabilityZone: aws.String("ap-southeast-2a")}, "acct-2")
|
||||
require.NoError(t, err)
|
||||
|
||||
out, err := svc.DescribeVolumes(ctx, &ec2.DescribeVolumesInput{VolumeIds: []*string{otherTenantVol.VolumeId}}, "acct-1")
|
||||
require.Error(t, err)
|
||||
assert.EqualError(t, err, awserrors.ErrorInvalidVolumeNotFound)
|
||||
assert.Nil(t, out)
|
||||
}
|
||||
|
||||
// TestDescribeVolumes_Provider_FastPath_UnknownIDIsNotFound covers a
|
||||
// requested ID with no ebsmetadata document at all.
|
||||
func TestDescribeVolumes_Provider_FastPath_UnknownIDIsNotFound(t *testing.T) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
ctx := context.Background()
|
||||
|
||||
out, err := svc.DescribeVolumes(ctx, &ec2.DescribeVolumesInput{VolumeIds: []*string{aws.String("vol-doesnotexist0")}}, "acct-1")
|
||||
require.Error(t, err)
|
||||
assert.EqualError(t, err, awserrors.ErrorInvalidVolumeNotFound)
|
||||
assert.Nil(t, out)
|
||||
}
|
||||
|
||||
// TestDescribeVolumes_Provider_FastPath_FiltersApply confirms parsedFilters
|
||||
// still apply to directly-fetched volumes exactly as they do to enumerated
|
||||
// ones: a requested ID that fails the filter is dropped, not returned.
|
||||
func TestDescribeVolumes_Provider_FastPath_FiltersApply(t *testing.T) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
ctx := context.Background()
|
||||
|
||||
vol, err := svc.CreateVolume(ctx, &ec2.CreateVolumeInput{Size: aws.Int64(8), AvailabilityZone: aws.String("ap-southeast-2a")}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
t.Run("matching filter keeps the volume", func(t *testing.T) {
|
||||
out, err := svc.DescribeVolumes(ctx, &ec2.DescribeVolumesInput{
|
||||
VolumeIds: []*string{vol.VolumeId},
|
||||
Filters: []*ec2.Filter{{Name: aws.String("size"), Values: []*string{aws.String("8")}}},
|
||||
}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, out.Volumes, 1)
|
||||
})
|
||||
|
||||
t.Run("non-matching filter excludes the volume", func(t *testing.T) {
|
||||
out, err := svc.DescribeVolumes(ctx, &ec2.DescribeVolumesInput{
|
||||
VolumeIds: []*string{vol.VolumeId},
|
||||
Filters: []*ec2.Filter{{Name: aws.String("size"), Values: []*string{aws.String("16")}}},
|
||||
}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, out.Volumes)
|
||||
})
|
||||
}
|
||||
|
||||
// TestDescribeVolumes_Provider_FastPath_MultipleIDs covers fetching more
|
||||
// than one directly-requested ID in the same call.
|
||||
func TestDescribeVolumes_Provider_FastPath_MultipleIDs(t *testing.T) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
ctx := context.Background()
|
||||
|
||||
volA, err := svc.CreateVolume(ctx, &ec2.CreateVolumeInput{Size: aws.Int64(8), AvailabilityZone: aws.String("ap-southeast-2a")}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
volB, err := svc.CreateVolume(ctx, &ec2.CreateVolumeInput{Size: aws.Int64(16), AvailabilityZone: aws.String("ap-southeast-2a")}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
out, err := svc.DescribeVolumes(ctx, &ec2.DescribeVolumesInput{VolumeIds: []*string{volA.VolumeId, volB.VolumeId}}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, out.Volumes, 2)
|
||||
|
||||
ids := map[string]bool{}
|
||||
for _, v := range out.Volumes {
|
||||
ids[aws.StringValue(v.VolumeId)] = true
|
||||
}
|
||||
assert.True(t, ids[aws.StringValue(volA.VolumeId)])
|
||||
assert.True(t, ids[aws.StringValue(volB.VolumeId)])
|
||||
}
|
||||
|
||||
// TestDescribeVolumes_Provider_NoIDs_StillEnumerates locks that the no-IDs
|
||||
// case keeps using ListVolumes, not the direct-fetch fast path.
|
||||
func TestDescribeVolumes_Provider_NoIDs_StillEnumerates(t *testing.T) {
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
ctx := context.Background()
|
||||
|
||||
volA, err := svc.CreateVolume(ctx, &ec2.CreateVolumeInput{Size: aws.Int64(8), AvailabilityZone: aws.String("ap-southeast-2a")}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
_, err = svc.CreateVolume(ctx, &ec2.CreateVolumeInput{Size: aws.Int64(16), AvailabilityZone: aws.String("ap-southeast-2a")}, "acct-2")
|
||||
require.NoError(t, err)
|
||||
|
||||
out, err := svc.DescribeVolumes(ctx, &ec2.DescribeVolumesInput{}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, out.Volumes, 1)
|
||||
assert.Equal(t, aws.StringValue(volA.VolumeId), aws.StringValue(out.Volumes[0].VolumeId))
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func newProviderVolumeService(t *testing.T) *VolumeServiceImpl {
|
||||
t.Helper()
|
||||
svc := newTestVolumeService("ap-southeast-2a")
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
return svc
|
||||
}
|
||||
|
||||
// TestGetVolumeMetadata_Provider_ResolvesProviderCreatedVolume is the regression
|
||||
// for the attach 404: a provider-created volume has no vol-*/config.json, so
|
||||
// reading the legacy key made the daemon report a healthy volume as
|
||||
// InvalidVolume.NotFound while DescribeVolumes showed it available.
|
||||
func TestGetVolumeMetadata_Provider_ResolvesProviderCreatedVolume(t *testing.T) {
|
||||
svc := newProviderVolumeService(t)
|
||||
|
||||
volumeID := "vol-provider00001"
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: volumeID,
|
||||
TenantID: "acct-provider",
|
||||
CapacityGiB: 8,
|
||||
State: "available",
|
||||
CreatedAt: time.Now(),
|
||||
AvailabilityZone: "ap-southeast-2a",
|
||||
VolumeType: "gp3",
|
||||
IOPS: 3000,
|
||||
Encrypted: true,
|
||||
}))
|
||||
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err, "a provider-created volume must resolve without a legacy config.json")
|
||||
assert.Equal(t, volumeID, meta.VolumeID)
|
||||
assert.Equal(t, "acct-provider", meta.TenantID)
|
||||
assert.Equal(t, "available", meta.State)
|
||||
assert.Equal(t, "ap-southeast-2a", meta.AvailabilityZone)
|
||||
assert.Equal(t, uint64(8), meta.CapacityGiB)
|
||||
}
|
||||
|
||||
// The daemon's attach validation reads State/AttachedInstance/DeviceName off
|
||||
// this config to decide VolumeInUse versus idempotent re-attach, so the
|
||||
// document's attachment fields have to survive the projection.
|
||||
func TestGetVolumeMetadata_Provider_ReflectsAttachmentState(t *testing.T) {
|
||||
svc := newProviderVolumeService(t)
|
||||
|
||||
volumeID := "vol-provider00002"
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: "acct-provider", CapacityGiB: 1,
|
||||
State: "available", AvailabilityZone: "ap-southeast-2a", VolumeType: "gp3",
|
||||
}))
|
||||
|
||||
require.NoError(t, svc.UpdateVolumeState(volumeID, "in-use", "i-0123456789abcdef0", "/dev/sdf"))
|
||||
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "in-use", meta.State)
|
||||
assert.Equal(t, "i-0123456789abcdef0", meta.AttachedInstance)
|
||||
assert.Equal(t, "/dev/sdf", meta.DeviceName)
|
||||
}
|
||||
|
||||
func TestGetVolumeMetadata_Provider_UnknownVolumeIsNotFound(t *testing.T) {
|
||||
svc := newProviderVolumeService(t)
|
||||
|
||||
_, err := svc.GetVolumeMetadata("vol-doesnotexist0")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidVolumeNotFound, err.Error())
|
||||
}
|
||||
@@ -11,10 +11,9 @@ import (
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/handlers/ec2/volumestate"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -42,22 +41,29 @@ func (s *recordingObjectStore) PutObject(ctx context.Context, input *s3.PutObjec
|
||||
return s.ObjectStore.PutObject(ctx, input)
|
||||
}
|
||||
|
||||
// seedVolume writes a volume config.json into the store so listAllVolumeIDs /
|
||||
// GetVolumeConfig see it. CreateVolume cannot be used in unit tests — it needs a
|
||||
// live viperblock S3 backend.
|
||||
// seedVolume writes the control-plane document the service reads, and gives the
|
||||
// provider the matching volume. CreateVolume cannot be used in unit tests — it
|
||||
// needs the cluster's encryption key file.
|
||||
func seedVolume(t *testing.T, svc *VolumeServiceImpl, volID, state, attachedInstance string) {
|
||||
t.Helper()
|
||||
cfg := &viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: volID,
|
||||
TenantID: testVolAccountID,
|
||||
SizeGiB: 8,
|
||||
State: state,
|
||||
AttachedInstance: attachedInstance,
|
||||
AvailabilityZone: "ap-southeast-2a",
|
||||
},
|
||||
}
|
||||
require.NoError(t, svc.putVolumeConfig(context.Background(), volID, cfg))
|
||||
seedTestProviderVolume(t, svc, volID, 8)
|
||||
require.NoError(t, svc.metadata.PutVolume(context.Background(), ebsmetadata.Volume{
|
||||
VolumeID: volID,
|
||||
TenantID: testVolAccountID,
|
||||
CapacityGiB: 8,
|
||||
State: state,
|
||||
AttachedInstance: attachedInstance,
|
||||
AvailabilityZone: "ap-southeast-2a",
|
||||
VolumeType: "gp3",
|
||||
}))
|
||||
}
|
||||
|
||||
// metadataKey is the object key the control-plane document for volumeID lives at.
|
||||
func metadataKey(t *testing.T, volumeID string) string {
|
||||
t.Helper()
|
||||
key, err := ebsmetadata.VolumeKey(volumeID)
|
||||
require.NoError(t, err)
|
||||
return key
|
||||
}
|
||||
|
||||
// TestRLC1_VolumeDeleteNotFoundOnAbsent enforces the Common Resource Lifecycle
|
||||
@@ -91,23 +97,23 @@ func TestRLC3_VolumeDeleteRequiresDetach(t *testing.T) {
|
||||
"ADR-0005 §2: deleting an attached volume must return VolumeInUse (detach-before-delete, rule #3)")
|
||||
}
|
||||
|
||||
// TestVolumeTagMirror_WritesOnlyControlPlaneTagsObject locks the single-writer
|
||||
// TestVolumeTagMirror_WritesOnlyControlPlaneDocument locks the single-writer
|
||||
// invariant: tag mutations must never rewrite config.json or route an encrypted
|
||||
// volume through the ebs.config keyholder queue.
|
||||
func TestVolumeTagMirror_WritesOnlyControlPlaneTagsObject(t *testing.T) {
|
||||
func TestVolumeTagMirror_WritesOnlyControlPlaneDocument(t *testing.T) {
|
||||
memoryStore := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", memoryStore)
|
||||
svc.natsConn = startTestNATS(t)
|
||||
volumeID := "vol-enc-tag-writer"
|
||||
seedEncryptedConfig(t, memoryStore, volumeID)
|
||||
require.NoError(t, svc.putVolumeState(context.Background(), volumeID, volumestate.Record{
|
||||
State: "in-use",
|
||||
AttachedInstance: "i-live0000000000",
|
||||
DeviceName: "/dev/nbd0",
|
||||
}))
|
||||
seedVolumeDocument(t, memoryStore, ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testVolAccountID, CapacityGiB: 10, State: "available",
|
||||
})
|
||||
require.NoError(t, svc.UpdateVolumeState(volumeID, "in-use", "i-live0000000000", "/dev/nbd0"))
|
||||
before := getStoredConfig(t, memoryStore, volumeID)
|
||||
store := &recordingObjectStore{ObjectStore: memoryStore}
|
||||
svc.store = store
|
||||
svc.metadata = ebsmetadata.NewStore(store, "test-bucket")
|
||||
|
||||
var configRequests atomic.Int32
|
||||
sub, err := svc.natsConn.Subscribe("ebs.config", func(msg *nats.Msg) {
|
||||
@@ -130,27 +136,30 @@ func TestVolumeTagMirror_WritesOnlyControlPlaneTagsObject(t *testing.T) {
|
||||
Tags: []*ec2.Tag{{Key: aws.String("owner"), Value: aws.String("control-plane")}},
|
||||
}, testVolAccountID))
|
||||
|
||||
assert.Equal(t, []string{volumeTagsKey(volumeID), volumeTagsKey(volumeID)}, store.putKeys,
|
||||
"tag mutations must write only tags.json")
|
||||
key := metadataKey(t, volumeID)
|
||||
assert.Equal(t, []string{key, key}, store.putKeys,
|
||||
"tag mutations must write only the control-plane document")
|
||||
assert.Equal(t, int32(0), configRequests.Load(),
|
||||
"tag mutations must never issue an ebs.config request")
|
||||
assert.Equal(t, before, getStoredConfig(t, memoryStore, volumeID),
|
||||
"tag mutations must leave the sealed config.json untouched")
|
||||
}
|
||||
|
||||
// TestVolumeTagMirror_ReturnsTagsObjectWriteFailure prevents tag persistence
|
||||
// TestVolumeTagMirror_ReturnsDocumentWriteFailure prevents tag persistence
|
||||
// errors from becoming false-successful CreateTags responses.
|
||||
func TestVolumeTagMirror_ReturnsTagsObjectWriteFailure(t *testing.T) {
|
||||
func TestVolumeTagMirror_ReturnsDocumentWriteFailure(t *testing.T) {
|
||||
memoryStore := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", memoryStore)
|
||||
volumeID := "vol-tag-write-error"
|
||||
seedVolume(t, svc, volumeID, "in-use", "i-live0000000000")
|
||||
svc.store = &recordingObjectStore{
|
||||
failing := &recordingObjectStore{
|
||||
ObjectStore: memoryStore,
|
||||
putError: map[string]error{
|
||||
volumeTagsKey(volumeID): errors.New("tags store unavailable"),
|
||||
metadataKey(t, volumeID): errors.New("tags store unavailable"),
|
||||
},
|
||||
}
|
||||
svc.store = failing
|
||||
svc.metadata = ebsmetadata.NewStore(failing, "test-bucket")
|
||||
|
||||
err := svc.ApplyRecordTags(&ec2.CreateTagsInput{
|
||||
Resources: []*string{aws.String(volumeID)},
|
||||
@@ -182,24 +191,24 @@ func TestRLC5_VolumeGCMarksLeakedVolumeNeverDeletes(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, marked, "ADR-0005 §3: the GC must surface exactly the leaked volume")
|
||||
|
||||
leaked, err := svc.GetVolumeConfig("vol-leaked0000000")
|
||||
leaked, err := svc.GetVolumeMetadata("vol-leaked0000000")
|
||||
require.NoErrorf(t, err, "ADR-0005 §3: the GC must NOT delete the leaked volume — it carries data; it may only mark + alarm")
|
||||
assert.NotEmpty(t, leaked.VolumeMetadata.Tags[orphanTagKey],
|
||||
assert.NotEmpty(t, leaked.Tags[orphanTagKey],
|
||||
"ADR-0005 §3: a leaked volume must be marked orphaned")
|
||||
assert.Equal(t, "preserved", leaked.VolumeMetadata.Tags["customer"],
|
||||
"orphan marking must preserve tags from the authoritative tags.json")
|
||||
assert.Equal(t, "preserved", leaked.Tags["customer"],
|
||||
"orphan marking must preserve the volume's existing tags")
|
||||
|
||||
// A volume attached to an instance NOT in the leaked set must be untouched —
|
||||
// node-local detection never false-marks another node's live-instance volume.
|
||||
live, err := svc.GetVolumeConfig("vol-live000000000")
|
||||
live, err := svc.GetVolumeMetadata("vol-live000000000")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, live.VolumeMetadata.Tags[orphanTagKey],
|
||||
assert.Empty(t, live.Tags[orphanTagKey],
|
||||
"a volume on a live instance must never be marked orphaned")
|
||||
|
||||
// Idempotent: a second sweep re-marks nothing and still deletes nothing.
|
||||
marked, err = reaper.Sweep(context.Background())
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, marked, "an already-marked orphan must not be re-marked")
|
||||
_, err = svc.GetVolumeConfig("vol-leaked0000000")
|
||||
_, err = svc.GetVolumeMetadata("vol-leaked0000000")
|
||||
require.NoError(t, err, "ADR-0005 §3: the volume must still exist after repeated sweeps")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
handlers_ec2_instance "github.com/mulgadc/spinifex/spinifex/handlers/ec2/instance"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
spxtypes "github.com/mulgadc/spinifex/spinifex/types"
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const providerTestAZ = "ap-southeast-2a"
|
||||
|
||||
// snapshotSourceProvider mirrors viperblockd's handleCreateVolume validation:
|
||||
// a clone resolves its base blocks against the source volume's prefix, so a
|
||||
// snapshot source without a source volume is rejected outright.
|
||||
type snapshotSourceProvider struct {
|
||||
*ebsprovider.MemoryProvider
|
||||
|
||||
lastRequest ebsprovider.CreateVolumeRequest
|
||||
}
|
||||
|
||||
func (p *snapshotSourceProvider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
p.lastRequest = req
|
||||
if req.SourceSnapshotID != "" && req.SourceSnapshotVolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: source_snapshot_volume_id is required with source_snapshot_id", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
// The snapshot exists only in the control plane's metadata for this test,
|
||||
// so bypass the in-memory provider's own snapshot bookkeeping.
|
||||
req.SourceSnapshotID = ""
|
||||
return p.MemoryProvider.CreateVolume(ctx, req)
|
||||
}
|
||||
|
||||
func newSnapshotSourceProvider() *snapshotSourceProvider {
|
||||
return &snapshotSourceProvider{MemoryProvider: ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})}
|
||||
}
|
||||
|
||||
// writeSnapshotMetadata publishes the control-plane record CreateVolume reads
|
||||
// to learn which volume a snapshot was taken from. The owner must be set:
|
||||
// CreateVolume fails closed on a snapshot whose owner it cannot match.
|
||||
func writeSnapshotMetadata(t *testing.T, store objectstore.ObjectStore, snapshotID, sourceVolumeID, ownerID string, sizeGiB int64) {
|
||||
t.Helper()
|
||||
data, err := json.Marshal(snapshotMetadata{VolumeID: sourceVolumeID, VolumeSize: sizeGiB, OwnerID: ownerID})
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(snapshotID + "/metadata.json"),
|
||||
Body: strings.NewReader(string(data)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// TestCreateVolume_Provider_FromSnapshotSendsSourceVolumeID locks that a
|
||||
// snapshot-backed CreateVolume names the snapshot's source volume. Without it
|
||||
// the provider rejects the request and create-from-snapshot cannot work at all.
|
||||
func TestCreateVolume_Provider_FromSnapshotSendsSourceVolumeID(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{AZ: providerTestAZ, Predastore: config.PredastoreConfig{Bucket: "test-bucket"}}
|
||||
svc := NewVolumeServiceImplWithStore(cfg, store, nil)
|
||||
provider := newSnapshotSourceProvider()
|
||||
svc.SetEBSProvider(provider)
|
||||
|
||||
writeSnapshotMetadata(t, store, "snap-src", "vol-origin", "acct-1", 8)
|
||||
|
||||
created, err := svc.CreateVolume(context.Background(), &ec2.CreateVolumeInput{
|
||||
Size: aws.Int64(8), AvailabilityZone: aws.String(providerTestAZ), SnapshotId: aws.String("snap-src"),
|
||||
}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, created)
|
||||
|
||||
assert.Equal(t, "snap-src", provider.lastRequest.SourceSnapshotID)
|
||||
assert.Equal(t, "vol-origin", provider.lastRequest.SourceSnapshotVolumeID,
|
||||
"the snapshot's source volume must travel with the snapshot ID")
|
||||
}
|
||||
|
||||
// TestCreateVolume_Provider_WithoutSnapshotSendsNoSource locks that an ordinary
|
||||
// blank volume does not gain a spurious source volume.
|
||||
func TestCreateVolume_Provider_WithoutSnapshotSendsNoSource(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{AZ: providerTestAZ, Predastore: config.PredastoreConfig{Bucket: "test-bucket"}}
|
||||
svc := NewVolumeServiceImplWithStore(cfg, store, nil)
|
||||
provider := newSnapshotSourceProvider()
|
||||
svc.SetEBSProvider(provider)
|
||||
|
||||
_, err := svc.CreateVolume(context.Background(), &ec2.CreateVolumeInput{
|
||||
Size: aws.Int64(8), AvailabilityZone: aws.String(providerTestAZ),
|
||||
}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, provider.lastRequest.SourceSnapshotID)
|
||||
assert.Empty(t, provider.lastRequest.SourceSnapshotVolumeID)
|
||||
}
|
||||
|
||||
// fixedAMILoader resolves one AMI to a snapshot on a known source volume.
|
||||
type fixedAMILoader struct {
|
||||
meta ebsmetadata.AMI
|
||||
sourceVolumeID string
|
||||
}
|
||||
|
||||
func (l *fixedAMILoader) GetAMIConfig(_ context.Context, _ string) (ebsmetadata.AMI, error) {
|
||||
return l.meta, nil
|
||||
}
|
||||
|
||||
func (l *fixedAMILoader) GetAMISourceVolumeID(_ context.Context, _ string) (string, error) {
|
||||
return l.sourceVolumeID, nil
|
||||
}
|
||||
|
||||
// TestDescribeVolumes_Provider_SeesInstanceRootVolume is the end-to-end guard
|
||||
// for the launch path: a root volume allocated through the provider carries no
|
||||
// legacy config.json, so it is visible only if the instance service records an
|
||||
// ebsmetadata document for it.
|
||||
func TestDescribeVolumes_Provider_SeesInstanceRootVolume(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
cfg := &config.Config{
|
||||
AZ: providerTestAZ,
|
||||
WalDir: t.TempDir(),
|
||||
Predastore: config.PredastoreConfig{Bucket: "test-bucket", Host: "127.0.0.1:1", Region: "us-east-1"},
|
||||
}
|
||||
provider := ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{})
|
||||
|
||||
// The AMI's snapshot must already exist in the provider, as it would after
|
||||
// the image was registered.
|
||||
const amiBytes = 4 * 1024 * 1024 * 1024
|
||||
_, err := provider.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), VolumeID: "vol-origin",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: amiBytes},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
_, err = provider.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{
|
||||
Versioned: ebsprovider.NewVersioned(), SnapshotID: "snap-src", VolumeID: "vol-origin",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
instanceSvc := handlers_ec2_instance.NewInstanceServiceImpl(cfg, nil, nil, store, nil, nil, nil)
|
||||
instanceSvc.SetEBSProvider(provider)
|
||||
instanceSvc.SetRunInstancesDeps(&fixedAMILoader{
|
||||
meta: ebsmetadata.AMI{ImageID: "ami-1", SnapshotID: "snap-src", VolumeSizeGiB: 4},
|
||||
sourceVolumeID: "vol-origin",
|
||||
}, nil, nil, nil)
|
||||
|
||||
instance := &vm.VM{AccountID: "acct-1"}
|
||||
volumeInfos, err := instanceSvc.GenerateVolumes(context.Background(), &ec2.RunInstancesInput{
|
||||
ImageId: aws.String("ami-1"),
|
||||
}, instance)
|
||||
require.NoError(t, err, "the launch must allocate its root volume through the provider")
|
||||
require.Len(t, volumeInfos, 1)
|
||||
rootVolumeID := volumeInfos[0].VolumeId
|
||||
|
||||
volumeSvc := NewVolumeServiceImplWithStore(cfg, store, nil)
|
||||
volumeSvc.SetEBSProvider(provider)
|
||||
|
||||
out, err := volumeSvc.DescribeVolumes(context.Background(), &ec2.DescribeVolumesInput{}, "acct-1")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, out.Volumes, 1, "the instance root volume must be visible to DescribeVolumes")
|
||||
|
||||
got := out.Volumes[0]
|
||||
assert.Equal(t, rootVolumeID, aws.StringValue(got.VolumeId))
|
||||
assert.Equal(t, "available", aws.StringValue(got.State))
|
||||
assert.Equal(t, providerTestAZ, aws.StringValue(got.AvailabilityZone))
|
||||
assert.Equal(t, spxtypes.VolumeTypeGP3, aws.StringValue(got.VolumeType))
|
||||
assert.Equal(t, int64(spxtypes.DefaultGP3IOPS), aws.Int64Value(got.Iops))
|
||||
assert.Equal(t, "snap-src", aws.StringValue(got.SnapshotId))
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -26,10 +27,10 @@ func TestVolumeRecordTagsMirror(t *testing.T) {
|
||||
},
|
||||
}, testVolAccountID))
|
||||
|
||||
cfg, err := svc.GetVolumeConfig("vol-tagmirror0001")
|
||||
meta, err := svc.GetVolumeMetadata("vol-tagmirror0001")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "yes", cfg.VolumeMetadata.Tags["keep"])
|
||||
assert.Equal(t, "v", cfg.VolumeMetadata.Tags["drop"])
|
||||
assert.Equal(t, "yes", meta.Tags["keep"])
|
||||
assert.Equal(t, "v", meta.Tags["drop"])
|
||||
|
||||
// Value-mismatched delete is a no-op; matched delete removes.
|
||||
require.NoError(t, svc.RemoveRecordTags(&ec2.DeleteTagsInput{
|
||||
@@ -40,10 +41,10 @@ func TestVolumeRecordTagsMirror(t *testing.T) {
|
||||
},
|
||||
}, testVolAccountID))
|
||||
|
||||
cfg, err = svc.GetVolumeConfig("vol-tagmirror0001")
|
||||
meta, err = svc.GetVolumeMetadata("vol-tagmirror0001")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "yes", cfg.VolumeMetadata.Tags["keep"])
|
||||
_, ok := cfg.VolumeMetadata.Tags["drop"]
|
||||
assert.Equal(t, "yes", meta.Tags["keep"])
|
||||
_, ok := meta.Tags["drop"]
|
||||
assert.False(t, ok)
|
||||
}
|
||||
|
||||
@@ -69,11 +70,15 @@ func TestApplyRecordTags_AttachedVolumePersistsForTagFilterDiscovery(t *testing.
|
||||
assert.Equal(t, volumeID, aws.StringValue(out.Volumes[0].VolumeId))
|
||||
}
|
||||
|
||||
// A tag must survive the live nbdkit VB rewriting config.json from its own
|
||||
// mount-time state: the tag lives in the control-plane document, which that
|
||||
// writer does not own.
|
||||
func TestApplyRecordTags_SurvivesStaleConfigRewrite(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
volumeID := "vol-staleconfigtag"
|
||||
seedVolume(t, svc, volumeID, "in-use", "i-live0000000000")
|
||||
seedVolume(t, svc, volumeID, "available", "")
|
||||
seedProviderConfig(t, store, volumeID)
|
||||
staleConfig := getStoredConfig(t, store, volumeID)
|
||||
|
||||
require.NoError(t, svc.ApplyRecordTags(&ec2.CreateTagsInput{
|
||||
@@ -89,9 +94,9 @@ func TestApplyRecordTags_SurvivesStaleConfigRewrite(t *testing.T) {
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
cfg, err := svc.GetVolumeConfig(volumeID)
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "retain", cfg.VolumeMetadata.Tags["sweep"])
|
||||
assert.Equal(t, "retain", meta.Tags["sweep"])
|
||||
|
||||
out, err := svc.DescribeVolumes(context.Background(), &ec2.DescribeVolumesInput{
|
||||
Filters: []*ec2.Filter{{
|
||||
@@ -104,55 +109,51 @@ func TestApplyRecordTags_SurvivesStaleConfigRewrite(t *testing.T) {
|
||||
assert.Equal(t, volumeID, aws.StringValue(out.Volumes[0].VolumeId))
|
||||
}
|
||||
|
||||
func TestApplyRecordTags_FirstWriteSeedsEmbeddedTags(t *testing.T) {
|
||||
// A tag write must merge into the tags already on the document rather than
|
||||
// replacing the set wholesale.
|
||||
func TestApplyRecordTags_MergesWithExistingTags(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
volumeID := "vol-seedembedded"
|
||||
seedVolumeWithEmbeddedTags(t, svc, volumeID, map[string]string{"created-with": "volume"})
|
||||
seedVolume(t, svc, volumeID, "available", "")
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testVolAccountID, CapacityGiB: 8, State: "available",
|
||||
Tags: map[string]string{"created-with": "volume"},
|
||||
})
|
||||
|
||||
require.NoError(t, svc.ApplyRecordTags(&ec2.CreateTagsInput{
|
||||
Resources: []*string{aws.String(volumeID)},
|
||||
Tags: []*ec2.Tag{{Key: aws.String("added-later"), Value: aws.String("yes")}},
|
||||
}, testVolAccountID))
|
||||
|
||||
cfg, err := svc.GetVolumeConfig(volumeID)
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, map[string]string{
|
||||
"created-with": "volume",
|
||||
"added-later": "yes",
|
||||
}, cfg.VolumeMetadata.Tags)
|
||||
}, meta.Tags)
|
||||
}
|
||||
|
||||
func TestRemoveRecordTags_EmptyTagsJSONOverridesEmbeddedTags(t *testing.T) {
|
||||
// Removing the last tag must leave an authoritative empty tag set rather than
|
||||
// any other source of tags resurrecting the deleted one.
|
||||
func TestRemoveRecordTags_LastTagLeavesEmptySet(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
volumeID := "vol-emptytagsjson"
|
||||
seedVolumeWithEmbeddedTags(t, svc, volumeID, map[string]string{"legacy": "tag"})
|
||||
seedVolume(t, svc, volumeID, "available", "")
|
||||
seedVolumeDocument(t, store, ebsmetadata.Volume{
|
||||
VolumeID: volumeID, TenantID: testVolAccountID, CapacityGiB: 8, State: "available",
|
||||
Tags: map[string]string{"legacy": "tag"},
|
||||
})
|
||||
|
||||
require.NoError(t, svc.RemoveRecordTags(&ec2.DeleteTagsInput{
|
||||
Resources: []*string{aws.String(volumeID)},
|
||||
Tags: []*ec2.Tag{{Key: aws.String("legacy"), Value: aws.String("tag")}},
|
||||
}, testVolAccountID))
|
||||
|
||||
tags, found, err := svc.getVolumeTags(context.Background(), volumeID)
|
||||
meta, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, found, "removing the final tag must leave an authoritative tags.json")
|
||||
assert.Empty(t, tags)
|
||||
|
||||
cfg, err := svc.GetVolumeConfig(volumeID)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, cfg.VolumeMetadata.Tags, "present empty tags.json must not fall back to embedded tags")
|
||||
}
|
||||
|
||||
// seedVolumeWithEmbeddedTags creates a legacy or create-time tagged volume with
|
||||
// no tags.json so tests exercise the migration fallback.
|
||||
func seedVolumeWithEmbeddedTags(t *testing.T, svc *VolumeServiceImpl, volumeID string, tags map[string]string) {
|
||||
t.Helper()
|
||||
seedVolume(t, svc, volumeID, "available", "")
|
||||
cfg, err := svc.GetVolumeConfig(volumeID)
|
||||
require.NoError(t, err)
|
||||
cfg.VolumeMetadata.Tags = tags
|
||||
require.NoError(t, svc.putVolumeConfig(context.Background(), volumeID, cfg))
|
||||
assert.Empty(t, meta.Tags, "the deleted tag must not come back from anywhere")
|
||||
}
|
||||
|
||||
func TestVolumeRecordTagsMirror_CrossTenantNoMutation(t *testing.T) {
|
||||
@@ -166,9 +167,9 @@ func TestVolumeRecordTagsMirror_CrossTenantNoMutation(t *testing.T) {
|
||||
Tags: []*ec2.Tag{{Key: aws.String("evil"), Value: aws.String("1")}},
|
||||
}, "999999999999"))
|
||||
|
||||
cfg, err := svc.GetVolumeConfig("vol-tenantguard01")
|
||||
meta, err := svc.GetVolumeMetadata("vol-tenantguard01")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, cfg.VolumeMetadata.Tags)
|
||||
assert.Empty(t, meta.Tags)
|
||||
}
|
||||
|
||||
func TestVolumeRecordTagsMirror_UnownedNoError(t *testing.T) {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -2,78 +2,33 @@ package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestGetVolumeConfig_CorruptJSON(t *testing.T) {
|
||||
// A document that does not decode must surface as an error rather than
|
||||
// resolving to an empty volume, which would read as a volume that exists with
|
||||
// no owner, no size and no attachment.
|
||||
func TestGetVolumeMetadata_CorruptJSON(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
|
||||
const volumeID = "vol-bad-json"
|
||||
// Body is not a valid VBState (BlockSize stays 0 → falls through) and is
|
||||
// not a valid volumeConfigWrapper either (unmarshal fails) — must surface
|
||||
// "failed to unmarshal config" from getVolumeConfigAndEncryption.
|
||||
_, err := store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader("not valid json {{{"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
putRawVolumeDocument(t, store, volumeID, "not valid json {{{")
|
||||
|
||||
_, err = svc.GetVolumeConfig(volumeID)
|
||||
_, err := svc.GetVolumeMetadata(volumeID)
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "failed to unmarshal config")
|
||||
}
|
||||
|
||||
func TestMergeVolumeConfig_RefusesEncryptedVBState(t *testing.T) {
|
||||
store := objectstore.NewMemoryObjectStore()
|
||||
svc := newTestVolumeServiceWithStore("ap-southeast-2a", store)
|
||||
|
||||
const volumeID = "vol-encrypted"
|
||||
const configKey = volumeID + "/config.json"
|
||||
|
||||
// Seed an encrypted-at-rest VBState. mergeVolumeConfig must refuse to
|
||||
// re-marshal it because spinifex does not currently hold the master key
|
||||
// and a tag-less rewrite would brick the volume.
|
||||
state := viperblock.VBState{
|
||||
VolumeName: volumeID,
|
||||
VolumeSize: 1024 * 1024 * 1024,
|
||||
BlockSize: 4096,
|
||||
EncryptionEnabled: true,
|
||||
VolumeConfig: viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{
|
||||
VolumeID: volumeID,
|
||||
SizeGiB: 1,
|
||||
},
|
||||
},
|
||||
}
|
||||
data, err := json.Marshal(state)
|
||||
require.NoError(t, err)
|
||||
_, err = store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(configKey),
|
||||
Body: strings.NewReader(string(data)),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.mergeVolumeConfig(context.Background(), configKey, &viperblock.VolumeConfig{
|
||||
VolumeMetadata: viperblock.VolumeMetadata{VolumeID: volumeID, SizeGiB: 2},
|
||||
})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "refusing to merge encrypted VBState")
|
||||
assert.ErrorIs(t, err, ebsmetadata.ErrCorruptDocument)
|
||||
}
|
||||
|
||||
// newTestVolumeServiceWithEncryptionKey wires CreateVolume to a configured
|
||||
@@ -91,13 +46,15 @@ func newTestVolumeServiceWithEncryptionKey(az, keyFile string) *VolumeServiceImp
|
||||
Viperblock: config.ViperblockConfig{EncryptionKeyFile: keyFile},
|
||||
WalDir: "/tmp/test-wal",
|
||||
}
|
||||
return NewVolumeServiceImplWithStore(cfg, objectstore.NewMemoryObjectStore(), nil)
|
||||
svc := NewVolumeServiceImplWithStore(cfg, objectstore.NewMemoryObjectStore(), nil)
|
||||
svc.SetEBSProvider(ebsprovider.NewMemoryProvider(ebsprovider.Capabilities{}))
|
||||
return svc
|
||||
}
|
||||
|
||||
func TestCreateVolume_EncryptionKeyLoadError(t *testing.T) {
|
||||
// Point EncryptionKeyFile at a non-existent path so LoadViperblockMasterKey
|
||||
// fails — CreateVolume must abort with ServerInternal, before any backend
|
||||
// init / SaveState call.
|
||||
// fails — CreateVolume must abort with ServerInternal and roll the
|
||||
// provider's allocation back rather than leave an orphaned volume.
|
||||
missing := filepath.Join(t.TempDir(), "absent.key")
|
||||
svc := newTestVolumeServiceWithEncryptionKey("ap-southeast-2a", missing)
|
||||
|
||||
@@ -107,4 +64,8 @@ func TestCreateVolume_EncryptionKeyLoadError(t *testing.T) {
|
||||
}, "")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
|
||||
out, err := svc.DescribeVolumes(context.Background(), &ec2.DescribeVolumesInput{}, "")
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, out.Volumes, "a failed create must not leave a volume behind")
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,86 @@
|
||||
package handlers_ec2_volume
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsmetadata"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/nats-io/nats-server/v2/server"
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// startTestNATS spins up an in-process NATS server and returns a connected client.
|
||||
func startTestNATS(t *testing.T) *nats.Conn {
|
||||
t.Helper()
|
||||
opts := &server.Options{
|
||||
Host: "127.0.0.1",
|
||||
Port: -1,
|
||||
JetStream: true,
|
||||
StoreDir: t.TempDir(),
|
||||
NoLog: true,
|
||||
NoSigs: true,
|
||||
}
|
||||
ns, err := server.NewServer(opts)
|
||||
require.NoError(t, err)
|
||||
go ns.Start()
|
||||
require.True(t, ns.ReadyForConnections(5*time.Second))
|
||||
t.Cleanup(func() { ns.Shutdown() })
|
||||
|
||||
nc, err := nats.Connect(ns.ClientURL())
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { nc.Close() })
|
||||
return nc
|
||||
}
|
||||
|
||||
// seedVolumeDocument writes the control-plane document that makes a volume
|
||||
// exist as far as the control plane is concerned.
|
||||
func seedVolumeDocument(t *testing.T, store objectstore.ObjectStore, volume ebsmetadata.Volume) {
|
||||
t.Helper()
|
||||
require.NoError(t, ebsmetadata.NewStore(store, "test-bucket").PutVolume(context.Background(), volume))
|
||||
}
|
||||
|
||||
// seedProviderConfig writes provider-owned state at the volume's config.json.
|
||||
// Its contents are deliberately opaque here: the control plane must neither
|
||||
// read nor rewrite them, so the tests only care that the bytes survive.
|
||||
func seedProviderConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string) {
|
||||
t.Helper()
|
||||
putProviderConfig(t, store, volumeID, `{"VolumeName":"`+volumeID+`","VolumeSize":10737418240,"BlockSize":4096,"SeqNum":7,"State":"available"}`)
|
||||
}
|
||||
|
||||
// seedEncryptedConfig writes a sealed provider config.json: an opaque
|
||||
// ciphertext blob a rewrite would corrupt rather than merely overwrite.
|
||||
func seedEncryptedConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string) {
|
||||
t.Helper()
|
||||
putProviderConfig(t, store, volumeID, `{"EncryptionEnabled":true,"StateSeqNum":7,"Sealed":"3q2+7wAAAAAAAAAAAAAAAA=="}`)
|
||||
}
|
||||
|
||||
func putProviderConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID, body string) {
|
||||
t.Helper()
|
||||
_, err := store.PutObject(context.Background(), &s3.PutObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
Body: strings.NewReader(body),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// getStoredConfig reads the raw config.json bytes from the memory store.
|
||||
func getStoredConfig(t *testing.T, store *objectstore.MemoryObjectStore, volumeID string) []byte {
|
||||
t.Helper()
|
||||
res, err := store.GetObject(context.Background(), &s3.GetObjectInput{
|
||||
Bucket: aws.String("test-bucket"),
|
||||
Key: aws.String(volumeID + "/config.json"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
defer res.Body.Close()
|
||||
data, err := io.ReadAll(res.Body)
|
||||
require.NoError(t, err)
|
||||
return data
|
||||
}
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/vm"
|
||||
"github.com/mulgadc/viperblock/viperblock"
|
||||
)
|
||||
|
||||
// orphanTagKey marks a volume the GC has surfaced as orphaned. It is a
|
||||
@@ -67,26 +66,27 @@ func (r *VolumeLeakReaper) Sweep(ctx context.Context) (int, error) {
|
||||
default:
|
||||
}
|
||||
|
||||
cfg, err := r.svc.getVolumeConfig(ctx, id)
|
||||
meta, err := r.svc.metadata.GetVolume(ctx, id)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
md := &cfg.VolumeMetadata
|
||||
if md.AttachedInstance == "" || !leaked[md.AttachedInstance] {
|
||||
attachedInstance, tags, sizeGiB := meta.AttachedInstance, meta.Tags, meta.CapacityGiB
|
||||
|
||||
if attachedInstance == "" || !leaked[attachedInstance] {
|
||||
continue
|
||||
}
|
||||
if md.Tags[orphanTagKey] != "" {
|
||||
if tags[orphanTagKey] != "" {
|
||||
continue // already surfaced; idempotent
|
||||
}
|
||||
|
||||
if err := r.svc.markVolumeOrphaned(ctx, id, cfg); err != nil {
|
||||
if err := r.svc.markVolumeOrphaned(ctx, id, tags, attachedInstance); err != nil {
|
||||
slog.Error("volume-leak: failed to mark orphaned volume", "volumeId", id, "err", err)
|
||||
continue
|
||||
}
|
||||
// ALARM — surface the retained orphan to the operator. NEVER delete:
|
||||
// reclamation is the explicit --purge / operator path (ADR-0005 §3).
|
||||
slog.Warn("DATA-SAFETY ALARM: orphaned volume retained, not deleted",
|
||||
"volumeId", id, "attachedInstance", md.AttachedInstance, "sizeGiB", md.SizeGiB)
|
||||
"volumeId", id, "attachedInstance", attachedInstance, "sizeGiB", sizeGiB)
|
||||
|
||||
// Reconcile the attachment record only — never the volume data. The
|
||||
// owning instance is definitively terminated (this node's leaked
|
||||
@@ -100,13 +100,20 @@ func (r *VolumeLeakReaper) Sweep(ctx context.Context) (int, error) {
|
||||
return marked, nil
|
||||
}
|
||||
|
||||
// markVolumeOrphaned tags the volume as orphaned in the control-plane-owned
|
||||
// tags.json. The volume's data and state remain untouched.
|
||||
func (s *VolumeServiceImpl) markVolumeOrphaned(ctx context.Context, volumeID string, cfg *viperblock.VolumeConfig) error {
|
||||
if cfg.VolumeMetadata.Tags == nil {
|
||||
cfg.VolumeMetadata.Tags = make(map[string]string)
|
||||
// markVolumeOrphaned tags the volume as orphaned. tags is the volume's
|
||||
// current, already-authoritative tag set; the volume's data and state remain
|
||||
// untouched.
|
||||
func (s *VolumeServiceImpl) markVolumeOrphaned(ctx context.Context, volumeID string, tags map[string]string, attachedInstance string) error {
|
||||
if tags == nil {
|
||||
tags = make(map[string]string)
|
||||
}
|
||||
cfg.VolumeMetadata.Tags[orphanTagKey] = time.Now().UTC().Format(time.RFC3339)
|
||||
cfg.VolumeMetadata.Tags[orphanInstanceTagKey] = cfg.VolumeMetadata.AttachedInstance
|
||||
return s.putVolumeTags(ctx, volumeID, cfg.VolumeMetadata.Tags)
|
||||
tags[orphanTagKey] = time.Now().UTC().Format(time.RFC3339)
|
||||
tags[orphanInstanceTagKey] = attachedInstance
|
||||
|
||||
meta, err := s.metadata.GetVolume(ctx, volumeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
meta.Tags = tags
|
||||
return s.metadata.PutVolume(ctx, meta)
|
||||
}
|
||||
|
||||
@@ -28,12 +28,12 @@ func TestVolumeLeakReaper_ReconcilesAttachmentNeverDeletesData(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, marked)
|
||||
|
||||
cfg, err := svc.GetVolumeConfig("vol-leaked-reconcile")
|
||||
meta, err := svc.GetVolumeMetadata("vol-leaked-reconcile")
|
||||
require.NoError(t, err, "the reaper must never delete the volume, only mark + reconcile it")
|
||||
assert.NotEmpty(t, cfg.VolumeMetadata.Tags[orphanTagKey], "the volume must still be marked orphaned")
|
||||
assert.Equal(t, "available", cfg.VolumeMetadata.State,
|
||||
assert.NotEmpty(t, meta.Tags[orphanTagKey], "the volume must still be marked orphaned")
|
||||
assert.Equal(t, "available", meta.State,
|
||||
"the reaper must reconcile the attachment record to available so DeleteVolume can proceed")
|
||||
assert.Empty(t, cfg.VolumeMetadata.AttachedInstance,
|
||||
assert.Empty(t, meta.AttachedInstance,
|
||||
"the reaper must clear the stale attachment left by the terminated instance")
|
||||
|
||||
// The reconcile only clears control-plane attachment state — this is
|
||||
@@ -41,7 +41,7 @@ func TestVolumeLeakReaper_ReconcilesAttachmentNeverDeletesData(t *testing.T) {
|
||||
// AttachedInstance=="" and State in {"available",""}. Confirm the reaper
|
||||
// leaves the volume passing that guard, without exercising DeleteVolume's
|
||||
// full path here (which needs a snapshotKV this unit fixture omits).
|
||||
assert.True(t, cfg.VolumeMetadata.AttachedInstance == "" && (cfg.VolumeMetadata.State == "available" || cfg.VolumeMetadata.State == ""),
|
||||
assert.True(t, meta.AttachedInstance == "" && (meta.State == "available" || meta.State == ""),
|
||||
"the reconciled volume must pass DeleteVolume's in-use guard so an operator can delete it")
|
||||
}
|
||||
|
||||
@@ -62,8 +62,8 @@ func TestVolumeLeakReaper_LiveOwnerAttachmentUntouched(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, marked)
|
||||
|
||||
cfg, err := svc.GetVolumeConfig("vol-live-reconcile")
|
||||
meta, err := svc.GetVolumeMetadata("vol-live-reconcile")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "in-use", cfg.VolumeMetadata.State, "a live-owner volume's attachment must never be reconciled")
|
||||
assert.Equal(t, "i-running00000000", cfg.VolumeMetadata.AttachedInstance)
|
||||
assert.Equal(t, "in-use", meta.State, "a live-owner volume's attachment must never be reconciled")
|
||||
assert.Equal(t, "i-running00000000", meta.AttachedInstance)
|
||||
}
|
||||
|
||||
@@ -23,6 +23,12 @@ const (
|
||||
KVBucketENIsVersion = 1
|
||||
)
|
||||
|
||||
// AutoENIDescriptionPrefix marks an ENI the launch path created for an
|
||||
// instance, as opposed to one a caller created through the API. It is the only
|
||||
// thing distinguishing the two before the attachment lands, which is what lets
|
||||
// a sweep tell an abandoned launch from a deliberate standalone ENI.
|
||||
const AutoENIDescriptionPrefix = "Primary network interface for "
|
||||
|
||||
// ENIRecord represents a stored Elastic Network Interface.
|
||||
type ENIRecord struct {
|
||||
NetworkInterfaceId string `json:"network_interface_id"`
|
||||
@@ -760,6 +766,70 @@ func (s *VPCServiceImpl) listInstanceENIs(ctx context.Context, accountID, instan
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// AccountENI pairs a record with the account its key is scoped by, which a
|
||||
// cluster-wide sweep needs because it has no single account to scope to.
|
||||
type AccountENI struct {
|
||||
AccountID string
|
||||
Record ENIRecord
|
||||
}
|
||||
|
||||
// ListAbandonedInstanceENIs returns auto-created instance ENIs that never
|
||||
// reached an attachment and are older than minAge.
|
||||
//
|
||||
// A launch attaches within seconds of creating the ENI, and stopping an
|
||||
// instance leaves the attachment in place, so an unattached one this old is
|
||||
// the residue of a launch that died in between. Such a record keeps its
|
||||
// security group undeletable forever, since the SG dependency check counts it.
|
||||
func (s *VPCServiceImpl) ListAbandonedInstanceENIs(ctx context.Context, minAge time.Duration) ([]AccountENI, error) {
|
||||
keys, err := s.eniKV.Keys(ctx)
|
||||
if err != nil {
|
||||
if errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
var out []AccountENI
|
||||
for _, key := range keys {
|
||||
if key == utils.VersionKey {
|
||||
continue
|
||||
}
|
||||
accountID, _, found := strings.Cut(key, ".")
|
||||
if !found || accountID == "" {
|
||||
continue
|
||||
}
|
||||
entry, err := s.eniKV.Get(ctx, key)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var record ENIRecord
|
||||
if err := json.Unmarshal(entry.Value(), &record); err != nil {
|
||||
continue
|
||||
}
|
||||
if !eniIsAbandonedLaunch(&record, minAge) {
|
||||
continue
|
||||
}
|
||||
out = append(out, AccountENI{AccountID: accountID, Record: record})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// eniIsAbandonedLaunch reports whether a record is an instance ENI left behind
|
||||
// by a launch that never attached it. A zero CreatedAt is not reapable: age is
|
||||
// the only guard against deleting an ENI a launch is still using.
|
||||
func eniIsAbandonedLaunch(record *ENIRecord, minAge time.Duration) bool {
|
||||
if !strings.HasPrefix(record.Description, AutoENIDescriptionPrefix) {
|
||||
return false
|
||||
}
|
||||
if record.InstanceId != "" || record.AttachmentId != "" || record.Status == "in-use" {
|
||||
return false
|
||||
}
|
||||
if record.CreatedAt.IsZero() {
|
||||
return false
|
||||
}
|
||||
return time.Since(record.CreatedAt) >= minAge
|
||||
}
|
||||
|
||||
// FindENIByAttachment scans the ENI bucket for the record with the given
|
||||
// AttachmentId. Used by DetachNetworkInterface which identifies by attachment ID.
|
||||
func (s *VPCServiceImpl) FindENIByAttachment(accountID, attachmentId string) (ENIRecord, error) {
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
package handlers_ec2_vpc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/awserrors"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// createAutoENI creates an ENI the way the launch path does, then backdates it
|
||||
// so the age guard treats it as settled rather than a launch still in flight.
|
||||
func createAutoENI(t *testing.T, svc *VPCServiceImpl, subnetID, instanceID string, sgIDs []string, age time.Duration) string {
|
||||
t.Helper()
|
||||
groups := make([]*string, 0, len(sgIDs))
|
||||
for _, id := range sgIDs {
|
||||
groups = append(groups, aws.String(id))
|
||||
}
|
||||
out, err := svc.CreateNetworkInterface(context.Background(), &ec2.CreateNetworkInterfaceInput{
|
||||
SubnetId: aws.String(subnetID),
|
||||
Description: aws.String(AutoENIDescriptionPrefix + instanceID),
|
||||
Groups: groups,
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
eniID := *out.NetworkInterface.NetworkInterfaceId
|
||||
require.NoError(t, svc.UpdateENI(testAccountID, eniID, func(rec *ENIRecord) {
|
||||
rec.CreatedAt = time.Now().Add(-age)
|
||||
}))
|
||||
return eniID
|
||||
}
|
||||
|
||||
func TestListAbandonedInstanceENIs(t *testing.T) {
|
||||
svc := setupTestVPCService(t)
|
||||
vpcID := createTestVPC(t, svc, "10.0.0.0/16")
|
||||
subnetID := createTestSubnet(t, svc, vpcID, "10.0.1.0/24")
|
||||
|
||||
abandoned := createAutoENI(t, svc, subnetID, "i-abandoned", nil, time.Hour)
|
||||
createAutoENI(t, svc, subnetID, "i-inflight", nil, time.Minute)
|
||||
|
||||
// A standalone ENI is indistinguishable by age or attachment; only the
|
||||
// description separates one a caller asked for from launch residue.
|
||||
standalone := createTestENI(t, svc, subnetID)
|
||||
require.NoError(t, svc.UpdateENI(testAccountID, standalone, func(rec *ENIRecord) {
|
||||
rec.CreatedAt = time.Now().Add(-time.Hour)
|
||||
}))
|
||||
|
||||
attached := createAutoENI(t, svc, subnetID, "i-running", nil, time.Hour)
|
||||
_, err := svc.AttachENI(testAccountID, attached, "i-running", 0)
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := svc.ListAbandonedInstanceENIs(context.Background(), 15*time.Minute)
|
||||
require.NoError(t, err)
|
||||
|
||||
ids := make([]string, 0, len(got))
|
||||
for _, o := range got {
|
||||
assert.Equal(t, testAccountID, o.AccountID)
|
||||
ids = append(ids, o.Record.NetworkInterfaceId)
|
||||
}
|
||||
assert.Equal(t, []string{abandoned}, ids,
|
||||
"only the aged, unattached, launch-created ENI is residue; the others are live or deliberate")
|
||||
}
|
||||
|
||||
// TestAbandonedENIBlocksSecurityGroupDelete pins the consequence that makes this
|
||||
// worth sweeping: the SG dependency check counts an abandoned ENI, so the group
|
||||
// stays undeletable for as long as the record survives.
|
||||
func TestAbandonedENIBlocksSecurityGroupDelete(t *testing.T) {
|
||||
svc := setupTestVPCService(t)
|
||||
vpcID := createTestVPC(t, svc, "10.0.0.0/16")
|
||||
subnetID := createTestSubnet(t, svc, vpcID, "10.0.1.0/24")
|
||||
sgID := createTestSG(t, svc, vpcID, "runner-sg")
|
||||
|
||||
eniID := createAutoENI(t, svc, subnetID, "i-abandoned", []string{sgID}, time.Hour)
|
||||
|
||||
_, err := svc.DeleteSecurityGroup(context.Background(), &ec2.DeleteSecurityGroupInput{
|
||||
GroupId: aws.String(sgID),
|
||||
}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorDependencyViolation, err.Error())
|
||||
|
||||
orphans, err := svc.ListAbandonedInstanceENIs(context.Background(), 15*time.Minute)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, orphans, 1)
|
||||
require.Equal(t, eniID, orphans[0].Record.NetworkInterfaceId)
|
||||
|
||||
_, err = svc.DeleteNetworkInterface(context.Background(), &ec2.DeleteNetworkInterfaceInput{
|
||||
NetworkInterfaceId: aws.String(eniID),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.DeleteSecurityGroup(context.Background(), &ec2.DeleteSecurityGroupInput{
|
||||
GroupId: aws.String(sgID),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err, "with the residue gone the group must delete")
|
||||
}
|
||||
@@ -21,13 +21,24 @@ Single source of truth for the current schema version of every config file Spini
|
||||
|---|---|---|
|
||||
| `spinifex.toml` | `3` → `4`: rename `[nodes.*.predastore]` `node_id` → `host_id` | `spinifex/migrate/005_spinifex_predastore_host_id.go` |
|
||||
|
||||
## Registered object-store migrations
|
||||
|
||||
These migrate data in Predastore rather than a config file. Predastore has no conditional write, so their version is stamped in a shared JetStream KV bucket instead of in an object; see `RunObject`.
|
||||
|
||||
| Target | Migration | File | Version bucket |
|
||||
|---|---|---|---|
|
||||
| `ebsmetadata` | `0` → `1`: backfill `spinifex/ebsmetadata/v1` documents from legacy viperblock `config.json` | `spinifex/migrate/ebsmetadatabackfill/006_ebsmetadata_backfill.go` | `spinifex-ebsmetadata-migrate` |
|
||||
|
||||
Unlike config migrations, these do not run from `spx admin upgrade`. The ebsmetadata backfill runs from `Daemon.configureEBSProvider`, and only when `[ebs] provider = "viperblockd"` is selected: backfilling while the embedded engine is still authoritative would produce documents that immediately go stale.
|
||||
|
||||
## Where migrations live
|
||||
|
||||
Register a `ConfigMigration` against `DefaultRegistry` in a new numbered file under `spinifex/migrate/`, and bump the version in both the template and the table above.
|
||||
|
||||
## Framework
|
||||
|
||||
The framework (`migrate.go`, `version_readers.go`) covers both kinds of target:
|
||||
The framework (`migrate.go`, `version_readers.go`) covers three kinds of target:
|
||||
|
||||
- `RunKV` is called from service-startup paths to stamp NATS KV bucket versions.
|
||||
- `RunConfig` / `RunAllConfig` / `PendingConfig` back `spx admin upgrade`, which `scripts/setup.sh` runs with `--yes` while the services are stopped.
|
||||
- `RunObject` migrates object-store data, stamping its version in a caller-supplied JetStream KV bucket. The stamp is a read-then-write, not a compare-and-swap, so two nodes can race to run the same step — every `ObjectMigration.Run` must be safe to re-execute.
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/kvutil"
|
||||
"github.com/mulgadc/spinifex/spinifex/objectstore"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
@@ -38,6 +39,25 @@ type ConfigMigration struct {
|
||||
Run func(ctx ConfigContext) error
|
||||
}
|
||||
|
||||
// ObjectMigration represents a versioned transformation of object-store data
|
||||
// (Predastore/S3 objects), such as backfilling a new document shape from an
|
||||
// older one written by a different subsystem.
|
||||
type ObjectMigration struct {
|
||||
FromVersion int
|
||||
ToVersion int
|
||||
Description string
|
||||
Run func(ctx context.Context, octx ObjectContext) error
|
||||
}
|
||||
|
||||
// ObjectContext provides object-store migration functions access to the
|
||||
// bucket being migrated. Objects is the same object-store abstraction
|
||||
// handlers use, so a migration reads/writes exactly as a handler would.
|
||||
type ObjectContext struct {
|
||||
Objects objectstore.ObjectStore
|
||||
Bucket string
|
||||
Logger *slog.Logger
|
||||
}
|
||||
|
||||
// ConfigContext provides config migration functions access to the filesystem.
|
||||
type ConfigContext struct {
|
||||
ConfigDir string // path to /etc/spinifex or equivalent
|
||||
@@ -64,6 +84,7 @@ type Registry struct {
|
||||
kvMigrations map[string][]KVMigration
|
||||
configMigrations map[string][]ConfigMigration
|
||||
configTargets map[string]configTarget
|
||||
objectMigrations map[string][]ObjectMigration
|
||||
}
|
||||
|
||||
// DefaultRegistry is the global migration registry. Migrations self-register
|
||||
@@ -76,6 +97,7 @@ func NewRegistry() *Registry {
|
||||
kvMigrations: make(map[string][]KVMigration),
|
||||
configMigrations: make(map[string][]ConfigMigration),
|
||||
configTargets: make(map[string]configTarget),
|
||||
objectMigrations: make(map[string][]ObjectMigration),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -101,6 +123,82 @@ func (r *Registry) RegisterConfig(target string, m ConfigMigration) {
|
||||
})
|
||||
}
|
||||
|
||||
// RegisterObject adds an object-store migration. Migrations are kept sorted by FromVersion.
|
||||
func (r *Registry) RegisterObject(target string, m ObjectMigration) {
|
||||
r.objectMigrations[target] = append(r.objectMigrations[target], m)
|
||||
sort.Slice(r.objectMigrations[target], func(i, j int) bool {
|
||||
return r.objectMigrations[target][i].FromVersion < r.objectMigrations[target][j].FromVersion
|
||||
})
|
||||
}
|
||||
|
||||
// RunObject applies pending object-store migrations for target up to
|
||||
// targetVersion. Predastore has no conditional write, so progress is stamped
|
||||
// in JetStream KV instead, exactly as RunKV stamps a KV bucket's own version.
|
||||
//
|
||||
// Callers must pass a KV bucket shared by every node. The stamp is a
|
||||
// read-then-write rather than a compare-and-swap, so two nodes can still race
|
||||
// to run a step; each Run must be safe to re-execute for that to be harmless.
|
||||
func (r *Registry) RunObject(ctx context.Context, target string, objects objectstore.ObjectStore, bucket string, versionKV jetstream.KeyValue, targetVersion int) error {
|
||||
current, err := kvutil.ReadVersion(ctx, versionKV)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read version for %s: %w", target, err)
|
||||
}
|
||||
|
||||
if current >= targetVersion {
|
||||
return nil
|
||||
}
|
||||
|
||||
all := r.objectMigrations[target]
|
||||
|
||||
// Fresh target, no migrations: stamp directly (common first-init path).
|
||||
if current == 0 && len(all) == 0 {
|
||||
return kvutil.WriteVersion(ctx, versionKV, targetVersion)
|
||||
}
|
||||
|
||||
// Fresh target with migrations: no v0 schema by convention; start at chain bottom.
|
||||
if current == 0 {
|
||||
current = all[0].FromVersion
|
||||
}
|
||||
|
||||
// Require a complete chain from current to target.
|
||||
var pending []ObjectMigration
|
||||
for _, m := range all {
|
||||
if m.FromVersion >= current && m.ToVersion <= targetVersion {
|
||||
pending = append(pending, m)
|
||||
}
|
||||
}
|
||||
|
||||
if len(pending) == 0 {
|
||||
return fmt.Errorf("no migrations registered for %s from version %d to %d", target, current, targetVersion)
|
||||
}
|
||||
|
||||
// Validate contiguous chain.
|
||||
expected := current
|
||||
for _, m := range pending {
|
||||
if m.FromVersion != expected {
|
||||
return fmt.Errorf("migration chain gap for %s: expected from %d, got from %d", target, expected, m.FromVersion)
|
||||
}
|
||||
expected = m.ToVersion
|
||||
}
|
||||
if expected != targetVersion {
|
||||
return fmt.Errorf("migration chain for %s ends at version %d, target is %d", target, expected, targetVersion)
|
||||
}
|
||||
|
||||
logger := slog.Default()
|
||||
for _, m := range pending {
|
||||
logger.Info("Running object migration", "target", target, "from", m.FromVersion, "to", m.ToVersion, "description", m.Description)
|
||||
octx := ObjectContext{Objects: objects, Bucket: bucket, Logger: logger}
|
||||
if err := m.Run(ctx, octx); err != nil {
|
||||
return fmt.Errorf("object migration %s %d→%d failed: %w", target, m.FromVersion, m.ToVersion, err)
|
||||
}
|
||||
if err := kvutil.WriteVersion(ctx, versionKV, m.ToVersion); err != nil {
|
||||
return fmt.Errorf("stamp version %d on %s: %w", m.ToVersion, target, err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// RunKVWithJetStream is RunKV with a JetStream handle attached to each
|
||||
// migration's KVContext, enabling cross-bucket reads (e.g. owner-attribution
|
||||
// during a backfill). Prefer plain RunKV when the migration is self-contained.
|
||||
|
||||
@@ -44,8 +44,26 @@ type NBDKitConfig struct {
|
||||
// plugin's gc_enabled param, gating chunk garbage collection on the VB
|
||||
// this nbdkit process constructs. Default false, matching ShardWAL.
|
||||
GCEnabled bool `json:"gc_enabled"`
|
||||
|
||||
// ReadOnly serves the export read-only via nbdkit -r, which both sets the
|
||||
// NBD read-only transmission flag and passes readonly=true to the plugin's
|
||||
// Open. The plugin refuses writes on that flag.
|
||||
ReadOnly bool `json:"read_only"`
|
||||
}
|
||||
|
||||
// Block size constraints advertised to clients through nbdkit's
|
||||
// blocksize-policy filter. The plugin itself has no way to advertise them:
|
||||
// the nbdkit Go binding has no block-size callback.
|
||||
//
|
||||
// The minimum is 1 because viperblock's WriteAt handles unaligned requests
|
||||
// with a read-modify-write; the preferred size is its 4KB block, which is the
|
||||
// size that avoids that cycle. The maximum matches what qemu-nbd advertises.
|
||||
const (
|
||||
blockSizeMinimum = 1
|
||||
blockSizePreferred = 4096
|
||||
blockSizeMaximum = 32 << 20
|
||||
)
|
||||
|
||||
// buildArgs constructs the nbdkit command-line arguments from the config.
|
||||
func (cfg *NBDKitConfig) buildArgs() ([]string, error) {
|
||||
args := []string{
|
||||
@@ -53,6 +71,12 @@ func (cfg *NBDKitConfig) buildArgs() ([]string, error) {
|
||||
"--pidfile", cfg.PidFile,
|
||||
}
|
||||
|
||||
if cfg.ReadOnly {
|
||||
args = append(args, "-r")
|
||||
}
|
||||
|
||||
args = append(args, "--filter=blocksize-policy")
|
||||
|
||||
// Add transport-specific arguments
|
||||
if cfg.UseTCP {
|
||||
// TCP transport - for remote/DPU scenarios
|
||||
@@ -75,6 +99,9 @@ func (cfg *NBDKitConfig) buildArgs() ([]string, error) {
|
||||
// they are passed via cmd.Env in Execute so they never appear in argv,
|
||||
// which is world-readable via /proc/<pid>/cmdline.
|
||||
pluginArgs := []string{
|
||||
fmt.Sprintf("blocksize-minimum=%d", blockSizeMinimum),
|
||||
fmt.Sprintf("blocksize-preferred=%d", blockSizePreferred),
|
||||
fmt.Sprintf("blocksize-maximum=%d", blockSizeMaximum),
|
||||
fmt.Sprintf("size=%d", cfg.Size),
|
||||
fmt.Sprintf("volume=%s", cfg.Volume),
|
||||
fmt.Sprintf("bucket=%s", cfg.Bucket),
|
||||
|
||||
@@ -33,8 +33,12 @@ func TestBuildArgs_TCPTransport(t *testing.T) {
|
||||
expected := []string{
|
||||
"-f",
|
||||
"--pidfile", "/tmp/nbd.pid",
|
||||
"--filter=blocksize-policy",
|
||||
"-p", "10809",
|
||||
"/usr/lib/nbdkit/plugins/vb.so",
|
||||
"blocksize-minimum=1",
|
||||
"blocksize-preferred=4096",
|
||||
"blocksize-maximum=33554432",
|
||||
"size=1073741824",
|
||||
"volume=vol-abc123",
|
||||
"bucket=my-bucket",
|
||||
@@ -76,8 +80,12 @@ func TestBuildArgs_UnixSocketTransport(t *testing.T) {
|
||||
expected := []string{
|
||||
"-f",
|
||||
"--pidfile", "/tmp/nbd.pid",
|
||||
"--filter=blocksize-policy",
|
||||
"--unix", "/tmp/nbd.sock",
|
||||
"/usr/lib/nbdkit/plugins/vb.so",
|
||||
"blocksize-minimum=1",
|
||||
"blocksize-preferred=4096",
|
||||
"blocksize-maximum=33554432",
|
||||
"size=536870912",
|
||||
"volume=vol-def456",
|
||||
"bucket=bucket-2",
|
||||
@@ -378,3 +386,44 @@ func assertArgs(t *testing.T, expected, got []string) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestBuildArgs_ReadOnly checks -r reaches nbdkit before the plugin path, so
|
||||
// it is parsed as an nbdkit flag rather than a plugin parameter.
|
||||
func TestBuildArgs_ReadOnly(t *testing.T) {
|
||||
cfg := &NBDKitConfig{
|
||||
Socket: "/tmp/nbd.sock",
|
||||
PidFile: "/tmp/nbd.pid",
|
||||
PluginPath: "/plugin.so",
|
||||
ReadOnly: true,
|
||||
}
|
||||
|
||||
args, err := cfg.buildArgs()
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
|
||||
readOnlyIdx := slices.Index(args, "-r")
|
||||
if readOnlyIdx < 0 {
|
||||
t.Fatalf("expected -r in args, got: %v", args)
|
||||
}
|
||||
if pluginIdx := slices.Index(args, "/plugin.so"); readOnlyIdx > pluginIdx {
|
||||
t.Errorf("-r must precede the plugin path, got: %v", args)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildArgs_ReadOnlyOmittedByDefault(t *testing.T) {
|
||||
cfg := &NBDKitConfig{
|
||||
Socket: "/tmp/nbd.sock",
|
||||
PidFile: "/tmp/nbd.pid",
|
||||
PluginPath: "/plugin.so",
|
||||
}
|
||||
|
||||
args, err := cfg.buildArgs()
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if slices.Index(args, "-r") >= 0 {
|
||||
t.Errorf("-r must be absent unless ReadOnly is set, got: %v", args)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"github.com/mulgadc/spinifex/spinifex/services/nats"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/northstar"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/predastore"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/qemunbdd"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/qmpcollector"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/spinifex"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/spinifexui"
|
||||
@@ -27,6 +28,7 @@ var (
|
||||
_ Service = (*northstar.Service)(nil)
|
||||
_ Service = (*predastore.Service)(nil)
|
||||
_ Service = (*viperblockd.Service)(nil)
|
||||
_ Service = (*qemunbdd.Service)(nil)
|
||||
_ Service = (*spinifex.Service)(nil)
|
||||
_ Service = (*awsgw.Service)(nil)
|
||||
_ Service = (*spinifexui.Service)(nil)
|
||||
@@ -48,6 +50,9 @@ func New(btype string, config any) (Service, error) {
|
||||
case "viperblock":
|
||||
return viperblockd.New(config)
|
||||
|
||||
case "qemunbd":
|
||||
return qemunbdd.New(config)
|
||||
|
||||
case "spinifex":
|
||||
return spinifex.New(config)
|
||||
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
package qemunbdd_test
|
||||
|
||||
// The control for the A/B. qemunbdd shares no code with viperblock and stores
|
||||
// real qcow2 files, so what it costs above the null provider is what a
|
||||
// straightforward local implementation of these verbs costs.
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/conformance"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/natsserve"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// benchVolumeBytes matches the size the live viperblockd benchmark uses, so
|
||||
// the two sets of numbers are comparable rather than merely adjacent.
|
||||
const benchVolumeBytes int64 = 64 << 20
|
||||
|
||||
func BenchmarkQEMUNBDProvider_InProcess(b *testing.B) {
|
||||
requireQEMUTools(b)
|
||||
conformance.RunBenchSuite(b, newQEMUProviderTB(b), conformance.BenchConfig{
|
||||
VolumeBytes: benchVolumeBytes,
|
||||
})
|
||||
}
|
||||
|
||||
// BenchmarkQEMUNBDProvider_OverNATS is the shape the control plane actually
|
||||
// calls: same verbs, same transport, a provider that does real filesystem
|
||||
// work. The gap to the in-process run should be the seam and nothing else.
|
||||
func BenchmarkQEMUNBDProvider_OverNATS(b *testing.B) {
|
||||
requireQEMUTools(b)
|
||||
_, conn := testutil.StartTestNATS(b)
|
||||
stop, err := natsserve.Serve(b.Context(), conn, newQEMUProviderTB(b), natsserve.Options{})
|
||||
require.NoError(b, err)
|
||||
b.Cleanup(stop)
|
||||
|
||||
conformance.RunBenchSuite(b, ebsprovider.NewNATSProvider(conn, 120*time.Second), conformance.BenchConfig{
|
||||
VolumeBytes: benchVolumeBytes,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package qemunbdd_test
|
||||
|
||||
// The point of this provider: the same contract suite that covers
|
||||
// MemoryProvider, run unmodified against an implementation that shares no
|
||||
// code with viperblock and stores real qcow2 files.
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/conformance"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/natsserve"
|
||||
"github.com/mulgadc/spinifex/spinifex/services/qemunbdd"
|
||||
"github.com/mulgadc/spinifex/spinifex/testutil"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// requireQEMUTools skips when the host has no qemu, so the suite stays
|
||||
// runnable on a machine that cannot exercise the real binaries. CI sets
|
||||
// SPINIFEX_REQUIRE_CONFORMANCE_TOOLS so a runner image that loses qemu fails
|
||||
// loudly instead of quietly leaving MemoryProvider as the only implementation
|
||||
// the contract suite judges.
|
||||
func requireQEMUTools(t testing.TB) {
|
||||
t.Helper()
|
||||
for _, bin := range []string{"qemu-img", "qemu-nbd", "qemu-io"} {
|
||||
if _, err := exec.LookPath(bin); err != nil {
|
||||
if os.Getenv("SPINIFEX_REQUIRE_CONFORMANCE_TOOLS") != "" {
|
||||
t.Fatalf("%s not installed, but SPINIFEX_REQUIRE_CONFORMANCE_TOOLS is set", bin)
|
||||
}
|
||||
t.Skipf("%s not installed", bin)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// shortTempDir is used instead of t.TempDir because the test name is part of
|
||||
// that path, and AF_UNIX truncates a socket path at 108 bytes. Real base
|
||||
// directories are short; only the test names are not.
|
||||
func shortTempDir(t testing.TB) string {
|
||||
t.Helper()
|
||||
dir, err := os.MkdirTemp("", "qnbd") //nolint:usetesting // t.TempDir embeds the test name and overflows the 108-byte socket path limit
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = os.RemoveAll(dir) })
|
||||
return dir
|
||||
}
|
||||
|
||||
func newQEMUProvider(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
return newQEMUProviderTB(t)
|
||||
}
|
||||
|
||||
func newQEMUProviderTB(t testing.TB) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
provider, err := qemunbdd.NewProvider(shortTempDir(t))
|
||||
require.NoError(t, err)
|
||||
return provider
|
||||
}
|
||||
|
||||
func TestQEMUNBDProviderConformance(t *testing.T) {
|
||||
requireQEMUTools(t)
|
||||
conformance.RunSuite(t, newQEMUProvider)
|
||||
}
|
||||
|
||||
// TestQEMUNBDExternalNBDClient checks the half of the boundary our own Go
|
||||
// client can never check: whether the export we publish is usable by an NBD
|
||||
// client that knows nothing about this codebase.
|
||||
func TestQEMUNBDExternalNBDClient(t *testing.T) {
|
||||
requireQEMUTools(t)
|
||||
conformance.RunNBDClientSuite(t, newQEMUProvider)
|
||||
}
|
||||
|
||||
// TestQEMUNBDProviderNATSConformance is the swappability claim itself: the
|
||||
// same wire contract, the same neutral server, a provider with no viperblock
|
||||
// in it, and a client that cannot tell the difference.
|
||||
func TestQEMUNBDProviderNATSConformance(t *testing.T) {
|
||||
requireQEMUTools(t)
|
||||
conformance.RunSuite(t, func(t *testing.T) ebsprovider.EBSProvider {
|
||||
t.Helper()
|
||||
_, conn := testutil.StartTestNATS(t)
|
||||
stop, err := natsserve.Serve(t.Context(), conn, newQEMUProvider(t), natsserve.Options{})
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(stop)
|
||||
return ebsprovider.NewNATSProvider(conn, 15*time.Second)
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
// This test proves the qemu-nbd EBS provider works end to end against a
|
||||
// real qemunbdd daemon over a live cluster's NATS. It is opt-in only: run it
|
||||
// with SPINIFEX_QEMUNBD_LIVE=1 on the node the daemon is running on.
|
||||
package qemunbdd_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/admin"
|
||||
"github.com/mulgadc/spinifex/spinifex/config"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// liveEnvVar opts this test into running against a real cluster; unset (the
|
||||
// default) it always skips, so it never runs under make preflight.
|
||||
const liveEnvVar = "SPINIFEX_QEMUNBD_LIVE"
|
||||
|
||||
// defaultConfigPath mirrors the path the qemunbd systemd unit sets for
|
||||
// SPINIFEX_CONFIG_PATH when the operator does not override it.
|
||||
const defaultConfigPath = "/etc/spinifex/spinifex.toml"
|
||||
|
||||
const liveVolumeBytes int64 = 64 * 1024 * 1024
|
||||
|
||||
// TestLive_QEMUNBDLifecycle drives a real qemunbdd daemon end to end through
|
||||
// ebsprovider.NewNATSProvider, the same client shape the control plane uses,
|
||||
// over the cluster's real NATS.
|
||||
func TestLive_QEMUNBDLifecycle(t *testing.T) {
|
||||
if os.Getenv(liveEnvVar) == "" {
|
||||
t.Skipf("skipping: set %s=1 to run this test against a real qemunbdd daemon over a live cluster's NATS", liveEnvVar)
|
||||
}
|
||||
|
||||
cfgPath := os.Getenv("SPINIFEX_CONFIG_PATH")
|
||||
if cfgPath == "" {
|
||||
cfgPath = defaultConfigPath
|
||||
}
|
||||
t.Logf("loading cluster config from %s", cfgPath)
|
||||
|
||||
clusterConfig, err := config.LoadConfig(cfgPath)
|
||||
require.NoError(t, err, "load cluster config")
|
||||
|
||||
nodeID := clusterConfig.Node
|
||||
require.NotEmpty(t, nodeID, "cluster config has no node name set")
|
||||
|
||||
nodeConfig, ok := clusterConfig.Nodes[nodeID]
|
||||
require.True(t, ok, "cluster config has no entry for node %q", nodeID)
|
||||
|
||||
t.Logf("node %q: base dir %s", nodeID, nodeConfig.Predastore.BaseDir)
|
||||
|
||||
// NATS token and CA cert are read from config and handed straight to the
|
||||
// connect helper. Never log them, and never let a failure message embed
|
||||
// the config value that produced it.
|
||||
nc, err := utils.ConnectNATSWithRetry(admin.DialTarget(nodeConfig.NATS.Host), nodeConfig.NATS.ACL.Token, nodeConfig.NATS.CACert)
|
||||
require.NoError(t, err, "connect to NATS")
|
||||
t.Cleanup(nc.Close)
|
||||
t.Logf("connected to NATS")
|
||||
|
||||
provider := ebsprovider.NewNATSProvider(nc, 60*time.Second)
|
||||
|
||||
runID := strconv.FormatInt(time.Now().UnixNano(), 10)
|
||||
volumeID := "live-qnbd-vol-" + runID
|
||||
snapshotID := "live-qnbd-snap-" + runID
|
||||
restoredVolumeID := "live-qnbd-restore-" + runID
|
||||
pattern := byte(0xAB)
|
||||
|
||||
t.Cleanup(func() {
|
||||
cleanupCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
cleanupBestEffort(t, "unpublish restored volume", func() error {
|
||||
return provider.UnpublishVolume(cleanupCtx, ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: restoredVolumeID, NodeID: nodeID})
|
||||
})
|
||||
cleanupBestEffort(t, "delete restored volume", func() error {
|
||||
return provider.DeleteVolume(cleanupCtx, ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: restoredVolumeID})
|
||||
})
|
||||
cleanupBestEffort(t, "unpublish volume", func() error {
|
||||
return provider.UnpublishVolume(cleanupCtx, ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: nodeID})
|
||||
})
|
||||
cleanupBestEffort(t, "delete snapshot", func() error {
|
||||
return provider.DeleteSnapshot(cleanupCtx, ebsprovider.DeleteSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID})
|
||||
})
|
||||
cleanupBestEffort(t, "delete volume", func() error {
|
||||
return provider.DeleteVolume(cleanupCtx, ebsprovider.DeleteVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID})
|
||||
})
|
||||
})
|
||||
|
||||
ctx := t.Context()
|
||||
|
||||
vol, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: volumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: liveVolumeBytes},
|
||||
})
|
||||
require.NoError(t, err, "create volume")
|
||||
t.Logf("created volume %s: %d bytes, state=%s", vol.ID, vol.CapacityBytes, vol.State)
|
||||
|
||||
pub, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: nodeID})
|
||||
require.NoError(t, err, "publish volume")
|
||||
t.Logf("published volume %s to node %s: %s", pub.VolumeID, pub.NodeID, pub.NBDURI)
|
||||
sockPath := requireUnixSocketPath(t, pub.NBDURI)
|
||||
|
||||
writePattern(t, sockPath, pattern, liveVolumeBytes)
|
||||
readAndVerifyPattern(t, sockPath, pattern, liveVolumeBytes)
|
||||
|
||||
require.NoError(t, provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: volumeID, NodeID: nodeID}), "unpublish volume")
|
||||
t.Logf("unpublished volume %s", volumeID)
|
||||
|
||||
snap, err := provider.CreateSnapshot(ctx, ebsprovider.CreateSnapshotRequest{Versioned: ebsprovider.NewVersioned(), SnapshotID: snapshotID, VolumeID: volumeID})
|
||||
require.NoError(t, err, "create snapshot")
|
||||
t.Logf("created snapshot %s from volume %s: state=%s, size=%d", snap.ID, volumeID, snap.State, snap.SizeBytes)
|
||||
require.Equal(t, ebsprovider.SnapshotStateCompleted, snap.State, "snapshot must report completed")
|
||||
|
||||
restoredVol, err := provider.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: ebsprovider.NewVersioned(),
|
||||
VolumeID: restoredVolumeID,
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: liveVolumeBytes},
|
||||
SourceSnapshotID: snapshotID,
|
||||
})
|
||||
require.NoError(t, err, "create volume from snapshot")
|
||||
t.Logf("created volume %s from snapshot %s: %d bytes, state=%s", restoredVol.ID, snapshotID, restoredVol.CapacityBytes, restoredVol.State)
|
||||
|
||||
restoredPub, err := provider.PublishVolume(ctx, ebsprovider.PublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: restoredVolumeID, NodeID: nodeID})
|
||||
require.NoError(t, err, "publish restored volume")
|
||||
t.Logf("published restored volume %s to node %s: %s", restoredPub.VolumeID, restoredPub.NodeID, restoredPub.NBDURI)
|
||||
restoredSockPath := requireUnixSocketPath(t, restoredPub.NBDURI)
|
||||
|
||||
// The real proof: bytes written before the snapshot survive both the
|
||||
// snapshot and the restore into an independent volume.
|
||||
readAndVerifyPattern(t, restoredSockPath, pattern, liveVolumeBytes)
|
||||
|
||||
require.NoError(t, provider.UnpublishVolume(ctx, ebsprovider.UnpublishVolumeRequest{Versioned: ebsprovider.NewVersioned(), VolumeID: restoredVolumeID, NodeID: nodeID}), "unpublish restored volume")
|
||||
t.Logf("unpublished restored volume %s", restoredVolumeID)
|
||||
}
|
||||
|
||||
// cleanupBestEffort logs a cleanup step's outcome without failing the test:
|
||||
// cleanup runs after the real assertions and must never mask their result.
|
||||
func cleanupBestEffort(t *testing.T, step string, fn func() error) {
|
||||
t.Helper()
|
||||
if err := fn(); err != nil {
|
||||
t.Logf("cleanup: %s: %v", step, err)
|
||||
} else {
|
||||
t.Logf("cleanup: %s: ok", step)
|
||||
}
|
||||
}
|
||||
|
||||
// requireUnixSocketPath asserts nbdURI is a nbd:unix: URI and returns its
|
||||
// socket path, parsed with the same helper the daemon uses to format it
|
||||
// rather than by slicing the string.
|
||||
func requireUnixSocketPath(t *testing.T, nbdURI string) string {
|
||||
t.Helper()
|
||||
serverType, path, _, _, err := utils.ParseNBDURI(nbdURI)
|
||||
require.NoError(t, err, "parse NBD URI %q", nbdURI)
|
||||
require.Equal(t, "unix", serverType, "published volume must be a nbd:unix: URI, got %q", nbdURI)
|
||||
require.NotEmpty(t, path, "NBD URI %q has an empty socket path", nbdURI)
|
||||
return path
|
||||
}
|
||||
|
||||
// qemuIOTarget builds the client-facing NBD URI qemu-io understands.
|
||||
// qemu-nbd exports the guest view raw regardless of the qcow2 file backing
|
||||
// it, so the client always dials with -f raw.
|
||||
func qemuIOTarget(sockPath string) string {
|
||||
return fmt.Sprintf("nbd+unix:///?socket=%s", sockPath)
|
||||
}
|
||||
|
||||
// runQEMUIO execs one qemu-io command against sockPath as its own process,
|
||||
// which is itself a fresh NBD connection independent of any prior exec.
|
||||
func runQEMUIO(t *testing.T, sockPath, ioCmd string) (string, error) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "qemu-io", "-f", "raw", "-c", ioCmd, qemuIOTarget(sockPath)).CombinedOutput()
|
||||
return string(out), err
|
||||
}
|
||||
|
||||
// writePattern writes a byte pattern across the whole volume through NBD.
|
||||
func writePattern(t *testing.T, sockPath string, pattern byte, size int64) {
|
||||
t.Helper()
|
||||
ioCmd := fmt.Sprintf("write -P 0x%02x 0 %d", pattern, size)
|
||||
out, err := runQEMUIO(t, sockPath, ioCmd)
|
||||
require.NoError(t, err, "qemu-io write failed: %s", strings.TrimSpace(out))
|
||||
t.Logf("wrote pattern 0x%02x across %d bytes via qemu-io", pattern, size)
|
||||
}
|
||||
|
||||
// readAndVerifyPattern reads the volume back through NBD and asserts every
|
||||
// byte matches pattern, checking both qemu-io's exit status and its output
|
||||
// for a mismatch complaint.
|
||||
func readAndVerifyPattern(t *testing.T, sockPath string, pattern byte, size int64) {
|
||||
t.Helper()
|
||||
ioCmd := fmt.Sprintf("read -P 0x%02x 0 %d", pattern, size)
|
||||
out, err := runQEMUIO(t, sockPath, ioCmd)
|
||||
require.NoError(t, err, "qemu-io read failed: %s", strings.TrimSpace(out))
|
||||
require.NotContains(t, out, "Pattern verification failed", "qemu-io reported a pattern mismatch: %s", strings.TrimSpace(out))
|
||||
t.Logf("verified pattern 0x%02x across %d bytes via qemu-io", pattern, size)
|
||||
}
|
||||
@@ -0,0 +1,769 @@
|
||||
package qemunbdd
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
)
|
||||
|
||||
// nbdSharedClients bounds how many simultaneous NBD connections qemu-nbd
|
||||
// accepts against one export. A volume has exactly one attaching client, so
|
||||
// this stays at qemu-nbd's own default rather than inviting concurrent writers through the same socket.
|
||||
const nbdSharedClients = 1
|
||||
|
||||
// maxUnixSocketPath is the kernel's sun_path limit for an AF_UNIX address.
|
||||
// A base directory deep enough to push a socket past it must fail with that
|
||||
// reason rather than with qemu-nbd's raw exit status.
|
||||
const maxUnixSocketPath = 108
|
||||
|
||||
// capabilities is fixed: this provider offers crash consistent snapshots and
|
||||
// base:allocation extents but, unlike viperblockd, cannot expand a published
|
||||
// volume or answer for a volume over an owner subject.
|
||||
var capabilities = ebsprovider.Capabilities{
|
||||
CrashConsistentSnapshot: true,
|
||||
OnlineExpansion: false,
|
||||
SparseExtentReporting: true,
|
||||
VolumeSeeding: true,
|
||||
ReadOnlyPublish: true,
|
||||
OwnerRouting: false,
|
||||
VolumeEnumeration: true,
|
||||
SnapshotEnumeration: true,
|
||||
// Publication is tracked in this process's own map, so a second publish
|
||||
// is refused here and nowhere else. Another node running against the same
|
||||
// baseDir would not be seen, which is why this is node and not cluster.
|
||||
Exclusion: ebsprovider.ExclusionSemantics{Scope: ebsprovider.ExclusionScopeNode},
|
||||
}
|
||||
|
||||
// volumeMeta carries the request attributes CreateVolume's idempotency check
|
||||
// needs but a qcow2 file has no field for: a bare image has no notion of
|
||||
// availability zone or opaque caller parameters. Kept in memory only, so it does not survive a provider restart.
|
||||
type volumeMeta struct {
|
||||
availabilityZone string
|
||||
parameters []byte
|
||||
}
|
||||
|
||||
// snapshotMeta records a snapshot's source volume. CreateSnapshot copies the
|
||||
// volume file outright rather than leaving a backing-file pointer (the
|
||||
// snapshot must survive the source volume's deletion), so this is the only place that link is recorded; it is memory-only like volumeMeta.
|
||||
type snapshotMeta struct {
|
||||
sourceVolumeID string
|
||||
}
|
||||
|
||||
// publication is a running qemu-nbd export: its socket, its node, and the PID
|
||||
// captured from --pid-file so UnpublishVolume can stop it. qemu-nbd forks
|
||||
// into the background on success (--fork), so this PID is the only handle this process retains on it.
|
||||
type publication struct {
|
||||
nodeID string
|
||||
socketPath string
|
||||
pid int
|
||||
}
|
||||
|
||||
// Provider is an EBSProvider backed by qcow2 files and qemu-nbd. One volume
|
||||
// is one qcow2 file; a snapshot is an independent copy, not a chunk
|
||||
// reference, so DeleteVolume never has to consult a snapshot the way viperblock does.
|
||||
type Provider struct {
|
||||
baseDir string
|
||||
run runner
|
||||
|
||||
mu sync.Mutex
|
||||
published map[string]*publication
|
||||
volumeMeta map[string]volumeMeta
|
||||
snapshotMeta map[string]snapshotMeta
|
||||
}
|
||||
|
||||
var _ ebsprovider.EBSProvider = (*Provider)(nil)
|
||||
|
||||
// NewProvider creates a Provider rooted at baseDir, creating its volumes,
|
||||
// snapshots, sockets and tmp subdirectories if they do not already exist.
|
||||
func NewProvider(baseDir string) (*Provider, error) {
|
||||
return newProvider(baseDir, execRunner{})
|
||||
}
|
||||
|
||||
func newProvider(baseDir string, run runner) (*Provider, error) {
|
||||
if baseDir == "" {
|
||||
return nil, fmt.Errorf("qemunbdd: base directory is required")
|
||||
}
|
||||
absBaseDir, err := filepath.Abs(baseDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("qemunbdd: resolve base directory: %w", err)
|
||||
}
|
||||
for _, sub := range []string{"volumes", "snapshots", "sockets", "tmp"} {
|
||||
if err := os.MkdirAll(filepath.Join(absBaseDir, sub), 0o750); err != nil {
|
||||
return nil, fmt.Errorf("qemunbdd: create %s directory: %w", sub, err)
|
||||
}
|
||||
}
|
||||
return &Provider{
|
||||
baseDir: absBaseDir,
|
||||
run: run,
|
||||
published: make(map[string]*publication),
|
||||
volumeMeta: make(map[string]volumeMeta),
|
||||
snapshotMeta: make(map[string]snapshotMeta),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (p *Provider) volumePath(volumeID string) string {
|
||||
return filepath.Join(p.baseDir, "volumes", volumeID+".qcow2")
|
||||
}
|
||||
|
||||
func (p *Provider) snapshotPath(snapshotID string) string {
|
||||
return filepath.Join(p.baseDir, "snapshots", snapshotID+".qcow2")
|
||||
}
|
||||
|
||||
func (p *Provider) socketPath(volumeID string) string {
|
||||
return filepath.Join(p.baseDir, "sockets", volumeID+".sock")
|
||||
}
|
||||
|
||||
func (p *Provider) pidFilePath(volumeID string) string {
|
||||
return filepath.Join(p.baseDir, "sockets", volumeID+".pid")
|
||||
}
|
||||
|
||||
// checkVersion mirrors ebsprovider's own unexported check: the wire schema
|
||||
// version must match exactly, since this provider does not translate between
|
||||
// versions.
|
||||
func checkVersion(version uint16) error {
|
||||
if version != ebsprovider.SchemaVersion {
|
||||
return fmt.Errorf("%w: got %d, want %d", ebsprovider.ErrUnsupportedVersion, version, ebsprovider.SchemaVersion)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validCapacityRange(cr ebsprovider.CapacityRange) bool {
|
||||
return cr.RequiredBytes > 0 && (cr.LimitBytes == 0 || cr.RequiredBytes <= cr.LimitBytes)
|
||||
}
|
||||
|
||||
func (p *Provider) GetCapabilities(_ context.Context, req ebsprovider.GetCapabilitiesRequest) (*ebsprovider.GetCapabilitiesResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ebsprovider.GetCapabilitiesResponse{Versioned: ebsprovider.NewVersioned(), Capabilities: capabilities}, nil
|
||||
}
|
||||
|
||||
// qemuImgInfo is the subset of `qemu-img info --output=json` this provider
|
||||
// reads: the image's logical size, and the backing file a CoW clone points
|
||||
// at, which stands in for a Go-side "created from snapshot X" record.
|
||||
type qemuImgInfo struct {
|
||||
VirtualSize int64 `json:"virtual-size"`
|
||||
BackingFilename string `json:"backing-filename"`
|
||||
}
|
||||
|
||||
// imgInfo reads an image's metadata. --force-share is required because a
|
||||
// published volume is held under qemu-nbd's write lock, and describing a
|
||||
// volume must not depend on whether it is currently attached.
|
||||
func (p *Provider) imgInfo(ctx context.Context, path string) (*qemuImgInfo, error) {
|
||||
out, err := p.run.Run(ctx, "qemu-img", "info", "--output=json", "--force-share", path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("inspect %s: %w", path, err)
|
||||
}
|
||||
var info qemuImgInfo
|
||||
if err := json.Unmarshal(out, &info); err != nil {
|
||||
return nil, fmt.Errorf("parse qemu-img info for %s: %w", path, err)
|
||||
}
|
||||
return &info, nil
|
||||
}
|
||||
|
||||
func (p *Provider) CreateVolume(ctx context.Context, req ebsprovider.CreateVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" || !validCapacityRange(req.CapacityRange) {
|
||||
return nil, fmt.Errorf("%w: invalid volume ID or capacity range", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
if req.SourceSnapshotID != "" && req.SourceSnapshotVolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: source_snapshot_volume_id is required with source_snapshot_id", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
if err := ebsprovider.ValidateSeedData(req.SeedData); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(req.SeedData)) > req.CapacityRange.RequiredBytes {
|
||||
return nil, fmt.Errorf("%w: seed data is larger than the requested capacity", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
volPath := p.volumePath(req.VolumeID)
|
||||
expectedBacking := ""
|
||||
if req.SourceSnapshotID != "" {
|
||||
expectedBacking = p.snapshotPath(req.SourceSnapshotID)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(volPath); err == nil {
|
||||
info, err := p.imgInfo(ctx, volPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
meta := p.volumeMeta[req.VolumeID]
|
||||
if info.VirtualSize != req.CapacityRange.RequiredBytes ||
|
||||
meta.availabilityZone != req.AvailabilityZone ||
|
||||
info.BackingFilename != expectedBacking ||
|
||||
!bytes.Equal(meta.parameters, req.Parameters) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrAlreadyExists, req.VolumeID)
|
||||
}
|
||||
return p.buildVolume(req.VolumeID, info.VirtualSize, meta.availabilityZone), nil
|
||||
} else if !errors.Is(err, os.ErrNotExist) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
sizeArg := strconv.FormatInt(req.CapacityRange.RequiredBytes, 10)
|
||||
switch {
|
||||
case req.SourceSnapshotID != "":
|
||||
snapPath := p.snapshotPath(req.SourceSnapshotID)
|
||||
snapInfo, err := p.statSnapshot(ctx, snapPath)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ebsprovider.ErrNotFound, req.SourceSnapshotID)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if req.CapacityRange.RequiredBytes < snapInfo.VirtualSize {
|
||||
return nil, fmt.Errorf("%w: requested capacity is smaller than snapshot %s", ebsprovider.ErrInvalidArgument, req.SourceSnapshotID)
|
||||
}
|
||||
// Only refuse an origin we can disprove: a snapshot file left by an
|
||||
// earlier process has no snapshotMeta entry, so its origin is unknown
|
||||
// rather than mismatched.
|
||||
if snapMeta, ok := p.snapshotMeta[req.SourceSnapshotID]; ok && snapMeta.sourceVolumeID != req.SourceSnapshotVolumeID {
|
||||
return nil, fmt.Errorf("%w: snapshot %s was taken from volume %q, not %q",
|
||||
ebsprovider.ErrInvalidArgument, req.SourceSnapshotID, snapMeta.sourceVolumeID, req.SourceSnapshotVolumeID)
|
||||
}
|
||||
if _, err := p.run.Run(ctx, "qemu-img", "create", "-f", "qcow2", "-b", snapPath, "-F", "qcow2", volPath, sizeArg); err != nil {
|
||||
return nil, fmt.Errorf("create volume %s from snapshot %s: %w", req.VolumeID, req.SourceSnapshotID, err)
|
||||
}
|
||||
|
||||
default:
|
||||
if _, err := p.run.Run(ctx, "qemu-img", "create", "-f", "qcow2", volPath, sizeArg); err != nil {
|
||||
return nil, fmt.Errorf("create volume %s: %w", req.VolumeID, err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := p.writeSeed(ctx, volPath, req.SeedData); err != nil {
|
||||
_ = os.Remove(volPath)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
p.volumeMeta[req.VolumeID] = volumeMeta{availabilityZone: req.AvailabilityZone, parameters: bytes.Clone(req.Parameters)}
|
||||
return p.buildVolume(req.VolumeID, req.CapacityRange.RequiredBytes, req.AvailabilityZone), nil
|
||||
}
|
||||
|
||||
// statSnapshot resolves os.ErrNotExist through os.Stat before shelling out,
|
||||
// so a missing source snapshot is reported without invoking qemu-img on a
|
||||
// path that cannot exist.
|
||||
func (p *Provider) statSnapshot(ctx context.Context, snapPath string) (*qemuImgInfo, error) {
|
||||
if _, err := os.Stat(snapPath); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return p.imgInfo(ctx, snapPath)
|
||||
}
|
||||
|
||||
// writeSeed stages SeedData to a temporary file and writes it at offset 0
|
||||
// with qemu-io. qemu-img has no operation for arbitrary byte writes, and
|
||||
// qemu-io's write command reads its buffer from a file (-s), not an inline argument, so the bytes must round-trip through disk first.
|
||||
func (p *Provider) writeSeed(ctx context.Context, volPath string, seed []byte) error {
|
||||
if len(seed) == 0 {
|
||||
return nil
|
||||
}
|
||||
tmp, err := os.CreateTemp(filepath.Join(p.baseDir, "tmp"), "seed-*.bin")
|
||||
if err != nil {
|
||||
return fmt.Errorf("stage seed data: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
defer os.Remove(tmpPath)
|
||||
if _, err := tmp.Write(seed); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("stage seed data: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("stage seed data: %w", err)
|
||||
}
|
||||
writeCmd := fmt.Sprintf("write -s %s 0 %d", tmpPath, len(seed))
|
||||
if _, err := p.run.Run(ctx, "qemu-io", "-f", "qcow2", "-c", writeCmd, volPath); err != nil {
|
||||
return fmt.Errorf("write seed data: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// buildVolume must be called with p.mu held: it consults the published map.
|
||||
func (p *Provider) buildVolume(volumeID string, size int64, az string) *ebsprovider.Volume {
|
||||
state := ebsprovider.VolumeStateAvailable
|
||||
if _, ok := p.published[volumeID]; ok {
|
||||
state = ebsprovider.VolumeStateInUse
|
||||
}
|
||||
return &ebsprovider.Volume{
|
||||
ID: volumeID,
|
||||
CapacityBytes: size,
|
||||
State: state,
|
||||
Handle: p.volumePath(volumeID),
|
||||
AvailabilityZone: az,
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Provider) GetVolume(ctx context.Context, req ebsprovider.GetVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: volume ID is required", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
volPath := p.volumePath(req.VolumeID)
|
||||
if _, err := os.Stat(volPath); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if req.Handle != "" && req.Handle != volPath {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
info, err := p.imgInfo(ctx, volPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
meta := p.volumeMeta[req.VolumeID]
|
||||
return p.buildVolume(req.VolumeID, info.VirtualSize, meta.availabilityZone), nil
|
||||
}
|
||||
|
||||
// ListVolumes reads the volumes directory rather than the in-memory maps.
|
||||
// The qcow2 files are what actually exists: volumeMeta does not survive a
|
||||
// restart, and a volume the provider has forgotten is exactly what a caller
|
||||
// enumerating is looking for.
|
||||
func (p *Provider) ListVolumes(_ context.Context, req ebsprovider.ListVolumesRequest) (*ebsprovider.ListVolumesResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries, err := os.ReadDir(filepath.Join(p.baseDir, "volumes"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("qemunbdd: read volumes directory: %w", err)
|
||||
}
|
||||
|
||||
pageSize := int(req.PageSize())
|
||||
response := &ebsprovider.ListVolumesResponse{Versioned: ebsprovider.NewVersioned()}
|
||||
// ReadDir returns entries sorted by filename, so resuming after the token
|
||||
// walks the same order the previous page ended on.
|
||||
for _, entry := range entries {
|
||||
volumeID := strings.TrimSuffix(entry.Name(), ".qcow2")
|
||||
if entry.IsDir() || volumeID == entry.Name() || volumeID <= req.StartingToken {
|
||||
continue
|
||||
}
|
||||
if len(response.Volumes) == pageSize {
|
||||
response.NextToken = response.Volumes[pageSize-1].ID
|
||||
break
|
||||
}
|
||||
response.Volumes = append(response.Volumes, ebsprovider.VolumeRef{
|
||||
ID: volumeID,
|
||||
Handle: p.volumePath(volumeID),
|
||||
})
|
||||
}
|
||||
return response, nil
|
||||
}
|
||||
|
||||
func (p *Provider) ExpandVolume(ctx context.Context, req ebsprovider.ExpandVolumeRequest) (*ebsprovider.Volume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" || !validCapacityRange(req.CapacityRange) {
|
||||
return nil, fmt.Errorf("%w: invalid volume ID or capacity range", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
volPath := p.volumePath(req.VolumeID)
|
||||
if _, err := os.Stat(volPath); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if req.Handle != "" && req.Handle != volPath {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
info, err := p.imgInfo(ctx, volPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.CapacityRange.RequiredBytes < info.VirtualSize {
|
||||
return nil, fmt.Errorf("%w: volume expansion is grow-only", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
// ebsprovider/errors.go has no failed-precondition sentinel, so a
|
||||
// published volume this provider cannot expand online is reported as
|
||||
// ErrVolumeInUse, matching memory.go's identical refusal.
|
||||
if _, published := p.published[req.VolumeID]; published && req.CapacityRange.RequiredBytes > info.VirtualSize {
|
||||
return nil, fmt.Errorf("%w: provider does not support online expansion", ebsprovider.ErrVolumeInUse)
|
||||
}
|
||||
if req.CapacityRange.RequiredBytes != info.VirtualSize {
|
||||
if _, err := p.run.Run(ctx, "qemu-img", "resize", volPath, strconv.FormatInt(req.CapacityRange.RequiredBytes, 10)); err != nil {
|
||||
return nil, fmt.Errorf("expand volume %s: %w", req.VolumeID, err)
|
||||
}
|
||||
}
|
||||
meta := p.volumeMeta[req.VolumeID]
|
||||
return p.buildVolume(req.VolumeID, req.CapacityRange.RequiredBytes, meta.availabilityZone), nil
|
||||
}
|
||||
|
||||
func (p *Provider) DeleteVolume(_ context.Context, req ebsprovider.DeleteVolumeRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.VolumeID == "" {
|
||||
return fmt.Errorf("%w: volume ID is required", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
volPath := p.volumePath(req.VolumeID)
|
||||
if _, err := os.Stat(volPath); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
if req.Handle != "" && req.Handle != volPath {
|
||||
return nil
|
||||
}
|
||||
if _, published := p.published[req.VolumeID]; published {
|
||||
return fmt.Errorf("%w: volume %s", ebsprovider.ErrVolumeInUse, req.VolumeID)
|
||||
}
|
||||
if err := os.Remove(volPath); err != nil {
|
||||
return fmt.Errorf("delete volume %s: %w", req.VolumeID, err)
|
||||
}
|
||||
delete(p.volumeMeta, req.VolumeID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Provider) buildSnapshot(ctx context.Context, snapshotID, sourceVolumeID, path string) (*ebsprovider.Snapshot, error) {
|
||||
info, err := p.imgInfo(ctx, path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fi, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("stat snapshot %s: %w", snapshotID, err)
|
||||
}
|
||||
return &ebsprovider.Snapshot{
|
||||
ID: snapshotID,
|
||||
SourceVolumeID: sourceVolumeID,
|
||||
SizeBytes: info.VirtualSize,
|
||||
CreatedAt: fi.ModTime().UTC(),
|
||||
State: ebsprovider.SnapshotStateCompleted,
|
||||
Handle: path,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (p *Provider) CreateSnapshot(ctx context.Context, req ebsprovider.CreateSnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.SnapshotID == "" || req.VolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: snapshot and volume IDs are required", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
snapPath := p.snapshotPath(req.SnapshotID)
|
||||
if meta, ok := p.snapshotMeta[req.SnapshotID]; ok {
|
||||
if meta.sourceVolumeID != req.VolumeID {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ebsprovider.ErrAlreadyExists, req.SnapshotID)
|
||||
}
|
||||
return p.buildSnapshot(ctx, req.SnapshotID, meta.sourceVolumeID, snapPath)
|
||||
}
|
||||
// A file on disk with no matching snapshotMeta entry means the provider
|
||||
// restarted since it was created: the source-volume link lived only in
|
||||
// memory and cannot be recovered, so the ID is refused rather than silently overwritten or accepted without provenance.
|
||||
if _, err := os.Stat(snapPath); err == nil {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ebsprovider.ErrAlreadyExists, req.SnapshotID)
|
||||
} else if !errors.Is(err, os.ErrNotExist) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
volPath := p.volumePath(req.VolumeID)
|
||||
if _, err := os.Stat(volPath); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeHandle != "" && req.VolumeHandle != volPath {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
|
||||
if _, err := p.run.Run(ctx, "qemu-img", "convert", "-O", "qcow2", volPath, snapPath); err != nil {
|
||||
return nil, fmt.Errorf("create snapshot %s: %w", req.SnapshotID, err)
|
||||
}
|
||||
p.snapshotMeta[req.SnapshotID] = snapshotMeta{sourceVolumeID: req.VolumeID}
|
||||
return p.buildSnapshot(ctx, req.SnapshotID, req.VolumeID, snapPath)
|
||||
}
|
||||
|
||||
// ListSnapshots walks the snapshots directory rather than snapshotMeta, so a
|
||||
// snapshot that outlived the process still appears. Its SourceVolumeID comes
|
||||
// back empty in that case: the link was only ever in memory.
|
||||
func (p *Provider) ListSnapshots(_ context.Context, req ebsprovider.ListSnapshotsRequest) (*ebsprovider.ListSnapshotsResponse, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries, err := os.ReadDir(filepath.Join(p.baseDir, "snapshots"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("qemunbdd: read snapshots directory: %w", err)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
pageSize := int(req.PageSize())
|
||||
response := &ebsprovider.ListSnapshotsResponse{Versioned: ebsprovider.NewVersioned()}
|
||||
// ReadDir returns entries sorted by filename, so resuming after the token
|
||||
// walks the same order the previous page ended on.
|
||||
for _, entry := range entries {
|
||||
snapshotID := strings.TrimSuffix(entry.Name(), ".qcow2")
|
||||
if entry.IsDir() || snapshotID == entry.Name() || snapshotID <= req.StartingToken {
|
||||
continue
|
||||
}
|
||||
if len(response.Snapshots) == pageSize {
|
||||
response.NextToken = response.Snapshots[pageSize-1].ID
|
||||
break
|
||||
}
|
||||
response.Snapshots = append(response.Snapshots, ebsprovider.SnapshotRef{
|
||||
ID: snapshotID,
|
||||
SourceVolumeID: p.snapshotMeta[snapshotID].sourceVolumeID,
|
||||
Handle: p.snapshotPath(snapshotID),
|
||||
})
|
||||
}
|
||||
return response, nil
|
||||
}
|
||||
|
||||
// snapshotInUse reports whether any volume file still points at snapPath as
|
||||
// its qcow2 backing file, the same relationship CreateVolume establishes for
|
||||
// a SourceSnapshotID clone.
|
||||
func (p *Provider) snapshotInUse(ctx context.Context, snapPath string) (bool, error) {
|
||||
volumesDir := filepath.Join(p.baseDir, "volumes")
|
||||
entries, err := os.ReadDir(volumesDir)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("list volumes: %w", err)
|
||||
}
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
info, err := p.imgInfo(ctx, filepath.Join(volumesDir, entry.Name()))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if info.BackingFilename == snapPath {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
||||
func (p *Provider) DeleteSnapshot(ctx context.Context, req ebsprovider.DeleteSnapshotRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.SnapshotID == "" {
|
||||
return fmt.Errorf("%w: snapshot ID is required", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
snapPath := p.snapshotPath(req.SnapshotID)
|
||||
if _, err := os.Stat(snapPath); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
if req.Handle != "" && req.Handle != snapPath {
|
||||
return nil
|
||||
}
|
||||
inUse, err := p.snapshotInUse(ctx, snapPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if inUse {
|
||||
// Same sentinel-substitution as ExpandVolume/DeleteVolume: no
|
||||
// failed-precondition error exists, so a snapshot a volume still
|
||||
// backs onto is reported as ErrVolumeInUse.
|
||||
return fmt.Errorf("%w: snapshot %s is a backing file for another volume", ebsprovider.ErrVolumeInUse, req.SnapshotID)
|
||||
}
|
||||
if err := os.Remove(snapPath); err != nil {
|
||||
return fmt.Errorf("delete snapshot %s: %w", req.SnapshotID, err)
|
||||
}
|
||||
delete(p.snapshotMeta, req.SnapshotID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Provider) CopySnapshot(ctx context.Context, req ebsprovider.CopySnapshotRequest) (*ebsprovider.Snapshot, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.SourceSnapshotID == "" || req.DestinationSnapshotID == "" || req.VolumeID == "" {
|
||||
return nil, fmt.Errorf("%w: source snapshot, destination snapshot, and volume IDs are required", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
if req.SourceSnapshotID == req.DestinationSnapshotID {
|
||||
return nil, fmt.Errorf("%w: destination snapshot must differ from the source", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
srcMeta, ok := p.snapshotMeta[req.SourceSnapshotID]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ebsprovider.ErrNotFound, req.SourceSnapshotID)
|
||||
}
|
||||
if srcMeta.sourceVolumeID != req.VolumeID {
|
||||
return nil, fmt.Errorf("%w: snapshot %s belongs to volume %s, not %s", ebsprovider.ErrInvalidArgument, req.SourceSnapshotID, srcMeta.sourceVolumeID, req.VolumeID)
|
||||
}
|
||||
dstPath := p.snapshotPath(req.DestinationSnapshotID)
|
||||
if _, ok := p.snapshotMeta[req.DestinationSnapshotID]; ok {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ebsprovider.ErrAlreadyExists, req.DestinationSnapshotID)
|
||||
}
|
||||
// A stray file with no metadata entry is the same restart-orphan case
|
||||
// CreateSnapshot refuses: without provenance it cannot be told apart
|
||||
// from a legitimate destination, so it is treated as already existing.
|
||||
if _, err := os.Stat(dstPath); err == nil {
|
||||
return nil, fmt.Errorf("%w: snapshot %s", ebsprovider.ErrAlreadyExists, req.DestinationSnapshotID)
|
||||
} else if !errors.Is(err, os.ErrNotExist) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
srcPath := p.snapshotPath(req.SourceSnapshotID)
|
||||
if err := copyFile(srcPath, dstPath); err != nil {
|
||||
return nil, fmt.Errorf("copy snapshot %s to %s: %w", req.SourceSnapshotID, req.DestinationSnapshotID, err)
|
||||
}
|
||||
p.snapshotMeta[req.DestinationSnapshotID] = snapshotMeta{sourceVolumeID: req.VolumeID}
|
||||
return p.buildSnapshot(ctx, req.DestinationSnapshotID, req.VolumeID, dstPath)
|
||||
}
|
||||
|
||||
// copyFile duplicates a snapshot file directly: CopySnapshot's job is to
|
||||
// produce a second, independent file, not a CoW clone, so this is plain
|
||||
// filesystem I/O and never touches the runner.
|
||||
func copyFile(srcPath, dstPath string) error {
|
||||
src, err := os.Open(srcPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer src.Close()
|
||||
dst, err := os.OpenFile(dstPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer dst.Close()
|
||||
if _, err := io.Copy(dst, src); err != nil {
|
||||
return err
|
||||
}
|
||||
return dst.Close()
|
||||
}
|
||||
|
||||
func (p *Provider) PublishVolume(ctx context.Context, req ebsprovider.PublishVolumeRequest) (*ebsprovider.PublishedVolume, error) {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if req.VolumeID == "" || req.NodeID == "" {
|
||||
return nil, fmt.Errorf("%w: volume and node IDs are required", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
volPath := p.volumePath(req.VolumeID)
|
||||
if _, err := os.Stat(volPath); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if req.Handle != "" && req.Handle != volPath {
|
||||
return nil, fmt.Errorf("%w: volume %s", ebsprovider.ErrNotFound, req.VolumeID)
|
||||
}
|
||||
if pub, ok := p.published[req.VolumeID]; ok {
|
||||
if pub.nodeID != req.NodeID {
|
||||
return nil, fmt.Errorf("%w: volume %s is published to %s", ebsprovider.ErrVolumeInUse, req.VolumeID, pub.nodeID)
|
||||
}
|
||||
return &ebsprovider.PublishedVolume{VolumeID: req.VolumeID, NodeID: pub.nodeID, NBDURI: utils.FormatNBDSocketURI(pub.socketPath)}, nil
|
||||
}
|
||||
|
||||
sockPath := p.socketPath(req.VolumeID)
|
||||
if len(sockPath) >= maxUnixSocketPath {
|
||||
return nil, fmt.Errorf("%w: socket path %s is %d bytes, over the %d-byte kernel limit",
|
||||
ebsprovider.ErrInvalidArgument, sockPath, len(sockPath), maxUnixSocketPath)
|
||||
}
|
||||
pidPath := p.pidFilePath(req.VolumeID)
|
||||
_ = os.Remove(sockPath)
|
||||
_ = os.Remove(pidPath)
|
||||
|
||||
args := []string{
|
||||
"--socket", sockPath,
|
||||
"--format", "qcow2",
|
||||
"--persistent",
|
||||
"--shared=" + strconv.Itoa(nbdSharedClients),
|
||||
"--fork",
|
||||
"--pid-file", pidPath,
|
||||
}
|
||||
if req.ReadOnly {
|
||||
args = append(args, "-r")
|
||||
}
|
||||
args = append(args, volPath)
|
||||
if _, err := p.run.Run(ctx, "qemu-nbd", args...); err != nil {
|
||||
return nil, fmt.Errorf("publish volume %s: %w", req.VolumeID, err)
|
||||
}
|
||||
|
||||
pid := readPID(pidPath)
|
||||
p.published[req.VolumeID] = &publication{nodeID: req.NodeID, socketPath: sockPath, pid: pid}
|
||||
return &ebsprovider.PublishedVolume{VolumeID: req.VolumeID, NodeID: req.NodeID, NBDURI: utils.FormatNBDSocketURI(sockPath)}, nil
|
||||
}
|
||||
|
||||
// readPID best-effort reads the PID qemu-nbd's --fork wrote to pidPath. A
|
||||
// missing or unparsable file (as in tests, where no real qemu-nbd runs)
|
||||
// yields 0, meaning UnpublishVolume will skip signaling a process.
|
||||
func readPID(pidPath string) int {
|
||||
data, err := os.ReadFile(pidPath)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
pid, err := strconv.Atoi(string(bytes.TrimSpace(data)))
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
return pid
|
||||
}
|
||||
|
||||
func (p *Provider) UnpublishVolume(_ context.Context, req ebsprovider.UnpublishVolumeRequest) error {
|
||||
if err := checkVersion(req.SchemaVersion); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.VolumeID == "" || req.NodeID == "" {
|
||||
return fmt.Errorf("%w: volume and node IDs are required", ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
|
||||
pub, ok := p.published[req.VolumeID]
|
||||
if !ok || pub.nodeID != req.NodeID {
|
||||
return nil
|
||||
}
|
||||
if pub.pid > 0 {
|
||||
if proc, err := os.FindProcess(pub.pid); err == nil {
|
||||
_ = proc.Signal(syscall.SIGTERM)
|
||||
}
|
||||
}
|
||||
_ = os.Remove(pub.socketPath)
|
||||
_ = os.Remove(p.pidFilePath(req.VolumeID))
|
||||
delete(p.published, req.VolumeID)
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,662 @@
|
||||
package qemunbdd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// call records one invocation the fake runner captured, so tests can assert
|
||||
// the exact argv a provider method built.
|
||||
type call struct {
|
||||
name string
|
||||
args []string
|
||||
}
|
||||
|
||||
// fakeRunner stands in for qemu-img/qemu-io/qemu-nbd. It mimics the side
|
||||
// effect a real "create" or "convert" has (leaving a file behind) so
|
||||
// existence checks with os.Stat behave the same as against a real qemu-img, without needing qemu installed.
|
||||
type fakeRunner struct {
|
||||
mu sync.Mutex
|
||||
calls []call
|
||||
|
||||
// infoResponses maps a path to the canned `qemu-img info --output=json`
|
||||
// body returned when that path is queried.
|
||||
infoResponses map[string]string
|
||||
|
||||
// failOn, when non-empty, makes Run return err for every call whose
|
||||
// command name matches.
|
||||
failOn string
|
||||
err error
|
||||
}
|
||||
|
||||
func newFakeRunner() *fakeRunner {
|
||||
return &fakeRunner{infoResponses: make(map[string]string)}
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Run(_ context.Context, name string, args ...string) ([]byte, error) {
|
||||
f.mu.Lock()
|
||||
f.calls = append(f.calls, call{name: name, args: append([]string(nil), args...)})
|
||||
f.mu.Unlock()
|
||||
|
||||
if f.err != nil && name == f.failOn {
|
||||
return nil, f.err
|
||||
}
|
||||
|
||||
if name == "qemu-img" && len(args) > 0 {
|
||||
switch args[0] {
|
||||
case "create":
|
||||
// The size argument is always last; the target path is the
|
||||
// element before it.
|
||||
_ = os.WriteFile(args[len(args)-2], nil, 0o640)
|
||||
case "convert":
|
||||
_ = os.WriteFile(args[len(args)-1], nil, 0o640)
|
||||
case "info":
|
||||
path := args[len(args)-1]
|
||||
f.mu.Lock()
|
||||
resp, ok := f.infoResponses[path]
|
||||
f.mu.Unlock()
|
||||
if ok {
|
||||
return []byte(resp), nil
|
||||
}
|
||||
return []byte(`{"virtual-size":0}`), nil
|
||||
}
|
||||
}
|
||||
return []byte("ok"), nil
|
||||
}
|
||||
|
||||
func (f *fakeRunner) callsSnapshot() []call {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]call(nil), f.calls...)
|
||||
}
|
||||
|
||||
func newTestProvider(t *testing.T) (*Provider, *fakeRunner) {
|
||||
t.Helper()
|
||||
fr := newFakeRunner()
|
||||
p, err := newProvider(t.TempDir(), fr)
|
||||
require.NoError(t, err)
|
||||
return p, fr
|
||||
}
|
||||
|
||||
func TestNewProvider_CreatesDirectories(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
_, err := newProvider(dir, newFakeRunner())
|
||||
require.NoError(t, err)
|
||||
for _, sub := range []string{"volumes", "snapshots", "sockets", "tmp"} {
|
||||
info, err := os.Stat(filepath.Join(dir, sub))
|
||||
require.NoError(t, err)
|
||||
assert.True(t, info.IsDir())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewProvider_RequiresBaseDir(t *testing.T) {
|
||||
_, err := newProvider("", newFakeRunner())
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func versioned() ebsprovider.Versioned { return ebsprovider.NewVersioned() }
|
||||
|
||||
func TestCreateVolume_DefaultArgv(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
ctx := context.Background()
|
||||
|
||||
vol, err := p.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-1",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, vol)
|
||||
assert.Equal(t, "vol-1", vol.ID)
|
||||
assert.Equal(t, int64(1<<30), vol.CapacityBytes)
|
||||
assert.Equal(t, ebsprovider.VolumeStateAvailable, vol.State)
|
||||
assert.Equal(t, p.volumePath("vol-1"), vol.Handle)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 1)
|
||||
assert.Equal(t, "qemu-img", calls[0].name)
|
||||
assert.Equal(t, []string{"create", "-f", "qcow2", p.volumePath("vol-1"), "1073741824"}, calls[0].args)
|
||||
}
|
||||
|
||||
func TestCreateVolume_FromSnapshotArgv(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
ctx := context.Background()
|
||||
|
||||
snapPath := p.snapshotPath("snap-1")
|
||||
require.NoError(t, os.WriteFile(snapPath, nil, 0o640))
|
||||
fr.infoResponses[snapPath] = `{"virtual-size":536870912}`
|
||||
|
||||
vol, err := p.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-clone",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SourceSnapshotID: "snap-1",
|
||||
SourceSnapshotVolumeID: "vol-origin",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, vol)
|
||||
|
||||
volPath := p.volumePath("vol-clone")
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 2)
|
||||
assert.Equal(t, []string{"info", "--output=json", "--force-share", snapPath}, calls[0].args)
|
||||
assert.Equal(t, []string{"create", "-f", "qcow2", "-b", snapPath, "-F", "qcow2", volPath, "1073741824"}, calls[1].args)
|
||||
}
|
||||
|
||||
func TestCreateVolume_FromMissingSnapshotNotFound(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-orphan",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SourceSnapshotID: "snap-missing",
|
||||
SourceSnapshotVolumeID: "vol-origin",
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
// TestCreateVolume_SnapshotSourceRequiresOriginVolume covers the contract rule
|
||||
// that a snapshot source must name the volume it was taken from, so a provider
|
||||
// needing it to resolve blocks gets it rather than guessing.
|
||||
func TestCreateVolume_SnapshotSourceRequiresOriginVolume(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-noorigin",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SourceSnapshotID: "snap-any",
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestCreateVolume_IdempotentRepeatSameParameters(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
ctx := context.Background()
|
||||
req := ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-idem",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
}
|
||||
|
||||
first, err := p.CreateVolume(ctx, req)
|
||||
require.NoError(t, err)
|
||||
fr.infoResponses[p.volumePath("vol-idem")] = `{"virtual-size":1073741824}`
|
||||
|
||||
second, err := p.CreateVolume(ctx, req)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, first, second)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 2, "the repeat must only inspect the file, not recreate it")
|
||||
assert.Equal(t, "create", calls[0].args[0])
|
||||
assert.Equal(t, "info", calls[1].args[0])
|
||||
}
|
||||
|
||||
func TestCreateVolume_ConflictingCapacityIsAlreadyExists(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
ctx := context.Background()
|
||||
req := ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-conflict",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
}
|
||||
_, err := p.CreateVolume(ctx, req)
|
||||
require.NoError(t, err)
|
||||
fr.infoResponses[p.volumePath("vol-conflict")] = `{"virtual-size":1073741824}`
|
||||
|
||||
req2 := req
|
||||
req2.CapacityRange.RequiredBytes = 2 << 30
|
||||
_, err = p.CreateVolume(ctx, req2)
|
||||
require.ErrorIs(t, err, ebsprovider.ErrAlreadyExists)
|
||||
}
|
||||
|
||||
func TestCreateVolume_InvalidArgumentOnEmptyID(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(), CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestCreateVolume_UnsupportedVersion(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
VolumeID: "vol-noversion", CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrUnsupportedVersion)
|
||||
}
|
||||
|
||||
func TestCreateVolume_SeedDataWritesViaQemuIO(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
ctx := context.Background()
|
||||
seed := []byte("firmware-vars-template")
|
||||
|
||||
_, err := p.CreateVolume(ctx, ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-seeded",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 4096},
|
||||
SeedData: seed,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 2)
|
||||
assert.Equal(t, "qemu-img", calls[0].name)
|
||||
seedCall := calls[1]
|
||||
assert.Equal(t, "qemu-io", seedCall.name)
|
||||
require.Len(t, seedCall.args, 5)
|
||||
assert.Equal(t, []string{"-f", "qcow2"}, seedCall.args[0:2])
|
||||
assert.Equal(t, "-c", seedCall.args[2])
|
||||
assert.Contains(t, seedCall.args[3], "write -s ")
|
||||
assert.Contains(t, seedCall.args[3], " 0 22")
|
||||
assert.Equal(t, p.volumePath("vol-seeded"), seedCall.args[4])
|
||||
}
|
||||
|
||||
func TestCreateVolume_SeedDataExceedsMax(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-seed-toobig",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
SeedData: make([]byte, ebsprovider.MaxSeedBytes+1),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestCreateVolume_SeedDataExceedsCapacity(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CreateVolume(context.Background(), ebsprovider.CreateVolumeRequest{
|
||||
Versioned: versioned(),
|
||||
VolumeID: "vol-seed-overcap",
|
||||
CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 512},
|
||||
SeedData: make([]byte, 4096),
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestGetVolume_Success(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-get")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
fr.infoResponses[volPath] = `{"virtual-size":268435456}`
|
||||
|
||||
vol, err := p.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{Versioned: versioned(), VolumeID: "vol-get"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(268435456), vol.CapacityBytes)
|
||||
assert.Equal(t, ebsprovider.VolumeStateAvailable, vol.State)
|
||||
}
|
||||
|
||||
func TestGetVolume_NotFound(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{Versioned: versioned(), VolumeID: "vol-never-existed"})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
func TestGetVolume_InvalidArgumentOnEmptyID(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.GetVolume(context.Background(), ebsprovider.GetVolumeRequest{Versioned: versioned()})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestExpandVolume_Argv(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-expand")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
fr.infoResponses[volPath] = `{"virtual-size":1073741824}`
|
||||
|
||||
vol, err := p.ExpandVolume(context.Background(), ebsprovider.ExpandVolumeRequest{
|
||||
Versioned: versioned(), VolumeID: "vol-expand", CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 2 << 30},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(2<<30), vol.CapacityBytes)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 2)
|
||||
assert.Equal(t, []string{"resize", volPath, "2147483648"}, calls[1].args)
|
||||
}
|
||||
|
||||
func TestExpandVolume_ShrinkIsInvalidArgument(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-shrink")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
fr.infoResponses[volPath] = `{"virtual-size":2147483648}`
|
||||
|
||||
_, err := p.ExpandVolume(context.Background(), ebsprovider.ExpandVolumeRequest{
|
||||
Versioned: versioned(), VolumeID: "vol-shrink", CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestExpandVolume_PublishedRefusesOnlineExpansion(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-expand-inuse")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
fr.infoResponses[volPath] = `{"virtual-size":1073741824}`
|
||||
|
||||
_, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-expand-inuse", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = p.ExpandVolume(context.Background(), ebsprovider.ExpandVolumeRequest{
|
||||
Versioned: versioned(), VolumeID: "vol-expand-inuse", CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 2 << 30},
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse)
|
||||
}
|
||||
|
||||
func TestExpandVolume_NotFound(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.ExpandVolume(context.Background(), ebsprovider.ExpandVolumeRequest{
|
||||
Versioned: versioned(), VolumeID: "vol-never-existed", CapacityRange: ebsprovider.CapacityRange{RequiredBytes: 1 << 30},
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
func TestDeleteVolume_Success(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-del")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
|
||||
require.NoError(t, p.DeleteVolume(context.Background(), ebsprovider.DeleteVolumeRequest{Versioned: versioned(), VolumeID: "vol-del"}))
|
||||
_, err := os.Stat(volPath)
|
||||
require.True(t, os.IsNotExist(err))
|
||||
}
|
||||
|
||||
func TestDeleteVolume_AbsentIsIdempotent(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
require.NoError(t, p.DeleteVolume(context.Background(), ebsprovider.DeleteVolumeRequest{Versioned: versioned(), VolumeID: "vol-never-existed"}))
|
||||
// A second delete of the same, already-absent target must also succeed.
|
||||
require.NoError(t, p.DeleteVolume(context.Background(), ebsprovider.DeleteVolumeRequest{Versioned: versioned(), VolumeID: "vol-never-existed"}))
|
||||
}
|
||||
|
||||
func TestDeleteVolume_PublishedRefuses(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-del-inuse")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
fr.infoResponses[volPath] = `{"virtual-size":1073741824}`
|
||||
|
||||
_, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-del-inuse", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
err = p.DeleteVolume(context.Background(), ebsprovider.DeleteVolumeRequest{Versioned: versioned(), VolumeID: "vol-del-inuse"})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse)
|
||||
}
|
||||
|
||||
func TestCreateSnapshot_Argv(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-snap-src")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
|
||||
snap, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-1", VolumeID: "vol-snap-src"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, ebsprovider.SnapshotStateCompleted, snap.State)
|
||||
assert.Equal(t, "vol-snap-src", snap.SourceVolumeID)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
snapPath := p.snapshotPath("snap-1")
|
||||
require.Len(t, calls, 2)
|
||||
assert.Equal(t, []string{"convert", "-O", "qcow2", volPath, snapPath}, calls[0].args)
|
||||
assert.Equal(t, []string{"info", "--output=json", "--force-share", snapPath}, calls[1].args)
|
||||
}
|
||||
|
||||
func TestCreateSnapshot_NotFoundSourceVolume(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-orphan", VolumeID: "vol-never-existed"})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
func TestCreateSnapshot_IdempotentSameVolume(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-snap-idem")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
|
||||
first, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-idem", VolumeID: "vol-snap-idem"})
|
||||
require.NoError(t, err)
|
||||
second, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-idem", VolumeID: "vol-snap-idem"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, first.SourceVolumeID, second.SourceVolumeID)
|
||||
}
|
||||
|
||||
func TestCreateSnapshot_ConflictingSourceVolumeIsAlreadyExists(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volA := p.volumePath("vol-snap-a")
|
||||
volB := p.volumePath("vol-snap-b")
|
||||
require.NoError(t, os.WriteFile(volA, nil, 0o640))
|
||||
require.NoError(t, os.WriteFile(volB, nil, 0o640))
|
||||
|
||||
_, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-conflict", VolumeID: "vol-snap-a"})
|
||||
require.NoError(t, err)
|
||||
_, err = p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-conflict", VolumeID: "vol-snap-b"})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrAlreadyExists)
|
||||
}
|
||||
|
||||
func TestDeleteSnapshot_Success(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-delsnap-src")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
_, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-delete-ok", VolumeID: "vol-delsnap-src"})
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, p.DeleteSnapshot(context.Background(), ebsprovider.DeleteSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-delete-ok"}))
|
||||
_, err = os.Stat(p.snapshotPath("snap-delete-ok"))
|
||||
require.True(t, os.IsNotExist(err))
|
||||
}
|
||||
|
||||
func TestDeleteSnapshot_AbsentIsIdempotent(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
require.NoError(t, p.DeleteSnapshot(context.Background(), ebsprovider.DeleteSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-never-existed"}))
|
||||
require.NoError(t, p.DeleteSnapshot(context.Background(), ebsprovider.DeleteSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-never-existed"}))
|
||||
}
|
||||
|
||||
func TestDeleteSnapshot_RefusedWhenBackingFileInUse(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
snapPath := p.snapshotPath("snap-backing")
|
||||
require.NoError(t, os.WriteFile(snapPath, nil, 0o640))
|
||||
// The snapshotInUse scan lists every file under volumes/ and checks its
|
||||
// reported backing-filename, so a bare marker file with a canned info
|
||||
// response is enough to simulate a CoW clone without a real qemu-img.
|
||||
clonePath := p.volumePath("vol-clone")
|
||||
require.NoError(t, os.WriteFile(clonePath, nil, 0o640))
|
||||
fr.infoResponses[clonePath] = `{"virtual-size":1073741824,"backing-filename":"` + snapPath + `"}`
|
||||
|
||||
err := p.DeleteSnapshot(context.Background(), ebsprovider.DeleteSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-backing"})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse)
|
||||
_, statErr := os.Stat(snapPath)
|
||||
require.NoError(t, statErr, "a refused delete must leave the snapshot file in place")
|
||||
}
|
||||
|
||||
func TestCopySnapshot_Success(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-copysnap-src")
|
||||
require.NoError(t, os.WriteFile(volPath, []byte("volume-bytes"), 0o640))
|
||||
_, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-copysnap-src", VolumeID: "vol-copysnap-src"})
|
||||
require.NoError(t, err)
|
||||
|
||||
copied, err := p.CopySnapshot(context.Background(), ebsprovider.CopySnapshotRequest{
|
||||
Versioned: versioned(), SourceSnapshotID: "snap-copysnap-src", DestinationSnapshotID: "snap-copysnap-dst", VolumeID: "vol-copysnap-src",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "snap-copysnap-dst", copied.ID)
|
||||
assert.Equal(t, "vol-copysnap-src", copied.SourceVolumeID)
|
||||
|
||||
srcBytes, err := os.ReadFile(p.snapshotPath("snap-copysnap-src"))
|
||||
require.NoError(t, err)
|
||||
dstBytes, err := os.ReadFile(p.snapshotPath("snap-copysnap-dst"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, srcBytes, dstBytes)
|
||||
}
|
||||
|
||||
func TestCopySnapshot_NotFoundSource(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CopySnapshot(context.Background(), ebsprovider.CopySnapshotRequest{
|
||||
Versioned: versioned(), SourceSnapshotID: "snap-missing", DestinationSnapshotID: "snap-dst", VolumeID: "vol-a",
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
func TestCopySnapshot_AlreadyExistsDestination(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-copysnap-conflict")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
_, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-a", VolumeID: "vol-copysnap-conflict"})
|
||||
require.NoError(t, err)
|
||||
_, err = p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-b", VolumeID: "vol-copysnap-conflict"})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = p.CopySnapshot(context.Background(), ebsprovider.CopySnapshotRequest{
|
||||
Versioned: versioned(), SourceSnapshotID: "snap-a", DestinationSnapshotID: "snap-b", VolumeID: "vol-copysnap-conflict",
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrAlreadyExists)
|
||||
}
|
||||
|
||||
func TestCopySnapshot_InvalidArgumentWhenIDsMatch(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.CopySnapshot(context.Background(), ebsprovider.CopySnapshotRequest{
|
||||
Versioned: versioned(), SourceSnapshotID: "snap-same", DestinationSnapshotID: "snap-same", VolumeID: "vol-a",
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestCopySnapshot_InvalidArgumentWrongOwner(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-owner")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
_, err := p.CreateSnapshot(context.Background(), ebsprovider.CreateSnapshotRequest{Versioned: versioned(), SnapshotID: "snap-owned", VolumeID: "vol-owner"})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = p.CopySnapshot(context.Background(), ebsprovider.CopySnapshotRequest{
|
||||
Versioned: versioned(), SourceSnapshotID: "snap-owned", DestinationSnapshotID: "snap-owned-dst", VolumeID: "vol-foreign",
|
||||
})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrInvalidArgument)
|
||||
}
|
||||
|
||||
func TestPublishVolume_ArgvReadWrite(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-pub")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
|
||||
pub, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-pub", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "vol-pub", pub.VolumeID)
|
||||
assert.Equal(t, "node-1", pub.NodeID)
|
||||
assert.Equal(t, "nbd+unix:///?socket="+p.socketPath("vol-pub"), pub.NBDURI)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 1)
|
||||
assert.Equal(t, "qemu-nbd", calls[0].name)
|
||||
assert.Equal(t, []string{
|
||||
"--socket", p.socketPath("vol-pub"),
|
||||
"--format", "qcow2",
|
||||
"--persistent",
|
||||
"--shared=1",
|
||||
"--fork",
|
||||
"--pid-file", p.pidFilePath("vol-pub"),
|
||||
volPath,
|
||||
}, calls[0].args)
|
||||
assert.NotContains(t, calls[0].args, "-r")
|
||||
}
|
||||
|
||||
func TestPublishVolume_ReadOnlyAddsFlag(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-pub-ro")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
|
||||
_, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-pub-ro", NodeID: "node-1", ReadOnly: true})
|
||||
require.NoError(t, err)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 1)
|
||||
assert.Contains(t, calls[0].args, "-r")
|
||||
// -r must precede the trailing volume path argument.
|
||||
assert.Equal(t, "-r", calls[0].args[len(calls[0].args)-2])
|
||||
}
|
||||
|
||||
func TestPublishVolume_IdempotentRepublishSameNode(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-pub-idem")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
|
||||
first, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-pub-idem", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
second, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-pub-idem", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, first, second)
|
||||
|
||||
calls := fr.callsSnapshot()
|
||||
require.Len(t, calls, 1, "a republish to the same node must not start a second qemu-nbd")
|
||||
}
|
||||
|
||||
func TestPublishVolume_ConflictDifferentNode(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-pub-conflict")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
|
||||
_, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-pub-conflict", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
_, err = p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-pub-conflict", NodeID: "node-2"})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrVolumeInUse)
|
||||
}
|
||||
|
||||
func TestPublishVolume_NotFound(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
_, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-never-existed", NodeID: "node-1"})
|
||||
require.ErrorIs(t, err, ebsprovider.ErrNotFound)
|
||||
}
|
||||
|
||||
func TestUnpublishVolume_Success(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-unpub")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
_, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-unpub", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, p.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-unpub", NodeID: "node-1"}))
|
||||
_, err = os.Stat(p.socketPath("vol-unpub"))
|
||||
require.True(t, os.IsNotExist(err))
|
||||
}
|
||||
|
||||
func TestUnpublishVolume_AbsentIsIdempotent(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
require.NoError(t, p.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-never-existed", NodeID: "node-1"}))
|
||||
}
|
||||
|
||||
func TestUnpublishVolume_RepeatIsIdempotent(t *testing.T) {
|
||||
p, fr := newTestProvider(t)
|
||||
volPath := p.volumePath("vol-unpub-repeat")
|
||||
require.NoError(t, os.WriteFile(volPath, nil, 0o640))
|
||||
_, err := p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-unpub-repeat", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, p.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-unpub-repeat", NodeID: "node-1"}))
|
||||
require.NoError(t, p.UnpublishVolume(context.Background(), ebsprovider.UnpublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-unpub-repeat", NodeID: "node-1"}))
|
||||
|
||||
// Republishing after unpublish must start a fresh qemu-nbd.
|
||||
_, err = p.PublishVolume(context.Background(), ebsprovider.PublishVolumeRequest{Versioned: versioned(), VolumeID: "vol-unpub-repeat", NodeID: "node-1"})
|
||||
require.NoError(t, err)
|
||||
calls := fr.callsSnapshot()
|
||||
nbdCalls := 0
|
||||
for _, c := range calls {
|
||||
if c.name == "qemu-nbd" {
|
||||
nbdCalls++
|
||||
}
|
||||
}
|
||||
assert.Equal(t, 2, nbdCalls)
|
||||
}
|
||||
|
||||
func TestGetCapabilities_Fixed(t *testing.T) {
|
||||
p, _ := newTestProvider(t)
|
||||
resp, err := p.GetCapabilities(context.Background(), ebsprovider.GetCapabilitiesRequest{Versioned: versioned()})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, capabilities, resp.Capabilities)
|
||||
assert.True(t, resp.Capabilities.CrashConsistentSnapshot)
|
||||
assert.True(t, resp.Capabilities.VolumeSeeding)
|
||||
assert.True(t, resp.Capabilities.SparseExtentReporting, "qemu-nbd advertises base:allocation")
|
||||
assert.True(t, resp.Capabilities.ReadOnlyPublish, "PublishVolume passes -r to qemu-nbd")
|
||||
assert.False(t, resp.Capabilities.OnlineExpansion)
|
||||
assert.False(t, resp.Capabilities.OwnerRouting)
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
// Package qemunbdd implements ebsprovider.EBSProvider over qcow2 files and
|
||||
// qemu-nbd. It shares no code with viperblock: every operation shells out to
|
||||
// qemu-img, qemu-io or qemu-nbd, and a volume's only state is its file.
|
||||
package qemunbdd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// runner executes a host command and returns its combined output. Every
|
||||
// qemu-img, qemu-io and qemu-nbd invocation goes through this seam so unit
|
||||
// tests can assert argv without a qemu installation.
|
||||
type runner interface {
|
||||
Run(ctx context.Context, name string, args ...string) ([]byte, error)
|
||||
}
|
||||
|
||||
// execRunner is the production runner, shelling out via os/exec.
|
||||
type execRunner struct{}
|
||||
|
||||
func (execRunner) Run(ctx context.Context, name string, args ...string) ([]byte, error) {
|
||||
cmd := exec.CommandContext(ctx, name, args...)
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("%s %s: %w: %s", name, strings.Join(args, " "), err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package qemunbdd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/admin"
|
||||
"github.com/mulgadc/spinifex/spinifex/ebsprovider/natsserve"
|
||||
"github.com/mulgadc/spinifex/spinifex/utils"
|
||||
)
|
||||
|
||||
// serviceName roots this daemon's PID file: baseDir/qemunbd.pid.
|
||||
var serviceName = "qemunbd"
|
||||
|
||||
// Config is the qemunbdd service's startup configuration, mirroring
|
||||
// viperblockd.Config's NATS and base-directory fields. NodeName scopes the
|
||||
// natsserve PublishVolume/UnpublishVolume subjects to this node.
|
||||
type Config struct {
|
||||
BaseDir string
|
||||
NatsHost string
|
||||
NatsToken string
|
||||
NatsCACert string
|
||||
NodeName string
|
||||
Debug bool
|
||||
}
|
||||
|
||||
// Service runs the qcow2/qemu-nbd EBSProvider behind natsserve. It and
|
||||
// viperblockd both answer ebs.provider.v1.*, so only one of the two may be
|
||||
// pointed at the same NATS cluster at a time.
|
||||
type Service struct {
|
||||
Config *Config
|
||||
}
|
||||
|
||||
func New(config any) (svc *Service, err error) {
|
||||
cfg, ok := config.(*Config)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("invalid config type for qemunbdd service")
|
||||
}
|
||||
return &Service{Config: cfg}, nil
|
||||
}
|
||||
|
||||
func (svc *Service) Start() (int, error) {
|
||||
if err := utils.WritePidFileTo(svc.Config.BaseDir, serviceName, os.Getpid()); err != nil {
|
||||
return 0, fmt.Errorf("write pid file: %w", err)
|
||||
}
|
||||
if err := launchService(svc.Config); err != nil {
|
||||
slog.Error("Failed to launch service", "err", err)
|
||||
return 0, err
|
||||
}
|
||||
return os.Getpid(), nil
|
||||
}
|
||||
|
||||
func (svc *Service) Stop() (err error) {
|
||||
return utils.StopProcessAt(svc.Config.BaseDir, serviceName)
|
||||
}
|
||||
|
||||
func (svc *Service) Status() (string, error) {
|
||||
return utils.ServiceStatus(svc.Config.BaseDir, serviceName)
|
||||
}
|
||||
|
||||
func (svc *Service) Shutdown() (err error) {
|
||||
return svc.Stop()
|
||||
}
|
||||
|
||||
func (svc *Service) Reload() (err error) {
|
||||
return nil
|
||||
}
|
||||
|
||||
// launchService connects to NATS, roots a qcow2 provider at cfg.BaseDir and
|
||||
// serves ebs.provider.v1.* until SIGINT/SIGTERM, then unsubscribes. It blocks
|
||||
// for the life of the process.
|
||||
func launchService(cfg *Config) error {
|
||||
nc, err := utils.ConnectNATSWithRetry(admin.DialTarget(cfg.NatsHost), cfg.NatsToken, cfg.NatsCACert)
|
||||
if err != nil {
|
||||
return fmt.Errorf("connect to NATS: %w", err)
|
||||
}
|
||||
defer nc.Close()
|
||||
|
||||
provider, err := NewProvider(cfg.BaseDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("construct qemunbd provider: %w", err)
|
||||
}
|
||||
|
||||
stop, err := natsserve.Serve(context.Background(), nc, provider, natsserve.Options{NodeID: cfg.NodeName})
|
||||
if err != nil {
|
||||
return fmt.Errorf("serve ebs.provider.v1: %w", err)
|
||||
}
|
||||
|
||||
slog.Info("qemunbdd: waiting for EBS provider events", "node", cfg.NodeName)
|
||||
|
||||
sigChan := make(chan os.Signal, 1)
|
||||
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
|
||||
<-sigChan
|
||||
slog.Info("qemunbdd: shutting down gracefully...")
|
||||
stop()
|
||||
|
||||
return nil
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user