mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
feat: add EC2-owned platform cleanup capability for the proven-safe subset (#1224)
Add domains/ec2/accountcleanup, an unwired EC2-owned capability that inventories key pairs, placement groups and launch templates for one account, applies their owner delete rules, and proves each disposition (deleted, blocked, failed, residue) by owner re-scan, never by Describe. Cleanup deletes only the proven-safe subset: placement groups with no members and key pairs that no non-terminated instance, open spot request or live launch-template version references, plus unreachable launch-template residue. Live launch templates are always blocked while the EKS nodegroup reference is unresolved. Unknown or unreadable reference state fails closed. Reference readers are injected; no action adapter, NATS subject, IAM or accountteardown wiring is added. Owner delete corrections shared by the public Delete* actions: - DeleteKeyPair keeps metadata when the public-key delete fails, removes leftover public-key objects by name, and fails closed on unreadable or undecodable metadata. - DeleteLaunchTemplate removes all version bodies and the name index before the revision-guarded header delete, and returns InternalError when a step fails. - CreateLaunchTemplateVersion commits through the header revision, so a concurrent delete leaves no headerless body. - DeletePlacementGroup is revision-guarded against concurrent reservations and maps store read errors to InternalError. - Placement-group and launch-template deletes clear their tag-index entries.
This commit is contained in:
Josh Siviour
@@ -446,7 +446,7 @@ These residuals were recorded by individual slices and must be resolved once the
|
||||
- `providers/ebs/viperblock` → `runtime/compute/nbd` and `runtime/host/process` (dependency, moved unchanged from `services/viperblockd`): the adapter formats NBD endpoints and manages nbdkit processes through runtime packages. ADR-0001 does not list providers → runtime as forbidden and the layering test does not enforce it (same category as `providers/ebs/conformance` → `runtime/compute/nbd`), but a provider should not depend on node runtime. Clears when NBD endpoint naming has a provider-facing home (a candidate for `contracts/ebs/v1`) and process supervision is injected by the role. The adapter also imports `providers/objectstore`, a provider-to-provider edge that is allowed.
|
||||
- `gateway/ec2.go` (registration): the EC2 dispatch map stays a `GatewayConfig` method in `gateway` and imports every `domains/ec2/awsapi/*` package; `gateway/ec2_idempotency.go` also imports `domains/ec2/awsapi/idem` (`KVBucket`, `TTL`, `TokenAndParams`) for the dispatch-time idempotency wrapper. ADR-0001 makes the domain own its action registration; moving the map waits on the generic ingress registration seam and must not be done as a package move.
|
||||
- `gateway/requestaudit.go`: the outermost request-audit middleware records AWS fields and is updated from four places in the gateway's authentication flow through the gateway's own context key. Moving it would export about seven names used only by that flow, so it stays in `gateway` until the generic ingress registration seam defines how ingress middleware receives authentication results.
|
||||
- `accountteardown` (dependency): its EC2 reapers call `domains/ec2/awsapi/*` action adapters directly, as they did under `gateway/ec2`. The ADR's `accountteardown` current-to-target map row replaces this with domain-owned cleanup capabilities before the package moves. It also holds a read-only consumer of the EC2 platform-inventory contract (`contracts/ec2/v1`, `ec2.v1.PlatformInventory`), served by `domains/ec2/accountinventory`; only the explicit dry run (`Engine.DryRun`) calls it, to log a shadow comparison, and the key-pair, placement-group and launch-template reapers still use the `awsapi` adapters.
|
||||
- `accountteardown` (dependency): its EC2 reapers call `domains/ec2/awsapi/*` action adapters directly, as they did under `gateway/ec2`. The ADR's `accountteardown` current-to-target map row replaces this with domain-owned cleanup capabilities before the package moves. It also holds a read-only consumer of the EC2 platform-inventory contract (`contracts/ec2/v1`, `ec2.v1.PlatformInventory`), served by `domains/ec2/accountinventory`; only the explicit dry run (`Engine.DryRun`) calls it, to log a shadow comparison, and the key-pair, placement-group and launch-template reapers still use the `awsapi` adapters. A domain-owned cleanup capability for the proven-safe subset of those three kinds (key pairs and empty placement groups no instance, spot request or live launch-template version names, and launch-template residue; never live launch templates) exists as `domains/ec2/accountcleanup` (a Go interface with injected owners and reference readers, no NATS subject); nothing calls it yet.
|
||||
- `domains/admission/quota` (dependency, moved from `handlers/quota` unchanged): `live.go` imports `gateway/elbv2` and `handlers/rds` for its live ELBv2 and RDS counters; `records.go` imports `runtime/compute/vm` for the instance-record vCPU sweep; `live.go`, `vcpu.go` and `storage.go` call `domains/ec2/awsapi/{vpc,eip,volume}` describe adapters to count VPCs, subnets, addresses and volumes, as they did under `gateway/ec2`. The ADR's `handlers/quota` current-to-target map row replaces these with reader capabilities once their final locations exist.
|
||||
- `domains/ec2/awsapi/instance` (dependency): imports `handlers/iam` (`IAMService` in `DescribeInstances.go`, `RunInstances.go` and `IamInstanceProfileAssociation.go`; `InstanceProfile` in the latter two), `runtime/compute/cache` (`NodeState`, `NodeUnknown`, `NodeLive`) and `runtime/compute/vm` (`VM`), both in `status_synthesis.go`. The IAM edge needs an IAM capability; the runtime edges are domain-to-runtime and need the instance record and node liveness exposed through an EC2-owned projection.
|
||||
- `domains/ec2/awsapi/spotinstance` (dependency): imports `handlers/iam.IAMService` and `domains/admission/quota.Service` in `RequestSpotInstances.go`.
|
||||
|
||||
@@ -6,12 +6,14 @@ package accountteardown
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
ec2v1 "github.com/mulgadc/spinifex/contracts/ec2/v1"
|
||||
"github.com/mulgadc/spinifex/internal/testkit"
|
||||
"github.com/mulgadc/spinifex/spinifex/domains/ec2/accountinventory"
|
||||
@@ -68,9 +70,13 @@ func TestEC2PlatformInventoryClientReturnsEC2sReport(t *testing.T) {
|
||||
KeyName: aws.String("deploy"), PublicKeyMaterial: []byte("ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
o.store.failDelete = func(key string) bool { return key == "keys/"+testAccountID+"/deploy" }
|
||||
_, err = o.keys.DeleteKeyPair(ctx, &ec2.DeleteKeyPairInput{KeyName: aws.String("deploy")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
// A public key without metadata, as an older release's failed delete left.
|
||||
_, err = o.store.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(platformBucket), Key: aws.String("keys/" + testAccountID + "/deploy"), Body: strings.NewReader("ssh-ed25519 AAAA"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
reply := newEC2PlatformInventoryClient(o.nc).PlatformInventory(ctx, testAccountID, "req-1")
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
ec2v1 "github.com/mulgadc/spinifex/contracts/ec2/v1"
|
||||
"github.com/mulgadc/spinifex/internal/testkit"
|
||||
"github.com/mulgadc/spinifex/spinifex/domains/ec2/accountinventory"
|
||||
@@ -455,9 +456,13 @@ func TestDryRunShadowOverTheOwnerStores(t *testing.T) {
|
||||
_, err = o.keys.ImportKeyPair(ctx, &ec2.ImportKeyPairInput{KeyName: aws.String(name), PublicKeyMaterial: material}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
o.store.failDelete = func(key string) bool { return key == "keys/"+testAccountID+"/secret-orphan-key" }
|
||||
_, err = o.keys.DeleteKeyPair(ctx, &ec2.DeleteKeyPairInput{KeyName: aws.String("secret-orphan-key")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
// Residue an older release's failed deletes left behind.
|
||||
_, err = o.store.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(platformBucket), Key: aws.String("keys/" + testAccountID + "/secret-orphan-key"), Body: bytes.NewReader(material),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = o.groups.CreatePlacementGroup(ctx, &ec2.CreatePlacementGroupInput{
|
||||
GroupName: aws.String("secret-group-name"), Strategy: aws.String(ec2.PlacementStrategySpread),
|
||||
@@ -480,6 +485,11 @@ func TestDryRunShadowOverTheOwnerStores(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
_, err = o.groups.DeletePlacementGroup(ctx, &ec2.DeletePlacementGroupInput{GroupName: aws.String("secret-doomed-group")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
_, err = o.tags.CreateTags(ctx, &ec2.CreateTagsInput{
|
||||
Resources: []*string{aws.String(doomedID)},
|
||||
Tags: []*ec2.Tag{{Key: aws.String("secret-tag-key"), Value: aws.String("secret-tag-value")}},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
engine, logs := shadowEngine(newEC2PlatformInventoryClient(o.nc), platformReapers(o.nc)...)
|
||||
result, err := engine.DryRun(ctx, testAccountID, "req-owners")
|
||||
|
||||
@@ -34,27 +34,11 @@ import (
|
||||
|
||||
const platformBucket = "test-bucket"
|
||||
|
||||
// deleteFailingStore fails object deletes its predicate selects.
|
||||
type deleteFailingStore struct {
|
||||
*objectstore.MemoryObjectStore
|
||||
|
||||
failDelete func(key string) bool
|
||||
}
|
||||
|
||||
var _ objectstore.ObjectStore = (*deleteFailingStore)(nil)
|
||||
|
||||
func (s *deleteFailingStore) DeleteObject(ctx context.Context, in *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error) {
|
||||
if s.failDelete != nil && s.failDelete(aws.StringValue(in.Key)) {
|
||||
return nil, errors.New("injected store failure")
|
||||
}
|
||||
return s.MemoryObjectStore.DeleteObject(ctx, in)
|
||||
}
|
||||
|
||||
// platformOwners runs the real key, placement-group, launch-template and tag
|
||||
// services behind the subjects the platform reapers call.
|
||||
type platformOwners struct {
|
||||
nc *nats.Conn
|
||||
store *deleteFailingStore
|
||||
store *objectstore.MemoryObjectStore
|
||||
keys *ec2key.KeyServiceImpl
|
||||
groups *ec2placementgroup.PlacementGroupServiceImpl
|
||||
templates *ec2launchtemplate.LaunchTemplateServiceImpl
|
||||
@@ -81,7 +65,7 @@ func newPlatformOwners(t *testing.T) *platformOwners {
|
||||
t.Helper()
|
||||
_, nc, js := testutil.StartTestJetStream(t)
|
||||
ctx := t.Context()
|
||||
o := &platformOwners{nc: nc, store: &deleteFailingStore{MemoryObjectStore: objectstore.NewMemoryObjectStore()}}
|
||||
o := &platformOwners{nc: nc, store: objectstore.NewMemoryObjectStore()}
|
||||
|
||||
var err error
|
||||
o.tagsKV, err = ec2tags.GetOrCreateTagsBucket(ctx, js)
|
||||
@@ -92,8 +76,10 @@ func newPlatformOwners(t *testing.T) *platformOwners {
|
||||
o.keys.SetCentralTagStore(o.tags)
|
||||
o.groups, err = ec2placementgroup.NewPlacementGroupServiceImplWithNATS(ctx, nil, nc)
|
||||
require.NoError(t, err)
|
||||
o.groups.SetCentralTagStore(o.tags)
|
||||
o.templates, err = ec2launchtemplate.NewLaunchTemplateServiceImplWithNATS(ctx, nil, nc)
|
||||
require.NoError(t, err)
|
||||
o.templates.SetCentralTagStore(o.tags)
|
||||
o.pgKV, err = js.KeyValue(ctx, ec2placementgroup.KVBucketPlacementGroups)
|
||||
require.NoError(t, err)
|
||||
o.ltKV, err = js.KeyValue(ctx, ec2launchtemplate.KVBucketLaunchTemplates)
|
||||
@@ -172,9 +158,18 @@ func TestCurrentBehaviour_AccountTermination_PlatformReapersReportDrainedOverHid
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Residue each owner's Describe skips.
|
||||
publicKey := "keys/" + testAccountID + "/deploy"
|
||||
o.store.failDelete = func(key string) bool { return key == publicKey }
|
||||
// Residue each owner's Describe skips, as left by older releases or a
|
||||
// crash between writes.
|
||||
publicKey := "keys/" + testAccountID + "/old-deploy"
|
||||
_, err = o.store.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(platformBucket), Key: aws.String(publicKey), Body: strings.NewReader("ssh-ed25519 AAAA"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
staleTagged := "pg-0stale000000000"
|
||||
_, err = o.tags.CreateTags(ctx, &ec2.CreateTagsInput{
|
||||
Resources: []*string{aws.String(staleTagged)}, Tags: []*ec2.Tag{{Key: aws.String("team"), Value: aws.String("blue")}},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
_, err = o.store.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(platformBucket), Key: aws.String("keys/" + testAccountID + "/key-0badbadbadbadbad0.json"),
|
||||
Body: strings.NewReader("{not json"),
|
||||
@@ -210,10 +205,6 @@ func TestCurrentBehaviour_AccountTermination_PlatformReapersReportDrainedOverHid
|
||||
assert.ElementsMatch(t, []string{publicKey, "keys/" + testAccountID + "/key-0badbadbadbadbad0.json"}, o.objectKeys(t))
|
||||
assert.Equal(t, []string{testAccountID + ".broken"}, accountKVKeys(t, o.pgKV))
|
||||
assert.Equal(t, []string{orphanBody}, accountKVKeys(t, o.ltKV))
|
||||
assert.Equal(t, sortedStrings(testAccountID+"."+pgID, testAccountID+"."+ltID), accountKVKeys(t, o.tagsKV))
|
||||
}
|
||||
|
||||
func sortedStrings(s ...string) []string {
|
||||
slices.Sort(s)
|
||||
return s
|
||||
// The owner deletes clear their own tags; only the stale entry remains.
|
||||
assert.Equal(t, []string{testAccountID + "." + staleTagged}, accountKVKeys(t, o.tagsKV))
|
||||
}
|
||||
|
||||
@@ -1825,9 +1825,6 @@ func (d *Daemon) startCluster() error {
|
||||
return fmt.Errorf("failed to get tags KV bucket: %w", err)
|
||||
}
|
||||
d.tagsService = ec2tags.NewTagsServiceImpl(d.config, tagsKV)
|
||||
// Key pairs keep their creation tags in their own metadata; project them so
|
||||
// describe-tags sees them, and clear them when the key pair is deleted.
|
||||
d.keyService.SetCentralTagStore(d.tagsService)
|
||||
|
||||
d.eigwService, err = initServiceWithRetry("EIGW service", func() (*ec2eigw.EgressOnlyIGWServiceImpl, error) {
|
||||
return ec2eigw.NewEgressOnlyIGWServiceImplWithNATS(d.ctx, d.config, d.natsConn)
|
||||
@@ -1856,6 +1853,7 @@ func (d *Daemon) startCluster() error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to initialize launch template service: %w", err)
|
||||
}
|
||||
d.wirePlatformTagIndex()
|
||||
|
||||
d.spotInstanceService, err = initServiceWithRetry("spot instance service", func() (*ec2spotinstance.SpotInstanceServiceImpl, error) {
|
||||
return ec2spotinstance.NewSpotInstanceServiceImplWithNATS(d.ctx, d.config, d.natsConn)
|
||||
@@ -3628,3 +3626,12 @@ func probeGPU() gpuProbeResult {
|
||||
r.Capable = len(r.Devices) > 0 && ((r.IOMMUActive && r.VFIOPresent) || hasMIG)
|
||||
return r
|
||||
}
|
||||
|
||||
// wirePlatformTagIndex gives the key-pair, placement-group and launch-template
|
||||
// owners the tag index their deletes clear; key pairs also project their
|
||||
// metadata-held creation tags into it so describe-tags sees them.
|
||||
func (d *Daemon) wirePlatformTagIndex() {
|
||||
d.keyService.SetCentralTagStore(d.tagsService)
|
||||
d.placementGroupService.SetCentralTagStore(d.tagsService)
|
||||
d.launchTemplateService.SetCentralTagStore(d.tagsService)
|
||||
}
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -56,11 +55,11 @@ func newDisclosureCluster(t *testing.T) *disclosureCluster {
|
||||
require.NoError(t, err)
|
||||
d.keyService = ec2key.NewKeyServiceImplWithStore(store, disclosureBucket)
|
||||
d.tagsService = ec2tags.NewTagsServiceImplWithStore(cfg, store, tagsKV)
|
||||
d.keyService.SetCentralTagStore(d.tagsService)
|
||||
d.placementGroupService, err = ec2placementgroup.NewPlacementGroupServiceImplWithNATS(t.Context(), nil, nc)
|
||||
require.NoError(t, err)
|
||||
d.launchTemplateService, err = ec2launchtemplate.NewLaunchTemplateServiceImplWithNATS(t.Context(), nil, nc)
|
||||
require.NoError(t, err)
|
||||
d.wirePlatformTagIndex()
|
||||
require.NoError(t, d.subscribeAll())
|
||||
|
||||
caller, err := nats.Connect(ns.ClientURL())
|
||||
@@ -225,12 +224,13 @@ func TestCurrentBehaviour_AccountTermination_DirectNATSCallerCannotSeeHiddenResi
|
||||
assert.Equal(t, before.taggedResources, after.taggedResources)
|
||||
}
|
||||
|
||||
// Pins current behaviour that differs from AWS: deleting a placement group or
|
||||
// launch template leaves its tag-index entry, which DescribeTags keeps
|
||||
// returning with its values. A key-pair delete clears its entry.
|
||||
func TestCurrentBehaviour_AccountTermination_PlatformDeletesLeaveTagIndexEntries(t *testing.T) {
|
||||
// Deleting a key pair, placement group or launch template clears its
|
||||
// tag-index entry, so DescribeTags no longer returns its rows, as in AWS.
|
||||
func TestPlatformDeletesClearTagIndexEntries(t *testing.T) {
|
||||
c := newDisclosureCluster(t)
|
||||
s := c.seed(t, disclosureAccount, "a")
|
||||
before := c.describe(t, disclosureAccount)
|
||||
require.Equal(t, sortedIDs(s.keyID, s.groupID, s.templateID), before.taggedResources)
|
||||
|
||||
c.call(t, "ec2.DeleteKeyPair", disclosureAccount, &ec2.DeleteKeyPairInput{KeyName: aws.String(s.keyName)}, nil)
|
||||
c.call(t, "ec2.DeletePlacementGroup", disclosureAccount, &ec2.DeletePlacementGroupInput{GroupName: aws.String(s.groupName)}, nil)
|
||||
@@ -240,13 +240,8 @@ func TestCurrentBehaviour_AccountTermination_PlatformDeletesLeaveTagIndexEntries
|
||||
assert.Empty(t, got.keyPairs)
|
||||
assert.Empty(t, got.placementGroups)
|
||||
assert.Empty(t, got.launchTemplates)
|
||||
assert.Equal(t, sortedIDs(s.groupID, s.templateID), got.taggedResources)
|
||||
|
||||
assert.Empty(t, got.taggedResources)
|
||||
var tags ec2.DescribeTagsOutput
|
||||
c.call(t, "ec2.DescribeTags", disclosureAccount, &ec2.DescribeTagsInput{}, &tags)
|
||||
for _, row := range tags.Tags {
|
||||
assert.Equal(t, "owner", aws.StringValue(row.Key))
|
||||
assert.Equal(t, "a", aws.StringValue(row.Value))
|
||||
assert.True(t, strings.HasPrefix(aws.StringValue(row.ResourceId), "pg-") || strings.HasPrefix(aws.StringValue(row.ResourceId), "lt-"))
|
||||
}
|
||||
assert.Empty(t, tags.Tags)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
// Package accountcleanup deletes the proven-safe subset of one account's EC2
|
||||
// platform resources through the owners' own delete paths: placement groups
|
||||
// with no members that no instance, spot request or live launch-template
|
||||
// version names; key pairs none of those name; and launch-template residue no
|
||||
// header owns. Live launch templates are never deleted: an EKS nodegroup may
|
||||
// name one and EC2 cannot see it. A reference check that cannot complete
|
||||
// deletes nothing of the kinds it guards. Every disposition is proved by an
|
||||
// owner-store re-scan, never by Describe.
|
||||
//
|
||||
// It adds no caller, checks no authority and holds no store or transport of
|
||||
// its own: the owners and reference readers are injected. Whoever binds it
|
||||
// decides who may invoke it. It is not exposed on any NATS subject.
|
||||
package accountcleanup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
ec2v1 "github.com/mulgadc/spinifex/contracts/ec2/v1"
|
||||
ec2key "github.com/mulgadc/spinifex/spinifex/domains/ec2/key"
|
||||
ec2launchtemplate "github.com/mulgadc/spinifex/spinifex/domains/ec2/launchtemplate"
|
||||
ec2placementgroup "github.com/mulgadc/spinifex/spinifex/domains/ec2/placementgroup"
|
||||
ec2tags "github.com/mulgadc/spinifex/spinifex/domains/ec2/tags"
|
||||
awserrors "github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
|
||||
awsidentifiers "github.com/mulgadc/spinifex/spinifex/foundation/aws/identifiers"
|
||||
otelsetup "github.com/mulgadc/spinifex/spinifex/foundation/telemetry"
|
||||
)
|
||||
|
||||
// PlatformCleaner deletes one account's EC2 platform resources and reports
|
||||
// what the owner stores hold afterwards. It is destructive.
|
||||
type PlatformCleaner interface {
|
||||
CleanupPlatform(ctx context.Context, req *Request) (*Result, error)
|
||||
}
|
||||
|
||||
var _ PlatformCleaner = (*Cleanup)(nil)
|
||||
|
||||
// KeyPairOwner is the key-pair owner's account scan and cleanup.
|
||||
type KeyPairOwner interface {
|
||||
ScanAccount(ctx context.Context, accountID string) (*ec2key.AccountScan, error)
|
||||
CleanupAccount(ctx context.Context, accountID string, settledBefore time.Time, referenced func(keyName string) string) (*ec2key.AccountCleanup, error)
|
||||
}
|
||||
|
||||
// PlacementGroupOwner is the placement-group owner's account scan and cleanup.
|
||||
type PlacementGroupOwner interface {
|
||||
ScanAccount(ctx context.Context, accountID string) (*ec2placementgroup.AccountScan, error)
|
||||
CleanupAccount(ctx context.Context, accountID string, referenced func(groupID, groupName string) string) (*ec2placementgroup.AccountCleanup, error)
|
||||
}
|
||||
|
||||
// LaunchTemplateOwner is the launch-template owner's account scan and residue
|
||||
// cleanup; live templates are never deleted by account cleanup.
|
||||
type LaunchTemplateOwner interface {
|
||||
ScanAccount(ctx context.Context, accountID string) (*ec2launchtemplate.AccountScan, error)
|
||||
CleanupAccountResidue(ctx context.Context, accountID string, settledBefore time.Time) (*ec2launchtemplate.AccountCleanup, error)
|
||||
}
|
||||
|
||||
// TagIndexOwner lists and clears the account's tag-index entries.
|
||||
type TagIndexOwner interface {
|
||||
ScanAccountResourceIDs(ctx context.Context, accountID string, prefixes ...string) ([]string, error)
|
||||
DeleteAllTags(ctx context.Context, accountID, resourceID string) error
|
||||
}
|
||||
|
||||
var (
|
||||
_ KeyPairOwner = (*ec2key.KeyServiceImpl)(nil)
|
||||
_ PlacementGroupOwner = (*ec2placementgroup.PlacementGroupServiceImpl)(nil)
|
||||
_ LaunchTemplateOwner = (*ec2launchtemplate.LaunchTemplateServiceImpl)(nil)
|
||||
_ TagIndexOwner = (*ec2tags.TagsServiceImpl)(nil)
|
||||
)
|
||||
|
||||
// DefaultSettleWindow is how old headerless residue must be before it is
|
||||
// removed; anything younger may be a create still in flight.
|
||||
const DefaultSettleWindow = 10 * time.Minute
|
||||
|
||||
// Cleanup composes the owner cleanups and re-scans.
|
||||
type Cleanup struct {
|
||||
keys KeyPairOwner
|
||||
groups PlacementGroupOwner
|
||||
templates LaunchTemplateOwner
|
||||
tags TagIndexOwner
|
||||
refs ReferenceReaders
|
||||
|
||||
// SettleWindow overrides DefaultSettleWindow when positive.
|
||||
SettleWindow time.Duration
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// New builds a cleanup over the four owners and the reference readers. A nil
|
||||
// reader counts as a failed check, so the kinds it guards delete nothing.
|
||||
func New(keys KeyPairOwner, groups PlacementGroupOwner, templates LaunchTemplateOwner, tags TagIndexOwner, refs ReferenceReaders) *Cleanup {
|
||||
return &Cleanup{keys: keys, groups: groups, templates: templates, tags: tags, refs: refs, now: time.Now}
|
||||
}
|
||||
|
||||
// Resource-ID prefixes of the three kinds in the tag index.
|
||||
const (
|
||||
keyPairIDPrefix = "key-"
|
||||
placementGroupIDPrefix = "pg-"
|
||||
launchTemplateIDPrefix = "lt-"
|
||||
)
|
||||
|
||||
// CleanupPlatform deletes the account's key pairs, empty placement groups and
|
||||
// launch templates, removes settled residue and orphaned tag-index entries,
|
||||
// then re-scans every owner store to build the result. It is safe to re-run.
|
||||
func (c *Cleanup) CleanupPlatform(ctx context.Context, req *Request) (*Result, error) {
|
||||
if req == nil || req.AccountID == "" {
|
||||
return nil, awserrors.Errorf(awserrors.ErrorMissingParameter, "The request must specify an account ID.")
|
||||
}
|
||||
if !awsidentifiers.IsAccountID(req.AccountID) {
|
||||
return nil, awserrors.Errorf(awserrors.ErrorInvalidParameterValue, "The account ID must be 12 digits.")
|
||||
}
|
||||
start := time.Now()
|
||||
window := c.SettleWindow
|
||||
if window <= 0 {
|
||||
window = DefaultSettleWindow
|
||||
}
|
||||
settledBefore := c.now().Add(-window)
|
||||
acct := req.AccountID
|
||||
res := &Result{AccountID: acct, CorrelationRef: req.CorrelationRef}
|
||||
|
||||
refs := c.readReferences(ctx, acct)
|
||||
keyRun := c.cleanKeyPairs(ctx, acct, settledBefore, refs)
|
||||
groupRun := c.cleanPlacementGroups(ctx, acct, refs)
|
||||
templateRun := c.cleanLaunchTemplates(ctx, acct, settledBefore)
|
||||
|
||||
// Tags are listed before the owner re-scans, so a resource created in
|
||||
// between is seen as owned and keeps its tags.
|
||||
tagKind := c.startKind(ec2v1.PlatformKindTagIndex)
|
||||
tagged, tagErr := c.tags.ScanAccountResourceIDs(ctx, acct, keyPairIDPrefix, placementGroupIDPrefix, launchTemplateIDPrefix)
|
||||
if tagErr != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: tag index listing failed", "err", tagErr)
|
||||
tagKind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
}
|
||||
|
||||
owners := map[string]ownedIDs{
|
||||
keyPairIDPrefix: c.finishKeyPairs(ctx, acct, keyRun, res),
|
||||
placementGroupIDPrefix: c.finishPlacementGroups(ctx, acct, groupRun, res),
|
||||
launchTemplateIDPrefix: c.finishLaunchTemplates(ctx, acct, templateRun, res),
|
||||
}
|
||||
res.Kinds = []KindReport{keyRun.kind.report, groupRun.kind.report, templateRun.kind.report}
|
||||
res.Kinds = append(res.Kinds, c.cleanTagIndex(ctx, acct, tagKind, tagged, tagErr == nil, owners, res))
|
||||
|
||||
slog.InfoContext(ctx, "Platform cleanup completed", "accountID", acct, "correlationRef", req.CorrelationRef,
|
||||
"items", len(res.Items), "duration_ms", otelsetup.Millis(time.Since(start)))
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// ownedIDs is the set of IDs whose owner record a re-scan found, live or
|
||||
// unreadable; resolved is false unless the re-scan listed and read every
|
||||
// record, so an absent ID is not proof its tags are orphaned.
|
||||
type ownedIDs struct {
|
||||
ids map[string]bool
|
||||
resolved bool
|
||||
}
|
||||
|
||||
// kindRun accumulates one kind's report across its cleanup and re-scan.
|
||||
type kindRun struct {
|
||||
report KindReport
|
||||
found bool
|
||||
}
|
||||
|
||||
func (c *Cleanup) startKind(kind string) *kindRun {
|
||||
return &kindRun{report: KindReport{Kind: kind, StartedAt: c.now().UTC()}}
|
||||
}
|
||||
|
||||
func (k *kindRun) incomplete(reason string) {
|
||||
if !slices.Contains(k.report.IncompleteReasons, reason) {
|
||||
k.report.IncompleteReasons = append(k.report.IncompleteReasons, reason)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Cleanup) finishKind(k *kindRun) {
|
||||
k.report.EndedAt = c.now().UTC()
|
||||
switch {
|
||||
case len(k.report.IncompleteReasons) > 0:
|
||||
k.report.Status = ec2v1.PlatformStatusScanIncomplete
|
||||
case k.found:
|
||||
k.report.Status = ec2v1.PlatformStatusRecordsFound
|
||||
default:
|
||||
k.report.Status = ec2v1.PlatformStatusObservedClean
|
||||
}
|
||||
}
|
||||
|
||||
// attempts is what an owner said it did, by resource ID. kept holds items
|
||||
// left on purpose, with the fixed reason.
|
||||
type attempts struct {
|
||||
deleted, failed map[string]bool
|
||||
blocked map[string]int
|
||||
kept map[string]string
|
||||
}
|
||||
|
||||
func newAttempts() attempts {
|
||||
return attempts{deleted: map[string]bool{}, failed: map[string]bool{}, blocked: map[string]int{}, kept: map[string]string{}}
|
||||
}
|
||||
|
||||
// itemDisposition decides one ID from what the owner did and whether the
|
||||
// re-scan still found it. Only a complete re-scan that misses an ID is deleted;
|
||||
// an item kept on purpose stays blocked unless that re-scan shows it gone.
|
||||
func itemDisposition(id string, did attempts, present, rescanned bool) (string, string) {
|
||||
switch {
|
||||
case did.kept[id] != "" && (present || !rescanned):
|
||||
return DispositionBlocked, did.kept[id]
|
||||
case !rescanned:
|
||||
return DispositionFailed, ReasonAbsenceUnproven
|
||||
case !present:
|
||||
return DispositionDeleted, ""
|
||||
case did.blocked[id] > 0:
|
||||
return DispositionBlocked, ReasonInstanceMembers
|
||||
case did.failed[id]:
|
||||
return DispositionFailed, ReasonOwnerDeleteFailed
|
||||
case did.deleted[id]:
|
||||
return DispositionFailed, ReasonStillPresent
|
||||
default:
|
||||
return DispositionResidue, ReasonAppearedDuringCleanup
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Result) addItem(k *kindRun, id, disposition, reason string, members int) {
|
||||
if disposition != DispositionDeleted {
|
||||
k.found = true
|
||||
}
|
||||
r.Items = append(r.Items, Item{ResourceID: id, Kind: k.report.Kind, Disposition: disposition, Reason: reason, MemberCount: members})
|
||||
}
|
||||
|
||||
func (r *Result) addCount(k *kindRun, class, disposition, reason string, n int) {
|
||||
if n <= 0 {
|
||||
return
|
||||
}
|
||||
if disposition != DispositionDeleted {
|
||||
k.found = true
|
||||
}
|
||||
r.Counts = append(r.Counts, Count{Kind: k.report.Kind, Class: class, Disposition: disposition, Reason: reason, Count: n})
|
||||
}
|
||||
|
||||
// residueCounts reports one headerless class: the proven decrease as deleted
|
||||
// (never more than the owner removed), and what remains split by cause, kept
|
||||
// records first, by reason in a fixed order.
|
||||
func (r *Result) residueCounts(k *kindRun, class string, before, after, removed, failed, unsettled int, kept map[string]int) {
|
||||
r.addCount(k, class, DispositionDeleted, "", min(removed, max(0, before-after)))
|
||||
left := after
|
||||
for _, reason := range keptReasons {
|
||||
n := min(left, kept[reason])
|
||||
r.addCount(k, class, DispositionBlocked, reason, n)
|
||||
left -= n
|
||||
}
|
||||
failedLeft := min(left, failed)
|
||||
unsettledLeft := min(left-failedLeft, unsettled)
|
||||
r.addCount(k, class, DispositionFailed, ReasonOwnerDeleteFailed, failedLeft)
|
||||
r.addCount(k, class, DispositionResidue, ReasonRecentlyWritten, unsettledLeft)
|
||||
r.addCount(k, class, DispositionFailed, ReasonStillPresent, left-failedLeft-unsettledLeft)
|
||||
}
|
||||
|
||||
// keptReasons are the reasons an owner may keep a record, in report order.
|
||||
var keptReasons = []string{ReasonReferenceCheckFailed, ReasonInstanceReference, ReasonSpotRequestReference, ReasonLaunchTemplateReference}
|
||||
|
||||
// sortedUnion returns the distinct IDs of all lists, sorted.
|
||||
func sortedUnion(lists ...[]string) []string {
|
||||
var out []string
|
||||
for _, l := range lists {
|
||||
for _, id := range l {
|
||||
if !slices.Contains(out, id) {
|
||||
out = append(out, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
slices.Sort(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func idPrefix(id string) string {
|
||||
for _, prefix := range []string{keyPairIDPrefix, placementGroupIDPrefix, launchTemplateIDPrefix} {
|
||||
if strings.HasPrefix(id, prefix) {
|
||||
return prefix
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,777 @@
|
||||
package accountcleanup
|
||||
|
||||
//test:in-package — the clock is injected to place residue outside the settle
|
||||
// window without waiting for it.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
ec2v1 "github.com/mulgadc/spinifex/contracts/ec2/v1"
|
||||
testutil "github.com/mulgadc/spinifex/internal/testkit"
|
||||
"github.com/mulgadc/spinifex/spinifex/bootstrap/config"
|
||||
ec2key "github.com/mulgadc/spinifex/spinifex/domains/ec2/key"
|
||||
ec2launchtemplate "github.com/mulgadc/spinifex/spinifex/domains/ec2/launchtemplate"
|
||||
ec2placementgroup "github.com/mulgadc/spinifex/spinifex/domains/ec2/placementgroup"
|
||||
ec2spotinstance "github.com/mulgadc/spinifex/spinifex/domains/ec2/spotinstance"
|
||||
ec2tags "github.com/mulgadc/spinifex/spinifex/domains/ec2/tags"
|
||||
awserrors "github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
|
||||
"github.com/mulgadc/spinifex/spinifex/providers/objectstore"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
const (
|
||||
testAccount = "111122223333"
|
||||
otherAccount = "999988887777"
|
||||
testBucket = "test-bucket"
|
||||
testPubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"
|
||||
memberID = "i-0123456789abcdef0"
|
||||
staleTagID = "pg-0stale000000000"
|
||||
)
|
||||
|
||||
var errInjected = errors.New("injected failure")
|
||||
|
||||
// faultyStore fails object deletes its predicate selects.
|
||||
type faultyStore struct {
|
||||
*objectstore.MemoryObjectStore
|
||||
|
||||
failDelete func(key string) bool
|
||||
}
|
||||
|
||||
var _ objectstore.ObjectStore = (*faultyStore)(nil)
|
||||
|
||||
func (f *faultyStore) DeleteObject(ctx context.Context, in *s3.DeleteObjectInput) (*s3.DeleteObjectOutput, error) {
|
||||
if f.failDelete != nil && f.failDelete(aws.StringValue(in.Key)) {
|
||||
return nil, errInjected
|
||||
}
|
||||
return f.MemoryObjectStore.DeleteObject(ctx, in)
|
||||
}
|
||||
|
||||
// owners are the real EC2 owner services over an embedded JetStream, wired to
|
||||
// the tag index as the daemon wires them.
|
||||
type owners struct {
|
||||
store *faultyStore
|
||||
keys *ec2key.KeyServiceImpl
|
||||
groups *ec2placementgroup.PlacementGroupServiceImpl
|
||||
templates *ec2launchtemplate.LaunchTemplateServiceImpl
|
||||
tags *ec2tags.TagsServiceImpl
|
||||
spot *ec2spotinstance.SpotInstanceServiceImpl
|
||||
instances *fakeInstances
|
||||
js jetstream.JetStream
|
||||
}
|
||||
|
||||
// fakeInstances stands in for the cluster-wide instance listing, which needs
|
||||
// every daemon to answer; its own refusal is covered separately.
|
||||
type fakeInstances struct {
|
||||
refs *References
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeInstances) AccountInstanceReferences(context.Context, string) (*References, error) {
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
return f.refs, nil
|
||||
}
|
||||
|
||||
func newOwners(t *testing.T) *owners {
|
||||
t.Helper()
|
||||
_, nc, js := testutil.StartTestJetStream(t)
|
||||
o := &owners{store: &faultyStore{MemoryObjectStore: objectstore.NewMemoryObjectStore()}}
|
||||
tagsKV, err := ec2tags.GetOrCreateTagsBucket(t.Context(), js)
|
||||
require.NoError(t, err)
|
||||
o.tags = ec2tags.NewTagsServiceImplWithStore(&config.Config{Predastore: config.PredastoreConfig{Bucket: testBucket}}, o.store, tagsKV)
|
||||
o.keys = ec2key.NewKeyServiceImplWithStore(o.store, testBucket)
|
||||
o.keys.SetCentralTagStore(o.tags)
|
||||
o.groups, err = ec2placementgroup.NewPlacementGroupServiceImplWithNATS(t.Context(), nil, nc)
|
||||
require.NoError(t, err)
|
||||
o.groups.SetCentralTagStore(o.tags)
|
||||
o.templates, err = ec2launchtemplate.NewLaunchTemplateServiceImplWithNATS(t.Context(), nil, nc)
|
||||
require.NoError(t, err)
|
||||
o.templates.SetCentralTagStore(o.tags)
|
||||
o.spot, err = ec2spotinstance.NewSpotInstanceServiceImplWithNATS(t.Context(), nil, nc)
|
||||
require.NoError(t, err)
|
||||
o.instances = &fakeInstances{refs: newReferences()}
|
||||
o.js = js
|
||||
return o
|
||||
}
|
||||
|
||||
// readers are the real spot and launch-template checks and the instance fake.
|
||||
func (o *owners) readers() ReferenceReaders {
|
||||
return ReferenceReaders{Instances: o.instances, Spot: o.spot, Templates: o.templates}
|
||||
}
|
||||
|
||||
func (o *owners) newCleanup() *Cleanup {
|
||||
return New(o.keys, o.groups, o.templates, o.tags, o.readers())
|
||||
}
|
||||
|
||||
// cleanup runs over the owners with residue of any age treated as settled.
|
||||
func (o *owners) cleanup(t *testing.T) *Result {
|
||||
t.Helper()
|
||||
return run(t, o.newCleanup())
|
||||
}
|
||||
|
||||
func run(t *testing.T, c *Cleanup) *Result {
|
||||
t.Helper()
|
||||
c.now = func() time.Time { return time.Now().Add(time.Hour) }
|
||||
res, err := c.CleanupPlatform(t.Context(), &Request{AccountID: testAccount, CorrelationRef: "ref-1"})
|
||||
require.NoError(t, err)
|
||||
return res
|
||||
}
|
||||
|
||||
func (o *owners) importKey(t *testing.T, account, name string) string {
|
||||
t.Helper()
|
||||
out, err := o.keys.ImportKeyPair(t.Context(), &ec2.ImportKeyPairInput{
|
||||
KeyName: aws.String(name), PublicKeyMaterial: []byte(testPubKey),
|
||||
TagSpecifications: []*ec2.TagSpecification{{ResourceType: aws.String("key-pair"), Tags: secretTags()}},
|
||||
}, account)
|
||||
require.NoError(t, err)
|
||||
return aws.StringValue(out.KeyPairId)
|
||||
}
|
||||
|
||||
func (o *owners) createGroup(t *testing.T, account, name string) string {
|
||||
t.Helper()
|
||||
out, err := o.groups.CreatePlacementGroup(t.Context(), &ec2.CreatePlacementGroupInput{
|
||||
GroupName: aws.String(name), Strategy: aws.String(ec2.PlacementStrategyCluster),
|
||||
}, account)
|
||||
require.NoError(t, err)
|
||||
id := aws.StringValue(out.PlacementGroup.GroupId)
|
||||
o.tag(t, account, id)
|
||||
return id
|
||||
}
|
||||
|
||||
func (o *owners) addMember(t *testing.T, group string) {
|
||||
t.Helper()
|
||||
reserved, err := o.groups.ReserveClusterNode(t.Context(), &ec2placementgroup.ReserveClusterNodeInput{
|
||||
GroupName: group, EligibleNodes: []string{"node-1"},
|
||||
}, testAccount)
|
||||
require.NoError(t, err)
|
||||
_, err = o.groups.FinalizeClusterInstances(t.Context(), &ec2placementgroup.FinalizeClusterInstancesInput{
|
||||
GroupName: group, NodeInstances: map[string][]string{reserved.TargetNode: {memberID}},
|
||||
}, testAccount)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func (o *owners) removeMember(t *testing.T, group string) {
|
||||
t.Helper()
|
||||
_, err := o.groups.RemoveInstance(t.Context(), &ec2placementgroup.RemoveInstanceInput{
|
||||
GroupName: group, NodeName: "node-1", InstanceID: memberID,
|
||||
}, testAccount)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func (o *owners) createTemplate(t *testing.T, account, name string) string {
|
||||
t.Helper()
|
||||
out, err := o.templates.CreateLaunchTemplate(t.Context(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String(name),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
|
||||
ImageId: aws.String("ami-123"),
|
||||
UserData: aws.String(base64.StdEncoding.EncodeToString([]byte("secret-user-data"))),
|
||||
},
|
||||
}, account)
|
||||
require.NoError(t, err)
|
||||
id := aws.StringValue(out.LaunchTemplate.LaunchTemplateId)
|
||||
o.tag(t, account, id)
|
||||
return id
|
||||
}
|
||||
|
||||
func secretTags() []*ec2.Tag {
|
||||
return []*ec2.Tag{{Key: aws.String("secret-tag-key"), Value: aws.String("secret-tag-value")}}
|
||||
}
|
||||
|
||||
func (o *owners) tag(t *testing.T, account, id string) {
|
||||
t.Helper()
|
||||
_, err := o.tags.CreateTags(t.Context(), &ec2.CreateTagsInput{Resources: []*string{aws.String(id)}, Tags: secretTags()}, account)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func (o *owners) putObject(t *testing.T, key, body string) {
|
||||
t.Helper()
|
||||
_, err := o.store.PutObject(t.Context(), &s3.PutObjectInput{
|
||||
Bucket: aws.String(testBucket), Key: aws.String("keys/" + testAccount + "/" + key), Body: strings.NewReader(body),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func statuses(res *Result) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, k := range res.Kinds {
|
||||
out[k.Kind] = k.Status
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func allStatus(status string) map[string]string {
|
||||
return map[string]string{
|
||||
ec2v1.PlatformKindKeyPair: status, ec2v1.PlatformKindPlacementGroup: status,
|
||||
ec2v1.PlatformKindLaunchTemplate: status, ec2v1.PlatformKindTagIndex: status,
|
||||
}
|
||||
}
|
||||
|
||||
func item(id, kind, disposition, reason string) Item {
|
||||
return Item{ResourceID: id, Kind: kind, Disposition: disposition, Reason: reason}
|
||||
}
|
||||
|
||||
// One run deletes every unreferenced key pair and empty placement group with
|
||||
// its tag-index entry, removes settled hidden residue and orphaned tags,
|
||||
// blocks on a group holding an instance, and leaves another account alone.
|
||||
// The next run, once the member leaves, converges to clean.
|
||||
func TestCleanupPlatformDeletesBlocksAndConverges(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
keyID := o.importKey(t, testAccount, "secret-key-name")
|
||||
emptyID := o.createGroup(t, testAccount, "secret-empty-group")
|
||||
heldID := o.createGroup(t, testAccount, "secret-held-group")
|
||||
o.addMember(t, "secret-held-group")
|
||||
o.putObject(t, "secret-orphan-key", testPubKey)
|
||||
o.tag(t, testAccount, staleTagID)
|
||||
theirKey := o.importKey(t, otherAccount, "theirs")
|
||||
o.createGroup(t, otherAccount, "theirs")
|
||||
o.createTemplate(t, otherAccount, "theirs")
|
||||
|
||||
res := o.cleanup(t)
|
||||
|
||||
assert.Equal(t, testAccount, res.AccountID)
|
||||
assert.Equal(t, "ref-1", res.CorrelationRef)
|
||||
assert.ElementsMatch(t, []Item{
|
||||
item(keyID, ec2v1.PlatformKindKeyPair, DispositionDeleted, ""),
|
||||
item(emptyID, ec2v1.PlatformKindPlacementGroup, DispositionDeleted, ""),
|
||||
{ResourceID: heldID, Kind: ec2v1.PlatformKindPlacementGroup, Disposition: DispositionBlocked, Reason: ReasonInstanceMembers, MemberCount: 1},
|
||||
item(staleTagID, ec2v1.PlatformKindTagIndex, DispositionDeleted, ""),
|
||||
}, res.Items)
|
||||
assert.Equal(t, []Count{{
|
||||
Kind: ec2v1.PlatformKindKeyPair, Class: ec2v1.PlatformClassPublicKeyWithoutMetadata, Disposition: DispositionDeleted, Count: 1,
|
||||
}}, res.Counts)
|
||||
// The held group's own tags are its, not residue.
|
||||
assert.Equal(t, map[string]string{
|
||||
ec2v1.PlatformKindKeyPair: ec2v1.PlatformStatusObservedClean,
|
||||
ec2v1.PlatformKindPlacementGroup: ec2v1.PlatformStatusRecordsFound,
|
||||
ec2v1.PlatformKindLaunchTemplate: ec2v1.PlatformStatusObservedClean,
|
||||
ec2v1.PlatformKindTagIndex: ec2v1.PlatformStatusRecordsFound,
|
||||
}, statuses(res))
|
||||
|
||||
theirs, err := o.keys.ScanAccount(t.Context(), otherAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{theirKey}, theirs.LiveKeyPairIDs)
|
||||
theirTags, err := o.tags.ScanAccountResourceIDs(t.Context(), otherAccount, "key-", "pg-", "lt-")
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, theirTags, 3)
|
||||
|
||||
o.removeMember(t, "secret-held-group")
|
||||
res = o.cleanup(t)
|
||||
assert.Equal(t, []Item{item(heldID, ec2v1.PlatformKindPlacementGroup, DispositionDeleted, "")}, res.Items,
|
||||
"items finished on an earlier run are not reported again")
|
||||
assert.Empty(t, res.Counts)
|
||||
assert.Equal(t, allStatus(ec2v1.PlatformStatusObservedClean), statuses(res))
|
||||
|
||||
res = o.cleanup(t)
|
||||
assert.Empty(t, res.Items)
|
||||
assert.Equal(t, allStatus(ec2v1.PlatformStatusObservedClean), statuses(res))
|
||||
}
|
||||
|
||||
// The result carries IDs, kinds, dispositions, fixed reasons and counts only:
|
||||
// no names, tags, key material, user data, member instance IDs or error text.
|
||||
func TestCleanupPlatformResultDisclosesNoTenantData(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
o.importKey(t, testAccount, "secret-key-name")
|
||||
o.createGroup(t, testAccount, "secret-held-group")
|
||||
o.addMember(t, "secret-held-group")
|
||||
o.createTemplate(t, testAccount, "secret-template-name")
|
||||
o.putObject(t, "secret-orphan-key", testPubKey)
|
||||
o.putObject(t, "key-0badbadbadbadbad0.json", "{secret-broken")
|
||||
failing := o.importKey(t, testAccount, "secret-failing-key")
|
||||
o.store.failDelete = func(key string) bool { return strings.HasSuffix(key, failing+".json") }
|
||||
o.importKey(t, testAccount, "secret-instance-key")
|
||||
o.instances.refs.addKeys("secret-instance-key")
|
||||
o.importKey(t, testAccount, "secret-spot-key")
|
||||
o.openSpotRequest(t, "sir-0secret000000001", "secret-spot-key", "")
|
||||
o.importKey(t, testAccount, "secret-template-key")
|
||||
_, err := o.templates.CreateLaunchTemplate(t.Context(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("secret-referencing-template"),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{KeyName: aws.String("secret-template-key")},
|
||||
}, testAccount)
|
||||
require.NoError(t, err)
|
||||
|
||||
body, err := json.Marshal(o.cleanup(t))
|
||||
require.NoError(t, err)
|
||||
for _, leak := range []string{
|
||||
"secret", "AAAAC3Nz", memberID, "node-1", "injected", "sir-",
|
||||
base64.StdEncoding.EncodeToString([]byte("secret-user-data")),
|
||||
} {
|
||||
assert.NotContains(t, string(body), leak)
|
||||
}
|
||||
assert.Contains(t, string(body), failing, "the IDs themselves are reported")
|
||||
}
|
||||
|
||||
// A delete that fails part way is failed, not deleted; the re-run completes
|
||||
// the partly deleted key pair and reports it deleted exactly once.
|
||||
func TestCleanupPlatformInterruptedRunConverges(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
keyID := o.importKey(t, testAccount, "deploy")
|
||||
o.store.failDelete = func(key string) bool { return strings.HasSuffix(key, ".json") }
|
||||
|
||||
res := o.cleanup(t)
|
||||
assert.Equal(t, []Item{item(keyID, ec2v1.PlatformKindKeyPair, DispositionFailed, ReasonOwnerDeleteFailed)}, res.Items)
|
||||
assert.Equal(t, ec2v1.PlatformStatusRecordsFound, statuses(res)[ec2v1.PlatformKindKeyPair])
|
||||
scan, err := o.keys.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{keyID}, scan.LiveKeyPairIDs, "the metadata anchors the unfinished delete")
|
||||
|
||||
o.store.failDelete = nil
|
||||
res = o.cleanup(t)
|
||||
assert.Equal(t, []Item{item(keyID, ec2v1.PlatformKindKeyPair, DispositionDeleted, "")}, res.Items)
|
||||
assert.Equal(t, allStatus(ec2v1.PlatformStatusObservedClean), statuses(res))
|
||||
}
|
||||
|
||||
// Records the owner cannot read are never deleted and stay reported as residue,
|
||||
// and orphaned tags whose owner may be one of them are left.
|
||||
func TestCleanupPlatformLeavesUnreadableRecords(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
o.putObject(t, "key-0badbadbadbadbad0.json", "{not json")
|
||||
o.putObject(t, "orphan", testPubKey)
|
||||
|
||||
res := o.cleanup(t)
|
||||
assert.Empty(t, res.Items)
|
||||
assert.ElementsMatch(t, []Count{
|
||||
{Kind: ec2v1.PlatformKindKeyPair, Class: ec2v1.PlatformClassUnreadableRecord, Disposition: DispositionResidue, Reason: ReasonUnreadableRecord, Count: 1},
|
||||
{Kind: ec2v1.PlatformKindKeyPair, Class: ec2v1.PlatformClassPublicKeyUnresolved, Disposition: DispositionResidue, Reason: ReasonOwnerRecordsUnresolved, Count: 1},
|
||||
}, res.Counts)
|
||||
assert.Equal(t, ec2v1.PlatformStatusRecordsFound, statuses(res)[ec2v1.PlatformKindKeyPair])
|
||||
scan, err := o.keys.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"key-0badbadbadbadbad0"}, scan.UnreadableKeyPairIDs)
|
||||
}
|
||||
|
||||
// Residue younger than the settle window may be a create in flight: it is
|
||||
// reported, not removed.
|
||||
func TestCleanupPlatformKeepsUnsettledResidue(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
o.putObject(t, "orphan", testPubKey)
|
||||
c := o.newCleanup()
|
||||
|
||||
res, err := c.CleanupPlatform(t.Context(), &Request{AccountID: testAccount})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []Count{{
|
||||
Kind: ec2v1.PlatformKindKeyPair, Class: ec2v1.PlatformClassPublicKeyWithoutMetadata,
|
||||
Disposition: DispositionResidue, Reason: ReasonRecentlyWritten, Count: 1,
|
||||
}}, res.Counts)
|
||||
}
|
||||
|
||||
// lyingKeys reports every key pair deleted without deleting anything.
|
||||
type lyingKeys struct{ *ec2key.KeyServiceImpl }
|
||||
|
||||
func (l lyingKeys) CleanupAccount(ctx context.Context, account string, _ time.Time, _ func(string) string) (*ec2key.AccountCleanup, error) {
|
||||
scan, err := l.ScanAccount(ctx, account)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ec2key.AccountCleanup{DeletedKeyPairIDs: scan.LiveKeyPairIDs}, nil
|
||||
}
|
||||
|
||||
// An owner that reports a delete is not believed: the re-scan still finds the
|
||||
// record, so it is failed and still present.
|
||||
func TestCleanupPlatformProvesDeletesByRescan(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
keyID := o.importKey(t, testAccount, "deploy")
|
||||
|
||||
res := run(t, New(lyingKeys{o.keys}, o.groups, o.templates, o.tags, o.readers()))
|
||||
assert.Contains(t, res.Items, item(keyID, ec2v1.PlatformKindKeyPair, DispositionFailed, ReasonStillPresent))
|
||||
assert.Equal(t, ec2v1.PlatformStatusRecordsFound, statuses(res)[ec2v1.PlatformKindKeyPair])
|
||||
}
|
||||
|
||||
// failingRescan fails every scan after the first, so the cleanup deletes but
|
||||
// cannot prove it.
|
||||
type failingRescan struct {
|
||||
*ec2launchtemplate.LaunchTemplateServiceImpl
|
||||
|
||||
scans int
|
||||
}
|
||||
|
||||
func (f *failingRescan) ScanAccount(ctx context.Context, account string) (*ec2launchtemplate.AccountScan, error) {
|
||||
if f.scans++; f.scans > 1 {
|
||||
return nil, errInjected
|
||||
}
|
||||
return f.LaunchTemplateServiceImpl.ScanAccount(ctx, account)
|
||||
}
|
||||
|
||||
// A failed re-scan is incomplete, never clean: the template stays blocked, and
|
||||
// a tag of that kind no reported resource owns stays, as residue.
|
||||
func TestCleanupPlatformFailedRescanIsIncomplete(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
ltID := o.createTemplate(t, testAccount, "web")
|
||||
o.tag(t, testAccount, "lt-0stale000000000")
|
||||
|
||||
res := run(t, New(o.keys, o.groups, &failingRescan{LaunchTemplateServiceImpl: o.templates}, o.tags, o.readers()))
|
||||
assert.Equal(t, []Item{
|
||||
item(ltID, ec2v1.PlatformKindLaunchTemplate, DispositionBlocked, ReasonReferenceUnknown),
|
||||
item("lt-0stale000000000", ec2v1.PlatformKindTagIndex, DispositionResidue, ReasonOwnerRecordsUnresolved),
|
||||
}, res.Items)
|
||||
assert.Equal(t, ec2v1.PlatformStatusScanIncomplete, statuses(res)[ec2v1.PlatformKindLaunchTemplate])
|
||||
assert.Equal(t, ec2v1.PlatformStatusScanIncomplete, statuses(res)[ec2v1.PlatformKindTagIndex])
|
||||
tagged, err := o.tags.ScanAccountResourceIDs(t.Context(), testAccount, "lt-")
|
||||
require.NoError(t, err)
|
||||
assert.ElementsMatch(t, []string{"lt-0stale000000000", ltID}, tagged)
|
||||
}
|
||||
|
||||
// failingTags fails every tag-index clear.
|
||||
type failingTags struct{ *ec2tags.TagsServiceImpl }
|
||||
|
||||
func (failingTags) DeleteAllTags(context.Context, string, string) error { return errInjected }
|
||||
|
||||
// A tag-index entry that cannot be cleared is failed and the kind keeps
|
||||
// records, never clean.
|
||||
func TestCleanupPlatformReportsUnclearedTags(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
o.tag(t, testAccount, staleTagID)
|
||||
|
||||
res := run(t, New(o.keys, o.groups, o.templates, failingTags{o.tags}, o.readers()))
|
||||
assert.Equal(t, []Item{item(staleTagID, ec2v1.PlatformKindTagIndex, DispositionFailed, ReasonOwnerDeleteFailed)}, res.Items)
|
||||
assert.Equal(t, ec2v1.PlatformStatusRecordsFound, statuses(res)[ec2v1.PlatformKindTagIndex])
|
||||
}
|
||||
|
||||
// failingCleanup fails the owner cleanup's own listing.
|
||||
type failingCleanup struct {
|
||||
*ec2placementgroup.PlacementGroupServiceImpl
|
||||
}
|
||||
|
||||
func (failingCleanup) CleanupAccount(context.Context, string, func(string, string) string) (*ec2placementgroup.AccountCleanup, error) {
|
||||
return nil, errInjected
|
||||
}
|
||||
|
||||
// An owner cleanup that cannot list deletes nothing and is incomplete; the
|
||||
// group it could not reach is still reported, from the re-scan.
|
||||
func TestCleanupPlatformOwnerListingFailureIsIncomplete(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
groupID := o.createGroup(t, testAccount, "web")
|
||||
|
||||
res := run(t, New(o.keys, failingCleanup{o.groups}, o.templates, o.tags, o.readers()))
|
||||
assert.Equal(t, ec2v1.PlatformStatusScanIncomplete, statuses(res)[ec2v1.PlatformKindPlacementGroup])
|
||||
assert.Contains(t, res.Items, item(groupID, ec2v1.PlatformKindPlacementGroup, DispositionResidue, ReasonAppearedDuringCleanup))
|
||||
}
|
||||
|
||||
func TestCleanupPlatformValidatesTheRequest(t *testing.T) {
|
||||
c := New(nil, nil, nil, nil, ReferenceReaders{})
|
||||
for name, tc := range map[string]struct {
|
||||
req *Request
|
||||
code string
|
||||
}{
|
||||
"nil": {nil, awserrors.ErrorMissingParameter},
|
||||
"empty": {&Request{}, awserrors.ErrorMissingParameter},
|
||||
"malformed": {&Request{AccountID: "acct-1"}, awserrors.ErrorInvalidParameterValue},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
_, err := c.CleanupPlatform(t.Context(), tc.req)
|
||||
require.Error(t, err)
|
||||
got, _ := awserrors.ResolveErrorCode(err)
|
||||
assert.Equal(t, tc.code, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (o *owners) openSpotRequest(t *testing.T, id, keyName, groupName string) {
|
||||
t.Helper()
|
||||
spec := &ec2.LaunchSpecification{ImageId: aws.String("ami-123"), KeyName: aws.String(keyName)}
|
||||
if groupName != "" {
|
||||
spec.Placement = &ec2.SpotPlacement{GroupName: aws.String(groupName)}
|
||||
}
|
||||
_, err := o.spot.PutSpotInstanceRequests(t.Context(), &ec2spotinstance.PutSpotRequestsInput{Requests: []*ec2.SpotInstanceRequest{{
|
||||
SpotInstanceRequestId: aws.String(id), State: aws.String(ec2.SpotInstanceStateOpen), LaunchSpecification: spec,
|
||||
}}}, testAccount)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func (o *owners) putTemplateRecord(t *testing.T, key, value string) {
|
||||
t.Helper()
|
||||
kv, err := o.js.KeyValue(t.Context(), ec2launchtemplate.KVBucketLaunchTemplates)
|
||||
require.NoError(t, err)
|
||||
_, err = kv.Put(t.Context(), key, []byte(value))
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// A live launch template is never deleted: an EKS nodegroup may name it and
|
||||
// EC2 cannot read EKS records. It stays blocked, described and tagged on every
|
||||
// run; headerless version bodies and names, which nothing can resolve, are
|
||||
// removed once settled.
|
||||
func TestCleanupPlatformNeverDeletesLiveLaunchTemplates(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
ltID := o.createTemplate(t, testAccount, "web")
|
||||
o.putTemplateRecord(t, testAccount+".lt-0orphan000000000.v1", `{"version_number":1}`)
|
||||
|
||||
for range 2 {
|
||||
res := o.cleanup(t)
|
||||
assert.Equal(t, []Item{item(ltID, ec2v1.PlatformKindLaunchTemplate, DispositionBlocked, ReasonReferenceUnknown)}, res.Items)
|
||||
assert.Equal(t, ec2v1.PlatformStatusRecordsFound, statuses(res)[ec2v1.PlatformKindLaunchTemplate])
|
||||
assert.Equal(t, ec2v1.PlatformStatusRecordsFound, statuses(res)[ec2v1.PlatformKindTagIndex], "the template keeps its tags")
|
||||
}
|
||||
scan, err := o.templates.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{ltID}, scan.LiveTemplateIDs)
|
||||
assert.Zero(t, scan.OrphanVersionBodies)
|
||||
out, err := o.templates.DescribeLaunchTemplateVersions(t.Context(), &ec2.DescribeLaunchTemplateVersionsInput{LaunchTemplateId: aws.String(ltID)}, testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, out.LaunchTemplateVersions, 1, "the version body is untouched")
|
||||
}
|
||||
|
||||
// A key pair a non-terminated instance names is blocked and kept, and so is an
|
||||
// orphan public key the instance names, because IMDS serves it by name. A
|
||||
// placement group a stopped instance names is blocked though it has no members.
|
||||
func TestCleanupPlatformBlocksOnInstanceReferences(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
usedKey := o.importKey(t, testAccount, "in-use")
|
||||
freeKey := o.importKey(t, testAccount, "free")
|
||||
o.putObject(t, "served-orphan", testPubKey)
|
||||
usedGroup := o.createGroup(t, testAccount, "stopped-member")
|
||||
o.instances.refs.addKeys("in-use", "served-orphan")
|
||||
o.instances.refs.addGroups("stopped-member")
|
||||
|
||||
res := o.cleanup(t)
|
||||
assert.ElementsMatch(t, []Item{
|
||||
item(usedKey, ec2v1.PlatformKindKeyPair, DispositionBlocked, ReasonInstanceReference),
|
||||
item(freeKey, ec2v1.PlatformKindKeyPair, DispositionDeleted, ""),
|
||||
item(usedGroup, ec2v1.PlatformKindPlacementGroup, DispositionBlocked, ReasonInstanceReference),
|
||||
}, res.Items)
|
||||
assert.Equal(t, []Count{{
|
||||
Kind: ec2v1.PlatformKindKeyPair, Class: ec2v1.PlatformClassPublicKeyWithoutMetadata,
|
||||
Disposition: DispositionBlocked, Reason: ReasonInstanceReference, Count: 1,
|
||||
}}, res.Counts)
|
||||
scan, err := o.keys.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{usedKey}, scan.LiveKeyPairIDs)
|
||||
assert.Equal(t, 1, scan.OrphanPublicKeys)
|
||||
material, err := o.keys.GetPublicKeyMaterial(testAccount, "served-orphan")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, testPubKey, material)
|
||||
}
|
||||
|
||||
// An open spot request's launch specification blocks the key pair and the
|
||||
// placement group it names until the request is cancelled.
|
||||
func TestCleanupPlatformBlocksOnSpotRequests(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
keyID := o.importKey(t, testAccount, "spot-key")
|
||||
groupID := o.createGroup(t, testAccount, "spot-group")
|
||||
o.openSpotRequest(t, "sir-0000000000000001", "spot-key", "spot-group")
|
||||
|
||||
res := o.cleanup(t)
|
||||
assert.ElementsMatch(t, []Item{
|
||||
item(keyID, ec2v1.PlatformKindKeyPair, DispositionBlocked, ReasonSpotRequestReference),
|
||||
item(groupID, ec2v1.PlatformKindPlacementGroup, DispositionBlocked, ReasonSpotRequestReference),
|
||||
}, res.Items)
|
||||
|
||||
_, err := o.spot.CancelSpotInstanceRequests(t.Context(), &ec2.CancelSpotInstanceRequestsInput{
|
||||
SpotInstanceRequestIds: []*string{aws.String("sir-0000000000000001")},
|
||||
}, testAccount)
|
||||
require.NoError(t, err)
|
||||
res = o.cleanup(t)
|
||||
assert.ElementsMatch(t, []Item{
|
||||
item(keyID, ec2v1.PlatformKindKeyPair, DispositionDeleted, ""),
|
||||
item(groupID, ec2v1.PlatformKindPlacementGroup, DispositionDeleted, ""),
|
||||
}, res.Items)
|
||||
}
|
||||
|
||||
// failingTemplates fails the launch-template reference check.
|
||||
type failingTemplates struct{}
|
||||
|
||||
func (failingTemplates) AccountTemplateReferences(context.Context, string) (*ec2launchtemplate.TemplateReferences, error) {
|
||||
return nil, errInjected
|
||||
}
|
||||
|
||||
// failingSpot fails the spot reference check.
|
||||
type failingSpot struct{}
|
||||
|
||||
func (failingSpot) AccountLaunchReferences(context.Context, string) (*ec2spotinstance.LaunchReferences, error) {
|
||||
return nil, errInjected
|
||||
}
|
||||
|
||||
// A reference check that cannot complete is never read as "no references":
|
||||
// key pairs, orphan public keys and placement groups are all kept, both kinds
|
||||
// are incomplete, and launch-template residue, which no check guards, is
|
||||
// still removed.
|
||||
func TestCleanupPlatformFailedReferenceCheckDeletesNothing(t *testing.T) {
|
||||
for name, build := range map[string]func(o *owners) *Cleanup{
|
||||
"instance listing fails": func(o *owners) *Cleanup {
|
||||
o.instances.err = errInjected
|
||||
return o.newCleanup()
|
||||
},
|
||||
"spot listing fails": func(o *owners) *Cleanup {
|
||||
r := o.readers()
|
||||
r.Spot = failingSpot{}
|
||||
return New(o.keys, o.groups, o.templates, o.tags, r)
|
||||
},
|
||||
"launch-template reading fails": func(o *owners) *Cleanup {
|
||||
r := o.readers()
|
||||
r.Templates = failingTemplates{}
|
||||
return New(o.keys, o.groups, o.templates, o.tags, r)
|
||||
},
|
||||
"instance reader absent": func(o *owners) *Cleanup {
|
||||
r := o.readers()
|
||||
r.Instances = nil
|
||||
return New(o.keys, o.groups, o.templates, o.tags, r)
|
||||
},
|
||||
"launch-template reader absent": func(o *owners) *Cleanup {
|
||||
r := o.readers()
|
||||
r.Templates = nil
|
||||
return New(o.keys, o.groups, o.templates, o.tags, r)
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
keyID := o.importKey(t, testAccount, "deploy")
|
||||
o.putObject(t, "orphan", testPubKey)
|
||||
groupID := o.createGroup(t, testAccount, "web")
|
||||
o.putTemplateRecord(t, testAccount+".lt-0orphan000000000.v1", `{"version_number":1}`)
|
||||
|
||||
res := run(t, build(o))
|
||||
assert.ElementsMatch(t, []Item{
|
||||
item(keyID, ec2v1.PlatformKindKeyPair, DispositionBlocked, ReasonReferenceCheckFailed),
|
||||
item(groupID, ec2v1.PlatformKindPlacementGroup, DispositionBlocked, ReasonReferenceCheckFailed),
|
||||
}, res.Items)
|
||||
assert.ElementsMatch(t, []Count{
|
||||
{Kind: ec2v1.PlatformKindKeyPair, Class: ec2v1.PlatformClassPublicKeyWithoutMetadata, Disposition: DispositionBlocked, Reason: ReasonReferenceCheckFailed, Count: 1},
|
||||
{Kind: ec2v1.PlatformKindLaunchTemplate, Class: ec2v1.PlatformClassVersionBodyWithoutHeader, Disposition: DispositionDeleted, Count: 1},
|
||||
}, res.Counts)
|
||||
for _, k := range res.Kinds {
|
||||
if k.Kind == ec2v1.PlatformKindKeyPair || k.Kind == ec2v1.PlatformKindPlacementGroup {
|
||||
assert.Equal(t, ec2v1.PlatformStatusScanIncomplete, k.Status, k.Kind)
|
||||
assert.Equal(t, []string{ReasonReferenceCheckFailed}, k.IncompleteReasons, k.Kind)
|
||||
}
|
||||
}
|
||||
keys, err := o.keys.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{keyID}, keys.LiveKeyPairIDs)
|
||||
assert.Equal(t, 1, keys.OrphanPublicKeys)
|
||||
groups, err := o.groups.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, groups.Groups, 1)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A key pair or placement group named by any version of a live launch
|
||||
// template, by name or group ID, is blocked: RunInstances may resolve any
|
||||
// version by $Default, $Latest or number. The template itself stays blocked.
|
||||
func TestCleanupPlatformBlocksOnLaunchTemplateReferences(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
usedKey := o.importKey(t, testAccount, "template-key")
|
||||
freeKey := o.importKey(t, testAccount, "free")
|
||||
byName := o.createGroup(t, testAccount, "template-group")
|
||||
byID := o.createGroup(t, testAccount, "by-id")
|
||||
ltID := o.createTemplate(t, testAccount, "web")
|
||||
_, err := o.templates.CreateLaunchTemplateVersion(t.Context(), &ec2.CreateLaunchTemplateVersionInput{
|
||||
LaunchTemplateId: aws.String(ltID),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
|
||||
KeyName: aws.String("template-key"),
|
||||
Placement: &ec2.LaunchTemplatePlacementRequest{GroupName: aws.String("template-group")},
|
||||
},
|
||||
}, testAccount)
|
||||
require.NoError(t, err)
|
||||
_, err = o.templates.CreateLaunchTemplateVersion(t.Context(), &ec2.CreateLaunchTemplateVersionInput{
|
||||
LaunchTemplateId: aws.String(ltID),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{Placement: &ec2.LaunchTemplatePlacementRequest{GroupId: aws.String(byID)}},
|
||||
}, testAccount)
|
||||
require.NoError(t, err)
|
||||
|
||||
res := o.cleanup(t)
|
||||
assert.ElementsMatch(t, []Item{
|
||||
item(usedKey, ec2v1.PlatformKindKeyPair, DispositionBlocked, ReasonLaunchTemplateReference),
|
||||
item(freeKey, ec2v1.PlatformKindKeyPair, DispositionDeleted, ""),
|
||||
item(byName, ec2v1.PlatformKindPlacementGroup, DispositionBlocked, ReasonLaunchTemplateReference),
|
||||
item(byID, ec2v1.PlatformKindPlacementGroup, DispositionBlocked, ReasonLaunchTemplateReference),
|
||||
item(ltID, ec2v1.PlatformKindLaunchTemplate, DispositionBlocked, ReasonReferenceUnknown),
|
||||
}, res.Items)
|
||||
keys, err := o.keys.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{usedKey}, keys.LiveKeyPairIDs)
|
||||
groups, err := o.groups.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, groups.Groups, 2)
|
||||
}
|
||||
|
||||
// A live template version the owner cannot decode may name any key pair or
|
||||
// group, so both kinds are blocked and incomplete and nothing is deleted.
|
||||
func TestCleanupPlatformUndecodableTemplateVersionDeletesNothing(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
keyID := o.importKey(t, testAccount, "deploy")
|
||||
groupID := o.createGroup(t, testAccount, "web")
|
||||
ltID := o.createTemplate(t, testAccount, "web")
|
||||
o.putTemplateRecord(t, testAccount+"."+ltID+".v2", "{not json")
|
||||
|
||||
res := o.cleanup(t)
|
||||
assert.Contains(t, res.Items, item(keyID, ec2v1.PlatformKindKeyPair, DispositionBlocked, ReasonReferenceCheckFailed))
|
||||
assert.Contains(t, res.Items, item(groupID, ec2v1.PlatformKindPlacementGroup, DispositionBlocked, ReasonReferenceCheckFailed))
|
||||
assert.Equal(t, ec2v1.PlatformStatusScanIncomplete, statuses(res)[ec2v1.PlatformKindKeyPair])
|
||||
assert.Equal(t, ec2v1.PlatformStatusScanIncomplete, statuses(res)[ec2v1.PlatformKindPlacementGroup])
|
||||
keys, err := o.keys.ScanAccount(t.Context(), testAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{keyID}, keys.LiveKeyPairIDs)
|
||||
}
|
||||
|
||||
// A tag-index entry whose owner re-scan holds an unreadable record is not
|
||||
// cleared, because that record may be its owner; it is residue, not deleted.
|
||||
func TestCleanupPlatformUnresolvedOwnerKeepsTags(t *testing.T) {
|
||||
for name, seed := range map[string]func(t *testing.T, o *owners) string{
|
||||
"key pair": func(t *testing.T, o *owners) string {
|
||||
o.putObject(t, "key-0badbadbadbadbad0.json", "{not json")
|
||||
return "key-0stale000000000"
|
||||
},
|
||||
"placement group": func(t *testing.T, o *owners) string {
|
||||
kv, err := o.js.KeyValue(t.Context(), ec2placementgroup.KVBucketPlacementGroups)
|
||||
require.NoError(t, err)
|
||||
_, err = kv.Put(t.Context(), testAccount+".broken", []byte("{not json"))
|
||||
require.NoError(t, err)
|
||||
return staleTagID
|
||||
},
|
||||
"launch template": func(t *testing.T, o *owners) string {
|
||||
o.putTemplateRecord(t, testAccount+".lt-0broken000000000", "{not json")
|
||||
return "lt-0stale000000000"
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
stale := seed(t, o)
|
||||
o.tag(t, testAccount, stale)
|
||||
|
||||
res := o.cleanup(t)
|
||||
assert.Contains(t, res.Items, item(stale, ec2v1.PlatformKindTagIndex, DispositionResidue, ReasonOwnerRecordsUnresolved))
|
||||
assert.Equal(t, ec2v1.PlatformStatusScanIncomplete, statuses(res)[ec2v1.PlatformKindTagIndex])
|
||||
tagged, err := o.tags.ScanAccountResourceIDs(t.Context(), testAccount, stale[:3])
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{stale}, tagged)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Settled launch-template residue the owner fails to remove is reported
|
||||
// failed from the re-scan, never deleted.
|
||||
func TestCleanupPlatformResidueRemovalFailureIsFailed(t *testing.T) {
|
||||
o := newOwners(t)
|
||||
o.putTemplateRecord(t, testAccount+".lt-0orphan000000000.v1", `{"version_number":1}`)
|
||||
|
||||
res := run(t, New(o.keys, o.groups, failingResidue{o.templates}, o.tags, o.readers()))
|
||||
assert.Equal(t, []Count{{
|
||||
Kind: ec2v1.PlatformKindLaunchTemplate, Class: ec2v1.PlatformClassVersionBodyWithoutHeader,
|
||||
Disposition: DispositionFailed, Reason: ReasonOwnerDeleteFailed, Count: 1,
|
||||
}}, res.Counts)
|
||||
assert.Equal(t, ec2v1.PlatformStatusRecordsFound, statuses(res)[ec2v1.PlatformKindLaunchTemplate])
|
||||
}
|
||||
|
||||
// failingResidue reports every settled orphan as a failed removal and removes nothing.
|
||||
type failingResidue struct {
|
||||
*ec2launchtemplate.LaunchTemplateServiceImpl
|
||||
}
|
||||
|
||||
func (f failingResidue) CleanupAccountResidue(ctx context.Context, acct string, _ time.Time) (*ec2launchtemplate.AccountCleanup, error) {
|
||||
scan, err := f.ScanAccount(ctx, acct)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ec2launchtemplate.AccountCleanup{KeptTemplateIDs: scan.LiveTemplateIDs, OrphanBodiesFailed: scan.OrphanVersionBodies}, nil
|
||||
}
|
||||
@@ -0,0 +1,411 @@
|
||||
package accountcleanup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"slices"
|
||||
"time"
|
||||
|
||||
ec2v1 "github.com/mulgadc/spinifex/contracts/ec2/v1"
|
||||
ec2key "github.com/mulgadc/spinifex/spinifex/domains/ec2/key"
|
||||
ec2launchtemplate "github.com/mulgadc/spinifex/spinifex/domains/ec2/launchtemplate"
|
||||
ec2placementgroup "github.com/mulgadc/spinifex/spinifex/domains/ec2/placementgroup"
|
||||
)
|
||||
|
||||
// Each kind runs scan, owner cleanup, then re-scan. The first scan only bounds
|
||||
// how much nameless residue a run can claim to have removed.
|
||||
|
||||
type keyPairRun struct {
|
||||
kind *kindRun
|
||||
// unchecked: the reference check failed, so nothing was attempted.
|
||||
unchecked bool
|
||||
before *ec2key.AccountScan
|
||||
cleaned *ec2key.AccountCleanup
|
||||
did attempts
|
||||
}
|
||||
|
||||
// cleanKeyPairs deletes only when the reference check completed; otherwise
|
||||
// every key pair and orphan public key is kept and the kind is incomplete.
|
||||
func (c *Cleanup) cleanKeyPairs(ctx context.Context, acct string, settledBefore time.Time, refs referenceCheck) *keyPairRun {
|
||||
run := &keyPairRun{kind: c.startKind(ec2v1.PlatformKindKeyPair), did: newAttempts()}
|
||||
if before, err := c.keys.ScanAccount(ctx, acct); err == nil {
|
||||
run.before = before
|
||||
}
|
||||
if !refs.complete {
|
||||
run.unchecked = true
|
||||
run.kind.incomplete(ReasonReferenceCheckFailed)
|
||||
return run
|
||||
}
|
||||
cleaned, err := c.keys.CleanupAccount(ctx, acct, settledBefore, refs.keyReason)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: key-pair cleanup failed", "err", err)
|
||||
run.kind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
return run
|
||||
}
|
||||
run.cleaned = cleaned
|
||||
markAll(run.did.deleted, cleaned.DeletedKeyPairIDs)
|
||||
markAll(run.did.failed, cleaned.FailedKeyPairIDs)
|
||||
maps.Copy(run.did.kept, cleaned.Referenced)
|
||||
return run
|
||||
}
|
||||
|
||||
func (c *Cleanup) finishKeyPairs(ctx context.Context, acct string, run *keyPairRun, res *Result) ownedIDs {
|
||||
defer c.finishKind(run.kind)
|
||||
after, err := c.keys.ScanAccount(ctx, acct)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: key-pair re-scan failed", "err", err)
|
||||
run.kind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
after = nil
|
||||
}
|
||||
present := map[string]bool{}
|
||||
if after != nil {
|
||||
markAll(present, after.LiveKeyPairIDs)
|
||||
markAll(present, after.UnreadableKeyPairIDs)
|
||||
if after.ReadFailures > 0 {
|
||||
run.kind.incomplete(ec2v1.PlatformReasonRecordReadFailed)
|
||||
}
|
||||
}
|
||||
var beforeLive, afterLive, deleted, failed []string
|
||||
if run.before != nil {
|
||||
beforeLive = run.before.LiveKeyPairIDs
|
||||
}
|
||||
if after != nil {
|
||||
afterLive = after.LiveKeyPairIDs
|
||||
}
|
||||
if run.cleaned != nil {
|
||||
deleted, failed = run.cleaned.DeletedKeyPairIDs, run.cleaned.FailedKeyPairIDs
|
||||
}
|
||||
if run.unchecked {
|
||||
keepAll(run.did.kept, beforeLive, ReasonReferenceCheckFailed)
|
||||
keepAll(run.did.kept, afterLive, ReasonReferenceCheckFailed)
|
||||
}
|
||||
for _, id := range sortedUnion(beforeLive, deleted, failed, afterLive) {
|
||||
disposition, reason := itemDisposition(id, run.did, present[id], after != nil)
|
||||
res.addItem(run.kind, id, disposition, reason, 0)
|
||||
}
|
||||
if after == nil {
|
||||
return ownedIDs{}
|
||||
}
|
||||
res.addCount(run.kind, ec2v1.PlatformClassUnreadableRecord, DispositionResidue, ReasonUnreadableRecord, len(after.UnreadableKeyPairIDs))
|
||||
res.addCount(run.kind, ec2v1.PlatformClassPublicKeyUnresolved, DispositionResidue, ReasonOwnerRecordsUnresolved, after.UnresolvedPublicKeys)
|
||||
var before, removed, failedN, unsettled int
|
||||
kept := map[string]int{}
|
||||
if run.before != nil {
|
||||
before = run.before.OrphanPublicKeys
|
||||
}
|
||||
if run.cleaned != nil {
|
||||
removed, failedN, unsettled = run.cleaned.OrphansRemoved, run.cleaned.OrphansFailed, run.cleaned.OrphansUnsettled
|
||||
kept = run.cleaned.OrphansReferenced
|
||||
} else if run.unchecked {
|
||||
kept[ReasonReferenceCheckFailed] = after.OrphanPublicKeys
|
||||
}
|
||||
res.residueCounts(run.kind, ec2v1.PlatformClassPublicKeyWithoutMetadata, before, after.OrphanPublicKeys, removed, failedN, unsettled, kept)
|
||||
return ownedIDs{ids: present, resolved: after.ReadFailures == 0 && len(after.UnreadableKeyPairIDs) == 0}
|
||||
}
|
||||
|
||||
type placementGroupRun struct {
|
||||
kind *kindRun
|
||||
unchecked bool
|
||||
before *ec2placementgroup.AccountScan
|
||||
cleaned *ec2placementgroup.AccountCleanup
|
||||
did attempts
|
||||
}
|
||||
|
||||
// cleanPlacementGroups deletes only when the reference check completed.
|
||||
func (c *Cleanup) cleanPlacementGroups(ctx context.Context, acct string, refs referenceCheck) *placementGroupRun {
|
||||
run := &placementGroupRun{kind: c.startKind(ec2v1.PlatformKindPlacementGroup), did: newAttempts()}
|
||||
if before, err := c.groups.ScanAccount(ctx, acct); err == nil {
|
||||
run.before = before
|
||||
}
|
||||
if !refs.complete {
|
||||
run.unchecked = true
|
||||
run.kind.incomplete(ReasonReferenceCheckFailed)
|
||||
return run
|
||||
}
|
||||
cleaned, err := c.groups.CleanupAccount(ctx, acct, refs.groupReason)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: placement-group cleanup failed", "err", err)
|
||||
run.kind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
return run
|
||||
}
|
||||
run.cleaned = cleaned
|
||||
markAll(run.did.deleted, cleaned.DeletedGroupIDs)
|
||||
markAll(run.did.failed, cleaned.FailedGroupIDs)
|
||||
for _, b := range cleaned.Blocked {
|
||||
run.did.blocked[b.GroupID] = b.MemberCount
|
||||
}
|
||||
maps.Copy(run.did.kept, cleaned.Referenced)
|
||||
return run
|
||||
}
|
||||
|
||||
// finishPlacementGroups reports a group that still names instances as
|
||||
// blocked with its member count, from the re-scan when it can.
|
||||
func (c *Cleanup) finishPlacementGroups(ctx context.Context, acct string, run *placementGroupRun, res *Result) ownedIDs {
|
||||
defer c.finishKind(run.kind)
|
||||
after, err := c.groups.ScanAccount(ctx, acct)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: placement-group re-scan failed", "err", err)
|
||||
run.kind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
after = nil
|
||||
}
|
||||
present := map[string]bool{}
|
||||
members := map[string]int{}
|
||||
var ids []string
|
||||
if run.before != nil {
|
||||
for _, g := range run.before.Groups {
|
||||
ids = append(ids, g.GroupID)
|
||||
}
|
||||
}
|
||||
if run.cleaned != nil {
|
||||
ids = append(ids, run.cleaned.DeletedGroupIDs...)
|
||||
ids = append(ids, run.cleaned.FailedGroupIDs...)
|
||||
for _, b := range run.cleaned.Blocked {
|
||||
ids = append(ids, b.GroupID)
|
||||
}
|
||||
for id := range run.cleaned.Referenced {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
if after != nil {
|
||||
for _, g := range after.Groups {
|
||||
present[g.GroupID] = true
|
||||
members[g.GroupID] = g.MemberCount
|
||||
ids = append(ids, g.GroupID)
|
||||
}
|
||||
if after.ReadFailures > 0 {
|
||||
run.kind.incomplete(ec2v1.PlatformReasonRecordReadFailed)
|
||||
}
|
||||
}
|
||||
if run.unchecked {
|
||||
keepAll(run.did.kept, ids, ReasonReferenceCheckFailed)
|
||||
}
|
||||
for _, id := range sortedUnion(ids) {
|
||||
if present[id] && (members[id] > 0 || run.did.blocked[id] > 0) {
|
||||
count := members[id]
|
||||
if count == 0 {
|
||||
count = run.did.blocked[id]
|
||||
}
|
||||
res.addItem(run.kind, id, DispositionBlocked, ReasonInstanceMembers, count)
|
||||
continue
|
||||
}
|
||||
disposition, reason := itemDisposition(id, run.did, present[id], after != nil)
|
||||
res.addItem(run.kind, id, disposition, reason, 0)
|
||||
}
|
||||
if after == nil {
|
||||
return ownedIDs{}
|
||||
}
|
||||
res.addCount(run.kind, ec2v1.PlatformClassUnreadableRecord, DispositionResidue, ReasonUnreadableRecord, after.Unreadable)
|
||||
return ownedIDs{ids: present, resolved: after.ReadFailures == 0 && after.Unreadable == 0}
|
||||
}
|
||||
|
||||
type launchTemplateRun struct {
|
||||
kind *kindRun
|
||||
before *ec2launchtemplate.AccountScan
|
||||
cleaned *ec2launchtemplate.AccountCleanup
|
||||
did attempts
|
||||
}
|
||||
|
||||
func (c *Cleanup) cleanLaunchTemplates(ctx context.Context, acct string, settledBefore time.Time) *launchTemplateRun {
|
||||
run := &launchTemplateRun{kind: c.startKind(ec2v1.PlatformKindLaunchTemplate), did: newAttempts()}
|
||||
if before, err := c.templates.ScanAccount(ctx, acct); err == nil {
|
||||
run.before = before
|
||||
}
|
||||
cleaned, err := c.templates.CleanupAccountResidue(ctx, acct, settledBefore)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: launch-template residue cleanup failed", "err", err)
|
||||
run.kind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
return run
|
||||
}
|
||||
run.cleaned = cleaned
|
||||
keepAll(run.did.kept, cleaned.KeptTemplateIDs, ReasonReferenceUnknown)
|
||||
return run
|
||||
}
|
||||
|
||||
func (c *Cleanup) finishLaunchTemplates(ctx context.Context, acct string, run *launchTemplateRun, res *Result) ownedIDs {
|
||||
defer c.finishKind(run.kind)
|
||||
after, err := c.templates.ScanAccount(ctx, acct)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: launch-template re-scan failed", "err", err)
|
||||
run.kind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
after = nil
|
||||
}
|
||||
present := map[string]bool{}
|
||||
if after != nil {
|
||||
markAll(present, after.LiveTemplateIDs)
|
||||
markAll(present, after.UnreadableTemplateIDs)
|
||||
if after.ReadFailures > 0 {
|
||||
run.kind.incomplete(ec2v1.PlatformReasonRecordReadFailed)
|
||||
}
|
||||
}
|
||||
var beforeLive, afterLive, kept []string
|
||||
if run.before != nil {
|
||||
beforeLive = run.before.LiveTemplateIDs
|
||||
}
|
||||
if after != nil {
|
||||
afterLive = after.LiveTemplateIDs
|
||||
}
|
||||
if run.cleaned != nil {
|
||||
kept = run.cleaned.KeptTemplateIDs
|
||||
}
|
||||
// Live templates are never deleted here, so every one still present is
|
||||
// blocked on the unknown EKS nodegroup reference, whenever it appeared.
|
||||
for _, id := range afterLive {
|
||||
run.did.kept[id] = ReasonReferenceUnknown
|
||||
}
|
||||
for _, id := range sortedUnion(beforeLive, kept, afterLive) {
|
||||
disposition, reason := itemDisposition(id, run.did, present[id], after != nil)
|
||||
res.addItem(run.kind, id, disposition, reason, 0)
|
||||
}
|
||||
if after == nil {
|
||||
return ownedIDs{}
|
||||
}
|
||||
res.addCount(run.kind, ec2v1.PlatformClassUnreadableRecord, DispositionResidue, ReasonUnreadableRecord,
|
||||
len(after.UnreadableTemplateIDs)+after.UnreadableNameIndexEntries)
|
||||
res.addCount(run.kind, ec2v1.PlatformClassUnrecognisedTemplateEntry, DispositionResidue, ReasonUnrecognisedRecord, after.UnrecognisedEntries)
|
||||
var c0 ec2launchtemplate.AccountCleanup
|
||||
if run.cleaned != nil {
|
||||
c0 = *run.cleaned
|
||||
}
|
||||
var beforeBodies, beforeNames int
|
||||
if run.before != nil {
|
||||
beforeBodies, beforeNames = run.before.OrphanVersionBodies, run.before.OrphanNameIndexEntries
|
||||
}
|
||||
res.residueCounts(run.kind, ec2v1.PlatformClassVersionBodyWithoutHeader, beforeBodies, after.OrphanVersionBodies,
|
||||
c0.OrphanBodiesRemoved, c0.OrphanBodiesFailed, c0.OrphanBodiesUnsettled, nil)
|
||||
res.residueCounts(run.kind, ec2v1.PlatformClassNameIndexWithoutHeader, beforeNames, after.OrphanNameIndexEntries,
|
||||
c0.OrphanNamesRemoved, c0.OrphanNamesFailed, c0.OrphanNamesUnsettled, nil)
|
||||
return ownedIDs{ids: present, resolved: after.ReadFailures == 0 && len(after.UnreadableTemplateIDs) == 0 &&
|
||||
after.UnreadableNameIndexEntries == 0 && after.UnrecognisedEntries == 0}
|
||||
}
|
||||
|
||||
// cleanTagIndex clears entries whose owner re-scan resolved and holds no
|
||||
// record for the ID, then re-lists the index to prove each clear. Entries of
|
||||
// resources that still exist are theirs, not residue, and are left alone.
|
||||
func (c *Cleanup) cleanTagIndex(ctx context.Context, acct string, kind *kindRun, tagged []string, listed bool, owners map[string]ownedIDs, res *Result) KindReport {
|
||||
if listed {
|
||||
c.clearOrphanTags(ctx, acct, kind, tagged, owners, res)
|
||||
}
|
||||
c.finishKind(kind)
|
||||
return kind.report
|
||||
}
|
||||
|
||||
func (c *Cleanup) clearOrphanTags(ctx context.Context, acct string, kind *kindRun, tagged []string, owners map[string]ownedIDs, res *Result) {
|
||||
// A resource this run reports as not deleted owns its tags even when its
|
||||
// kind's re-scan failed.
|
||||
reported := map[string]bool{}
|
||||
for _, it := range res.Items {
|
||||
if it.Disposition != DispositionDeleted {
|
||||
reported[it.ResourceID] = true
|
||||
}
|
||||
}
|
||||
var cleared []string
|
||||
for _, id := range tagged {
|
||||
owner := owners[idPrefix(id)]
|
||||
switch {
|
||||
case owner.ids[id], reported[id]:
|
||||
kind.found = true
|
||||
case !owner.resolved:
|
||||
kind.incomplete(ec2v1.PlatformReasonOwnerRecordsUnresolved)
|
||||
res.addItem(kind, id, DispositionResidue, ReasonOwnerRecordsUnresolved, 0)
|
||||
default:
|
||||
if err := c.tags.DeleteAllTags(ctx, acct, id); err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: tag-index clear failed", "resourceId", id, "err", err)
|
||||
res.addItem(kind, id, DispositionFailed, ReasonOwnerDeleteFailed, 0)
|
||||
continue
|
||||
}
|
||||
cleared = append(cleared, id)
|
||||
}
|
||||
}
|
||||
after, err := c.tags.ScanAccountResourceIDs(ctx, acct, keyPairIDPrefix, placementGroupIDPrefix, launchTemplateIDPrefix)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: tag index re-scan failed", "err", err)
|
||||
kind.incomplete(ec2v1.PlatformReasonStoreListFailed)
|
||||
}
|
||||
for _, id := range cleared {
|
||||
disposition, reason := itemDisposition(id, attempts{deleted: map[string]bool{id: true}}, slices.Contains(after, id), err == nil)
|
||||
res.addItem(kind, id, disposition, reason, 0)
|
||||
}
|
||||
if len(after) > 0 {
|
||||
kind.found = true
|
||||
}
|
||||
}
|
||||
|
||||
func keepAll(set map[string]string, ids []string, reason string) {
|
||||
for _, id := range ids {
|
||||
set[id] = reason
|
||||
}
|
||||
}
|
||||
|
||||
// referenceCheck is the outcome of reading what instance records, spot
|
||||
// requests and launch-template versions name. complete is false when any
|
||||
// reader failed or is absent.
|
||||
type referenceCheck struct {
|
||||
complete bool
|
||||
instances, spot, template *References
|
||||
}
|
||||
|
||||
// readReferences runs every reference check. A check that cannot complete is
|
||||
// never read as "no references".
|
||||
func (c *Cleanup) readReferences(ctx context.Context, acct string) referenceCheck {
|
||||
r := c.refs
|
||||
if r.Instances == nil || r.Spot == nil || r.Templates == nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: reference reader not configured")
|
||||
return referenceCheck{}
|
||||
}
|
||||
instances, err := r.Instances.AccountInstanceReferences(ctx, acct)
|
||||
if err != nil || instances == nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: instance reference check failed", "err", err)
|
||||
return referenceCheck{}
|
||||
}
|
||||
launch, err := r.Spot.AccountLaunchReferences(ctx, acct)
|
||||
if err != nil || launch == nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: spot reference check failed", "err", err)
|
||||
return referenceCheck{}
|
||||
}
|
||||
tmpl, err := r.Templates.AccountTemplateReferences(ctx, acct)
|
||||
if err != nil || tmpl == nil {
|
||||
slog.WarnContext(ctx, "Platform cleanup: launch-template reference check failed", "err", err)
|
||||
return referenceCheck{}
|
||||
}
|
||||
inst := newReferences()
|
||||
inst.merge(instances)
|
||||
spot := newReferences()
|
||||
spot.addKeys(launch.KeyNames...)
|
||||
spot.addGroups(launch.PlacementGroupNames...)
|
||||
template := newReferences()
|
||||
template.addKeys(tmpl.KeyNames...)
|
||||
template.addGroups(tmpl.PlacementGroupNames...)
|
||||
template.addGroupIDs(tmpl.PlacementGroupIDs...)
|
||||
return referenceCheck{complete: true, instances: inst, spot: spot, template: template}
|
||||
}
|
||||
|
||||
func (r referenceCheck) keyReason(name string) string {
|
||||
switch {
|
||||
case r.instances.KeyNames[name]:
|
||||
return ReasonInstanceReference
|
||||
case r.spot.KeyNames[name]:
|
||||
return ReasonSpotRequestReference
|
||||
case r.template.KeyNames[name]:
|
||||
return ReasonLaunchTemplateReference
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (r referenceCheck) groupReason(id, name string) string {
|
||||
switch {
|
||||
case r.instances.hasGroup(id, name):
|
||||
return ReasonInstanceReference
|
||||
case r.spot.hasGroup(id, name):
|
||||
return ReasonSpotRequestReference
|
||||
case r.template.hasGroup(id, name):
|
||||
return ReasonLaunchTemplateReference
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func markAll(set map[string]bool, ids []string) {
|
||||
for _, id := range ids {
|
||||
set[id] = true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package accountcleanup
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
ec2launchtemplate "github.com/mulgadc/spinifex/spinifex/domains/ec2/launchtemplate"
|
||||
ec2spotinstance "github.com/mulgadc/spinifex/spinifex/domains/ec2/spotinstance"
|
||||
)
|
||||
|
||||
// References are the key-pair and placement-group names, and placement-group
|
||||
// IDs, that other EC2 records still name. They select what to keep and are
|
||||
// never reported.
|
||||
type References struct {
|
||||
KeyNames map[string]bool
|
||||
PlacementGroupNames map[string]bool
|
||||
PlacementGroupIDs map[string]bool
|
||||
}
|
||||
|
||||
// InstanceReferenceReader reads the key-pair and placement-group names the
|
||||
// account's non-terminated instance records carry, stopped ones included. It
|
||||
// is complete or an error, never a partial answer. A later wiring supplies it.
|
||||
type InstanceReferenceReader interface {
|
||||
AccountInstanceReferences(ctx context.Context, accountID string) (*References, error)
|
||||
}
|
||||
|
||||
// SpotReferenceReader reads the names the account's open or active spot
|
||||
// requests would launch with, complete or an error.
|
||||
type SpotReferenceReader interface {
|
||||
AccountLaunchReferences(ctx context.Context, accountID string) (*ec2spotinstance.LaunchReferences, error)
|
||||
}
|
||||
|
||||
// TemplateReferenceReader reads the names and IDs every version of every
|
||||
// live launch template would launch with, complete or an error.
|
||||
type TemplateReferenceReader interface {
|
||||
AccountTemplateReferences(ctx context.Context, accountID string) (*ec2launchtemplate.TemplateReferences, error)
|
||||
}
|
||||
|
||||
var (
|
||||
_ SpotReferenceReader = (*ec2spotinstance.SpotInstanceServiceImpl)(nil)
|
||||
_ TemplateReferenceReader = (*ec2launchtemplate.LaunchTemplateServiceImpl)(nil)
|
||||
)
|
||||
|
||||
// ReferenceReaders are the checks run before key pairs and placement groups
|
||||
// are deleted. A nil reader counts as a failed check.
|
||||
type ReferenceReaders struct {
|
||||
Instances InstanceReferenceReader
|
||||
Spot SpotReferenceReader
|
||||
Templates TemplateReferenceReader
|
||||
}
|
||||
|
||||
func newReferences() *References {
|
||||
return &References{KeyNames: map[string]bool{}, PlacementGroupNames: map[string]bool{}, PlacementGroupIDs: map[string]bool{}}
|
||||
}
|
||||
|
||||
func (r *References) addKeys(names ...string) {
|
||||
for _, name := range names {
|
||||
if name != "" {
|
||||
r.KeyNames[name] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *References) addGroups(names ...string) {
|
||||
for _, name := range names {
|
||||
if name != "" {
|
||||
r.PlacementGroupNames[name] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *References) addGroupIDs(ids ...string) {
|
||||
for _, id := range ids {
|
||||
if id != "" {
|
||||
r.PlacementGroupIDs[id] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (r *References) merge(o *References) {
|
||||
for name := range o.KeyNames {
|
||||
r.addKeys(name)
|
||||
}
|
||||
for name := range o.PlacementGroupNames {
|
||||
r.addGroups(name)
|
||||
}
|
||||
for id := range o.PlacementGroupIDs {
|
||||
r.addGroupIDs(id)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *References) hasGroup(id, name string) bool {
|
||||
return r.PlacementGroupIDs[id] || r.PlacementGroupNames[name]
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
package accountcleanup
|
||||
|
||||
import "time"
|
||||
|
||||
// Request names the account to clean. CorrelationRef is an opaque caller
|
||||
// reference echoed for logs; it grants nothing and is not a fence.
|
||||
type Request struct {
|
||||
AccountID string `json:"account_id"`
|
||||
CorrelationRef string `json:"correlation_ref,omitempty"`
|
||||
}
|
||||
|
||||
// Result is what the owner stores hold after one run. Kinds reuse the ec2.v1
|
||||
// platform kind and status vocabulary; Items and Counts carry resource IDs,
|
||||
// fixed reasons and counts only, never names, tags, key material or data.
|
||||
type Result struct {
|
||||
AccountID string `json:"account_id"`
|
||||
CorrelationRef string `json:"correlation_ref,omitempty"`
|
||||
Kinds []KindReport `json:"kinds"`
|
||||
Items []Item `json:"items,omitempty"`
|
||||
Counts []Count `json:"counts,omitempty"`
|
||||
}
|
||||
|
||||
// KindReport is one kind's status after its re-scan. Clean means the re-scan
|
||||
// read every store completely and found nothing; writers are not fenced, so
|
||||
// it is a snapshot, not evidence of absence.
|
||||
type KindReport struct {
|
||||
Kind string `json:"kind"`
|
||||
Status string `json:"status"`
|
||||
IncompleteReasons []string `json:"incomplete_reasons,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
EndedAt time.Time `json:"ended_at"`
|
||||
}
|
||||
|
||||
// Item is one resource by its AWS resource ID. MemberCount is set only for a
|
||||
// placement group blocked by members; instance and request IDs are never reported.
|
||||
type Item struct {
|
||||
ResourceID string `json:"resource_id"`
|
||||
Kind string `json:"kind"`
|
||||
Disposition string `json:"disposition"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
MemberCount int `json:"member_count,omitempty"`
|
||||
}
|
||||
|
||||
// Count is the number of records of one class with one disposition, for
|
||||
// records that have no public identity.
|
||||
type Count struct {
|
||||
Kind string `json:"kind"`
|
||||
Class string `json:"class"`
|
||||
Disposition string `json:"disposition"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Count int `json:"count"`
|
||||
}
|
||||
|
||||
// Dispositions. Only a complete owner re-scan that no longer finds a record
|
||||
// makes it deleted.
|
||||
const (
|
||||
DispositionDeleted = "deleted"
|
||||
DispositionBlocked = "blocked"
|
||||
DispositionFailed = "failed"
|
||||
DispositionResidue = "residue"
|
||||
)
|
||||
|
||||
// Reasons are fixed categories, never store error text.
|
||||
const (
|
||||
// ReasonInstanceMembers: a placement group still names instances.
|
||||
ReasonInstanceMembers = "instance-members"
|
||||
// ReasonOwnerDeleteFailed: the owner delete hit a store failure.
|
||||
ReasonOwnerDeleteFailed = "owner-delete-failed"
|
||||
// ReasonStillPresent: the owner reported a delete but the re-scan found it.
|
||||
ReasonStillPresent = "still-present"
|
||||
// ReasonAbsenceUnproven: the re-scan failed, so absence is not shown.
|
||||
ReasonAbsenceUnproven = "absence-unproven"
|
||||
// ReasonUnreadableRecord: a record the owner cannot read is never deleted.
|
||||
ReasonUnreadableRecord = "unreadable-record"
|
||||
// ReasonUnrecognisedRecord: an entry matching no owner record shape.
|
||||
ReasonUnrecognisedRecord = "unrecognised-record"
|
||||
// ReasonOwnerRecordsUnresolved: the record may belong to an unreadable one.
|
||||
ReasonOwnerRecordsUnresolved = "owner-records-unresolved"
|
||||
// ReasonRecentlyWritten: residue younger than the settle window, which may
|
||||
// be a create still in flight.
|
||||
ReasonRecentlyWritten = "recently-written"
|
||||
// ReasonAppearedDuringCleanup: a record the run did not see until its re-scan.
|
||||
ReasonAppearedDuringCleanup = "appeared-during-cleanup"
|
||||
// ReasonInstanceReference: a non-terminated instance record names it.
|
||||
ReasonInstanceReference = "instance-reference"
|
||||
// ReasonSpotRequestReference: an open or active spot request names it.
|
||||
ReasonSpotRequestReference = "spot-request-reference"
|
||||
// ReasonLaunchTemplateReference: a version of a live launch template names it.
|
||||
ReasonLaunchTemplateReference = "launch-template-reference"
|
||||
// ReasonReferenceUnknown: a record EC2 cannot read may name it, so it is
|
||||
// never deleted by account cleanup.
|
||||
ReasonReferenceUnknown = "reference-unknown"
|
||||
// ReasonReferenceCheckFailed: an instance, spot or launch-template reference
|
||||
// check did not complete, so nothing it guards was deleted. Also a kind
|
||||
// incomplete reason.
|
||||
ReasonReferenceCheckFailed = "reference-check-failed"
|
||||
)
|
||||
@@ -0,0 +1,75 @@
|
||||
package key
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AccountCleanup is what one CleanupAccount run did. It holds key-pair IDs and
|
||||
// counts only; it is the owner's account of its attempts, not proof of
|
||||
// absence, which only a later ScanAccount gives.
|
||||
type AccountCleanup struct {
|
||||
// DeletedKeyPairIDs completed the same owner delete DeleteKeyPair runs.
|
||||
DeletedKeyPairIDs []string
|
||||
// FailedKeyPairIDs had a store failure part way; the metadata remains.
|
||||
FailedKeyPairIDs []string
|
||||
// Referenced maps a kept key-pair ID to the reason the caller's predicate
|
||||
// gave for keeping it.
|
||||
Referenced map[string]string
|
||||
// Orphan public-key objects removed, that failed to delete, or left because
|
||||
// they were written after settledBefore and may belong to an in-flight create.
|
||||
OrphansRemoved, OrphansFailed, OrphansUnsettled int
|
||||
// OrphansReferenced counts public-key objects kept, by predicate reason.
|
||||
OrphansReferenced map[string]int
|
||||
}
|
||||
|
||||
// CleanupAccount deletes every readable key pair of the account through the
|
||||
// owner delete, and removes public-key objects no metadata names once they
|
||||
// were last written before settledBefore. A key pair or public-key object
|
||||
// for whose name referenced returns a reason is kept: IMDS serves the public
|
||||
// key by name to any instance naming it. Unreadable metadata, and public keys that
|
||||
// may belong to it, are left for the owner scan to report. A listing failure
|
||||
// is returned as an error and nothing is deleted.
|
||||
func (s *KeyServiceImpl) CleanupAccount(ctx context.Context, accountID string, settledBefore time.Time, referenced func(keyName string) string) (*AccountCleanup, error) {
|
||||
records, err := s.classifyAccount(ctx, accountID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if referenced == nil {
|
||||
referenced = func(string) string { return "" }
|
||||
}
|
||||
out := &AccountCleanup{Referenced: map[string]string{}, OrphansReferenced: map[string]int{}}
|
||||
for _, kp := range records.live {
|
||||
if reason := referenced(kp.name); reason != "" {
|
||||
out.Referenced[kp.id] = reason
|
||||
continue
|
||||
}
|
||||
if err := s.deleteKeyPairRecords(ctx, accountID, kp.name, kp.id); err != nil {
|
||||
out.FailedKeyPairIDs = append(out.FailedKeyPairIDs, kp.id)
|
||||
continue
|
||||
}
|
||||
out.DeletedKeyPairIDs = append(out.DeletedKeyPairIDs, kp.id)
|
||||
}
|
||||
for _, orphan := range records.orphans {
|
||||
if reason := referenced(orphan.name); reason != "" {
|
||||
out.OrphansReferenced[reason]++
|
||||
continue
|
||||
}
|
||||
if orphan.modified == nil || !orphan.modified.Before(settledBefore) {
|
||||
out.OrphansUnsettled++
|
||||
continue
|
||||
}
|
||||
if err := s.deleteObject(ctx, orphan.key); err != nil {
|
||||
slog.WarnContext(ctx, "Key-pair account cleanup: orphan public key delete failed", "err", err)
|
||||
out.OrphansFailed++
|
||||
continue
|
||||
}
|
||||
out.OrphansRemoved++
|
||||
}
|
||||
slog.InfoContext(ctx, "Key-pair account cleanup completed", "accountID", accountID,
|
||||
"deleted", len(out.DeletedKeyPairIDs), "failed", len(out.FailedKeyPairIDs), "referenced", len(out.Referenced),
|
||||
"orphansRemoved", out.OrphansRemoved, "orphansFailed", out.OrphansFailed,
|
||||
"orphansUnsettled", out.OrphansUnsettled)
|
||||
return out, nil
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/s3"
|
||||
@@ -29,10 +30,49 @@ type AccountScan struct {
|
||||
ReadFailures int
|
||||
}
|
||||
|
||||
// liveKeyPair is a readable metadata record and the name it gives its key.
|
||||
type liveKeyPair struct {
|
||||
id, name string
|
||||
}
|
||||
|
||||
// orphanObject is a public-key object no metadata names, the key name it is
|
||||
// served under, and its last write time when the listing reported one.
|
||||
type orphanObject struct {
|
||||
key, name string
|
||||
modified *time.Time
|
||||
}
|
||||
|
||||
// accountRecords is the private classification ScanAccount and CleanupAccount
|
||||
// share. It holds object keys and names, so it never leaves this package.
|
||||
type accountRecords struct {
|
||||
live []liveKeyPair
|
||||
unreadable []string
|
||||
orphans []orphanObject
|
||||
unresolved int
|
||||
readFailures int
|
||||
}
|
||||
|
||||
// ScanAccount lists and classifies every object under keys/<accountID>/. A
|
||||
// listing failure is returned as an error; per-object read failures are
|
||||
// counted. It reads only objects named as metadata (key-*.json).
|
||||
func (s *KeyServiceImpl) ScanAccount(ctx context.Context, accountID string) (*AccountScan, error) {
|
||||
records, err := s.classifyAccount(ctx, accountID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
scan := &AccountScan{
|
||||
UnreadableKeyPairIDs: records.unreadable,
|
||||
OrphanPublicKeys: len(records.orphans),
|
||||
UnresolvedPublicKeys: records.unresolved,
|
||||
ReadFailures: records.readFailures,
|
||||
}
|
||||
for _, kp := range records.live {
|
||||
scan.LiveKeyPairIDs = append(scan.LiveKeyPairIDs, kp.id)
|
||||
}
|
||||
return scan, nil
|
||||
}
|
||||
|
||||
func (s *KeyServiceImpl) classifyAccount(ctx context.Context, accountID string) (*accountRecords, error) {
|
||||
prefix := fmt.Sprintf("keys/%s/", accountID)
|
||||
objects, _, err := objectstore.ListAll(ctx, s.store, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
@@ -42,9 +82,9 @@ func (s *KeyServiceImpl) ScanAccount(ctx context.Context, accountID string) (*Ac
|
||||
return nil, fmt.Errorf("list key-pair objects: %w", err)
|
||||
}
|
||||
|
||||
scan := &AccountScan{}
|
||||
records := &accountRecords{}
|
||||
liveNames := make(map[string]bool)
|
||||
var rest []string
|
||||
var rest []*s3.Object
|
||||
unreadable := make(map[string]bool)
|
||||
for _, obj := range objects {
|
||||
if obj == nil || obj.Key == nil {
|
||||
@@ -53,24 +93,24 @@ func (s *KeyServiceImpl) ScanAccount(ctx context.Context, accountID string) (*Ac
|
||||
name := strings.TrimPrefix(*obj.Key, prefix)
|
||||
id, isJSON := strings.CutSuffix(name, ".json")
|
||||
if !isJSON || !strings.HasPrefix(id, "key-") {
|
||||
rest = append(rest, name)
|
||||
rest = append(rest, obj)
|
||||
continue
|
||||
}
|
||||
metadata, found, readErr := s.readScanMetadata(ctx, *obj.Key)
|
||||
switch {
|
||||
case readErr != nil:
|
||||
scan.ReadFailures++
|
||||
records.readFailures++
|
||||
unreadable[name] = true
|
||||
case !found:
|
||||
// Removed between the listing and the read.
|
||||
case metadata == nil || aws.StringValue(metadata.KeyPairId) != id || aws.StringValue(metadata.KeyName) == "":
|
||||
unreadable[name] = true
|
||||
default:
|
||||
scan.LiveKeyPairIDs = append(scan.LiveKeyPairIDs, id)
|
||||
records.live = append(records.live, liveKeyPair{id: id, name: *metadata.KeyName})
|
||||
liveNames[*metadata.KeyName] = true
|
||||
}
|
||||
if unreadable[name] {
|
||||
rest = append(rest, name)
|
||||
rest = append(rest, obj)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -81,19 +121,20 @@ func (s *KeyServiceImpl) ScanAccount(ctx context.Context, accountID string) (*Ac
|
||||
delete(unreadable, name)
|
||||
}
|
||||
}
|
||||
for _, name := range rest {
|
||||
for _, obj := range rest {
|
||||
name := strings.TrimPrefix(*obj.Key, prefix)
|
||||
switch {
|
||||
case liveNames[name]:
|
||||
// The public key of a live key pair.
|
||||
case unreadable[name]:
|
||||
scan.UnreadableKeyPairIDs = append(scan.UnreadableKeyPairIDs, strings.TrimSuffix(name, ".json"))
|
||||
records.unreadable = append(records.unreadable, strings.TrimSuffix(name, ".json"))
|
||||
case len(unreadable) > 0:
|
||||
scan.UnresolvedPublicKeys++
|
||||
records.unresolved++
|
||||
default:
|
||||
scan.OrphanPublicKeys++
|
||||
records.orphans = append(records.orphans, orphanObject{key: *obj.Key, name: name, modified: obj.LastModified})
|
||||
}
|
||||
}
|
||||
return scan, nil
|
||||
return records, nil
|
||||
}
|
||||
|
||||
// readScanMetadata reads and decodes one metadata object. A decode failure is
|
||||
|
||||
@@ -33,15 +33,12 @@ func TestScanAccountClassifiesEveryObjectUnderTheAccountPrefix(t *testing.T) {
|
||||
live := importKey(t, svc, ctAccount, "live")
|
||||
dotted := importKey(t, svc, ctAccount, "web.json")
|
||||
|
||||
importKey(t, svc, ctAccount, "deleted")
|
||||
store.failDelete = func(key string) bool { return key == prefix+"deleted" }
|
||||
_, err := svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("deleted")}, ctAccount)
|
||||
require.NoError(t, err)
|
||||
store.failDelete = nil
|
||||
// A public key whose metadata is gone, as an interrupted delete leaves it.
|
||||
putObject(t, store, prefix+"deleted", ctPubKey+"\n")
|
||||
|
||||
store.failPut = func(key string) bool { return strings.HasSuffix(key, ".json") }
|
||||
store.failDelete = func(string) bool { return true }
|
||||
_, err = svc.CreateKeyPair(context.Background(), &ec2.CreateKeyPairInput{KeyName: aws.String("half")}, ctAccount)
|
||||
_, err := svc.CreateKeyPair(context.Background(), &ec2.CreateKeyPairInput{KeyName: aws.String("half")}, ctAccount)
|
||||
require.Error(t, err)
|
||||
store.failPut, store.failDelete = nil, nil
|
||||
|
||||
@@ -50,7 +47,7 @@ func TestScanAccountClassifiesEveryObjectUnderTheAccountPrefix(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.ElementsMatch(t, []string{aws.StringValue(live.KeyPairId), aws.StringValue(dotted.KeyPairId)}, scan.LiveKeyPairIDs)
|
||||
assert.Equal(t, 2, scan.OrphanPublicKeys, "the failed-delete and failed-create public keys")
|
||||
assert.Equal(t, 2, scan.OrphanPublicKeys, "the interrupted-delete and failed-create public keys")
|
||||
assert.Empty(t, scan.UnreadableKeyPairIDs)
|
||||
assert.Zero(t, scan.UnresolvedPublicKeys)
|
||||
assert.Zero(t, scan.ReadFailures)
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
@@ -19,8 +20,8 @@ import (
|
||||
)
|
||||
|
||||
// The TestCurrentBehaviour_AccountTermination_ tests pin how key pairs are
|
||||
// stored and what a delete or Describe leaves behind today. They describe
|
||||
// current behaviour, not the target design; a failure means it changed.
|
||||
// stored and what Describe leaves behind today; a failure means it changed.
|
||||
// The TestAccountTermination_ tests pin the corrected owner delete.
|
||||
|
||||
const (
|
||||
ctBucket = "test-bucket"
|
||||
@@ -152,35 +153,97 @@ func TestCurrentBehaviour_AccountTermination_KeyPairStorageLayout(t *testing.T)
|
||||
}
|
||||
}
|
||||
|
||||
// Pins current behaviour that differs from AWS: a failed public-key delete is
|
||||
// logged, the metadata delete proceeds and the call reports success. The
|
||||
// leftover object still blocks the name and passes launch validation.
|
||||
func TestCurrentBehaviour_AccountTermination_DeleteKeyPairLeavesPublicKeyWhenItsDeleteFails(t *testing.T) {
|
||||
// A key pair is deleted public key first and metadata last, so a failed step
|
||||
// returns an error and leaves the key pair described and found by name; a
|
||||
// retried delete then removes both objects and frees the name.
|
||||
func TestAccountTermination_DeleteKeyPairKeepsTheKeyPairWhenAStepFails(t *testing.T) {
|
||||
prefix := "keys/" + ctAccount + "/"
|
||||
tests := []struct {
|
||||
name string
|
||||
failOn func(keyPairID string) string
|
||||
left func(keyPairID string) []string
|
||||
}{
|
||||
{"public key delete fails", func(string) string { return prefix + "deploy" },
|
||||
func(id string) []string { return sortedStrings(prefix+"deploy", prefix+id+".json") }},
|
||||
{"metadata delete fails", func(id string) string { return prefix + id + ".json" },
|
||||
func(id string) []string { return []string{prefix + id + ".json"} }},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
store := newFaultyStore()
|
||||
svc := ec2key.NewKeyServiceImplWithStore(store, ctBucket)
|
||||
id := aws.StringValue(importKey(t, svc, ctAccount, "deploy").KeyPairId)
|
||||
failing := tc.failOn(id)
|
||||
store.failDelete = func(key string) bool { return key == failing }
|
||||
|
||||
_, err := svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Equal(t, tc.left(id), accountObjects(t, store, ctAccount))
|
||||
assert.Equal(t, []string{"deploy"}, describedKeyNames(t, svc, ctAccount))
|
||||
|
||||
store.failDelete = nil
|
||||
out, err := svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, id, aws.StringValue(out.KeyPairId))
|
||||
assert.Empty(t, accountObjects(t, store, ctAccount))
|
||||
assert.ErrorContains(t, svc.ValidateKeyPairExists(context.Background(), ctAccount, "deploy"), awserrors.ErrorInvalidKeyPairNotFound)
|
||||
|
||||
_, err = svc.CreateKeyPair(context.Background(), &ec2.CreateKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
assert.NoError(t, err, "the name is free again")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A public key left with no metadata is that name's key: a delete by name
|
||||
// removes it and reports success without a key pair ID, freeing the name.
|
||||
func TestAccountTermination_DeleteKeyPairByNameRemovesALeftoverPublicKey(t *testing.T) {
|
||||
store := newFaultyStore()
|
||||
svc := ec2key.NewKeyServiceImplWithStore(store, ctBucket)
|
||||
imported := importKey(t, svc, ctAccount, "deploy")
|
||||
pubKeyPath := "keys/" + ctAccount + "/deploy"
|
||||
store.failDelete = func(key string) bool { return key == pubKeyPath }
|
||||
putObject(t, store, "keys/"+ctAccount+"/orphan", ctPubKey+"\n")
|
||||
|
||||
out, err := svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
out, err := svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("orphan")}, ctAccount)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, aws.BoolValue(out.Return))
|
||||
assert.Equal(t, aws.StringValue(imported.KeyPairId), aws.StringValue(out.KeyPairId))
|
||||
assert.Nil(t, out.KeyPairId)
|
||||
assert.Empty(t, accountObjects(t, store, ctAccount))
|
||||
_, err = svc.CreateKeyPair(context.Background(), &ec2.CreateKeyPairInput{KeyName: aws.String("orphan")}, ctAccount)
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
assert.Equal(t, []string{pubKeyPath}, accountObjects(t, store, ctAccount), "only the public key object remains")
|
||||
assert.Empty(t, describedKeyNames(t, svc, ctAccount))
|
||||
assert.NoError(t, svc.ValidateKeyPairExists(context.Background(), ctAccount, "deploy"))
|
||||
// A by-name delete that cannot read some metadata cannot prove the name is
|
||||
// absent, so it fails rather than reporting an idempotent success.
|
||||
func TestAccountTermination_DeleteKeyPairByNameFailsWhenMetadataIsUnreadable(t *testing.T) {
|
||||
store := newFaultyStore()
|
||||
svc := ec2key.NewKeyServiceImplWithStore(store, ctBucket)
|
||||
id := aws.StringValue(importKey(t, svc, ctAccount, "deploy").KeyPairId)
|
||||
store.failGet = func(key string) bool { return strings.HasSuffix(key, id+".json") }
|
||||
|
||||
// A retried delete finds no metadata, reports success and removes nothing.
|
||||
store.failDelete = nil
|
||||
_, err = svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{pubKeyPath}, accountObjects(t, store, ctAccount))
|
||||
|
||||
_, err = svc.CreateKeyPair(context.Background(), &ec2.CreateKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
_, err := svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
require.Error(t, err)
|
||||
code, _ := awserrors.ResolveErrorCode(err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidKeyPairDuplicate, code)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Len(t, accountObjects(t, store, ctAccount), 2, "nothing was deleted")
|
||||
}
|
||||
|
||||
// Undecodable metadata may be the named key pair, so a by-name delete fails
|
||||
// and leaves both the metadata and the same-name public key in place.
|
||||
func TestAccountTermination_DeleteKeyPairByNameFailsOnUndecodableMetadata(t *testing.T) {
|
||||
store := newFaultyStore()
|
||||
svc := ec2key.NewKeyServiceImplWithStore(store, ctBucket)
|
||||
broken := "keys/" + ctAccount + "/key-0badbadbadbadbad0.json"
|
||||
pub := "keys/" + ctAccount + "/deploy"
|
||||
putObject(t, store, broken, "{not json")
|
||||
putObject(t, store, pub, ctPubKey+"\n")
|
||||
|
||||
_, err := svc.DeleteKeyPair(context.Background(), &ec2.DeleteKeyPairInput{KeyName: aws.String("deploy")}, ctAccount)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Equal(t, sortedStrings(broken, pub), accountObjects(t, store, ctAccount))
|
||||
}
|
||||
|
||||
func sortedStrings(s ...string) []string {
|
||||
slices.Sort(s)
|
||||
return s
|
||||
}
|
||||
|
||||
// A create whose metadata write fails and whose cleanup delete also fails
|
||||
@@ -231,3 +294,79 @@ func TestCurrentBehaviour_AccountTermination_DescribeKeyPairsHidesUnreadableReco
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Account cleanup deletes each readable key pair through the owner delete and
|
||||
// removes a settled public key no metadata names. A younger orphan may be a
|
||||
// create in flight and is left; unreadable metadata is never deleted.
|
||||
func TestAccountCleanup_DeletesKeyPairsAndSettledOrphans(t *testing.T) {
|
||||
store := newFaultyStore()
|
||||
svc := ec2key.NewKeyServiceImplWithStore(store, ctBucket)
|
||||
importKey(t, svc, ctAccount, "deploy")
|
||||
importKey(t, svc, "999988887777", "theirs")
|
||||
putObject(t, store, "keys/"+ctAccount+"/old-orphan", ctPubKey)
|
||||
broken := "keys/" + ctAccount + "/key-0badbadbadbadbad0.json"
|
||||
putObject(t, store, broken, "{not json")
|
||||
|
||||
out, err := svc.CleanupAccount(context.Background(), ctAccount, time.Now().Add(-time.Hour), nil)
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, out.DeletedKeyPairIDs, 1)
|
||||
assert.Zero(t, out.OrphansRemoved, "orphans are left while a public key may belong to unreadable metadata")
|
||||
|
||||
store.Clear()
|
||||
importKey(t, svc, ctAccount, "deploy")
|
||||
putObject(t, store, "keys/"+ctAccount+"/old-orphan", ctPubKey)
|
||||
out, err = svc.CleanupAccount(context.Background(), ctAccount, time.Now().Add(-time.Hour), nil)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, out.OrphansUnsettled, "an orphan written inside the settle window is kept")
|
||||
assert.Equal(t, []string{"keys/" + ctAccount + "/old-orphan"}, accountObjects(t, store, ctAccount))
|
||||
|
||||
out, err = svc.CleanupAccount(context.Background(), ctAccount, time.Now().Add(time.Second), nil)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, out.OrphansRemoved)
|
||||
assert.Empty(t, accountObjects(t, store, ctAccount))
|
||||
}
|
||||
|
||||
// A key pair, and an orphan public key, whose name the reference check names
|
||||
// are kept with the given reason: IMDS serves the public key by name.
|
||||
func TestAccountCleanup_KeepsReferencedKeyPairsAndPublicKeys(t *testing.T) {
|
||||
store := newFaultyStore()
|
||||
svc := ec2key.NewKeyServiceImplWithStore(store, ctBucket)
|
||||
kp := importKey(t, svc, ctAccount, "in-use")
|
||||
importKey(t, svc, ctAccount, "free")
|
||||
putObject(t, store, "keys/"+ctAccount+"/served-orphan", ctPubKey)
|
||||
referenced := func(name string) string {
|
||||
if name == "in-use" || name == "served-orphan" {
|
||||
return "instance-reference"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
out, err := svc.CleanupAccount(context.Background(), ctAccount, time.Now().Add(time.Second), referenced)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, map[string]string{aws.StringValue(kp.KeyPairId): "instance-reference"}, out.Referenced)
|
||||
assert.Len(t, out.DeletedKeyPairIDs, 1)
|
||||
assert.Equal(t, map[string]int{"instance-reference": 1}, out.OrphansReferenced)
|
||||
assert.Zero(t, out.OrphansRemoved)
|
||||
assert.Equal(t, []string{"in-use"}, describedKeyNames(t, svc, ctAccount))
|
||||
assert.Contains(t, accountObjects(t, store, ctAccount), "keys/"+ctAccount+"/served-orphan")
|
||||
}
|
||||
|
||||
// A key pair whose delete fails part way is reported failed with its metadata
|
||||
// kept; the next run completes it.
|
||||
func TestAccountCleanup_InterruptedKeyPairDeleteCompletesOnRerun(t *testing.T) {
|
||||
store := newFaultyStore()
|
||||
svc := ec2key.NewKeyServiceImplWithStore(store, ctBucket)
|
||||
kp := importKey(t, svc, ctAccount, "deploy")
|
||||
store.failDelete = func(key string) bool { return strings.HasSuffix(key, ".json") }
|
||||
|
||||
out, err := svc.CleanupAccount(context.Background(), ctAccount, time.Now(), nil)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{aws.StringValue(kp.KeyPairId)}, out.FailedKeyPairIDs)
|
||||
assert.Equal(t, []string{"deploy"}, describedKeyNames(t, svc, ctAccount))
|
||||
|
||||
store.failDelete = nil
|
||||
out, err = svc.CleanupAccount(context.Background(), ctAccount, time.Now(), nil)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{aws.StringValue(kp.KeyPairId)}, out.DeletedKeyPairIDs)
|
||||
assert.Empty(t, accountObjects(t, store, ctAccount))
|
||||
}
|
||||
|
||||
@@ -500,8 +500,12 @@ func (s *KeyServiceImpl) findKeyPairIdFromKeyName(ctx context.Context, accountID
|
||||
Key: obj.Key,
|
||||
})
|
||||
if err != nil {
|
||||
slog.Debug("Failed to get metadata file", "key", *obj.Key, "err", err)
|
||||
continue
|
||||
if objectstore.IsNoSuchKeyError(err) {
|
||||
continue
|
||||
}
|
||||
// Unread metadata may be the key pair asked for, so absence is unproven.
|
||||
slog.ErrorContext(ctx, "Failed to get metadata file", "key", *obj.Key, "err", err)
|
||||
return "", fmt.Errorf("read metadata %s: %w", *obj.Key, err)
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(getResult.Body)
|
||||
@@ -509,14 +513,15 @@ func (s *KeyServiceImpl) findKeyPairIdFromKeyName(ctx context.Context, accountID
|
||||
slog.Debug("Failed to close metadata body", "key", *obj.Key, "err", err)
|
||||
}
|
||||
if err != nil {
|
||||
slog.Debug("Failed to read metadata body", "key", *obj.Key, "err", err)
|
||||
continue
|
||||
slog.ErrorContext(ctx, "Failed to read metadata body", "key", *obj.Key, "err", err)
|
||||
return "", fmt.Errorf("read metadata %s: %w", *obj.Key, err)
|
||||
}
|
||||
|
||||
metadata, err := decodeKeyPairMetadata(body)
|
||||
if err != nil {
|
||||
slog.Debug("Failed to unmarshal metadata", "key", *obj.Key, "err", err)
|
||||
continue
|
||||
// Undecodable metadata may be this name's key pair, so absence is unproven.
|
||||
slog.ErrorContext(ctx, "Undecodable key-pair metadata", "key", *obj.Key, "err", err)
|
||||
return "", fmt.Errorf("decode metadata %s: %w", *obj.Key, err)
|
||||
}
|
||||
|
||||
// Check if this metadata matches the keyName
|
||||
@@ -611,7 +616,7 @@ func (s *KeyServiceImpl) describedPublicKey(accountID, keyName string) (string,
|
||||
return fmt.Sprintf("%s %s\n", bytes.TrimSpace(ssh.MarshalAuthorizedKey(publicKey)), keyName), nil
|
||||
}
|
||||
|
||||
// DeleteKeyPair removes a key pair (both public key and metadata from S3).
|
||||
// DeleteKeyPair removes a key pair: its public key, metadata and tag-index entry.
|
||||
func (s *KeyServiceImpl) DeleteKeyPair(ctx context.Context, input *ec2.DeleteKeyPairInput, accountID string) (*ec2.DeleteKeyPairOutput, error) {
|
||||
if input == nil {
|
||||
return nil, errors.New(awserrors.ErrorMissingParameter)
|
||||
@@ -641,7 +646,7 @@ func (s *KeyServiceImpl) DeleteKeyPair(ctx context.Context, input *ec2.DeleteKey
|
||||
return &ec2.DeleteKeyPairOutput{Return: aws.Bool(true)}, nil
|
||||
}
|
||||
slog.ErrorContext(ctx, "Failed to get keyName from keyPairId", "keyPairId", keyPairID, "err", err)
|
||||
return nil, err
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
} else if input.KeyName != nil && *input.KeyName != "" {
|
||||
// KeyName provided - validate and find the keyPairId
|
||||
@@ -655,50 +660,70 @@ func (s *KeyServiceImpl) DeleteKeyPair(ctx context.Context, input *ec2.DeleteKey
|
||||
|
||||
keyPairID, err = s.findKeyPairIdFromKeyName(ctx, accountID, keyName)
|
||||
if err != nil {
|
||||
// AWS DeleteKeyPair is idempotent — return success for non-existent keys
|
||||
// AWS DeleteKeyPair is idempotent. A public key left with no metadata
|
||||
// is this name's key, so it goes too and the name becomes reusable.
|
||||
if err.Error() == awserrors.ErrorInvalidKeyPairNotFound {
|
||||
if rmErr := s.deleteObject(ctx, publicKeyPath(accountID, keyName)); rmErr != nil {
|
||||
slog.ErrorContext(ctx, "DeleteKeyPair: failed to remove leftover public key", "keyName", keyName, "err", rmErr)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
slog.DebugContext(ctx, "DeleteKeyPair: key pair not found, returning success (idempotent)", "keyName", keyName)
|
||||
return &ec2.DeleteKeyPairOutput{Return: aws.Bool(true)}, nil
|
||||
}
|
||||
slog.ErrorContext(ctx, "Failed to find keyPairId from keyName", "keyName", keyName, "err", err)
|
||||
return nil, err
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
} else {
|
||||
return nil, errors.New(awserrors.ErrorMissingParameter)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "Deleting key pair", "keyName", keyName, "keyPairId", keyPairID)
|
||||
|
||||
// Delete public key
|
||||
publicKeyPath := fmt.Sprintf("keys/%s/%s", accountID, keyName)
|
||||
_, err = s.store.DeleteObject(ctx, &s3.DeleteObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(publicKeyPath),
|
||||
})
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to delete public key", "path", publicKeyPath, "err", err)
|
||||
// Continue to try deleting metadata even if public key deletion fails
|
||||
}
|
||||
|
||||
// Delete metadata file (stored with keyPairID)
|
||||
metadataPath := fmt.Sprintf("keys/%s/%s.json", accountID, keyPairID)
|
||||
_, err = s.store.DeleteObject(ctx, &s3.DeleteObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(metadataPath),
|
||||
})
|
||||
if err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to delete metadata", "path", metadataPath, "err", err)
|
||||
if err := s.deleteKeyPairRecords(ctx, accountID, keyName, keyPairID); err != nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
s.clearRecordTags(ctx, accountID, keyPairID)
|
||||
|
||||
slog.InfoContext(ctx, "Key pair deleted successfully", "keyName", keyName, "keyPairId", keyPairID)
|
||||
|
||||
// AWS names the key only when one was actually deleted.
|
||||
return &ec2.DeleteKeyPairOutput{Return: aws.Bool(true), KeyPairId: aws.String(keyPairID)}, nil
|
||||
}
|
||||
|
||||
// deleteKeyPairRecords removes the public key, then the metadata, then the
|
||||
// tag-index entry. Metadata goes last: while it exists the key pair is still
|
||||
// described and found by name or ID, so a retried delete finishes the job.
|
||||
func (s *KeyServiceImpl) deleteKeyPairRecords(ctx context.Context, accountID, keyName, keyPairID string) error {
|
||||
if err := s.deleteObject(ctx, publicKeyPath(accountID, keyName)); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to delete public key", "keyName", keyName, "keyPairId", keyPairID, "err", err)
|
||||
return fmt.Errorf("delete public key: %w", err)
|
||||
}
|
||||
if err := s.deleteObject(ctx, metadataPath(accountID, keyPairID)); err != nil {
|
||||
slog.ErrorContext(ctx, "Failed to delete key pair metadata", "keyPairId", keyPairID, "err", err)
|
||||
return fmt.Errorf("delete metadata: %w", err)
|
||||
}
|
||||
s.clearRecordTags(ctx, accountID, keyPairID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// deleteObject deletes one object; an absent object is already deleted.
|
||||
func (s *KeyServiceImpl) deleteObject(ctx context.Context, key string) error {
|
||||
_, err := s.store.DeleteObject(ctx, &s3.DeleteObjectInput{
|
||||
Bucket: aws.String(s.bucketName),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
if err != nil && !objectstore.IsNoSuchKeyError(err) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func publicKeyPath(accountID, keyName string) string {
|
||||
return fmt.Sprintf("keys/%s/%s", accountID, keyName)
|
||||
}
|
||||
|
||||
func metadataPath(accountID, keyPairID string) string {
|
||||
return fmt.Sprintf("keys/%s/%s.json", accountID, keyPairID)
|
||||
}
|
||||
|
||||
// DescribeKeyPairs lists available key pairs by reading metadata files from S3
|
||||
// describeKeyPairsValidFilters defines the set of filter names accepted by DescribeKeyPairs.
|
||||
var describeKeyPairsValidFilters = map[string]bool{
|
||||
@@ -769,7 +794,7 @@ func (s *KeyServiceImpl) DescribeKeyPairs(ctx context.Context, input *ec2.Descri
|
||||
Key: obj.Key,
|
||||
})
|
||||
if err != nil {
|
||||
slog.DebugContext(ctx, "Failed to get metadata file", "key", *obj.Key, "err", err)
|
||||
slog.WarnContext(ctx, "DescribeKeyPairs: skipping metadata the store cannot read", "key", *obj.Key, "err", err)
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -778,13 +803,14 @@ func (s *KeyServiceImpl) DescribeKeyPairs(ctx context.Context, input *ec2.Descri
|
||||
slog.DebugContext(ctx, "Failed to close metadata body", "key", *obj.Key, "err", err)
|
||||
}
|
||||
if err != nil {
|
||||
slog.DebugContext(ctx, "Failed to read metadata body", "key", *obj.Key, "err", err)
|
||||
slog.WarnContext(ctx, "DescribeKeyPairs: skipping metadata whose body cannot be read", "key", *obj.Key, "err", err)
|
||||
continue
|
||||
}
|
||||
|
||||
// The decoder's message is not logged: it can quote the stored bytes.
|
||||
metadata, err := decodeKeyPairMetadata(body)
|
||||
if err != nil {
|
||||
slog.DebugContext(ctx, "Failed to unmarshal metadata", "key", *obj.Key, "err", err)
|
||||
slog.WarnContext(ctx, "DescribeKeyPairs: skipping undecodable metadata", "key", *obj.Key)
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package launchtemplate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/foundation/state/kvutil"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// AccountCleanup is what one CleanupAccountResidue run did. It holds template
|
||||
// IDs and counts only; it is the owner's account of its attempts, not proof of
|
||||
// absence, which only a later ScanAccount gives.
|
||||
type AccountCleanup struct {
|
||||
// KeptTemplateIDs are live templates; account cleanup never deletes them.
|
||||
KeptTemplateIDs []string
|
||||
// Version bodies and name-index entries with no header: removed, failed,
|
||||
// or left because they were written after settledBefore and may belong to
|
||||
// a create still in flight.
|
||||
OrphanBodiesRemoved, OrphanBodiesFailed, OrphanBodiesUnsettled int
|
||||
OrphanNamesRemoved, OrphanNamesFailed, OrphanNamesUnsettled int
|
||||
}
|
||||
|
||||
// CleanupAccountResidue removes the account's version bodies and name-index
|
||||
// entries whose header is absent, once written before settledBefore: without
|
||||
// a header no ID, name or launch can resolve them. Live templates are listed,
|
||||
// not deleted, because not every record that may name one is visible to EC2.
|
||||
// Unreadable headers, their bodies and unrecognised entries are left for the
|
||||
// owner scan to report. A listing failure is returned and nothing is removed.
|
||||
func (s *LaunchTemplateServiceImpl) CleanupAccountResidue(ctx context.Context, accountID string, settledBefore time.Time) (*AccountCleanup, error) {
|
||||
keys, err := kvutil.Keys(ctx, s.kv)
|
||||
if err != nil && !errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, fmt.Errorf("list launch-template keys: %w", err)
|
||||
}
|
||||
|
||||
prefix := accountID + "."
|
||||
present := make(map[string]bool)
|
||||
out := &AccountCleanup{}
|
||||
var bodies, names []string
|
||||
for _, key := range keys {
|
||||
rest, ok := strings.CutPrefix(key, prefix)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case isHeaderKey(key, prefix):
|
||||
entry, err := s.kv.Get(ctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
continue
|
||||
}
|
||||
// An unreadable header still owns its bodies and name.
|
||||
present[rest] = true
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var header LaunchTemplateHeader
|
||||
if json.Unmarshal(entry.Value(), &header) == nil && header.LaunchTemplateId == rest {
|
||||
out.KeptTemplateIDs = append(out.KeptTemplateIDs, rest)
|
||||
}
|
||||
case strings.HasPrefix(rest, "name."):
|
||||
names = append(names, key)
|
||||
case versionBodyTemplateID(rest) != "":
|
||||
bodies = append(bodies, key)
|
||||
}
|
||||
}
|
||||
|
||||
for _, key := range bodies {
|
||||
if present[versionBodyTemplateID(strings.TrimPrefix(key, prefix))] {
|
||||
continue
|
||||
}
|
||||
tallyOrphan(s.removeSettled(ctx, key, settledBefore, nil), &out.OrphanBodiesRemoved, &out.OrphanBodiesFailed, &out.OrphanBodiesUnsettled)
|
||||
}
|
||||
for _, key := range names {
|
||||
// A name still naming a present header is that template's, not residue.
|
||||
owned := func(value []byte) bool { return present[string(value)] }
|
||||
tallyOrphan(s.removeSettled(ctx, key, settledBefore, owned), &out.OrphanNamesRemoved, &out.OrphanNamesFailed, &out.OrphanNamesUnsettled)
|
||||
}
|
||||
slog.InfoContext(ctx, "Launch-template account residue cleanup completed", "accountID", accountID,
|
||||
"kept", len(out.KeptTemplateIDs), "orphanBodiesRemoved", out.OrphanBodiesRemoved, "orphanNamesRemoved", out.OrphanNamesRemoved)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// orphanOutcome is what removeSettled did with one headerless record.
|
||||
type orphanOutcome int
|
||||
|
||||
const (
|
||||
orphanGone orphanOutcome = iota
|
||||
orphanRemoved
|
||||
orphanFailed
|
||||
orphanUnsettled
|
||||
orphanOwned
|
||||
)
|
||||
|
||||
// removeSettled deletes key at the revision read when it was written before
|
||||
// settledBefore and owned does not claim its value. The value is never decoded
|
||||
// or logged: a version body holds template data and user data.
|
||||
func (s *LaunchTemplateServiceImpl) removeSettled(ctx context.Context, key string, settledBefore time.Time, owned func([]byte) bool) orphanOutcome {
|
||||
entry, err := s.kv.Get(ctx, key)
|
||||
switch {
|
||||
case errors.Is(err, jetstream.ErrKeyNotFound):
|
||||
return orphanGone
|
||||
case err != nil:
|
||||
slog.WarnContext(ctx, "Launch-template account cleanup: orphan read failed", "err", err)
|
||||
return orphanFailed
|
||||
case owned != nil && owned(entry.Value()):
|
||||
return orphanOwned
|
||||
case !entry.Created().Before(settledBefore):
|
||||
return orphanUnsettled
|
||||
}
|
||||
if err := s.kv.Delete(ctx, key, jetstream.LastRevision(entry.Revision())); err != nil {
|
||||
slog.WarnContext(ctx, "Launch-template account cleanup: orphan delete failed", "err", err)
|
||||
return orphanFailed
|
||||
}
|
||||
return orphanRemoved
|
||||
}
|
||||
|
||||
func tallyOrphan(outcome orphanOutcome, removed, failed, unsettled *int) {
|
||||
switch outcome {
|
||||
case orphanRemoved:
|
||||
*removed++
|
||||
case orphanFailed:
|
||||
*failed++
|
||||
case orphanUnsettled:
|
||||
*unsettled++
|
||||
case orphanGone, orphanOwned:
|
||||
}
|
||||
}
|
||||
@@ -6,7 +6,6 @@ package launchtemplate
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
@@ -43,17 +42,17 @@ func TestScanAccountClassifiesLaunchTemplateRecords(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
live := createWithUserData(t, svc, "live")
|
||||
|
||||
// A header removed alone, as a delete interrupted after it leaves the rest.
|
||||
deleted := createWithUserData(t, svc, "deleted")
|
||||
inner := svc.kv
|
||||
deletedHeader, deletedName := kvutil.AccountKey(testAccountID, deleted), nameKey(testAccountID, "deleted")
|
||||
svc.kv = &faultyKV{KeyValue: inner, failDelete: func(key string) bool {
|
||||
return strings.HasPrefix(key, deletedHeader+".v") || key == deletedName
|
||||
}}
|
||||
_, err := svc.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(deleted)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, inner.Delete(t.Context(), kvutil.AccountKey(testAccountID, deleted)))
|
||||
|
||||
svc.kv = &faultyKV{KeyValue: inner, failPut: func(key string) bool { return isHeaderKey(key, testAccountID+".") }}
|
||||
_, err = svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
// A create whose header write and clean-up both fail.
|
||||
svc.kv = &faultyKV{KeyValue: inner,
|
||||
failPut: func(key string) bool { return isHeaderKey(key, testAccountID+".") },
|
||||
failDelete: func(string) bool { return true },
|
||||
}
|
||||
_, err := svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("interrupted"),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{UserData: aws.String(base64.StdEncoding.EncodeToString([]byte(ctUserData)))},
|
||||
}, testAccountID)
|
||||
|
||||
+240
-60
@@ -9,7 +9,9 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
@@ -21,8 +23,8 @@ import (
|
||||
)
|
||||
|
||||
// The TestCurrentBehaviour_AccountTermination_ tests pin how launch templates
|
||||
// are stored and what Describe and Delete leave behind today. They describe
|
||||
// current behaviour, not the target design; a failure means it changed.
|
||||
// are stored and what Describe leaves behind today; a failure means it
|
||||
// changed. The TestAccountTermination_ tests pin the corrected owner delete.
|
||||
|
||||
var errInjected = errors.New("injected store failure")
|
||||
|
||||
@@ -31,10 +33,31 @@ var errInjected = errors.New("injected store failure")
|
||||
type faultyKV struct {
|
||||
jetstream.KeyValue
|
||||
|
||||
failGet func(key string) bool
|
||||
failPut func(key string) bool
|
||||
failDelete func(key string) bool
|
||||
failKeys func() bool
|
||||
failGet func(key string) bool
|
||||
failPut func(key string) bool
|
||||
failDelete func(key string) bool
|
||||
failKeys func() bool
|
||||
failUpdate func(key string) bool
|
||||
beforeDelete func(key string)
|
||||
beforeCreate func(key string)
|
||||
beforeUpdate func(key string)
|
||||
}
|
||||
|
||||
func (k *faultyKV) Create(ctx context.Context, key string, value []byte, opts ...jetstream.KVCreateOpt) (uint64, error) {
|
||||
if k.beforeCreate != nil {
|
||||
k.beforeCreate(key)
|
||||
}
|
||||
return k.KeyValue.Create(ctx, key, value, opts...)
|
||||
}
|
||||
|
||||
func (k *faultyKV) Update(ctx context.Context, key string, value []byte, revision uint64) (uint64, error) {
|
||||
if k.beforeUpdate != nil {
|
||||
k.beforeUpdate(key)
|
||||
}
|
||||
if k.failUpdate != nil && k.failUpdate(key) {
|
||||
return 0, errInjected
|
||||
}
|
||||
return k.KeyValue.Update(ctx, key, value, revision)
|
||||
}
|
||||
|
||||
func (k *faultyKV) Get(ctx context.Context, key string) (jetstream.KeyValueEntry, error) {
|
||||
@@ -52,6 +75,9 @@ func (k *faultyKV) Put(ctx context.Context, key string, value []byte) (uint64, e
|
||||
}
|
||||
|
||||
func (k *faultyKV) Delete(ctx context.Context, key string, opts ...jetstream.KVDeleteOpt) error {
|
||||
if k.beforeDelete != nil {
|
||||
k.beforeDelete(key)
|
||||
}
|
||||
if k.failDelete != nil && k.failDelete(key) {
|
||||
return errInjected
|
||||
}
|
||||
@@ -138,40 +164,44 @@ func TestCurrentBehaviour_AccountTermination_LaunchTemplateStorageLayout(t *test
|
||||
assert.Equal(t, ctUserData, storedUserData(t, svc.kv, header+".v2"), "a version copied from v1 keeps its user data")
|
||||
}
|
||||
|
||||
// Pins current behaviour that differs from AWS: the header delete alone
|
||||
// decides success, and a failed name-index or version-body cleanup leaves
|
||||
// those records, user data included, with no header to find them by.
|
||||
func TestCurrentBehaviour_AccountTermination_DeleteLaunchTemplateLeavesRecordsWhenCleanupFails(t *testing.T) {
|
||||
// The header is deleted last, so a failed body, listing, name-index or header
|
||||
// delete returns an error with the template still described, and a retried
|
||||
// delete by ID removes every record, user data included.
|
||||
func TestAccountTermination_DeleteLaunchTemplateKeepsTheTemplateWhenAStepFails(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inject func(kv *faultyKV, header, nameIdx string)
|
||||
residual func(header, nameIdx string) []string
|
||||
name string
|
||||
inject func(kv *faultyKV, header, nameIdx string)
|
||||
left func(header, nameIdx string) []string
|
||||
}{
|
||||
{
|
||||
name: "version body deletes fail",
|
||||
inject: func(kv *faultyKV, header, _ string) {
|
||||
kv.failDelete = func(key string) bool { return key == header+".v1" || key == header+".v2" }
|
||||
},
|
||||
residual: func(header, _ string) []string { return []string{header + ".v1", header + ".v2"} },
|
||||
left: func(header, nameIdx string) []string {
|
||||
return []string{header, header + ".v1", header + ".v2", nameIdx}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "version listing fails",
|
||||
inject: func(kv *faultyKV, _, _ string) { kv.failKeys = func() bool { return true } },
|
||||
left: func(header, nameIdx string) []string {
|
||||
return []string{header, header + ".v1", header + ".v2", nameIdx}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "name index delete fails",
|
||||
inject: func(kv *faultyKV, _, nameIdx string) {
|
||||
kv.failDelete = func(key string) bool { return key == nameIdx }
|
||||
},
|
||||
residual: func(_, nameIdx string) []string { return []string{nameIdx} },
|
||||
left: func(header, nameIdx string) []string { return []string{header, nameIdx} },
|
||||
},
|
||||
{
|
||||
name: "version listing fails after the header delete",
|
||||
name: "header delete fails",
|
||||
inject: func(kv *faultyKV, header, _ string) {
|
||||
headerGone := false
|
||||
kv.failDelete = func(key string) bool {
|
||||
headerGone = headerGone || key == header
|
||||
return false
|
||||
}
|
||||
kv.failKeys = func() bool { return headerGone }
|
||||
kv.failDelete = func(key string) bool { return key == header }
|
||||
},
|
||||
residual: func(header, _ string) []string { return []string{header + ".v1", header + ".v2"} },
|
||||
left: func(header, _ string) []string { return []string{header} },
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
@@ -184,56 +214,85 @@ func TestCurrentBehaviour_AccountTermination_DeleteLaunchTemplateLeavesRecordsWh
|
||||
tc.inject(faulty, header, nameIdx)
|
||||
svc.kv = faulty
|
||||
|
||||
_, err := svc.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(id)}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
|
||||
svc.kv = inner
|
||||
assert.Equal(t, sortedKeys(append(tc.left(header, nameIdx), kvutil.VersionKey)...), storedKeys(t, inner))
|
||||
assert.Equal(t, []string{id}, describedTemplateIDs(t, svc, testAccountID))
|
||||
|
||||
out, err := svc.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(id)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, id, aws.StringValue(out.LaunchTemplate.LaunchTemplateId))
|
||||
|
||||
svc.kv = inner
|
||||
want := append(tc.residual(header, nameIdx), kvutil.VersionKey)
|
||||
assert.Equal(t, sortedKeys(want...), storedKeys(t, inner))
|
||||
assert.Equal(t, []string{kvutil.VersionKey}, storedKeys(t, inner))
|
||||
assert.Empty(t, describedTemplateIDs(t, svc, testAccountID))
|
||||
if slices.Contains(want, header+".v1") {
|
||||
assert.Equal(t, ctUserData, storedUserData(t, inner, header+".v1"))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A create interrupted before its header write leaves the name claim and
|
||||
// version 1 behind. A retry under the same name reclaims the name, and the
|
||||
// first attempt's body is never reachable again.
|
||||
func TestCurrentBehaviour_AccountTermination_InterruptedCreateLeavesAnOrphanBody(t *testing.T) {
|
||||
// A header rewritten and a version added while the delete runs send it back
|
||||
// through the bodies at the new revision, and the late body is removed too.
|
||||
func TestAccountTermination_DeleteLaunchTemplateRestartsWhenTheHeaderChanges(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
inner := svc.kv
|
||||
svc.kv = &faultyKV{KeyValue: inner, failPut: func(key string) bool { return isHeaderKey(key, testAccountID+".") }}
|
||||
|
||||
_, err := svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("web"),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
|
||||
UserData: aws.String(base64.StdEncoding.EncodeToString([]byte(ctUserData))),
|
||||
},
|
||||
}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
|
||||
svc.kv = inner
|
||||
keys := storedKeys(t, inner)
|
||||
require.Len(t, keys, 3, "name index, version 1 and the bucket version key")
|
||||
nameEntry, err := inner.Get(t.Context(), nameKey(testAccountID, "web"))
|
||||
header := testAccountID + "." + id
|
||||
headerEntry, err := inner.Get(t.Context(), header)
|
||||
require.NoError(t, err)
|
||||
orphanBody := testAccountID + "." + string(nameEntry.Value()) + ".v1"
|
||||
assert.Contains(t, keys, orphanBody)
|
||||
assert.Equal(t, ctUserData, storedUserData(t, inner, orphanBody))
|
||||
assert.Empty(t, describedTemplateIDs(t, svc, testAccountID))
|
||||
|
||||
retried := createTemplate(t, svc, "web", "t3.micro")
|
||||
assert.NotEqual(t, string(nameEntry.Value()), aws.StringValue(retried.LaunchTemplateId))
|
||||
assert.Contains(t, storedKeys(t, inner), orphanBody)
|
||||
assert.Equal(t, []string{aws.StringValue(retried.LaunchTemplateId)}, describedTemplateIDs(t, svc, testAccountID))
|
||||
raced := false
|
||||
svc.kv = &faultyKV{KeyValue: inner, beforeDelete: func(key string) {
|
||||
if key != header || raced {
|
||||
return
|
||||
}
|
||||
raced = true
|
||||
_, err := inner.Put(t.Context(), header, headerEntry.Value())
|
||||
require.NoError(t, err)
|
||||
_, err = inner.Put(t.Context(), header+".v3", []byte(`{"version_number":3}`))
|
||||
require.NoError(t, err)
|
||||
}}
|
||||
|
||||
_, err = svc.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(id)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, raced)
|
||||
assert.Equal(t, []string{kvutil.VersionKey}, storedKeys(t, inner))
|
||||
}
|
||||
|
||||
// Describe skips a header it cannot read or decode, and still succeeds. An
|
||||
// undecodable header also cannot be removed through DeleteLaunchTemplate.
|
||||
// A create that fails after claiming the name removes what it wrote, so no
|
||||
// body is orphaned and a retry under the same name succeeds.
|
||||
func TestAccountTermination_FailedCreateRemovesWhatItWrote(t *testing.T) {
|
||||
for _, failing := range []string{"version", "header"} {
|
||||
t.Run(failing+" write fails", func(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
inner := svc.kv
|
||||
svc.kv = &faultyKV{KeyValue: inner, failPut: func(key string) bool {
|
||||
if failing == "header" {
|
||||
return isHeaderKey(key, testAccountID+".")
|
||||
}
|
||||
return versionBodyTemplateID(strings.TrimPrefix(key, testAccountID+".")) != ""
|
||||
}}
|
||||
|
||||
_, err := svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("web"),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
|
||||
UserData: aws.String(base64.StdEncoding.EncodeToString([]byte(ctUserData))),
|
||||
},
|
||||
}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
|
||||
svc.kv = inner
|
||||
assert.Equal(t, []string{kvutil.VersionKey}, storedKeys(t, inner))
|
||||
retried := createTemplate(t, svc, "web", "t3.micro")
|
||||
assert.Equal(t, []string{aws.StringValue(retried.LaunchTemplateId)}, describedTemplateIDs(t, svc, testAccountID))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Describe skips a header it cannot read or decode, and still succeeds. Such
|
||||
// a header cannot be removed through DeleteLaunchTemplate either, and a read
|
||||
// failure is an internal error, never a missing template.
|
||||
func TestCurrentBehaviour_AccountTermination_DescribeLaunchTemplatesHidesUnreadableHeaders(t *testing.T) {
|
||||
const brokenID = "lt-0badbadbadbadbad0"
|
||||
brokenHeader := kvutil.AccountKey(testAccountID, brokenID)
|
||||
@@ -252,7 +311,7 @@ func TestCurrentBehaviour_AccountTermination_DescribeLaunchTemplatesHidesUnreada
|
||||
_, err = svc.kv.Put(t.Context(), brokenHeader, data)
|
||||
require.NoError(t, err)
|
||||
svc.kv = &faultyKV{KeyValue: svc.kv, failGet: func(key string) bool { return key == brokenHeader }}
|
||||
}, awserrors.ErrorInvalidLaunchTemplateIdNotFound},
|
||||
}, awserrors.ErrorServerInternal},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -271,3 +330,124 @@ func TestCurrentBehaviour_AccountTermination_DescribeLaunchTemplatesHidesUnreada
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Account cleanup never deletes a live template: every key of a readable
|
||||
// template stays and its ID is reported kept. Headerless bodies and names are
|
||||
// removed once settled; a record younger than the settle window, and an
|
||||
// unreadable header with its bodies and name, are kept.
|
||||
func TestAccountCleanup_KeepsTemplatesAndRemovesSettledResidue(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
live := createWithUserData(t, svc, "web")
|
||||
liveKeys := accountKeys(t, svc.kv, testAccountID)
|
||||
other := "999988887777"
|
||||
_, err := svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("theirs"), LaunchTemplateData: &ec2.RequestLaunchTemplateData{ImageId: aws.String("ami-123")},
|
||||
}, other)
|
||||
require.NoError(t, err)
|
||||
orphanBody := versionKey(testAccountID, "lt-0orphan000000000", 1)
|
||||
orphanName := nameKey(testAccountID, "gone")
|
||||
brokenHeader := kvutil.AccountKey(testAccountID, "lt-0broken000000000")
|
||||
brokenBody := versionKey(testAccountID, "lt-0broken000000000", 1)
|
||||
brokenName := nameKey(testAccountID, "broken")
|
||||
for key, value := range map[string]string{
|
||||
orphanBody: `{"version_number":1}`, orphanName: "lt-0orphan000000000",
|
||||
brokenHeader: "{not json", brokenBody: `{"version_number":1}`, brokenName: "lt-0broken000000000",
|
||||
} {
|
||||
_, err := svc.kv.Put(t.Context(), key, []byte(value))
|
||||
require.NoError(t, err)
|
||||
}
|
||||
keep := sortedKeys(append(liveKeys, brokenHeader, brokenBody, brokenName)...)
|
||||
|
||||
out, err := svc.CleanupAccountResidue(t.Context(), testAccountID, time.Now().Add(-time.Hour))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{live}, out.KeptTemplateIDs)
|
||||
assert.Equal(t, 1, out.OrphanBodiesUnsettled)
|
||||
assert.Equal(t, 1, out.OrphanNamesUnsettled)
|
||||
assert.Equal(t, sortedKeys(append(keep, orphanBody, orphanName)...), accountKeys(t, svc.kv, testAccountID))
|
||||
|
||||
out, err = svc.CleanupAccountResidue(t.Context(), testAccountID, time.Now().Add(time.Second))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{live}, out.KeptTemplateIDs)
|
||||
assert.Equal(t, 1, out.OrphanBodiesRemoved)
|
||||
assert.Equal(t, 1, out.OrphanNamesRemoved)
|
||||
assert.Equal(t, keep, accountKeys(t, svc.kv, testAccountID))
|
||||
assert.Equal(t, []string{live}, describedTemplateIDs(t, svc, testAccountID))
|
||||
assert.Len(t, describedTemplateIDs(t, svc, other), 1)
|
||||
}
|
||||
|
||||
// A public delete interrupted at its header leaves the template live, so the
|
||||
// account cleanup keeps it rather than finishing it; the public retry does.
|
||||
func TestAccountCleanup_InterruptedTemplateDeleteIsLeftToThePublicRetry(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
header := kvutil.AccountKey(testAccountID, id)
|
||||
kv := svc.kv
|
||||
svc.kv = &faultyKV{KeyValue: kv, failDelete: func(key string) bool { return key == header }}
|
||||
_, err := svc.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(id)}, testAccountID)
|
||||
require.Error(t, err)
|
||||
svc.kv = kv
|
||||
require.Equal(t, []string{header}, accountKeys(t, kv, testAccountID), "only the header, the delete's anchor, remains")
|
||||
|
||||
out, err := svc.CleanupAccountResidue(t.Context(), testAccountID, time.Now().Add(time.Second))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{id}, out.KeptTemplateIDs)
|
||||
assert.Equal(t, []string{header}, accountKeys(t, kv, testAccountID))
|
||||
|
||||
_, err = svc.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(id)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, accountKeys(t, kv, testAccountID))
|
||||
}
|
||||
|
||||
// A listing failure is returned and nothing is deleted.
|
||||
func TestAccountCleanup_TemplateListingFailureDeletesNothing(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
kv := svc.kv
|
||||
svc.kv = &faultyKV{KeyValue: kv, failKeys: func() bool { return true }}
|
||||
|
||||
_, err := svc.CleanupAccountResidue(t.Context(), testAccountID, time.Now())
|
||||
require.Error(t, err)
|
||||
svc.kv = kv
|
||||
assert.Equal(t, []string{id}, describedTemplateIDs(t, svc, testAccountID))
|
||||
}
|
||||
|
||||
func accountKeys(t *testing.T, kv jetstream.KeyValue, account string) []string {
|
||||
t.Helper()
|
||||
var mine []string
|
||||
for _, key := range storedKeys(t, kv) {
|
||||
if strings.HasPrefix(key, account+".") {
|
||||
mine = append(mine, key)
|
||||
}
|
||||
}
|
||||
return mine
|
||||
}
|
||||
|
||||
// A settled orphan the store cannot read or delete is counted failed, never
|
||||
// removed, and is still there for the next run.
|
||||
func TestAccountCleanup_OrphanReadOrDeleteFailureIsCountedFailed(t *testing.T) {
|
||||
for name, fault := range map[string]func(kv *faultyKV, key string){
|
||||
"read": func(kv *faultyKV, key string) { kv.failGet = func(k string) bool { return k == key } },
|
||||
"delete": func(kv *faultyKV, key string) { kv.failDelete = func(k string) bool { return k == key } },
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
body := versionKey(testAccountID, "lt-0orphan000000000", 1)
|
||||
nameIdx := nameKey(testAccountID, "gone")
|
||||
for key, value := range map[string]string{body: `{"version_number":1}`, nameIdx: "lt-0orphan000000000"} {
|
||||
_, err := svc.kv.Put(t.Context(), key, []byte(value))
|
||||
require.NoError(t, err)
|
||||
}
|
||||
inner := svc.kv
|
||||
faulty := &faultyKV{KeyValue: inner}
|
||||
fault(faulty, body)
|
||||
svc.kv = faulty
|
||||
|
||||
out, err := svc.CleanupAccountResidue(t.Context(), testAccountID, time.Now().Add(time.Second))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 1, out.OrphanBodiesFailed)
|
||||
assert.Zero(t, out.OrphanBodiesRemoved)
|
||||
assert.Equal(t, 1, out.OrphanNamesRemoved, "a fault on one record does not stop the rest")
|
||||
assert.Equal(t, []string{body}, accountKeys(t, inner, testAccountID))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -52,6 +52,9 @@ type LaunchTemplateServiceImpl struct {
|
||||
config *config.Config
|
||||
natsConn *nats.Conn
|
||||
kv jetstream.KeyValue
|
||||
|
||||
// Optional: nil leaves the central tag index untouched by delete.
|
||||
centralTags CentralTagStore
|
||||
}
|
||||
|
||||
// NewLaunchTemplateServiceImplWithNATS creates a launch template service with NATS JetStream.
|
||||
@@ -103,7 +106,12 @@ func versionPrefix(accountID, ltID string) string {
|
||||
func (s *LaunchTemplateServiceImpl) getHeaderByID(ctx context.Context, accountID, ltID string) (*LaunchTemplateHeader, jetstream.KeyValueEntry, error) {
|
||||
entry, err := s.kv.Get(ctx, kvutil.AccountKey(accountID, ltID))
|
||||
if err != nil {
|
||||
return nil, nil, errors.New(awserrors.ErrorInvalidLaunchTemplateIdNotFound)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil, nil, errors.New(awserrors.ErrorInvalidLaunchTemplateIdNotFound)
|
||||
}
|
||||
// A header the store cannot return now is not proven absent.
|
||||
slog.ErrorContext(ctx, "Failed to read launch template header", "launchTemplateId", ltID, "err", err)
|
||||
return nil, nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
var h LaunchTemplateHeader
|
||||
if err := json.Unmarshal(entry.Value(), &h); err != nil {
|
||||
@@ -117,12 +125,19 @@ func (s *LaunchTemplateServiceImpl) getHeaderByID(ctx context.Context, accountID
|
||||
func (s *LaunchTemplateServiceImpl) getHeaderByName(ctx context.Context, accountID, name string) (*LaunchTemplateHeader, jetstream.KeyValueEntry, error) {
|
||||
entry, err := s.kv.Get(ctx, nameKey(accountID, name))
|
||||
if err != nil {
|
||||
return nil, nil, errors.New(awserrors.ErrorInvalidLaunchTemplateNameNotFoundException)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil, nil, errors.New(awserrors.ErrorInvalidLaunchTemplateNameNotFoundException)
|
||||
}
|
||||
slog.ErrorContext(ctx, "Failed to read launch template name index", "err", err)
|
||||
return nil, nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
ltID := string(entry.Value())
|
||||
h, hentry, err := s.getHeaderByID(ctx, accountID, ltID)
|
||||
if err != nil {
|
||||
return nil, nil, errors.New(awserrors.ErrorInvalidLaunchTemplateNameNotFoundException)
|
||||
if err.Error() == awserrors.ErrorInvalidLaunchTemplateIdNotFound {
|
||||
return nil, nil, errors.New(awserrors.ErrorInvalidLaunchTemplateNameNotFoundException)
|
||||
}
|
||||
return nil, nil, err
|
||||
}
|
||||
return h, hentry, nil
|
||||
}
|
||||
@@ -291,6 +306,7 @@ func (s *LaunchTemplateServiceImpl) CreateLaunchTemplate(ctx context.Context, in
|
||||
Data: data,
|
||||
}
|
||||
if err := s.putVersion(ctx, accountID, ltID, &rec); err != nil {
|
||||
s.abandonCreate(ctx, accountID, name, ltID, false)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -304,6 +320,7 @@ func (s *LaunchTemplateServiceImpl) CreateLaunchTemplate(ctx context.Context, in
|
||||
Tags: awstags.Extract(input.TagSpecifications, launchTemplateTagResourceType),
|
||||
}
|
||||
if err := s.putHeader(ctx, accountID, &header); err != nil {
|
||||
s.abandonCreate(ctx, accountID, name, ltID, true)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -314,6 +331,39 @@ func (s *LaunchTemplateServiceImpl) CreateLaunchTemplate(ctx context.Context, in
|
||||
}, nil
|
||||
}
|
||||
|
||||
// abandonCreate removes what a failed create wrote, newest first. A failure
|
||||
// is logged; what remains has no header, and the account cleanup removes it
|
||||
// once it is older than an in-flight create could be.
|
||||
func (s *LaunchTemplateServiceImpl) abandonCreate(ctx context.Context, accountID, name, ltID string, wroteVersion bool) {
|
||||
if wroteVersion {
|
||||
if err := s.kv.Delete(ctx, versionKey(accountID, ltID, 1)); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateLaunchTemplate: failed to remove version 1 after a failed create", "launchTemplateId", ltID, "err", err)
|
||||
}
|
||||
}
|
||||
if err := s.releaseName(ctx, accountID, name, ltID); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateLaunchTemplate: failed to release the name after a failed create", "launchTemplateId", ltID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// releaseName removes the name index only while it still names ltID, at the
|
||||
// revision read, so a name another template now holds is never released.
|
||||
func (s *LaunchTemplateServiceImpl) releaseName(ctx context.Context, accountID, name, ltID string) error {
|
||||
key := nameKey(accountID, name)
|
||||
entry, err := s.kv.Get(ctx, key)
|
||||
switch {
|
||||
case errors.Is(err, jetstream.ErrKeyNotFound):
|
||||
return nil
|
||||
case err != nil:
|
||||
return fmt.Errorf("read name index: %w", err)
|
||||
case string(entry.Value()) != ltID:
|
||||
return nil
|
||||
}
|
||||
if err := s.kv.Delete(ctx, key, jetstream.LastRevision(entry.Revision())); err != nil {
|
||||
return fmt.Errorf("delete name index: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// errNameInUse is AWS's error for a launch template name already taken.
|
||||
func errNameInUse() error {
|
||||
return awserrors.Errorf(awserrors.ErrorInvalidLaunchTemplateNameAlreadyExistsException, "Launch template name already in use.")
|
||||
@@ -377,7 +427,7 @@ func (s *LaunchTemplateServiceImpl) CreateLaunchTemplateVersion(ctx context.Cont
|
||||
if input.LaunchTemplateData == nil {
|
||||
return nil, errors.New(awserrors.ErrorMissingParameter)
|
||||
}
|
||||
header, _, err := s.resolveHeader(ctx, accountID, input.LaunchTemplateId, input.LaunchTemplateName)
|
||||
header, entry, err := s.resolveHeader(ctx, accountID, input.LaunchTemplateId, input.LaunchTemplateName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -429,6 +479,13 @@ func (s *LaunchTemplateServiceImpl) CreateLaunchTemplateVersion(ctx context.Cont
|
||||
slog.ErrorContext(ctx, "CreateLaunchTemplateVersion: version-key create failed", "next", next, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if err := s.commitVersion(ctx, accountID, header.LaunchTemplateId, entry); err != nil {
|
||||
s.abandonVersion(ctx, accountID, header.LaunchTemplateId, next)
|
||||
if errors.Is(err, errTemplateGone) {
|
||||
return nil, templateNotFound(input.LaunchTemplateId, input.LaunchTemplateName)
|
||||
}
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
slog.InfoContext(ctx, "CreateLaunchTemplateVersion completed", "launchTemplateId", header.LaunchTemplateId, "version", next, "accountID", accountID)
|
||||
return &ec2.CreateLaunchTemplateVersionOutput{
|
||||
LaunchTemplateVersion: versionToEC2(header, &rec),
|
||||
@@ -439,6 +496,53 @@ func (s *LaunchTemplateServiceImpl) CreateLaunchTemplateVersion(ctx context.Cont
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// errTemplateGone: the header was deleted while a version was being added.
|
||||
var errTemplateGone = errors.New("launch template deleted")
|
||||
|
||||
// commitVersion rewrites the header's current value under a revision guard
|
||||
// after a new body is written. A delete that removed the header at any earlier
|
||||
// revision makes it fail, so the body is never left without its template.
|
||||
func (s *LaunchTemplateServiceImpl) commitVersion(ctx context.Context, accountID, ltID string, entry jetstream.KeyValueEntry) error {
|
||||
key := kvutil.AccountKey(accountID, ltID)
|
||||
for range maxVersionRetries {
|
||||
_, err := s.kv.Update(ctx, key, entry.Value(), entry.Revision())
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
if !errors.Is(err, jetstream.ErrKeyRevisionMismatch) && !errors.Is(err, jetstream.ErrKeyExists) {
|
||||
slog.ErrorContext(ctx, "CreateLaunchTemplateVersion: header commit failed", "launchTemplateId", ltID, "err", err)
|
||||
return err
|
||||
}
|
||||
next, gerr := s.kv.Get(ctx, key)
|
||||
if errors.Is(gerr, jetstream.ErrKeyNotFound) {
|
||||
return errTemplateGone
|
||||
}
|
||||
if gerr != nil {
|
||||
return gerr
|
||||
}
|
||||
entry = next
|
||||
}
|
||||
return errors.New("header commit retries exhausted")
|
||||
}
|
||||
|
||||
// abandonVersion removes a body whose header commit failed. A failure is
|
||||
// logged and leaves a headerless body, which no ID, name or launch resolves.
|
||||
func (s *LaunchTemplateServiceImpl) abandonVersion(ctx context.Context, accountID, ltID string, n int64) {
|
||||
if err := s.kv.Delete(ctx, versionKey(accountID, ltID, n)); err != nil {
|
||||
slog.ErrorContext(ctx, "CreateLaunchTemplateVersion: failed to remove an uncommitted version", "launchTemplateId", ltID, "version", n, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// templateNotFound is the NotFound error for the template the request named.
|
||||
func templateNotFound(ltID, name *string) error {
|
||||
if id := aws.StringValue(ltID); id != "" {
|
||||
return awserrors.Errorf(awserrors.ErrorInvalidLaunchTemplateIdNotFound,
|
||||
"The specified launch template, with template ID %s, does not exist.", id)
|
||||
}
|
||||
return awserrors.Errorf(awserrors.ErrorInvalidLaunchTemplateNameNotFoundException,
|
||||
"The specified launch template, with template name %s, does not exist.", aws.StringValue(name))
|
||||
}
|
||||
|
||||
// --- ModifyLaunchTemplate ---
|
||||
|
||||
// ModifyLaunchTemplate implements the EC2 action. ClientToken is accepted but not stored: this only
|
||||
@@ -481,8 +585,11 @@ func (s *LaunchTemplateServiceImpl) ModifyLaunchTemplate(ctx context.Context, in
|
||||
|
||||
// --- DeleteLaunchTemplate ---
|
||||
|
||||
// DeleteLaunchTemplate deletes a template and every version. A store failure
|
||||
// returns an error with the header still in place, so the template stays
|
||||
// visible and a retried delete finishes it.
|
||||
func (s *LaunchTemplateServiceImpl) DeleteLaunchTemplate(ctx context.Context, input *ec2.DeleteLaunchTemplateInput, accountID string) (*ec2.DeleteLaunchTemplateOutput, error) {
|
||||
header, _, err := s.resolveHeader(ctx, accountID, input.LaunchTemplateId, input.LaunchTemplateName)
|
||||
header, entry, err := s.resolveHeader(ctx, accountID, input.LaunchTemplateId, input.LaunchTemplateName)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -493,28 +600,92 @@ func (s *LaunchTemplateServiceImpl) DeleteLaunchTemplate(ctx context.Context, in
|
||||
}
|
||||
out := headerToEC2(header, latest)
|
||||
|
||||
// Delete the header first: the template immediately vanishes from every
|
||||
// describe. Version bodies and the name index are best-effort cleanup.
|
||||
if err := s.kv.Delete(ctx, kvutil.AccountKey(accountID, header.LaunchTemplateId)); err != nil {
|
||||
if err := s.deleteTemplate(ctx, accountID, header, entry); err != nil {
|
||||
slog.ErrorContext(ctx, "DeleteLaunchTemplate failed", "launchTemplateId", header.LaunchTemplateId, "accountID", accountID, "err", err)
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
if err := s.kv.Delete(ctx, nameKey(accountID, header.LaunchTemplateName)); err != nil {
|
||||
slog.WarnContext(ctx, "DeleteLaunchTemplate: name index cleanup failed", "name", header.LaunchTemplateName, "err", err)
|
||||
}
|
||||
nums, err := s.listVersionNumbers(ctx, accountID, header.LaunchTemplateId)
|
||||
if err != nil {
|
||||
slog.WarnContext(ctx, "DeleteLaunchTemplate: version enumeration failed, bodies left in place", "launchTemplateId", header.LaunchTemplateId, "err", err)
|
||||
}
|
||||
for _, n := range nums {
|
||||
if err := s.kv.Delete(ctx, versionKey(accountID, header.LaunchTemplateId, n)); err != nil {
|
||||
slog.WarnContext(ctx, "DeleteLaunchTemplate: version cleanup failed", "launchTemplateId", header.LaunchTemplateId, "version", n, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "DeleteLaunchTemplate completed", "launchTemplateId", header.LaunchTemplateId, "accountID", accountID)
|
||||
return &ec2.DeleteLaunchTemplateOutput{LaunchTemplate: out}, nil
|
||||
}
|
||||
|
||||
// maxHeaderDeleteAttempts bounds restarts when the header changes mid-delete.
|
||||
const maxHeaderDeleteAttempts = 3
|
||||
|
||||
// deleteTemplate removes every version body, then the name index, then the
|
||||
// header at the revision read, and last any body a concurrent version create
|
||||
// added meanwhile. The header goes last so every earlier failure is retryable
|
||||
// through the public delete; a header changed in between restarts the sweep.
|
||||
func (s *LaunchTemplateServiceImpl) deleteTemplate(ctx context.Context, accountID string, header *LaunchTemplateHeader, entry jetstream.KeyValueEntry) error {
|
||||
ltID := header.LaunchTemplateId
|
||||
headerKey := kvutil.AccountKey(accountID, ltID)
|
||||
for attempt := 1; ; attempt++ {
|
||||
if err := s.deleteVersionBodies(ctx, accountID, ltID); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.releaseName(ctx, accountID, header.LaunchTemplateName, ltID); err != nil {
|
||||
return err
|
||||
}
|
||||
err := s.kv.Delete(ctx, headerKey, jetstream.LastRevision(entry.Revision()))
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if (!errors.Is(err, jetstream.ErrKeyRevisionMismatch) && !errors.Is(err, jetstream.ErrKeyExists)) || attempt >= maxHeaderDeleteAttempts {
|
||||
return fmt.Errorf("delete header: %w", err)
|
||||
}
|
||||
next, gerr := s.kv.Get(ctx, headerKey)
|
||||
if errors.Is(gerr, jetstream.ErrKeyNotFound) {
|
||||
break
|
||||
}
|
||||
if gerr != nil {
|
||||
return fmt.Errorf("re-read header: %w", gerr)
|
||||
}
|
||||
entry = next
|
||||
}
|
||||
s.clearRecordTags(ctx, accountID, ltID)
|
||||
// A version create in flight fails its header commit and removes its own
|
||||
// body; this sweep catches one that committed nothing before stopping.
|
||||
return s.deleteVersionBodies(ctx, accountID, ltID)
|
||||
}
|
||||
|
||||
// deleteVersionBodies deletes every version body of ltID, stopping at the
|
||||
// first failure so the caller can keep the header for a retry.
|
||||
func (s *LaunchTemplateServiceImpl) deleteVersionBodies(ctx context.Context, accountID, ltID string) error {
|
||||
nums, err := s.listVersionNumbers(ctx, accountID, ltID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list versions: %w", err)
|
||||
}
|
||||
for _, n := range nums {
|
||||
if err := s.kv.Delete(ctx, versionKey(accountID, ltID, n)); err != nil {
|
||||
return fmt.Errorf("delete version %d: %w", n, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CentralTagStore clears a deleted template's entry from the central EC2 tag
|
||||
// index. Implemented by domains/ec2/tags.TagsServiceImpl.
|
||||
type CentralTagStore interface {
|
||||
DeleteAllTags(ctx context.Context, accountID, resourceID string) error
|
||||
}
|
||||
|
||||
// SetCentralTagStore injects the central tag index that delete clears.
|
||||
func (s *LaunchTemplateServiceImpl) SetCentralTagStore(st CentralTagStore) {
|
||||
s.centralTags = st
|
||||
}
|
||||
|
||||
// clearRecordTags drops a deleted template's tag-index entry. It runs only
|
||||
// after the header delete committed; a failure is logged, as a retried delete
|
||||
// would find nothing, and the entry is left as tag residue.
|
||||
func (s *LaunchTemplateServiceImpl) clearRecordTags(ctx context.Context, accountID, ltID string) {
|
||||
if s.centralTags == nil {
|
||||
return
|
||||
}
|
||||
if err := s.centralTags.DeleteAllTags(ctx, accountID, ltID); err != nil {
|
||||
slog.ErrorContext(ctx, "central tag store clear failed", "launchTemplateId", ltID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// --- DeleteLaunchTemplateVersions ---
|
||||
|
||||
func (s *LaunchTemplateServiceImpl) DeleteLaunchTemplateVersions(ctx context.Context, input *ec2.DeleteLaunchTemplateVersionsInput, accountID string) (*ec2.DeleteLaunchTemplateVersionsOutput, error) {
|
||||
|
||||
@@ -0,0 +1,235 @@
|
||||
package launchtemplate
|
||||
|
||||
//test:in-package — the interleavings are forced by wrapping the service's
|
||||
// unexported KV handle.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/foundation/aws/errors"
|
||||
"github.com/mulgadc/spinifex/spinifex/foundation/state/kvutil"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// raceDelete runs a whole DeleteLaunchTemplate on the unwrapped bucket, so the
|
||||
// hook that calls it is not re-entered.
|
||||
func raceDelete(t *testing.T, svc *LaunchTemplateServiceImpl, id string) {
|
||||
t.Helper()
|
||||
other := &LaunchTemplateServiceImpl{kv: svc.kv.(*faultyKV).KeyValue}
|
||||
_, err := other.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(id)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func newVersionInput(id string) *ec2.CreateLaunchTemplateVersionInput {
|
||||
return &ec2.CreateLaunchTemplateVersionInput{
|
||||
LaunchTemplateId: aws.String(id),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{InstanceType: aws.String("t3.small")},
|
||||
}
|
||||
}
|
||||
|
||||
// A version create paused before it writes its body or before it commits the
|
||||
// header, while a whole delete runs, fails NotFound and leaves nothing behind.
|
||||
func TestAccountTermination_VersionCreateRacingADeleteLeavesNoBody(t *testing.T) {
|
||||
for _, pause := range []string{"before-body", "before-commit"} {
|
||||
t.Run(pause, func(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
header := kvutil.AccountKey(testAccountID, id)
|
||||
raced := false
|
||||
race := func(key string, at string) {
|
||||
if raced || pause != at {
|
||||
return
|
||||
}
|
||||
if (at == "before-body" && key == versionKey(testAccountID, id, 3)) || (at == "before-commit" && key == header) {
|
||||
raced = true
|
||||
raceDelete(t, svc, id)
|
||||
}
|
||||
}
|
||||
svc.kv = &faultyKV{
|
||||
KeyValue: svc.kv,
|
||||
beforeCreate: func(key string) { race(key, "before-body") },
|
||||
beforeUpdate: func(key string) { race(key, "before-commit") },
|
||||
}
|
||||
|
||||
_, err := svc.CreateLaunchTemplateVersion(context.Background(), newVersionInput(id), testAccountID)
|
||||
require.True(t, raced)
|
||||
require.Error(t, err)
|
||||
code, _ := awserrors.ResolveErrorCode(err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidLaunchTemplateIdNotFound, code)
|
||||
assert.Empty(t, accountKeys(t, svc.kv, testAccountID), "no headerless body is left")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A version that commits before the delete removes the header moves the
|
||||
// header revision, so the delete restarts and removes the new body too.
|
||||
func TestAccountTermination_DeleteRacingACommittedVersionRemovesIt(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
inner := svc.kv
|
||||
header := kvutil.AccountKey(testAccountID, id)
|
||||
raced := false
|
||||
svc.kv = &faultyKV{KeyValue: inner, beforeDelete: func(key string) {
|
||||
if key != header || raced {
|
||||
return
|
||||
}
|
||||
raced = true
|
||||
other := &LaunchTemplateServiceImpl{kv: inner}
|
||||
_, err := other.CreateLaunchTemplateVersion(context.Background(), newVersionInput(id), testAccountID)
|
||||
require.NoError(t, err)
|
||||
}}
|
||||
|
||||
_, err := svc.DeleteLaunchTemplate(context.Background(), &ec2.DeleteLaunchTemplateInput{LaunchTemplateId: aws.String(id)}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, raced)
|
||||
assert.Empty(t, accountKeys(t, inner, testAccountID))
|
||||
}
|
||||
|
||||
// A header commit that fails removes the body it guarded and returns an
|
||||
// error; the template keeps exactly the versions it had.
|
||||
func TestAccountTermination_FailedVersionCommitRemovesTheBody(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
before := accountKeys(t, svc.kv, testAccountID)
|
||||
header := kvutil.AccountKey(testAccountID, id)
|
||||
svc.kv = &faultyKV{KeyValue: svc.kv, failUpdate: func(key string) bool { return key == header }}
|
||||
|
||||
_, err := svc.CreateLaunchTemplateVersion(context.Background(), newVersionInput(id), testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorServerInternal, err.Error())
|
||||
assert.Equal(t, before, accountKeys(t, svc.kv, testAccountID))
|
||||
}
|
||||
|
||||
// A version create racing a default-version change commits at the new header
|
||||
// revision and keeps the change.
|
||||
func TestAccountTermination_VersionCommitKeepsAConcurrentHeaderChange(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
inner := svc.kv
|
||||
header := kvutil.AccountKey(testAccountID, id)
|
||||
raced := false
|
||||
svc.kv = &faultyKV{KeyValue: inner, beforeUpdate: func(key string) {
|
||||
if key != header || raced {
|
||||
return
|
||||
}
|
||||
raced = true
|
||||
other := &LaunchTemplateServiceImpl{kv: inner}
|
||||
_, err := other.ModifyLaunchTemplate(context.Background(), &ec2.ModifyLaunchTemplateInput{
|
||||
LaunchTemplateId: aws.String(id), DefaultVersion: aws.String("2"),
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
}}
|
||||
|
||||
out, err := svc.CreateLaunchTemplateVersion(context.Background(), newVersionInput(id), testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, raced)
|
||||
assert.Equal(t, int64(3), aws.Int64Value(out.LaunchTemplateVersion.VersionNumber))
|
||||
h, _, err := svc.getHeaderByID(context.Background(), testAccountID, id)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int64(2), h.DefaultVersionNumber)
|
||||
}
|
||||
|
||||
// Template data carrying a key name and a placement group reaches the
|
||||
// RunInstances input for $Default, $Latest and an explicit version.
|
||||
func TestCurrentBehaviour_AccountTermination_TemplateKeyAndGroupReachRunInstances(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
out, err := svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("web"),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
|
||||
ImageId: aws.String("ami-123"), KeyName: aws.String("v1-key"),
|
||||
Placement: &ec2.LaunchTemplatePlacementRequest{GroupName: aws.String("v1-group")},
|
||||
},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
id := aws.StringValue(out.LaunchTemplate.LaunchTemplateId)
|
||||
_, err = svc.CreateLaunchTemplateVersion(context.Background(), &ec2.CreateLaunchTemplateVersionInput{
|
||||
LaunchTemplateId: aws.String(id),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
|
||||
ImageId: aws.String("ami-123"), KeyName: aws.String("v2-key"),
|
||||
Placement: &ec2.LaunchTemplatePlacementRequest{GroupName: aws.String("v2-group")},
|
||||
},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
for version, want := range map[string]string{"$Default": "v1", "$Latest": "v2", "2": "v2", "1": "v1"} {
|
||||
in := &ec2.RunInstancesInput{LaunchTemplate: &ec2.LaunchTemplateSpecification{LaunchTemplateId: aws.String(id), Version: aws.String(version)}}
|
||||
require.NoError(t, ExpandRunInstances(context.Background(), svc, in, testAccountID), version)
|
||||
assert.Equal(t, want+"-key", aws.StringValue(in.KeyName), version)
|
||||
require.NotNil(t, in.Placement, version)
|
||||
assert.Equal(t, want+"-group", aws.StringValue(in.Placement.GroupName), version)
|
||||
}
|
||||
}
|
||||
|
||||
// The reference reader returns what every version of every live template
|
||||
// names, and nothing from headerless bodies or other accounts.
|
||||
func TestAccountTemplateReferences_ReadsEveryVersionOfLiveTemplates(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
out, err := svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("web"),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{KeyName: aws.String("default-key")},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
id := aws.StringValue(out.LaunchTemplate.LaunchTemplateId)
|
||||
_, err = svc.CreateLaunchTemplateVersion(context.Background(), &ec2.CreateLaunchTemplateVersionInput{
|
||||
LaunchTemplateId: aws.String(id),
|
||||
LaunchTemplateData: &ec2.RequestLaunchTemplateData{
|
||||
KeyName: aws.String("later-key"),
|
||||
Placement: &ec2.LaunchTemplatePlacementRequest{GroupName: aws.String("later-group"), GroupId: aws.String("pg-0123456789abcdef0")},
|
||||
},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
_, err = svc.CreateLaunchTemplate(context.Background(), &ec2.CreateLaunchTemplateInput{
|
||||
LaunchTemplateName: aws.String("theirs"), LaunchTemplateData: &ec2.RequestLaunchTemplateData{KeyName: aws.String("their-key")},
|
||||
}, "999988887777")
|
||||
require.NoError(t, err)
|
||||
_, err = svc.kv.Put(t.Context(), versionKey(testAccountID, "lt-0orphan000000000", 1), []byte(`{"version_number":1,"data":{"KeyName":"orphan-key"}}`))
|
||||
require.NoError(t, err)
|
||||
|
||||
refs, err := svc.AccountTemplateReferences(t.Context(), testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.ElementsMatch(t, []string{"default-key", "later-key"}, refs.KeyNames)
|
||||
assert.Equal(t, []string{"later-group"}, refs.PlacementGroupNames)
|
||||
assert.Equal(t, []string{"pg-0123456789abcdef0"}, refs.PlacementGroupIDs)
|
||||
}
|
||||
|
||||
// A header or version the reader cannot read or decode, or a listing
|
||||
// failure, is an error: the unread record may name the resource.
|
||||
func TestAccountTemplateReferences_FailsClosed(t *testing.T) {
|
||||
tests := map[string]func(t *testing.T, svc *LaunchTemplateServiceImpl, id string){
|
||||
"listing fails": func(t *testing.T, svc *LaunchTemplateServiceImpl, _ string) {
|
||||
svc.kv = &faultyKV{KeyValue: svc.kv, failKeys: func() bool { return true }}
|
||||
},
|
||||
"version unreadable": func(t *testing.T, svc *LaunchTemplateServiceImpl, id string) {
|
||||
svc.kv = &faultyKV{KeyValue: svc.kv, failGet: func(key string) bool { return key == versionKey(testAccountID, id, 2) }}
|
||||
},
|
||||
"version undecodable": func(t *testing.T, svc *LaunchTemplateServiceImpl, id string) {
|
||||
_, err := svc.kv.Put(t.Context(), versionKey(testAccountID, id, 2), []byte("{not json"))
|
||||
require.NoError(t, err)
|
||||
},
|
||||
"version without data": func(t *testing.T, svc *LaunchTemplateServiceImpl, id string) {
|
||||
_, err := svc.kv.Put(t.Context(), versionKey(testAccountID, id, 2), []byte(`{"version_number":2}`))
|
||||
require.NoError(t, err)
|
||||
},
|
||||
"header unreadable": func(t *testing.T, svc *LaunchTemplateServiceImpl, id string) {
|
||||
svc.kv = &faultyKV{KeyValue: svc.kv, failGet: func(key string) bool { return key == kvutil.AccountKey(testAccountID, id) }}
|
||||
},
|
||||
"header undecodable": func(t *testing.T, svc *LaunchTemplateServiceImpl, _ string) {
|
||||
_, err := svc.kv.Put(t.Context(), kvutil.AccountKey(testAccountID, "lt-0broken000000000"), []byte("{not json"))
|
||||
require.NoError(t, err)
|
||||
},
|
||||
}
|
||||
for name, seed := range tests {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
id := createWithUserData(t, svc, "web")
|
||||
seed(t, svc, id)
|
||||
refs, err := svc.AccountTemplateReferences(t.Context(), testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Nil(t, refs)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
package launchtemplate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/mulgadc/spinifex/spinifex/foundation/state/kvutil"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// TemplateReferences are the key-pair names, placement-group names and
|
||||
// placement-group IDs that the account's live launch-template versions carry.
|
||||
// They stay inside EC2; callers must not report them.
|
||||
type TemplateReferences struct {
|
||||
KeyNames []string
|
||||
PlacementGroupNames []string
|
||||
PlacementGroupIDs []string
|
||||
}
|
||||
|
||||
// AccountTemplateReferences reads every version body of every live template
|
||||
// in the account, since RunInstances may resolve any of them by $Default,
|
||||
// $Latest or number. It is complete or an error: an unlistable bucket, or a
|
||||
// header or body it cannot read or decode, is an error, because the unread
|
||||
// record may name the resource. Headerless bodies resolve from no launch.
|
||||
func (s *LaunchTemplateServiceImpl) AccountTemplateReferences(ctx context.Context, accountID string) (*TemplateReferences, error) {
|
||||
keys, err := kvutil.Keys(ctx, s.kv)
|
||||
if err != nil && !errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, fmt.Errorf("list launch-template keys: %w", err)
|
||||
}
|
||||
prefix := accountID + "."
|
||||
var live []string
|
||||
for _, key := range keys {
|
||||
if !isHeaderKey(key, prefix) {
|
||||
continue
|
||||
}
|
||||
entry, err := s.kv.Get(ctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return nil, errors.New("launch-template header unreadable")
|
||||
}
|
||||
var header LaunchTemplateHeader
|
||||
if json.Unmarshal(entry.Value(), &header) != nil || prefix+header.LaunchTemplateId != key {
|
||||
return nil, errors.New("launch-template header undecodable")
|
||||
}
|
||||
live = append(live, header.LaunchTemplateId)
|
||||
}
|
||||
|
||||
out := &TemplateReferences{}
|
||||
for _, ltID := range live {
|
||||
bodyPrefix := versionPrefix(accountID, ltID)
|
||||
for _, key := range keys {
|
||||
if !strings.HasPrefix(key, bodyPrefix) {
|
||||
continue
|
||||
}
|
||||
if err := s.addVersionReferences(ctx, key, out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// addVersionReferences adds what one version body would launch with. The
|
||||
// body is never logged: it holds template data and user data.
|
||||
func (s *LaunchTemplateServiceImpl) addVersionReferences(ctx context.Context, key string, out *TemplateReferences) error {
|
||||
entry, err := s.kv.Get(ctx, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return errors.New("launch-template version unreadable")
|
||||
}
|
||||
var rec LaunchTemplateVersionRec
|
||||
if json.Unmarshal(entry.Value(), &rec) != nil || rec.Data == nil {
|
||||
return errors.New("launch-template version undecodable")
|
||||
}
|
||||
if name := aws.StringValue(rec.Data.KeyName); name != "" {
|
||||
out.KeyNames = append(out.KeyNames, name)
|
||||
}
|
||||
if p := rec.Data.Placement; p != nil {
|
||||
if name := aws.StringValue(p.GroupName); name != "" {
|
||||
out.PlacementGroupNames = append(out.PlacementGroupNames, name)
|
||||
}
|
||||
if id := aws.StringValue(p.GroupId); id != "" {
|
||||
out.PlacementGroupIDs = append(out.PlacementGroupIDs, id)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package placementgroup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/foundation/state/kvutil"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// BlockedGroup is a group the owner refused to delete because its member map
|
||||
// still names instances. Only the count is kept; member IDs never leave.
|
||||
type BlockedGroup struct {
|
||||
GroupID string
|
||||
MemberCount int
|
||||
}
|
||||
|
||||
// AccountCleanup is what one CleanupAccount run did, by group ID. It is the
|
||||
// owner's account of its attempts, not proof of absence, which only a later
|
||||
// ScanAccount gives.
|
||||
type AccountCleanup struct {
|
||||
// DeletedGroupIDs were removed by the owner delete or were already gone.
|
||||
DeletedGroupIDs []string
|
||||
// FailedGroupIDs hit a store failure; the record remains.
|
||||
FailedGroupIDs []string
|
||||
// Blocked groups still hold instances and are never forced.
|
||||
Blocked []BlockedGroup
|
||||
// Referenced maps a kept group ID to the reason the caller's predicate gave.
|
||||
Referenced map[string]string
|
||||
}
|
||||
|
||||
// CleanupAccount deletes every readable, empty placement group of the account
|
||||
// through the same revision-guarded delete DeletePlacementGroup runs. A group
|
||||
// holding instances is reported blocked, and one for whose ID or name
|
||||
// referenced returns a reason is kept; unreadable records are left for the owner scan to report. A
|
||||
// listing failure is returned and nothing is deleted.
|
||||
func (s *PlacementGroupServiceImpl) CleanupAccount(ctx context.Context, accountID string, referenced func(groupID, groupName string) string) (*AccountCleanup, error) {
|
||||
keys, err := kvutil.Keys(ctx, s.kv)
|
||||
if err != nil && !errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, fmt.Errorf("list placement-group keys: %w", err)
|
||||
}
|
||||
|
||||
prefix := accountID + "."
|
||||
out := &AccountCleanup{Referenced: map[string]string{}}
|
||||
for _, key := range keys {
|
||||
if key == kvutil.VersionKey || !strings.HasPrefix(key, prefix) {
|
||||
continue
|
||||
}
|
||||
entry, err := s.kv.Get(ctx, key)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var record PlacementGroupRecord
|
||||
if err := json.Unmarshal(entry.Value(), &record); err != nil || record.GroupId == "" {
|
||||
continue
|
||||
}
|
||||
if referenced != nil {
|
||||
reason := referenced(record.GroupId, record.GroupName)
|
||||
if reason == "" {
|
||||
reason = referenced(record.GroupId, strings.TrimPrefix(key, prefix))
|
||||
}
|
||||
if reason != "" {
|
||||
out.Referenced[record.GroupId] = reason
|
||||
continue
|
||||
}
|
||||
}
|
||||
members, err := s.deleteGroup(ctx, accountID, key)
|
||||
switch {
|
||||
case err == nil, errors.Is(err, errPGAbsent):
|
||||
out.DeletedGroupIDs = append(out.DeletedGroupIDs, record.GroupId)
|
||||
case errors.Is(err, errPGInUse):
|
||||
out.Blocked = append(out.Blocked, BlockedGroup{GroupID: record.GroupId, MemberCount: members})
|
||||
default:
|
||||
slog.WarnContext(ctx, "Placement-group account cleanup: delete failed", "groupId", record.GroupId, "err", err)
|
||||
out.FailedGroupIDs = append(out.FailedGroupIDs, record.GroupId)
|
||||
}
|
||||
}
|
||||
slog.InfoContext(ctx, "Placement-group account cleanup completed", "accountID", accountID,
|
||||
"deleted", len(out.DeletedGroupIDs), "failed", len(out.FailedGroupIDs), "blocked", len(out.Blocked), "referenced", len(out.Referenced))
|
||||
return out, nil
|
||||
}
|
||||
+168
-18
@@ -21,16 +21,18 @@ import (
|
||||
)
|
||||
|
||||
// The TestCurrentBehaviour_AccountTermination_ tests pin how placement groups
|
||||
// are stored and what Describe and Delete do with them today. They describe
|
||||
// current behaviour, not the target design; a failure means it changed.
|
||||
// are stored and what Describe and Delete do with them today; a failure means
|
||||
// it changed. The TestAccountTermination_ tests pin the revision-guarded delete.
|
||||
|
||||
// hookedKV fails reads of failGet and runs afterGet once, after the first read
|
||||
// of key returns, standing in for a writer on another node.
|
||||
// hookedKV fails reads of failGet and runs afterGet once, after read number
|
||||
// skip+1 of key returns, standing in for a writer on another node.
|
||||
type hookedKV struct {
|
||||
jetstream.KeyValue
|
||||
|
||||
failGet string
|
||||
key string
|
||||
skip int
|
||||
reads int
|
||||
once sync.Once
|
||||
afterGet func()
|
||||
}
|
||||
@@ -41,7 +43,9 @@ func (k *hookedKV) Get(ctx context.Context, key string) (jetstream.KeyValueEntry
|
||||
}
|
||||
entry, err := k.KeyValue.Get(ctx, key)
|
||||
if key == k.key && k.afterGet != nil {
|
||||
k.once.Do(k.afterGet)
|
||||
if k.reads++; k.reads > k.skip {
|
||||
k.once.Do(k.afterGet)
|
||||
}
|
||||
}
|
||||
return entry, err
|
||||
}
|
||||
@@ -92,8 +96,9 @@ func TestCurrentBehaviour_AccountTermination_PlacementGroupStorageLayout(t *test
|
||||
assert.Empty(t, record.NodeInstances)
|
||||
}
|
||||
|
||||
// Describe skips a record it cannot read or decode, and still succeeds. An
|
||||
// undecodable record also cannot be removed through DeletePlacementGroup.
|
||||
// Describe skips a record it cannot read or decode, and still succeeds. Such
|
||||
// a record cannot be removed through DeletePlacementGroup either, and a read
|
||||
// failure is an internal error, never an unknown group.
|
||||
func TestCurrentBehaviour_AccountTermination_DescribePlacementGroupsHidesUnreadableRecords(t *testing.T) {
|
||||
brokenKey := kvutil.AccountKey(testAccountID, "broken")
|
||||
tests := []struct {
|
||||
@@ -108,7 +113,7 @@ func TestCurrentBehaviour_AccountTermination_DescribePlacementGroupsHidesUnreada
|
||||
{"record the store cannot read", func(t *testing.T, svc *PlacementGroupServiceImpl) {
|
||||
createTestGroup(t, svc, "broken", ec2.PlacementStrategySpread)
|
||||
svc.kv = &hookedKV{KeyValue: svc.kv, failGet: brokenKey}
|
||||
}, awserrors.ErrorInvalidPlacementGroupUnknown},
|
||||
}, awserrors.ErrorServerInternal},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -128,10 +133,10 @@ func TestCurrentBehaviour_AccountTermination_DescribePlacementGroupsHidesUnreada
|
||||
}
|
||||
}
|
||||
|
||||
// Pins current behaviour that differs from AWS: the delete reads the record,
|
||||
// checks membership, then deletes without a revision guard. A launch that
|
||||
// records an instance between the two is deleted with the group.
|
||||
func TestCurrentBehaviour_AccountTermination_DeletePlacementGroupDropsAConcurrentMember(t *testing.T) {
|
||||
// The delete commits only at the revision whose membership it checked. A
|
||||
// launch that records an instance between the check and the delete wins: the
|
||||
// delete re-reads, refuses with InUse, and the member stays.
|
||||
func TestAccountTermination_DeletePlacementGroupLosesToAConcurrentMember(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
createTestGroup(t, svc, "cluster-a", ec2.PlacementStrategyCluster)
|
||||
key := kvutil.AccountKey(testAccountID, "cluster-a")
|
||||
@@ -151,17 +156,162 @@ func TestCurrentBehaviour_AccountTermination_DeletePlacementGroupDropsAConcurren
|
||||
}}
|
||||
|
||||
_, err := svc.DeletePlacementGroup(context.Background(), &ec2.DeletePlacementGroupInput{GroupName: aws.String("cluster-a")}, testAccountID)
|
||||
require.NoError(t, err, "the delete succeeds although the group gained a member")
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidPlacementGroupInUse, err.Error())
|
||||
require.NoError(t, reserveErr)
|
||||
require.NoError(t, finalizeErr, "the interleaved launch committed its member")
|
||||
|
||||
_, err = launcher.kv.Get(t.Context(), key)
|
||||
assert.ErrorIs(t, err, jetstream.ErrKeyNotFound)
|
||||
assert.Empty(t, describedGroupNames(t, launcher, testAccountID))
|
||||
entry, err := launcher.kv.Get(t.Context(), key)
|
||||
require.NoError(t, err)
|
||||
var record PlacementGroupRecord
|
||||
require.NoError(t, json.Unmarshal(entry.Value(), &record))
|
||||
assert.Equal(t, map[string][]string{"node-1": {"i-0123456789abcdef0"}}, record.NodeInstances)
|
||||
|
||||
// The member's later shutdown finds no group and reports success.
|
||||
// Once the member leaves, the same delete succeeds.
|
||||
_, err = launcher.RemoveInstance(context.Background(), &RemoveInstanceInput{
|
||||
GroupName: "cluster-a", NodeName: "node-1", InstanceID: "i-0123456789abcdef0",
|
||||
}, testAccountID)
|
||||
assert.NoError(t, err)
|
||||
require.NoError(t, err)
|
||||
_, err = launcher.DeletePlacementGroup(context.Background(), &ec2.DeletePlacementGroupInput{GroupName: aws.String("cluster-a")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, describedGroupNames(t, launcher, testAccountID))
|
||||
}
|
||||
|
||||
// A reservation placeholder is a launch in flight, not a member, so the
|
||||
// delete may win; the launch's finalize then finds no group and fails, which
|
||||
// makes the launcher roll its instances back rather than lose them.
|
||||
func TestAccountTermination_DeletePlacementGroupBeforeFinalizeFailsTheLaunch(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
createTestGroup(t, svc, "cluster-a", ec2.PlacementStrategyCluster)
|
||||
reserved, err := svc.ReserveClusterNode(context.Background(), &ReserveClusterNodeInput{
|
||||
GroupName: "cluster-a", EligibleNodes: []string{"node-1"},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.DeletePlacementGroup(context.Background(), &ec2.DeletePlacementGroupInput{GroupName: aws.String("cluster-a")}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.FinalizeClusterInstances(context.Background(), &FinalizeClusterInstancesInput{
|
||||
GroupName: "cluster-a", NodeInstances: map[string][]string{reserved.TargetNode: {"i-0123456789abcdef0"}},
|
||||
}, testAccountID)
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, awserrors.ErrorInvalidPlacementGroupUnknown, err.Error())
|
||||
assert.Empty(t, describedGroupNames(t, svc, testAccountID))
|
||||
}
|
||||
|
||||
// addMember reserves a node in a cluster group and records one instance on it.
|
||||
func addMember(t *testing.T, svc *PlacementGroupServiceImpl, group, instanceID string) error {
|
||||
t.Helper()
|
||||
reserved, err := svc.ReserveClusterNode(context.Background(), &ReserveClusterNodeInput{
|
||||
GroupName: group, EligibleNodes: []string{"node-1"},
|
||||
}, testAccountID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = svc.FinalizeClusterInstances(context.Background(), &FinalizeClusterInstancesInput{
|
||||
GroupName: group, NodeInstances: map[string][]string{reserved.TargetNode: {instanceID}},
|
||||
}, testAccountID)
|
||||
return err
|
||||
}
|
||||
|
||||
// Account cleanup deletes empty groups through the revision-guarded delete,
|
||||
// reports a group holding instances as blocked with its member count only,
|
||||
// and leaves records it cannot read for the owner scan to report.
|
||||
func TestAccountCleanup_DeletesEmptyGroupsAndBlocksOnMembers(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
empty := createTestGroup(t, svc, "empty", ec2.PlacementStrategySpread)
|
||||
held := createTestGroup(t, svc, "held", ec2.PlacementStrategyCluster)
|
||||
require.NoError(t, addMember(t, svc, "held", "i-0123456789abcdef0"))
|
||||
_, err := svc.kv.Put(t.Context(), kvutil.AccountKey(testAccountID, "broken"), []byte("{not json"))
|
||||
require.NoError(t, err)
|
||||
other := "999988887777"
|
||||
_, err = svc.CreatePlacementGroup(context.Background(), &ec2.CreatePlacementGroupInput{
|
||||
GroupName: aws.String("theirs"), Strategy: aws.String(ec2.PlacementStrategySpread),
|
||||
}, other)
|
||||
require.NoError(t, err)
|
||||
|
||||
out, err := svc.CleanupAccount(t.Context(), testAccountID, nil)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{aws.StringValue(empty.GroupId)}, out.DeletedGroupIDs)
|
||||
assert.Equal(t, []BlockedGroup{{GroupID: aws.StringValue(held.GroupId), MemberCount: 1}}, out.Blocked)
|
||||
assert.Empty(t, out.FailedGroupIDs)
|
||||
|
||||
assert.Equal(t, []string{"held"}, describedGroupNames(t, svc, testAccountID))
|
||||
assert.Equal(t, []string{"theirs"}, describedGroupNames(t, svc, other))
|
||||
_, err = svc.kv.Get(t.Context(), kvutil.AccountKey(testAccountID, "broken"))
|
||||
require.NoError(t, err, "an unreadable record is never deleted")
|
||||
}
|
||||
|
||||
// A launch that records a member between the cleanup's membership check and
|
||||
// its delete wins: the cleanup reports the group blocked and the member stays.
|
||||
func TestAccountCleanup_ConcurrentMemberBlocksTheDelete(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
group := createTestGroup(t, svc, "cluster-a", ec2.PlacementStrategyCluster)
|
||||
key := kvutil.AccountKey(testAccountID, "cluster-a")
|
||||
launcher := &PlacementGroupServiceImpl{kv: svc.kv, region: svc.region}
|
||||
var memberErr error
|
||||
// Read 1 is the cleanup's listing read; read 2 is the delete's check.
|
||||
svc.kv = &hookedKV{KeyValue: svc.kv, key: key, skip: 1, afterGet: func() {
|
||||
memberErr = addMember(t, launcher, "cluster-a", "i-0123456789abcdef0")
|
||||
}}
|
||||
|
||||
out, err := svc.CleanupAccount(t.Context(), testAccountID, nil)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, memberErr, "the interleaved launch committed its member")
|
||||
assert.Equal(t, []BlockedGroup{{GroupID: aws.StringValue(group.GroupId), MemberCount: 1}}, out.Blocked)
|
||||
assert.Empty(t, out.DeletedGroupIDs)
|
||||
assert.Equal(t, map[string][]string{"node-1": {"i-0123456789abcdef0"}}, readRecord(t, launcher, "cluster-a").NodeInstances)
|
||||
}
|
||||
|
||||
// A group the caller's reference check names is kept with the given reason,
|
||||
// whether it is matched by its name or its group ID.
|
||||
func TestAccountCleanup_KeepsReferencedGroups(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
named := createTestGroup(t, svc, "named", ec2.PlacementStrategySpread)
|
||||
byID := createTestGroup(t, svc, "by-id", ec2.PlacementStrategySpread)
|
||||
free := createTestGroup(t, svc, "free", ec2.PlacementStrategySpread)
|
||||
referenced := func(id, name string) string {
|
||||
switch {
|
||||
case name == "named":
|
||||
return "spot-request-reference"
|
||||
case id == aws.StringValue(byID.GroupId):
|
||||
return "launch-template-reference"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
out, err := svc.CleanupAccount(t.Context(), testAccountID, referenced)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, map[string]string{
|
||||
aws.StringValue(named.GroupId): "spot-request-reference",
|
||||
aws.StringValue(byID.GroupId): "launch-template-reference",
|
||||
}, out.Referenced)
|
||||
assert.Equal(t, []string{aws.StringValue(free.GroupId)}, out.DeletedGroupIDs)
|
||||
assert.ElementsMatch(t, []string{"named", "by-id"}, describedGroupNames(t, svc, testAccountID))
|
||||
}
|
||||
|
||||
// A listing failure is returned and nothing is deleted.
|
||||
func TestAccountCleanup_ListingFailureDeletesNothing(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
createTestGroup(t, svc, "empty", ec2.PlacementStrategySpread)
|
||||
kv := svc.kv
|
||||
svc.kv = &listFailingKV{KeyValue: kv}
|
||||
|
||||
_, err := svc.CleanupAccount(t.Context(), testAccountID, nil)
|
||||
require.Error(t, err)
|
||||
svc.kv = kv
|
||||
assert.Equal(t, []string{"empty"}, describedGroupNames(t, svc, testAccountID))
|
||||
}
|
||||
|
||||
// listFailingKV fails every key listing.
|
||||
type listFailingKV struct {
|
||||
jetstream.KeyValue
|
||||
}
|
||||
|
||||
func (k *listFailingKV) ListKeys(context.Context, ...jetstream.WatchOpt) (jetstream.KeyLister, error) {
|
||||
return nil, errors.New("injected listing failure")
|
||||
}
|
||||
|
||||
func (k *listFailingKV) Keys(context.Context, ...jetstream.WatchOpt) ([]string, error) {
|
||||
return nil, errors.New("injected listing failure")
|
||||
}
|
||||
|
||||
@@ -52,6 +52,32 @@ type PlacementGroupServiceImpl struct {
|
||||
natsConn *nats.Conn
|
||||
kv jetstream.KeyValue
|
||||
region string
|
||||
|
||||
// Optional: nil leaves the central tag index untouched by delete.
|
||||
centralTags CentralTagStore
|
||||
}
|
||||
|
||||
// CentralTagStore clears a deleted group's entry from the central EC2 tag
|
||||
// index. Implemented by domains/ec2/tags.TagsServiceImpl.
|
||||
type CentralTagStore interface {
|
||||
DeleteAllTags(ctx context.Context, accountID, resourceID string) error
|
||||
}
|
||||
|
||||
// SetCentralTagStore injects the central tag index that delete clears.
|
||||
func (s *PlacementGroupServiceImpl) SetCentralTagStore(st CentralTagStore) {
|
||||
s.centralTags = st
|
||||
}
|
||||
|
||||
// clearRecordTags drops a deleted group's tag-index entry. It runs only after
|
||||
// the record delete committed; a failure is logged, as the group is gone and a
|
||||
// retried delete would find nothing, and the entry is left as tag residue.
|
||||
func (s *PlacementGroupServiceImpl) clearRecordTags(ctx context.Context, accountID, groupID string) {
|
||||
if s.centralTags == nil || groupID == "" {
|
||||
return
|
||||
}
|
||||
if err := s.centralTags.DeleteAllTags(ctx, accountID, groupID); err != nil {
|
||||
slog.ErrorContext(ctx, "central tag store clear failed", "groupId", groupID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// NewPlacementGroupServiceImplWithNATS creates a placement group service with NATS JetStream.
|
||||
@@ -142,37 +168,19 @@ func (s *PlacementGroupServiceImpl) CreatePlacementGroup(ctx context.Context, in
|
||||
}, nil
|
||||
}
|
||||
|
||||
// DeletePlacementGroup deletes a placement group.
|
||||
// DeletePlacementGroup deletes a placement group that holds no instances.
|
||||
func (s *PlacementGroupServiceImpl) DeletePlacementGroup(ctx context.Context, input *ec2.DeletePlacementGroupInput, accountID string) (*ec2.DeletePlacementGroupOutput, error) {
|
||||
if input.GroupName == nil || *input.GroupName == "" {
|
||||
return nil, errors.New(awserrors.ErrorMissingParameter)
|
||||
}
|
||||
|
||||
groupName := *input.GroupName
|
||||
key := kvutil.AccountKey(accountID, groupName)
|
||||
|
||||
entry, err := s.kv.Get(ctx, key)
|
||||
if err != nil {
|
||||
// Lower case, unlike DescribePlacementGroups' message for the same code.
|
||||
return nil, awserrors.Errorf(awserrors.ErrorInvalidPlacementGroupUnknown, "The placement group '%s' is unknown.", groupName)
|
||||
}
|
||||
|
||||
var record PlacementGroupRecord
|
||||
if err := json.Unmarshal(entry.Value(), &record); err != nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
}
|
||||
|
||||
// Check for running instances
|
||||
instanceCount := 0
|
||||
for _, ids := range record.NodeInstances {
|
||||
instanceCount += len(ids)
|
||||
}
|
||||
if instanceCount > 0 {
|
||||
return nil, errors.New(awserrors.ErrorInvalidPlacementGroupInUse)
|
||||
}
|
||||
|
||||
if err := s.kv.Delete(ctx, key); err != nil {
|
||||
return nil, errors.New(awserrors.ErrorServerInternal)
|
||||
if _, err := s.deleteGroup(ctx, accountID, kvutil.AccountKey(accountID, groupName)); err != nil {
|
||||
if errors.Is(err, errPGAbsent) {
|
||||
// Lower case, unlike DescribePlacementGroups' message for the same code.
|
||||
return nil, awserrors.Errorf(awserrors.ErrorInvalidPlacementGroupUnknown, "The placement group '%s' is unknown.", groupName)
|
||||
}
|
||||
return nil, mapCASError(ctx, "DeletePlacementGroup", groupName, accountID, err)
|
||||
}
|
||||
|
||||
slog.InfoContext(ctx, "DeletePlacementGroup completed", "groupName", groupName, "accountID", accountID)
|
||||
@@ -180,6 +188,26 @@ func (s *PlacementGroupServiceImpl) DeletePlacementGroup(ctx context.Context, in
|
||||
return &ec2.DeletePlacementGroupOutput{}, nil
|
||||
}
|
||||
|
||||
// deleteGroup removes the record at key only while it names no instance, at
|
||||
// the revision that check read, so a reservation committed in between sends
|
||||
// the delete back to re-check. Its tag-index entry is cleared afterwards.
|
||||
// The returned count is the members seen by the last check.
|
||||
func (s *PlacementGroupServiceImpl) deleteGroup(ctx context.Context, accountID, key string) (int, error) {
|
||||
members := 0
|
||||
record, err := kvutil.DeleteIf(ctx, s.kv, key, casConfig(), func(record *PlacementGroupRecord) error {
|
||||
members = memberCount(record.NodeInstances)
|
||||
if members > 0 {
|
||||
return errPGInUse
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return members, err
|
||||
}
|
||||
s.clearRecordTags(ctx, accountID, record.GroupId)
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
var describePlacementGroupsValidFilters = map[string]bool{
|
||||
"group-id": true,
|
||||
"strategy": true,
|
||||
@@ -324,11 +352,12 @@ func pgMatchesFilters(record *PlacementGroupRecord, filters map[string][]string)
|
||||
const maxCASRetries = 5
|
||||
|
||||
// errPGAbsent and errPGContended mark the two kvutil outcomes that map to
|
||||
// specific AWS errors rather than a generic internal failure. Neither
|
||||
// escapes this file.
|
||||
// specific AWS errors rather than a generic internal failure, and neither
|
||||
// escapes this file. errPGInUse is the AWS error itself.
|
||||
var (
|
||||
errPGAbsent = errors.New("placementgroup: record absent")
|
||||
errPGContended = errors.New("placementgroup: CAS retries exhausted")
|
||||
errPGInUse = errors.New(awserrors.ErrorInvalidPlacementGroupInUse)
|
||||
)
|
||||
|
||||
// mapCASError turns a kvutil failure into the AWS error the caller returns.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
package spotinstance
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/foundation/state/kvutil"
|
||||
"github.com/nats-io/nats.go/jetstream"
|
||||
)
|
||||
|
||||
// LaunchReferences are the key-pair and placement-group names that an
|
||||
// account's open or active spot requests would launch with. They stay inside
|
||||
// EC2; callers must not report them.
|
||||
type LaunchReferences struct {
|
||||
KeyNames []string
|
||||
PlacementGroupNames []string
|
||||
}
|
||||
|
||||
// AccountLaunchReferences reads every open or active spot request of the
|
||||
// account. It is complete or an error: an unlistable bucket or an unreadable
|
||||
// request is an error, because the unread request may name the resource.
|
||||
func (s *SpotInstanceServiceImpl) AccountLaunchReferences(ctx context.Context, accountID string) (*LaunchReferences, error) {
|
||||
keys, err := kvutil.Keys(ctx, s.activeKV)
|
||||
if err != nil && !errors.Is(err, jetstream.ErrNoKeysFound) {
|
||||
return nil, fmt.Errorf("list spot requests: %w", err)
|
||||
}
|
||||
prefix := accountID + "."
|
||||
out := &LaunchReferences{}
|
||||
for _, key := range keys {
|
||||
if key == kvutil.VersionKey || !strings.HasPrefix(key, prefix) {
|
||||
continue
|
||||
}
|
||||
record, err := s.readRecord(ctx, s.activeKV, key)
|
||||
if errors.Is(err, jetstream.ErrKeyNotFound) {
|
||||
continue
|
||||
}
|
||||
if err != nil || record.Request == nil {
|
||||
return nil, errors.New("spot request unreadable")
|
||||
}
|
||||
switch aws.StringValue(record.Request.State) {
|
||||
case ec2.SpotInstanceStateClosed, ec2.SpotInstanceStateCancelled, ec2.SpotInstanceStateFailed:
|
||||
continue
|
||||
}
|
||||
spec := launchSpec(record.Request)
|
||||
if name := aws.StringValue(spec.KeyName); name != "" {
|
||||
out.KeyNames = append(out.KeyNames, name)
|
||||
}
|
||||
if spec.Placement != nil {
|
||||
if name := aws.StringValue(spec.Placement.GroupName); name != "" {
|
||||
out.PlacementGroupNames = append(out.PlacementGroupNames, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package spotinstance
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/aws/aws-sdk-go/aws"
|
||||
"github.com/aws/aws-sdk-go/service/ec2"
|
||||
"github.com/mulgadc/spinifex/spinifex/foundation/state/kvutil"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// Open and active requests contribute their key and group names; cancelled
|
||||
// requests and other accounts' requests do not.
|
||||
func TestAccountLaunchReferences_ReadsOpenAndActiveRequests(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
active := makeSIR("sir-active01", "i-0000000000000001")
|
||||
active.LaunchSpecification.Placement = &ec2.SpotPlacement{GroupName: aws.String("pg-spread")}
|
||||
open := makeSIR("sir-open0001", "")
|
||||
open.State = aws.String(ec2.SpotInstanceStateOpen)
|
||||
open.LaunchSpecification.KeyName = aws.String("open-key")
|
||||
putSIRs(t, svc, active, open)
|
||||
cancelled := makeSIR("sir-cancel01", "i-0000000000000002")
|
||||
cancelled.LaunchSpecification.KeyName = aws.String("cancelled-key")
|
||||
putSIRs(t, svc, cancelled)
|
||||
_, err := svc.CancelSpotInstanceRequests(t.Context(), &ec2.CancelSpotInstanceRequestsInput{
|
||||
SpotInstanceRequestIds: []*string{aws.String("sir-cancel01")},
|
||||
}, testAccountID)
|
||||
require.NoError(t, err)
|
||||
_, err = svc.PutSpotInstanceRequests(t.Context(), &PutSpotRequestsInput{Requests: []*ec2.SpotInstanceRequest{makeSIR("sir-other001", "")}}, "999988887777")
|
||||
require.NoError(t, err)
|
||||
|
||||
refs, err := svc.AccountLaunchReferences(t.Context(), testAccountID)
|
||||
require.NoError(t, err)
|
||||
assert.ElementsMatch(t, []string{"my-key", "open-key"}, refs.KeyNames)
|
||||
assert.Equal(t, []string{"pg-spread"}, refs.PlacementGroupNames)
|
||||
}
|
||||
|
||||
// An unreadable request may name anything, so the check fails rather than
|
||||
// answer without it.
|
||||
func TestAccountLaunchReferences_UnreadableRequestFailsTheCheck(t *testing.T) {
|
||||
svc := setupTestService(t)
|
||||
putSIRs(t, svc, makeSIR("sir-active01", "i-0000000000000001"))
|
||||
_, err := svc.activeKV.Put(t.Context(), kvutil.AccountKey(testAccountID, "sir-broken01"), []byte("{not json"))
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = svc.AccountLaunchReferences(t.Context(), testAccountID)
|
||||
require.Error(t, err)
|
||||
}
|
||||
@@ -189,8 +189,9 @@ func (s *S3ObjectStore) EnsureBucket(ctx context.Context, bucket string) error {
|
||||
|
||||
// MemoryObjectStore implements ObjectStore using in-memory storage for testing.
|
||||
type MemoryObjectStore struct {
|
||||
objects map[string][]byte // key: bucket/key -> value: object data
|
||||
mu sync.RWMutex
|
||||
objects map[string][]byte // key: bucket/key -> value: object data
|
||||
modified map[string]time.Time // key: bucket/key -> last write time
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
var _ ObjectStore = (*MemoryObjectStore)(nil)
|
||||
@@ -198,7 +199,8 @@ var _ ObjectStore = (*MemoryObjectStore)(nil)
|
||||
// NewMemoryObjectStore creates an in-memory ObjectStore for testing.
|
||||
func NewMemoryObjectStore() *MemoryObjectStore {
|
||||
return &MemoryObjectStore{
|
||||
objects: make(map[string][]byte),
|
||||
objects: make(map[string][]byte),
|
||||
modified: make(map[string]time.Time),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -259,6 +261,7 @@ func (m *MemoryObjectStore) PutObject(_ context.Context, input *s3.PutObjectInpu
|
||||
}
|
||||
|
||||
m.objects[storageKey] = data
|
||||
m.modified[storageKey] = time.Now().UTC()
|
||||
|
||||
return &s3.PutObjectOutput{}, nil
|
||||
}
|
||||
@@ -272,6 +275,7 @@ func (m *MemoryObjectStore) DeleteObject(_ context.Context, input *s3.DeleteObje
|
||||
|
||||
storageKey := makeKey(*input.Bucket, *input.Key)
|
||||
delete(m.objects, storageKey)
|
||||
delete(m.modified, storageKey)
|
||||
|
||||
return &s3.DeleteObjectOutput{}, nil
|
||||
}
|
||||
@@ -311,8 +315,9 @@ func (m *MemoryObjectStore) ListObjectsV2(_ context.Context, input *s3.ListObjec
|
||||
}
|
||||
|
||||
contents = append(contents, &s3.Object{
|
||||
Key: aws.String(objectKey),
|
||||
Size: aws.Int64(int64(len(data))),
|
||||
Key: aws.String(objectKey),
|
||||
Size: aws.Int64(int64(len(data))),
|
||||
LastModified: aws.Time(m.modified[key]),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -340,6 +345,7 @@ func (m *MemoryObjectStore) Clear() {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.objects = make(map[string][]byte)
|
||||
m.modified = make(map[string]time.Time)
|
||||
}
|
||||
|
||||
// Count returns the number of objects in the memory store.
|
||||
|
||||
Reference in New Issue
Block a user