* feat: add OVSDB RAFT cluster primitives to setup-ovn.sh
Add --db-cluster-local-addr / --db-cluster-remote-addr flags so a
management node can run the OVN NB/SB databases as a native OVSDB RAFT
cluster instead of a single non-clustered ovn-central. When a local addr
is given, write a shared OVN_CTL_OPTS to /etc/default/ovn-central (NB
RAFT 6643 / SB RAFT 6644) and restart the per-DB units directly, since
the ovn-central aggregator is ExecStart=/bin/true and won't restart its
children. Empty remote addr creates the cluster (init node); a set remote
addr joins it. NB/SB client set-connection runs only on the create node
(RAFT replicates it cluster-wide).
The single-node --management path is unchanged when no cluster addrs are
supplied, so dev boxes and existing single-node clusters keep working.
--ovn-remote already passes through verbatim, so a comma-separated SB
endpoint list works for compute nodes; documented in usage.
* feat: OVN client multi-endpoint failover for RAFT clusters
Accept a comma-separated OVN NB/SB endpoint list so vpcd can target a
clustered DB. config.ParseEndpoints splits the list (trimming whitespace,
dropping empties); live.go Connect passes one libovsdb WithEndpoint per
endpoint, so the client tries them in order, fails over on disconnect, and
follows RAFT leader redirects. Single-endpoint strings still yield one
endpoint, so single-node dev is unchanged.
ovn-sbctl --db= already accepts the raw comma list, so the SB CLI
consumers (discoverChassis, gateway_claim) need no change.
* feat: write clustered OVN NB/SB endpoints across the RAFT quorum
spx admin init/join now emit comma-separated OVN NB/SB endpoint lists for
the first three quorum nodes (formation.BuildOVNDBAddrs) instead of loopback
or a single leader address, so vpcd fails over across the cluster.
firstboot defers OVN bring-up to the provisioning controller when it owns
formation (SkipFormation): a standalone single-node .db would otherwise block
the controller's clustered create/join.
* test: add multinode OVN RAFT cluster + failover e2e; document clustered topology
ovn_raft_test.go asserts NB/SB cluster/status reports a 3-server quorum with
one leader on every DB node, then stops the NB leader's ovn-central and proves
an NB write still lands via a surviving endpoint.
Docs: vpc-networking gains the multi-endpoint vpcd config and a RAFT
cluster-status debugging section; network-connections lists the 6643/6644 RAFT
ports.
vpcd's libovsdb NB client was a single long-lived connection with no
reconnect logic. After the NB server restarted (host reboot, service
restart) every vpcd holding a dead/half-open socket never re-established
it, so OVN-write handlers blocked until the caller timeout and CreateVpc
returned ServerInternal intermittently (queue-distributed across vpcd).
Add WithInactivityCheck to the OVSDB client: a periodic Echo probe
detects the dead peer and triggers reconnect against the configured
endpoint, with infinite backoff so a daemon keeps retrying across a long
outage. Bound the SG subscriber handlers with a timeout context so a
stuck OVN op frees the worker instead of hanging on context.Background.
NLBs hard-rejected customer security groups and always minted a managed SG that
auto-opened listener ports, so aws-load-balancer-controller (which assigns SGs to
type=external NLBs) got a 400 and looped. Accept up to 5 customer SGs on an NLB:
when supplied they replace the managed SG on the ENIs and the caller owns the
listener-port rules (CreateListener already no-ops its auto-open when there is no
managed SG); without them the managed-SG default is unchanged. SetSecurityGroups
follows AWS create-time semantics — replace SGs on an NLB created with SGs, reject
adding them to one created without.
CreateVolume hardcoded iops to 3000, silently dropping a caller-supplied Iops,
and rejected non-gp3 VolumeType with the generic InvalidParameterValue. Honor
input.Iops within the gp3 envelope (3000 baseline, 500 IOPS/GiB, 16000 ceiling)
and return the volume-type-specific UnknownVolumeType for unsupported types.
Closes EBS CSI API gaps #3-4 (siv-380); gp3 Throughput (gap #2) deferred.
* feat: IMDS-capable gateway signer for lb-agent + EKS clients
Add internal/gwsign, a SigV4 signer that draws credentials from either
fixed static keys or the AWS SDK default credential chain. In a guest VM
the chain falls back to the in-VM IMDS endpoint, yielding scoped,
auto-rotating instance-role credentials; a session token is signed as
X-Amz-Security-Token so the gateway can verify temporary (ASIA) creds.
Wire it into the lb-agent (lbagent.Agent) and the EKS on-VM client
(internal/eksgw). When static keys are absent the clients now sign with
IMDS credentials instead of failing; when present, behaviour is
unchanged (static path). This is the client-side foundation for dropping
long-lived secrets from VM user-data.
Credentials are retrieved per request so rotated IMDS creds take effect
without an agent restart.
* feat: IMDS instance-role creds for EKS server + LB VMs (Phase B/C)
Attach a Spinifex-managed system instance profile to the k3s control-plane VM
and the elbv2 LB VM so both authenticate to the gateway with scoped, rotating
IMDS instance-role credentials instead of a long-lived static secret baked into
user-data.
- Add a shared handlers/iam.EnsureSystemInstanceProfile that find-or-creates a
role (EC2 trust) plus its instance profile and asserts an inline permission
policy. The gateway enforces per-action authz on assumed-role principals, so
the roles carry exactly the actions each VM calls:
- EKS server: eks:PublishInternal, eks:ListInternalAddons.
- LB agent: elasticloadbalancing:LBAgentHeartbeat, GetLBConfig.
- Plumb IamInstanceProfileArn through sysinstance.SystemInstanceInput and both
daemon launch paths (BootAMI via PrepareRunInstances->RunInstance; direct-boot
microVM via RunInstance), which persist it so the in-VM IMDS endpoint serves
the role.
- Drop the baked secrets when a profile is attached: omit EKS_ACCESS_KEY/
EKS_SECRET_KEY from the k3s first-boot env and LB_ACCESS_KEY/LB_SECRET_KEY from
the lb-agent fw_cfg env; relax k3s server validation to accept profile-only;
drop the EKS key requires from the first-boot and state-report scripts.
- Both verticals fall back to baked static creds when IAM is unwired, so an
absent master key never blocks a launch.
* fix: resolve IMDS instance-role creds under the instance account for system VMs
Three boot-path defects kept the LB VM's lb-agent from authenticating to the
gateway with IMDS-served instance-role creds; each produced a silent empty IMDS
role list and a heartbeat failing with "unexpected empty EC2 IMDS role list".
Found and fixed against env19.
- Build the system IAM ensurer lazily on the first system-VM launch instead of
eagerly at daemon startup, where it raced the NATS KV backend and fell back to
baked static creds permanently. ELBv2 and the EKS deps now hold an IAMProvider
resolved at launch time.
- Create the LB instance profile in the system account the LB VM runs under
(utils.GlobalAccountID), not the LB-owner account. IMDS resolves the profile
under the instance's account, so a customer-account role was invisible to it.
- Record the attached instance's owning account on the ENI (InstanceOwnerId,
mirroring AWS Attachment.InstanceOwnerId) and resolve the instance and every
IAM/STS lookup under it. The LB ENI lives in the customer account while the
system LB VM and its role live in the system account; keying the whole IMDS
chain off the ENI account left the instance invisible and the role lookup
missing. VPC leaves (subnet/VPC/SG CIDR) keep using the ENI account.
A fresh LB on env19 then authenticated end-to-end: IMDS serves a non-empty role
list plus ASIA creds, and the gateway accepts the heartbeat as an assumed-role
principal with no static keys in user-data.
* test(e2e): assert LB VM authenticates via IMDS instance-role creds
The LB suite's targets-healthy assertion already depends on the lb-agent
authenticating to the gateway, and with static keys dropped from user-data that
auth is IMDS-only. Add an explicit check that a recent gateway authentication
for the LB used an assumed-role (ASIA) principal, so a regression to the
instance-account IMDS resolution fails with a clear message instead of a
mysterious datapath timeout. The LB VM is a system-account instance invisible to
the customer AWS API, so the assertion observes the gateway journal on the
cluster nodes; it is time-bounded to avoid the stale-console-log false-fail and
skips when no node is SSH-reachable.
Cell 19 bootstraps with --external-mode=nat plus the CI static pool as
the public wan pool and runs the natuplink suite, covering the Tier-1
transit datapath and Tier-2 EIP ingress. The run step exits 0 when
natuplink.test is absent from the tests tar so scheduled runs against
branches predating the suite stay green.
Add a second build+publish step to the EKS AMI workflow so the same
manual run also builds the ecs-agent system AMI and uploads its qcow2 +
.sha256 sidecar to R2. Reuses the shared checkout, Go, and tooling setup;
renames the workflow to cover both node images.
Flip the k3s server apiserver to anonymous-auth=false. The only
consumer of the anonymous path was mulga-eks-state-report's /healthz
probe; switch it to an authenticated kubectl get --raw=/healthz through
the node admin kubeconfig (an unauthenticated curl now returns 401).
The NLB target group uses a TCP health check, and the host-side
reconciler rides the authenticated NATS state-report, so neither
depends on the anonymous apiserver path.
* fix: free host-netns :80 for ECS bridge/host tasks via cred-endpoint DNAT
bridge and host network-mode tasks share the VM host netns where the
ecs-agent credential endpoint bound 169.254.170.2:80, so a task binding
0.0.0.0:80 hit EADDRINUSE and crash-looped. Bind the listener on the high
proxy port 51679 and DNAT 169.254.170.2:80 to it (OUTPUT for host/bridge
task traffic, PREROUTING for the awsvpc credential veth), keeping the dst
IP on the endpoint so no route_localnet is needed. Nothing holds a socket
on :80, so host/bridge tasks can bind it.
e2e runs nginx on :80 in all three network modes; the settle guard catches
a regression where a crash-looping :80 bind flaps RUNNING then STOPPED.
* test: validate ECS host-netns :80 binding via bridge task
The bridge subtest now runs nginx on :80 to prove the cred-endpoint DNAT
frees the host netns (pre-fix it crash-looped on EADDRINUSE). host mode
keeps a no-port command: it shares the same host netns, so two :80 tasks
cannot coexist on one node (one :80 per host, as on AWS), and StopTask
does not synchronously reap the prior container's port.
When NATS disables JetStream at runtime (e.g. a filesystem permission error
flushing stream state) it keeps the process alive and serves KV as read-only.
That is not a process failure, so spinifex-nats.service's Restart=on-failure
never fires, and KV-backed startup (awsgw IAM bucket init) stalls for minutes
until an operator restarts NATS by hand.
Add a JetStream liveness watchdog:
- build/scripts/nats-js-watchdog.sh probes the NATS monitoring endpoint
(http://127.0.0.1:8222/healthz?js-enabled-only=true). Only when the service
is active and JS fails every sample of a short multi-sample window (so
transient startup/quorum 503s are ignored) does it systemctl try-restart
spinifex-nats.
- spinifex-nats-watchdog.{service,timer} run it as a oneshot every 30s, with a
60s boot delay so cluster quorum settles first.
- spinifex.target Wants= the timer so it starts with the platform.
The NATS-internal permission race itself is a server quirk outside spinifex's
control; this restores service automatically when it recurs.
The in-cluster AWS LoadBalancer Controller provisions its own ALBs
(k8s-toc-*) and security groups (k8s-traffic-toc-*) in the customer VPC
via the awsgw ELBv2/EC2 APIs. Spinifex never tracks these, so cluster
teardown left them orphaned: DeleteSecurityGroup / DeleteVpc then failed
with DependencyViolation and the cluster wedged in DELETING.
purgeClusterInfra now reaps them by the AWS-standard ownership tag
elbv2.k8s.aws/cluster=<name>, scoped to the customer VPC, after the
cluster NLB teardown and before the SG / VPC block:
- ReapLBCLoadBalancers lists account LBs, filters to the customer VPC,
confirms the ownership tag via DescribeTags, and deletes by ARN.
- DeleteClusterSGs deletes any VPC SG carrying the ownership tag (reusing
the DependencyViolation retry budget), in addition to the named cluster
SGs, after rules are revoked.
- DescribeTags is added to the narrow nlbProvisioner interface.
Errors join teardownErrs so a failed reap keeps the cluster in DELETING
for a retry rather than completing with leaked, VPC-pinning infra.
TerminateWorkerInstances routes terminates via the per-instance ec2.cmd.<id>
topic, whose subscription is torn down when a VM stops. A stopped (incl.
stopped+wedged, volume remount broken) worker therefore has no responder, and
the old code treated ErrNoResponders as already-gone and returned nil. The
worker was still in shared KV with its ENI intact, so the nodegroup record was
deleted while the ENI kept pinning the customer subnet/VPC, wedging DeleteCluster
in DELETING on DependencyViolation until a manual ec2 terminate-instances.
On ErrNoResponders, fall back to the ec2.terminate queue group (mirroring the
gateway TerminateInstances path), which runs TerminateStoppedInstance against
shared KV — deleting the worker's volumes, IP, and ENI without a running QEMU.
NotFound or no responder is idempotent success; any other error surfaces so the
teardown backstop retries.
TerminateInstances marks the EC2 record terminated and drives node-local
teardown, but when that teardown runs on a node other than the one hosting
QEMU (per-instance subscription lost, request falls back to the ec2.terminate
broadcast), the process survives and keeps holding its OVN logical port / IP,
blocking rebuilds.
Add OrphanQEMUReaper (node-local, ScopeNodeLocal): it inspects only the
terminated KV bucket and reaps a live process found via the instance's local
PID file. Running, pending, and stopped instances are never touched. Whichever
node holds the orphan PID file is the one that reaps it; every other node's
lookup misses and is a no-op.
Add utils.ProcessAlive (signal-0 liveness) and utils.ForceKillProcess
(SIGKILL + bounded wait), guard shutdownAndUnmount's force-kill on liveness,
and register the reaper in the daemon GC loop.
reconnectInstance re-asserted the instance running state but never the boot
volume's in-use state, so after a daemon restart describe-instances showed the
VM running while describe-volumes showed its root volume available — a
split-brain that makes start-instances attempt to double-launch a live QEMU.
Extract the launch path's boot-volume promotion into markBootVolumesInUse and
call it from reconnectInstance so launch and recovery keep volume state
consistent. Add a reconnect unit test asserting the boot volume is marked
in-use and non-boot volumes are left to their own attach flow.
The daemon-side in-use gate trusts persisted volume State/AttachedInstance,
which the attach/detach lifecycle can leave desynced: a running root volume
was observed with State:"" and AttachedInstance:"" while the instance's
BlockDeviceMappings still listed it, so DeleteVolume passed the gate and
removed the durable backing under a live QEMU.
Add an authoritative guard at the gateway DeleteVolume path: fan out the
strict DescribeInstances survey and refuse with VolumeInUse when any
non-terminated instance's BlockDeviceMappings reference the volume. Fail
closed on an incomplete cluster view rather than delete against a partial
survey. Terminate-time cleanup uses the daemon volume service directly and is
unaffected.
Drives the ECS data plane against the local awsgw: a customer VPC plus a real
container instance launched from the spinifex-ecs-node AMI, then standalone
tasks in all three network modes and an awsvpc service fronted by an ALB target
group. Adds an ECS client to the harness, a Makefile target, and a
service-interfaces suite entry.
bridge/host tasks share the host netns where the agent credential endpoint owns
169.254.170.2:80, so they run a no-port command and assert placement + mode;
awsvpc and the ELB service prove real :80 serving. waitTaskRunning re-checks
after a settle so a transient RUNNING cannot pass.
Wire crash detection and host-resource signals through to the AWS-faithful
DescribeInstanceStatus API and the spx CLI, closing out the EC2 health
monitoring / self-healing work.
Phase C — QMP health monitoring:
- qmpHeartbeat now tracks consecutive query-status failures; at
QMPMaxConsecutiveFailures it checks process liveness, triggering HandleCrash
on a dead QEMU and marking an alive-but-wedged process impaired.
- InstanceHealthState gains QMPConsecutiveFailures, LastQMPSuccess, ImpairedSince.
Phase E — DescribeInstanceStatus mapping:
- buildInstanceStatus maps running VMs to ok / impaired (QMP-unresponsive,
with ImpairedSince) / initializing (2-min launch grace), non-running to
not-applicable.
- SystemStatus reflects node memory pressure via ResourceManager
HostUnderMemoryPressure (live MemAvailable below the host reserve).
Phase C2 — observability:
- spx get vms gains HEALTH and CRASHES columns derived from vmHealthLabel.
Tests cover crash/QMP helpers, status mapping, memory-pressure system status,
and the health label.
On startup the agent had no reconcile pass: Run registered, heartbeat, and
immediately polled assignments with an in-memory dedup map. After a restart the
gateway re-delivers the instance's assignments and the agent re-ran them,
orphaning containers still running from before and double-starting tasks.
Add a reconcile() pass that runs once before polling: it lists the runtime's
containers, and for each task still running in this cluster re-registers its IAM
credentials, re-attaches the exit-wait, and refreshes RUNNING on the gateway.
The adopted task set seeds pollAssignments so re-delivered assignments are acked
but not re-run.
- runtime: add a label-agnostic List(ctx) []Container to the Runner seam, with a
containerd implementation and fake support.
- taskLabels: stamp credID / taskRoleArn / eniMac (when set) so a container is
self-describing enough to re-register creds and tear down its netns on adoption.
- pollAssignments takes the seed dedup map.
Best-effort: a nil runner or a List error degrades to today's poll+run behaviour.
* docs: add ECS (Elastic Container Service) guide + register ecs-quickstart workbook
Add a Container Services -> ECS doc mirroring the EKS/ECR shape: concepts
(cluster, task definition, task, service, container instances via the
ECS-AMI EC2 launch type), an aws ecs / console / Terraform quickstart,
task IAM roles via the credential endpoint, awsvpc networking, and the
honest v1 limitations (no service discovery, awslogs discarded, no rolling
deployments, no capacity providers, no distinct execution role).
Convert the ecs-quickstart workbook README to the doc-page format
(frontmatter + overview/prerequisites/instructions/troubleshooting) so it
renders as a Terraform Workbooks entry alongside the EKS workbooks.
* docs: add node-image prerequisite banner + infra diagram to EKS and ECS pages
Add a GitHub-style [!IMPORTANT] banner to the EKS and ECS prerequisites
flagging the required node-image import (eks-node / spinifex-ecs-node),
mirroring the install page's WAN-bridge banner, each with a describe-images
verification command.
Add a basic Spinifex topology diagram (SVG) to each Overview, in the
tf-nginx-alb visual style: the EKS managed control plane + node-group
workers, and the ECS ALB + container instances with the gateway/agent
and task-credential path.
End-to-end ECS stack against the Spinifex gateway: cluster, task definition
with a task role, container instances from the spinifex-ecs-node AMI, and an
awsvpc service fronted by an ALB target group. Unlike the console's provision-
capacity action, the workbook spells out the two things a Spinifex container
instance needs that AWS hides: a LAN-reachable gateway URL and the gateway CA,
both seeded into cloud-init user-data; instance creds come from IMDS via the
ecsInstanceRole profile. Live-verified apply (HTTP 200 through the ALB) and
destroy round-trip.
The agent heartbeats by periodically re-registering through the gateway
(RegisterContainerInstance). After a control-plane restart, the heartbeat
reaper marks instances DRAINING for the missed-heartbeat window, but the
re-register path only refreshed LastSeen and never restored the status, so a
reaped instance stayed DRAINING forever even though its agent was healthy and
re-registering.
Mark reaper-induced DRAINING with a Reaped flag and restore the instance to
ACTIVE on the next re-register, clearing the flag. An operator
UpdateContainerInstancesState drain clears Reaped, so an intentional drain is
never undone by the live agent.
* feat(ecs): provision EC2 container-instance capacity via IMDS instance-role
Adds a ProvisionCapacity gateway action that launches spinifex-ecs-node
EC2 instances which register as ECS container instances, fixing
RESOURCE:placement when a cluster has no capacity.
Agent control-plane now SigV4-signs gateway calls with instance-role
credentials read from IMDS (no static keys seeded in user-data). The
gateway already verifies STS/temp creds, so the instance role carries an
ecs:* policy + ec2 trust; ensureECSInstanceProfile find-or-creates the
ecsInstanceRole instance profile attached at launch.
Backend resolves the spinifex-ecs-node AMI by spinifex:managed-by=ecs
tag, renders keyless user-data, and launches via the customer
RunInstances path. Frontend adds a hand-signed JSON-1.1 ProvisionCapacity
call, a provision-capacity dialog on the container-instances tab, and the
ECS console section nav + system-image gating.
* feat: auto-assign EIPs to awsvpc service tasks; add ECS task/service detail pages
Wire AssignPublicIp=ENABLED through the service reconciler so awsvpc service
tasks automatically allocate and associate an Elastic IP on the RUNNING
transition, and release it on STOP. assignTaskPublicIP/releaseTaskPublicIP
mirror the target-group registration hooks in recordTaskState and forceStopTask;
the public IP and allocation ID are persisted on the task record and surfaced in
DescribeTasks attachments as publicIPv4Address.
Add AWS-parity ECS task and service detail pages showing private and public
endpoints, container state, and networking configuration, reachable by clicking
a task or service in the cluster tabs.
* test: cover natsEIPManager allocate/associate/release over NATS
ListTaskDefinitions appended every revision regardless of the requested
status, so deregistered (INACTIVE) task definitions never dropped off the
default listing and the console could not clear them. Filter by status,
defaulting to ACTIVE to match AWS.
Add an ECS console mirroring the EKS/ECR route shape: a single ECS nav
entry, cluster list and detail with Services/Tasks/Task Definitions/
Container Instances tabs, plus create and teardown paths.
Adds three full-page wizards to drive container deploys end-to-end:
Register Task Definition, Run Task, and Create Service. RunTask surfaces
a 200-with-failures placement result as an error instead of silently
navigating away.
Includes the @aws-sdk/client-ecs client singleton, ecs queries and
mutations, zod form schemas, and co-located tests.
STOPPED task records were never pruned: ListTasks enumerated every task a
cluster ever ran and the per-account KV grew unbounded. Add a leader-gated sweep
tick to the scheduler cron (alongside the reaper/reconcile) that deletes task
records STOPPED longer than a 1h retention window — keeping a just-stopped task
describable for DescribeTasks / UI exit reason, matching AWS's STOPPED window.
A task missing its StoppedAt timestamp is never pruned (defensive). Single-writer
(leader only), reuses the reap() bucket walk.
Wire the four NotImplemented ECS teardown actions so an IaC stack and the
console can reverse a deployment:
- DeleteCluster cascades: force-stops every task (releasing ENIs, deregistering
TG targets, returning capacity), then sweeps the cluster's whole KV subtree.
Unlike AWS it cascades rather than rejecting a non-empty cluster, so terraform
destroy round-trips regardless of teardown ordering.
- DeregisterTaskDefinition marks a specific family:revision INACTIVE (kept
describable); a bare family is rejected, matching AWS.
- DeregisterContainerInstance removes an instance, force-stopping its tasks when
Force is set and rejecting a non-empty instance otherwise.
- UpdateContainerInstancesState sets ACTIVE/DRAINING; draining force-stops the
instance's service-owned tasks so the reconciler relaunches them elsewhere.
Reuses the existing forceStopTask primitive and the reaper drain pattern; adds a
deleteKeysWithPrefix KV helper and INACTIVE status constants.
Serve task-role credentials to containers the AWS-faithful way: an HTTP
endpoint at 169.254.170.2:80 inside the ecs-agent (host VM netns) serving
short-lived credentials at /v2/credentials/{credID}. Containers find it via
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, injected per container.
The agent mints credentials by SigV4-signing sts:AssumeRole(taskRoleARN)
against the gateway with its instance-role credentials, mirroring
internal/ecrauth. This reconciles ecs-v1 Q4/Q10 (STS over NATS + scheduler
JWT assertion) with the 4f rewire that moved the agent entirely onto the
gateway: the agent has no NATS connection and already holds instance creds,
so it assumes the task role directly. Credentials are cached per credID and
refreshed near expiry.
Each awsvpc task netns gets a credentials veth into the host netns and a
host-routed /32 to the endpoint, keeping it off the task ENI's VPC route.
The scheduler now persists taskRoleArn on the task definition and carries it
in the assign payload so the endpoint activates end to end.
* fix: serialize EBS hot-plug per instance to prevent PCIe port race
Concurrent AttachVolume calls to one instance both allocated the lowest free
hot-plug PCIe root port via nextHotplugEBSPort (which reads live QEMU state),
then issued QMP device_add on the same hotplug-ebs bus. The second device_add
failed with 'PCI: slot 0 function 0 already occupied by virtio-blk-pci',
surfacing as nats:timeout -> ServerInternal and a CSI retry-storm
('retry quota exceeded'); volumes never attached.
Add a per-instance attachMu and hold it across the whole AttachVolume and
DetachVolume critical sections, mirroring the ENI hot-plug serialization.
The mutex is non-persisted and shared via the Manager's stable *VM.
* fix: allocate EBS hot-plug PCIe port from in-memory accounting
AttachVolume derived the free PCIe hot-plug root port from a live
query-block scan (nextHotplugEBSPort), extracting the bus with a
hotplug-ebs(\d+) regex on each device's QDev. Real QEMU reports a
hot-plugged blockdev by id (/machine/peripheral/vdisk-vol-xxx/
virtio-backend), with no hotplug-ebs component, so the regex never
matched, the used set stayed empty, and the allocator always returned
port 1. Only the first EBS volume per instance attached; every later
attach collided on "PCI: slot 0 function 0 already occupied by
virtio-blk-pci", rolling back the blockdev, killing NBDKit, and
surfacing as nats: timeout / ServerInternal to the CSI driver, which
then exhausted its retry quota.
Allocate the port from authoritative in-memory accounting instead:
record the chosen port as EBSRequest.HotplugPort (persisted, so it
survives a daemon restart) and pick the lowest unclaimed port with
freeHotplugEBSPort over the instance's recorded requests. Hold
attachMu across allocate->device_add so concurrent CSI attaches cannot
pick the same port. DetachVolume already removes the request, freeing
the port. Drops the query-block round-trip from the attach path.
Julian Sommer