fix: stop the detach seal, EIP teardown and key-pair races failing the nightly (#1003)

* fix: stop the detach seal, EIP teardown and key-pair races failing the nightly

Four independent faults the full-suite nightly surfaced, each of which had
been masked while the heavy cells were skipped.

unmountVolume SIGKILLed nbdkit the moment the request arrived. Removing the
guest device has already started the plugin's own Close, which drains to the
backend and leaves a seal receipt, so the kill landed 61ms into that flush and
threw it away — the receipt path never fired once across a whole run. What was
left was 74 orphaned WAL files for the fallback seal to replay, which ran past
the caller's 180s deadline and returned DetachVolume ServerInternal with the
volume still attached. It now terminates gracefully first, bounded by the
plugin's own drain budget, and only escalates if that does not take.

DeleteEIP returned early when the dnat_and_snat row was already absent, before
the neighbour flush and the host unbind. An absent row is not a reassignment —
nobody owns the address — so the /32 route and proxy-ARP entry were orphaned
and a disassociate left the address still answering from the host. The
reassignment cases keep their early return.

EnsureKeyPair was not atomic: a failure after CreateKeyPair left the pair on
the cluster with no cleanup registered, so every later caller failed with
InvalidKeyPair.Duplicate in a test that never ran. The describe that follows a
create also had no tolerance for the propagation window on a multi-node
cluster; it now retries, and still reports a duplicate the moment it sees one.

An admission refusal could not name its cause. The static vCPU/memory budget
and the live MemAvailable clamp both surface as InsufficientInstanceCapacity,
and they have different fixes; the subscription transition now records which
gate closed.

* fix: identify the default-deny by priority, not by being a drop

DenyIsNotEthertypeScoped bucketed every drop ACL by direction, so the last one
seen won each direction. The platform WAN egress block added at priority 1049
is also a drop and is legitimately ip4-scoped, so it displaced the default-deny
in the map and the assertion fired on its qualifier instead:

  from-lport deny on sg_c94870c0d68288dc6 is IPv4-scoped:
  "inport == @sg_c94870c0d68288dc6 && ip4 && ip4.dst != {10.0.0.0/8, ...}"

Select the two default-denies by their own priorities. What the test asserts is
unchanged — the default-deny still must carry no ethertype qualifier, for the
reason denyIngressACL states: OVN allows on no-match, so an ip4-scoped deny
lets every other ethertype through.

* fix: drain every EIP on an ENI at teardown, not just the first

findByENI returned the first matching record and stopped, so an ENI holding
more than one EIP had all but one left marked associated to an interface being
deleted — with the host route still installed and no record left naming the ENI
to find them. Teardown now clears them all.

* ci: serialise e2e-nightly runs so two do not share the hypervisors

Only iso-single and iso-multi were guarded, by the iso-qemu-rack group. Every
other job could overlap a second run of the workflow, with both sets of cells
building clusters on the same five hosts. Queue rather than cancel: a dispatch
is usually a verification run whose result someone is waiting on.

.github/workflows/e2e-nightly.yml

@@ -16,6 +16,14 @@ permissions:
   # context exposes the id itself.
   actions: read
 
+# Every cell builds its cluster on the same five hypervisors, so two runs in
+# flight at once compete for their disks and their libvirt guests. Queue the
+# second rather than cancelling it: a dispatch is usually a verification run
+# and discarding it loses the result someone is waiting on.
+concurrency:
+  group: e2e-nightly
+  cancel-in-progress: false
+
 # Scheduled runs are loaded from the default branch (main), so the schedule
 # trigger must stay in sync on main for cron to fire.
 env: