Files
spinifex/contracts
Bryn Mailer 020319c44a fix(ec2): classify a volume held elsewhere by the provider code, not lease text (#1496)
* fix(ec2): classify a volume held elsewhere by the provider code, not lease text

The instance service and the EKS cluster reconciler recognised a start
refused because another node holds a volume only by viperblockd's lease
text, so a provider that words the refusal differently was reported as
ServerInternal and counted against the reconciler's restart budget.

The instance service now matches the EBS provider's volume_held_elsewhere
code, on both the stopped-instance claim and the live-owner start. The
owning node marks such a refusal with an X-Volume-Held-Elsewhere reply
header, the ec2.start forwarder relays it, and StartInstances wraps
ec2v1.ErrVolumeHeldElsewhere around the rebuilt error, which the
reconciler matches with errors.Is. The AWS code stays IncorrectState.

The lease text stays as a fallback for replies that carry no code: the
legacy ebs.mount reply, a publish below schema version 3 (where
volume_in_use also means a read-only mismatch), and an owner node that
predates the header.

* fix(daemon): relay every header of the owner's start reply on both forwarder paths

The ec2.start forwarder copied only the held-elsewhere header onto a
relayed reply, and its unsealed-writes branch answered with the body
alone, so any other header the owning node set was lost in the relay.

Both relay branches now go through relayStartReply, which forwards the
owner's reply body and every header unchanged.

* test(daemon): pin that only a held volume gets the held-elsewhere header

The owner's reply header is the one signal that a start was refused
because another node holds a volume, and every other start refusal in
the same IncorrectState code must reach the caller without it.

* refactor(ec2): classify a volume held elsewhere only by the provider code

Every node sends provider requests at the one contract version, at which
every provider refuses a volume held by another node with the
volume_held_elsewhere code. The lease-text fallbacks in the instance
service and the EKS cluster reconciler matched replies that no longer
exist, so they are removed: a start is a held-volume refusal only when
the mount error carries the provider code, or when the start error wraps
ec2v1.ErrVolumeHeldElsewhere, which StartInstances rebuilds from the
owner's header.

The ec2instance.ErrVolumeHeldElsewhere alias is removed, so the sentinel
has one name, ec2v1.ErrVolumeHeldElsewhere. The reconciler's budget test
now proves an IncorrectState without the sentinel still counts, whether
its message is unrelated or carries the lease text.

* fix(ec2): log a held-volume start refusal at Warn with the instance ID

A start refused because another node holds a volume is returned as
IncorrectState, a client error, yet was logged at Error without the
instance it concerned. It is now logged at Warn, and every vmMgr.Start
failure log names the instance.
2026-11-04 10:13:23 +10:00
..