mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
4793b5ac09
* fix(eks-node): stop the DHCP MTU overriding the pinned link MTU
Pinning eth0 to 1320 was never enough. This image runs dhcpcd, which
honours DHCP option 26 and stamps the advertised MTU onto every route it
installs:
default via 10.32.101.1 dev eth0 proto dhcp src 10.32.101.7 \
metric 1002 mtu 1442
Linux prefers a route's MTU metric over the device MTU, so the node kept
advertising an MSS derived from 1442 while the real path MTU through the
NAT gateway is 1408. Small packets pass and full-MSS inbound segments —
a TLS ServerHello with its certificate chain, image layers — are
silently dropped. It presents as `net/http: TLS handshake timeout` on
image pulls, not as anything MTU-shaped, and it only affects the host
netns, so containerd pulls fail while pod traffic sized off cni0 is
fine. Per-destination PMTU exceptions expiring in and out of the route
cache made it look intermittent and host-specific.
Three changes, innermost first:
- dhcpcd.conf gets `nooption interface_mtu`, so no lease can stamp a
route MTU. mulga-vpc-mtu is the single source of truth for the MTU
of these nodes.
- mulga-vpc-mtu re-stamps any route that already carries a different
MTU, covering an image built before this fix or a dhcpcd already
running. Routes with no MTU metric inherit the device MTU and are
left alone.
- the subnet DHCP option drops 1442 to 1408. 1442 is geneve-only and
ignores the IPsec ESP overhead on the NAT egress path, so it
overstates the usable MTU for every guest, not just these nodes.
Also corrects the header comment, which claimed Alpine's udhcpc ignores
option 26 — this image runs dhcpcd, which honours it. That wrong premise
is why the route metric was never considered.
Adds a stubbed-`ip` test covering the rewrite, the dev token that
`ip route show dev X` omits, idempotency, and routes without an MTU
metric.
* fix(eks): sweep the CP VPC by identity so teardown can finish
DeleteClusterCPVPC finds the managed control-plane VPC's subnets and SGs
by role tag. A create that fails partway — the first control-plane
server failing to launch, say — leaves resources behind that never got
tagged, so the sweep cannot see them.
DeleteVpc rejects while any subnet or non-default SG remains, and
teardown is re-driven on a timer, so the cluster sits in DELETING
forever retrying the same doomed delete every couple of minutes:
eks-deleting: re-driving wedged DELETING teardown cluster=toc
eks-deleting: re-drive teardown failed
err="delete managed CP VPC: eks: delete cp vpc vpc-...: DependencyViolation"
Nothing converges: the cluster name cannot be reused and terraform's
delete wait can never pass. Observed on env19, wedged for ~11 hours
until this change went in, at which point the next re-drive cleared it.
After the tagged sweep, clear anything still in the VPC scoped by
vpc-id: subnets, and non-default SGs (the default SG goes with the VPC
cascade). Teardown had no SG surface, so CPVPCDeps gains an optional SG
provisioner — nil is tolerated, since only teardown uses it.
* chore: pin viperblock + predastore to the merged storage fixes
viperblock f8cfd77 (#42) makes read-modify-write atomic per block,
removing the lost update that silently reverted a 4096-byte block to a
previous generation of itself. Confirmed as the env19 corruption cause
by an A/B: the pre-fix build corrupted image layers within ~15 minutes
(failed to extract layer / invalid tar header / unpigz: corrupted) while
the fixed build ran 5+ hours clean, absorbing 6407 same-block RMW
conflicts on the worker root disks.
predastore c0a90d2 (#63) batches FSM.Restore. Without it, any predastore
restart on a store whose metadata set outgrew one badger transaction
fails with "Txn is too big to fit into one request" on every node at
once, taking the metadata plane — and with it the AMI catalogue and
cluster creation — down permanently.
Both were verified in the built artifact rather than from go.mod alone:
go version -m reports the new pseudo-versions, and the rmw_conflicts
counter plus the batched-restore error strings are present in the
binary. Pinning by go.mod while the submodule pointer says otherwise is
how an earlier analysis ended up reading dead code.
* test: record the corrected DHCP MTU in the kitchen-sink golden
The NB golden pins the DHCP options published per subnet, including the
advertised MTU. Dropping the subnet option from 1442 to 1408 changes
what the reconciler writes, so the golden diverged and the live scenario
test failed — the test doing exactly its job.
Regenerated with UPDATE_GOLDEN=1. The diff is only mtu:1442 -> mtu:1408
on the two subnet DHCP records; nothing else in the snapshot moves.