mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-19 10:52:04 +00:00
Josh Siviour
527308c342
* feat(elbv2): localize internal-LB heartbeat failures for baremetal debug Add diagnostics so a stuck-in-provisioning internal LB can be pinned to one of three points on the heartbeat path: - lb-agent failure log now carries the gateway target + region, so the microVM serial console shows whether the heartbeat hit a transport error (never reached AWSGW) or an HTTP-level rejection. - AWSGW LBAgentHeartbeat logs receipt (debug) and warns on the previously-silent not-found/account-mismatch reject path, so the journal distinguishes 'heartbeat arrived but cannot transition' from 'heartbeat never arrived'. * test(e2e/lb): fail fast on stuck internal LB + probe fw_cfg on microvm boot - lb suite: gate the remaining internal subtests on Internal_ALB. The internal LBs share one mgmt-return datapath and fixture, so if the first never reaches active the rest time out identically — skip them instead of burning ~5m per subtest (~20m saved on a known-bad run). - microvm init.sh: widen the fw_cfg netcfg poll window and, on the not-available path, dump DMI / loaded modules / qemu_fw_cfg by_name listing / dmesg fw_cfg to the serial console before init exits, so a boot panic is diagnosable from the captured console log. * fix(microvm): insmod fallback for fw_cfg + fail build on bad modules.dep Baremetal-built microVM images shipped an unusable modules.dep (the build host's depmod errors were swallowed), so busybox modprobe in the guest init silently no-oped, qemu_fw_cfg never loaded, /sys/firmware/qemu_fw_cfg was absent, init could not read opt/spinifex/netcfg, and the LB microVM panicked before lb-agent ran — leaving internal LBs stuck in provisioning. init.sh: load_mod() tries modprobe, then falls back to insmod by explicit path (.ko/.ko.gz) which needs no modules.dep, so the guest boots regardless of which host built the image. build-microvm-image.sh: require depmod, drop the error swallow, and assert modules.dep is non-empty after depmod so a boot-broken image fails the build loudly instead of silently. * fix(microvm): decompress kernel modules before depmod Debian trixie's kmod 34 depmod silently emits an empty modules.dep when fed Alpine's gzip-compressed (.ko.gz) modules — reproduced: 928 modules in, 0-byte modules.dep out. The guest's busybox modprobe then no-ops, qemu_fw_cfg never loads, and internal LB microVMs panic before lb-agent runs (stuck provisioning on bare metal; GitHub-hosted runners ship a kmod that reads .gz, so normal CI was unaffected). Decompress kept modules to plain .ko after the keep-list strip, before depmod. Host depmod then produces a usable map regardless of its gzip support, and the guest loads modules without a decompressor. * feat(e2e): capacity-aware scheduling for dense single-node runs Denominate daemon schedulable capacity in physical cores instead of logical threads, and auto-serialise e2e launches when the cluster is at capacity, so a single hyperthreaded node stops over-packing guests. Daemon: NewResourceManager sizes hostVCPU from physicalCoreCount() (distinct physical-id:core-id pairs in /proc/cpuinfo) rather than runtime.NumCPU(). Falls back to logical CPUs when topology is absent (non-Linux, container, ARM cpuinfo without core id). On a hyperthreaded box this halves the over-pack: with the unchanged 2 vCPU reserve the i5-12600H schedules 10 vCPU (12 cores - 2) instead of 14, keeping the boot-spike CPU within real cores. Harness: capacityRetryEC2 decorates the EC2 client and, on InsufficientInstanceCapacity, waits and retries RunInstances instead of failing — turning the daemon admission gate into back-pressure so concurrent subtests queue. AWSClient.EC2 is now the wrapped ec2iface.EC2API; EC2Conf keeps the raw *ec2.EC2 for .Config access. Adds unit tests for parsePhysicalCores (hyperthreaded/hybrid/dual-socket/ fallback) and the retry decorator (retry-then-succeed, non-capacity passthrough, deadline). * fix(daemon): SPINIFEX_HOST_VCPU override so small CI hosts can start Denominating capacity by physical cores starves tiny hosts: a 2-physical- core CI runner leaves 0 schedulable after the default 2-vCPU reserve, so NewResourceManager refuses to start and every daemon-constructing unit test fails. The previous runtime.NumCPU() count (4 logical on the same runner) incidentally cleared the reserve floor. Add a SPINIFEX_HOST_VCPU escape hatch (symmetric with the existing SPINIFEX_RESERVED_* overrides) that pins the detected core count for hosts whose /proc/cpuinfo topology misreports, and pin it to 4 in the daemon TestMain so unit tests are deterministic regardless of runner topology. Reserve stays at the default 2.
72 lines
2.2 KiB
Go
72 lines
2.2 KiB
Go
//go:build e2e
|
|
|
|
package harness
|
|
|
|
import (
|
|
"errors"
|
|
"log/slog"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/aws/aws-sdk-go/aws/awserr"
|
|
"github.com/aws/aws-sdk-go/service/ec2"
|
|
"github.com/aws/aws-sdk-go/service/ec2/ec2iface"
|
|
)
|
|
|
|
// capacityRetryEC2 wraps the EC2 client so concurrent e2e subtests serialise
|
|
// against real cluster capacity instead of overscheduling. When the daemon
|
|
// reports InsufficientInstanceCapacity (the cluster is momentarily full) it
|
|
// waits and retries RunInstances rather than failing the test; the launch
|
|
// proceeds as soon as a running guest is torn down and frees vCPU/memory. All
|
|
// other calls pass straight through to the embedded SDK client.
|
|
type capacityRetryEC2 struct {
|
|
ec2iface.EC2API
|
|
// interval / maxWait override the defaults; zero means use the constants.
|
|
// Exposed only so tests can drive the retry loop without real-time sleeps.
|
|
interval time.Duration
|
|
maxWait time.Duration
|
|
}
|
|
|
|
var _ ec2iface.EC2API = (*capacityRetryEC2)(nil)
|
|
|
|
// capacityRetryMax bounds the wait so a genuinely wedged cluster (never frees
|
|
// capacity) still fails rather than hanging until the test timeout.
|
|
const (
|
|
capacityRetryMax = 5 * time.Minute
|
|
capacityRetryInterval = 5 * time.Second
|
|
)
|
|
|
|
func (c *capacityRetryEC2) RunInstances(in *ec2.RunInstancesInput) (*ec2.Reservation, error) {
|
|
interval, maxWait := c.interval, c.maxWait
|
|
if interval <= 0 {
|
|
interval = capacityRetryInterval
|
|
}
|
|
if maxWait <= 0 {
|
|
maxWait = capacityRetryMax
|
|
}
|
|
deadline := time.Now().Add(maxWait)
|
|
for attempt := 1; ; attempt++ {
|
|
out, err := c.EC2API.RunInstances(in)
|
|
if err == nil || !isInsufficientCapacity(err) || time.Now().After(deadline) {
|
|
return out, err
|
|
}
|
|
slog.Debug("RunInstances: cluster at capacity, waiting to retry",
|
|
"attempt", attempt, "interval", interval, "err", err)
|
|
time.Sleep(interval)
|
|
}
|
|
}
|
|
|
|
// isInsufficientCapacity reports whether err is the daemon's
|
|
// InsufficientInstanceCapacity admission rejection — the signal to wait for a
|
|
// guest to free resources rather than fail.
|
|
func isInsufficientCapacity(err error) bool {
|
|
if err == nil {
|
|
return false
|
|
}
|
|
var ae awserr.Error
|
|
if errors.As(err, &ae) {
|
|
return ae.Code() == "InsufficientInstanceCapacity"
|
|
}
|
|
return strings.Contains(err.Error(), "InsufficientInstanceCapacity")
|
|
}
|