mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
53b31cb76ca12f8a48288132bee97d30a8348156
317 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Julian Sommer
|
53b31cb76c | fix: keep the RDS master password out of the server and console logs (#754) | ||
|
Josh Siviour
|
d178ef5992 |
feat: version-aware host-asset preflight for binary-swap deploys (#750)
A binary-swap deploy (make deploy or a manual spx copy) ships a new binary whose host-config prerequisites — the privileged helper scripts and their sudoers grant, placed only by setup.sh / setup-ovn.sh — it does not ship. The node looks healthy and the gap surfaces later as a cryptic launch failure or a silent security regression. Add a preflight package that checks the host against what this build expects, comparing asset content, not existence, so a stale-but-present asset is caught too. The managed helpers install from quoted heredocs, so their on-disk bytes equal the heredoc body verbatim; the binary carries the canonical sha256s and compares. - spinifex/preflight: CheckHost / CheckHostAt, content hash of the two managed helpers and a NOPASSWD grant-line check for the sysctl helper. - Generated canonicalHashes plus a drift guard that re-extracts the heredocs and fails if the committed manifest no longer matches. - Three surfaces: spx admin preflight, a make deploy gate (SKIP_ASSET_PREFLIGHT=1 to bypass), and a non-fatal daemon boot warning. |
||
|
Josh Siviour
|
bc1dd3413e |
fix(bedrock): make the reaper's decision visible to an operator (#746)
Three gaps found while verifying idle reclaim on live hardware. The reclaim log line passed a time.Duration, which slog renders as an int64 nanosecond count in JSON: the line read idleFor:135000000000 rather than 2m15s, in journalctl and in the metrics sink alike. It now logs an integer second count under a unit-suffixed field, which is both readable and still aggregatable — a string would have cost the latter. The same shape appears at around ten other sites across the fleet and is tracked separately. The endpoint command's help still told operators the gateway could not request endpoints itself, which stopped being true when dynamic resolution landed. It now describes the lifecycle that actually runs, and says what the manual commands are still for. describe showed nothing the idle sweep observed, so an operator asking why an endpoint was or was not reclaimed had to read the KV. It now reports in-flight count, last activity, idle duration, pinned status and consecutive scrape failures for a READY endpoint, and nothing for any other state, since no other state is swept. LastActive moves onto EndpointRecord so the idle clock, the LRU eviction order and this output all read one definition rather than three. |
||
|
Josh Siviour
|
0a1d837f40 |
feat(bedrock): gate Ochre models behind per-account access grants (#741)
* feat(bedrock): gate Ochre models behind per-account access grants Self-host models were invokable by any authenticated account: tieredCatalog returned them unconditionally and no runtime router checked the caller. GetFoundationModel also described models the caller could not use. Add a bedrock-model-access JetStream KV bucket holding presence-only grants, resolved through a new AccessResolver. Access is deny-by-default: a gateway with no store configured falls back to DenyAllAccessResolver, so an unconfigured deployment serves nothing rather than everything. The system account bypasses grants, matching how quotas exempt it. All four runtime routers share one grantedCatalogEntry gate so they cannot drift on who may invoke what. List and describe agree: an ungranted model is absent from ListFoundationModels and reported as ResourceNotFoundException by GetFoundationModel, so the error cannot confirm a model exists. Administration lands on two surfaces: Grant/Revoke/ListModelAccess actions on the spinifex gateway service for the UI, and `spx admin ochre access` for operators and provisioning, which opens the bucket over NATS directly rather than requiring a SigV4 client the CLI does not have. * feat(bedrock): seed the admin account's model grants at gateway start Deny-by-default left a fresh install with a catalog nobody could see, and the first cut filled that gap from ansible — which is a dev tool, so any install that never ran it came up unusable. Seed from the product instead: on start awsgw grants the platform admin account the full catalog, once, behind a marker key in the same bucket. The marker makes it a bootstrap step rather than a policy: tenant accounts still begin with no access, and an operator who revokes a grant does not have it restored by the next restart. Grants are written before the marker so a crash midway repeats the idempotent grants rather than leaving a half-seeded account, and the marker is a conditional create so only the first node to reach it need win. Seeding is best-effort: a KV failure logs and leaves the gateway serving, and the next start retries because the marker is written only on success. * test(bedrock): cover the model-access CLI and gateway admin actions Route the access CLI runners through the same loadConfigAndConnectFn / ochreExit seam the weights runners use, so grant, revoke and list can be driven against an embedded JetStream server, and add gateway tests for the grant admin actions including the non-admin denial. |
||
|
Julian Sommer
|
3379b16586 | fix: encrypt and acknowledge the RDS bootstrap password (#739) | ||
|
Josh Siviour
|
272b4b22e8 |
feat(ochre): add the vLLM serving AMI and give the daemon a route to it (#740)
* feat(images): add a vllm-serving AMI as a stacked mkosi profile
LaunchServingVM resolves its AMI by the tags spinifex:managed-by=bedrock and
spinifex:bedrock-role=vllm-serving. No image carried either, so every Ensure
failed at resolveServingAMI. The only GPU image ships the NVIDIA driver and
Docker but no Python, no vLLM and no serving unit, so tagging it would have
moved the failure from admission to first boot.
Add a vllm profile that stacks onto gpu-nvidia and owns the whole guest side
of the launcher contract: the runtime, vllm-serve.service, the weights mount
and the tags. vLLM is baked into a uv-managed venv rather than pulled as a
container at boot — a runtime docker pull of vllm/vllm-openai is known to
stall this platform's write plane, and mkosi's unprivileged build has no
daemon to bake an image from anyway.
Also bring images/mkosi.profiles/ under make test-images. Its scripts have no
.sh suffix, so the existing name glob skipped exactly the files worth linting;
sources are found by shebang instead.
* fix(images): make the vllm postinst survive the build sandbox
Two failures found by actually running the build.
uv installs to /usr/local/bin, which mkosi's postinst PATH need not carry, so
`command -v uv` reported a good install as missing. Call it by absolute path,
and check it the same way.
`vllm --version` initialises device config and dies with "Failed to infer
device type" on a builder with no GPU, so it could never have verified
anything. Assert the pinned version through package metadata instead, which
also catches a version mismatch rather than merely a broken entrypoint.
* fix(images): pin the serving AMI datasource and mirror cloud-init to the console
A serving VM boots through LaunchSystemInstance, so it comes up dual-NIC with a
fw_cfg netcfg blob and takes its user-data solely from IMDS — the same shape as
the EKS control plane. That path carries a cloud-init datasource pin; the vllm
profile never got one, and the base config deliberately leaves datasource_list
unpinned, so the serving image was the only system VM booting without it.
Reuse the EKS pin rather than copying it. The 99- prefix is load-bearing:
read_conf_d() applies cloud.cfg.d in REVERSE and mergemanydict() is
first-source-wins, so a 90- name loses to cloud-init's own 90_dpkg.cfg ('_'
sorts above '-') and the pin is discarded in silence. The postinst asserts the
MERGED datasource_list via cloud-init's own read_conf_with_confd rather than
checking the file exists, because it existed at 90- too and did nothing there.
Also mirror every cloud-init stage to the serial console. A serving VM is
headless, and a guest that wedges before handover never surrenders
/var/log/cloud-init-output.log, so the console get-console-output reads is the
only channel an operator has.
The images_test.go case covers the two tags resolveServingAMI filters on; it
matches on those alone and never on name, so a typo hides the image entirely.
* fix(images): make the dhcpcd data-NIC lease genuinely one-shot
dhcp_oneshot() promised to configure the link and exit "leaving no daemon
to fight cloud-init", but the dhcpcd branch passed only -q, which is quiet
rather than one-shot. Every Ubuntu image takes that branch, so a dhcpcd
daemon survived on the data NIC holding the control socket.
cloud-init's Ec2 datasource then runs its own ephemeral dhcpcd on the same
NIC. Instead of spawning, dhcpcd forwards the invocation to the resident
daemon as a control command and exits 0, and --dumplease returns the
resident daemon's lease, so discovery looks successful. Cleanup then waits
for a pid file at /run/dhcpcd/<iface>-4.pid that no process ever wrote,
spinning its full 30000 x 10ms budget: init-local stalls for exactly 300s
on every system-instance boot.
Adding -1 exits once the interface is configured and -p keeps the address
and routes across that exit. Both are needed; -1 alone tears the link back
down. The test now models the real clients — the stub stays resident unless
given the flag that makes its client exit, and the dhcpcd branch is forced
by dropping udhcpc from PATH rather than left to whether the host happens
to have /usr/share/udhcpc/default.script.
* fix(images): serve without a CUDA toolkit and stop vllm-serve crash-looping
vLLM picks FlashInfer for top-k/top-p sampling, and FlashInfer JIT-compiles
that kernel on first use. The serving image ships the driver and the
container toolkit but no nvcc, so the sampler warmup died with "Could not
find nvcc and default cuda_home='/usr/local/cuda' doesn't exist" minutes
into a GPU cold start, after the model had already loaded.
VLLM_USE_FLASHINFER_SAMPLER=0 selects vLLM's native top-k/top-p sampler,
which is the same operation with no CUDA toolchain at runtime. Everything
else on the serving path is already precompiled in the wheels: the attention
backend resolves to FLASH_ATTN and CUDA graph capture completes, both
without nvcc. Baking FlashInfer's AOT modules is the alternative if the
fused kernel is ever wanted, and it keeps the toolchain in the builder
rather than the appliance.
The unit had Restart=on-failure with no bound, so an unfixable start failure
restarted forever and buried its own first traceback under identical copies
on the serial console — the only view an operator has of a headless guest.
StartLimitIntervalSec/StartLimitBurst make it report instead.
The postinst asserts the installed unit carries the sampler setting, for the
same reason the datasource pin is asserted there: the alternative is finding
out at runtime, on a GPU, five minutes in.
Also corrects the shared datasource pin's header, which still claimed it
installs as 90-spinifex-ec2-imds.cfg. It is installed at 99- by the vllm
profile precisely because a 90- name loses the reverse-sorted, first-source-
wins cloud.cfg.d merge to cloud-init's own 90_dpkg.cfg.
* fix(bedrock): raise the serving-endpoint readiness bound to 15 minutes
The 5 minute bound was chosen before any cold start had been measured, and
its own comment asked to be corrected once a model actually ran on wattle.
It now has: Llama 3.2 1B on an RTX A1000 reached "Application startup
complete" 257.6s after launch, leaving ~42s of margin on the smallest model
this platform can serve, with a cold torch.compile accounting for 26.3s of
it. A bigger card and a bigger model are unmeasured.
The two failure modes are not symmetric. Too low tears down an endpoint that
was about to come up and reports it as a failed launch; too high only makes
a genuinely stuck launch take longer to give up. So the bound is set well
clear of the single data point rather than close to it.
The CLI's own wait moves to 20 minutes to stay above it. The daemon starts
its clock when the VM is launched, up to a minute after the CLI starts its
own, so leaving both at 15 would expire the client first and report a bare
"timed out" in place of the daemon's abort, which carries the reason.
* fix: give the Ochre daemon a routed port in the Bedrock system VPC
The readiness probe and every InvokeModel call dial a serving VM's
system-VPC address from inside the daemon, but that subnet lives in OVN
on br-int and the host has no route to it, so a healthy VM never reached
READY.
Each daemon now owns an ENI in the system VPC's private subnet, bound to
an OVS internal port on br-int by iface-id, exactly as ovn-controller
binds a guest tap. The ENI address is installed at the subnet's prefix
length, which is what supplies the connected route. BaseURL is unchanged:
the address it already carried is now reachable.
Describe-or-create keyed on a per-node description, since ENIs carry no
tag filter, so a restart or a second endpoint reuses one address instead
of leaking one per launch. The port is ensured before anything else the
launch creates: a VM the daemon cannot dial holds a GPU for the whole
readiness window before unwinding.
|
||
|
Julian Sommer
|
e3b5701f15 | fix: authorize RDS data-volume formatting by exact volume identity (#737) | ||
|
Josh Siviour
|
688ce5ccfd | feat: add AWS model conformance coverage (#736) | ||
|
Julian Sommer
|
506a8126b7 | fix: propogate rds heartbeat failures | ||
|
Julian Sommer
|
68671cdf19 | fix: prevent parameter groups bricking RDS instances (#731) | ||
|
Julian Sommer
|
871513dc6c |
Reapply "feat: match new predastore config surface area (#591)"
This reverts commit
|
||
|
Julian Sommer
|
b4701c78de |
Revert "feat: match new predastore config surface area (#591)"
This reverts commit
|
||
|
Julian Sommer
|
615578a2e7 | refactor: simplify RDS internals (#692) | ||
|
Julian Sommer
|
2e09461080 | chore: cleanup code comments | ||
|
Josh Siviour
|
43975b01e6 |
feat: add spx admin ochre endpoint ensure/describe/list/delete (#691)
* feat: add spx admin ochre endpoint ensure/describe/list/delete The daemon subscribes all four bedrock.endpoint.* subjects, but nothing in production calls their NATS client, so the serving-endpoint lifecycle could not be exercised at all in a running system. This gives an operator a direct way to drive it. ensure --wait polls describe until READY and reports the elapsed time, since the daemon replies STARTING as soon as it claims the model and the launch outlives the request. That is the measured cold start. Also corrects the weights stage tokenizer rule in COMMANDS.md, which still described both tokenizer files as required rather than either. * test: cover the ochre endpoint Run wrappers' connect and exit paths * fix: treat an empty endpoint bucket as no endpoints, not an error ListEndpoints was the only kvutil.Keys caller not handling ErrNoKeysFound, so 'ochre endpoint list' failed on a cluster that had never launched one. |
||
|
Josh Siviour
|
49c5a04bed |
fix: register ochre weights snapshot with the EC2 control plane (#690)
weights stage created the snapshot at the viperblock layer only, writing the block checkpoint and config.json but not the metadata.json that CreateVolume resolves a SnapshotId through. The endpoint launcher clones the weights volume via ec2.CreateVolume, so every launch failed with InvalidSnapshot.NotFound against a snapshot that was present on disk. |
||
|
Julian Sommer
|
050dbdb060 | fix: disallow reserved master names during rds password apply | ||
|
Julian Sommer
|
8cf44f3310 | fix: harden RDS beta provisioning and replies (#689) | ||
|
Ben Duncan
|
b76aab8b50 | feat(installer): three-plane networking, ZFS root, distributed NAT and multi-node install (#663) | ||
|
Josh Siviour
|
f3dd3c251e |
feat: add spx admin ochre weights stage/list/remove (#680)
* feat: add spx admin ochre weights stage/list/remove Give operators a supported path to stage a self-hosted model's weights: point at a predastore S3 prefix holding ordinary Hugging Face files, and stage materialises them into a viperblock volume, snapshots it, and records the source URI and snapshot ID in the bedrock-weights KV bucket. stage refuses before downloading anything if the model is not a self-host catalog entry, or if the prefix is missing any required file. Re-staging the same source is a no-op; a different source replaces the entry and reports the previous snapshot for separate reclamation. WeightsStore gains GetWeights and a source-URI field on its record to support this. LookupServingSpec (catalog.go) is reused unchanged to validate --model-id. * fix: unify ServingSpec after rebase onto dev's daemon-launcher spec Rebasing onto dev pulled in mulga-vmfng.7.5's daemon-side ServingSpec and LookupServingSpec, textually duplicating the ochre weights stage CLI's own versions in catalog.go. Collapse both into one type carrying ModelID and VLLMArgs, and one three-value LookupServingSpec so callers distinguish an unknown model (found=false) from a provider-tier one (selfHost=false). Update the daemon launcher's call site accordingly. * test: cover ochre weights stage/list/remove command behaviour Splits runOchreWeightsStage/List/Remove into testable decision functions (runStageWeights, listWeightsOutput, removeWeights) separate from cobra and NATS wiring, and makes mkfs.ext4 execution and the connect/exit steps swappable so the full command flow can be driven against an embedded JetStream server instead of live infrastructure. Also relaxes the tokenizer file requirement to accept either tokenizer.json or tokenizer.model, since canonical Llama 3.x repos ship only the former at the prefix root. * fix: resolve mkfs.ext4 from sbin when it is absent from PATH e2fsprogs installs mkfs.ext4 into sbin, which a non-login shell omits, so weights staging failed on an unprivileged operator's otherwise fine host. * fix: size the weights volume to a whole GiB so the tail block fits viperblock rejects a write whose block overruns VolumeSize, so importing an image of unaligned length failed on its last block. Rounding up also makes the volume's byte size and its manifest SizeGiB describe the same volume. Name the volume in the import error so a part-written import is traceable. |
||
|
Josh Siviour
|
8be25c93f9 |
fix: surface faults on the AMI-read and service-ownership paths (#679)
DescribeImages skipped any ami-* prefix whose config.json could not be fetched, read, or parsed, logging each at DEBUG. At the daemon's default INFO level the result was an empty Images list and a completion line reading count=0, indistinguishable from a healthy empty registry. Promote the three fault branches plus the body-close failure to WARN, leave NoSuchKey at DEBUG since a prefix with no config is a legitimate non-AMI directory, and report prefixes examined alongside images returned so a mismatch is visible without raising the log level. SetServiceOwnership chowned per-service directories to service users without propagating a missing user, so spx admin init reported success while leaving /etc/spinifex/northstar root-owned 0600. ChownRecursive and SetServiceOwnership now return errors, failures are aggregated across every service rather than stopping at the first, and finalizeNodeSetup prints the unapplied paths instead of a success mark. |
||
|
Josh Siviour
|
b5fa0ffb7a |
fix: stop viperblockd discarding the configured S3 region (#677)
The viperblock start command applies --s3-host, --s3-bucket and --s3-region over spinifex.toml whenever viper reports a non-empty value. s3-bucket and s3-region carried non-empty defaults, so the override fired on every start even though the service unit passes neither flag nor environment variable, and the region read from the config was replaced with ap-southeast-2. Any cluster deployed to another region could not launch an instance: viperblockd signed its S3 requests with ap-southeast-2, predastore verified them against its own region and rejected them as AuthorizationHeaderMalformed, the volume mount failed and RunInstances ended in launch_failed. The bare-metal e2e cell, which installs with --region bare-metal, failed every instance-backed test this way. Default all three to empty, matching s3-host, so an unset flag leaves the configured value alone while an explicit flag or SPINIFEX_VIPERBLOCK_* still wins. The predastore.toml template also pinned the main bucket to a literal ap-southeast-2 while the top-level key and the northstar bucket both rendered the configured region; it now renders it too. |
||
|
Julian Sommer
|
5e17169661 | feat(e2e): complete the RDS end-to-end suite (#608) | ||
|
Bryn Mailer
|
c2b80a9b93 |
feat: match new predastore config surface area (#591)
* match new predastore config surface area
* fix(migrate): carry existing installs onto the predastore v5 topology
The cluster topology refactor breaks an install three ways at once. The old
config carries a top-level host key, which the new schema binds to the [[host]]
table array, so TOML rejects the document and the daemon dies before it reads
any topology. Node state moved from distributed/{db,nodes}/node-N to a per-host
data directory keyed by a node id that is now unique across roles. And because
object metadata records which node holds each shard, the shard nodes had to
keep their ids, so the state replicas took new ones and every replica's
persisted raft configuration stopped listing it.
predastore.toml gains a 4 to 5 migration covering all three: it rewrites the
tables, relocates whichever node directories exist on this machine, and
recovers each relocated state replica onto the new member set. The rewrite is
line-oriented rather than a re-render from the template, so buckets, auth,
rate limits and iam settings an operator edited survive.
spinifex.toml gains a 3 to 4 migration renaming the predastore node_id to
host_id, which predastore-start.sh greps to select the host this machine runs.
The predastore migration lives in its own package rather than in migrate,
which every handler that stamps a KV bucket version imports; reaching the
cluster runtime from there would pull raft, badger and the QUIC client into all
of them. admin_upgrade.go blank-imports it, so only the binary that runs the
upgrade links it, and a test pins both registrations.
* test(predastore): cover the service runtime, gateway and shutdown paths
The service wrapper's new seams had no unit tests: buildRuntime, serve and
the errgroup Start runs the cluster and the S3 gateway under. CI's diff
coverage gate measured the PR at 34.6% against a 60% threshold as a result.
Cover every buildRuntime error wrap and its host selection, both ways the
gateway can fail serve, and the full lifecycle: Start brings the nodes and
the gateway up together, Shutdown drains both by cancelling the context
they share. Topologies are written per test as real config, with node ids
from a process-wide counter — the pipe transport's listener registry is
global, and this package's integration tests hold a fixture daemon on the
low ids for the life of the test binary.
|
||
|
Josh Siviour
|
bda08e1afd |
fix: namespace colliding viper keys and bind viperblock flags to their own command (#661)
* fix: namespace remaining viper keys shared across service subcommands
predastore, nats and awsgw registered "port", "host", "debug", "tls-cert",
"tls-key", "encryption-key-file" and "node-id" as bare, unqualified viper
keys. viper.BindPFlag overwrites the same key's registered pflag object on
each call, so only the last subcommand registered in init() kept its CLI
flag live; the others read a foreign flag's (usually unset) value.
Bare keys are also vulnerable to AutomaticEnv: viper checks the
SPINIFEX_<KEY> derived name before an explicit BindEnv target, so a
same-named env var set for one subcommand also feeds viper.Get calls in
every other subcommand sharing that key, or a generic SPINIFEX_DEBUG-style
var could shadow all of them at once.
Namespace each affected key per subcommand ("predastore-port",
"nats-debug", "awsgw-host", etc.) following the pattern already used for
predastore-host/base-path/config-path and spinifex-ui-*, so each key's
AutomaticEnv-derived name matches its own BindEnv target. CLI flag names
and environment variable names are unchanged. nats's own bind-host flag
uses "nats-service-host" rather than "nats-host", since that key is already
bound by rootCmd's own cluster-wide --nats-host override.
* fix: bind viperblock flags through the command that declares them
s3-host, s3-bucket, s3-region and plugin-path are declared on viperblockCmd
but were looked up on predastoreCmd, which declares none of them. The lookup
returned nil, BindPFlag rejected it silently, and the discarded error hid it.
Both the flags and their declared defaults were invisible to viper, so
s3-bucket resolved "" rather than "predastore".
|
||
|
Josh Siviour
|
bbf1ca7f0b |
fix: stop viper AutomaticEnv from shadowing predastore env vars (#659)
* fix: stop viper AutomaticEnv from shadowing predastore's own env vars Viper resolves AutomaticEnv before explicit BindEnv, so predastore's bare "config-path", "host" and "base-path" viper keys auto-derived to SPINIFEX_CONFIG_PATH, SPINIFEX_HOST and SPINIFEX_BASE_PATH - coincidentally the same names used elsewhere (the global cluster config path, and root's global --host override). Setting SPINIFEX_CONFIG_PATH for the cluster config (needed by derivePredastoreBind) silently made predastore parse the cluster config as its own S3 config, breaking auth for every signed request. Rename predastore's host/base-path/config-path viper keys to "predastore-"-prefixed keys whose AutomaticEnv-derived name is identical to their explicit BindEnv target, so no shadow is possible regardless of what else is set in the environment. * style: trim predastore env-binding comments to the 3-line limit |
||
|
Josh Siviour
|
5de598e85a |
ci: trim unused E2E builds, move IMDS to the SDK, derive predastore bind in spx (#656)
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs These two jobs immediately rebuild suite-aware binaries after Build Install Artifacts and never consume the tar's _bin contents, so the cross-compiled suite build was pure waste. build-install-artifacts.sh gains an optional --skip-e2e-build flag (positional contract unchanged); e2e-nightly.yml and the other callers keep building the full set. * refactor: replace hand-rolled IMDS client with AWS SDK v2 internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials provider used a hand-written IMDSv2 token/GET flow with manual RFC3339 expiry parsing. A malformed Expiration silently fell back to "never expires" instead of failing. Replace it with feature/ec2/imds, credentials/ec2rolecreds and aws.CredentialsCache, following the internal/gwsign/signer.go precedent. An unparseable Expiration is now a hard error from Fetch, covered by TestFetch_BadExpirationErrors. Output shapes to callers are unchanged, and IMDS base URLs ending in /latest keep working. * refactor: derive predastore bind config from spinifex.toml in spx itself predastore-start.sh awk-parsed spinifex.toml for the bind host/port and node_id, exported them as env vars, then exec'd spx. Move that derivation into predastoreStartCmd via derivePredastoreBind, which reads the raw (non-DIAL-normalized) nodes.<node>.predastore section, defaulting node_id to -1 (co-located) when the key is absent, same as the script. An explicit --host/--port/--node-id flag or SPINIFEX_PREDASTORE_* env var still overrides the derived value. Delete the now-unused script and point the systemd unit's ExecStart directly at spx service predastore start, matching the spinifex-viperblock.service precedent. |
||
|
Josh Siviour
|
031f0a4cdd |
fix: preserve running guests across target restart, harden state write (#654)
* fix: preserve running guests across target restart, harden state write spinifex-shutdown.service's ExecStop fired on any spinifex.target stop, including the stop half of `systemctl restart spinifex.target`, draining every guest via QMP even though the host never went down. Gate the drain behind `systemctl is-system-running` == "stopping", which only holds once systemd is genuinely unwinding into shutdown.target (reboot/poweroff), so a target restart or plain `systemctl stop` now leaves QEMU/nbdkit running (KillMode=process) for the daemon to reconnect to on the way back up. Daemon.WriteState returned as soon as the local state file write failed, skipping the independent JetStream KV write entirely. A crash in that window left a live VM with no durable record anywhere. Attempt the KV write regardless of local disk health, then still surface the local write error to callers, mirroring finalizeTerminated's handling of the same fault class on the terminate path. * fix: gate the shutdown drain via an spx flag, not a shell script Replace the shutdown-drain-gate.sh gate with a --only-if-host-stopping flag on `spx admin node drain`. The unit now points at /usr/local/bin/spx directly (already required for the node to function) instead of a script installed under the data directory, which a partial rebuild of that directory could silently drop without ever failing ExecStop. The gate logic also becomes a plain Go function with injectable state lookup, so it gets real table-driven coverage instead of only a live reboot. hostIsStopping() drains only on a literal "stopping" from `systemctl is-system-running`. Every other recognized state (running/degraded/...) and any unrecognized output skip the drain; a fully unreachable systemctl fails toward draining, since a skipped drain on a real shutdown hard-kills guests and loses unflushed writes, while a spurious drain on a restart only costs an extra graceful stop/relaunch cycle. Run by hand without the flag, the command still drains unconditionally as it always has. * refactor: split the shutdown-gate comments to sit with the code they explain |
||
|
Josh Siviour
|
ae06234009 |
fix(spx): detect a production install from a marker a reset cannot remove (#651)
isProductionLayout() keyed off /etc/spinifex alone. A node reset legitimately removes that directory, so spx silently fell back to the dev layout and rebuilt the whole node — configs, keys, nats.conf — under the invoking user's home while the systemd units kept reading /etc/spinifex. Cluster formation reported success and every service then failed to start. Detection now also accepts the installed systemd target, which setup.sh writes and a reset never touches, so the layout survives the reset that caused this. init and join no longer mkdir the production config dir either. Only setup.sh can rebuild the per-service subtree and its ownership, so recreating the top level would still leave a node that inits cleanly and dies at service start. They now refuse with the remedy instead. |
||
|
Josh Siviour
|
57a93fdee3 |
fix(nats): detect the JetStream write latch the health endpoint cannot see (#645)
The watchdog probed only healthz?js-enabled-only=true, which reports the server-wide JetStream state. A per-message-block write error is recorded on the block and returned on every later write, and nothing clears it short of a restart -- the server-wide state is never touched, so healthz stays 200 while writes fail indefinitely after the disk recovers. spx admin node js-probe round-trips a canary key through the cluster-state bucket, which is the only operation that observes the latch. It probes that bucket rather than one of its own because a dedicated bucket is a separate stream that can be placed elsewhere and stay writable. |
||
|
Josh Siviour
|
dad5d702e1 |
fix: stop lbagent heartbeats on positive LB deletion, and actually enable the JetStream watchdog (#643)
* fix(lbagent): stop heartbeating a load balancer once it is confirmed deleted A LoadBalancerNotFound response is the gateway's positive confirmation that the store has no record for the LB, not a connectivity failure — unlike a dial error, timeout, or 5xx, which leave its existence unknown and must keep retrying. Past the startup registration grace window (or after any prior successful heartbeat), it can no longer be the record's own replication race, so the agent now stops rather than retrying forever and filling the log with heartbeat noise after teardown. * fix(nats): enable the JetStream ENOSPC-latch watchdog timer nats-server disables JetStream process-wide on ENOSPC (server/jetstream.go handleOutOfSpace -> ShutdownJetStream) and never re-enables it itself — that decision is nats-server's own internal state, not spinifex's, so a restart is the only way back. The spinifex-nats-watchdog.timer + service already probe /healthz?js-enabled-only=true and try-restart on a sustained failure, but the timer's WantedBy=timers.target never took effect: neither setup.sh nor firstboot enabled it, so the watchdog shipped inert and a full disk needed a manual restart forever. Enable --now it alongside the rest of the unit set. |
||
|
Josh Siviour
|
59db7d1e52 |
fix: stop calling RemoveSystemImage's deleted bytes "freed" (#638)
BytesFreed counted logical object sizes deleted from the S3 namespace, but predastore only reclaims that disk space later via background compaction. The CLI printed "freed X GiB" immediately on delete, which made a compaction outage that never reclaimed the space invisible. Rename to BytesDeleted and reword the CLI output to say the bytes are marked for deletion and reclaimed by background compaction, not freed immediately. |
||
|
Josh Siviour
|
9f5f877e50 |
fix: batched hygiene — slog key, ui pid file, race-scaled timeouts, GPU console capture (#637)
* fix(ec2): rename ambiguous 'code' slog key to firmwarePath The EFI variable store prep log used 'code' for the OVMF firmware path, which reads as an error/status code in log search. Rename to firmwarePath to avoid confusion. * fix(spinifexui): write pid file to a service-owned BaseDir spinifexui.Start() called the legacy utils.WritePidFile, which has no MkdirAll and falls back to $HOME/spinifex. Under the spinifex-ui unit's ProtectHome=yes this directory is neither guaranteed to exist nor writable. Add Config.BaseDir, switch to utils.WritePidFileTo (which does MkdirAll) for Start/Stop/Status, wire it through the CLI as a spinifex-ui-base-dir flag/SPINIFEX_UI_BASE_DIR env var, provision /var/lib/spinifex/spinifex-ui in setup.sh, and add it to the unit's ReadWritePaths. * test(viperblockd): replace fixed sleep with a readiness poll TestEBSConfigQueueGroup_DetachedOpenFails slept 500ms before assuming launchService's ebs.config subscriber was bound, then raced a hardcoded 5s request deadline. Poll with a cheap malformed request (rejected at json.Unmarshal, before the slow S3 path) until the subscriber answers, then issue the real request with room for openVolumeVB's S3 backoff. * fix(e2e): capture GPU worker console log on test failure Wire harness.OnFailure into TestEKSGPUPodExposure to dump the GPU worker's serial console via the established EC2 GetConsoleOutput path, matching lb_test.go's captureLBConsoleOnFailure. A driver or device-plugin hang after boot only ever surfaces on the guest console. |
||
|
Josh Siviour
|
2983160374 |
fix(lbagent): emit JSON logs so console attributes reach the sink as fields (#617)
lb-agent configured no logger at all, so it fell back to slog's default text handler. That folds attributes into the message, leaving the cause of a failure trapped in a string: 'Heartbeat failed err="gateway returned 400"' arrived greppable but never queryable. No OTLP export: the agent's only egress is the gateway it is watching, so the host-side serial console remains the path that survives a blackout. |
||
|
Josh Siviour
|
eff11b611b |
fix: stop upstream DHCP leases leaking out of the external pool (#602)
* fix(dhcp): release stranded leases and label them with the owning node Two paths dropped a lease from KV without returning it to the upstream server, so a source="dhcp" pool leaked addresses permanently. Manager.Start deleted an expired entry outright. Dropping the entry alone abandons the binding rather than returning it, and a server that ages leases by liveness holds the address for as long as anything answers ARP for it. Adoption now sends a bounded, best-effort RELEASE first, skipping leases with no stored offer/ack because nclient4 cannot rebuild those. Release-by-IP answered SUCCESS when the IP was absent from KV, telling the caller the address was freed when nothing went on the wire. It now returns an error. Every ReleaseIP call site logs and continues, so this surfaces the strand without wedging teardown. Leases were also unattributable upstream: chaddr is a derived HashMAC and option 61 carries that same MAC, so no field identified the host that took the lease. Option 12 is now "<node>-<client-id>" and option 60 is "mulga-spinifex/<node>", giving the lease table a per-lease identity and a per-host column to group on. HashMAC is untouched — encoding the node there would cut per-node entropy and reverse a documented decision. Observed live: 20 office-LAN addresses held by one node's orphaned OVN gateway LRPs, with no lease records and no renewals. * fix: rebind the datapath when a DHCP lease moves to a new address A lease re-issued on a different IP was persisted to KV with no notification of any kind. doAcquire replaced the lease after an expiry and doRenew accepted whatever a rebind answered with, while the gateway LRP's address is written once at attach and never revisited. Each such event burned one address permanently: the old one squatted by OVN's ARP responder with no renewer and no DHCPRELEASE, the new one renewed forever attached to nothing. Route both paths through applyLease, which releases the superseded address and invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg so the VPC SNAT and host ingress route follow, and waits on the flows barrier. EIP, ENI-public and NATGW addresses are recorded outside vpcd in DescribeAddresses and DescribeInstances, so the hook reports those rather than repointing a datapath the control plane no longer agrees with. The release half still applies to every purpose, so the leak itself is closed either way. A failed rebind does not fail the lease loop: the new address is held regardless, and dropping it would strand both. * fix: reconcile API-visible records when their DHCP lease moves An EIP or auto-assigned public IP whose lease was re-issued elsewhere left DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had already released. The datapath kept the old dnat_and_snat too, so the address was dead on both sides while the upstream server was free to hand it to someone else. vpcd cannot fix this alone: it holds the lease, but the records naming those addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and the daemon serves, dispatching to the service owning the moved address. RebindPublicIP and RebindENIPublicIP commit the new external IP through vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP, so one commit is enough — and only then move the record, so a NAT failure never leaves a record advertising an address that does not route. Both refuse to move a record naming neither address, so a stale request cannot overwrite an allocation the lease does not own, and both are idempotent under duplicate delivery. A failed or unanswered reconcile propagates back to the lease loop rather than reading as success. vpc.dhcp.lease-expired, listed in the service manifest but never implemented, is replaced by the subject that now exists. * fix: dedupe retried AllocateAddress so one call takes one address EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can outlast the AWS client read timeout, so every SDK-level retry minted a fresh allocation ID and drew another upstream DHCP lease. The caller kept one EIP and the rest stayed leased with no record naming them, so nothing ever released them. Retries are now identified by the SDK's amz-sdk-invocation-id, which is stable across attempts of one call. The gateway lifts it off the request, NATSRequest forwards it as a header, and the daemon puts it back on the context, so the EIP service sees the same token every attempt. singleflight joins a retry to the call still in flight — the common case — and a TTL'd KV record answers retries that arrive after the first call completed. * fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones Three leaks that all end the same way: an address stays leased upstream with nothing renewing it on purpose and nothing that will ever release it. Manager.Start deleted any lease that expired while vpcd was down, after releasing it. The resource it was taken for still existed, so the datapath kept forwarding on an address the upstream server now considers free — a duplicate-IP conflict waiting for the server to reissue it — and nothing ever re-acquired. Expired leases now get a renewal loop like any other, which re-DORAs on its first pass; the existing IP-change hook rebinds the datapath when the new address differs. The reaffirm RENEW is skipped for these, since a lease the server has already aged out will only fail it. Nothing reconciled leases against the resources that justify them, so a resource deleted out from under its lease left the address renewed forever. ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check whether the EIP, ENI or VPC still exists, and releases the lease when it does not. A gateway lease also has a router port to remove, so DetachIGW runs before the address goes back — releasing it while OVN still answers ARP for it invites the same conflict. Anything the daemon cannot resolve is reported unknown and kept: an unreachable daemon must never read as a fleet-wide deletion. The acquire ladder ran to ~62s, just past the 60s read timeout botocore defaults to, which is what turned one AllocateAddress into two leases. It now totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the outcome instead of giving up and retrying. This is a mitigation, not a fix, for callers that send no retry token: botocore emits no amz-sdk-invocation-id, so the dedupe added earlier cannot see their retries at all. Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj. * feat: add spx get dhcp-leases so held external addresses are visible The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV bucket with no read path, so working out which addresses were held, by what, and whether the owner still existed meant writing a NATS client or reading journals. That was the main obstacle to diagnosing every leak in this series. The command lists each lease with its purpose, VPC, address, MAC, holding node and remaining time, and resolves the owner over vpc.dhcp.owner-check so an orphan is visible as such. --orphans narrows the output to leaks; --skip-owner-check drops the lookup when only the addresses matter. Fixes mulga-655iu. * fix(vpcd): reconcile gateway ports onto their lease address at startup A gateway LRP configured from a static pool keeps its address when the pool is switched to source="dhcp", so the VPC forwards on an address nothing leases while the lease it holds goes unused. No lease event ever fires to correct it. vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the address the lease actually holds. RebindGatewayIP is idempotent, so a port already in agreement costs one OVSDB read. * test: cover dhcp lease owner-check handlers and the lease CLI The owner-check path decides whether the reaper releases an address, so every inconclusive branch is asserted to report unknown rather than gone. * fix(dhcp): make the outcome of a stale lease release observable A stale RELEASE was logged only when the transport failed. A lease with no raw offer/ack was skipped in silence, and a RELEASE the server ignored looked identical to one it honoured, so an address left bound upstream could not be traced to either cause. The skip now warns, and a successful release logs the chaddr it used, which is what a server matches a RELEASE against. * fix(dhcp): take the RELEASE chaddr from the lease being released A server matches a RELEASE to a binding on chaddr, and the release path re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed since, that sends an address the binding never had: the server drops the RELEASE and says nothing, so the address stays held with no renewer. The chaddr now comes from the lease record, which carries what took the lease out, falling back to live resolution only when the record has none. |
||
|
Josh Siviour
|
3e195506d3 |
feat(images): mkosi system-image migration — CPU eks-node/ecs-node + GPU variants (#605)
* fix(eks): GPU node AMI declares uefi boot mode
The spinifex-eks-node-gpu catalog entry declared BootMode: bios, but
the Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under
legacy BIOS no OVMF/pflash firmware is attached, so the guest hangs at
firmware with zero serial console output.
* fix(images): install open-kernel NVIDIA DKMS driver for Blackwell GPUs
Blackwell-generation GPUs (e.g. RTX Pro 6000 Blackwell) refuse to
initialize under the closed/proprietary kernel module: dmesg shows
RmInitAdapter failing with "requires use of the NVIDIA open kernel
modules" (NVRM error 0x22). Switch from nvidia-dkms-<ver>-server to
nvidia-dkms-<ver>-server-open in both the shared ubuntu-gpu-nvidia
recipe and its verbatim duplicate in eks-node-gpu. nvidia-utils has no
separate open variant (userspace tools only), so it is unchanged.
* fix(eks): tolerate CP and GPU taints in ebs-csi-controller/lbc addons
A cluster with a GPU-only nodegroup has no untainted node: the CP
carries CriticalAddonsOnly (EKS parity keeps user workloads off it)
and every worker carries nvidia.com/gpu. Without tolerations for both,
ebs-csi-controller and aws-load-balancer-controller have nowhere to
schedule and stay Pending forever. Real EKS never needs this since its
control plane isn't a schedulable node, so the stock upstream
manifests carry no such toleration.
* fix(images): give the libguestfs appliance a working resolver
The guest's /etc/resolv.conf symlinks to systemd-resolved's stub
(127.0.0.53), which isn't running inside the appliance chroot, so DNS
is dead until package installs need it. Point it at a real resolver
before package installs, then restore the stub symlink afterward so
the shipped AMI still gets its DNS from systemd-resolved at boot like
a stock cloud image.
* fix(gpu): skip upstream PCIe bridge devices during VFIO bind/unbind
A GPU's IOMMU group can include an upstream PCIe bridge when the
platform lacks ACS isolation between them. vfio-pci refuses to bind
non-endpoint bridge devices, so Claim/ReclaimByAddress must filter
bridge-class (0x06) members out of the group before touching driver
bindings, rather than failing the whole claim or leaving the bridge
rebound incorrectly.
* fix(ecs): GPU node AMI declares uefi boot mode
The spinifex-ecs-node-gpu catalog entry declared BootMode: bios, but the
Ubuntu 26.04 GPU appliance image is built GPT/EFI-only. Under legacy BIOS
no OVMF/pflash firmware is attached, so the guest hangs at firmware with
zero serial console output.
Same defect and same fix as the eks-node-gpu entry: both images are built
by the same script from the same stock Ubuntu cloudimg, so both were
EFI-only and only one was corrected.
* fix(images): open-kernel NVIDIA module on ecs-agent-gpu, pin driver branch
ecs-agent-gpu still installed nvidia-dkms-<N>-server, the proprietary
kernel module. NVIDIA ships no proprietary module for Blackwell (GB202)
and later, so ECS GPU container-instances could not bind those cards on
any driver branch. It is the third verbatim copy of the routine and was
missed when the other two were fixed.
Also pins the driver branch across all three copies. The branch was
resolved at build time via apt-cache search | sort -rn | head -1, which
sorts on the number in the package name — but that number is a branch
alias, not a driver version (nvidia-dkms-535-server installs 580.126.20),
so the ordering key need not match what installs. Resolving also let
identical source produce different drivers on different days.
The pin lives in setup.sh rather than manifest.conf because manifest
variables are not passed into the guest: the setup script is uploaded and
executed by virt-customize --run, with no environment carried across.
* fix(eks): tolerate CP and GPU taints in argocd addon
All six argocd Deployments and its StatefulSet carried no tolerations. A
cluster with a GPU-only nodegroup has no untainted node to place them on:
the control plane carries CriticalAddonsOnly and every worker carries
nvidia.com/gpu, so the pods stay Pending forever.
Same defect the ebs-csi-controller and load-balancer-controller addons
already had. Real EKS never hits it because its control plane is not a
schedulable node, so the stock upstream manifests carry no such
toleration and the divergence is ours to maintain.
* refactor(images): extract eks-node initd logic into init-agnostic helpers
The OpenRC runscripts carried their start_pre logic inline, so porting a
service to systemd meant re-deriving that logic under a distro change
rather than just rewriting the init wiring. Move it into /bin/sh helpers
that either init system can drive, keeping eks-node on Alpine/OpenRC.
k3s-prestart takes the restore-block guard, config.yaml derivation,
konnectivity-agent and CoreDNS staging, and the token-webhook wait.
konnectivity-server-prestart takes the CA wait and cert mint, plus the
server-count derivation that used to run at parse time. Both keep every
guard, bound and message as-is.
Values flow back the way the GPU image already does it: the helper writes
shell assignments to a /run file and the init system sources them, so
command_args is built in start_pre where it used to be a top-level
assignment.
k3s-agent-prestart moves to eks-node and becomes the single copy. The GPU
manifest claimed its helpers were reused verbatim from eks-node while
hand-copying this one; it now points at the shared file, so the two
cannot drift.
k3s-agent.initd still sources agent.env and exports itself rather than
delegating: OpenRC has no EnvironmentFile= equivalent, and the daemon
inherits only what the runscript exports, so a helper subprocess sourcing
it would not reach k3s.
The helpers run under set -eu, which the runscripts did not. The awk over
first-boot.env is guarded on the konnectivity manifest, not on the env
file, so a missing env file exits awk nonzero; without || true that turned
a warn-and-continue into a refusal to start k3s. Covered by a test that
fails on rc=2 if the guard is dropped.
* feat(images): build system images in a pinned mkosi container
Move the image-build toolchain into a container so a runner needs only
Docker. The toolchain cannot drift between runners, and the same build
runs identically on a developer's machine.
mkosi builds in a user namespace, which two of Docker's default
confinements block: the seccomp profile masks CLONE_NEWUSER, and the
docker-default AppArmor profile separately blocks unsharing the mount
namespace. Both are relaxed; neither is a privilege grant, as
capabilities stay dropped and no devices are exposed. mkosi also runs as
an ordinary user rather than root, since as root it takes a path that
needs real CAP_SYS_ADMIN.
Verified end to end: an Ubuntu 26.04 Format=disk image builds from the
Debian container and the artefact is a valid GPT disk, with the package
cache on a named volume cutting a forced rebuild from ~2min to ~34s.
* feat(images): publish system images from the build container
Convert mkosi's raw output to a compressed qcow2, write the .sha256
sidecar the spx catalog verifier matches, and upload both to R2 using the
same pinned toolchain as the build, so publishing needs nothing on the
host either. The toolchain image build moves to a shared helper rather
than being duplicated across the two entry points.
Publishing needs no relaxed confinement, unlike the build: qemu-img and
aws create no namespaces, so it keeps Docker's default seccomp and
AppArmor profiles.
The object path carries no version and the catalog resolves newest-wins,
so an upload overwrites the live image with no way back. Copy the current
object to a timestamped archive key first, making a bad publish a restore
rather than an unrecoverable loss.
* fix(images): build the image on the output's mount, not the cache volume
mkosi assembles the image in its workspace and moves the result to the
output directory. The workspace lived under the cache volume, which is a
different mount from the output bind mount even when both sit on one
filesystem, so the move failed EXDEV and mkosi fell back to copying —
requiring the finished image to exist twice at once.
That is invisible on a small image and fatal on a real one: a ~4G GPU
image needed ~8G to land and died with ENOSPC after a successful build.
Point the workspace at the output's own mount so the move is a rename
again and the image only exists once. The package cache stays on the
volume, which is the part worth persisting across runs.
* feat(images): add mkosi base config and gpu-nvidia profile
Establish images/ with a base mkosi.conf plus a gpu-nvidia profile,
replacing the throwaway smoke-test config. The base builds a bootable
UEFI disk; the profile adds the NVIDIA stack.
Most of ubuntu-gpu-nvidia-setup.sh does not carry over. Its kernel-version
hunt, the do_bootloader=no kernel-img.conf hack and the manual
update-initramfs all exist only because that pipeline chroots into a
prebuilt cloud image that already has a kernel. mkosi installs the kernel
and owns the bootloader, so the version is known rather than detected, and
the postinst asserts exactly one kernel instead of searching for it. The
static configuration (nouveau blacklist, unattended-upgrades blacklist,
cloud-init docker group) becomes files under mkosi.extra/ rather than
heredocs.
linux-image-generic drags in every linux-firmware split package, none of
it relevant to a headless VM guest; left at the default the modules initrd
embeds all of it into the UKI, which grew to ~880MB and overflowed the
512M ESP. The initrd module set is trimmed to mkosi-initrd's curated
default, which is a boot fix and not only a size one.
Verified in the built artefact, not just the build log: nvidia.ko.zst and
the four sibling modules under /usr/lib/modules/<kver>/updates/dkms, the
nouveau blacklist, and the nvidia runtime in docker's daemon.json.
* fix(images): unblock guest boot on wait-online and firstboot
A guest built from these images stalled twice on the way to a login prompt,
both times on a unit waiting for input that never arrives in a VM.
systemd-networkd-wait-online holds network-online.target until every managed
link is up, and docker.service pulls that target in with Wants=, so any image
carrying docker inherits the wait whether or not it wants it. A guest whose
links are slow or absent ran the unit to its full 120s default with
multi-user.target queued behind it. Override it to --any with a short timeout:
one link up is what a single-NIC guest actually needs.
systemd-firstboot separately prompts on the console for anything the image
does not already carry, before multi-user.target. With no timezone baked in,
boot stopped dead at "Please enter the new timezone name" and waited for a
keystroke. Setting locale, keymap and timezone at build time leaves firstboot
with nothing to ask.
Both are baked into the base rather than delivered as first-boot drop-ins:
an image built from scratch has no cloud-init to write them.
Measured on wattle with an A1000 passed through: login at 42.8s, down from
~210s, and the firstboot prompt is gone. nvidia-smi in the guest reports
"NVIDIA RTX A1000, 595.71.05, 8188 MiB".
Also widen the images gitignore to output-*/ so a scratch build cannot leave
multi-GB artefacts stageable.
* feat(images): make the base image reachable — cloud-init, sshd, console
An image built from scratch launched unreachable. The images this replaces
start from a stock Ubuntu cloud image and inherit cloud-init, sshd, sudo, a
network renderer, dbus and a serial console from it; mkosi installs only what
is listed, so none of it was there and nothing said so.
Guests get their SSH key and user-data from IMDS via cloud-init, so without
the package a launched instance has no authorized_keys at all — no error, just
no way in. It also made any cloud-init-delivered config inert, including the
gpu profile's docker-group drop-in.
Each addition is load-bearing on its own:
cloud-init consumes IMDS: keys, user-data, hostname, network
openssh-server the key needs a sshd to be installed into
netplan.io cloud-init's renderer on Ubuntu; with no renderer the NIC is
never configured, IMDS is never reached, and cloud-init
falls back to DataSourceNone
dbus systemd only Recommends it and mkosi installs no Recommends,
so systemctl could not reach the system bus — and the EKS and
ECS bootstraps drive guests with systemctl from user-data
lsb-release cloud-init shells out to it; without it every boot reports
"degraded" forever and the status stops being a usable signal
The kernel also needs console=ttyS0 baked in. The launcher gives every guest a
serial port and get-console-output reads it, but a kernel with no console=
writes to none of it: the console stopped after the UEFI bootloader and every
boot failure was invisible. ttyS0 goes last because Linux makes the final
console= the controlling /dev/console.
manage_etc_hosts is set because cloud-init renames the guest per launch but at
its default never writes a matching hosts entry, so sudo warned on every call.
datasource_list is deliberately NOT pinned: the guest identifies the platform
from the SMBIOS system UUID the launcher spoofs, ds-identify narrows to Ec2 on
its own, and pinning per image is the customisation this migration deletes.
Verified on wattle with an A1000 passed through: ssh in as ubuntu with
spinifex-key, cloud-id "aws", cloud-init status done (not degraded), no failed
units, nvidia-smi reports the A1000 on 595.71.05, docker active, boot 26.7s.
* feat(images): let the guest grow its root into the volume it was given
mkosi sizes the disk to its content, so the image ships with a root
partition that fills the disk to its last sector and ~1.4G free. Nothing
in it can expand that: cloud-init's growpart module is enabled, but the
growpart binary lives in cloud-guest-utils, which is not installed and is
not pulled in as a Recommends by anything in the package set. systemd-repart
is not an alternative — Ubuntu's systemd package carries no repart binary,
unit or definitions at all.
The result was a node handed a larger volume that it could only ever use
the first few GiB of, with the rest allocated and unreachable. That is
fatal for a worker whose job is to pull container images.
gdisk is deliberately left out despite being a cloud-init Recommends:
growpart resizes GPT through sfdisk, which arrives via the hard fdisk
dependency, and e2fsprogs brings the resize2fs that the resizefs module
runs afterwards.
Verified on wattle: the gpu-nvidia image launched with a 20G root reports
20G on df -h / (was 4.4G), via
"'/' resized: changed (/dev/vda2) from 4770582528 to 20936900096", with
nvidia-smi still reporting the A1000 and no failed units.
* feat(images): move eks-node-gpu and ecs-agent-gpu onto mkosi profiles
Adds the eks-common, eks-agent and ecs profiles, so both GPU node images
build from the composed base rather than a hand-authored tree, and the three
verbatim copies of install_nvidia_gpu_stack() collapse into gpu-nvidia alone.
Splits Docker out of gpu-nvidia into its own profile. Docker belongs to the
standalone GPU AMI, whose purpose is `docker run --gpus all`; the EKS and ECS
node images run k3s or containerd and reach the GPU through CDI. Fusing the
two was fatal for ECS rather than merely wasteful: gpu-nvidia installed
containerd.io from Docker's repository, which Conflicts/Replaces the Ubuntu
containerd the ecs profile needs, so gpu-nvidia + ecs could not resolve at
all. nvidia-container-toolkit stays in gpu-nvidia — every GPU image needs
nvidia-ctk to generate its CDI spec.
Closes three gaps the pilot left, each of which only shows up on a booted
guest: /etc/modules-load.d/nvidia.conf, without which nothing loads the
modules on a headless node and CDI generation finds no GPU;
mulga-cdi-generate.service, which no profile shipped; and an apt-mark hold on
the toolkit packages, which the unattended-upgrades blacklist does not cover
for a plain `apt upgrade`.
mkosi-build.sh gains --image, naming an output and expanding it to an ordered
profile list. The order is load-bearing: mkosi runs each profile's postinst in
the order the profiles are passed, so docker's nvidia-ctk wiring must follow
gpu-nvidia. Leaving that to callers meant an argument-order mistake produced a
healthy-looking image with no GPU wiring, so docker's postinst also fails
loudly when gpu-nvidia is selected but nvidia-ctk is absent.
Profiles reference build inputs where they already live via ExtraTrees=
instead of copying them, keeping one copy of the helpers shared with the
Alpine image. Go binaries are built on the host into a staging tree, since the
builder container carries no Go on purpose: go.mod is the single pin for the
toolchain and a second pin in the Dockerfile could disagree with it silently.
* fix(images): install systemd-resolved so the guest has a DNS resolver
mkosi installs only the packages named in Packages= plus their hard dependency
closure, and systemd-resolved is merely a Recommends of systemd. So it was never
installed: the image shipped a 0-byte /etc/resolv.conf that nothing populated,
because netplan renders to systemd-networkd and networkd alone does not write
resolv.conf. glibc then fell back to localhost and every lookup failed with
"[::1]:53 connection refused", which reads as a network fault rather than a
missing package.
A node that cannot resolve cannot pull a container image. The ECS GPU node
booted, registered as a container instance, then failed every task at image
pull. The EKS image carries the same defect but is masked by the cloud-init
bootcmd in nodegroup_userdata.go that writes resolv.conf at launch; the ECS
launch path has no equivalent, so only ECS exposed it.
* fix(images): name build output by image and never report a skipped build as success
Two defects in the build wrapper that let the wrong bytes reach a boot test.
Every composition wrote the same spinifex.raw, taken from the base config's
ImageId, so an artefact carried no record of which profiles produced it and an
EKS image was distinguishable from an ECS one only by which build ran last.
Passing --image-id names the output after the image, and as a side effect the
name lands in the guest's /usr/lib/os-release. Publishing correspondingly loses
its default input: --name is the published object and --image the local
artefact, and with nothing tying the two together a defaulted input meant
whatever raw happened to be in the output directory could be uploaded under the
name the caller asked for.
mkosi also skips the build whenever the output path merely exists. It does not
compare inputs, so it never notices an edited mkosi.conf or a restaged binary,
prints "Use --force to rebuild" and exits 0 — a stale image indistinguishable
from a fresh one. Builds are now always forced, and the run additionally asserts
the expected raw was written during this invocation rather than trusting the
exit status. The cache worth keeping is the package download volume, which
--force does not touch.
* feat: add gpu-amd mkosi profile for the ROCm AMI
Ports scripts/images/ubuntu-gpu-amd-setup.sh so ubuntu-gpu-amd builds under
mkosi as 'gpu-amd docker'. Without it the legacy scripts/images/ tree has to
stay alive for one remaining image.
Pins the ROCm repo by apt priority rather than only adding it: Ubuntu 26.04
universe ships its own vendored rocminfo, which unpinned apt preferred over
AMD's, mixing release trains within one image.
Uses a systemd unit for render/video group membership instead of a cloud-init
runcmd, because cloud-init's config merge takes a duplicated runcmd key
wholesale from one fragment and silently discards the other, which would have
dropped the docker profile's own usermod.
Runtime GPU verification is deferred — no AMD hardware is available.
* feat(images): add eks-server, ec2-node and CPU eks-node/ecs-node mkosi images
Ports the K3s server side of EKS off the legacy Alpine/OpenRC builder: nine
systemd units, a templated etcd-snapshot service with daily and 15-minute
timers, the addon/coredns/konnectivity assets, and the six server binaries.
konnectivity-server is not an in-repo package, so it gets a pinned host-side
clone-and-build rather than an entry in profile_binaries().
Splits ec2-node out of eks-common. mulga-ebs-byid mints the by-id links for any
attached volume and mulga-mgmt-net configures any multi-NIC system VM, so
neither belongs behind an EKS profile that a database image would have to
compose to reach them.
Start-at-boot moves into preset files. mkosi runs preset-all after every
postinst, so a postinst enable is discarded before the image is finished, and
with no policy file the fallthrough is enable — which built an image that ran a
control plane and a worker at once.
Installs the cloud-init datasource pin as 99- rather than 90-. cloud-init-base
ships its own 90_dpkg.cfg declaring datasource_list, read_conf_d sorts the
directory in reverse and the merge keeps the first source, so a 90- prefixed
file lost the key silently. The postinst now asserts the merged result.
* refactor(eks-node): make the role selector init-agnostic
eks-node-role and k3s-first-boot drove OpenRC directly, so on the systemd image
the selector enabled nothing and the node never started k3s. Both now dispatch
through svc_enable/svc_start/svc_disable on a detected init, with EKS_NODE_INIT
overriding detection so either branch is testable on a host that has only one.
Detection prefers rc-update over systemctl: the test suite stubs rc-update on
PATH, and preferring systemctl would have silently rerouted those cases to a
path they were not written for.
Enable and start stay two separate passes. The server path starts
mulga-eks-k3s-recovery before k3s so a recovery directive applies before the
apiserver comes up, and collapsing to systemctl enable --now would interleave
them.
The dispatch is duplicated in both scripts rather than shared: they are
single-file deliverables with no library-path convention between them.
Tests cover both branches across all three roles, including that server-join
still gets no k3s-first-boot, and that rc-update wins when both inits are
present.
* fix(eks-node): enqueue role-selected units instead of blocking on them
eks-node-role.service declares Before=k3s.service, and
mulga-eks-k3s-recovery.service declares After=eks-node-role.service, so a
blocking systemctl start deadlocks: systemd will not run the queued job while
the selector is still activating, and the selector cannot finish until that job
runs. Type=oneshot defaults TimeoutStartSec=infinity, so nothing breaks the tie
and the node wedges mid-boot with the role written but no control plane. On a
booted guest the selector sat in activating for 261s until the VM was torn down.
Start with --no-block on systemd. The ordering the blocking call was implicitly
providing is already declared between the units themselves, so systemd
sequences them correctly on its own. OpenRC runs the script directly rather
than scheduling it, so that branch keeps blocking and keeps surfacing failures
through set -e.
* fix: escape $ in konnectivity-server ExecStart so server-count survives
systemd expands $VAR/${VAR} in Exec*= from the unit's own environment before
the command runs, substituting an empty string for a name it does not know.
KONN_SERVER_COUNT is not a unit variable — konnectivity-server-prestart writes
it to /run/konnectivity-server-args.env, which the wrapper shell sources at
runtime — so systemd rewrote the flag to `--server-count=` before /bin/sh saw
it. konnectivity-server then died on flag parsing, which is why the unit logged
Started and produced nothing despite --logtostderr=true and journal+console,
and why the apiserver's cluster egress selector found no UDS for the life of
the cluster.
This is a port artefact. konnectivity-server.initd is flag-for-flag identical
and shares the same prestart script, but under OpenRC the shell was the only
thing reading ${KONN_SERVER_COUNT}, so the same text was correct.
Escape it as $$ to defer expansion to the wrapper shell, and add
lint_unit_expansions() to reject any unescaped $VAR/${VAR} in an Exec*= line of
a profile's units. The linter joins line continuations first, since the
offending reference sat mid-way through a wrapped multi-line ExecStart, and
runs on the host before the container starts so a bad unit fails the build in
seconds rather than at first boot. $(cmd) is deliberately not flagged: systemd
leaves it literal.
Verified on env12 with a rebuilt image: TestEKS/IRSAPod and
TestEKS/EBSCSIVolume, which both previously failed on `kubectl exec ... timed
out` through the egress tunnel, now pass, with zero UDS connect errors across
the run.
* fix: refresh the stale qcow2 mkosi-build leaves beside a fresh raw
mkosi only emits the raw. A qcow2 next to it came from a previous run's hand
conversion, and nothing about its name says which build it belongs to, so it
survives a rebuild and silently ships the old image to a node while the raw
beside it carries the new one. Observed on a rebuild that left the raw at 15:34
carrying a fix and the qcow2 at 12:45 carrying the pre-fix unit.
This is the same failure the existing freshness assertion guards against, one
file over, so extend that block rather than add a second check. Refresh rather
than delete: the qcow2 is what gets copied to a node and imported, so removing
it would push the manual conversion, and this trap, back onto the caller.
Convert via a temp name and rename so an interrupted run leaves the old file
rather than a half-written one that looks current.
* fix: resolve the awsvpc task DHCP client instead of hardcoding udhcpc
ecs-agent leases a task ENI's OVN-assigned address inside a per-task netns,
which systemd-networkd does not manage, so it shells out to a DHCP client. That
call was hardcoded to busybox udhcpc, which Alpine ships and Ubuntu does not.
`ip netns exec` exits 1 rather than 127 when it cannot exec the binary, so the
failure surfaced as an opaque "network setup failed" with the task stuck in
STOPPED, and it took every awsvpc-mode workload with it: TaskAwsvpc and
ServiceWithELB both failed on the mkosi image, the latter as runningCount=0
because its target group targets task ENI IPs.
Resolve the client at setup time, preferring udhcpc so Alpine behaviour is
unchanged and falling back to dhcpcd. Presence of the udhcpc binary is not
sufficient to select it: Debian and Ubuntu's busybox ship the applet without
/usr/share/udhcpc/default.script, and udhcpc then takes the lease and exits
nonzero having configured nothing. Same order and same reasoning as
dhcp_oneshot() in mulga-mgmt-net.sh. Both invocations stay one-shot, leaving no
daemon behind in the netns.
Install dhcpcd-base in base, because neither client was present in any profile.
That also repairs mulga-mgmt-net.sh, whose own udhcpc/dhcpcd fallback resolved
to nothing on Ubuntu and would have failed the first time a cross-host
control-plane data NIC needed it — the S5 path the step-6 plan notes has never
been exercised.
The DHCP-client choice is injected so the tests assert a named platform's
branch rather than whatever happens to be installed on the machine running
them; without that the same test exercises different code on an Alpine CI box
and an Ubuntu dev box.
* refactor: retire the ec2-node profile, fold its helpers into the base image
ec2-node carried mulga-ebs-byid and mulga-mgmt-net, and every EKS composition
had to remember to list it — eks-common even guarded on its presence and
aborted the build without it. ECS did not list it, so the ECS image shipped
with no by-id symlinks for hot-attached volumes at all.
Neither helper is answerable at build time: any image may later be attached a
volume, and any image may later be launched as a control plane. Both are inert
when unused (mulga-ebs-byid is udev-driven; mulga-mgmt-net exits 0 when its
fw_cfg blob is absent), so the base image is where they belong. That deletes
the composition rule, the guard, and the ECS gap in one move.
Also drops ec2-node from the two eks image compositions in mkosi-build.sh.
Verified on freshly built spinifex-ecs-node and spinifex-eks-node raws: both
carry the helpers and the udev rule, mulga-ebs-byid is enabled via
multi-user.target.wants and mulga-mgmt-net via cloud-init.target.wants.
* fix(eks): force UseMTU=false on netplan units so DHCP renewal can't reinflate the pinned link
mulga-vpc-mtu only won the boot race by ordering: netplan renders
UseMTU=true (it does not inherit networkd's own false default), so a
DHCP lease renewal at T1 could silently re-apply the advertised MTU to
the link. usemtu_off() forces UseMTU=false via a runtime drop-in in
the systemd-documented [DHCPv4] section for every rendered unit and
reloads networkd, removing the dependency on boot ordering entirely.
assert_pinned() then verifies the link and its routes actually match
before exiting, so any regression is a loud boot-time failure instead
of a silent one. Both are no-ops on the Alpine/dhcpcd image, which has
no systemd-networkd.
* fix(images): state UseMTU=false under both the current section and the legacy alias
netplan renders the DHCP settings under [DHCP] while systemd documents
[DHCPv4]. A drop-in naming only one parses cleanly and does nothing if the
running systemd disagrees, which is the silent no-op the drop-in exists to
prevent.
* fix(ecs): install iptables on both ecs-node images, cover task-role credential fetch
credendpoint.go DNATs the ECS task credential endpoint via iptables; neither
the mkosi ecs profile nor the legacy Alpine manifest installed it, so the
endpoint silently failed to start and any task with a task role could not
fetch credentials while the instance still looked healthy.
Also fixes a latent bug in the e2e task-role trust policy (Service-principal-only,
which this platform's HTTPS AssumeRole path can never match) and adds a
TaskRoleCredentials subtest that fetches real credentials from inside a task
container, so a dead endpoint fails the suite instead of passing unnoticed.
* fix(images): ship preset files for base, eks-common and ecs start-at-boot
mkosi runs systemctl preset-all after every profile's postinst, discarding
whatever systemctl enable a postinst performed. eks-server and eks-agent
already carry preset files for this reason; base, eks-common and ecs still
relied on the default-enable fallthrough, which happens to match today but
is one upstream package change from reversing.
Adds 10-mulga-base.preset (mulga-ebs-byid, mulga-mgmt-net),
30-mulga-eks-common.preset (mulga-eks-provider-id, mulga-vpc-mtu) and
40-mulga-ecs.preset (containerd, ecs-ca-install, ecs-agent; also disables
cni-dhcp.service/socket, which containernetworking-plugins enables by
default with nothing to serve under the baked host-local IPAM conflist).
Removes the now-redundant systemctl enable calls from each postinst,
matching the eks-server/eks-agent precedent.
* fix(sts): attribute ecs-tasks.amazonaws.com to container-instance callers
The HTTPS AssumeRole path passed an empty principalSource, so matchServicePrincipal
denied every Service clause other than the IMDS-only EC2 one. A role carrying the
trust policy AWS documents for an ECS task role was unassumable by construction.
Service attribution is now derived from the authenticated SigV4 caller: an
assumed-role session of the account's ecsInstanceRole — the only credential the
in-guest agent holds — is attributed ecs-tasks.amazonaws.com, and only for roles
in its own account. Nothing in the request body contributes, so a task holding
its own task-role session cannot claim the service principal.
* fix(images): remove baked SSH host keys, pin Seed= for reproducibility
openssh-server's postinst bakes an identical host key pair into every
build, making every guest from one build mutually MITM-able; sshd-keygen
already regenerates keys on first boot (ConditionFirstBoot=yes), so the
baked pair is pure non-determinism with no purpose. Also pin Seed= so
systemd-repart stops assigning fresh partition UUIDs on every build.
Verified on a built spinifex-ecs-node image: no ssh_host_* files ship,
cloud-init still lists ssh in cloud_init_modules, and sshd-keygen.service
is wired via ssh.service.wants/ (redundant regeneration path). A two-build
comparison dropped content-differing files from 9 to 1 (only the UKI,
whose zstd internals remain sensitive to unrelated input changes).
---------
Co-authored-by: tomnewton-mulga <tom.newton@mulgadc.com>
|
||
|
Josh Siviour
|
2db1034436 |
feat(acm): managed certificate issuance and renewal from a tenant Private CA (#606)
* fix: fan out certificate re-import to referencing load balancers Re-importing new material under an existing ACM ARN previously updated only the stored CertRecord; no load balancer referencing that ARN ever re-rendered its config, so HAProxy kept serving the old leaf until it expired while the API reported success. Add an InUseBy reverse index on CertRecord, maintained via snapshot-diff around every listener create/modify/delete and cert attach/detach in handlers/elbv2. On re-import, ACMServiceImpl invokes a fan-out hook (wired in daemon.go to avoid an import cycle) that re-runs updateStoredConfig for every load balancer in the ARN's InUseBy set, mirroring the existing target-group fan-out. lbagent is unchanged: it already polls ConfigHash and reloads on change. * fix: make InUseBy index updates CAS-safe under concurrent writers AddInUseBy/RemoveInUseBy previously did a plain get/mutate/put, which loses concurrent writers silently: two listeners on different load balancers attaching the same certificate at once could drop one entry with no error. Because InUseBy is the sole mechanism that fans a renewed certificate out to the data plane, a lost entry means that load balancer never re-renders and its certificate silently expires in HAProxy while ACM reports it as healthy. Switch both to a JetStream KV revision-based compare-and-swap loop with jittered backoff, retrying on a concurrent writer instead of overwriting it. Replace the old concurrency test, which only asserted no errors/panics and so could not detect a lost index entry, with one that asserts exact final state across N concurrent writers. * fix: encrypt ACM certificate private keys at rest CertRecord.PrivateKey was persisted to the spinifex-acm JetStream KV bucket as plaintext PEM. Encrypt it with AES-256-GCM via the existing handlers_iam.EncryptSecret/DecryptSecret helpers, keyed by master.key. Store.PutCert encrypts when a master key is set; GetCert/ListCerts decrypt, falling back to a narrowly-bounded legacy-plaintext read (only when the stored value is unambiguously a PEM private key block) so pre-existing records keep working and are re-encrypted on next write. A Store never given a key (ELBv2's independent read-only acmStore) behaves exactly as before — encryption is opt-in per Store instance via the new SetMasterKey method, so no other package needs to change. NewACMServiceImplWithNATS now requires a master key and fails construction without one; daemon.go loads master.key inside the initServiceWithRetry closure so a missing key fails daemon startup after the retry window instead of leaving the service permanently degraded. * fix: wire ELBv2's ACM store with the deployment master key ELBv2 constructed its own handlers_acm.Store without a master key, so once ACM started encrypting CertRecord.PrivateKey at rest, ELBv2 could no longer resolve listener certificates: it read back ciphertext where it expected PEM. Require NewStore(ctx, nc, masterKey) to always take a key so an unkeyed Store can no longer be constructed, thread the deployment master key through NewELBv2ServiceImplWithNATS and the daemon's ELBv2 init closure (mirroring the ACM closure), and make ACM store construction failure fatal instead of silently degrading every HTTPS listener. * feat(acm): tenant Private CA with x509 name constraints Adds an independent tenant root CA (separate from the platform CA) that signs leaf certificates only within its baked-in permitted domains. Name constraints (PermittedDNSDomains, excluded IP ranges, excluded email/URI) are frozen at creation and drift is rejected rather than silently regenerated, so a device's trust store is never invalidated without an explicit operator action. * docs(acm): record OpenSSL verification of the empty email/URI exclusions Confirms the zero-length ExcludedEmailAddresses/ExcludedURIDomains entries parse cleanly and enforce as expected under OpenSSL 3.5.6, not just Go's own verifier, since this CA is meant for device trust stores. * feat: implement RequestCertificate with PRIVATE_CA managed issuance Adds the ACM RequestCertificate API surface: mode derivation (PROVIDER_API/MANUAL_TXT/PRIVATE_CA), synchronous PRIVATE_CA issuance via an injected CertAuthority, and correctly-shaped PENDING_VALIDATION records for the deferred modes. Extends CertRecord with issuance/validation/renewal state, wires the gateway and daemon NATS dispatch, and updates COMMANDS.md. * feat: wire tenant CA into the running spinifex daemon Add LoadTenantCA as a load-only entry point (never creates), refactor LoadOrCreateTenantCA to share the load path. Add 'spx admin cert create-tenant-ca' to create the tenant CA with explicit permitted domains, idempotent unless --regenerate is passed. Wire tenant CA loading into daemon startup: missing tenant CA logs a warning and degrades PRIVATE_CA issuance only, it does not block startup like the master key does. Fold tenant CA paths into admin's CA path helpers, keeping filenames distinct from the platform CA (tenant-ca.pem/.key vs ca.pem/.key) so the two roots can never be confused. Also give ACM's DeleteCertificate in-use refusal its own awserrors alias instead of reusing the EKS-labelled one directly, and correct COMMANDS.md to reflect that the tagging endpoints are implemented. * fix: derive ACM validation mode from the requested domain, not a global flag Mode derivation tested cfg.Northstar.DefaultDomain != "", which says only that northstar is configured, never whether it is authoritative for the domain being requested. Any deployment with a default domain set therefore returned MANUAL_TXT for every request, including domains hosted nowhere, with a TXT record the caller could not publish. Resolve the requested domain against northstar's actual hosted zones, and put MANUAL_TXT last. A name-constrained tenant CA is a definite local capability the operator declared explicitly, whereas MANUAL_TXT assumes the domain is publicly delegated so a public CA can resolve the challenge - which is never true of an internal northstar zone. A tenant CA covering every requested domain now wins, and a configured DNS provider still takes precedence over the manual path so operators with a real domain are not downgraded to a private certificate. * feat: renew PRIVATE_CA certificates automatically before they expire Managed issuance signed a leaf once and never renewed it. With 90-day leaves every certificate silently expired a quarter after the first apply, which is the treadmill managed issuance exists to remove. Renewal is scoped to PRIVATE_CA: re-signing from the tenant root is fully offline, so this needs no ACME client and no network access. The window is proportional - two thirds of the certificate's own lifetime, floored at a minimum - so it self-tunes as issuance lifetimes change rather than degenerating into 'renew immediately, forever' the way a fixed window would. A deterministic per-ARN jitter keeps certificates issued in one apply from all renewing in the same minute. Reissue keeps the certificate ARN and generates a fresh leaf key, then pushes through the ELBv2 fan-out. The push is mandatory rather than incidental: GetLBConfig returns stored configuration and never re-resolves certificate material, so a renewal that did not push would leave HAProxy serving the old leaf until something unrelated changed the config hash. A per-certificate lease keyed on the ARN means one node drives a given renewal without serialising all renewals behind a single leader. Certificates whose domains have fallen outside the tenant CA's name constraints record a failure reason instead of retrying in a tight loop. The renewal scan reads metadata only and never decrypts stored private keys; the lease is a raw compare-and-swap that leaves the encrypted key untouched. * fix: carry handler error messages over the daemon NATS transport handleNATSRequest responded with the sanitized error code alone, so any handler error without a registered code collapsed to a bare ServerInternal and its reason was visible only in the daemon log. utils.ServeNATSRequestCtx has always preserved the message; the two implementations had drifted. Adds respondWithServiceError and routes the 16 service-error sites through it, plus transport-level tests pinning that the message survives for an uncoded error, that a wrapped coded error keeps its real code, and that a bare code carries no message. * feat: implement ACM GetCertificate and stop decrypting keys on read paths GetCertificate was unimplemented end to end: it now returns the PEM certificate body and chain for an owned ARN, ResourceNotFound for an unknown or cross-account ARN, and RequestInProgressException for a certificate that has a record but no body yet, wired through the service interface, the NATS client, the daemon subscription and the gateway action map. Store gains GetCertMetadata alongside GetCert, and ListCerts becomes ListCertMetadata. Every read that answers a question about a certificate rather than serving it - ownership checks, Describe, Delete, tag reads, GetCertificate and listing - now leaves PrivateKey empty instead of running an AES-GCM open per record to produce plaintext key material no caller reads. * fix: reconcile the ACM master-key wiring with dev's new tests The InUseBy and cert fan-out tests that landed on dev construct the ACM and ELBv2 services directly, and both constructors now take the deployment master key. Pass one shared key in the fan-out test - two services over the same bucket disagreeing on the key is the exact failure the wiring exists to stop. A Store is never unkeyed now, so the two test helpers collapse into one. * fix: rebuild the ACM InUseBy index at daemon startup The index that drives certificate fan-out is written only by listener mutations, so a listener whose certificates have not been touched since it was created has no entry. A certificate with an empty index then neither fans out on renewal nor trips DeleteCertificate's in-use guard, and both failures are silent: the renewal reports success while the load balancer keeps serving the old leaf, and a certificate a live listener depends on can be deleted. Reconcile the index from the persisted listener records once at startup. Walking every certificate rather than only the referenced ones makes it a rebuild rather than a backfill, so it also drops entries naming a load balancer that no longer exists. Writing only differences keeps it idempotent, and the index writes are already CAS-guarded, so the nodes of a cluster reconciling the same bucket converge. * fix: deliver listener SNI certificates to HAProxy Add/RemoveListenerCertificates persisted the listener record and then returned, so the config was never re-rendered and lbagent had nothing to poll for. Re-render through reloadListenerLB, the same helper the rule mutations use. That alone changed nothing, because the renderer dropped SNI certificates: resolveFrontendCert returned only the default certificate and the bind line emitted a single crt, with the PEM path keyed on the listener so a second certificate had nowhere to live. Render every attached certificate instead, default first so it stays the no-SNI fallback, each with its own path. HAProxy accepts crt more than once on a bind line and selects by SNI. The default certificate keeps its existing filename so an upgrade does not move a deployed PEM, and the nginx renderer keeps the single-certificate resolver it still relies on. * fix: prune stale TLS certs from the lb-agent cert dir A delivered PEM is cert + chain + private key. writeCertFiles only ever wrote, so every certificate detached from a listener left its key on the load balancer VM's disk indefinitely. Prune after a successful reload, scoped to .pem files in the cert dir's immediate entries. * fix: serve every attached certificate on the nginx NLB engine nginx allows one ssl_certificate per key type in a server block, and the NLB config is a stream block where server_name does not exist. Select by SNI instead: give ssl_certificate a variable and map $ssl_server_name to each certificate's path, keyed on the leaf's DNS SANs. A single-certificate listener keeps a literal path, since a variable costs a per-handshake read. The cert-path helpers were HAProxy-only by hard-coding lbagent.CertDir, so they now take the dir and both engines share one multi-cert resolver. * fix: emit ValidationMethod in DomainValidationOptions DescribeCertificate returned each domain validation entry without a ValidationMethod, so the AWS provider had nothing to compare against aws_acm_certificate.validation_method. Emit DNS for the DNS-based modes. PRIVATE_CA stays empty on purpose: it validates nothing, and real ACM omits the method on a private certificate, where the provider reads it as NONE whatever the request asked for. |
||
|
Josh Siviour
|
2638a792c5 |
fix(ecs): task-definition round-trip fidelity and a task that dies while the agent is down (#603)
Persist and echo requiresCompatibilities on task definitions, so a Terraform round trip no longer reads null as drift and force-replaces the definition on every apply. Report a task whose containers all exited while the ecs-agent was down. reconcile only re-adopted running containers, so the gateway held such a task RUNNING forever, its service never replaced it, and the load balancer target stayed unhealthy. |
||
|
Julian Sommer
|
d2eb3b11dd | feat: add manual DB snapshots and restore (#601) | ||
|
Julian Sommer
|
d4d9db9e0b | feat: add DB subnet groups and DB parameter groups (#600) | ||
|
Julian Sommer
|
23de2ad85d | feat: ModifyDBInstance and the shared VM-replace helper (#597) | ||
|
Julian Sommer
|
fc6e055ca0 | feat: RDS lifecycle core — reboot, start, stop, delete, events (#596) | ||
|
Julian Sommer
|
32d80b473a | feat: RDS v1 foundations (#592) | ||
|
Josh Siviour
|
6cf9ee9d03 |
fix(dhcp): retransmit within a DORA transaction, resolve the chaddr on every path, stop masking IPAM errors (#593)
* fix(dhcp): retransmit within a DORA transaction instead of restarting it nclient4's retryFn re-sends the same packet under the same xid with a doubling read deadline, which is RFC 2131 4.1 retransmission. WithRetry(1) runs that loop exactly once, so the client transmitted a single DISCOVER per attempt and never retransmitted it. socketTimeout then handed that lone transmission the caller's whole budget, so it sat silent for 4s, then 8s, then 16s, then 32s. Manager does retry, on that schedule, but each attempt builds a fresh client and therefore a fresh xid. Every attempt is a new transaction rather than a retransmission of the previous one, so an OFFER answering an earlier exchange is discarded as belonging to an unknown transaction. Send six transmissions per exchange starting from a 1s read, spanning a 63s ladder — longer than the widest window Manager schedules, so an attempt ends on the caller's context rather than on an exhausted ladder. socketTimeout now clamps to the caller's remaining budget instead of consuming it in one silent read. Observed live recovering an exchange whose OFFER arrived on the third transmission, which a single-shot client would have missed. Both allocation sites also stop replacing the allocator's error with a bare InsufficientAddressCapacity. ResolveErrorCode unwraps, so a genuinely exhausted static pool still surfaces that code while a DHCP fault reports as itself. The flat mapping is why every failure on this path read as a capacity problem regardless of cause. * chore(dhcptest): add --broadcast so the probe can mirror vpcd's Acquire path * fix(dhcp): resolve the chaddr on renew and release, not just on acquire Acquire derived a chaddr when the request carried none. Renew and Release read lease.HWAddr straight out of the store and handed it to nclient4, so a lease persisted before the chaddr was carried renewed with all zeros in chaddr and option 61 fell back to its untyped form. The lease then lands upstream with no hardware address against it — the outcome option 61 was fixed to stop, reached by a different route — and because the server sees an unknown client each time, every renewal takes a new lease. resolveHWAddr now serves all three paths and treats all-zeros as unusable rather than merely empty: six zero bytes passes a width check yet is the exact value that produces the unattributable entry. * test(dhcp): cover chaddr resolution and the unmasked allocation errors |
||
|
Julian Sommer
|
95973eeb3c | refactor: migrate spinifex off legacy nats.JetStreamContext onto nats.go/jetstream (#584) | ||
|
Julian Sommer
|
e79ca59d4b | fix: configure host DNS from spx admin init/join (#582) | ||
|
Julian Sommer
|
12085f029c | feat: render import progress bars in human-readable units (#581) | ||
|
Julian Sommer
|
1d7d5e1750 | fix: disable debug log | ||
|
Julian Sommer
|
1d55404a73 |
revert: host DNS setup on resolved/resolvconf hosts (#580)
This reverts commit
|
||
|
Julian Sommer
|
f9c9f0b38d | fix: configure host DNS for Spinifex zones on resolved and resolvconf hosts (#580) |