Compare commits

..

3 Commits

Author SHA1 Message Date
Ben Duncan 3e0c24e245 fix(ci): release the ISO build's root-owned cache when the ISO cells end (#1393)
The ISO cells build the rootfs as root inside the workspace and only cleaned
it at the start of their own job, so the next job on the same runner from any
workflow failed its checkout on files it could not remove.
2026-11-01 10:46:41 -06:00
Ben Duncan 3f7d3c2c13 fix(eks): delete the Node of a terminated nodegroup worker (#1392)
* fix(eks): delete the Node of a terminated nodegroup worker

K3s runs no AWS cloud provider, and its own reports every instance as
existing, so a worker terminated by DeleteNodegroup, scale-down or reclaim
stayed a NotReady Node forever. Its pods stayed bound to it, and once most
Nodes were NotReady the node lifecycle controller stopped evicting, leaving
metrics-server on a dead Node.

Add an internal-workers route the control plane pulls the cluster's live
nodegroup workers from, and a mulga-eks-node-gc agent on every server that
deletes a nodegroup Node only when it is not Ready and its instance is no
longer one of those workers, as the AWS cloud controller does.

* test(e2e): check the live worker's Node and wait out nodegroup deletes

WorkerNodeReady accepted any Ready Node, the tolerate-all DaemonSet check
dropped pods with no Ready condition, and nodegroup cleanup returned before
the nodegroup was gone. Scope the Ready check to the CPIsolation nodegroup,
count a pod with no Ready condition as not Ready, wait for each nodegroup
delete, and assert that deleted nodegroups leave no Node behind.

* test(integration): account for the internal ListInternalWorkers route in the cross-tenant table
2026-11-01 10:46:36 -06:00
Ben Duncan 49593b0c98 fix(setup): keep NetworkManager from replacing the resolved stub link (#1391)
NetworkManager restarted while /etc/resolv.conf is a plain file runs in
rc-manager=symlink mode and writes resolv.conf on its first DNS update, which
can land after the installer links the stub. Write a conf.d drop-in
(dns=systemd-resolved, rc-manager=unmanaged) before the restart, refuse a
later file that overrides rc-manager, wait for NetworkManager to settle, and
assert the link holds with bounded relinks. A rerun after an older handover
adds the drop-in, reloads NetworkManager and relinks the stub.
2026-11-01 10:30:43 -06:00
2 changed files with 21 additions and 1 deletions
+20
View File
@@ -1203,6 +1203,16 @@ jobs:
sleep 2
rm -f "$TMPDIR"/tmp.*.qcow2 "$TMPDIR"/tmp.*.log "$TMPDIR"/tmp.*-node.log
# The ISO build runs as root inside the workspace, and the next job on
# this runner, from any workflow, cannot remove what it leaves there.
- name: Release the workspace
if: always()
run: |
sudo umount -R -l "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache/rootfs 2>/dev/null || true
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache \
"$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/dist
sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"
iso-multi:
name: cell-22 (iso/dhcp/three-node)
# After cell-21 rather than beside it: same concurrency group, and running
@@ -1354,6 +1364,16 @@ jobs:
rm -rf /tmp/cell22-artifacts-$GITHUB_RUN_ID /tmp/cell22-$GITHUB_RUN_ID.log \
"$WORK_DIR"
# The ISO build runs as root inside the workspace, and the next job on
# this runner, from any workflow, cannot remove what it leaves there.
- name: Release the workspace
if: always()
run: |
sudo umount -R -l "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache/rootfs 2>/dev/null || true
sudo rm -rf --one-file-system "$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/.cache \
"$GITHUB_WORKSPACE"/mulga/scripts/iso-builder/dist
sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"
# Predastore throughput ("cell 23"). Predastore has no runners of its own and
# spinifex depends on it, so the benchmark lives here and checks predastore
# out as a sibling.
+1 -1
View File
@@ -431,7 +431,7 @@ func eksUncased() map[string]string {
// These name the cluster's account in the path or body. AuthorizeInternal
// refuses every tenant principal outright and binds a control-plane VM to
// the cluster it serves, so no tenant ARN reaches the evaluator.
for _, action := range []string{"GetRecoveryDirective", "ListInternalAddons", "PublishInternal", "WebhookTokenReview"} {
for _, action := range []string{"GetRecoveryDirective", "ListInternalAddons", "ListInternalWorkers", "PublishInternal", "WebhookTokenReview"} {
uncased[action] = "internal control-plane route: refused to tenants, bound to the caller's own cluster"
}
for _, action := range []string{