feat: EBS export tunables, teardown reapers, fan-out and OVN fixes (#800)

Merges `feat/ebs-export-tunables` (83 commits) into dev. The branch could not be rebase-merged because it carries five merges from dev, so it lands as a squash.

## What is in it

- **EBS export tunables** and the nbdkit/viperblock export path they configure.
- **Teardown reapers and backoff** for S3 buckets, load balancers, gateways, spot requests, certificates, endpoints, elastic IPs and network interfaces, so a dependent that keeps failing is backed off rather than retried hot.
- **Retry and tolerance paths**: ECS IAM service init, ECR signing keys deleted between the list and the read, an unreadable signing key at awsgw startup, and an ECS instance-profile policy attach converging on a stale read.
- **Key pair validation reads the material by name** instead of listing the account, closing the `InvalidKeyPair.NotFound` race that a single-replica LIST could produce.
- **Concurrency and fan-out**: concurrent metadata document fetches, bounded metadata reads for describes, and the node-discovery timeout removed from every fan-out call.
- **OVN correctness against the configured NB cluster**: the flows barrier, nexthop MAC binding, and stopping a compute node from running the package's standalone database.
- **nats-js-watchdog can re-arm itself.** The escalation check ran above the probe loop while the only line that cleared the counter sat below it, so a latched watchdog could never observe a recovery. It now probes first and decides second, and says so once per latch rather than once per tick.
- **CI**: 19 commits of nightly and build-pipeline work.

## Verification

`make preflight` green locally. Build and Test, Integration Tests, Race Detection, and Lint and Security all green on `c2da0385`.

## Known red, not introduced here

Two nightly cells fail on this branch and are pre-existing:

- **cell-10** — a partial OVN IPsec enable leaves `NB_Global ipsec=true` asserted cluster-wide while two of three chassis failed their local reconcile, black-holing cross-chassis guest traffic for roughly 5.5 minutes while every health signal stays green.
- **cell-19** — a total layer-3 egress blackhole below a correctly and promptly programmed NAT/EIP datapath, aggravated by a natuplink probe whose retry budget outlives the test's own wait, so it can never emit its FAIL verdict.

Both are filed separately and should be weighed before a release is cut.
This commit is contained in:
Ben Duncan
2026-08-17 17:42:27 -05:00
committed by GitHub
parent 2ced9154bd
commit 225bf65a8e
109 changed files with 13298 additions and 618 deletions
+1 -1
View File
@@ -398,7 +398,7 @@ else
echo " --file $OUTPUT_RAW \\"
NAME_HINT=""
if [[ -n "${AMI_NAME:-}" ]]; then
NAME_HINT=" \\\n --ami-name ${AMI_NAME}"
NAME_HINT=" \\\\\n --ami-name ${AMI_NAME}"
fi
if [[ -n "${SYSTEM_TAG:-}" ]]; then
echo -e " --distro ${DISTRO} --version ${DISTRO_VERSION} --arch x86_64 --boot-mode ${BOOT_MODE}${NAME_HINT} \\"
+19
View File
@@ -572,6 +572,25 @@ EOF
echo " OVN SB DB listening on tcp:127.0.0.1:6642"
fi
fi
else
# A compute node must not run a database. The ovn-central package starts a
# standalone one on install, and nothing here used to stop it, so every
# node past the quorum kept an unclustered NB/SB that no chassis reports
# into. It serves nothing, but it answers on the local socket, which is
# enough for any ovn-nbctl without an explicit --db to confirm against the
# wrong database — a `--wait=hv` there can never be acknowledged and burns
# its whole timeout instead.
if systemctl is-enabled ovn-central >/dev/null 2>&1 ||
systemctl is-active ovn-central >/dev/null 2>&1; then
sudo systemctl stop ovn-central ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb 2>/dev/null || true
sudo systemctl disable ovn-central ovn-northd ovn-ovsdb-server-nb ovn-ovsdb-server-sb 2>/dev/null || true
echo " ovn-central: stopped and disabled (compute node runs no database)"
fi
for db in "$OVN_DBDIR/ovnnb_db.db" "$OVN_DBDIR/ovnsb_db.db"; do
[ -f "$db" ] || continue
sudo mv "$db" "$db.standalone-disabled"
echo " moved aside $db (left by the package's standalone ovn-central)"
done
fi
# --- Step 3: Create and configure br-int ---