mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-27 06:18:43 +00:00
Chris
b06923beaf
* test(e2e): add scanner/heal topology protection matrix Add the S00 protection baseline for the Scanner/Heal program (rustfs/backlog#2873, program rustfs/backlog#2872): seven scanner_heal_topology_ cases in the distributed e2e lane that pin pool expansion, rebalance and decommission behavior before the runtime changes. The fixture seeds a deterministic inventory (empty, inline, external, multipart, special and nested keys; a versioned bucket with an explicit null version, three history versions and a delete marker per key; an Object Lock bucket with legal hold and COMPLIANCE retention), records every version with its requested and listed version id, length and SHA-256, verifies the manifest through every node, reads physical placement from xl.meta on the pool drives, and reads the strict durability mode back from the admin API and the node logs. Fault cases hold the mover at proven windows through file barriers that compile only with the e2e-test-hooks feature: before the multipart target commit and before source cleanup (shared by decommission and rebalance), plus the existing pool-metadata before_publish cutpoint. An unreached window fails the case. DistCluster gains capture_logs_under so a case can read node startup events without a global environment variable. docs/testing/scanner-heal-topology-gates.md maps the gates to the tests and records the timeout basis. * test(e2e): keep the barrier failure message static The error(format!) ratchet rejects new formatted io::Error::other call sites in ecstore. The reached observation already records the stage and object, so the injected failure needs no per-call detail. * test(e2e): require e2e-test-hooks for topology fault cases The four scanner_heal_topology_ fault cases inject at server barriers that exist only with the e2e-test-hooks feature. Against a default server they waited for the barrier timeout and reported it as not reached. Check the verified binary's features first so the missing prerequisite fails immediately and is named. * ci(e2e): build the distributed lane with e2e-test-hooks Every other e2e lane builds and runs the fault-test hooks server. The distributed lane alone used the default server, so the topology fault cases could never reach their barriers there. Build and run with the same feature so the receipt matches, and update the documented local commands. The upgrade cases spawn the pinned previous release directly and do not compare feature sets. * test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR The topology fixture captured node logs, barrier observations and evidence in a temp dir removed on drop, so the distributed workflow uploaded nothing for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow uploads, when no explicit evidence directory is set. * test(e2e): remove retired pools before restart; race the pre-ACK start The queue case restarted with decommissioned pools 0 and 1 still on the command line, which the server rightly refuses. Add remove_single_node_pools and restart without them, then expect the remaining pools renumbered to 0 and 1. The pre-ACK case now races the start request against the barrier. A start that returns first, or stays in flight past the documented 60 s budget, is reported as such instead of as an unreached barrier, and the evidence records the reach latency. capture_logs_under now overrides a log path assigned from RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too. * test(e2e): bound fault-window waits by the movement budget On Linux CI a busy target capacity gate delayed the first fault object by about two minutes, past the 120 s wait borrowed from the server's release poll. Wait for a movement window within the 240 s movement budget instead, and fail at once when the decommission or rebalance ends without entering the window, so an uninjected fault is never counted. * test(e2e): bound status probes while waiting for a fault window The admin decommission status request can wait on a mover held at a test barrier until the barrier's own 120 s release timeout. The window wait issued that request each loop, so a reached barrier went unnoticed until the server timed it out and failed the decommission, which surfaced as a false resume failure. Bound each status probe to 5 s and treat a slow probe as still running. * test(e2e): retry a busy decommission start A decommission start that meets a busy movement gate can answer a retryable 503 instead of waiting. The harness treated any non-proof failure as fatal. Share one start helper that retries 503 within the case's movement budget and the fleet-proof 500 for 45 s, and keeps every other status fatal. The pre-ACK case retries 503 within its window budget.
Workflow runners: what is billed and where a job belongs
Use this when: adding a job, changing a runs-on, moving a lane between runner pools, or deciding whether a new check is worth its runner cost.
Source of truth: runs-on in each workflow file for current placement; .github/actionlint.yaml for the declared self-hosted labels; the organisation's Settings → Billing → Usage page for what is actually charged; docs/testing/ci-gates.md for which checks block a merge (that does not change when a runner changes).
Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (ubuntu-latest, windows-latest, macos-latest) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for by the hour whether or not a job is running well:
| Label | Paid by | What it is | Use it for |
|---|---|---|---|
ubuntu-latest |
nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, docker, gh, curl, wget, jq, Python. Can mount tmpfs and loop devices; io_uring works |
the default for every job |
windows-latest, macos-latest |
nobody | GitHub-hosted standard runners | Windows and macOS targets |
sm-standard-2 |
RustFS | ARC pod, Ubuntu 24.04, minimal image: no gh, no C toolchain; Docker only on the dind- label |
only a job that a hosted runner cannot run (see the placement rule) |
sm-standard-4 |
RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
sm-standard-4-arm |
RustFS | ARC pod, aarch64; reached through the build.yml matrix, so actionlint.yaml does not list it |
native aarch64 release builds |
dind-sm-standard-2 |
RustFS | ARC pod with Docker-in-Docker | image builds that need buildx and a registry push |
smoke-testing, pf-testing |
RustFS | long-lived lab machines | the packaged functional chain and performance suites |
The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; ci.yml builds the debug binary on sm-standard-4 and runs it on both pools.
Placement rule
A job runs on ubuntu-latest unless it has one of these reasons, and the reason goes in a comment next to its runs-on:
- Disk. It restores or produces the workspace
target/:cargo buildorcargo testwith--all-targets,cargo nextest run --all, or any reader of theci-dev/ci-feat-*caches. The workspace test build leaves about 64 GB intarget/; a hosted runner has about 14 GB. - Critical-path CPU. It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (
build-rustfs-debug-binarygates the e2e and s3-tests lanes). - Hardware. Native aarch64, Docker-in-Docker, or the lab machines.
- Cache ownership. It reads a
rust-cachekey that a self-hosted job writes (see the cache rule below).
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, cargo fmt, cargo deny, typos, actionlint, API calls (gh, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a cargo build -p rustfs with its own cache key. e2e-distributed.yml and e2e-upgrade.yml build the server and the e2e_test harness on ubuntu-latest inside their budgets, so "it compiles something" alone is not a reason.
A hosted runner is also the only choice for a few things the pods cannot do: mount -o loop and mount -t tmpfs fail inside the ARC pods, and a container runtime cannot guarantee that io_uring_setup is allowed. uring-integration and e2e-distributed are on ubuntu-latest for that reason, not for cost.
Cache rule
Swatinem/rust-cache builds its key from the OS, architecture, toolchain and lockfile. It does not include the runner label or image, so a cache written on an sm-standard-4 pod and one written on ubuntu-latest would overwrite each other under the same shared-key.
- Every
shared-keyhas exactly one writer.cache-warm.ymlownsci-dev,ci-feat-rio,ci-feat-protoandci-uring; every other./.github/actions/setupcall statescache-save-ifexplicitly, andscripts/security/check_cache_save_if.shfails the build when one does not. - All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (
ci-uringis warmed onubuntu-latest;ci-e2e-distributed-hosted,e2e-upgrade-server-hosted,e2e-upgrade-tests-hosted). - Moving a job between pools therefore means renaming its key. The new key is empty until a run from
main(a scheduled run, or a manual dispatch of that workflow frommain) saves it; until then that lane does a cold build. CARGO_BUILD_JOBSand every otherCARGO*,CC*,CFLAGS*,CXX*,CMAKE*andRUST*variable is hashed into the key.ci.ymlsetsCARGO_BUILD_JOBSat step level for that reason; promoting it to job level rotates the key for every lane that shares it.
Where self-hosted minutes leak
These are the patterns that have cost real money here without validating anything more:
- An
if: always()summary or alert job on a self-hosted label inside aworkflow_run-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it onubuntu-latest. - Per-push workflows on
main. A merge triggers the fullci.ymlmatrix,cache-warm.ymland thebuild.ymldevelopment build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind apathsfilter. apt-get installof tools the hosted image already ships. It is only needed on the minimal pod images.- A throttled
CARGO_BUILD_JOBSon a 14-CPU pod. The limit exists for link-storm reasons documented inci.yml; measure before raising it, and measure before lowering it further. - Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
Measuring a run
To see which pool each job of a run used and for how long:
gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
Only rows whose label starts with sm-, dind-, smoke- or pf- cost anything. At the October 2026 review, a full pull-request run of ci.yml spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Checklist for a new or moved job
runs-onis a literal label. A new self-hosted label is declared in.github/actionlint.yamlin the same change (.github/AGENTS.md).- The job declares
timeout-minutes(scripts/security/check_job_timeouts.sh). actionlintreports nothing.- A self-hosted
runs-oncarries a comment naming its reason from the placement rule. - Every
./.github/actions/setupcall statescache-save-if, and a key read from one pool is not written from the other. - Moving a job does not change its check name, so the
mainruleset anddocs/testing/ci-gates.mdstay as they are; renaming it does, and then that document's change checklist applies.