hector
e494dc2b4b
* ci(validation): sweep the migrated canonical validation cases nightly The canonical release-validation cases migrated into auto-testing by rustfs/auto-testing#113 (66 implemented, all non-heal) currently run only through the manual validation flow; the last fleet executions stopped on 2026-10-27 and nothing schedules them. Add a scheduled validation sweep that closes the gap: - nightly at 19:30 UTC (after the functional chain settles), rotating windows of the implemented catalog cases so the whole catalog is covered roughly weekly, with a bounded batch per night - workflow_dispatch for a single case (--case), one module (--module), or an explicit batch (--limit), mirroring how the lanes are testable standalone - fleet admission through functional_environment enter/finish with the same quarantine discipline as the lanes, inter-case node cleanup, per-case logs + reports retained as a sweep artifact - a markdown sweep report uploaded to rustfs/dashboard under functional-reports/validation/ so the cases appear next to the lane reports, plus an alert-on-failure job for scheduled runs - heal.* cases stay on the interactive validation flow: many need the reviewed native/tool collectors built by separate validation-tools runs before fleet execution The registry gains the workflow responsibility (monitor: workflow, max-age 60h so the watchdog notices a dead sweep) and the wiring test baseline count moves with it. Verified locally: check_wiring, check_test_wiring, test_validation_registry (9/9), test_functional_chain + test_functional_chain_health + test_security_workflow (99/99). * ci(validation): group the sweep report by catalog module The validation cases cover distinct areas (pool lifecycle, ILM expiration, bucket/site replication, kms, data services, identity, s3 compatibility), so a flat case table hides the category breakdown. Render one section per catalog module with its own case table and a Subtotal line, and keep the single grand - Total:/- PASS:/- FAIL: block for the dashboard overview parser. Subtotals deliberately use a different bullet prefix so the overview regexes cannot mistake a module subtotal for the grand total. * ci(validation): sweep the eight planned cases whose executors exist The planned catalog entries are not all unimplemented: all eight (kms.sse-kms-round-trip, tiering.transition-and-restore, events.delivery, audit.request-correlation, iam.policy-enforcement, oidc.login-and-session, s3.basic-object-round-trip and scanner.usage-convergence) have dedicated executors routed in run_validation_case.sh and live PASS evidence recorded in the migration map. Widen the sweep filter to include them: the rotating nightly batch now covers 74 cases (66 implemented + these 8), and the fresh evidence each sweep produces feeds their acceptance review. heal.* procedure-ready cases remain excluded - their executors need the reviewed native/tool collectors built by separate validation-tools runs before fleet execution. --------- Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Workflow runners: what is billed and where a job belongs
Use this when: adding a job, changing a runs-on, moving a lane between runner pools, or deciding whether a new check is worth its runner cost.
Source of truth: runs-on in each workflow file for current placement; .github/actionlint.yaml for the declared self-hosted labels; the organisation's Settings → Billing → Usage page for what is actually charged; docs/testing/ci-gates.md for which checks block a merge (that does not change when a runner changes).
Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (ubuntu-latest, windows-latest, macos-latest) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
| Label | Paid by | What it is | Use it for |
|---|---|---|---|
ubuntu-latest |
nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, docker, gh, curl, wget, jq, Python. Can mount tmpfs and loop devices; io_uring works |
the default for every job |
windows-latest, macos-latest |
nobody | GitHub-hosted standard runners | Windows and macOS targets |
sm-standard-2 |
RustFS | ARC pod, Ubuntu 24.04, minimal image: no gh, no C toolchain; Docker only on the dind- label |
only a job that a hosted runner cannot run (see the placement rule) |
sm-standard-4 |
RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
ubicloud-standard-16-arm |
RustFS | Ubicloud VM, alias of ubicloud-standard-16-arm-ubuntu-2404: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the build.yml matrix, so actionlint.yaml does not list it |
native aarch64 release builds (linux-aarch64-musl, linux-aarch64-gnu) |
dind-sm-standard-2 |
RustFS | ARC pod with Docker-in-Docker | Docker workloads with a measured resource or environment requirement beyond the hosted runner |
smoke-testing, pf-testing |
RustFS | long-lived lab machines | the packaged functional chain and performance suites |
The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; ci.yml builds the debug binary on sm-standard-4 and runs it on both pools.
Placement rule
A job runs on ubuntu-latest unless it has one of these reasons, and the reason goes in a comment next to its runs-on:
- Disk. It restores or produces the workspace
target/:cargo buildorcargo testwith--all-targets,cargo nextest run --all, or any reader of theci-dev/ci-feat-*caches. The workspace test build leaves about 64 GB intarget/; a hosted runner has about 14 GB. - Critical-path CPU. It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (
build-rustfs-debug-binarygates the e2e and s3-tests lanes). - Hardware or environment. Native aarch64 or access to the lab machines. Docker alone is not a reason: hosted Ubuntu ships Docker and supports the QEMU and Buildx setup actions.
- Cache ownership. It reads a
rust-cachekey that a self-hosted job writes (see the cache rule below).
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, cargo fmt, cargo deny, typos, actionlint, API calls (gh, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a cargo build -p rustfs with its own cache key. e2e-distributed.yml and e2e-upgrade.yml build the server and the e2e_test harness on ubuntu-latest inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's prepare and complete-chain jobs, the performance runner availability check, and runner hygiene use ubuntu-latest: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses ubuntu-latest so a lab outage cannot also stop its independent monitor. Docker image packaging also uses ubuntu-latest for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
A hosted runner is also the only choice for a few things the pods cannot do: mount -o loop and mount -t tmpfs fail inside the ARC pods, and a container runtime cannot guarantee that io_uring_setup is allowed. uring-integration and e2e-distributed are on ubuntu-latest for that reason, not for cost.
Cache rule
Swatinem/rust-cache builds its key from the OS, architecture, toolchain and lockfile. It does not include the runner label or image, so a cache written on an sm-standard-4 pod and one written on ubuntu-latest would overwrite each other under the same shared-key.
- Every
shared-keyhas exactly one writer.cache-warm.ymlownsci-dev,ci-feat-rio,ci-feat-protoandci-uring; every other./.github/actions/setupcall statescache-save-ifexplicitly, andscripts/security/check_cache_save_if.shfails the build when one does not. - All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (
ci-uringis warmed onubuntu-latest;ci-e2e-distributed-hosted,e2e-upgrade-server-hosted,e2e-upgrade-tests-hosted). - Moving a job between pools therefore means renaming its key. The new key is empty until a run from
main(a scheduled run, or a manual dispatch of that workflow frommain) saves it; until then that lane does a cold build. Thebuild.ymlaarch64 release legs append-ubicloud-armfor this reason.cache-warm.ymldoes not write those keys. CARGO_BUILD_JOBSand every otherCARGO*,CC*,CFLAGS*,CXX*,CMAKE*andRUST*variable is hashed into the key.ci.ymlsetsCARGO_BUILD_JOBSat step level for that reason; promoting it to job level rotates the key for every lane that shares it.
Where self-hosted minutes leak
These are the patterns that have cost real money here without validating anything more:
- An
if: always()summary or alert job on a self-hosted label inside aworkflow_run-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it onubuntu-latest. - Per-push workflows on
main. A merge triggers the fullci.ymlmatrix,cache-warm.ymland thebuild.ymldevelopment build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind apathsfilter. apt-get installof tools the hosted image already ships. It is only needed on the minimal pod images.- A throttled
CARGO_BUILD_JOBSon a 14-CPU pod. The limit exists for link-storm reasons documented inci.yml; measure before raising it, and measure before lowering it further. - Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
Measuring a run
To see which pool each job of a run used and for how long:
gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
Only rows whose label starts with sm-, dind-, ubicloud-, smoke- or pf- cost anything. At the October 2026 review, a full pull-request run of ci.yml spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Checklist for a new or moved job
runs-onis a literal label. A new self-hosted label is declared in.github/actionlint.yamlin the same change (.github/AGENTS.md).- The job declares
timeout-minutes(scripts/security/check_job_timeouts.sh). actionlintreports nothing.- A self-hosted
runs-oncarries a comment naming its reason from the placement rule. - Every
./.github/actions/setupcall statescache-save-if, and a key read from one pool is not written from the other. - Moving a job does not change its check name, so the
mainruleset anddocs/testing/ci-gates.mdstay as they are; renaming it does, and then that document's change checklist applies.