* feat: S3-over-RDMA with NIXL and cuObject transports A signed S3 request with an empty body names a registered client buffer; the server moves the object bytes with one-sided RDMA (NIXL/UCX, or NVIDIA cuObject through the cuobject crate) instead of the HTTP body. Internode shard reads and writes can also move over NIXL. Applied onto main from #9305 (head0f3c19a0f); the stacked #9302-#9304 commits are already on main aseca95f297. Conflicts were resolved where this change met newer main: crates/ecstore/src/api/mod.rs (kept the internode-transport exports, added the nixl module), crates/rio/src/http_reader.rs (internode_http_request uses the pooled get_http_client::<false>), rustfs/src/storage/rpc/http_service.rs (write_authenticated_put_file stays a plain fn returning a future, now pub(super)), rustfs/src/app/multipart_usecase.rs (the RDMA UploadPart branch moved into multipart_usecase/upload_part.rs after the per-operation split), Cargo.lock (main's lock plus only this change's new packages). * fix(rdma): build the RDMA crates without vendor SDKs and pass guards Maintainer changes on top of the contributed S3-over-RDMA work: - rustfs-cuobject: cuobject is an optional, default-features = false dependency behind a new `server` feature (enabled by the rustfs `cuobject` feature); the crate root is #![cfg(feature = "server")], so without the feature it exports nothing and a workspace build needs no NVIDIA cuObject SDK. - The NIXL agent handle (OnceLock, install_nixl_server, nixl_server) moved from app/object/cuobject.rs to storage/rpc/nixl_read.rs; the app init installs it. The layer-dependency guard rejects infra -> app. - New code reaches ECStore through a cfg-gated `ecstore_rdma` facade module and s3s types through the existing re-export hubs (architecture-migration and s3s-footprint guards). - SAFETY notes at the two #[allow(unsafe_code)] sites, inside the window the unsafe-allowance guard checks. - transport_config_known_backends_are_current_oss_values loops over KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS again, expecting nixl to name itself. - New non-required workflow rdma-feature-check.yml runs `cargo check --locked -p rustfs --features nixl` on Linux (paths filter, weekly, manual), registered in scheduled-validations.json. * fix(ecstore): keep NIXL internode error messages constant The three io::Error::other(format!(..)) sites in nixl_internode.rs (read refused, write refused, write byte-count mismatch) now return constant messages; the status, response body length and byte counts go to a debug! line in the same place, the level this file already uses for a failed NIXL peer call. No response body is logged, as on the HTTP internode path. The error-format ratchet exists because per-disk formatted messages make otherwise identical errors compare unequal and split the quorum reduction; constant messages keep them in one bucket. A typed transport error would carry the detail without that problem; this takes the mechanical route. * fix(ecstore): flush staged NIXL writes before waiting for buffers * fix(rdma): build rustfs-nixl only on Linux nixl-sys's stub build links -lstdc++, which macOS does not ship, so `cargo test -p rustfs-nixl` (and a workspace test run) failed to link on macOS. rustfs-nixl's dependencies are now Linux target dependencies and its crate root is #![cfg(target_os = "linux")]; the rustfs `nixl` feature code that was not already Linux-gated (the storage/rpc nixl_read module, open_shard_contained, the internode route arms and the facade module) is now cfg(all(feature = "nixl", target_os = "linux")). Linux builds are unchanged: the workspace tests still build and run the crate's tests through the stub. * ci(rdma): alert on scheduled failures of the RDMA feature check The test-wiring guard requires every scheduled validation workflow to carry the shared alert-on-failure job; a failed weekly run now opens or updates the tracking issue like the other scheduled lanes. * fix(rdma): put rustfs-nixl behind a sys feature nixl-sys needs a C++ compiler and libclang (bindgen) to build, which the shared Linux runners do not have, so Workspace Clippy and Lint and Build failed once the crate compiled there. Its dependencies are now optional behind a 'sys' feature that the rustfs 'nixl' feature enables; without it (and off Linux) the crate is empty, so no default or workspace build touches vendor C/C++ tooling. The RDMA feature check now also runs the crate's unit tests with the feature on; they use a fake sysfs and host memory, no device. --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Rain Jiang <rain-jiang@outlook.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Workflow runners: what is billed and where a job belongs
Use this when: adding a job, changing a runs-on, moving a lane between runner pools, or deciding whether a new check is worth its runner cost.
Source of truth: runs-on in each workflow file for current placement; .github/actionlint.yaml for the declared self-hosted labels; the organisation's Settings → Billing → Usage page for what is actually charged; docs/testing/ci-gates.md for which checks block a merge (that does not change when a runner changes).
Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (ubuntu-latest, windows-latest, macos-latest) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the macos-latest-xlarge release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
| Label | Paid by | What it is | Use it for |
|---|---|---|---|
ubuntu-latest |
nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, docker, gh, curl, wget, jq, Python. Can mount tmpfs and loop devices; io_uring works |
the default for every job |
windows-latest, macos-latest |
nobody | GitHub-hosted standard runners | Windows and macOS targets |
macos-latest-xlarge |
RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
sm-standard-2 |
RustFS | ARC pod, Ubuntu 24.04, minimal image: no gh, no C toolchain; Docker only on the dind- label |
only a job that a hosted runner cannot run (see the placement rule) |
sm-standard-4 |
RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
ubicloud-standard-16-arm |
RustFS | Ubicloud VM, alias of ubicloud-standard-16-arm-ubuntu-2404: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the build.yml matrix, so actionlint.yaml does not list it |
native aarch64 release builds (linux-aarch64-musl, linux-aarch64-gnu) |
dind-sm-standard-2 |
RustFS | ARC pod with Docker-in-Docker | Docker workloads with a measured resource or environment requirement beyond the hosted runner |
smoke-testing, pf-testing |
RustFS | long-lived lab machines | the packaged functional chain and performance suites |
The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; ci.yml builds the debug binary on sm-standard-4 and runs it on both pools.
Placement rule
A job runs on ubuntu-latest unless it has one of these reasons, and the reason goes in a comment next to its runs-on:
- Disk. It restores or produces the workspace
target/:cargo buildorcargo testwith--all-targets,cargo nextest run --all, or any reader of theci-dev/ci-feat-*caches. The workspace test build leaves about 64 GB intarget/; a hosted runner has about 14 GB. - Critical-path CPU. It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (
build-rustfs-debug-binarygates the e2e and s3-tests lanes). - Hardware or environment. Native aarch64 or access to the lab machines. Docker alone is not a reason: hosted Ubuntu ships Docker and supports the QEMU and Buildx setup actions.
- Cache ownership. It reads a
rust-cachekey that a self-hosted job writes (see the cache rule below).
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, cargo fmt, cargo deny, typos, actionlint, API calls (gh, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a cargo build -p rustfs with its own cache key. e2e-distributed.yml and e2e-upgrade.yml build the server and the e2e_test harness on ubuntu-latest inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's prepare and complete-chain jobs, the performance runner availability check, and runner hygiene use ubuntu-latest: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses ubuntu-latest so a lab outage cannot also stop its independent monitor. Docker image packaging also uses ubuntu-latest for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
A hosted runner is also the only choice for a few things the pods cannot do: mount -o loop and mount -t tmpfs fail inside the ARC pods, and a container runtime cannot guarantee that io_uring_setup is allowed. uring-integration and e2e-distributed are on ubuntu-latest for that reason, not for cost.
Cache rule
Swatinem/rust-cache builds its key from the OS, architecture, toolchain and lockfile. It does not include the runner label or image, so a cache written on an sm-standard-4 pod and one written on ubuntu-latest would overwrite each other under the same shared-key.
- Every
shared-keyhas exactly one writer.cache-warm.ymlownsci-dev,ci-feat-rio,ci-feat-protoandci-uring; every other./.github/actions/setupcall statescache-save-ifexplicitly, andscripts/security/check_cache_save_if.shfails the build when one does not. - The
ci-devkey ends in a digest ofwarm-ci-dev's build commands. Changing what that job warms therefore needs a new key, or readers keep restoring the old artifacts:scripts/ci_gate.py --check-workflowprints the key the commands require and fails until the writer and every reader inci.ymlandreference-parity.ymlname it. - All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (
ci-uringis warmed onubuntu-latest;ci-e2e-distributed-hosted,e2e-upgrade-server-hosted,e2e-upgrade-tests-hosted). - Moving a job between pools therefore means renaming its key. The new key is empty until a run from
main(a scheduled run, or a manual dispatch of that workflow frommain) saves it; until then that lane does a cold build. Thebuild.ymlaarch64 release legs append-ubicloud-armfor this reason.cache-warm.ymldoes not write those keys. CARGO_BUILD_JOBSand every otherCARGO*,CC*,CFLAGS*,CXX*,CMAKE*andRUST*variable is hashed into the key.ci.ymlsetsCARGO_BUILD_JOBSat step level for that reason; promoting it to job level rotates the key for every lane that shares it.
Where self-hosted minutes leak
These are the patterns that have cost real money here without validating anything more:
- An
if: always()summary or alert job on a self-hosted label inside aworkflow_run-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it onubuntu-latest. - Per-push workflows on
main. A merge triggers cache warming and aci.ymlrun, which validates the merged tree in full unless a merge-queue run already validated that tree;build.ymldevelopment builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind apathsfilter. apt-get installof tools the hosted image already ships. It is only needed on the minimal pod images.- A throttled
CARGO_BUILD_JOBSon a 14-CPU pod. The limit exists for link-storm reasons documented inci.yml; measure before raising it, and measure before lowering it further. - Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
Measuring a run
To see which pool each job of a run used and for how long:
gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
Rows using macos-latest-xlarge or a label starting with sm-, dind-, ubicloud-, smoke- or pf- use paid capacity. At the October 2026 review, a full pull-request run of ci.yml spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Checklist for a new or moved job
runs-onis a literal label. A new self-hosted label is declared in.github/actionlint.yamlin the same change (.github/AGENTS.md).- The job declares
timeout-minutes(scripts/security/check_job_timeouts.sh). actionlintreports nothing.- A self-hosted
runs-oncarries a comment naming its reason from the placement rule. - Every
./.github/actions/setupcall statescache-save-if, and a key read from one pool is not written from the other. - Moving a job does not change its check name, so the
mainruleset anddocs/testing/ci-gates.mdstay as they are; renaming it does, and then that document's change checklist applies.
Release stages
The candidate release pipeline builds Linux first and defers paid desktop capacity until acceptance. The daily build.yml schedule selects Linux; the Sunday 16:17 UTC schedule selects macOS and Windows. Manual dispatch can still select any platform list. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see GitHub runner billing. Actual spending comes from organization billing, not runner labels or an estimated build duration.