Files
rustfs/.github
Chris An 645bb63809 feat: S3-over-RDMA with NIXL and cuObject transports (#9394)
* feat: S3-over-RDMA with NIXL and cuObject transports

A signed S3 request with an empty body names a registered client buffer; the server moves the object bytes with one-sided RDMA (NIXL/UCX, or NVIDIA cuObject through the cuobject crate) instead of the HTTP body. Internode shard reads and writes can also move over NIXL.

Applied onto main from #9305 (head 0f3c19a0f); the stacked #9302-#9304
commits are already on main as eca95f297. Conflicts were resolved where
this change met newer main: crates/ecstore/src/api/mod.rs (kept the
internode-transport exports, added the nixl module),
crates/rio/src/http_reader.rs (internode_http_request uses the pooled
get_http_client::<false>), rustfs/src/storage/rpc/http_service.rs
(write_authenticated_put_file stays a plain fn returning a future, now
pub(super)), rustfs/src/app/multipart_usecase.rs (the RDMA UploadPart
branch moved into multipart_usecase/upload_part.rs after the
per-operation split), Cargo.lock (main's lock plus only this change's
new packages).

* fix(rdma): build the RDMA crates without vendor SDKs and pass guards

Maintainer changes on top of the contributed S3-over-RDMA work:

- rustfs-cuobject: cuobject is an optional, default-features = false
  dependency behind a new `server` feature (enabled by the rustfs
  `cuobject` feature); the crate root is #![cfg(feature = "server")], so
  without the feature it exports nothing and a workspace build needs no
  NVIDIA cuObject SDK.
- The NIXL agent handle (OnceLock, install_nixl_server, nixl_server)
  moved from app/object/cuobject.rs to storage/rpc/nixl_read.rs; the app
  init installs it. The layer-dependency guard rejects infra -> app.
- New code reaches ECStore through a cfg-gated `ecstore_rdma` facade
  module and s3s types through the existing re-export hubs
  (architecture-migration and s3s-footprint guards).
- SAFETY notes at the two #[allow(unsafe_code)] sites, inside the window
  the unsafe-allowance guard checks.
- transport_config_known_backends_are_current_oss_values loops over
  KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS again, expecting nixl to name
  itself.
- New non-required workflow rdma-feature-check.yml runs
  `cargo check --locked -p rustfs --features nixl` on Linux (paths
  filter, weekly, manual), registered in scheduled-validations.json.

* fix(ecstore): keep NIXL internode error messages constant

The three io::Error::other(format!(..)) sites in nixl_internode.rs
(read refused, write refused, write byte-count mismatch) now return
constant messages; the status, response body length and byte counts go
to a debug! line in the same place, the level this file already uses for
a failed NIXL peer call. No response body is logged, as on the HTTP
internode path.

The error-format ratchet exists because per-disk formatted messages make
otherwise identical errors compare unequal and split the quorum
reduction; constant messages keep them in one bucket. A typed transport
error would carry the detail without that problem; this takes the
mechanical route.

* fix(ecstore): flush staged NIXL writes before waiting for buffers

* fix(rdma): build rustfs-nixl only on Linux

nixl-sys's stub build links -lstdc++, which macOS does not ship, so
`cargo test -p rustfs-nixl` (and a workspace test run) failed to link on
macOS. rustfs-nixl's dependencies are now Linux target dependencies and
its crate root is #![cfg(target_os = "linux")]; the rustfs `nixl`
feature code that was not already Linux-gated (the storage/rpc nixl_read
module, open_shard_contained, the internode route arms and the facade
module) is now cfg(all(feature = "nixl", target_os = "linux")). Linux
builds are unchanged: the workspace tests still build and run the
crate's tests through the stub.

* ci(rdma): alert on scheduled failures of the RDMA feature check

The test-wiring guard requires every scheduled validation workflow to carry the shared alert-on-failure job; a failed weekly run now opens or updates the tracking issue like the other scheduled lanes.

* fix(rdma): put rustfs-nixl behind a sys feature

nixl-sys needs a C++ compiler and libclang (bindgen) to build, which the shared Linux runners do not have, so Workspace Clippy and Lint and Build failed once the crate compiled there. Its dependencies are now optional behind a 'sys' feature that the rustfs 'nixl' feature enables; without it (and off Linux) the crate is empty, so no default or workspace build touches vendor C/C++ tooling. The RDMA feature check now also runs the crate's unit tests with the feature on; they use a fake sysfs and host memory, no device.

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Rain Jiang <rain-jiang@outlook.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-15 00:10:10 +08:00
..
2026-07-20 10:44:06 +08:00