* feat: S3-over-RDMA with NIXL and cuObject transports A signed S3 request with an empty body names a registered client buffer; the server moves the object bytes with one-sided RDMA (NIXL/UCX, or NVIDIA cuObject through the cuobject crate) instead of the HTTP body. Internode shard reads and writes can also move over NIXL. Applied onto main from #9305 (head0f3c19a0f); the stacked #9302-#9304 commits are already on main aseca95f297. Conflicts were resolved where this change met newer main: crates/ecstore/src/api/mod.rs (kept the internode-transport exports, added the nixl module), crates/rio/src/http_reader.rs (internode_http_request uses the pooled get_http_client::<false>), rustfs/src/storage/rpc/http_service.rs (write_authenticated_put_file stays a plain fn returning a future, now pub(super)), rustfs/src/app/multipart_usecase.rs (the RDMA UploadPart branch moved into multipart_usecase/upload_part.rs after the per-operation split), Cargo.lock (main's lock plus only this change's new packages). * fix(rdma): build the RDMA crates without vendor SDKs and pass guards Maintainer changes on top of the contributed S3-over-RDMA work: - rustfs-cuobject: cuobject is an optional, default-features = false dependency behind a new `server` feature (enabled by the rustfs `cuobject` feature); the crate root is #![cfg(feature = "server")], so without the feature it exports nothing and a workspace build needs no NVIDIA cuObject SDK. - The NIXL agent handle (OnceLock, install_nixl_server, nixl_server) moved from app/object/cuobject.rs to storage/rpc/nixl_read.rs; the app init installs it. The layer-dependency guard rejects infra -> app. - New code reaches ECStore through a cfg-gated `ecstore_rdma` facade module and s3s types through the existing re-export hubs (architecture-migration and s3s-footprint guards). - SAFETY notes at the two #[allow(unsafe_code)] sites, inside the window the unsafe-allowance guard checks. - transport_config_known_backends_are_current_oss_values loops over KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS again, expecting nixl to name itself. - New non-required workflow rdma-feature-check.yml runs `cargo check --locked -p rustfs --features nixl` on Linux (paths filter, weekly, manual), registered in scheduled-validations.json. * fix(ecstore): keep NIXL internode error messages constant The three io::Error::other(format!(..)) sites in nixl_internode.rs (read refused, write refused, write byte-count mismatch) now return constant messages; the status, response body length and byte counts go to a debug! line in the same place, the level this file already uses for a failed NIXL peer call. No response body is logged, as on the HTTP internode path. The error-format ratchet exists because per-disk formatted messages make otherwise identical errors compare unequal and split the quorum reduction; constant messages keep them in one bucket. A typed transport error would carry the detail without that problem; this takes the mechanical route. * fix(ecstore): flush staged NIXL writes before waiting for buffers * fix(rdma): build rustfs-nixl only on Linux nixl-sys's stub build links -lstdc++, which macOS does not ship, so `cargo test -p rustfs-nixl` (and a workspace test run) failed to link on macOS. rustfs-nixl's dependencies are now Linux target dependencies and its crate root is #![cfg(target_os = "linux")]; the rustfs `nixl` feature code that was not already Linux-gated (the storage/rpc nixl_read module, open_shard_contained, the internode route arms and the facade module) is now cfg(all(feature = "nixl", target_os = "linux")). Linux builds are unchanged: the workspace tests still build and run the crate's tests through the stub. * ci(rdma): alert on scheduled failures of the RDMA feature check The test-wiring guard requires every scheduled validation workflow to carry the shared alert-on-failure job; a failed weekly run now opens or updates the tracking issue like the other scheduled lanes. * fix(rdma): put rustfs-nixl behind a sys feature nixl-sys needs a C++ compiler and libclang (bindgen) to build, which the shared Linux runners do not have, so Workspace Clippy and Lint and Build failed once the crate compiled there. Its dependencies are now optional behind a 'sys' feature that the rustfs 'nixl' feature enables; without it (and off Linux) the crate is empty, so no default or workspace build touches vendor C/C++ tooling. The RDMA feature check now also runs the crate's unit tests with the feature on; they use a fake sysfs and host memory, no device. --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Rain Jiang <rain-jiang@outlook.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Documentation
Use the focused indexes rather than treating this directory as an unordered collection:
Operations
-
Candidate release pipeline: frozen inputs, staged acceptance, immutable publication and recovery.
-
Sampled local batch CPU diagnostics — default-off request-correlated allocation and worker wall/CPU diagnosis.
-
Retained MRF responsibilities — cluster observations, explicit disposition, integrity audit boundaries, and unreadable checkpoint recovery.
-
Read-only BTF declaration inspection — bounded prototypes and scalar member layouts, explicit void handling, and independent native references.
-
Internode HTTP streaming transport — protocol selection, pooled connection isolation, and distributed read validation.
-
cuObject S3-over-RDMA GET — optional
cuobjectfeature: GET straight into client GPU/host memory viax-amz-rdma-token; build, settings, wire contract. -
Erasure shard size —
RUSTFS_ERASURE_SHARD_BYTES: larger, aligned shards for new objects; rules and memory cost. -
Object data cache fill on PUT —
RUSTFS_OBJECT_DATA_CACHE_FILL_ON_PUT: insert plain streaming PUT bodies so the first GET hits. -
Persisted config compatibility — ignored MinIO settings, bounded diagnostics, alias retention, and safe resets.
-
Table maintenance periodic queueing — opt-in scheduling, limits, worker handoff, and recovery.
-
S3 Tables catalog lifecycle — disablement, fenced empty-table cleanup, and registration-cache boundaries.
For PUT cleanup admission, blocking-pool isolation, and optional CPU affinity, see PUT cleanup isolation.
-
External certificate issuance and TLS rotation — publication, native reload, trust separation and failure handling.
-
Architecture guard troubleshooting — diagnose repository guard failures.
-
Logging governance — broad event audits, migrations, and guardrail changes.
-
Multipart upload memory diagnosis — allocator attribution, Docker reproduction and sustained upload checks.
For the logical per-operation io_uring read cap, see io_uring read chunk size.
For bounded local read-backend startup and cancellation, see io_uring backend initialization.
For legacy protection assessment and bounded protected copies, see Object integrity inventory, audit, and migration.
For encrypted listing compatibility and older stored fingerprints, see Encrypted object ETags.
For durable hard quota accounting, node-outage behavior, and fence protocol upgrades, see Durable hard quotas.
Operational runbooks live under operations/. Replication
operators should start with:
| Runbook | Use it for |
|---|---|
| Site replication operations | Health fields, pending operations, outage recovery, re-pair admission, IAM/SSE boundaries, and upgrades. |
| Replication target check | Validating an S3 destination and version fidelity before enabling replication. |
| Replication object size limits | Multipart routing, large-object limits, and retry characteristics. |
| Replication outbound transport | Integrity headers, generic target behavior, and transport knobs. |
For the erasure-coded cluster lifecycle (planning, parity and EC:0,
expansion, rebalance, decommission, heal, drive replacement, restart
recovery, and the rc CLI mapping), start with
Cluster and erasure-coding lifecycle operations.
For concurrent bucket creation and deletion, see Bucket operation admission.
For persisted administrator bucket tasks and bucket recreation, see Bucket heal recovery.
For interrupted object deletions and returning drives, see Delete transaction recovery.
For disk replacement across VM restarts and schema 5/6 maintenance migration, see Replacement generation recovery.
For historical GET timeouts during PUT or Heal, see Object lock contention diagnostics.
Other runbooks remain grouped by filename in operations/;
architecture pages link to the relevant runbook where a cross-boundary
procedure is required.
For storage dashboards, see Storage metrics and observer selection: drive ownership, snapshot freshness, counter queries, and rolling upgrades.
For per-bucket capacity and object counts, see Bucket usage metrics: OTLP export, cluster deduplication, accounting semantics, and scanner freshness.
For optional shard commitments, see Independent shard integrity rollout: activation, legacy repair results, multipart mode changes, and rollback limits.
For crates.io publication of workspace crates, see Workspace Cargo Publish: dependency ordering, dry-run, publish, and failure handling.