The s3-tests lifecycle expiration cases (test_lifecycle_expiration,
test_lifecyclev2_expiration, test_lifecycle_deletemarker_expiration)
flaked with counts stalling one plateau behind (e.g. `assert 6 == 4`).
Root cause: a compacted directory is only re-descended -- and its
objects re-evaluated against ILM rules -- once every
DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the lane's
accelerated RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so
a Days=1 object due at debug_day (10s) is not actually expired until the
next ~32s boundary (~42s) -- just past the test's 4*lc_interval (40s)
poll window, so the list still returns the pre-expiry count.
Set RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 for this debug-only lane so
compacted directories are re-descended every cycle and ILM fires within
~2s of the due time, comfortably inside the poll window. This does not
touch the 4*lc_interval < 5*debug_day plateau invariant.
`full_object_plaintext_len` decides whether a body-cache hook hit may serve
bytes in place of the erasure read. It is a fail-closed allow-list: it excludes
every read whose `ReadPlan::build` applies some other transform (ranged/part,
raw/data-movement, restore, encrypted, remote) with an early `return None`,
then returns a `Some(..)` length only for the whole-plaintext cases. A newly
added `ReadPlan` branch that nobody teaches this gate about falls through to
`None` and safely bypasses the cache. Flip it to a deny-list and the same new
branch silently serves bytes in the wrong representation — the backlog#1108 /
#1109 / #1146 class of bug.
The existing unit and e2e tests only cover the branches that exist today. This
adds `scripts/check_body_cache_whitelist.sh`, a structural guard wired into
pre-commit / pre-pr / dev-check and CI, that asserts every exclusion predicate
and a `return None` still precede the first `Some(..)`. Reordering a predicate,
dropping one, moving the positive return ahead of the gate, or renaming/removing
the function all fail; wording, formatting, and adding a new exclusion in the
same gate do not. Mutation-tested against all four regression shapes.
This machine-enforces the structural invariant that backlog#1146 was kept open
to guard by hand.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): close the fd-cache open-then-insert race with a generation guard (rustfs/backlog#1176)
pread_uring's miss path opened a descriptor on the blocking pool and only then
inserted it into the moka cache. moka's invalidations cover only entries present
at call time, so a heal/delete commit that invalidated between the open and the
insert could not stop the just-opened stale inode from being cached afterwards —
serving the pre-heal/pre-delete inode for up to the TTL and defeating the heal.
Add an invalidation generation to FdCache, bumped by invalidate_exact and
invalidate_under before they touch moka. The read path snapshots the generation
before opening and inserts via insert_if_fresh, which refuses the insert if the
generation moved during the open and, with a post-insert re-check, removes the
entry if an invalidation raced the insert itself. Reads that never miss are
unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate the fd cache on the primary object-delete paths (rustfs/backlog#1175)
The fd-cache invalidation contract was only wired into DiskAPI::delete,
rename_file and rename_data, but object deletion almost never goes through
LocalDisk::delete — DeleteObject(s) reach delete_version, delete_versions ->
delete_versions_internal, and delete_paths, all of which remove a version's data
dir (move_to_trash / rename_all staging) with no invalidation. A cached io_uring
descriptor kept the deleted part.N inode readable for up to the TTL, so a GET in
that window could still return deleted data.
Invalidate every cached fd under the removed data dir at each site: in
delete_version and delete_versions_internal the data_dir uuid and object path are
in hand (invalidate_cached_fds_under(volume, "{path}/{uuid}")); delete_paths
invalidates under each removed path. A later rollback that restores a data dir
just causes the next read to re-open it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): close remaining fd-cache invalidation gaps (rustfs/backlog#1177)
Three residual paths could keep serving a stale descriptor:
- delete_volume removed the whole bucket tree (remove_dir_all/remove_dir) with no
invalidation, and the cache-hit read path skips the volume-access check, so a
cached fd kept a removed object readable. Add invalidate_cached_fds_for_volume
(a per-volume moka predicate) and call it after the bucket is removed.
- A retired LocalDisk instance (renew_disk on reconnect builds a fresh one) kept
its populated cache alive while still referenced by in-flight ops, so
invalidations through the new instance never reached it. close() now clears the
backend's cache via clear_cached_fds.
- rename_data's post-commit rollback (a commit-metadata fsync failure under
strict durability) restored the old data dir without dropping fds cached during
the committed window; the streaming branch now invalidates the dst part fds on
those rollback paths. The inline branch's rollback runs inside spawn_blocking
and is left to the TTL backstop.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): narrow the io_uring latch classes to match StdBackend (rustfs/backlog#1171)
The runtime degradation classification reused the probe-time restriction errnos,
which the driver's C7 contract explicitly warns against, so a single per-file
error could latch a whole disk off io_uring:
- is_io_uring_unsupported no longer includes EACCES: at read time on an
already-open fd it is per-file (an LSM hooks security_file_permission on every
read) and StdBackend hits the same denial, so falling back masks nothing and a
full-disk latch would be wrong. ENOSYS and EPERM (seccomp/LSM applied after
startup) remain. EOPNOTSUPP is now classified per-path by the caller.
- pread_uring_direct's read-error arm now mirrors StdBackend: an O_DIRECT-shape
error (EINVAL/EOPNOTSUPP) latches only direct_uring.supported so eligible reads
take StdBackend's aligned path, instead of over-latching the whole io_uring
backend or never latching a read-side EINVAL at all.
- try_new only negative-caches genuine restriction-class probe failures in
URING_UNSUPPORTED_DISKS; an unexpected (possibly transient) probe failure now
falls back without latching, so the next reconnect re-probes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): log when a disk latches io_uring off at runtime (rustfs/backlog#1172)
A probe-gated gray release was flying blind: the permanent per-disk `active`
latch flipped with no log and no metric, so the only message operators ever saw
was the startup "io_uring read backend enabled" line — which stayed true on
dashboards even after the very first read latched the disk back to StdBackend
forever.
Add latch_active_off, which flips the latch with `swap` and logs the true->false
transition exactly once at warn with a dedicated event constant, disk root, and
errno. Both the buffered and O_DIRECT read paths use it. A fallback/latch metric
counter and periodic export of the driver StatsSnapshot (cq_overflow,
cancel_already) remain as follow-ups that need rustfs_io_metrics plumbing.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(audit): correct the stale rustfs-uring license-allow rationale (rustfs/backlog#1181)
The dependency-review allow said rustfs-uring is "pulled as a git dependency",
but ecstore now pins it from crates.io. Update the rationale and scope the allow
to the exact pinned version (pkg:cargo/rustfs-uring@0.1.0) so a future version
bump forces a conscious re-review of the license/provenance claim instead of
being waved through on an outdated justification.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): offload io_uring driver teardown off the tokio worker (rustfs/backlog#1170)
UringBackend held Arc<UringDriver> and had no Drop, so when the last LocalDisk
reference dropped in async context (disk reconnect via renew_disk, or shutdown),
UringDriver's own Drop ran on that thread — sending Shutdown and joining each
shard thread, which can block up to the bounded-drain timeout (5s) on a hung /
D-state disk, stalling a tokio worker.
Wrap the driver in ManuallyDrop (deref is transparent, so read call sites are
unchanged) and add a Drop that takes the Arc and, when a runtime is present,
drops it on a blocking thread so the potentially-blocking join never runs on a
runtime worker. Off-runtime it drops inline.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): restore StdBackend read parity on the uring paths (rustfs/backlog#1173)
Two byte-for-byte parity breaks against StdBackend on the io_uring read paths:
- A zero-length read on an fd-cache hit returned Ok(empty) without any bounds
check, while StdBackend and the uring miss path return FileCorrupt for an
offset past EOF. Fstat the cached descriptor on the length==0 path and match.
- reclaim_read_range fadvise(DONTNEED)'d the raw unaligned [offset, offset+len)
range, but fadvise only drops fully-covered pages, so the head partial page
stayed resident — whereas StdBackend's mmap path reclaims the page-aligned
superset. Bitrot shards' 32-byte block headers keep offsets off page
boundaries, so this diverged on the common case. Page-align the reclaim window
to match the mmap path exactly.
(The third parity item from the audit — a failed reclaim fadvise failing the
read — is already parity: StdBackend's mmap path propagates the same fadvise
error with `?`, so no change is needed.)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): bound worst-case in-flight memory by chunking huge uring reads (rustfs/backlog#1174)
The driver's backpressure permits count operations, not bytes, and it zero-fills
a full-size buffer per op, so a single unbounded read could pin ~length bytes per
permit (128 permits x shards x up to ~2 GiB). ecstore passes a whole part's shard
range as one pread_bytes with no upstream chunking.
On the buffered path, split reads larger than URING_MAX_OP_LEN (128 MiB) into
sequential chunks, awaited one at a time, so worst-case in-flight memory is
bounded by permits x URING_MAX_OP_LEN per shard. The threshold is high enough
that ordinary shard reads keep the single-op, zero-copy fast path unchanged. The
O_DIRECT path (opt-in, alignment-constrained) is left for a follow-up.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): gate the io_uring fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178)
The fd cache holds up to FD_CACHE_CAPACITY (512) descriptors per disk, but
try_new cannot know the disk count and nothing checked the process fd budget. On
a bare-metal / non-systemd run with the common 1024 soft RLIMIT_NOFILE, two disks
would already exhaust fds with EMFILE surfacing on reads and probes.
Check the soft limit at try_new: enable the cache only with ample headroom
(>= 16384), otherwise log a warning once and fall back to open-per-read. The
packaged systemd unit sets 1,048,576, so tuned deployments are unaffected.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): make io_uring test skips visible and gate non-vacuity (rustfs/backlog#1179)
The ecstore io_uring tests degrade to a silent pass when io_uring is unavailable
(bare `return`s or plain eprintlns), so a CI leg on a restricted runner never
exercises the real UringBackend/FdCache/latch paths yet still goes green — an
integration regression could merge unseen.
Add uring_test_skip: it emits a grep-able `SKIP <name>` line and, when
RUSTFS_URING_TESTS_MUST_RUN is set (a CI leg that guarantees io_uring, e.g. a
seccomp=unconfined container), panics instead of skipping. Route the silent-skip
sites through it. Wiring a dedicated CI leg that sets that env on a capable
runner is tracked in the issue; this provides the enforcement mechanism.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): cover delete_paths fd-cache invalidation (rustfs/backlog#1180)
Add an end-to-end test that seeds the descriptor cache with a read, removes the
part via disk.delete_paths (one of the primary object-delete entry points that
does not go through LocalDisk::delete), and asserts the next read no longer
returns the removed inode — pinning the invalidation added in #1175. The sharded
cancel-routing half of #1180 is covered in the rustfs-uring PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* io_uring audit follow-ups: O_DIRECT chunking, inline invalidation, metrics, CI leg (backlog#1160) (#4729)
* fix(ecstore): chunk large O_DIRECT reads too, bounding in-flight memory (rustfs/backlog#1174)
The buffered read path already splits reads above URING_MAX_OP_LEN into
sequential chunks; do the same for the O_DIRECT path, which was left for a
follow-up. read_at_direct aligns each chunk's sub-range internally, and chunk
sizes are a multiple of URING_MAX_OP_LEN so a boundary re-read is at most one
block. Extract classify_direct_read_error so the single-op and chunked paths
share one copy of the EINVAL/EOPNOTSUPP-vs-subsystem latch classification rather
than duplicating it.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): invalidate cached fds on the inline rename_data rollback (rustfs/backlog#1177)
The streaming rename_data branch invalidates cached part fds on its post-commit
rollback paths, but the inline branch runs its commit and rollback inside a
single spawn_blocking closure where the async invalidate cannot be called, so it
was left to the TTL backstop.
Capture the closure's result instead of `??`-propagating it: on error (a
commit-metadata fsync failure under strict durability rolls the committed rename
back), invalidate the dst part paths at the async level before returning. Inline
objects keep their data in xl.meta rather than separate part inodes, so this is
largely defensive, but it removes the caveat and keeps the two branches
consistent.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ecstore): export io_uring latch/fallback and driver stats metrics (rustfs/backlog#1172)
Complete the gray-release observability. Beyond the warn log added earlier, emit
metrics so a dashboard can answer "how much traffic is on io_uring vs falling
back, and is any disk degrading":
- rustfs_io_uring_latch_off_total — a disk latching io_uring off at runtime.
- rustfs_io_uring_read_fallback_total — each io_uring -> StdBackend read
fallback (latched-off short-circuit, O_DIRECT error, buffered error).
- a low-frequency per-disk exporter of the driver StatsSnapshot as gauges
(in_flight, cq_overflow, cancel_already), spawned in try_new. It holds only a
Weak reference so it never keeps the driver alive, and drops any temporary
strong reference on the blocking pool so a last-reference UringDriver::Drop
join never runs on an async worker (rustfs/backlog#1170).
submit_errors is deliberately not exported yet: it is a field added in the
unreleased rustfs-uring 0.2.0, and ecstore still pins 0.1.0. It lands once the
dependency is bumped (rustfs/backlog#1181).
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci: add a real-io_uring integration leg on ubuntu-latest (rustfs/backlog#1179)
The existing self-hosted sm-standard runners cannot guarantee io_uring is
available (a container seccomp filter can block io_uring_setup), so the ecstore
uring tests degrade to a silent skip and never exercise the real
UringBackend/FdCache/latch paths in CI.
Add a job on GitHub-hosted ubuntu-latest, which runs a recent kernel with no
container seccomp filter, running the uring-named ecstore tests with
RUSTFS_IO_URING_READ_ENABLE=true and RUSTFS_URING_TESTS_MUST_RUN=1 — the
non-vacuity gate makes the leg fail rather than skip if io_uring is unavailable,
so an integration regression can no longer merge green behind a vacuous pass.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all
closed). Review of every migration-era test/gate measure found three things
actually retirable or broken — everything else is a live anti-regression
guard and stays.
Remove:
- scripts/check_metrics_migration_refs.sh — guards a migration that
finished: rustfs_metrics:: has zero hits, the metrics crate no longer
exists, and the script was never wired into CI or make (only reference
was one line in config-model-boundary-adr.md, also removed).
- crates/obs init_metrics_collectors — the "backward-compatible alias kept
during migration" the removed script was guarding. Zero callers; pure
delegate to init_metrics_runtime.
Archive (docs/superpowers/plans/, continuing the 2026-07 convention,
with the standard archived banner):
- startup-timeline.md, scheduler-baseline.md,
profiling-numa-capability-inventory.md,
kms-development-defaults-inventory.md — one-shot snapshots whose only
consumer is the already-archived migration-progress ledger (their
same-dir links there start resolving again after the move); zero script
pins; fed the closed backlog#660/#665 architecture-review ledger.
Fixed the one outbound link (startup-timeline -> readiness-matrix) that
the move would have broken — check_doc_paths.sh deliberately does not
scan plans/, so nothing else would have caught it.
Wire (found orphaned by the same review):
- scripts/check_extension_schema_boundaries.sh guards a live contract
crate but was never invoked anywhere. Add lint-fmt.mak target, include
in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job
already installs ripgrep), sync the CONTRIBUTING.md enumerated list,
and harden the script against a silently-passing rg probe when src/
is missing.
Keep (verified live, documented so the next cleanup pass does not repeat
this analysis):
- scripts/check_architecture_migration_rules.sh — added a header stating
it is a permanent boundary guard, not retirable migration scaffolding;
'migration' in the name is historical.
- check_migration_gate_count.sh + floor, delete-marker e2e proof, all
pinned docs, compat-cleanup-register sync, remaining inventories
(referenced by live docs).
Verification: all 7 guard scripts pass, actionlint clean,
cargo check --workspace (excl e2e) clean, cargo fmt --check clean.
Adversarially reviewed by two independent skeptic passes; their 7
findings (alias left behind, broken outbound link, missing banners,
wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing
header rationale) are all folded in.
* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)
The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.
These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.
First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
- test_lifecycle_expiration (prefix Days=1/Days=5, scanner delete)
- test_lifecyclev2_expiration (same via ListObjectsV2)
- test_lifecycle_noncur_expiration (NoncurrentVersionExpiration)
- test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
- test_lifecycle_multipart_expiration (AbortIncompleteMultipartUpload)
Dropped from the batch, kept excluded with reason:
- test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
while AWS/this test expect InvalidArgument. Real validation gap.
Changes:
- scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
- scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
can point at an alternate whitelist.
- .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
no start delay) and a 2s stale-multipart cleanup interval, running the new
list serially.
- scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
scope=all sweep (plain server) does not misclassify expected failures.
- scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
- docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
excluded 274, behavior 5) and document the lane.
Refs backlog#1148 (ilm-10), master plan backlog#1155.
* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
ci: run e2e smoke subset via nextest e2e-smoke profile (backlog#1149 ci-4)
The e2e-tests job previously ran a single test (delete_marker_migration
_semantics) out of ~400 in the e2e_test crate. Define a nextest
profile.e2e-smoke whose default-filter selects 17 fast single-node
dependency-free modules (63 tests) and run it in the job, reusing the
downloaded debug binary. The profile is the single wiring mechanism for
e2e tests in CI; admission criteria live in crates/e2e_test/README.md,
and docs/testing/e2e-suite-inventory.md records authoritative per-module
counts from cargo nextest list.
Implements ilm-1 from the ILM test-strategy plan: the 33 #[ignore]d ILM
integration tests (14 in crates/scanner/tests/lifecycle_integration_test.rs,
19 in rustfs/src/app/lifecycle_transition_api_test.rs) never ran anywhere.
This adds a dedicated serialized CI job and repairs the app-layer suite,
activating 29 of them.
- ci.yml: new test-ilm-integration-serial job running the two ILM test
surfaces with 'cargo nextest run -j1 --run-ignored ignored-only'. The
tests share process-global singletons and fixed ports (9002/9003);
serial_test's #[serial] does not serialize across nextest's
process-per-test boundary, so -j1 is the serialization mechanism.
- app suite repair: the 19 app tests rotted after the InstanceContext
migration (usecases resolve the store via AppContext; the test env never
installed one, so every store-touching call failed with InternalError
'Not init'). Add a test-only install_test_app_context helper behind the
sanctioned context/runtime_sources boundaries and switch the tests from
without_context() to from_global(). All 19 now pass.
- delete test_lifecycle_transition_basic (+3 orphaned helpers): required
an external MinIO at localhost:9000 and slept 1200s; superseded by the
mock-tier tests in the same file.
- fix a timing flake: poll free-version metadata removal instead of
asserting a single read right after remote-object absence.
- 3 scanner tests fail on main for product reasons (multipart restore
UnexpectedEof; noncurrent transition/expiry after immediate compensation
transition on versioned buckets); they keep #[ignore] with a backlog
reference and are excluded from the lane by name.
Lane: 29 tests, ~43s test time, green twice locally.
Refs rustfs/backlog#1148 (ilm-1), rustfs/backlog#1155.
The migration-proof step in ci.yml selects tests by name substring
(data_movement / rebalance / decommission / source_cleanup /
delete_marker), so a rename can silently thin the gate to zero with no
CI signal. Move the filter expression into
scripts/check_migration_gate_count.sh as its single source of truth:
the script counts the selected tests with cargo nextest list, fails if
the count drops below the committed floor in
.config/migration-gate-floor.txt, and then runs the gate with the same
filter so the check and the run cannot drift.
The floor equals the exact selected-test count at landing (571), so any
reduction fails CI and intentional shrinks must edit the floor file in
the same PR, keeping gate changes visible in diffs.
Refs rustfs/backlog#1153 (infra-12), rustfs/backlog#1155.
Signed-off-by: Zhengchao An <anzhengchao@gmail.com>
test(ci): add strict nextest ci profile with quarantine + flake policy
Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate
(ci-10, absorbs infra-15; backlog#1149).
- .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a
new race's first occurrence), fail-fast=false, and JUnit output at
target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get
retries=2 under the ci profile only; each entry links one OPEN issue. First
members are the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*), which keep their existing
ecstore-serial-flaky test-group serialization. Local default profile still
never retries.
- .github/workflows/ci.yml: run the main test step with --profile ci and upload
the JUnit report (if: always(), 3-day retention, run-number in name). The
migration-proof step stays on the default profile to avoid clobbering the ci
JUnit artifact (its tests are not quarantined).
- docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding
the flake policy: discover -> open issue within 24h -> quarantine with issue
link -> fix or delete within 30 days. AGENTS.md points to it.
Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
- build.yml: update latest.json only for stable release tags
(alpha/beta/rc tags previously overwrote the stable pointer with
release_type "stable"); drop the placeholder .asc files; build the
Linux targets only on main pushes (tags/schedule/dispatch keep the
full matrix); remove the unreachable --build clause and a needless
full-history clone
- ci.yml: event-scoped concurrency so merges stop cancelling the
weekly scheduled run; drop the redundant skip-duplicate-actions
gate job; run clippy before tests; trim the unused toolchain from
the typos job
- ci-docs-only.yml (new): satisfy the required "Test and Lint" check
on docs-only PRs that ci.yml skips via paths-ignore
- audit.yml: drop cargo-audit (cargo-deny advisories covers the same
RustSec database); same event-scoped concurrency fix
- docker.yml: job-level short-circuit for per-merge dev builds; send
the Trivy SARIF to code scanning; unify the upload-artifact pin
- performance-ab.yml: add concurrency; only re-run on labeled events
when the added label is perf-ab
- stagger the Sunday crons (build 01:00, audit 03:00,
nix-flake-update 05:00, mint 06:00)
- delete performance.yml: disabled since 2025-07; idle-server
profiling, no benchmark baseline, stale ecstore package name
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:
- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
449-test PR gate is reproducible; previously every run cloned upstream
master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
CI round reports every failure in parallel instead of stopping at the
first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
report_compat.py to diff junit results against the classification
lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
of truth; also fixes the broken config generation that left S3_PORT
empty), add a weekly scheduled full sweep that fails only on whitelist
regressions, and fix the multi-node topology to a real distributed
cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
update scripts/s3-tests/README.md, fix dead build_testexpr.sh
reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.
All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Speeds up the CI pipeline without changing what is tested:
- Drop the global CARGO_BUILD_JOBS=2 and per-job --jobs 2 flags, which
halved compiler parallelism on the 4-core runners.
- Stop embedding hashFiles(Cargo.lock) in rust-cache shared keys. The
action already fingerprints lockfiles internally; putting the hash in
the key prefix meant any PR that touched Cargo.lock started from a
completely cold cache instead of reusing unchanged dependencies.
- Give the e2e-tests job the full setup action with dependency caching.
Its migration-proof step compiles the e2e_test crate (which pulls in
most of the workspace) and previously did so cold on every run with
no cache, on a 2-core runner.
- Set profile.dev debug = "line-tables-only": keeps usable backtraces
while cutting compile/link time, target-dir size (faster cache
save/restore), and debug-binary artifact upload/download.
- Split fmt + repo-script checks into a compile-free quick-checks job
on ubuntu-latest so contributors get feedback in ~1 minute instead
of after the full test run.
- Collapse the five per-filter migration-proof cargo test reruns into
one filtered nextest invocation; the same tests already run in the
full nextest pass.
- Remove the touch rustfs/build.rs + forced rebuild from the debug
binary jobs (version stamping is irrelevant for e2e binaries) and
the unreachable Windows cross-compile branch in build.yml.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An