* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)
The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.
These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.
First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
- test_lifecycle_expiration (prefix Days=1/Days=5, scanner delete)
- test_lifecyclev2_expiration (same via ListObjectsV2)
- test_lifecycle_noncur_expiration (NoncurrentVersionExpiration)
- test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
- test_lifecycle_multipart_expiration (AbortIncompleteMultipartUpload)
Dropped from the batch, kept excluded with reason:
- test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
while AWS/this test expect InvalidArgument. Real validation gap.
Changes:
- scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
- scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
can point at an alternate whitelist.
- .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
no start delay) and a 2s stale-multipart cleanup interval, running the new
list serially.
- scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
scope=all sweep (plain server) does not misclassify expected failures.
- scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
- docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
excluded 274, behavior 5) and document the lane.
Refs backlog#1148 (ilm-10), master plan backlog#1155.
* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
ci: run e2e smoke subset via nextest e2e-smoke profile (backlog#1149 ci-4)
The e2e-tests job previously ran a single test (delete_marker_migration
_semantics) out of ~400 in the e2e_test crate. Define a nextest
profile.e2e-smoke whose default-filter selects 17 fast single-node
dependency-free modules (63 tests) and run it in the job, reusing the
downloaded debug binary. The profile is the single wiring mechanism for
e2e tests in CI; admission criteria live in crates/e2e_test/README.md,
and docs/testing/e2e-suite-inventory.md records authoritative per-module
counts from cargo nextest list.
Implements ilm-1 from the ILM test-strategy plan: the 33 #[ignore]d ILM
integration tests (14 in crates/scanner/tests/lifecycle_integration_test.rs,
19 in rustfs/src/app/lifecycle_transition_api_test.rs) never ran anywhere.
This adds a dedicated serialized CI job and repairs the app-layer suite,
activating 29 of them.
- ci.yml: new test-ilm-integration-serial job running the two ILM test
surfaces with 'cargo nextest run -j1 --run-ignored ignored-only'. The
tests share process-global singletons and fixed ports (9002/9003);
serial_test's #[serial] does not serialize across nextest's
process-per-test boundary, so -j1 is the serialization mechanism.
- app suite repair: the 19 app tests rotted after the InstanceContext
migration (usecases resolve the store via AppContext; the test env never
installed one, so every store-touching call failed with InternalError
'Not init'). Add a test-only install_test_app_context helper behind the
sanctioned context/runtime_sources boundaries and switch the tests from
without_context() to from_global(). All 19 now pass.
- delete test_lifecycle_transition_basic (+3 orphaned helpers): required
an external MinIO at localhost:9000 and slept 1200s; superseded by the
mock-tier tests in the same file.
- fix a timing flake: poll free-version metadata removal instead of
asserting a single read right after remote-object absence.
- 3 scanner tests fail on main for product reasons (multipart restore
UnexpectedEof; noncurrent transition/expiry after immediate compensation
transition on versioned buckets); they keep #[ignore] with a backlog
reference and are excluded from the lane by name.
Lane: 29 tests, ~43s test time, green twice locally.
Refs rustfs/backlog#1148 (ilm-1), rustfs/backlog#1155.
The migration-proof step in ci.yml selects tests by name substring
(data_movement / rebalance / decommission / source_cleanup /
delete_marker), so a rename can silently thin the gate to zero with no
CI signal. Move the filter expression into
scripts/check_migration_gate_count.sh as its single source of truth:
the script counts the selected tests with cargo nextest list, fails if
the count drops below the committed floor in
.config/migration-gate-floor.txt, and then runs the gate with the same
filter so the check and the run cannot drift.
The floor equals the exact selected-test count at landing (571), so any
reduction fails CI and intentional shrinks must edit the floor file in
the same PR, keeping gate changes visible in diffs.
Refs rustfs/backlog#1153 (infra-12), rustfs/backlog#1155.
Signed-off-by: Zhengchao An <anzhengchao@gmail.com>
test(ci): add strict nextest ci profile with quarantine + flake policy
Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate
(ci-10, absorbs infra-15; backlog#1149).
- .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a
new race's first occurrence), fail-fast=false, and JUnit output at
target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get
retries=2 under the ci profile only; each entry links one OPEN issue. First
members are the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*), which keep their existing
ecstore-serial-flaky test-group serialization. Local default profile still
never retries.
- .github/workflows/ci.yml: run the main test step with --profile ci and upload
the JUnit report (if: always(), 3-day retention, run-number in name). The
migration-proof step stays on the default profile to avoid clobbering the ci
JUnit artifact (its tests are not quarantined).
- docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding
the flake policy: discover -> open issue within 24h -> quarantine with issue
link -> fix or delete within 30 days. AGENTS.md points to it.
Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
- build.yml: update latest.json only for stable release tags
(alpha/beta/rc tags previously overwrote the stable pointer with
release_type "stable"); drop the placeholder .asc files; build the
Linux targets only on main pushes (tags/schedule/dispatch keep the
full matrix); remove the unreachable --build clause and a needless
full-history clone
- ci.yml: event-scoped concurrency so merges stop cancelling the
weekly scheduled run; drop the redundant skip-duplicate-actions
gate job; run clippy before tests; trim the unused toolchain from
the typos job
- ci-docs-only.yml (new): satisfy the required "Test and Lint" check
on docs-only PRs that ci.yml skips via paths-ignore
- audit.yml: drop cargo-audit (cargo-deny advisories covers the same
RustSec database); same event-scoped concurrency fix
- docker.yml: job-level short-circuit for per-merge dev builds; send
the Trivy SARIF to code scanning; unify the upload-artifact pin
- performance-ab.yml: add concurrency; only re-run on labeled events
when the added label is perf-ab
- stagger the Sunday crons (build 01:00, audit 03:00,
nix-flake-update 05:00, mint 06:00)
- delete performance.yml: disabled since 2025-07; idle-server
profiling, no benchmark baseline, stale ecstore package name
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:
- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
449-test PR gate is reproducible; previously every run cloned upstream
master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
CI round reports every failure in parallel instead of stopping at the
first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
report_compat.py to diff junit results against the classification
lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
of truth; also fixes the broken config generation that left S3_PORT
empty), add a weekly scheduled full sweep that fails only on whitelist
regressions, and fix the multi-node topology to a real distributed
cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
update scripts/s3-tests/README.md, fix dead build_testexpr.sh
reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.
All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Speeds up the CI pipeline without changing what is tested:
- Drop the global CARGO_BUILD_JOBS=2 and per-job --jobs 2 flags, which
halved compiler parallelism on the 4-core runners.
- Stop embedding hashFiles(Cargo.lock) in rust-cache shared keys. The
action already fingerprints lockfiles internally; putting the hash in
the key prefix meant any PR that touched Cargo.lock started from a
completely cold cache instead of reusing unchanged dependencies.
- Give the e2e-tests job the full setup action with dependency caching.
Its migration-proof step compiles the e2e_test crate (which pulls in
most of the workspace) and previously did so cold on every run with
no cache, on a 2-core runner.
- Set profile.dev debug = "line-tables-only": keeps usable backtraces
while cutting compile/link time, target-dir size (faster cache
save/restore), and debug-binary artifact upload/download.
- Split fmt + repo-script checks into a compile-free quick-checks job
on ubuntu-latest so contributors get feedback in ~1 minute instead
of after the full test run.
- Collapse the five per-filter migration-proof cargo test reruns into
one filtered nextest invocation; the same tests already run in the
full nextest pass.
- Remove the touch rustfs/build.rs + forced rebuild from the debug
binary jobs (version stamping is irrelevant for e2e binaries) and
the unreachable Windows cross-compile branch in build.yml.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An