Follow-up to #4772. After forcing every-cycle ILM evaluation the Days=1
plateau is reliable, but test_lifecycle_expiration / test_lifecyclev2_expiration
still flaked intermittently on the *second* assertion (`assert 4 == 2`):
the Days=5 (expire3) objects were not expired within their poll window.
Cause: _wait_for_lifecycle_count for expire3 starts its N*lc_interval
deadline only after the Days=1 poll returns (~1 debug-day in). With N=5
and lc_interval == debug_day, the 5*debug_day terms cancel and the slack
past the Days=5 due time is only ~1 debug-day (~9s) -- which a single slow
scanner cycle (observed ~13s spacing under CI load) can exceed, leaving the
count stalled at 4. Bumping the two expire3 windows to 8*lc_interval raises
the slack to ~39s, comfortably above the observed cycle jitter.
Test-only, lane-scoped: does not touch RUSTFS_ILM_DEBUG_DAY_SECS or the
4*lc_interval < 5*debug_day plateau invariant. The expire3 target count (2)
is terminal (nothing expires after it), so a wider window can never
over-expire. Also documents the #4772 RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1
knob in lifecycle_behavior_tests.txt.
* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)
The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.
These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.
First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
- test_lifecycle_expiration (prefix Days=1/Days=5, scanner delete)
- test_lifecyclev2_expiration (same via ListObjectsV2)
- test_lifecycle_noncur_expiration (NoncurrentVersionExpiration)
- test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
- test_lifecycle_multipart_expiration (AbortIncompleteMultipartUpload)
Dropped from the batch, kept excluded with reason:
- test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
while AWS/this test expect InvalidArgument. Real validation gap.
Changes:
- scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
- scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
can point at an alternate whitelist.
- .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
no start delay) and a 2s stale-multipart cleanup interval, running the new
list serially.
- scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
scope=all sweep (plain server) does not misclassify expected failures.
- scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
- docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
excluded 274, behavior 5) and document the lane.
Refs backlog#1148 (ilm-10), master plan backlog#1155.
* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
Follow-ups to the compatibility harness rework:
- Weekly full sweep now runs as a matrix over both topologies: single
node and the 4-node distributed cluster. Manual dispatch keeps the
test-mode input.
- New mint workflow (weekly + dispatch) runs MinIO Mint against RustFS:
functional suites of real client SDKs and tools (awscli, mc,
aws-sdk-*, minio-*, s3cmd, ...), catching client-specific signing and
streaming edge cases that boto3-only ceph/s3-tests cannot. Report-only
for test failures with a per-suite summary table; fails only when mint
produces no results.
- New scripts/s3-tests/api_coverage.py quantifies API surface coverage
by diffing the s3s S3 trait (at the Cargo.toml pinned revision)
against the methods overridden in impl S3 for FS, distinguishing
NotImplemented defaults from delegating defaults (e.g. post_object).
Current state: 76/101 operations covered (75 overridden, 1 delegated).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:
- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
449-test PR gate is reproducible; previously every run cloned upstream
master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
CI round reports every failure in parallel instead of stopping at the
first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
report_compat.py to diff junit results against the classification
lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
of truth; also fixes the broken config generation that left S3_PORT
empty), add a weekly scheduled full sweep that fails only on whitelist
regressions, and fix the multi-node topology to a real distributed
cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
update scripts/s3-tests/README.md, fix dead build_testexpr.sh
reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.
All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: harden local SSE-S3 fallback
* test: update managed SSE-S3 assumptions
* chore: keep issue plan local only
* test(ci): seed local SSE key for s3-tests
Zhengchao An