ci(ilm): add gated s3-tests lane for lifecycle expiration cases (#4715)

* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)

The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.

These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.

First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
  - test_lifecycle_expiration          (prefix Days=1/Days=5, scanner delete)
  - test_lifecyclev2_expiration        (same via ListObjectsV2)
  - test_lifecycle_noncur_expiration   (NoncurrentVersionExpiration)
  - test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
  - test_lifecycle_multipart_expiration    (AbortIncompleteMultipartUpload)

Dropped from the batch, kept excluded with reason:
  - test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
    200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
    while AWS/this test expect InvalidArgument. Real validation gap.

Changes:
  - scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
  - scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
    can point at an alternate whitelist.
  - .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
    starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
    no start delay) and a 2s stale-multipart cleanup interval, running the new
    list serially.
  - scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
    scope=all sweep (plain server) does not misclassify expected failures.
  - scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
    the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
  - docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
    excluded 274, behavior 5) and document the lane.

Refs backlog#1148 (ilm-10), master plan backlog#1155.

* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)

scripts/s3-tests/excluded_tests.txt

@@ -157,26 +157,47 @@ test_encryption_sse_c_multipart_invalid_chunks_1
 test_encryption_sse_c_multipart_invalid_chunks_2
 test_get_multipart_checksum_object_attributes
 test_head_bucket_usage
-test_lifecycle_cloud_multiple_transition
-test_lifecycle_cloud_transition
-test_lifecycle_cloud_transition_large_obj
-test_lifecycle_deletemarker_expiration
+# -----------------------------------------------------------------------------
+# Lifecycle behavior (backlog#1148 ilm-10, master plan #1155)
+#
+# The upstream "Vendor-specific" label on these was inaccurate: the real blocker
+# was the absence of a Ceph `lc_debug_interval` equivalent, so Days>=1 expiration
+# behavior could not be exercised in seconds. RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5)
+# now provides that. The first batch of expiration tests has been moved into the
+# dedicated behavior lane: scripts/s3-tests/lifecycle_behavior_tests.txt
+# (run by the ci.yml job `s3-lifecycle-behavior-tests`). The entries below are
+# NOT vendor-specific — they stay excluded from the DEFAULT single-server gate
+# for the specific reasons noted, and are queued for later batches.
+#
+# --- Batch 2 candidates: expiration with tag / size filters + noncurrent counts.
+#     These need the same debug/scanner lane as batch 1 plus tag/size-filter and
+#     NewerNoncurrentVersions coverage (see ilm-9). Queued for ilm-10 batch 2.
 test_lifecycle_deletemarker_expiration_with_days_tag
-test_lifecycle_expiration
-test_lifecycle_expiration_days0
 test_lifecycle_expiration_newer_noncurrent
 test_lifecycle_expiration_noncur_tags1
 test_lifecycle_expiration_size_gt
 test_lifecycle_expiration_size_lt
 test_lifecycle_expiration_tags2
 test_lifecycle_expiration_versioned_tags2
-test_lifecycle_multipart_expiration
-test_lifecycle_noncur_cloud_transition
-test_lifecycle_noncur_expiration
+#
+# --- Storage-class transition to a remote/warm tier. Needs a real tier backend
+#     (the hermetic second-RustFS cold layer from ilm-7) which is not wired into
+#     the s3-tests topology. Queued behind ilm-7.
 test_lifecycle_noncur_transition
 test_lifecycle_transition
+#
+# --- Single-rule multi-level Transition (hot->warm->cold). RustFS only honors
+#     transitions.first() (core.rs:676); multi-level semantics are undecided and
+#     tracked as a product item (ex-ilm-20). Do not enable until that lands.
 test_lifecycle_transition_single_rule_multi_trans
-test_lifecyclev2_expiration
+#
+# --- Cloud (S3/remote) transition + restore. Requires the [s3 cloud] endpoint
+#     the RustFS SNSD harness does not provision; depends on ilm-7 tier carrier.
+test_lifecycle_cloud_multiple_transition
+test_lifecycle_cloud_transition
+test_lifecycle_cloud_transition_large_obj
+test_lifecycle_noncur_cloud_transition
+# -----------------------------------------------------------------------------
 test_list_buckets_anonymous
 test_list_buckets_paginated
 test_list_multipart_upload

scripts/s3-tests/implemented_tests.txt

@@ -225,6 +225,8 @@ test_delete_bucket_encryption_kms
 test_delete_bucket_encryption_s3
 # Lifecycle tests
 test_lifecycle_delete
+# Negative validation: Expiration{Days:0} must return InvalidArgument (backlog#1148 ilm-10).
+test_lifecycle_expiration_days0
 test_lifecycle_get
 test_lifecycle_get_no_id
 test_lifecycle_id_too_long

scripts/s3-tests/lifecycle_behavior_tests.txt

@@ -0,0 +1,40 @@
+# Lifecycle behavior tests (ceph/s3-tests) — backlog#1148 ilm-10, master plan #1155
+# =================================================================================
+#
+# These upstream lifecycle tests exercise *real expiration behavior*: they PUT a
+# lifecycle rule with Days/NoncurrentDays/DaysAfterInitiation, sleep a multiple of
+# the debug interval, then assert objects/versions/uploads were actually removed by
+# the background scanner and the stale-multipart cleanup loop.
+#
+# They CANNOT run in the default single-server gate (implemented_tests.txt) because:
+#   1. The default harness disables the background scanner (run.sh sets
+#      RUSTFS_SCANNER_ENABLED=false) — nothing expires.
+#   2. A real day is 86400s; without acceleration a Days=1 rule takes 24h.
+#   3. Enabling RUSTFS_ILM_DEBUG_DAY_SECS *globally* would break the already-passing
+#      x-amz-expiration header tests (test_lifecycle_expiration_header_*), which
+#      assert `(expiry-date - now).days == Days` — the header is computed from the
+#      same debug-scaled expected_expiry_time().
+#
+# So this list is run by a DEDICATED lane (ci.yml job `s3-lifecycle-behavior-tests`)
+# against a server started with:
+#   RUSTFS_ILM_DEBUG_DAY_SECS=10   (1 "day" = 10s, matches s3-tests lc_debug_interval)
+#   RUSTFS_SCANNER_ENABLED=true
+#   RUSTFS_SCANNER_CYCLE=2         (scan every 2s, well under a debug-day)
+#   RUSTFS_SCANNER_START_DELAY_SECS=0
+#   RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL=2s  (multipart abort lane)
+#
+# Point run.sh at this file with IMPLEMENTED_TESTS_FILE=<this path>.
+#
+# First batch (ilm-10). Each entry was verified against the pinned upstream source
+# and the RustFS evaluator/scanner/stale-multipart execution paths.
+
+# Basic prefix expiration: Days=1 / Days=5 rules, scanner-driven deletion.
+test_lifecycle_expiration
+# Same as above but validates ListObjectsV2 sees the expirations.
+test_lifecyclev2_expiration
+# Versioned NoncurrentVersionExpiration (NoncurrentDays): noncurrent versions removed.
+test_lifecycle_noncur_expiration
+# ExpiredObjectDeleteMarker + NoncurrentVersionExpiration cascade on a versioned bucket.
+test_lifecycle_deletemarker_expiration
+# AbortIncompleteMultipartUpload (DaysAfterInitiation): stale-multipart cleanup loop.
+test_lifecycle_multipart_expiration

scripts/s3-tests/report_compat.py

@@ -36,6 +36,12 @@ LIST_FILES = {
     "implemented": "implemented_tests.txt",
     "unimplemented": "unimplemented_tests.txt",
     "excluded": "excluded_tests.txt",
+    # Lifecycle behavior lane (backlog#1148 ilm-10): gated by its own ci.yml job
+    # against a debug-accelerated server. In the full `scope=all` sweep these run
+    # against the plain server (no scanner / no RUSTFS_ILM_DEBUG_DAY_SECS) and are
+    # EXPECTED to fail there, so a failure here is neither a regression nor an
+    # unclassified failure.
+    "behavior": "lifecycle_behavior_tests.txt",
 }
 
 
@@ -122,6 +128,7 @@ def main() -> int:
     promotions: dict[str, list[str]] = {"unimplemented": [], "excluded": []}
     unclassified_passed: list[str] = []
     unclassified_failed: list[str] = []
+    behavior_passed: list[str] = []
     counts = {"passed": 0, "failed": 0, "error": 0, "skipped": 0}
 
     for name, status in results.items():
@@ -129,13 +136,19 @@ def main() -> int:
         if status in ("failed", "error"):
             if name in lists["implemented"]:
                 regressions.append(name)
-            elif name not in lists["unimplemented"] and name not in lists["excluded"]:
+            elif (
+                name not in lists["unimplemented"]
+                and name not in lists["excluded"]
+                and name not in lists["behavior"]
+            ):
                 unclassified_failed.append(name)
         elif status == "passed":
             if name in lists["unimplemented"]:
                 promotions["unimplemented"].append(name)
             elif name in lists["excluded"]:
                 promotions["excluded"].append(name)
+            elif name in lists["behavior"]:
+                behavior_passed.append(name)
             elif name not in lists["implemented"]:
                 unclassified_passed.append(name)
 
@@ -162,6 +175,12 @@ def main() -> int:
         promotions["excluded"],
         "Passing despite being excluded — re-evaluate the exclusion.",
     )
+    lines += render_section(
+        "Lifecycle behavior lane (passing in this run)",
+        behavior_passed,
+        "Gated by the dedicated `s3-lifecycle-behavior-tests` lane; expected to "
+        "fail in the plain `scope=all` sweep (no debug day / scanner).",
+    )
     lines += render_section(
         "Unclassified passes",
         unclassified_passed,

scripts/s3-tests/run.sh

@@ -159,7 +159,10 @@ PY
 
 # Test list files location
 TEST_LISTS_DIR="${SCRIPT_DIR}"
-IMPLEMENTED_TESTS_FILE="${TEST_LISTS_DIR}/implemented_tests.txt"
+# IMPLEMENTED_TESTS_FILE may be overridden to run an alternate exact-node-id
+# whitelist (e.g. lifecycle_behavior_tests.txt for the dedicated lifecycle
+# behavior lane, which needs RUSTFS_ILM_DEBUG_DAY_SECS + the scanner enabled).
+IMPLEMENTED_TESTS_FILE="${IMPLEMENTED_TESTS_FILE:-${TEST_LISTS_DIR}/implemented_tests.txt}"
 UNIMPLEMENTED_TESTS_FILE="${TEST_LISTS_DIR}/unimplemented_tests.txt"
 EXCLUDED_TESTS_FILE="${TEST_LISTS_DIR}/excluded_tests.txt"
 S3_TEST_FILE="s3tests/functional/test_s3.py"