mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
Zhengchao An
cf142e7fdd
Follow-up to #4772. After forcing every-cycle ILM evaluation the Days=1 plateau is reliable, but test_lifecycle_expiration / test_lifecyclev2_expiration still flaked intermittently on the *second* assertion (`assert 4 == 2`): the Days=5 (expire3) objects were not expired within their poll window. Cause: _wait_for_lifecycle_count for expire3 starts its N*lc_interval deadline only after the Days=1 poll returns (~1 debug-day in). With N=5 and lc_interval == debug_day, the 5*debug_day terms cancel and the slack past the Days=5 due time is only ~1 debug-day (~9s) -- which a single slow scanner cycle (observed ~13s spacing under CI load) can exceed, leaving the count stalled at 4. Bumping the two expire3 windows to 8*lc_interval raises the slack to ~39s, comfortably above the observed cycle jitter. Test-only, lane-scoped: does not touch RUSTFS_ILM_DEBUG_DAY_SECS or the 4*lc_interval < 5*debug_day plateau invariant. The expire3 target count (2) is terminal (nothing expires after it), so a wider window can never over-expire. Also documents the #4772 RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 knob in lifecycle_behavior_tests.txt.
54 lines
3.2 KiB
Plaintext
54 lines
3.2 KiB
Plaintext
# Lifecycle behavior tests (ceph/s3-tests) — backlog#1148 ilm-10, master plan #1155
|
|
# =================================================================================
|
|
#
|
|
# These upstream lifecycle tests exercise *real expiration behavior*: they PUT a
|
|
# lifecycle rule with Days/NoncurrentDays/DaysAfterInitiation, sleep a multiple of
|
|
# the debug interval, then assert objects/versions/uploads were actually removed by
|
|
# the background scanner and the stale-multipart cleanup loop.
|
|
#
|
|
# They CANNOT run in the default single-server gate (implemented_tests.txt) because:
|
|
# 1. The default harness disables the background scanner (run.sh sets
|
|
# RUSTFS_SCANNER_ENABLED=false) — nothing expires.
|
|
# 2. A real day is 86400s; without acceleration a Days=1 rule takes 24h.
|
|
# 3. Enabling RUSTFS_ILM_DEBUG_DAY_SECS *globally* would break the already-passing
|
|
# x-amz-expiration header tests (test_lifecycle_expiration_header_*), which
|
|
# assert `(expiry-date - now).days == Days` — the header is computed from the
|
|
# same debug-scaled expected_expiry_time().
|
|
#
|
|
# So this list is run by a DEDICATED lane (ci.yml job `s3-lifecycle-behavior-tests`)
|
|
# against a server started with:
|
|
# RUSTFS_ILM_DEBUG_DAY_SECS=10 (1 "day" = 10s, matches s3-tests lc_debug_interval)
|
|
# RUSTFS_SCANNER_ENABLED=true
|
|
# RUSTFS_SCANNER_CYCLE=2 (scan every 2s, well under a debug-day)
|
|
# RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 (re-descend compacted dirs every cycle so
|
|
# ILM is evaluated ~every 2s, not once per 16 cycles;
|
|
# without this a Days=1 object due at 10s is not
|
|
# expired until the next ~32s boundary — #4772)
|
|
# RUSTFS_SCANNER_START_DELAY_SECS=0
|
|
# RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL=2s (multipart abort lane)
|
|
#
|
|
# Polling windows (patches/0002): each assertion polls list_objects until the
|
|
# expected count is reached or a N*lc_interval deadline elapses. The Days=5
|
|
# (expire3) window uses 8*lc_interval rather than 5: its deadline only starts
|
|
# after the Days=1 poll returns (~1 debug-day in), so a 5*lc_interval window left
|
|
# only ~1 debug-day of slack past the Days=5 due time — a single slow scanner
|
|
# cycle (observed ~13s under CI load) could push the expiry just past it and
|
|
# stall the count at 4 (assert 4 == 2). 8*lc_interval restores comfortable slack;
|
|
# the target count (2) is terminal so a wider window never over-expires.
|
|
#
|
|
# Point run.sh at this file with IMPLEMENTED_TESTS_FILE=<this path>.
|
|
#
|
|
# First batch (ilm-10). Each entry was verified against the pinned upstream source
|
|
# and the RustFS evaluator/scanner/stale-multipart execution paths.
|
|
|
|
# Basic prefix expiration: Days=1 / Days=5 rules, scanner-driven deletion.
|
|
test_lifecycle_expiration
|
|
# Same as above but validates ListObjectsV2 sees the expirations.
|
|
test_lifecyclev2_expiration
|
|
# Versioned NoncurrentVersionExpiration (NoncurrentDays): noncurrent versions removed.
|
|
test_lifecycle_noncur_expiration
|
|
# ExpiredObjectDeleteMarker + NoncurrentVersionExpiration cascade on a versioned bucket.
|
|
test_lifecycle_deletemarker_expiration
|
|
# AbortIncompleteMultipartUpload (DaysAfterInitiation): stale-multipart cleanup loop.
|
|
test_lifecycle_multipart_expiration
|