gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci(ilm): add gated s3-tests lane for lifecycle expiration cases (#4715)

Browse Source 
* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)

The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.

These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.

First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
  - test_lifecycle_expiration          (prefix Days=1/Days=5, scanner delete)
  - test_lifecyclev2_expiration        (same via ListObjectsV2)
  - test_lifecycle_noncur_expiration   (NoncurrentVersionExpiration)
  - test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
  - test_lifecycle_multipart_expiration    (AbortIncompleteMultipartUpload)

Dropped from the batch, kept excluded with reason:
  - test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
    200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
    while AWS/this test expect InvalidArgument. Real validation gap.

Changes:
  - scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
  - scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
    can point at an alternate whitelist.
  - .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
    starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
    no start delay) and a 2s stale-multipart cleanup interval, running the new
    list serially.
  - scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
    scope=all sweep (plain server) does not misclassify expected failures.
  - scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
    the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
  - docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
    excluded 274, behavior 5) and document the lane.

Refs backlog#1148 (ilm-10), master plan backlog#1155.

* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
This commit is contained in:
Zhengchao An
2026-07-11 13:16:55 +08:00
committed by GitHub
parent 4de73c0653
commit 4b83efaf36
10 changed files with 314 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/lifecycle/src/core.rs
+76 -15
View File
@@ -40,10 +40,11 @@ const _ERR_XML_NOT_WELL_FORMED: &str =
const ERR_LIFECYCLE_BUCKET_LOCKED: &str =
"ExpiredObjectAllVersions element and DelMarkerExpiration action cannot be used on an object locked bucket";
const ERR_LIFECYCLE_TOO_MANY_RULES: &str = "Lifecycle configuration should have at most 1000 rules";
const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "Lifecycle expiration days must not be negative";
const ERR_LIFECYCLE_INVALID_NONCURRENT_EXPIRATION_DAYS: &str = "Lifecycle noncurrent expiration days must not be negative";
const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "'Days' for Expiration action must be a positive integer";
const ERR_LIFECYCLE_INVALID_NONCURRENT_EXPIRATION_DAYS: &str =
"'NoncurrentDays' for NoncurrentVersionExpiration action must be a positive integer";
const ERR_LIFECYCLE_INVALID_ABORT_INCOMPLETE_MPU_DAYS: &str =
"DaysAfterInitiation must be 0 or greater when used with AbortIncompleteMultipartUpload";
"'DaysAfterInitiation' for AbortIncompleteMultipartUpload action must be a positive integer";
const ERR_LIFECYCLE_INVALID_EXPIRATION_DATE_NOT_MIDNIGHT: &str = "Expiration.Date must be at midnight UTC";
const ERR_LIFECYCLE_INVALID_EXPIRED_OBJECT_DELETE_MARKER: &str =
"ExpiredObjectDeleteMarker cannot be specified with Days or Date";
@@ -325,21 +326,29 @@ impl Lifecycle for BucketLifecycleConfiguration {
return Err(std::io::Error::other(ERR_LIFECYCLE_INVALID_EXPIRATION_DATE_NOT_MIDNIGHT));
}
}
// S3 requires Expiration.Days to be a positive integer (>= 1). AWS and the
// ceph s3-tests `test_lifecycle_expiration_days0` case reject Days == 0 with
// InvalidArgument; only Date-based rules may omit Days entirely.
if let Some(days) = expiration.days
&& days < 0
&& days < 1
{
return Err(std::io::Error::other(ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS));
}
}
// S3 requires NoncurrentVersionExpiration.NoncurrentDays to be a positive
// integer (>= 1); AWS rejects 0 with InvalidArgument.
if let Some(noncurrent_version_expiration) = &r.noncurrent_version_expiration
&& let Some(noncurrent_days) = noncurrent_version_expiration.noncurrent_days
&& noncurrent_days < 0
&& noncurrent_days < 1
{
return Err(std::io::Error::other(ERR_LIFECYCLE_INVALID_NONCURRENT_EXPIRATION_DAYS));
}
// S3 requires AbortIncompleteMultipartUpload.DaysAfterInitiation to be present
// and a positive integer (>= 1); AWS rejects 0 (and a missing value) with
// InvalidArgument.
if let Some(abort_incomplete_multipart_upload) = &r.abort_incomplete_multipart_upload {
match abort_incomplete_multipart_upload.days_after_initiation {
Some(days) if days >= 0 => {}
Some(days) if days >= 1 => {}
_ => return Err(std::io::Error::other(ERR_LIFECYCLE_INVALID_ABORT_INCOMPLETE_MPU_DAYS)),
}
}
@@ -1066,7 +1075,11 @@ mod tests {
#[tokio::test]
#[serial]
async fn validate_accepts_zero_expiration_days() {
async fn validate_rejects_zero_expiration_days() {
// S3 compatibility: Expiration.Days must be a positive integer (>= 1). AWS and
// the ceph s3-tests `test_lifecycle_expiration_days0` case reject Days == 0 with
// InvalidArgument. The PutBucketLifecycleConfiguration path maps this io::Error
// to s3_error!(InvalidArgument) (see execute_put_bucket_lifecycle_configuration).
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
@@ -1086,9 +1099,12 @@ mod tests {
}],
};
lc.validate(&ObjectLockConfiguration::default())
let err = lc
.validate(&ObjectLockConfiguration::default())
.await
.expect("zero-day expiration should be accepted");
.expect_err("zero-day expiration should be rejected");
assert_eq!(err.to_string(), ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS);
}
#[tokio::test]
@@ -1148,6 +1164,41 @@ mod tests {
.expect("expected validation to pass");
}
#[tokio::test]
#[serial]
async fn validate_accepts_one_day_boundary_values() {
// Pin the exact >= 1 boundary: a value of 1 is the smallest legal positive
// integer and must be accepted for every day-count field tightened for S3
// compatibility. This catches an off-by-one that would reject Days == 1.
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
..Default::default()
}),
abort_incomplete_multipart_upload: Some(s3s::dto::AbortIncompleteMultipartUpload {
days_after_initiation: Some(1),
}),
del_marker_expiration: None,
filter: None,
id: Some("one-day-boundary".to_string()),
noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
}),
noncurrent_version_transitions: None,
prefix: Some("boundary/".to_string()),
transitions: None,
}],
};
lc.validate(&ObjectLockConfiguration::default())
.await
.expect("one-day boundary values should be accepted");
}
#[tokio::test]
#[serial]
async fn has_active_rules_accepts_zero_day_expiration() {
@@ -1175,7 +1226,9 @@ mod tests {
#[tokio::test]
#[serial]
async fn validate_accepts_zero_noncurrent_expiration_days() {
async fn validate_rejects_zero_noncurrent_expiration_days() {
// S3 compatibility: NoncurrentVersionExpiration.NoncurrentDays must be a positive
// integer (>= 1); AWS rejects 0 with InvalidArgument.
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
@@ -1195,9 +1248,12 @@ mod tests {
}],
};
lc.validate(&ObjectLockConfiguration::default())
let err = lc
.validate(&ObjectLockConfiguration::default())
.await
.expect("zero-day noncurrent expiration should be accepted");
.expect_err("zero-day noncurrent expiration should be rejected");
assert_eq!(err.to_string(), ERR_LIFECYCLE_INVALID_NONCURRENT_EXPIRATION_DAYS);
}
#[tokio::test]
@@ -1258,7 +1314,9 @@ mod tests {
#[tokio::test]
#[serial]
async fn validate_accepts_zero_abort_incomplete_multipart_upload_days() {
async fn validate_rejects_zero_abort_incomplete_multipart_upload_days() {
// S3 compatibility: AbortIncompleteMultipartUpload.DaysAfterInitiation must be a
// positive integer (>= 1); AWS rejects 0 with InvalidArgument.
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
@@ -1277,9 +1335,12 @@ mod tests {
}],
};
lc.validate(&ObjectLockConfiguration::default())
let err = lc
.validate(&ObjectLockConfiguration::default())
.await
.expect("zero-day abort incomplete multipart upload should be accepted");
.expect_err("zero-day abort incomplete multipart upload should be rejected");
assert_eq!(err.to_string(), ERR_LIFECYCLE_INVALID_ABORT_INCOMPLETE_MPU_DAYS);
}
#[tokio::test]