Compare commits

...

7 Commits

Author SHA1 Message Date
Hauser Zhi 7852b0cfde style(notify): match the lifecycle module's log field shape
The quarantined-target warning carried its own message where every
sibling entry in this module labels itself "notify runtime lifecycle"
and lets the state field distinguish the outcome. Conforming keeps the
event stream uniform and drops the second copy of the sentence.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
2026-11-21 14:54:27 +08:00
Hauser Zhi efa8bc6d4b merge: sync with origin/main 2026-11-21 14:47:08 +08:00
Hauser Zhi fefd238d57 perf(notify): publish quarantined generations instead of rebuilding forever
is_converged() answered "did every target activate?" but was used as
"does the runtime need rebuilding?". One unreachable target therefore
left the generation permanently unpublished, and the 5s reconciler tore
the whole runtime down and rebuilt it on every tick: 9 full rebuilds
(measured: create + init probe + queue reopen + replay restart + close)
in 30s, which is where the delayed first events after a restart come from.

mark_published now always advances the applied generation and records
per-target completeness separately, so is_converged() means "published".
The reconciler's rebuild trigger and the admin availability gate are
correct with no call-site changes, and runtime_lifecycle_is_published()
is gone again. the one-shot initialize() API keeps its strict contract
through the new all_targets_activated() signal.

prepare_target no longer drops store-less targets whose init failed:
they stay quarantined like store-backed ones, so recovery never needs a
runtime rebuild and PrepareTargetResult::Failed disappears.

Measured on the issue #9545 scenario: 0 rebuilds in the 30s window that
previously saw 9, with the admin surface unchanged.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
2026-11-21 14:45:03 +08:00
overtrue 411caa97d7 test(storage): build object-lock helpers on the owned types
The fixtures keep the same Enabled, COMPLIANCE, GOVERNANCE, ON, and OFF spellings.
2026-11-21 14:22:24 +08:00
Hauser Zhi a8312c1157 merge: sync with origin/main 2026-11-21 14:20:51 +08:00
Hauser Zhi 323f308d57 fix(notify): quarantine unreachable targets instead of failing the runtime
A target that cannot activate is a per-target health outcome, not a
lifecycle transition failure. Turning it into a transition error left
the generation permanently unconverged, so every notification admin call
re-entered the failing reconcile and answered 500, and the broken target
could not be removed because only the /reset DELETE route existed.

Publish the generation with the failed target quarantined, gate admin
access on the shared runtime_lifecycle_is_published() predicate, and
register the bare DELETE /v3/target/{target_type}/{target_name} route.
The one-shot initialize() API keeps its strict retry contract.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
2026-11-21 14:16:44 +08:00
overtrue 317e895096 test(admin): pin the lifecycle staleness axis to the fixture instant
ExpiryUpdatedAt stays 2026-07-07T15:58:57.337315Z, the text in the fixture.
2026-11-21 13:40:35 +08:00
11 changed files with 158 additions and 173 deletions
+12 -1
View File
@@ -77,7 +77,18 @@ pub async fn initialize(config: Config) -> Result<(), NotificationError> {
let task_system = system.clone();
tokio::spawn(async move {
let result = task_system.set_targets_enabled(true, Some(config)).await;
// The one-shot API keeps its strict contract: report an activation that
// left quarantined targets so the caller can retry with a fixed config.
// The reconciliation path quarantines instead and stays admin-usable.
let result = task_system.set_targets_enabled(true, Some(config)).await.and_then(|()| {
if task_system.runtime_lifecycle_all_targets_activated() {
Ok(())
} else {
Err(NotificationError::Initialization(
"one or more notification targets failed to activate".to_string(),
))
}
});
let mut legacy = LEGACY_INITIALIZATION.lock().unwrap_or_else(|err| err.into_inner());
if matches!(
legacy.as_ref(),
+10 -1
View File
@@ -393,11 +393,20 @@ impl NotificationSystem {
self.services.config_manager.lifecycle().state()
}
/// Returns whether the latest accepted lifecycle generation is active.
/// Returns whether the latest accepted lifecycle generation has been handed
/// to the runtime. A published generation serves admin operations even when
/// some targets are quarantined; per-target health is reported by the target
/// list, and the reconciler must not rebuild a published generation.
pub fn runtime_lifecycle_is_converged(&self) -> bool {
self.services.config_manager.lifecycle().is_converged()
}
/// Returns whether the last published generation activated every configured
/// target. Used by the one-shot `initialize` API's strict contract.
pub fn runtime_lifecycle_all_targets_activated(&self) -> bool {
self.services.config_manager.lifecycle().all_targets_activated()
}
/// Validates a bucket notification configuration against the runtime targets
/// without publishing it, so callers can reject a request before persisting.
pub async fn validate_bucket_notification_config(
+44 -13
View File
@@ -21,12 +21,12 @@ use rustfs_config::server_config::Config;
use rustfs_targets::Target;
use std::future::Future;
use std::pin::Pin;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex as StdMutex};
use tokio::sync::{Mutex, RwLock};
use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;
use tracing::{debug, info};
use tracing::{debug, info, warn};
type CurrentGeneration = Arc<dyn Fn() -> bool + Send + Sync>;
type CommitGeneration = Box<dyn FnOnce(bool) + Send>;
@@ -238,11 +238,18 @@ impl LifecycleRuntime for DefaultLifecycleRuntime {
close_old?;
close_rejected?;
close_activation_rejected?;
// A target that cannot activate is quarantined and reported by the
// admin target list. Failing the whole generation here would leave
// the lifecycle permanently unconverged and block every admin call.
if !activation_failures.is_empty() {
return Err(NotificationError::Initialization(format!(
"one or more notification targets failed to activate: {}",
activation_failures.join("; ")
)));
warn!(
event = EVENT_NOTIFY_RUNTIME_LIFECYCLE,
component = LOG_COMPONENT_NOTIFY,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "degraded",
failures = %activation_failures.join("; "),
"notify runtime lifecycle"
);
}
Ok(())
})))
@@ -281,6 +288,7 @@ impl LifecycleRuntime for DefaultLifecycleRuntime {
struct LifecycleCoordinatorInner {
applied_generation: AtomicU64,
apply_lock: Mutex<()>,
all_targets_activated: AtomicBool,
desired: StdMutex<DesiredState>,
publication_gate: Arc<StdMutex<()>>,
runtime: Arc<dyn LifecycleRuntime>,
@@ -309,6 +317,7 @@ impl NotifyLifecycleCoordinator {
inner: Arc::new(LifecycleCoordinatorInner {
applied_generation: AtomicU64::new(0),
apply_lock: Mutex::new(()),
all_targets_activated: AtomicBool::new(true),
desired: StdMutex::new(DesiredState {
cancellation: CancellationToken::new(),
config: initial_config,
@@ -357,11 +366,22 @@ impl NotifyLifecycleCoordinator {
*self.inner.state.lock().unwrap_or_else(|err| err.into_inner())
}
/// Whether the latest accepted generation has been handed to the runtime.
/// This is the rebuild trigger and the admin availability gate: a published
/// generation stays published until the configuration changes, so one
/// unreachable target cannot keep the runtime churning through rebuilds.
pub(crate) fn is_converged(&self) -> bool {
let desired_generation = self.inner.desired.lock().unwrap_or_else(|err| err.into_inner()).generation;
self.inner.applied_generation.load(Ordering::Acquire) >= desired_generation
}
/// Whether the last published generation activated every configured target.
/// Per-target health is reported by the admin target list; this only drives
/// the one-shot `initialize` API's strict contract.
pub(crate) fn all_targets_activated(&self) -> bool {
self.inner.all_targets_activated.load(Ordering::Acquire)
}
fn accept<F>(&self, allow_terminated: bool, update: F) -> NotificationLifecycleTransition
where
F: FnOnce(&mut DesiredState) -> Result<bool, NotificationError>,
@@ -454,9 +474,8 @@ impl LifecycleCoordinatorInner {
fn mark_published(&self, generation: u64, state: NotificationRuntimeState, complete: bool) {
*self.state.lock().unwrap_or_else(|err| err.into_inner()) = state;
if complete {
self.applied_generation.store(generation, Ordering::Release);
}
self.all_targets_activated.store(complete, Ordering::Release);
self.applied_generation.store(generation, Ordering::Release);
}
}
@@ -979,7 +998,7 @@ mod tests {
}
#[tokio::test]
async fn partial_activation_reports_error_and_same_config_can_retry() {
async fn partial_activation_quarantines_failed_target_without_rebuild_churn() {
// The template's single-failure init budget is shared by every clone the factory hands
// out, so the "bad" instance fails once in the first generation and recovers on retry.
let bad_template = MockTarget::new("bad", RETRY_INIT_TARGET_TYPE).with_init_failures(1);
@@ -1006,14 +1025,25 @@ mod tests {
.insert("bad".to_string(), bad);
let (coordinator, runtime_view, _) = real_runtime(config.clone(), Arc::new(TargetRegistry::with_plugins(plugins)), None);
// A quarantined target is a per-target health outcome, so the mutation
// itself succeeds and admin callers are not locked out.
coordinator
.set_mode(true, Some(config.clone()))
.wait()
.await
.expect_err("the originating transition must report the failed target");
.expect("a quarantined target must not fail the lifecycle transition");
let partial = runtime_view.runtime_status_snapshot().await;
assert_eq!(partial.target_count, 1, "the healthy target should remain fault-isolated and active");
assert!(!coordinator.is_converged(), "a partial generation must remain retryable");
assert_eq!(partial.target_count, 2, "a quarantined target stays in the runtime beside healthy ones");
// A published generation must not look stale: otherwise the reconciler
// tears the whole runtime down and rebuilds it on every tick.
assert!(
coordinator.is_converged(),
"a quarantined target must not leave the generation unpublished"
);
assert!(
!coordinator.all_targets_activated(),
"the quarantined target is still reported as not fully activated"
);
assert_eq!(coordinator.state(), NotificationRuntimeState::TargetsEnabled { generation: 1 });
coordinator
@@ -1024,6 +1054,7 @@ mod tests {
let recovered = runtime_view.runtime_status_snapshot().await;
assert_eq!(recovered.target_count, 2);
assert!(coordinator.is_converged());
assert!(coordinator.all_targets_activated(), "the transient failure recovers on the next apply");
assert_eq!(coordinator.state(), NotificationRuntimeState::TargetsEnabled { generation: 2 });
coordinator
+5 -17
View File
@@ -162,7 +162,6 @@ where
) -> PreparedActivation<E> {
let mut prepared = Vec::with_capacity(targets.len());
let mut failures = Vec::new();
let mut rejected_targets = Vec::new();
let mut targets = targets.into_iter();
while let Some(target) = targets.next() {
match prepare_target(target, cancellation).await {
@@ -179,19 +178,6 @@ where
});
prepared.push(target);
}
PrepareTargetResult::Failed { error, target } => {
drop(error);
let target_id = target.id().to_string();
tracing::warn!(
target_id,
reason = "initialization_failed",
"Target initialization failed during lifecycle preparation"
);
failures.push(TargetActivationFailure {
detail: format!("{target_id}: initialization failed"),
});
rejected_targets.push(Arc::from(target));
}
PrepareTargetResult::Cancelled(target) => {
prepared.push(Arc::from(target));
prepared.extend(targets.map(Arc::from));
@@ -201,7 +187,7 @@ where
}
PreparedActivation {
failures,
rejected_targets,
rejected_targets: Vec::new(),
targets: prepared,
}
}
@@ -615,13 +601,15 @@ mod tests {
}
#[tokio::test]
async fn builtin_adapter_skips_non_store_target_when_init_fails() {
async fn builtin_adapter_keeps_non_store_target_when_init_fails() {
let adapter = builtin_adapter();
let target = MockTarget::new("primary", "webhook").with_init_failures(usize::MAX);
// A target without a queue store has nothing to replay, but it stays in
// the runtime so it self-heals and remains visible to the target list.
let activation = adapter.activate_with_replay(vec![Box::new(target)]).await;
assert!(activation.targets.is_empty());
assert_eq!(activation.targets.len(), 1);
assert!(activation.replay_workers.is_empty());
}
+5 -22
View File
@@ -87,14 +87,7 @@ where
E: PluginEvent,
{
Ready(SharedTarget<E>),
Degraded {
error: TargetError,
target: SharedTarget<E>,
},
Failed {
error: TargetError,
target: Box<dyn Target<E> + Send + Sync>,
},
Degraded { error: TargetError, target: SharedTarget<E> },
Cancelled(Box<dyn Target<E> + Send + Sync>),
}
@@ -537,12 +530,7 @@ where
) -> (mpsc::Sender<()>, JoinHandle<()>),
{
let shared = match prepare_target(target, None).await {
PrepareTargetResult::Ready(target) => target,
PrepareTargetResult::Degraded { target, .. } => target,
PrepareTargetResult::Failed { target, .. } => {
let _ = target.close().await;
return None;
}
PrepareTargetResult::Ready(target) | PrepareTargetResult::Degraded { target, .. } => target,
PrepareTargetResult::Cancelled(_) => unreachable!("preparation without a cancellation token cannot be cancelled"),
};
let target_id = shared.id().to_string();
@@ -567,7 +555,6 @@ where
E: PluginEvent,
{
let target_id = target.id().to_string();
let has_store = target.store().is_some();
let init_result = match cancellation {
Some(cancellation) => {
@@ -582,13 +569,9 @@ where
if let Err(err) = init_result {
tracing::error!(target_id = %target_id, reason = "initialization_failed", "Failed to initialize target");
if !has_store {
return PrepareTargetResult::Failed { error: err, target };
}
tracing::warn!(
target_id = %target_id,
"Proceeding with store-backed target despite init failure"
);
// Keep the target in the runtime whether or not it has a queue store: a
// quarantined target self-heals through delivery retries and is reported
// by the target list, so recovery never needs a runtime rebuild.
return PrepareTargetResult::Degraded {
error: err,
target: Arc::from(target),
+7
View File
@@ -221,6 +221,13 @@ pub fn register_notification_target_route(r: &mut S3Router<AdminOperation>) -> s
AdminOperation(&NotificationTarget {}),
)?;
r.insert(
Method::DELETE,
format!("{}{}", ADMIN_PREFIX, "/v3/target/{target_type}/{target_name}").as_str(),
AdminOperation(&RemoveNotificationTarget {}),
)?;
// Legacy spelling kept for consoles that still call the reset suffix.
r.insert(
Method::DELETE,
format!("{}{}", ADMIN_PREFIX, "/v3/target/{target_type}/{target_name}/reset").as_str(),
@@ -13921,18 +13921,18 @@ mod tests {
env!("CARGO_MANIFEST_DIR"),
"/../crates/ecstore/tests/fixtures/bucket-config-goldens/lifecycle/minio.in.xml"
));
let mut old_reader = s3s::xml::Deserializer::new(raw);
let old: s3s::dto::BucketLifecycleConfiguration =
s3s::xml::Deserialize::deserialize(&mut old_reader).expect("read the actual old lifecycle expiry axis");
old_reader.expect_eof().expect("the old expiry reader consumes the document");
let old_axis = OffsetDateTime::from(old.expiry_updated_at.expect("the old config contains its expiry axis"));
// Answer of s3s 5761ddfe (`Timestamp::parse(DateTime)` on this document's
// ExpiryUpdatedAt). The owned axis must keep that instant.
let legacy_axis =
parse_lifecycle_date("2026-07-07T15:58:57.337315Z").expect("the legacy expiry axis is the minio.in.xml value");
let new = deserialize(raw).expect("read the persisted lifecycle expiry axis");
let new_axis = local_lifecycle_staleness_axis(Some(&new), OffsetDateTime::UNIX_EPOCH)
.expect("the parsed lifecycle axis is readable");
let incoming = parse_lifecycle_date("2026-07-07T15:58:57.337100Z").expect("the incoming expiry axis is valid");
assert_eq!(new_axis, legacy_axis, "the parsed axis must keep the legacy submillisecond instant");
assert_eq!(
is_stale_update(new_axis, Some(incoming)),
is_stale_update(old_axis, Some(incoming)),
is_stale_update(legacy_axis, Some(incoming)),
"a peer update older than the full-precision local axis must remain stale"
);
}
+6
View File
@@ -305,6 +305,12 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
SET_BUCKET_TARGET,
RouteRiskLevel::High,
),
admin(
HttpMethod::Delete,
"/rustfs/admin/v3/target/{target_type}/{target_name}",
SET_BUCKET_TARGET,
RouteRiskLevel::High,
),
admin(
HttpMethod::Delete,
"/rustfs/admin/v3/target/{target_type}/{target_name}/reset",
@@ -212,6 +212,7 @@ fn expected_admin_route_matrix() -> Vec<RouteMatrixEntry> {
admin_route(Method::GET, "/v3/idp/builtin/policy-entities"),
admin_route(Method::GET, "/v3/target/list"),
admin_route_sample(Method::PUT, "/v3/target/{target_type}/{target_name}", "/v3/target/webhook/test-target"),
admin_route_sample(Method::DELETE, "/v3/target/{target_type}/{target_name}", "/v3/target/webhook/test-target"),
admin_route_sample(
Method::DELETE,
"/v3/target/{target_type}/{target_name}/reset",
+4
View File
@@ -167,6 +167,10 @@ fn ensure_live_events_initialized() -> std::sync::Arc<NotificationSystem> {
system
}
/// A published lifecycle generation is usable even when some targets are
/// quarantined: healthy targets already carry events and per-target health is
/// reported by the admin target list. Only an unpublished generation means the
/// runtime is not available yet.
fn ensure_event_notifier_converged(system: &NotificationSystem) -> Result<(), NotificationError> {
if system.runtime_lifecycle_is_converged() {
Ok(())
+58 -113
View File
@@ -55,32 +55,41 @@ mod tests {
use http::{HeaderMap, HeaderValue, StatusCode};
use rustfs_config::MI_B;
use rustfs_gateway_types::persistence::{
PersistedFilterRule, PersistedLambdaFunctionConfiguration, PersistedNotificationConfigurationFilter,
PersistedDefaultRetention, PersistedFilterRule, PersistedLambdaFunctionConfiguration,
PersistedNotificationConfigurationFilter, PersistedObjectLockConfiguration, PersistedObjectLockRule,
PersistedQueueConfiguration, PersistedS3KeyFilter, PersistedTopicConfiguration,
};
use rustfs_s3_types::{S3ErrorCode, S3Result};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
use rustfs_utils::http::{
AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
MINIO_INTERNAL_PREFIX, RUSTFS_INTERNAL_PREFIX, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,
SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, contains_key_str, get_str, insert_str,
};
use rustfs_zip::CompressionFormat;
use s3s::dto::{
DefaultRetention, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockLegalHold, ObjectLockLegalHoldStatus,
ObjectLockRetention, ObjectLockRetentionMode, ObjectLockRule,
};
use std::collections::HashMap;
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
fn parse_object_lock_retention(retention: Option<ObjectLockRetention>) -> S3Result<HashMap<String, String>> {
parse_object_lock_retention_native(retention.map(|r| RequestedObjectLockRetention {
mode: r.mode.map(|m| m.as_str().to_owned()),
retain_until: r.retain_until_date.map(OffsetDateTime::from),
}))
fn parse_object_lock_retention(retention: Option<RequestedObjectLockRetention>) -> S3Result<HashMap<String, String>> {
parse_object_lock_retention_native(retention)
}
fn parse_object_lock_legal_hold(legal_hold: Option<ObjectLockLegalHold>) -> S3Result<HashMap<String, String>> {
parse_object_lock_legal_hold_native(legal_hold.map(|h| RequestedObjectLegalHold(h.status.map(|s| s.as_str().to_owned()))))
fn parse_object_lock_legal_hold(legal_hold: Option<RequestedObjectLegalHold>) -> S3Result<HashMap<String, String>> {
parse_object_lock_legal_hold_native(legal_hold)
}
/// Bucket default retention already stored, the shape `apply_default_lock_retention_metadata` reads.
/// `Enabled` / `COMPLIANCE` / `GOVERNANCE` are the spellings s3s 5761ddfe wrote for these fixtures.
fn enabled_lock(retention: Option<(&str, i32)>) -> PersistedObjectLockConfiguration {
PersistedObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: retention.map(|(mode, days)| PersistedObjectLockRule {
default_retention: Some(PersistedDefaultRetention {
mode: Some(mode.to_owned()),
days: Some(days),
years: None,
}),
}),
}
}
#[test]
@@ -155,8 +164,6 @@ mod tests {
#[test]
fn test_s3_error_scenarios() {
use s3s::{S3Error, S3ErrorCode, s3_error};
// Test that we can create expected S3 errors for common validation cases
// Test incomplete body error
@@ -185,27 +192,12 @@ mod tests {
assert_eq!(gz_format.extension(), "gz");
}
/// The bucket default retention path reads the persisted shape; the request
/// literals in these tests stay on the wire DTO and convert once here.
fn persisted_lock(cfg: &ObjectLockConfiguration) -> rustfs_gateway_types::persistence::PersistedObjectLockConfiguration {
crate::storage::persisted_from_s3s(cfg).expect("the test configuration converts")
}
#[test]
fn test_apply_default_lock_retention_metadata_applies_bucket_default() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
days: Some(1),
years: None,
}),
}),
};
let cfg = enabled_lock(Some(("COMPLIANCE", 1)));
let mut metadata = std::collections::HashMap::new();
assert!(apply_default_lock_retention_metadata(Some(&persisted_lock(&cfg)), &mut metadata));
assert!(apply_default_lock_retention_metadata(Some(&cfg), &mut metadata));
assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER), Some(&"COMPLIANCE".to_string()));
let retain_until = metadata
.get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER)
@@ -224,22 +216,13 @@ mod tests {
fn test_apply_default_lock_retention_metadata_preserves_explicit_retention() {
use std::collections::HashMap;
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
days: Some(1),
years: None,
}),
}),
};
let cfg = enabled_lock(Some(("COMPLIANCE", 1)));
let mut metadata = HashMap::from([
(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), "GOVERNANCE".to_string()),
(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2030-01-01T00:00:00Z".to_string()),
]);
assert!(!apply_default_lock_retention_metadata(Some(&persisted_lock(&cfg)), &mut metadata));
assert!(!apply_default_lock_retention_metadata(Some(&cfg), &mut metadata));
assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER), Some(&"GOVERNANCE".to_string()));
assert_eq!(
metadata.get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER),
@@ -249,30 +232,17 @@ mod tests {
#[test]
fn test_apply_default_lock_retention_metadata_ignores_bucket_without_default() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: None,
};
let cfg = enabled_lock(None);
let mut metadata = std::collections::HashMap::new();
assert!(!apply_default_lock_retention_metadata(Some(&persisted_lock(&cfg)), &mut metadata));
assert!(!apply_default_lock_retention_metadata(Some(&cfg), &mut metadata));
assert!(metadata.is_empty());
}
#[test]
fn test_apply_bucket_default_lock_retention_applies_authoritative_default() {
let config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::GOVERNANCE)),
days: Some(1),
years: None,
}),
}),
};
let state = ObjectLockConfigState::Configured {
config: persisted_lock(&config),
config: enabled_lock(Some(("GOVERNANCE", 1))),
updated_at: OffsetDateTime::now_utc(),
};
let mut metadata = std::collections::HashMap::new();
@@ -286,19 +256,10 @@ mod tests {
#[test]
fn empty_retention_metadata_does_not_suppress_bucket_default() {
let config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::GOVERNANCE)),
days: Some(1),
years: None,
}),
}),
};
let config = enabled_lock(Some(("GOVERNANCE", 1)));
for name in [AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER] {
let mut metadata = HashMap::from([(name.to_string(), String::new())]);
assert!(apply_default_lock_retention_metadata(Some(&persisted_lock(&config)), &mut metadata));
assert!(apply_default_lock_retention_metadata(Some(&config), &mut metadata));
assert_eq!(metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), Some("GOVERNANCE"));
assert!(!metadata[AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER].is_empty());
}
@@ -306,18 +267,8 @@ mod tests {
#[test]
fn test_apply_bucket_default_lock_retention_preserves_explicit_retention_path() {
let config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
days: Some(1),
years: None,
}),
}),
};
let state = ObjectLockConfigState::Configured {
config: persisted_lock(&config),
config: enabled_lock(Some(("COMPLIANCE", 1))),
updated_at: OffsetDateTime::now_utc(),
};
let mut metadata = HashMap::new();
@@ -655,15 +606,15 @@ mod tests {
// [1] Normal case: No retention specified (empty metadata)
assert!(parse_object_lock_retention(None).is_ok());
assert!(parse_object_lock_retention(None).unwrap().is_empty());
let cleared = parse_object_lock_retention(Some(ObjectLockRetention::default())).unwrap();
let cleared = parse_object_lock_retention(Some(RequestedObjectLockRetention::default())).unwrap();
assert_eq!(cleared.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), Some(""));
assert_eq!(cleared.get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER).map(String::as_str), Some(""));
assert!(contains_key_str(&cleared, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP));
// [2] Normal case: Retention with valid COMPLIANCE mode (future date)
let valid_compliance_retention = ObjectLockRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
retain_until_date: Some(datetime!(2030-01-01 00:00:00 UTC).into()),
let valid_compliance_retention = RequestedObjectLockRetention {
mode: Some("COMPLIANCE".to_owned()),
retain_until: Some(datetime!(2030-01-01 00:00:00 UTC)),
};
let compliance_metadata = parse_object_lock_retention(Some(valid_compliance_retention)).unwrap();
assert_eq!(compliance_metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER).unwrap(), "COMPLIANCE");
@@ -678,26 +629,26 @@ mod tests {
assert!(retention_timestamp <= OffsetDateTime::now_utc());
// [3] Normal case: Retention with valid GOVERNANCE mode (future date)
let valid_governance_retention = ObjectLockRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::GOVERNANCE)),
retain_until_date: Some(datetime!(2030-01-01 00:00:00 UTC).into()),
let valid_governance_retention = RequestedObjectLockRetention {
mode: Some("GOVERNANCE".to_owned()),
retain_until: Some(datetime!(2030-01-01 00:00:00 UTC)),
};
let governance_metadata = parse_object_lock_retention(Some(valid_governance_retention)).unwrap();
assert_eq!(governance_metadata.get(AMZ_OBJECT_LOCK_MODE_LOWER).unwrap(), "GOVERNANCE");
// A retention update is either empty or has both fields.
let none_mode_retention = ObjectLockRetention {
let none_mode_retention = RequestedObjectLockRetention {
mode: None,
retain_until_date: Some(datetime!(2030-01-01 00:00:00 UTC).into()),
retain_until: Some(datetime!(2030-01-01 00:00:00 UTC)),
};
assert_eq!(
parse_object_lock_retention(Some(none_mode_retention)).unwrap_err().code(),
&S3ErrorCode::MalformedXML
);
let none_date_retention = ObjectLockRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
retain_until_date: None,
let none_date_retention = RequestedObjectLockRetention {
mode: Some("COMPLIANCE".to_owned()),
retain_until: None,
};
assert_eq!(
parse_object_lock_retention(Some(none_date_retention)).unwrap_err().code(),
@@ -705,9 +656,9 @@ mod tests {
);
// [6] Error case: Retention with invalid mode (non COMPLIANCE/GOVERNANCE)
let invalid_mode_retention = ObjectLockRetention {
mode: Some(ObjectLockRetentionMode::from_static("INVALID_MODE")),
retain_until_date: Some(datetime!(2030-01-01 00:00:00 UTC).into()),
let invalid_mode_retention = RequestedObjectLockRetention {
mode: Some("INVALID_MODE".to_owned()),
retain_until: Some(datetime!(2030-01-01 00:00:00 UTC)),
};
let err = parse_object_lock_retention(Some(invalid_mode_retention)).unwrap_err();
assert_eq!(err.code().as_str(), S3ErrorCode::MalformedXML.as_str());
@@ -717,9 +668,9 @@ mod tests {
);
// [7] Error case: Retention with past date should fail
let past_date_retention = ObjectLockRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
retain_until_date: Some(datetime!(2020-01-01 00:00:00 UTC).into()),
let past_date_retention = RequestedObjectLockRetention {
mode: Some("COMPLIANCE".to_owned()),
retain_until: Some(datetime!(2020-01-01 00:00:00 UTC)),
};
let err = parse_object_lock_retention(Some(past_date_retention)).unwrap_err();
assert_eq!(err.code().as_str(), S3ErrorCode::InvalidArgument.as_str());
@@ -730,9 +681,9 @@ mod tests {
.to_offset(UtcOffset::from_hms(5, 30, 45).expect("offset with seconds should be constructible"));
assert!(unsupported_rfc3339_date > OffsetDateTime::now_utc());
let unsupported_date_retention = ObjectLockRetention {
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
retain_until_date: Some(unsupported_rfc3339_date.into()),
let unsupported_date_retention = RequestedObjectLockRetention {
mode: Some("COMPLIANCE".to_owned()),
retain_until: Some(unsupported_rfc3339_date),
};
let err = parse_object_lock_retention(Some(unsupported_date_retention)).unwrap_err();
assert_eq!(err.code().as_str(), S3ErrorCode::InvalidArgument.as_str());
@@ -746,9 +697,7 @@ mod tests {
assert!(parse_object_lock_legal_hold(None).unwrap().is_empty());
// [2] Normal case: Legal hold with valid ON status
let valid_on_legal_hold = ObjectLockLegalHold {
status: Some(ObjectLockLegalHoldStatus::from_static(ObjectLockLegalHoldStatus::ON)),
};
let valid_on_legal_hold = RequestedObjectLegalHold(Some("ON".to_owned()));
let on_metadata = parse_object_lock_legal_hold(Some(valid_on_legal_hold)).unwrap();
assert_eq!(on_metadata.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).unwrap(), "ON");
let legal_hold_timestamp = get_str(&on_metadata, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP)
@@ -757,20 +706,16 @@ mod tests {
assert!(legal_hold_timestamp <= OffsetDateTime::now_utc());
// [3] Normal case: Legal hold with valid OFF status
let valid_off_legal_hold = ObjectLockLegalHold {
status: Some(ObjectLockLegalHoldStatus::from_static(ObjectLockLegalHoldStatus::OFF)),
};
let valid_off_legal_hold = RequestedObjectLegalHold(Some("OFF".to_owned()));
let off_metadata = parse_object_lock_legal_hold(Some(valid_off_legal_hold)).unwrap();
assert_eq!(off_metadata.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).unwrap(), "OFF");
let none_status_legal_hold = ObjectLockLegalHold { status: None };
let none_status_legal_hold = RequestedObjectLegalHold(None);
let err = parse_object_lock_legal_hold(Some(none_status_legal_hold)).expect_err("Status is required");
assert_eq!(err.code().as_str(), S3ErrorCode::MalformedXML.as_str());
// [5] Error case: Legal hold with invalid status (non ON/OFF)
let invalid_status_legal_hold = ObjectLockLegalHold {
status: Some(ObjectLockLegalHoldStatus::from_static("INVALID_STATUS")),
};
let invalid_status_legal_hold = RequestedObjectLegalHold(Some("INVALID_STATUS".to_owned()));
let err = parse_object_lock_legal_hold(Some(invalid_status_legal_hold)).unwrap_err();
assert_eq!(err.code().as_str(), S3ErrorCode::MalformedXML.as_str());
assert_eq!(
@@ -1094,7 +1039,7 @@ mod tests {
.source()
.is_some_and(|cause| cause.is::<http::header::InvalidHeaderValue>())
);
let wire = s3s::S3Error::from(error);
let wire = crate::app::storage_api::s3::S3Error::from(error);
assert_eq!(wire.status_code(), Some(StatusCode::INTERNAL_SERVER_ERROR));
assert!(
wire.source()