Compare commits

..

2 Commits

Author SHA1 Message Date
overtrue 6befa4d042 ci: correct the companion-workflow comments
The ci-docs-only header claimed the ruleset already requires "Quick Checks";
it does not, and this PR deliberately leaves it that way. Say what is true
today and why the job is mirrored ahead of the ruleset change.

Also move the byte-identical requirement onto ci.yml's quick-checks job, which
is the more likely edit site, instead of pointing at a comment that was not
there.
2026-08-01 10:49:34 +08:00
overtrue 06c70fcd14 ci: stop running expensive jobs that cannot inform the result
Three independent fixes that all avoid burning self-hosted runners on work
whose outcome is already determined. None of them changes what is tested.

- ci-docs-only: add a "Quick Checks" companion job. It is a prerequisite for
  gating ci.yml's expensive jobs behind quick-checks (rustfs/backlog#1599):
  once "Quick Checks" is a required check, a docs-only PR would otherwise wait
  on it forever. The steps are a byte-identical copy of ci.yml's quick-checks
  rather than an `echo`, so that on a mixed PR the two same-named check runs
  execute the same commands against the same merge ref and cannot disagree —
  GitHub has no written contract for how it picks between same-named required
  check runs, and the real job only takes 45-51s, leaving no timing margin to
  rely on.

- ci: guard uring-integration with the same `closed` check every other job
  already has. The pull_request trigger includes `closed` only so the
  concurrency group cancels in-flight runs; this job had no guard and no
  `needs`, so every closed or merged PR ran the full io_uring suite (4m17s,
  7m19s and 7m31s on runs 30678272341, 30678117601 and 30662728539).

- ci: gate s3-lifecycle-behavior-tests on e2e-tests, matching
  s3-implemented-tests. Both lanes only download the prebuilt debug binary, and
  s3-implemented-tests already finishes later, so a green PR's wall clock is
  unchanged; a red one stops holding a sm-standard-4 for up to 30 minutes.

Refs: rustfs/backlog#1598, rustfs/backlog#1599
2026-08-01 10:41:35 +08:00
16 changed files with 178 additions and 400 deletions
+2 -2
View File
@@ -23,8 +23,8 @@
# https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
#
# "Quick Checks" is mirrored here ahead of the ruleset change that will make it
# required too (rustfs/backlog#1599). Until that change lands this job is
# inert; mirroring it first is what lets the ruleset change happen without
# required too (rustfs/backlog#1599). It is not required yet, so today this job
# is inert; adding it first is what lets that ruleset change land without
# stranding docs-only PRs on a check nobody reports.
#
# Keep the paths list below in sync with the pull_request paths-ignore list
+1 -68
View File
@@ -144,26 +144,13 @@ jobs:
test-and-lint:
name: Test and Lint
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 90
# Both lines are required. Job-level `permissions` replaces the workflow
# block rather than merging with it, so declaring only `actions: write`
# would drop `contents: read` and break this job's checkout and the
# repo-token the setup action hands to setup-protoc.
permissions:
contents: read
actions: write
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
# This job's token can cancel runs and delete Actions caches. Checkout
# otherwise writes it into .git/config, where a PR's own build.rs or
# proc-macro could read it back out.
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -294,48 +281,6 @@ jobs:
- name: Run rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh
# Early stop. Once this job has failed the PR cannot merge, so the sibling
# lanes are burning runners on a result nobody can act on: on run
# 30674613104 three lanes had already failed while Test and Lint and the
# rio-v2 variant kept going past 70 minutes.
#
# Only this job may cancel. The lanes that are NOT required checks
# (protocols, ILM, e2e, s3-tests) must never hold that power: a flake in
# one of them would turn the required "Test and Lint" into `cancelled`,
# which blocks the merge. Today a maintainer can merge with sftp red, and
# that has to stay true.
#
# These steps run last so the `if: always()` artifact upload above still
# captures logs and diagnostics before the run goes away.
- name: Annotate early-stop reason
if: failure() && github.event_name == 'pull_request'
run: |
echo "## CI early-stop" >> "$GITHUB_STEP_SUMMARY"
echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners." >> "$GITHUB_STEP_SUMMARY"
echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure." >> "$GITHUB_STEP_SUMMARY"
# curl rather than `gh`: every existing `gh` call in this repo runs on
# ubuntu-latest, and the sm-standard-* images are custom and trimmed (they
# ship no C toolchain, see the e2e job below), so `gh` is not known to
# exist here.
#
# Fork PRs are excluded explicitly instead of relying on the error path:
# their GITHUB_TOKEN is forced read-only and job-level permissions cannot
# raise it, so the call would always 403. Skipping keeps their logs clean.
- name: Cancel run on failure (same-repo PR only)
if: >-
failure() && github.event_name == 'pull_request'
&& github.event.pull_request.head.repo.full_name == github.repository
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
curl -fsS -X POST \
-H "Authorization: Bearer ${GH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/cancel" || true
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
@@ -349,7 +294,6 @@ jobs:
test-ilm-integration-serial:
name: ILM Integration (serial)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 45
env:
@@ -387,7 +331,6 @@ jobs:
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
env:
@@ -415,17 +358,10 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
strategy:
# On a PR, one failing protocol leg is enough to know the PR is not ready,
# so stop the sibling leg instead of paying another ~40 minutes for it.
# Everywhere else (main pushes, the merge queue, the weekly schedule) keep
# the full signal: there we want to know whether swift AND sftp are broken,
# not just whichever failed first. This is the only part of the early-stop
# work that also covers fork PRs, since it needs no token.
fail-fast: ${{ github.event_name == 'pull_request' }}
fail-fast: false
matrix:
features:
- name: swift
@@ -457,7 +393,6 @@ jobs:
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -488,7 +423,6 @@ jobs:
build-rustfs-debug-binary-rio-v2:
name: Build RustFS Debug Binary (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -525,7 +459,6 @@ jobs:
# suite (measured 4m17s / 7m19s / 7m31s on runs 30678272341 / 30678117601 /
# 30662728539) and kept the cancellation run in progress for minutes.
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
# GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
# a container, applies no seccomp filter that would block io_uring_setup — so
# the probe succeeds and the tests exercise the real UringBackend/FdCache/
+2 -2
View File
@@ -184,8 +184,8 @@ pub mod bucket {
get_global_replication_stats, init_background_replication, read_durable_mrf_backlog, replication_state_to_filemeta,
replication_status_to_filemeta, replication_statuses_map, replication_target_arns, resync_start_conflict_id,
should_remove_replication_target, should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source, unsupported_replication_config_field,
validate_replication_config_target_arns, version_purge_status_to_filemeta,
should_use_existing_delete_replication_source, validate_replication_config_target_arns,
version_purge_status_to_filemeta,
};
}
+1 -1
View File
@@ -46,7 +46,7 @@ mod runtime_boundary;
pub use datatypes::ResyncStatusType;
pub use replication_config_boundary::{
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
should_remove_replication_target, unsupported_replication_config_field, validate_replication_config_target_arns,
should_remove_replication_target, validate_replication_config_target_arns,
};
#[cfg(test)]
pub(crate) use replication_filemeta_boundary::ReplicateTargetDecision;
@@ -14,5 +14,5 @@
pub use rustfs_replication::{
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
should_remove_replication_target, unsupported_replication_config_field, validate_replication_config_target_arns,
should_remove_replication_target, validate_replication_config_target_arns,
};
@@ -14,11 +14,8 @@
use std::{collections::HashMap, sync::Arc};
use super::replication_error_boundary::Result;
use super::replication_filemeta_boundary::{ReplicateDecision, ReplicationStatusType, ReplicationType};
use super::replication_object_config::{
check_replicate_delete, check_replicate_delete_strict, get_must_replicate_options, must_replicate,
};
use super::replication_object_config::{check_replicate_delete, get_must_replicate_options, must_replicate};
use super::replication_object_decision_boundary::MustReplicateOptions;
use super::replication_pool::{schedule_replication, schedule_replication_delete};
use super::replication_queue_boundary::DeletedObjectReplicationInfo;
@@ -53,16 +50,6 @@ impl ReplicationObjectBridge {
check_replicate_delete(bucket, object, source, opts, get_error).await
}
pub async fn check_delete_strict(
bucket: &str,
object: &ObjectToDelete,
source: &ObjectInfo,
opts: &ObjectOptions,
get_error: Option<String>,
) -> Result<ReplicateDecision> {
check_replicate_delete_strict(bucket, object, source, opts, get_error).await
}
pub async fn schedule_object<S: ReplicationStorage>(
object: ObjectInfo,
storage: Arc<S>,
@@ -169,8 +169,9 @@ pub(crate) async fn check_replicate_delete(
del_opts: &ObjectOptions,
gerr: Option<String>,
) -> ReplicateDecision {
match check_replicate_delete_strict(bucket, dobj, oi, del_opts, gerr).await {
Ok(decision) => decision,
let rcfg = match get_replication_config(bucket).await {
Ok(Some(config)) => config,
Ok(None) => return ReplicateDecision::default(),
Err(err) => {
error!(
event = EVENT_RESYNC_CONFIG_LOOKUP_SKIPPED,
@@ -181,30 +182,16 @@ pub(crate) async fn check_replicate_delete(
error = %err,
"Failed to look up replication config for delete replication"
);
ReplicateDecision::default()
return ReplicateDecision::default();
}
}
}
pub(crate) async fn check_replicate_delete_strict(
bucket: &str,
dobj: &ObjectToDelete,
oi: &ObjectInfo,
del_opts: &ObjectOptions,
gerr: Option<String>,
) -> Result<ReplicateDecision> {
let rcfg = match get_replication_config(bucket).await {
Ok(Some(config)) => config,
Ok(None) => return Ok(ReplicateDecision::default()),
Err(err) => return Err(err),
};
if del_opts.replication_request {
return Ok(ReplicateDecision::default());
return ReplicateDecision::default();
}
if !del_opts.versioned && !del_opts.version_suspended {
return Ok(ReplicateDecision::default());
if !del_opts.versioned {
return ReplicateDecision::default();
}
let replication_delete = object_to_delete_for_replication(dobj);
@@ -222,7 +209,7 @@ pub(crate) async fn check_replicate_delete_strict(
let mut dsc = ReplicateDecision::new();
if tgt_arns.is_empty() {
return Ok(dsc);
return dsc;
}
for tgt_arn in tgt_arns {
@@ -252,7 +239,7 @@ pub(crate) async fn check_replicate_delete_strict(
dsc.set(tgt_dsc);
}
Ok(dsc)
dsc
}
pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplicateOptions) -> ReplicateDecision {
@@ -1151,6 +1151,60 @@ pub async fn replicate_delete<S: ReplicationStorage>(dobj: DeletedObjectReplicat
dobj.delete_object.version_id
};
let _rcfg = match get_replication_config(&bucket).await {
Ok(Some(config)) => config,
Ok(None) => {
debug!(
event = EVENT_REPLICATION_DELETE_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
reason = "replication_config_missing",
"Skipping replication delete because replication config is missing"
);
send_local_event(EventArgs {
event_name: EventName::ObjectReplicationNotTracked.to_string(),
bucket_name: bucket.clone(),
object: ObjectInfo {
bucket: bucket.clone(),
name: dobj.delete_object.object_name.clone(),
version_id,
delete_marker: dobj.delete_object.delete_marker,
..Default::default()
},
user_agent: "Internal: [Replication]".to_string(),
..Default::default()
});
return;
}
Err(err) => {
debug!(
event = EVENT_REPLICATION_DELETE_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
error = %err,
reason = "replication_config_lookup_failed",
"Skipping replication delete because replication config lookup failed"
);
send_local_event(EventArgs {
event_name: EventName::ObjectReplicationNotTracked.to_string(),
bucket_name: bucket.clone(),
object: ObjectInfo {
bucket: bucket.clone(),
name: dobj.delete_object.object_name.clone(),
version_id,
delete_marker: dobj.delete_object.delete_marker,
..Default::default()
},
user_agent: "Internal: [Replication]".to_string(),
..Default::default()
});
return;
}
};
if dobj.delete_object.delete_marker
&& let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id
{
+6 -122
View File
@@ -51,28 +51,6 @@ pub enum ReplicationTargetValidationError {
StaleTarget,
}
pub fn unsupported_replication_config_field(config: &ReplicationConfiguration) -> Option<&'static str> {
for rule in &config.rules {
if rule
.source_selection_criteria
.as_ref()
.is_some_and(|criteria| criteria.sse_kms_encrypted_objects.is_some())
{
return Some("SourceSelectionCriteria.SseKmsEncryptedObjects");
}
if rule.destination.encryption_configuration.is_some() {
return Some("Destination.EncryptionConfiguration");
}
if rule.destination.metrics.is_some() {
return Some("Destination.Metrics");
}
if rule.destination.replication_time.is_some() {
return Some("Destination.ReplicationTime");
}
}
None
}
pub fn active_replication_rule_destination_arns(config: &ReplicationConfiguration) -> HashSet<String> {
let mut arns = HashSet::new();
@@ -239,6 +217,11 @@ impl ReplicationConfigurationExt for ReplicationConfiguration {
}
if obj.version_id.is_some() {
if obj.delete_marker {
return rule.delete_marker_replication.clone().is_some_and(|d| {
d.status == Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED))
});
}
return rule
.delete_replication
.clone()
@@ -322,11 +305,7 @@ impl ReplicationConfigurationExt for ReplicationConfiguration {
#[cfg(test)]
mod tests {
use super::*;
use s3s::dto::{
DeleteMarkerReplication, DeleteReplication, Destination, EncryptionConfiguration, ExistingObjectReplication, Metrics,
MetricsStatus, ReplicationRule, ReplicationTime, ReplicationTimeStatus, ReplicationTimeValue, SourceSelectionCriteria,
SseKmsEncryptedObjects, SseKmsEncryptedObjectsStatus,
};
use s3s::dto::{DeleteMarkerReplication, Destination, ExistingObjectReplication, ReplicationRule};
fn replication_rule(id: &str, arn: &str) -> ReplicationRule {
ReplicationRule {
@@ -626,99 +605,4 @@ mod tests {
"highest-priority rule disables delete-marker replication, so the delete marker must not replicate"
);
}
#[test]
fn version_purge_uses_delete_replication_for_object_and_marker_versions() {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule("delete", arn);
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
});
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
});
let mut config = ReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
let version_id = Some(Uuid::new_v4());
for delete_marker in [false, true] {
assert!(config.replicate(&ObjectOpts {
name: "object".to_string(),
version_id,
delete_marker,
op_type: ReplicationType::Delete,
..Default::default()
}));
}
assert!(!config.replicate(&ObjectOpts {
name: "object".to_string(),
delete_marker: true,
op_type: ReplicationType::Delete,
..Default::default()
}));
let rule = &mut config.rules[0];
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
});
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
});
for delete_marker in [false, true] {
assert!(!config.replicate(&ObjectOpts {
name: "object".to_string(),
version_id,
delete_marker,
op_type: ReplicationType::Delete,
..Default::default()
}));
}
assert!(config.replicate(&ObjectOpts {
name: "object".to_string(),
delete_marker: true,
op_type: ReplicationType::Delete,
..Default::default()
}));
}
#[test]
fn unsupported_replication_fields_are_reported_before_persistence() {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut config = ReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule("unsupported", arn)],
};
config.rules[0].source_selection_criteria = Some(SourceSelectionCriteria {
replica_modifications: None,
sse_kms_encrypted_objects: Some(SseKmsEncryptedObjects {
status: SseKmsEncryptedObjectsStatus::from_static(SseKmsEncryptedObjectsStatus::ENABLED),
}),
});
assert_eq!(
unsupported_replication_config_field(&config),
Some("SourceSelectionCriteria.SseKmsEncryptedObjects")
);
config.rules[0].source_selection_criteria = None;
config.rules[0].destination.encryption_configuration = Some(EncryptionConfiguration::default());
assert_eq!(unsupported_replication_config_field(&config), Some("Destination.EncryptionConfiguration"));
config.rules[0].destination.encryption_configuration = None;
config.rules[0].destination.metrics = Some(Metrics {
event_threshold: None,
status: MetricsStatus::from_static(MetricsStatus::ENABLED),
});
assert_eq!(unsupported_replication_config_field(&config), Some("Destination.Metrics"));
config.rules[0].destination.metrics = None;
config.rules[0].destination.replication_time = Some(ReplicationTime {
status: ReplicationTimeStatus::from_static(ReplicationTimeStatus::ENABLED),
time: ReplicationTimeValue { minutes: Some(15) },
});
assert_eq!(unsupported_replication_config_field(&config), Some("Destination.ReplicationTime"));
}
}
+1 -2
View File
@@ -30,8 +30,7 @@ pub mod tagging;
pub use config::{
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, active_replication_rule_destination_arns,
replication_target_arns, should_remove_replication_target, unsupported_replication_config_field,
validate_replication_config_target_arns,
replication_target_arns, should_remove_replication_target, validate_replication_config_target_arns,
};
pub use delete::{
DeletedObjectReplicationInfo, is_retryable_delete_replication_head_error, is_version_delete_replication,
+5 -7
View File
@@ -322,9 +322,9 @@ mod tests {
use crate::storage_api::ObjectToDelete;
use crate::{ReplicationStatusType, ReplicationType, VersionPurgeStatusType, target_reset_header};
use s3s::dto::{
DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
ExistingObjectReplication, ExistingObjectReplicationStatus, ReplicaModifications, ReplicaModificationsStatus,
ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, SourceSelectionCriteria,
DeleteMarkerReplication, DeleteMarkerReplicationStatus, Destination, ExistingObjectReplication,
ExistingObjectReplicationStatus, ReplicaModifications, ReplicaModificationsStatus, ReplicationConfiguration,
ReplicationRule, ReplicationRuleStatus, SourceSelectionCriteria,
};
use std::collections::HashMap;
use time::{Duration, OffsetDateTime};
@@ -433,9 +433,7 @@ mod tests {
delete_marker_replication: Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
}),
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
}),
delete_replication: None,
destination: Destination {
bucket: arn.to_string(),
..Default::default()
@@ -515,7 +513,7 @@ mod tests {
};
let state = delete_replication_state_from_config(&config, &source)
.expect("delete-marker version purge should honor delete replication rules");
.expect("delete-marker version purge should honor delete-marker replication rules");
let pending = format!("{arn}=PENDING;");
assert_eq!(state.version_purge_status_internal.as_deref(), Some(pending.as_str()));
-35
View File
@@ -179,20 +179,6 @@ impl RemoteTargetRequest {
return Err(s3_error!(InvalidRequest, "credentials.secretKey is required"));
}
for (unsupported, configured) in [
("disableProxy", self.disable_proxy),
("healthCheckDuration", self.health_check_duration != 0),
("edge", self.edge),
("edgeSyncBeforeExpiry", self.edge_sync_before_expiry),
] {
if configured {
return Err(s3_error!(
InvalidRequest,
"remote target field {unsupported} is not supported by this RustFS version"
));
}
}
Ok(BucketTarget {
source_bucket: self.source_bucket,
endpoint: self.endpoint,
@@ -1218,27 +1204,6 @@ mod tests {
assert!(err.to_string().contains("credentials.secretKey is required"));
}
#[test]
fn remote_target_request_rejects_unimplemented_fields() {
for (field, value) in [
("disableProxy", serde_json::json!(true)),
("healthCheckDuration", serde_json::json!(5)),
("edge", serde_json::json!(true)),
("edgeSyncBeforeExpiry", serde_json::json!(true)),
] {
let mut request = valid_remote_target_request();
request[field] = value;
let request: RemoteTargetRequest =
serde_json::from_value(request).expect("unsupported field should still deserialize");
let err = request
.into_bucket_target()
.expect_err("unimplemented remote target fields must not be persisted");
assert!(err.to_string().contains(field));
assert!(err.to_string().contains("not supported by this RustFS version"));
}
}
#[test]
fn remote_target_request_converts_to_bucket_target() {
let target = serde_json::from_value::<RemoteTargetRequest>(valid_remote_target_request())
+1 -32
View File
@@ -35,7 +35,7 @@ use super::storage_api::bucket_usecase::bucket::{
policy_sys::PolicySys,
replication::{
ReplicationTargetValidationError, replication_target_arns, should_remove_replication_target,
unsupported_replication_config_field, validate_replication_config_target_arns,
validate_replication_config_target_arns,
},
target::{BucketTargetType, BucketTargets},
utils::serialize,
@@ -556,16 +556,6 @@ fn validate_replication_config_targets(targets: &BucketTargets, config: &Replica
}
}
fn validate_replication_config_capabilities(config: &ReplicationConfiguration) -> S3Result<()> {
if let Some(field) = unsupported_replication_config_field(config) {
return Err(S3Error::with_message(
S3ErrorCode::InvalidRequest,
format!("replication field {field} is not supported by this RustFS version"),
));
}
Ok(())
}
async fn validate_bucket_replication_update(bucket: &str, config: &ReplicationConfiguration) -> S3Result<()> {
if !BucketVersioningSys::enabled(bucket).await {
return Err(s3_error!(
@@ -574,8 +564,6 @@ async fn validate_bucket_replication_update(bucket: &str, config: &ReplicationCo
));
}
validate_replication_config_capabilities(config)?;
let targets = metadata_sys::get_bucket_targets_config(bucket)
.await
.map_err(|err| match err {
@@ -3087,25 +3075,6 @@ mod tests {
validate_replication_config_targets(&targets, &config).expect("disabled rules should not require live targets");
}
#[test]
fn validate_replication_config_capabilities_names_unsupported_field() {
let mut rule = replication_rule_for_target("arn:rustfs:replication:us-east-1:target:bucket");
rule.destination.encryption_configuration = Some(s3s::dto::EncryptionConfiguration::default());
let config = ReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
let err = validate_replication_config_capabilities(&config)
.expect_err("destination encryption must be rejected until the execution path supports it");
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
assert!(
err.to_string()
.contains("Destination.EncryptionConfiguration is not supported")
);
}
#[test]
fn remove_replication_targets_from_config_targets_only_removes_referenced_replication_targets() {
let removed_arn = "arn:rustfs:replication:us-east-1:removed:bucket";
+76 -50
View File
@@ -44,8 +44,8 @@ use super::storage_api::object_usecase::bucket::{
REPLICATE_INCOMING_DELETE, ReplicationStatusType, VersionPurgeStatusType, check_replicate_delete,
delete_replication_state_from_config, delete_replication_version_id, deleted_object_has_pending_replication_delete,
must_replicate_object, schedule_object_replication, schedule_replication_delete, set_deleted_object_replication_state,
set_object_to_delete_version_purge_status, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source,
set_object_to_delete_version_purge_status, should_schedule_delete_replication,
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
},
tagging::decode_tags,
validate_restore_request,
@@ -6691,7 +6691,7 @@ impl DefaultObjectUsecase {
// the same early, advisory rejection as before.
let store_ref = &store;
let bucket_ref = bucket.as_str();
let admitted_deletes: Vec<Result<AdmittedDelete, ApiError>> =
let admitted_deletes: Vec<AdmittedDelete> =
futures::stream::iter(prepared_deletes.into_iter().map(|prepared| async move {
let PreparedDelete {
idx,
@@ -6716,7 +6716,7 @@ impl DefaultObjectUsecase {
&& let Some(block_reason) = check_object_lock_for_deletion(bucket_ref, &goi, bypass_governance).await
{
let blocked_key = object.object_name.clone();
return Ok(AdmittedDelete {
return AdmittedDelete {
idx,
object,
size: 0,
@@ -6727,11 +6727,15 @@ impl DefaultObjectUsecase {
message: Some(block_reason.error_message()),
version_id,
}),
});
};
}
let size = goi.size;
if is_dir_object(&object.object_name) && object.version_id.is_none() {
object.version_id = Some(Uuid::nil());
}
if replicate_deletes {
let dsc = check_replicate_delete(
bucket_ref,
@@ -6744,8 +6748,7 @@ impl DefaultObjectUsecase {
&opts,
gerr.clone(),
)
.await
.map_err(ApiError::from)?;
.await;
if dsc.replicate_any() {
if object.version_id.is_some() {
set_object_to_delete_version_purge_status(&mut object, VersionPurgeStatusType::Pending);
@@ -6757,23 +6760,18 @@ impl DefaultObjectUsecase {
}
}
if is_dir_object(&object.object_name) && object.version_id.is_none() {
object.version_id = Some(Uuid::nil());
}
let existing = (!skip_stat && gerr.is_none()).then_some(goi);
Ok(AdmittedDelete {
AdmittedDelete {
idx,
object,
size,
existing,
blocked: None,
})
}
}))
.buffered(DELETE_OBJECTS_PRE_STAT_CONCURRENCY)
.collect()
.await;
let admitted_deletes: Vec<AdmittedDelete> = admitted_deletes.into_iter().collect::<Result<_, ApiError>>()?;
// Phase 3 (serial): apply outcomes in the original request order so
// per-key success/failure reporting is unchanged.
@@ -7091,28 +7089,6 @@ impl DefaultObjectUsecase {
}
};
let delete_replication_state = if !replica && !force_delete && (opts.versioned || opts.version_suspended) {
let fallback_source;
let source = if let Some(source) = existing_object_info.as_ref() {
source
} else {
fallback_source = ObjectInfo {
name: key.clone(),
..Default::default()
};
&fallback_source
};
match metadata_sys::get_replication_config(&bucket).await {
Ok((config, _)) => {
delete_replication_state_from_config(&config, source, version_id_clone.as_ref().and(source.version_id), false)
}
Err(StorageError::ConfigNotFound) => None,
Err(err) => return Err(ApiError::from(err).into()),
}
} else {
None
};
let cache_adapter = self.object_data_cache();
// A force (delete_prefix) delete removes every object under `key` as a
// prefix, so invalidating only the exact key would strand every cached
@@ -7208,7 +7184,7 @@ impl DefaultObjectUsecase {
let schedule_delete_replication = if opts.replication_request && replica {
should_schedule_replica_delete_replication(&bucket, replication_state_source, delete_replication_version_id).await
} else {
delete_replication_state.is_some()
should_schedule_delete_replication(&opts, deleted_object_source, deleted_delete_marker_version)
};
if schedule_delete_replication {
@@ -7230,12 +7206,8 @@ impl DefaultObjectUsecase {
replication_state: None,
..Default::default()
};
if let Some(state) = delete_replication_state.as_ref() {
set_deleted_object_replication_state(&mut deleted_object, state);
} else {
set_deleted_object_replication_state(&mut deleted_object, &replication_state_source.replication_state());
enrich_delete_replication_state_if_needed(&bucket, &mut deleted_object, replication_state_source).await;
}
set_deleted_object_replication_state(&mut deleted_object, &replication_state_source.replication_state());
enrich_delete_replication_state_if_needed(&bucket, &mut deleted_object, replication_state_source).await;
schedule_replication_delete(deleted_object, bucket.clone(), REPLICATE_INCOMING_DELETE.to_string()).await;
}
@@ -8373,10 +8345,9 @@ mod tests {
use super::*;
use http::{Extensions, HeaderMap, HeaderName, HeaderValue, Method, Uri};
use s3s::dto::{
Delete, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
ExistingObjectReplication, ExistingObjectReplicationStatus, ObjectIdentifier, ReplicaModifications,
ReplicaModificationsStatus, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, RestoreRequest,
SourceSelectionCriteria,
Delete, DeleteMarkerReplication, DeleteMarkerReplicationStatus, Destination, ExistingObjectReplication,
ExistingObjectReplicationStatus, ObjectIdentifier, ReplicaModifications, ReplicaModificationsStatus,
ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, RestoreRequest, SourceSelectionCriteria,
};
use std::pin::Pin;
use std::sync::Arc;
@@ -13384,6 +13355,63 @@ mod tests {
assert!(!can_skip_delete_objects_pre_stat(false, false, &delete_marker_creating_opts(), false));
}
#[test]
fn should_schedule_delete_replication_skips_replica_requests() {
let opts = ObjectOptions {
replication_request: true,
version_id: Some(Uuid::new_v4().to_string()),
..Default::default()
};
let replication_source = ObjectInfo {
delete_marker: true,
replication_status: ReplicationStatusType::Completed,
..Default::default()
};
assert!(
!should_schedule_delete_replication(&opts, &replication_source, true),
"replica delete requests on target sites must not enqueue a second replication delete task"
);
}
#[test]
fn should_schedule_delete_replication_keeps_delete_marker_version_purge_from_source() {
let opts = ObjectOptions {
replication_request: false,
version_id: Some(Uuid::new_v4().to_string()),
..Default::default()
};
let replication_source = ObjectInfo {
delete_marker: true,
replication_status: ReplicationStatusType::Completed,
..Default::default()
};
assert!(
should_schedule_delete_replication(&opts, &replication_source, true),
"source-side delete-marker version purge still needs replication scheduling"
);
}
#[test]
fn should_schedule_delete_replication_keeps_object_version_purge_from_completed_source() {
let opts = ObjectOptions {
replication_request: false,
version_id: Some(Uuid::new_v4().to_string()),
..Default::default()
};
let replication_source = ObjectInfo {
delete_marker: false,
replication_status: ReplicationStatusType::Completed,
..Default::default()
};
assert!(
should_schedule_delete_replication(&opts, &replication_source, false),
"source-side object version purge must still enqueue delete replication after the original PUT completed"
);
}
#[tokio::test]
#[ignore = "requires isolated global object layer state"]
async fn execute_get_object_attributes_returns_internal_error_when_store_uninitialized() {
@@ -13648,9 +13676,7 @@ mod tests {
delete_marker_replication: Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
}),
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
}),
delete_replication: None,
destination: Destination {
bucket: arn.clone(),
..Default::default()
+17 -6
View File
@@ -649,8 +649,8 @@ pub(crate) mod bucket {
oi: &crate::storage::storage_api::StorageObjectInfo,
del_opts: &crate::storage::storage_api::StorageObjectOptions,
gerr: Option<String>,
) -> Result<ReplicateDecision, crate::storage::storage_api::StorageError> {
ReplicationObjectBridge::check_delete_strict(bucket, dobj, oi, del_opts, gerr).await
) -> ReplicateDecision {
ReplicationObjectBridge::check_delete(bucket, dobj, oi, del_opts, gerr).await
}
pub(crate) fn delete_replication_version_id(
@@ -751,6 +751,21 @@ pub(crate) mod bucket {
replication_contracts::should_remove_replication_target(target_arn, is_replication_service, target_arns)
}
pub(crate) fn should_schedule_delete_replication(
opts: &crate::storage::storage_api::StorageObjectOptions,
replication_source: &crate::storage::storage_api::StorageObjectInfo,
deleted_delete_marker_version: bool,
) -> bool {
replication_contracts::should_schedule_delete_replication(replication_contracts::ReplicationDeleteScheduleInput {
replication_request: opts.replication_request,
version_id_requested: opts.version_id.is_some(),
source_delete_marker: replication_source.delete_marker,
source_replication_status: &replication_source.replication_status,
source_version_purge_status: &replication_source.version_purge_status,
deleted_delete_marker_version,
})
}
pub(crate) fn should_use_existing_delete_replication_info(
opts: &crate::storage::storage_api::StorageObjectOptions,
) -> bool {
@@ -775,10 +790,6 @@ pub(crate) mod bucket {
) -> Result<(), ReplicationTargetValidationError> {
replication_contracts::validate_replication_config_target_arns(configured_arns, config)
}
pub(crate) fn unsupported_replication_config_field(config: &s3s::dto::ReplicationConfiguration) -> Option<&'static str> {
replication_contracts::unsupported_replication_config_field(config)
}
}
pub(crate) mod tagging {
+1 -36
View File
@@ -12,32 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc"))]
use std::alloc::{GlobalAlloc, Layout};
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc"))]
#[derive(Default)]
struct DefaultMiMalloc;
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc"))]
// SAFETY: allocation and deallocation are forwarded unchanged to MiMalloc, so
// MiMalloc's GlobalAlloc guarantees apply to every returned pointer and layout.
#[allow(unsafe_code)]
unsafe impl GlobalAlloc for DefaultMiMalloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
// SAFETY: the caller upholds GlobalAlloc's contract for layout.
unsafe { mimalloc::MiMalloc.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
// SAFETY: ptr and layout came from this allocator and are forwarded unchanged.
unsafe { mimalloc::MiMalloc.dealloc(ptr, layout) }
}
}
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc"))]
#[global_allocator]
static GLOBAL: hotpath::CountingAllocator<DefaultMiMalloc> = hotpath::CountingAllocator::new();
static GLOBAL: hotpath::CountingAllocator = hotpath::CountingAllocator::new();
#[cfg(not(all(feature = "hotpath", feature = "hotpath-alloc")))]
#[global_allocator]
@@ -48,15 +25,3 @@ fn main() {
rustfs::startup_entrypoint::run_process();
}
#[cfg(all(test, feature = "hotpath", feature = "hotpath-alloc"))]
mod tests {
#[test]
#[allow(unsafe_code)]
fn hotpath_allocator_uses_mimalloc() {
let allocation = Box::new([0_u8; 64]);
// SAFETY: the live Box pointer is valid to inspect for heap ownership.
assert!(unsafe { libmimalloc_sys::mi_is_in_heap_region(allocation.as_ptr().cast()) });
}
}