mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-22 20:36:38 +00:00
fix: address rc.1 release blockers (#5648)
* fix: address rc.1 release blockers
* fix: route release guards through architecture boundaries
* fix: close remaining rc.1 regression gaps
* refactor: group multipart listing options
* fix: resolve rc.1 CI regressions
* fix(ecstore): keep bucket-config writes off the caller's stack
A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.
Two CI lanes aborted with SIGABRT on this:
ILM Integration (serial)
rustfs app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions
Test and Lint (swift)
rustfs-protocols::swift_metadata_persistence::
swift_metadata_writes_are_durable
Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.
Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.
* fix(lifecycle): unbreak restore under the bucket generation fence
The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.
1. RestoreObject and ListMultipartUploads hard-required
`opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
deliberately leaves it unset when no guard extension is present — only the
S3 access layer installs one. Every direct caller therefore got
`InternalError: ... bucket generation guard is missing`. Resolve the
current generation instead, the way the copy path already does. The fence
is unaffected: RestoreObject still re-reads the incarnation from disk and
compares before admitting the restore, and the multipart listing is
filtered by the value it resolves.
2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
restored-copy expiry whose `restore_expires` had not already elapsed.
Whether the restored copy is due to expire is the ILM evaluator's
decision, made when it emitted DeleteRestoredAction; re-deriving it in
the set layer only adds a way for a legitimate action to be rejected.
The stale-event risk it appears to guard is already covered by the
surrounding snapshot match — a re-restore rewrites `restore_expires`,
so a replayed event fails the equality check. Drop the clause; the
fifteen identity clauses are unchanged.
Fixed:
rustfs app::lifecycle_transition_api_test::
restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
restore_object_usecase_completes_suspended_null_version_in_place
restore_object_usecase_reports_ongoing_conflict
rustfs-scanner::lifecycle_integration_test serial_tests::
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore
Verification: the CI ILM lane filter now runs 53/53 green locally.
* chore: address review follow-ups on this branch
Four items from the adversarial review that were still open.
- Restore the assertion `test_bucket_replication_replayed_delete_marker_
preserves_source_mtime_without_source_restart` is named for. The branch
had replaced the backlog#867 mtime check with `assert_replication_
converged`, which any successful replication satisfies, and deleted the
two helpers it needed — so the regression the test exists to catch would
now pass. This matters here specifically because the branch changes the
flag feeding `replication_delete_remove_options` and routes replay
through a new file and ordering.
- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
one real consumer calls the `_with_metadata` variant directly). Its test
stanza now exercises that variant, so the coverage moves to live code
rather than being deleted.
- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
2.7.0 before this branch and is untouched, so the caret range already
resolved there. Nothing in the diff uses the crate.
- Split the AGENTS.md "Adversarial Validation" policy change out of this
branch. The edit is defensible on its own, but it relaxes the review gate
that this branch has to pass, so it should land as its own PR reviewed on
its own merits rather than bundled with the change that benefits from it.
The reverted hunks are unchanged and ready to re-apply.
Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.
* test(ci): serialize the new bucket-incarnation tests
The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.
Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.
Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.
* test(ecstore): pin that a drained bucket is actually deletable
`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.
That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.
Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.
The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.
* fix(ecstore): let an explicit null-version delete purge its delete marker
Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.
On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:
explicit null-version purge of the null delete marker must succeed,
got [Some(MethodNotAllowed)]
so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.
The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.
This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.
Normalize both sides through `delete_file_info_version_id`.
The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.
* fix(app): drop the now-needless struct update
Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.
Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.
* test(e2e): assert the fresh-volume classification
four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.
Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.
* fix(bucket): apply the requested Object Lock to existing buckets
Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.
Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.
An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.
Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".
cargo nextest run -p rustfs-ecstore --lib: 3633 passed.
* fix(ecstore): box the generation-checked config mutation paths too
The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.
* fix(restore): keep the nil-version normalization the strip removed
Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:
- self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
+ self.version_id == fi.version_id
For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.
Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
This commit is contained in:
@@ -20,7 +20,7 @@ use crate::bucket::{
|
||||
use crate::error::is_err_bucket_not_found;
|
||||
use crate::runtime::sources as runtime_sources;
|
||||
use crate::set_disk::get_lock_acquire_timeout;
|
||||
use crate::storage_api_contracts::bucket::SRBucketDeleteOp;
|
||||
use crate::storage_api_contracts::bucket::{BUCKET_LIFECYCLE_LOCK_OBJECT, SRBucketDeleteOp};
|
||||
use crate::storage_api_contracts::namespace::NamespaceLocking as _;
|
||||
use futures::stream::{self, StreamExt};
|
||||
use std::collections::BTreeMap;
|
||||
@@ -113,6 +113,25 @@ where
|
||||
}
|
||||
}
|
||||
|
||||
async fn await_bucket_lifecycle_operation<T, F>(
|
||||
lifecycle_guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
namespace_guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
bucket: &str,
|
||||
operation: &'static str,
|
||||
future: F,
|
||||
) -> Result<T>
|
||||
where
|
||||
F: Future<Output = Result<T>>,
|
||||
{
|
||||
await_bucket_namespace_operation(
|
||||
lifecycle_guard,
|
||||
bucket,
|
||||
operation,
|
||||
await_bucket_namespace_operation(namespace_guard, bucket, operation, future),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn run_bucket_usage_cleanup<F>(guard: Option<&rustfs_lock::NamespaceLockGuard>, bucket: &str, future: F) -> Result<()>
|
||||
where
|
||||
F: Future<Output = Result<()>>,
|
||||
@@ -131,6 +150,50 @@ where
|
||||
}
|
||||
|
||||
impl ECStore {
|
||||
pub async fn bucket_incarnation_id(&self, bucket: &str) -> Result<Uuid> {
|
||||
metadata_sys::get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await
|
||||
}
|
||||
|
||||
pub async fn bucket_incarnation_id_from_disk(&self, bucket: &str) -> Result<Uuid> {
|
||||
metadata_sys::get_bucket_incarnation_id_in(&self.ctx, bucket).await
|
||||
}
|
||||
|
||||
/// The object commit path acquires this sentinel before bucket metadata and
|
||||
/// exact-object namespace locks.
|
||||
pub(crate) async fn acquire_bucket_lifecycle_read_lock(&self, bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
|
||||
let lock = self.new_ns_lock(bucket, BUCKET_LIFECYCLE_LOCK_OBJECT).await?;
|
||||
lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(|err| match err {
|
||||
rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
|
||||
StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "bucket_lifecycle_read",
|
||||
bucket: bucket.to_string(),
|
||||
object: BUCKET_LIFECYCLE_LOCK_OBJECT.to_string(),
|
||||
required,
|
||||
achieved,
|
||||
}
|
||||
}
|
||||
other => StorageError::Lock(other),
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) async fn acquire_bucket_lifecycle_write_lock(&self, bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard> {
|
||||
let lock = self.new_ns_lock(bucket, BUCKET_LIFECYCLE_LOCK_OBJECT).await?;
|
||||
lock.get_write_lock(get_lock_acquire_timeout())
|
||||
.await
|
||||
.map_err(|err| match err {
|
||||
rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
|
||||
StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "bucket_lifecycle_write",
|
||||
bucket: bucket.to_string(),
|
||||
object: BUCKET_LIFECYCLE_LOCK_OBJECT.to_string(),
|
||||
required,
|
||||
achieved,
|
||||
}
|
||||
}
|
||||
other => StorageError::Lock(other),
|
||||
})
|
||||
}
|
||||
|
||||
async fn mark_bucket_deleted(&self, bucket: &str) -> Result<()> {
|
||||
let marker_volume = bucket_deleted_marker_volume(bucket);
|
||||
|
||||
@@ -204,25 +267,39 @@ impl ECStore {
|
||||
}
|
||||
}
|
||||
|
||||
async fn rollback_failed_bucket_creation(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) {
|
||||
async fn rollback_failed_bucket_creation(
|
||||
&self,
|
||||
bucket: &str,
|
||||
lifecycle_guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
namespace_guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
) {
|
||||
let rollback_opts = DeleteBucketOptions {
|
||||
no_lock: true,
|
||||
no_recreate: true,
|
||||
..Default::default()
|
||||
};
|
||||
if let Err(err) = await_bucket_namespace_operation(guard, bucket, "failed bucket creation rollback", async {
|
||||
self.peer_sys
|
||||
.delete_bucket(bucket, &rollback_opts)
|
||||
.await
|
||||
.map_err(|rollback_err| to_object_err(rollback_err.into(), vec![bucket]))
|
||||
})
|
||||
.await
|
||||
if let Err(err) =
|
||||
await_bucket_lifecycle_operation(lifecycle_guard, namespace_guard, bucket, "failed bucket creation rollback", async {
|
||||
self.peer_sys
|
||||
.delete_bucket(bucket, &rollback_opts)
|
||||
.await
|
||||
.map_err(|rollback_err| to_object_err(rollback_err.into(), vec![bucket]))
|
||||
})
|
||||
.await
|
||||
{
|
||||
warn!(
|
||||
bucket = %bucket,
|
||||
error = ?err,
|
||||
"failed bucket creation rollback did not remove every physical bucket volume"
|
||||
);
|
||||
return;
|
||||
}
|
||||
if let Err(err) = self.cleanup_deleted_bucket_metadata(bucket, false, namespace_guard).await {
|
||||
warn!(
|
||||
bucket = %bucket,
|
||||
error = ?err,
|
||||
"failed bucket creation rollback left internal bucket metadata"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -234,40 +311,112 @@ impl ECStore {
|
||||
return Err(StorageError::BucketNameInvalid(err.to_string()));
|
||||
}
|
||||
|
||||
// Lock order: bucket lifecycle -> metadata transaction -> exact bucket namespace.
|
||||
let bucket_lifecycle_guard = if !opts.no_lock {
|
||||
Some(self.acquire_bucket_lifecycle_write_lock(bucket).await?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let metadata_transaction_guard = if !opts.no_lock && !is_meta_bucketname(bucket) {
|
||||
Some(metadata_sys::acquire_bucket_metadata_transaction_lock_in(&self.ctx, bucket).await?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let ns_guard = if !opts.no_lock {
|
||||
let ns_lock = self.new_ns_lock(bucket, bucket).await?;
|
||||
Some(
|
||||
ns_lock
|
||||
.get_write_lock(get_lock_acquire_timeout())
|
||||
.await
|
||||
.map_err(|e| match e {
|
||||
rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
|
||||
StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "write",
|
||||
bucket: bucket.to_string(),
|
||||
object: bucket.to_string(),
|
||||
required,
|
||||
achieved,
|
||||
await_bucket_namespace_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
bucket,
|
||||
"bucket namespace lock acquisition",
|
||||
async {
|
||||
ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| match e {
|
||||
rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
|
||||
StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "write",
|
||||
bucket: bucket.to_string(),
|
||||
object: bucket.to_string(),
|
||||
required,
|
||||
achieved,
|
||||
}
|
||||
}
|
||||
}
|
||||
other => StorageError::Lock(other),
|
||||
})?,
|
||||
other => StorageError::Lock(other),
|
||||
})
|
||||
},
|
||||
)
|
||||
.await?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let confirmed_missing = match self.peer_sys.get_bucket_info(bucket, &BucketOptions::default()).await {
|
||||
Ok(_) => false,
|
||||
let existing_bucket_info = match self.peer_sys.get_bucket_info(bucket, &BucketOptions::default()).await {
|
||||
Ok(info) => Some(info),
|
||||
Err(err) => {
|
||||
let err: StorageError = err.into();
|
||||
if is_err_bucket_not_found(&err) {
|
||||
true
|
||||
None
|
||||
} else {
|
||||
return Err(to_object_err(err, vec![bucket]));
|
||||
}
|
||||
}
|
||||
};
|
||||
let confirmed_missing = existing_bucket_info.is_none();
|
||||
let existing_metadata = if opts.force_create && !confirmed_missing && !is_meta_bucketname(bucket) {
|
||||
let (mut metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&self.ctx, bucket).await?;
|
||||
if !persisted {
|
||||
metadata = BucketMetadata::new(bucket);
|
||||
metadata.created = existing_bucket_info
|
||||
.as_ref()
|
||||
.and_then(|info| info.created)
|
||||
.unwrap_or(OffsetDateTime::UNIX_EPOCH);
|
||||
} else if !metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() {
|
||||
return Err(Error::other(format!(
|
||||
"bucket incarnation sidecar is missing for new-format metadata: {bucket}"
|
||||
)));
|
||||
} else if !metadata.bucket_incarnation_sidecar {
|
||||
metadata.bucket_incarnation_id = Uuid::new_v4();
|
||||
}
|
||||
Some(metadata)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let mut meta = existing_metadata.unwrap_or_else(|| BucketMetadata::new(bucket));
|
||||
let existing_incarnation_is_authoritative = meta.bucket_incarnation_sidecar;
|
||||
if confirmed_missing || is_meta_bucketname(bucket) {
|
||||
meta.set_created(opts.created_at);
|
||||
|
||||
if opts.versioning_enabled {
|
||||
meta.versioning_config_xml =
|
||||
crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
|
||||
}
|
||||
}
|
||||
|
||||
// Object Lock enable is one-way, and it must apply to an existing bucket
|
||||
// too. Site replication replays make-with-versioning carrying the
|
||||
// source's lockEnabled against a destination bucket that already exists;
|
||||
// gating this on `confirmed_missing` returned success while leaving the
|
||||
// replica unlocked, so replicated versions could be deleted without the
|
||||
// retention the source enforces.
|
||||
let lock_newly_enabled = opts.lock_enabled && !meta.lock_enabled;
|
||||
if opts.lock_enabled {
|
||||
meta.lock_enabled = true;
|
||||
meta.object_lock_config_xml =
|
||||
crate::bucket::utils::serialize::<ObjectLockConfiguration>(&ENABLED_OBJECT_LOCK_CONFIG)?;
|
||||
meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
|
||||
}
|
||||
|
||||
let metadata_persisted_before_physical = confirmed_missing && !is_meta_bucketname(bucket) && opts.lock_enabled;
|
||||
if metadata_persisted_before_physical {
|
||||
metadata_sys::set_new_bucket_metadata_in(&self.ctx, meta.clone()).await?;
|
||||
if bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
{
|
||||
return Err(Error::other(format!("bucket metadata creation intent lock was lost: {bucket}")));
|
||||
}
|
||||
}
|
||||
|
||||
if confirmed_missing && !is_meta_bucketname(bucket) {
|
||||
// Fence every scanner cycle that could have observed the namespace
|
||||
@@ -277,12 +426,23 @@ impl ECStore {
|
||||
self.cleanup_bucket_usage(bucket, ns_guard.as_ref()).await?;
|
||||
}
|
||||
|
||||
if let Err(err) = await_bucket_namespace_operation(ns_guard.as_ref(), bucket, "physical bucket creation", async {
|
||||
self.peer_sys
|
||||
.make_bucket(bucket, opts)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket]))
|
||||
})
|
||||
if let Err(err) = await_bucket_lifecycle_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
ns_guard.as_ref(),
|
||||
bucket,
|
||||
"physical bucket creation",
|
||||
await_bucket_namespace_operation(
|
||||
metadata_transaction_guard.as_ref(),
|
||||
bucket,
|
||||
"bucket creation metadata transaction",
|
||||
async {
|
||||
self.peer_sys
|
||||
.make_bucket(bucket, opts)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket]))
|
||||
},
|
||||
),
|
||||
)
|
||||
.await
|
||||
{
|
||||
if is_err_bucket_exists(&err)
|
||||
@@ -300,32 +460,38 @@ impl ECStore {
|
||||
}
|
||||
if !is_err_bucket_exists(&err) && ns_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) {
|
||||
error!("make bucket failed: {err}");
|
||||
self.rollback_failed_bucket_creation(bucket, ns_guard.as_ref()).await;
|
||||
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_ref())
|
||||
.await;
|
||||
}
|
||||
return Err(err);
|
||||
};
|
||||
|
||||
let mut meta = BucketMetadata::new(bucket);
|
||||
|
||||
meta.set_created(opts.created_at);
|
||||
|
||||
if opts.lock_enabled {
|
||||
meta.object_lock_config_xml =
|
||||
crate::bucket::utils::serialize::<ObjectLockConfiguration>(&ENABLED_OBJECT_LOCK_CONFIG)?;
|
||||
meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
|
||||
let metadata_result = async {
|
||||
if metadata_persisted_before_physical {
|
||||
return Ok(());
|
||||
}
|
||||
if is_meta_bucketname(bucket) {
|
||||
metadata_sys::set_bucket_metadata_in(&self.ctx, meta).await
|
||||
} else if existing_incarnation_is_authoritative && !lock_newly_enabled {
|
||||
metadata_sys::cache_bucket_metadata_in(&self.ctx, meta).await
|
||||
} else {
|
||||
metadata_sys::set_new_bucket_metadata_in(&self.ctx, meta).await
|
||||
}
|
||||
}
|
||||
|
||||
if opts.versioning_enabled {
|
||||
meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
|
||||
.await;
|
||||
let metadata_lock_lost = bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost());
|
||||
if let Err(err) = metadata_result {
|
||||
if !metadata_lock_lost {
|
||||
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_ref())
|
||||
.await;
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
if metadata_lock_lost {
|
||||
return Err(Error::other(format!("bucket metadata initialization lock was lost: {bucket}")));
|
||||
}
|
||||
|
||||
await_bucket_namespace_operation(
|
||||
ns_guard.as_ref(),
|
||||
bucket,
|
||||
"bucket metadata initialization",
|
||||
metadata_sys::set_bucket_metadata_in(&self.ctx, meta),
|
||||
)
|
||||
.await?;
|
||||
|
||||
if confirmed_missing && !is_meta_bucketname(bucket) {
|
||||
// A scanner may have sampled the first fence before the bucket
|
||||
@@ -454,24 +620,34 @@ impl ECStore {
|
||||
return Err(StorageError::BucketNameInvalid(err.to_string()));
|
||||
}
|
||||
|
||||
let bucket_lifecycle_guard = if !opts.no_lock {
|
||||
Some(self.acquire_bucket_lifecycle_write_lock(bucket).await?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let ns_guard = if !opts.no_lock {
|
||||
let ns_lock = self.new_ns_lock(bucket, bucket).await?;
|
||||
Some(
|
||||
ns_lock
|
||||
.get_write_lock(get_lock_acquire_timeout())
|
||||
.await
|
||||
.map_err(|e| match e {
|
||||
rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
|
||||
StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "write",
|
||||
bucket: bucket.to_string(),
|
||||
object: bucket.to_string(),
|
||||
required,
|
||||
achieved,
|
||||
await_bucket_namespace_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
bucket,
|
||||
"bucket namespace lock acquisition",
|
||||
async {
|
||||
ns_lock.get_write_lock(get_lock_acquire_timeout()).await.map_err(|e| match e {
|
||||
rustfs_lock::error::LockError::QuorumNotReached { required, achieved } => {
|
||||
StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "write",
|
||||
bucket: bucket.to_string(),
|
||||
object: bucket.to_string(),
|
||||
required,
|
||||
achieved,
|
||||
}
|
||||
}
|
||||
}
|
||||
other => StorageError::Lock(other),
|
||||
})?,
|
||||
other => StorageError::Lock(other),
|
||||
})
|
||||
},
|
||||
)
|
||||
.await?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
@@ -519,7 +695,8 @@ impl ECStore {
|
||||
}
|
||||
|
||||
if sr_mark_delete {
|
||||
await_bucket_namespace_operation(
|
||||
await_bucket_lifecycle_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
ns_guard.as_ref(),
|
||||
bucket,
|
||||
"bucket delete marker creation",
|
||||
@@ -528,12 +705,17 @@ impl ECStore {
|
||||
.await?;
|
||||
}
|
||||
|
||||
let delete_result = run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
|
||||
self.peer_sys
|
||||
.delete_bucket(bucket, &delete_opts)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket]))
|
||||
})
|
||||
let delete_result = await_bucket_namespace_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
bucket,
|
||||
"physical bucket deletion",
|
||||
run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
|
||||
self.peer_sys
|
||||
.delete_bucket(bucket, &delete_opts)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err.into(), vec![bucket]))
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
if let Err(err) = delete_result
|
||||
&& (!sr_delete || !is_err_strict_volume_not_found(&err))
|
||||
@@ -591,7 +773,6 @@ mod tests {
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, SystemTime};
|
||||
use time::OffsetDateTime;
|
||||
use tokio::io::AsyncReadExt;
|
||||
use tokio::sync::{Notify, OnceCell};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use uuid::Uuid;
|
||||
@@ -1229,6 +1410,86 @@ mod tests {
|
||||
assert!(!any_disk_path_exists(&disk_paths, bucket_deleted_marker_volume(&bucket)).await);
|
||||
}
|
||||
|
||||
/// Site replication replays make-with-versioning carrying the source's
|
||||
/// `lockEnabled` against a destination bucket that already exists. Gating the
|
||||
/// lock enable on `confirmed_missing` returned success while leaving the
|
||||
/// replica unlocked, so replicated versions could be deleted without the
|
||||
/// retention the source enforces.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
#[serial]
|
||||
async fn force_create_enables_object_lock_on_an_existing_bucket() {
|
||||
let (_disk_paths, ecstore) = setup_bucket_delete_test_env().await;
|
||||
let bucket = format!("bucket-force-lock-{}", Uuid::new_v4().simple());
|
||||
|
||||
ecstore
|
||||
.make_bucket(&bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("plain bucket should be created");
|
||||
assert!(
|
||||
!metadata_sys::get_in(&ecstore.ctx, &bucket)
|
||||
.await
|
||||
.expect("metadata should load")
|
||||
.lock_enabled,
|
||||
"test setup: the bucket must start unlocked"
|
||||
);
|
||||
|
||||
ecstore
|
||||
.make_bucket(
|
||||
&bucket,
|
||||
&MakeBucketOptions {
|
||||
force_create: true,
|
||||
lock_enabled: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("force create with lock_enabled should succeed on an existing bucket");
|
||||
|
||||
let meta = metadata_sys::get_in(&ecstore.ctx, &bucket)
|
||||
.await
|
||||
.expect("metadata should load after the lock enable");
|
||||
assert!(meta.lock_enabled, "Object Lock must be enabled on the existing bucket");
|
||||
assert!(
|
||||
!meta.object_lock_config_xml.is_empty(),
|
||||
"the Object Lock configuration must be persisted, not just the flag"
|
||||
);
|
||||
assert!(
|
||||
!meta.versioning_config_xml.is_empty(),
|
||||
"Object Lock requires versioning, so that must be persisted too"
|
||||
);
|
||||
}
|
||||
|
||||
/// `DeleteBucket`'s emptiness check is a raw disk scan (`has_xlmeta_files`),
|
||||
/// not an S3-level listing, so "the client drained the bucket" and "the
|
||||
/// bucket is deletable" are two different contracts. Nothing pinned the
|
||||
/// second one, which is how the s3-tests lane ended up failing 219 cases on
|
||||
/// `nuke_prefixed_buckets` while every test body passed.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
#[serial]
|
||||
async fn bucket_delete_succeeds_after_the_last_object_version_is_deleted() {
|
||||
let (disk_paths, ecstore) = setup_bucket_delete_test_env().await;
|
||||
let bucket = format!("bucket-delete-after-drain-{}", Uuid::new_v4().simple());
|
||||
let object = "object.txt";
|
||||
|
||||
create_bucket_with_object(&ecstore, &bucket, object).await;
|
||||
|
||||
ecstore
|
||||
.delete_object(&bucket, object, ObjectOptions::default())
|
||||
.await
|
||||
.expect("client delete of the only object should succeed");
|
||||
|
||||
assert!(
|
||||
!any_disk_has_object_metadata(&disk_paths, &bucket).await,
|
||||
"deleting the last version must not leave xl.meta on disk: DeleteBucket scans the raw \
|
||||
bucket directory, so residue here is reported to clients as BucketNotEmpty"
|
||||
);
|
||||
|
||||
ecstore
|
||||
.delete_bucket(&bucket, &DeleteBucketOptions::default())
|
||||
.await
|
||||
.expect("DeleteBucket must succeed once the client has drained the bucket");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn bucket_delete_default_s3_delete_still_rejects_non_empty_bucket() {
|
||||
@@ -1262,11 +1523,11 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn bucket_delete_preserves_put_committed_after_empty_scan() {
|
||||
async fn bucket_delete_fences_put_started_after_empty_scan() {
|
||||
let (_, ecstore) = setup_bucket_delete_test_env().await;
|
||||
let bucket = format!("bucket-delete-empty-scan-race-{}", Uuid::new_v4().simple());
|
||||
let object = "committed-after-empty-scan";
|
||||
let payload = b"object committed after DeleteBucket empty scan".to_vec();
|
||||
let payload = b"object started after DeleteBucket empty scan".to_vec();
|
||||
|
||||
ecstore
|
||||
.make_bucket(&bucket, &MakeBucketOptions::default())
|
||||
@@ -1283,30 +1544,29 @@ mod tests {
|
||||
});
|
||||
barrier.wait_until_paused().await;
|
||||
|
||||
let mut put_reader = PutObjReader::from_vec(payload.clone());
|
||||
ecstore
|
||||
.put_object(&bucket, object, &mut put_reader, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("PUT should commit while DeleteBucket is paused after its empty scan");
|
||||
let put_store = ecstore.clone();
|
||||
let put_bucket = bucket.clone();
|
||||
let mut put = tokio::spawn(async move {
|
||||
let mut put_reader = PutObjReader::from_vec(payload);
|
||||
put_store
|
||||
.put_object(&put_bucket, object, &mut put_reader, &ObjectOptions::default())
|
||||
.await
|
||||
});
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(100), &mut put).await.is_err(),
|
||||
"PUT must wait behind the DeleteBucket lifecycle fence"
|
||||
);
|
||||
|
||||
barrier.release();
|
||||
let err = delete
|
||||
delete
|
||||
.await
|
||||
.expect("DeleteBucket task should join")
|
||||
.expect_err("DeleteBucket must reject a PUT committed after its empty scan");
|
||||
assert!(matches!(err, StorageError::BucketNotEmpty(name) if name == bucket));
|
||||
|
||||
let mut reader = ecstore
|
||||
.get_object_reader(&bucket, object, None, http::HeaderMap::new(), &ObjectOptions::default())
|
||||
.expect("DeleteBucket should commit while holding the lifecycle fence");
|
||||
let err = put
|
||||
.await
|
||||
.expect("committed object should remain readable after DeleteBucket fails");
|
||||
let mut restored = Vec::new();
|
||||
reader
|
||||
.stream
|
||||
.read_to_end(&mut restored)
|
||||
.await
|
||||
.expect("object body should remain readable");
|
||||
assert_eq!(restored, payload);
|
||||
.expect("PUT task should join")
|
||||
.expect_err("PUT must not recreate an object in the deleted bucket");
|
||||
assert!(matches!(err, StorageError::BucketNotFound(name) if name == bucket));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -1441,8 +1701,11 @@ mod tests {
|
||||
crate::data_usage::store_data_usage_in_backend(snapshot, ecstore.clone())
|
||||
.await
|
||||
.expect("the usage fixture should be stored");
|
||||
crate::bucket::metadata::save_bucket_incarnation(ecstore.clone(), &bucket, Uuid::new_v4())
|
||||
.await
|
||||
.expect("partial create should have an incarnation sidecar");
|
||||
|
||||
ecstore.rollback_failed_bucket_creation(&bucket, None).await;
|
||||
ecstore.rollback_failed_bucket_creation(&bucket, None, None).await;
|
||||
|
||||
assert!(
|
||||
ecstore
|
||||
@@ -1452,6 +1715,13 @@ mod tests {
|
||||
.is_err(),
|
||||
"failed-create rollback should remove the partial physical bucket"
|
||||
);
|
||||
assert!(
|
||||
crate::bucket::metadata::load_bucket_incarnation(ecstore.clone(), &bucket)
|
||||
.await
|
||||
.expect("rollback sidecar lookup should succeed")
|
||||
.is_none(),
|
||||
"failed-create rollback must remove the orphan incarnation sidecar"
|
||||
);
|
||||
let persisted = crate::data_usage::load_data_usage_from_backend(ecstore.clone())
|
||||
.await
|
||||
.expect("the usage snapshot should remain readable");
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -5819,6 +5819,83 @@ impl Sets {
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
pub(crate) async fn inner_list_object_versions_for_recursive_delete(
|
||||
self: Arc<Self>,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
marker: Option<String>,
|
||||
version_marker: Option<String>,
|
||||
max_keys: i32,
|
||||
) -> Result<ListObjectVersionsInfo> {
|
||||
let max_keys = normalize_max_keys(max_keys);
|
||||
if marker.is_none() && version_marker.is_some() {
|
||||
return Err(StorageError::NotImplemented);
|
||||
}
|
||||
|
||||
let has_version_marker = version_marker.is_some();
|
||||
let version_marker = version_marker.map(parse_version_marker).transpose()?;
|
||||
let effective_max_keys = if max_keys <= 0 { 0 } else { max_keys_plus_one(max_keys, true) };
|
||||
let mut opts = ListPathOptions {
|
||||
bucket: bucket.to_owned(),
|
||||
prefix: prefix.to_owned(),
|
||||
limit: effective_max_keys,
|
||||
marker,
|
||||
incl_deleted: true,
|
||||
ask_disks: list_objects_quorum_from_env(),
|
||||
versioned: true,
|
||||
include_marker: has_version_marker,
|
||||
..Default::default()
|
||||
};
|
||||
opts.parse_marker();
|
||||
|
||||
let mut list_result = self
|
||||
.list_path_result(&opts)
|
||||
.await
|
||||
.unwrap_or_else(|err| MetaCacheEntriesSortedResult {
|
||||
err: Some(err.into()),
|
||||
..Default::default()
|
||||
});
|
||||
let next_cache_id = list_result.entries.as_ref().and_then(|entries| entries.list_id.clone());
|
||||
let disk_has_more = list_result.err.is_none();
|
||||
if let Some(err) = list_result.err.take()
|
||||
&& err != rustfs_filemeta::Error::Unexpected
|
||||
{
|
||||
return Err(to_object_err(err.into(), vec![bucket, prefix]));
|
||||
}
|
||||
if let Some(result) = list_result.entries.as_mut()
|
||||
&& !has_version_marker
|
||||
{
|
||||
result.forward_past(opts.marker.clone());
|
||||
}
|
||||
let version_marker = version_marker_for_entries(list_result.entries.as_ref(), opts.marker.as_deref(), version_marker);
|
||||
let last_scanned_key = last_scanned_entry_name(list_result.entries.as_ref());
|
||||
let entries = list_result.entries.unwrap_or_default();
|
||||
let get_objects = ObjectInfo::from_meta_cache_entries_sorted_versions_for_recursive_delete(
|
||||
&entries,
|
||||
bucket,
|
||||
prefix,
|
||||
None,
|
||||
version_marker,
|
||||
)
|
||||
.await?;
|
||||
let (objects, prefixes, is_truncated, next_marker, next_version_idmarker) = list_objects_paginate(
|
||||
get_objects,
|
||||
&None,
|
||||
max_keys,
|
||||
disk_has_more,
|
||||
next_cache_id.as_deref(),
|
||||
true,
|
||||
last_scanned_key.as_deref(),
|
||||
);
|
||||
Ok(ListObjectVersionsInfo {
|
||||
is_truncated,
|
||||
next_marker,
|
||||
next_version_idmarker,
|
||||
objects,
|
||||
prefixes,
|
||||
})
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub async fn inner_list_objects_v2(
|
||||
self: Arc<Self>,
|
||||
|
||||
@@ -656,8 +656,18 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for ECStore {
|
||||
delimiter: Option<String>,
|
||||
max_uploads: usize,
|
||||
) -> Result<ListMultipartsInfo> {
|
||||
self.handle_list_multipart_uploads(bucket, prefix, key_marker, upload_id_marker, delimiter, max_uploads)
|
||||
.await
|
||||
self.handle_list_multipart_uploads(
|
||||
bucket,
|
||||
multipart::MultipartUploadListRequest {
|
||||
prefix: prefix.to_string(),
|
||||
key_marker,
|
||||
upload_id_marker,
|
||||
delimiter,
|
||||
max_uploads,
|
||||
expected_incarnation_id: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[instrument(skip(self))]
|
||||
|
||||
@@ -16,8 +16,21 @@ use super::*;
|
||||
use crate::multipart_listing::paginate_multipart_listing;
|
||||
use crate::set_disk::get_lock_acquire_timeout;
|
||||
use crate::storage_api_contracts::multipart::MultipartOperations as _;
|
||||
use futures::{StreamExt, stream};
|
||||
use std::collections::HashSet;
|
||||
|
||||
const MULTIPART_LIST_SET_CONCURRENCY: usize = 4;
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub(super) struct MultipartUploadListRequest {
|
||||
pub(super) prefix: String,
|
||||
pub(super) key_marker: Option<String>,
|
||||
pub(super) upload_id_marker: Option<String>,
|
||||
pub(super) delimiter: Option<String>,
|
||||
pub(super) max_uploads: usize,
|
||||
pub(super) expected_incarnation_id: Option<Uuid>,
|
||||
}
|
||||
|
||||
fn map_multipart_namespace_lock_error(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
@@ -36,7 +49,137 @@ fn map_multipart_namespace_lock_error(
|
||||
}
|
||||
}
|
||||
|
||||
fn ensure_multipart_bucket_lifecycle_guard_held(
|
||||
guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
) -> Result<()> {
|
||||
if guard.is_some_and(rustfs_lock::NamespaceLockGuard::is_lock_lost) {
|
||||
return Err(StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "multipart_bucket_generation",
|
||||
bucket: bucket.to_string(),
|
||||
object: object.to_string(),
|
||||
required: 1,
|
||||
achieved: 0,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn list_pool_multipart_uploads_for_incarnation(
|
||||
pool: &crate::core::sets::Sets,
|
||||
bucket: &str,
|
||||
request: &MultipartUploadListRequest,
|
||||
) -> Result<ListMultipartsInfo> {
|
||||
let per_set_limit = request.max_uploads.saturating_add(1);
|
||||
let results = stream::iter(pool.disk_set.iter().cloned())
|
||||
.map(|set| {
|
||||
let request = request.clone();
|
||||
async move {
|
||||
set.list_multipart_uploads_for_incarnation(
|
||||
bucket,
|
||||
&request.prefix,
|
||||
request.key_marker,
|
||||
request.upload_id_marker,
|
||||
request.delimiter,
|
||||
per_set_limit,
|
||||
request.expected_incarnation_id,
|
||||
)
|
||||
.await
|
||||
}
|
||||
})
|
||||
.buffer_unordered(MULTIPART_LIST_SET_CONCURRENCY)
|
||||
.collect::<Vec<_>>()
|
||||
.await;
|
||||
|
||||
let mut uploads = Vec::new();
|
||||
let mut common_prefixes = HashSet::new();
|
||||
let mut source_truncated = false;
|
||||
for result in results {
|
||||
let page = result?;
|
||||
uploads.extend(page.uploads);
|
||||
common_prefixes.extend(page.common_prefixes);
|
||||
source_truncated |= page.is_truncated;
|
||||
}
|
||||
|
||||
let page = paginate_multipart_listing(
|
||||
uploads,
|
||||
common_prefixes.into_iter().collect(),
|
||||
request.key_marker.as_deref(),
|
||||
request.key_marker.as_ref().and(request.upload_id_marker.as_deref()),
|
||||
request.max_uploads,
|
||||
source_truncated,
|
||||
);
|
||||
|
||||
Ok(ListMultipartsInfo {
|
||||
key_marker: request.key_marker.clone(),
|
||||
upload_id_marker: request.upload_id_marker.clone(),
|
||||
next_key_marker: page.next_key_marker,
|
||||
next_upload_id_marker: page.next_upload_id_marker,
|
||||
max_uploads: request.max_uploads,
|
||||
is_truncated: page.is_truncated,
|
||||
uploads: page.uploads,
|
||||
common_prefixes: page.common_prefixes,
|
||||
prefix: request.prefix.clone(),
|
||||
delimiter: request.delimiter.clone(),
|
||||
})
|
||||
}
|
||||
|
||||
impl ECStore {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub async fn list_multipart_uploads_for_bucket_incarnation(
|
||||
&self,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
key_marker: Option<String>,
|
||||
upload_id_marker: Option<String>,
|
||||
delimiter: Option<String>,
|
||||
max_uploads: usize,
|
||||
expected_incarnation_id: Uuid,
|
||||
) -> Result<ListMultipartsInfo> {
|
||||
self.handle_list_multipart_uploads(
|
||||
bucket,
|
||||
MultipartUploadListRequest {
|
||||
prefix: prefix.to_string(),
|
||||
key_marker,
|
||||
upload_id_marker,
|
||||
delimiter,
|
||||
max_uploads,
|
||||
expected_incarnation_id: Some(expected_incarnation_id),
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Multipart lock order is bucket lifecycle, generation validation, then
|
||||
/// object/upload locks in the selected set.
|
||||
async fn guard_multipart_bucket_incarnation(
|
||||
&self,
|
||||
bucket: &str,
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<(ObjectOptions, Option<rustfs_lock::NamespaceLockGuard>)> {
|
||||
let mut opts = opts.clone();
|
||||
if is_meta_bucketname(bucket) {
|
||||
return Ok((opts, None));
|
||||
}
|
||||
if opts.expected_bucket_incarnation_id.is_none() {
|
||||
opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(bucket).await?);
|
||||
}
|
||||
let guard = if opts.bucket_lifecycle_lock_fence.is_some() {
|
||||
None
|
||||
} else {
|
||||
Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
|
||||
};
|
||||
if let Some(guard) = guard.as_ref() {
|
||||
opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
}
|
||||
let current = crate::bucket::metadata_sys::get_bucket_incarnation_id_in(&self.ctx, bucket).await?;
|
||||
if opts.expected_bucket_incarnation_id != Some(current) {
|
||||
return Err(StorageError::BucketNotFound(bucket.to_string()));
|
||||
}
|
||||
Ok((opts, guard))
|
||||
}
|
||||
|
||||
async fn acquire_list_parts_read_lock(
|
||||
&self,
|
||||
bucket: &str,
|
||||
@@ -66,6 +209,8 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<ListPartsInfo> {
|
||||
check_list_parts_args(bucket, object, upload_id)?;
|
||||
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
|
||||
let opts = &opts;
|
||||
|
||||
let _object_lock_guard = self.acquire_list_parts_read_lock(bucket, object, opts).await?;
|
||||
|
||||
@@ -100,22 +245,38 @@ impl ECStore {
|
||||
pub(super) async fn handle_list_multipart_uploads(
|
||||
&self,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
key_marker: Option<String>,
|
||||
upload_id_marker: Option<String>,
|
||||
delimiter: Option<String>,
|
||||
max_uploads: usize,
|
||||
request: MultipartUploadListRequest,
|
||||
) -> Result<ListMultipartsInfo> {
|
||||
check_list_multipart_args(bucket, prefix, &key_marker, &upload_id_marker, &delimiter)?;
|
||||
check_list_multipart_args(
|
||||
bucket,
|
||||
&request.prefix,
|
||||
&request.key_marker,
|
||||
&request.upload_id_marker,
|
||||
&request.delimiter,
|
||||
)?;
|
||||
let guard_opts = ObjectOptions {
|
||||
expected_bucket_incarnation_id: request.expected_incarnation_id,
|
||||
..Default::default()
|
||||
};
|
||||
let (opts, bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, &guard_opts).await?;
|
||||
let expected_incarnation_id = opts.expected_bucket_incarnation_id;
|
||||
|
||||
if prefix.is_empty() {
|
||||
if request.prefix.is_empty() {
|
||||
// TODO: return from cache
|
||||
}
|
||||
|
||||
if self.single_pool() {
|
||||
return self.pools[0]
|
||||
.list_multipart_uploads(bucket, prefix, key_marker, upload_id_marker, delimiter, max_uploads)
|
||||
.await;
|
||||
let result = list_pool_multipart_uploads_for_incarnation(
|
||||
&self.pools[0],
|
||||
bucket,
|
||||
&MultipartUploadListRequest {
|
||||
expected_incarnation_id,
|
||||
..request.clone()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
ensure_multipart_bucket_lifecycle_guard_held(bucket_lifecycle_guard.as_ref(), bucket, &request.prefix)?;
|
||||
return result;
|
||||
}
|
||||
|
||||
let mut uploads = Vec::new();
|
||||
@@ -126,16 +287,15 @@ impl ECStore {
|
||||
if self.is_suspended(pool.pool_idx).await {
|
||||
continue;
|
||||
}
|
||||
let res = pool
|
||||
.list_multipart_uploads(
|
||||
bucket,
|
||||
prefix,
|
||||
key_marker.clone(),
|
||||
upload_id_marker.clone(),
|
||||
delimiter.clone(),
|
||||
max_uploads,
|
||||
)
|
||||
.await?;
|
||||
let res = list_pool_multipart_uploads_for_incarnation(
|
||||
pool,
|
||||
bucket,
|
||||
&MultipartUploadListRequest {
|
||||
expected_incarnation_id,
|
||||
..request.clone()
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
uploads.extend(res.uploads);
|
||||
common_prefixes.extend(res.common_prefixes);
|
||||
source_truncated |= res.is_truncated;
|
||||
@@ -145,19 +305,21 @@ impl ECStore {
|
||||
// unordered across pools and may exceed the global cap. Re-sort, re-cap,
|
||||
// and derive the truncation markers so a bucket whose uploads span pools
|
||||
// pages correctly instead of being silently reported complete.
|
||||
let page = merge_multipart_upload_pages(uploads, common_prefixes.into_iter().collect(), max_uploads, source_truncated);
|
||||
let page =
|
||||
merge_multipart_upload_pages(uploads, common_prefixes.into_iter().collect(), request.max_uploads, source_truncated);
|
||||
ensure_multipart_bucket_lifecycle_guard_held(bucket_lifecycle_guard.as_ref(), bucket, &request.prefix)?;
|
||||
|
||||
Ok(ListMultipartsInfo {
|
||||
key_marker,
|
||||
upload_id_marker,
|
||||
key_marker: request.key_marker,
|
||||
upload_id_marker: request.upload_id_marker,
|
||||
next_key_marker: page.next_key_marker,
|
||||
next_upload_id_marker: page.next_upload_id_marker,
|
||||
max_uploads,
|
||||
max_uploads: request.max_uploads,
|
||||
is_truncated: page.is_truncated,
|
||||
uploads: page.uploads,
|
||||
common_prefixes: page.common_prefixes,
|
||||
prefix: prefix.to_owned(),
|
||||
delimiter: delimiter.to_owned(),
|
||||
prefix: request.prefix,
|
||||
delimiter: request.delimiter,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -180,6 +342,8 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<(MultipartUploadResult, usize)> {
|
||||
check_new_multipart_args(bucket, object)?;
|
||||
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
|
||||
let opts = &opts;
|
||||
|
||||
if self.single_pool() {
|
||||
return self.pools[0]
|
||||
@@ -192,9 +356,19 @@ impl ECStore {
|
||||
if self.is_suspended(idx).await || self.is_pool_rebalancing(idx).await {
|
||||
continue;
|
||||
}
|
||||
let res = pool
|
||||
.list_multipart_uploads(bucket, object, None, None, None, MAX_UPLOADS_LIST)
|
||||
.await?;
|
||||
let res = list_pool_multipart_uploads_for_incarnation(
|
||||
pool,
|
||||
bucket,
|
||||
&MultipartUploadListRequest {
|
||||
prefix: object.to_string(),
|
||||
key_marker: None,
|
||||
upload_id_marker: None,
|
||||
delimiter: None,
|
||||
max_uploads: MAX_UPLOADS_LIST,
|
||||
expected_incarnation_id: opts.expected_bucket_incarnation_id,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
|
||||
if !res.uploads.is_empty() {
|
||||
let res = self.pools[idx].new_multipart_upload(bucket, object, opts).await?;
|
||||
@@ -249,6 +423,8 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<PartInfo> {
|
||||
check_put_object_part_args(bucket, object, upload_id)?;
|
||||
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
|
||||
let opts = &opts;
|
||||
|
||||
if self.single_pool() {
|
||||
return self.pools[0]
|
||||
@@ -289,6 +465,8 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<MultipartInfo> {
|
||||
check_list_parts_args(bucket, object, upload_id)?;
|
||||
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
|
||||
let opts = &opts;
|
||||
if self.single_pool() {
|
||||
return self.pools[0].get_multipart_info(bucket, object, upload_id, opts).await;
|
||||
}
|
||||
@@ -322,6 +500,8 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<()> {
|
||||
check_abort_multipart_args(bucket, object, upload_id)?;
|
||||
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
|
||||
let opts = &opts;
|
||||
|
||||
// TODO: defer DeleteUploadID
|
||||
|
||||
@@ -360,6 +540,8 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<ObjectInfo> {
|
||||
check_complete_multipart_args(bucket, object, upload_id)?;
|
||||
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
|
||||
let opts = &opts;
|
||||
|
||||
if self.single_pool() {
|
||||
return self.pools[0]
|
||||
|
||||
@@ -23,9 +23,16 @@ use crate::bucket::lifecycle::{
|
||||
Jentry, attach_tier_delete_source, transitioned_delete_journal_entry_for_source, transitioned_force_delete_journal_entry,
|
||||
},
|
||||
};
|
||||
use crate::bucket::metadata_sys::{
|
||||
acquire_bucket_metadata_transaction_read_lock_in, get_bucket_incarnation_id_in, get_cached_bucket_incarnation_id_in,
|
||||
get_object_lock_config_and_incarnation_from_disk_in,
|
||||
};
|
||||
use crate::bucket::object_lock::objectlock_sys::{
|
||||
check_object_lock_for_deletion_with_state, ensure_recursive_force_delete_allowed_for_state,
|
||||
};
|
||||
use crate::bucket::replication::ReplicationObjectBridge;
|
||||
use crate::disk::OldCurrentSize;
|
||||
use crate::object_api::DeleteLockFence;
|
||||
use crate::object_api::{NamespaceLockFence, ObjectLockConfigSnapshot};
|
||||
use crate::set_disk::{
|
||||
get_lock_acquire_timeout, get_object_lock_diag_slow_acquire_threshold, get_object_lock_diag_slow_hold_threshold,
|
||||
is_lock_optimization_enabled, is_object_lock_diag_enabled,
|
||||
@@ -43,6 +50,10 @@ use std::{
|
||||
};
|
||||
use tokio::io::{AsyncRead, ReadBuf};
|
||||
|
||||
#[cfg(not(test))]
|
||||
const RECURSIVE_DELETE_VERSION_SCAN_PAGE_SIZE: i32 = 1000;
|
||||
#[cfg(test)]
|
||||
const RECURSIVE_DELETE_VERSION_SCAN_PAGE_SIZE: i32 = 2;
|
||||
const FORCE_DELETE_LIST_PAGE_SIZE: i32 = 1_000;
|
||||
|
||||
fn build_tier_delete_journal_entry(
|
||||
@@ -365,6 +376,13 @@ impl RestoreAcceptGuard {
|
||||
pub fn is_lock_lost(&self) -> bool {
|
||||
self.0.guard.is_lock_lost()
|
||||
}
|
||||
|
||||
pub fn add_namespace_lock_fence(&self, opts: &mut ObjectOptions) {
|
||||
opts.ensure_namespace_lock_fence();
|
||||
if let Some(signal) = self.0.lock_lost_signal() {
|
||||
opts.add_namespace_lock_lost_signal(signal);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ObjectLockDiagGuard {
|
||||
@@ -485,6 +503,77 @@ fn should_create_delete_marker_for_missing_object(opts: &ObjectOptions) -> bool
|
||||
opts.versioned && opts.version_id.is_none() && !opts.delete_marker && !opts.data_movement
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
struct DeleteAfterObjectLockSnapshotBarrierState {
|
||||
bucket: String,
|
||||
arrived: tokio::sync::Notify,
|
||||
release: tokio::sync::Notify,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct DeleteAfterObjectLockSnapshotBarrier {
|
||||
state: Arc<DeleteAfterObjectLockSnapshotBarrierState>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
static DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER: std::sync::OnceLock<
|
||||
std::sync::Mutex<Option<Arc<DeleteAfterObjectLockSnapshotBarrierState>>>,
|
||||
> = std::sync::OnceLock::new();
|
||||
|
||||
#[cfg(test)]
|
||||
impl DeleteAfterObjectLockSnapshotBarrier {
|
||||
pub(crate) fn install(bucket: &str) -> Self {
|
||||
let state = Arc::new(DeleteAfterObjectLockSnapshotBarrierState {
|
||||
bucket: bucket.to_string(),
|
||||
arrived: tokio::sync::Notify::new(),
|
||||
release: tokio::sync::Notify::new(),
|
||||
});
|
||||
let mut slot = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER
|
||||
.get_or_init(|| std::sync::Mutex::new(None))
|
||||
.lock()
|
||||
.expect("delete snapshot barrier mutex should not poison");
|
||||
assert!(slot.is_none(), "delete snapshot barrier must not already be installed");
|
||||
*slot = Some(Arc::clone(&state));
|
||||
Self { state }
|
||||
}
|
||||
|
||||
pub(crate) async fn wait_until_paused(&self) {
|
||||
self.state.arrived.notified().await;
|
||||
}
|
||||
|
||||
pub(crate) fn release(&self) {
|
||||
self.state.release.notify_one();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for DeleteAfterObjectLockSnapshotBarrier {
|
||||
fn drop(&mut self) {
|
||||
self.state.release.notify_one();
|
||||
if let Some(slot) = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER.get() {
|
||||
let mut slot = slot.lock().expect("delete snapshot barrier mutex should not poison");
|
||||
if slot.as_ref().is_some_and(|installed| Arc::ptr_eq(installed, &self.state)) {
|
||||
*slot = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
async fn pause_delete_after_object_lock_snapshot(bucket: &str) {
|
||||
let state = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER
|
||||
.get_or_init(|| std::sync::Mutex::new(None))
|
||||
.lock()
|
||||
.expect("delete snapshot barrier mutex should not poison")
|
||||
.as_ref()
|
||||
.filter(|state| state.bucket == bucket)
|
||||
.cloned();
|
||||
if let Some(state) = state {
|
||||
state.arrived.notify_one();
|
||||
state.release.notified().await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a delete-time lookup miss on a directory key should trigger an orphan
|
||||
/// empty-directory tree purge (issue #4189).
|
||||
///
|
||||
@@ -643,6 +732,54 @@ fn sorted_unique_delete_object_names(objects: &[ObjectToDelete]) -> Vec<&str> {
|
||||
}
|
||||
|
||||
impl ECStore {
|
||||
/// Captures Object Lock state once for a batch of PUTs to the same bucket.
|
||||
/// `handle_put_object` only reuses the token for the same store, bucket,
|
||||
/// bucket incarnation, and Object Lock configuration revision.
|
||||
pub async fn object_lock_config_snapshot(&self, bucket: &str) -> Result<Arc<ObjectLockConfigSnapshot>> {
|
||||
check_valid_bucket_name(bucket)?;
|
||||
let lifecycle_guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
|
||||
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
|
||||
let (state, bucket_incarnation_id, config_revision) =
|
||||
get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await?;
|
||||
if lifecycle_guard.is_lock_lost() || metadata_guard.is_lock_lost() {
|
||||
return Err(Error::other("bucket lifecycle lock was lost while loading the Object Lock snapshot"));
|
||||
}
|
||||
Ok(Arc::new(ObjectLockConfigSnapshot::for_guarded_store_bucket(
|
||||
self.id,
|
||||
bucket,
|
||||
bucket_incarnation_id,
|
||||
config_revision,
|
||||
state,
|
||||
lifecycle_guard,
|
||||
metadata_guard,
|
||||
)))
|
||||
}
|
||||
|
||||
async fn object_lock_config_snapshot_under_lifecycle_fence(
|
||||
&self,
|
||||
bucket: &str,
|
||||
lifecycle_fence: &NamespaceLockFence,
|
||||
) -> Result<Arc<ObjectLockConfigSnapshot>> {
|
||||
if lifecycle_fence.is_lock_lost() {
|
||||
return Err(Error::other("bucket lifecycle lock was lost before loading the Object Lock snapshot"));
|
||||
}
|
||||
let metadata_guard = acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?;
|
||||
let (state, bucket_incarnation_id, config_revision) =
|
||||
get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await?;
|
||||
if lifecycle_fence.is_lock_lost() || metadata_guard.is_lock_lost() {
|
||||
return Err(Error::other("bucket lock was lost while loading the Object Lock snapshot"));
|
||||
}
|
||||
Ok(Arc::new(ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence(
|
||||
self.id,
|
||||
bucket,
|
||||
bucket_incarnation_id,
|
||||
config_revision,
|
||||
state,
|
||||
lifecycle_fence.clone(),
|
||||
metadata_guard,
|
||||
)))
|
||||
}
|
||||
|
||||
/// Resolves a GET's object identity without constructing its body reader.
|
||||
///
|
||||
/// This is an additive two-stage counterpart to `get_object_reader`. The
|
||||
@@ -748,6 +885,10 @@ impl ECStore {
|
||||
}
|
||||
|
||||
let guard = self.acquire_object_write_lock(op, bucket, object).await?;
|
||||
if let Some(signal) = guard.lock_lost_signal() {
|
||||
opts.add_namespace_lock_lost_signal(signal);
|
||||
}
|
||||
opts.ensure_namespace_lock_fence();
|
||||
opts.no_lock = true;
|
||||
|
||||
Ok(Some(guard))
|
||||
@@ -788,9 +929,10 @@ impl ECStore {
|
||||
guards.push(self.acquire_object_write_lock("delete_objects", bucket, object).await?);
|
||||
}
|
||||
opts.no_lock = true;
|
||||
opts.delete_lock_fence = Some(DeleteLockFence::new(
|
||||
guards.iter().filter_map(ObjectLockDiagGuard::lock_lost_signal).collect(),
|
||||
));
|
||||
for signal in guards.iter().filter_map(ObjectLockDiagGuard::lock_lost_signal) {
|
||||
opts.add_namespace_lock_lost_signal(signal);
|
||||
}
|
||||
opts.ensure_namespace_lock_fence();
|
||||
|
||||
Ok(guards)
|
||||
}
|
||||
@@ -1057,17 +1199,40 @@ impl ECStore {
|
||||
check_put_object_args(bucket, object)?;
|
||||
|
||||
let object = encode_dir_object(object);
|
||||
let mut opts = opts.clone();
|
||||
if !is_meta_bucketname(bucket) && opts.expected_bucket_incarnation_id.is_none() {
|
||||
opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(bucket).await?);
|
||||
}
|
||||
if opts.overwrites_existing_version() && !is_meta_bucketname(bucket) {
|
||||
let expected_incarnation_id = opts
|
||||
.expected_bucket_incarnation_id
|
||||
.ok_or_else(|| Error::other("destructive PUT is missing its bucket incarnation"))?;
|
||||
if opts.object_lock_config_snapshot.is_none() {
|
||||
opts.object_lock_config_snapshot = Some(self.object_lock_config_snapshot(bucket).await?);
|
||||
}
|
||||
let snapshot = match opts.object_lock_config_snapshot.as_ref() {
|
||||
Some(snapshot) if snapshot.is_valid_for_destructive_put(self.id, bucket, expected_incarnation_id) => {
|
||||
Arc::clone(snapshot)
|
||||
}
|
||||
_ => {
|
||||
return Err(Error::other(
|
||||
"Object Lock snapshot does not hold valid target bucket generation and configuration fences",
|
||||
));
|
||||
}
|
||||
};
|
||||
snapshot.add_lock_fences(&mut opts);
|
||||
}
|
||||
|
||||
// Keep PUT atomic-read friendly: SetDisks takes the object write lock only
|
||||
// around precondition checks and the final rename/commit.
|
||||
if self.single_pool() {
|
||||
return self.pools[0]
|
||||
.put_object_with_old_current_size(bucket, object.as_str(), data, opts)
|
||||
.put_object_with_old_current_size(bucket, object.as_str(), data, &opts)
|
||||
.await;
|
||||
}
|
||||
|
||||
let idx = if opts.data_movement && opts.version_id.is_some() {
|
||||
self.select_data_movement_pool_idx(bucket, &object, data.size(), opts, false)
|
||||
self.select_data_movement_pool_idx(bucket, &object, data.size(), &opts, false)
|
||||
.await?
|
||||
} else if opts.no_lock {
|
||||
self.get_pool_idx_no_lock(bucket, &object, data.size()).await?
|
||||
@@ -1084,7 +1249,7 @@ impl ECStore {
|
||||
}
|
||||
|
||||
self.pools[idx]
|
||||
.put_object_with_old_current_size(bucket, &object, data, opts)
|
||||
.put_object_with_old_current_size(bucket, &object, data, &opts)
|
||||
.await
|
||||
}
|
||||
|
||||
@@ -1130,6 +1295,60 @@ impl ECStore {
|
||||
let cp_src_dst_same = path_join_buf(&[src_bucket, &src_object]) == path_join_buf(&[dst_bucket, &dst_object]);
|
||||
|
||||
let mut dst_opts = dst_opts.clone();
|
||||
if !is_meta_bucketname(dst_bucket) && dst_opts.expected_bucket_incarnation_id.is_none() {
|
||||
dst_opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(dst_bucket).await?);
|
||||
}
|
||||
let _bucket_lifecycle_guard = if is_meta_bucketname(dst_bucket) || dst_opts.bucket_lifecycle_lock_fence.is_some() {
|
||||
None
|
||||
} else {
|
||||
Some(self.acquire_bucket_lifecycle_read_lock(dst_bucket).await?)
|
||||
};
|
||||
let current_bucket_incarnation_id = if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
|
||||
dst_opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
let current_incarnation_id = get_bucket_incarnation_id_in(&self.ctx, dst_bucket).await?;
|
||||
if dst_opts
|
||||
.expected_bucket_incarnation_id
|
||||
.is_some_and(|expected| expected != current_incarnation_id)
|
||||
{
|
||||
return Err(StorageError::BucketNotFound(dst_bucket.to_string()));
|
||||
}
|
||||
Some(current_incarnation_id)
|
||||
} else {
|
||||
dst_opts.expected_bucket_incarnation_id
|
||||
};
|
||||
if dst_opts
|
||||
.bucket_lifecycle_lock_fence
|
||||
.as_ref()
|
||||
.is_some_and(NamespaceLockFence::is_lock_lost)
|
||||
{
|
||||
return Err(StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "copy_object_bucket_generation",
|
||||
bucket: dst_bucket.to_string(),
|
||||
object: dst_object.clone(),
|
||||
required: 1,
|
||||
achieved: 0,
|
||||
});
|
||||
}
|
||||
if dst_opts.overwrites_existing_version() && !is_meta_bucketname(dst_bucket) {
|
||||
let incarnation_id =
|
||||
current_bucket_incarnation_id.ok_or_else(|| Error::other("copy is missing its bucket incarnation snapshot"))?;
|
||||
let lifecycle_fence = dst_opts
|
||||
.bucket_lifecycle_lock_fence
|
||||
.as_ref()
|
||||
.ok_or_else(|| Error::other("copy is missing its bucket lifecycle fence"))?;
|
||||
let snapshot = match dst_opts.object_lock_config_snapshot.as_ref() {
|
||||
Some(snapshot) => Arc::clone(snapshot),
|
||||
None => {
|
||||
self.object_lock_config_snapshot_under_lifecycle_fence(dst_bucket, lifecycle_fence)
|
||||
.await?
|
||||
}
|
||||
};
|
||||
if !snapshot.is_valid_for_destructive_put(self.id, dst_bucket, incarnation_id) {
|
||||
return Err(Error::other("copy Object Lock snapshot does not match the target bucket generation"));
|
||||
}
|
||||
snapshot.add_lock_fences(&mut dst_opts);
|
||||
dst_opts.object_lock_config_snapshot = Some(snapshot);
|
||||
}
|
||||
let _dst_lock_guard = if cp_src_dst_same && dst_opts.expected_current_version_id.is_none() {
|
||||
self.acquire_object_write_lock_if_needed("copy_object", dst_bucket, &dst_object, &mut dst_opts)
|
||||
.await?
|
||||
@@ -1176,6 +1395,10 @@ impl ECStore {
|
||||
mod_time: dst_opts.mod_time,
|
||||
http_preconditions: dst_opts.http_preconditions.clone(),
|
||||
expected_current_version_id: dst_opts.expected_current_version_id.clone(),
|
||||
expected_bucket_incarnation_id: dst_opts.expected_bucket_incarnation_id,
|
||||
namespace_lock_fence: dst_opts.namespace_lock_fence.clone(),
|
||||
bucket_lifecycle_lock_fence: dst_opts.bucket_lifecycle_lock_fence.clone(),
|
||||
object_lock_config_snapshot: dst_opts.object_lock_config_snapshot.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
return if let Some(reader) = src_info.put_object_reader.as_mut() {
|
||||
@@ -1206,6 +1429,10 @@ impl ECStore {
|
||||
mod_time: dst_opts.mod_time,
|
||||
http_preconditions: dst_opts.http_preconditions.clone(),
|
||||
expected_current_version_id: dst_opts.expected_current_version_id.clone(),
|
||||
expected_bucket_incarnation_id: dst_opts.expected_bucket_incarnation_id,
|
||||
namespace_lock_fence: dst_opts.namespace_lock_fence.clone(),
|
||||
bucket_lifecycle_lock_fence: dst_opts.bucket_lifecycle_lock_fence.clone(),
|
||||
object_lock_config_snapshot: dst_opts.object_lock_config_snapshot.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
return self.pools[pool_idx]
|
||||
@@ -1233,6 +1460,10 @@ impl ECStore {
|
||||
mod_time: dst_opts.mod_time,
|
||||
http_preconditions: dst_opts.http_preconditions.clone(),
|
||||
expected_current_version_id: dst_opts.expected_current_version_id.clone(),
|
||||
expected_bucket_incarnation_id: dst_opts.expected_bucket_incarnation_id,
|
||||
namespace_lock_fence: dst_opts.namespace_lock_fence.clone(),
|
||||
bucket_lifecycle_lock_fence: dst_opts.bucket_lifecycle_lock_fence.clone(),
|
||||
object_lock_config_snapshot: dst_opts.object_lock_config_snapshot.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
@@ -1322,6 +1553,13 @@ impl ECStore {
|
||||
) -> Result<ObjectInfo> {
|
||||
check_del_obj_args(bucket, object)?;
|
||||
|
||||
let _bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
|
||||
None
|
||||
} else if opts.delete_prefix {
|
||||
Some(self.acquire_bucket_lifecycle_write_lock(bucket).await?)
|
||||
} else {
|
||||
Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
|
||||
};
|
||||
let object = if opts.delete_prefix && !opts.delete_prefix_object {
|
||||
object.to_owned()
|
||||
} else {
|
||||
@@ -1330,10 +1568,94 @@ impl ECStore {
|
||||
let object = object.as_str();
|
||||
let mut opts = opts;
|
||||
opts.tier_delete_journal_api = tier_journal_api.clone();
|
||||
if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
|
||||
opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
}
|
||||
|
||||
if !is_meta_bucketname(bucket) {
|
||||
get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await?;
|
||||
}
|
||||
let _object_lock_metadata_guard = if !is_meta_bucketname(bucket) {
|
||||
Some(acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some(guard) = _object_lock_metadata_guard.as_ref() {
|
||||
opts.add_namespace_lock_guard(guard);
|
||||
}
|
||||
let current_bucket_incarnation_id = if _object_lock_metadata_guard.is_some() {
|
||||
let (state, incarnation_id, config_revision) =
|
||||
get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await?;
|
||||
opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket(
|
||||
self.id,
|
||||
bucket,
|
||||
incarnation_id,
|
||||
config_revision,
|
||||
state,
|
||||
)));
|
||||
Some(incarnation_id)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let (Some(expected), Some(current)) = (opts.expected_bucket_incarnation_id, current_bucket_incarnation_id)
|
||||
&& expected != current
|
||||
{
|
||||
return Err(StorageError::BucketNotFound(bucket.to_string()));
|
||||
}
|
||||
#[cfg(test)]
|
||||
if current_bucket_incarnation_id.is_some() {
|
||||
pause_delete_after_object_lock_snapshot(bucket).await;
|
||||
}
|
||||
|
||||
if opts.delete_prefix && !opts.delete_prefix_object {
|
||||
// Prefix deletes cover multiple object keys; an exact lock on the prefix string
|
||||
// would not protect child objects.
|
||||
if !is_meta_bucketname(bucket) {
|
||||
let state = opts
|
||||
.object_lock_config_snapshot
|
||||
.as_deref()
|
||||
.ok_or_else(|| Error::other("recursive delete is missing its Object Lock configuration snapshot"))?
|
||||
.state();
|
||||
ensure_recursive_force_delete_allowed_for_state(bucket, state)?;
|
||||
let bypass_governance = opts
|
||||
.object_lock_delete
|
||||
.as_ref()
|
||||
.is_some_and(|delete_opts| delete_opts.bypass_governance);
|
||||
for pool in &self.pools {
|
||||
for set in &pool.disk_set {
|
||||
let mut marker = None;
|
||||
let mut version_marker = None;
|
||||
loop {
|
||||
let page = set
|
||||
.clone()
|
||||
.inner_list_object_versions_for_recursive_delete(
|
||||
bucket,
|
||||
object,
|
||||
marker.clone(),
|
||||
version_marker.clone(),
|
||||
RECURSIVE_DELETE_VERSION_SCAN_PAGE_SIZE,
|
||||
)
|
||||
.await?;
|
||||
for object_info in &page.objects {
|
||||
if check_object_lock_for_deletion_with_state(state, object_info, bypass_governance)?.is_some() {
|
||||
return Err(StorageError::PrefixAccessDenied(bucket.to_string(), object_info.name.clone()));
|
||||
}
|
||||
}
|
||||
if !page.is_truncated {
|
||||
break;
|
||||
}
|
||||
let next_marker = page.next_marker.ok_or_else(|| {
|
||||
Error::other("recursive delete version scan did not return a continuation marker")
|
||||
})?;
|
||||
if marker.as_ref() == Some(&next_marker) && version_marker == page.next_version_idmarker {
|
||||
return Err(Error::other("recursive delete version scan did not advance"));
|
||||
}
|
||||
marker = Some(next_marker);
|
||||
version_marker = page.next_version_idmarker;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
delete_prefix_with_tier_delete_journal(self, bucket, object, &opts, tier_journal_api.as_ref()).await?;
|
||||
return Ok(ObjectInfo::default());
|
||||
}
|
||||
@@ -1344,7 +1666,6 @@ impl ECStore {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
if opts.delete_prefix {
|
||||
delete_prefix_with_tier_delete_journal(self, bucket, object, &opts, tier_journal_api.as_ref()).await?;
|
||||
return Ok(ObjectInfo::default());
|
||||
@@ -1531,6 +1852,17 @@ impl ECStore {
|
||||
|
||||
let mut opts = opts;
|
||||
opts.tier_delete_journal_api = tier_journal_api;
|
||||
let _bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
|
||||
None
|
||||
} else {
|
||||
match self.acquire_bucket_lifecycle_read_lock(bucket).await {
|
||||
Ok(guard) => Some(guard),
|
||||
Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err),
|
||||
}
|
||||
};
|
||||
if let Some(guard) = _bucket_lifecycle_guard.as_ref() {
|
||||
opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
}
|
||||
if opts.delete_replication_config_snapshot.is_none() {
|
||||
match ReplicationObjectBridge::delete_request_config_in(&self.ctx, bucket).await {
|
||||
Ok(snapshot) => opts.delete_replication_config_snapshot = Some(Arc::new(snapshot)),
|
||||
@@ -1541,7 +1873,48 @@ impl ECStore {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !is_meta_bucketname(bucket)
|
||||
&& let Err(err) = get_cached_bucket_incarnation_id_in(&self.ctx, bucket).await
|
||||
{
|
||||
return return_batch_delete_lock_error(objects.as_slice(), err);
|
||||
}
|
||||
let _object_lock_metadata_guard = if is_meta_bucketname(bucket) {
|
||||
None
|
||||
} else {
|
||||
Some(match acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket).await {
|
||||
Ok(guard) => guard,
|
||||
Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err),
|
||||
})
|
||||
};
|
||||
if let Some(guard) = _object_lock_metadata_guard.as_ref() {
|
||||
opts.add_namespace_lock_guard(guard);
|
||||
}
|
||||
let current_bucket_incarnation_id = if _object_lock_metadata_guard.is_some() {
|
||||
let (state, incarnation_id, config_revision) =
|
||||
match get_object_lock_config_and_incarnation_from_disk_in(&self.ctx, bucket).await {
|
||||
Ok(snapshot) => snapshot,
|
||||
Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err),
|
||||
};
|
||||
opts.object_lock_config_snapshot = Some(Arc::new(ObjectLockConfigSnapshot::for_store_bucket(
|
||||
self.id,
|
||||
bucket,
|
||||
incarnation_id,
|
||||
config_revision,
|
||||
state,
|
||||
)));
|
||||
Some(incarnation_id)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let (Some(expected), Some(current)) = (opts.expected_bucket_incarnation_id, current_bucket_incarnation_id)
|
||||
&& expected != current
|
||||
{
|
||||
return return_batch_delete_lock_error(objects.as_slice(), StorageError::BucketNotFound(bucket.to_string()));
|
||||
}
|
||||
#[cfg(test)]
|
||||
if current_bucket_incarnation_id.is_some() {
|
||||
pause_delete_after_object_lock_snapshot(bucket).await;
|
||||
}
|
||||
let _object_lock_guards = match self.acquire_delete_objects_write_locks(bucket, &objects, &mut opts).await {
|
||||
Ok(guards) => guards,
|
||||
Err(err) => return return_batch_delete_lock_error(objects.as_slice(), err),
|
||||
@@ -1747,6 +2120,45 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<()> {
|
||||
let object = encode_dir_object(object);
|
||||
let mut opts = transition_restore_pool_opts(opts);
|
||||
if !is_meta_bucketname(bucket) && opts.expected_bucket_incarnation_id.is_none() {
|
||||
opts.expected_bucket_incarnation_id = Some(self.bucket_incarnation_id(bucket).await?);
|
||||
}
|
||||
let bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
|
||||
None
|
||||
} else {
|
||||
Some(self.acquire_bucket_lifecycle_read_lock(bucket).await?)
|
||||
};
|
||||
if let Some(guard) = bucket_lifecycle_guard.as_ref() {
|
||||
opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
}
|
||||
if !is_meta_bucketname(bucket) {
|
||||
let current_incarnation_id = get_bucket_incarnation_id_in(&self.ctx, bucket).await?;
|
||||
if opts.expected_bucket_incarnation_id != Some(current_incarnation_id) {
|
||||
return Err(StorageError::BucketNotFound(bucket.to_string()));
|
||||
}
|
||||
}
|
||||
if opts.overwrites_existing_version() && !is_meta_bucketname(bucket) {
|
||||
let expected_incarnation_id = opts
|
||||
.expected_bucket_incarnation_id
|
||||
.ok_or_else(|| Error::other("restore is missing its bucket incarnation snapshot"))?;
|
||||
let lifecycle_fence = opts
|
||||
.bucket_lifecycle_lock_fence
|
||||
.as_ref()
|
||||
.ok_or_else(|| Error::other("restore is missing its bucket lifecycle fence"))?;
|
||||
let snapshot = match opts.object_lock_config_snapshot.as_ref() {
|
||||
Some(snapshot) => Arc::clone(snapshot),
|
||||
None => {
|
||||
self.object_lock_config_snapshot_under_lifecycle_fence(bucket, lifecycle_fence)
|
||||
.await?
|
||||
}
|
||||
};
|
||||
if !snapshot.is_valid_for_destructive_put(self.id, bucket, expected_incarnation_id) {
|
||||
return Err(Error::other("restore Object Lock snapshot does not match the target bucket generation"));
|
||||
}
|
||||
snapshot.add_lock_fences(&mut opts);
|
||||
opts.object_lock_config_snapshot = Some(snapshot);
|
||||
}
|
||||
// Deliberately NOT holding the object write lock across the tier
|
||||
// copy-back (backlog#1304): non-SELECT restore-vs-restore is
|
||||
// serialized by the accept path's compare-and-set of the ongoing flag
|
||||
@@ -1762,10 +2174,12 @@ impl ECStore {
|
||||
// (#4877) blocked HEAD/get_object_info for the whole copy-back and
|
||||
// self-deadlocked on the inner commits.
|
||||
if self.single_pool() {
|
||||
return self.pools[0].clone().restore_transitioned_object(bucket, &object, opts).await;
|
||||
return self.pools[0]
|
||||
.clone()
|
||||
.restore_transitioned_object(bucket, &object, &opts)
|
||||
.await;
|
||||
}
|
||||
|
||||
let opts = transition_restore_pool_opts(opts);
|
||||
let (_, idx) = self
|
||||
.get_latest_accessible_object_info_with_idx(bucket, object.as_str(), &opts)
|
||||
.await?;
|
||||
@@ -1784,18 +2198,38 @@ impl ECStore {
|
||||
opts: &ObjectOptions,
|
||||
) -> Result<ObjectInfo> {
|
||||
let object = encode_dir_object(object);
|
||||
if self.single_pool() {
|
||||
return self.pools[0].put_object_metadata(bucket, object.as_str(), opts).await;
|
||||
}
|
||||
|
||||
let mut opts = opts.clone();
|
||||
opts.metadata_chg = true;
|
||||
let bucket_lifecycle_guard = if is_meta_bucketname(bucket) {
|
||||
None
|
||||
} else {
|
||||
let guard = self.acquire_bucket_lifecycle_read_lock(bucket).await?;
|
||||
let current_incarnation_id = get_bucket_incarnation_id_in(&self.ctx, bucket).await?;
|
||||
if opts
|
||||
.expected_bucket_incarnation_id
|
||||
.is_some_and(|expected| expected != current_incarnation_id)
|
||||
{
|
||||
return Err(StorageError::BucketNotFound(bucket.to_string()));
|
||||
}
|
||||
opts.expected_bucket_incarnation_id = Some(current_incarnation_id);
|
||||
opts.add_bucket_lifecycle_lock_guard(&guard);
|
||||
if guard.is_lock_lost() {
|
||||
return Err(Error::other("bucket lifecycle lock was lost before the metadata update"));
|
||||
}
|
||||
Some(guard)
|
||||
};
|
||||
|
||||
if self.single_pool() {
|
||||
return self.pools[0].put_object_metadata(bucket, object.as_str(), &opts).await;
|
||||
}
|
||||
|
||||
let (_, idx) = self
|
||||
.get_latest_accessible_object_info_with_idx(bucket, object.as_str(), &opts)
|
||||
.await?;
|
||||
|
||||
self.pools[idx].put_object_metadata(bucket, object.as_str(), &opts).await
|
||||
let result = self.pools[idx].put_object_metadata(bucket, object.as_str(), &opts).await;
|
||||
drop(bucket_lifecycle_guard);
|
||||
result
|
||||
}
|
||||
|
||||
#[instrument(skip(self))]
|
||||
@@ -2885,7 +3319,10 @@ mod tests {
|
||||
|
||||
assert_eq!(guards.len(), 2, "duplicate object names should share one namespace lock");
|
||||
assert!(opts.no_lock, "set layer should not reacquire locks already held by ECStore");
|
||||
assert!(opts.delete_lock_fence.is_some(), "set layer must receive the outer write-lock loss fence");
|
||||
assert!(
|
||||
opts.namespace_lock_fence.is_some(),
|
||||
"set layer must receive the outer write-lock loss fence"
|
||||
);
|
||||
|
||||
let alpha_lock = store
|
||||
.handle_new_ns_lock("bucket", "alpha")
|
||||
|
||||
@@ -202,13 +202,35 @@ impl ECStore {
|
||||
}
|
||||
|
||||
pub(super) async fn delete_prefix(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<()> {
|
||||
let mut first_error = None;
|
||||
let mut first_volume_error = None;
|
||||
let mut has_success = false;
|
||||
for pool in self.pools.iter() {
|
||||
let mut opts = opts.clone();
|
||||
opts.delete_prefix = true;
|
||||
pool.delete_object(bucket, object, opts).await?;
|
||||
match pool.delete_object(bucket, object, opts).await {
|
||||
Ok(_) => has_success = true,
|
||||
Err(err) if is_err_strict_volume_not_found(&err) => {
|
||||
if first_volume_error.is_none() {
|
||||
first_volume_error = Some(err);
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
if first_error.is_none() {
|
||||
first_error = Some(err);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
match first_error {
|
||||
Some(err) => Err(err),
|
||||
None if has_success => Ok(()),
|
||||
None => match first_volume_error {
|
||||
Some(err) => Err(err),
|
||||
None => Ok(()),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) async fn get_available_pool_idx(&self, bucket: &str, object: &str, size: i64) -> Option<usize> {
|
||||
@@ -748,9 +770,148 @@ impl ECStore {
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::storageclass::{CLASS_RRS, CLASS_STANDARD, lookup_config_for_pools_without_env};
|
||||
use crate::disk::error::DiskError;
|
||||
use crate::layout::endpoint::Endpoint;
|
||||
use crate::layout::endpoints::{EndpointServerPools, Endpoints, PoolEndpoints};
|
||||
use crate::storage_api_contracts::bucket::MakeBucketOptions;
|
||||
use arc_swap::ArcSwap;
|
||||
use rustfs_config::server_config::KVS;
|
||||
use std::sync::Arc;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_prefix_attempts_later_pools_after_an_earlier_pool_error() {
|
||||
let temp_dir = tempfile::tempdir().expect("multi-pool delete test directory should be created");
|
||||
let mut pools = Vec::with_capacity(2);
|
||||
for (pool_index, drives_per_set) in [2, 4].into_iter().enumerate() {
|
||||
let mut endpoints = Vec::with_capacity(drives_per_set);
|
||||
for disk_index in 0..drives_per_set {
|
||||
let disk_path = temp_dir.path().join(format!("pool{pool_index}-disk{disk_index}"));
|
||||
tokio::fs::create_dir_all(&disk_path)
|
||||
.await
|
||||
.expect("multi-pool delete test disk should be created");
|
||||
let mut endpoint =
|
||||
Endpoint::try_from(disk_path.to_str().expect("disk path should be utf8")).expect("endpoint should parse");
|
||||
endpoint.set_pool_index(pool_index);
|
||||
endpoint.set_set_index(0);
|
||||
endpoint.set_disk_index(disk_index);
|
||||
endpoints.push(endpoint);
|
||||
}
|
||||
pools.push(PoolEndpoints {
|
||||
legacy: false,
|
||||
set_count: 1,
|
||||
drives_per_set,
|
||||
endpoints: Endpoints::from(endpoints),
|
||||
cmd_line: format!("delete-prefix-pool-{pool_index}"),
|
||||
platform: "test".to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
let endpoint_pools = EndpointServerPools(pools);
|
||||
let instance_ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
|
||||
crate::store::init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools.clone())
|
||||
.await
|
||||
.expect("multi-pool local disks should initialize");
|
||||
let shutdown = CancellationToken::new();
|
||||
let store = ECStore::new_with_instance_ctx(
|
||||
"127.0.0.1:0".parse().expect("test address should parse"),
|
||||
endpoint_pools,
|
||||
shutdown.clone(),
|
||||
instance_ctx,
|
||||
)
|
||||
.await
|
||||
.expect("multi-pool store should initialize");
|
||||
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
let bucket = format!("delete-prefix-{}", Uuid::new_v4().simple());
|
||||
store
|
||||
.make_bucket(&bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("bucket should be created in both pools");
|
||||
|
||||
let first_pool_disks = store.pools[0].disk_set[0].disks.read().await.clone();
|
||||
for disk in first_pool_disks.iter().flatten() {
|
||||
disk.write_all(&bucket, "blocked", bytes::Bytes::from_static(b"not-a-directory"))
|
||||
.await
|
||||
.expect("first pool should contain a blocking parent file");
|
||||
}
|
||||
let later_pool_disks = store.pools[1].disk_set[0].disks.read().await.clone();
|
||||
let later_data_disk = later_pool_disks[0].clone().expect("later pool should have its first disk");
|
||||
later_data_disk
|
||||
.write_all(&bucket, "blocked/prefix/object", bytes::Bytes::from_static(b"data"))
|
||||
.await
|
||||
.expect("later pool should contain the prefix on its available disk");
|
||||
*store.pools[1].disk_set[0].disks.write().await = vec![Some(later_data_disk.clone()), None, None, None];
|
||||
|
||||
let err = store
|
||||
.delete_prefix(&bucket, "blocked/prefix", &ObjectOptions::default())
|
||||
.await
|
||||
.expect_err("the first pool's hard error must be returned");
|
||||
|
||||
assert!(
|
||||
matches!(err, StorageError::PrefixAccessDenied(ref error_bucket, ref error_prefix)
|
||||
if error_bucket == &bucket && error_prefix == "blocked/prefix"),
|
||||
"unexpected multi-pool delete error: {err:?}"
|
||||
);
|
||||
assert!(matches!(
|
||||
later_data_disk.read_all(&bucket, "blocked/prefix/object").await,
|
||||
Err(DiskError::FileNotFound)
|
||||
));
|
||||
|
||||
*store.pools[1].disk_set[0].disks.write().await = later_pool_disks.clone();
|
||||
for disk in first_pool_disks.iter().flatten() {
|
||||
disk.write_all(&bucket, "second-blocked", bytes::Bytes::from_static(b"not-a-directory"))
|
||||
.await
|
||||
.expect("first pool should contain a second blocking parent file");
|
||||
}
|
||||
for disk in later_pool_disks.iter().flatten() {
|
||||
disk.write_all(&bucket, "second-blocked/prefix/object", bytes::Bytes::from_static(b"data"))
|
||||
.await
|
||||
.expect("later pool should contain the second prefix");
|
||||
}
|
||||
let err = store
|
||||
.delete_prefix(&bucket, "second-blocked/prefix", &ObjectOptions::default())
|
||||
.await
|
||||
.expect_err("a successful later pool must not override the first pool's hard error");
|
||||
assert!(
|
||||
matches!(err, StorageError::PrefixAccessDenied(ref error_bucket, ref error_prefix)
|
||||
if error_bucket == &bucket && error_prefix == "second-blocked/prefix"),
|
||||
"unexpected hard-error plus success result: {err:?}"
|
||||
);
|
||||
for disk in later_pool_disks.iter().flatten() {
|
||||
assert!(matches!(
|
||||
disk.read_all(&bucket, "second-blocked/prefix/object").await,
|
||||
Err(DiskError::FileNotFound)
|
||||
));
|
||||
}
|
||||
|
||||
for disk in later_pool_disks.iter().flatten() {
|
||||
disk.delete_volume(&bucket, true)
|
||||
.await
|
||||
.expect("the bucket should be absent from the later pool");
|
||||
}
|
||||
let healthy_object = "healthy/prefix/object";
|
||||
for disk in first_pool_disks.iter().flatten() {
|
||||
disk.write_all(&bucket, healthy_object, bytes::Bytes::from_static(b"data"))
|
||||
.await
|
||||
.expect("the first pool should contain the healthy prefix");
|
||||
}
|
||||
store
|
||||
.delete_prefix(&bucket, "healthy/prefix", &ObjectOptions::default())
|
||||
.await
|
||||
.expect("one successful pool should make a partially missing bucket idempotent");
|
||||
for disk in first_pool_disks.iter().flatten() {
|
||||
assert!(matches!(disk.read_all(&bucket, healthy_object).await, Err(DiskError::FileNotFound)));
|
||||
}
|
||||
|
||||
let missing_bucket = format!("delete-prefix-missing-{}", Uuid::new_v4().simple());
|
||||
let err = store
|
||||
.delete_prefix(&missing_bucket, "missing/prefix", &ObjectOptions::default())
|
||||
.await
|
||||
.expect_err("a bucket missing from every pool must remain an error");
|
||||
assert_eq!(err, StorageError::BucketNotFound(missing_bucket));
|
||||
|
||||
shutdown.cancel();
|
||||
}
|
||||
|
||||
fn assert_backend_layout_empty(info: &rustfs_madmin::BackendInfo) {
|
||||
assert!(info.standard_sc_parities.is_empty());
|
||||
|
||||
Reference in New Issue
Block a user