mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-29 00:17:11 +00:00
refactor(rustfs): split object_usecase.rs into per-operation app/object modules (#6670)
* refactor(rustfs): carve app/object out of object_usecase.rs — shared, extract, test_support children (backlog#1841 step 1) Mechanical move-only split of rustfs/src/app/object_usecase.rs (19.7K lines). The file body moves to rustfs/src/app/object/mod.rs, and the first self-contained slices move into children: shared.rs (cross-cutting helpers: quota admission, response checksum injection, object-lock write validation, table-catalog mutation guard, deadlock request guard, proxy passthrough utilities), extract.rs (snowball auto-extract path incl. tar/pax helpers and execute_put_object_extract), and cfg(test) test_support.rs for cross-module test scaffolding. object_usecase.rs stays as a thin pub use facade so every existing crate::app::object_usecase:: path keeps working. No behavior change: items move verbatim; the only source edits are visibility widenings required by the new module boundaries (private -> pub(super); pub(super) -> pub(crate) for the three helpers multipart_usecase and the app gating tests import). Guard scripts that pinned rustfs/src/app/object_usecase.rs now scan the rustfs/src/app/object tree, and the table_catalog source-text guard test concatenates the split files. * refactor(rustfs): move the GetObject read path into app/object/get.rs (backlog#1841 step 2) Move-only continuation of the object_usecase split: cold-fill orchestration, disk-permit admission, streaming readers and resume control, stream-buffer tuning, execute_get_object / execute_get_object_attributes, the GET replication proxy helpers, and their unit tests move from app/object/mod.rs into app/object/get.rs. Items keep their original text; cross-module call sites rely on the visibility widenings introduced in step 1. * refactor(rustfs): move the PutObject and CopyObject paths into app/object (backlog#1841 step 3) Move-only continuation: put.rs takes the PUT body admission and timeout readers, zero-copy and eager-commit machinery, execute_put_object, and the PUT unit tests; copy.rs takes the copy namespace/lifecycle lock helpers and execute_copy_object with its tests. Two source edits beyond visibility widenings: PutObjectChecksums fields become pub(super) (read by shared::apply_trailing_checksums across the new module boundary) and one relative super::storage_api call in the copy path becomes crate::app::storage_api since super now resolves to app::object. The table_catalog source-text guard concatenates the new files. * refactor(rustfs): finish the object_usecase split — delete, head, restore modules (backlog#1841 step 4) Move-only completion: delete.rs takes the delete helpers, cfg(test) delete hooks, and execute_delete_object/execute_delete_objects; head.rs takes execute_head_object with the HEAD replication proxy helpers; restore.rs takes execute_restore_object. app/object/mod.rs is now just the shared import prelude, module wiring, and the DefaultObjectUsecase struct with its constructors, accessors, and the execute_select_object_content delegation; the emptied tests module is gone. The delete re-export glob is cfg(test)-gated because its only cross-module consumers are the delete test hooks. The table_catalog source-text guard now isolates the delete entrypoints from app/object/delete.rs, and doc/comment references that pointed at rustfs/src/app/object_usecase.rs internals now point at the per-operation modules.
This commit is contained in:
+1
-1
@@ -73,7 +73,7 @@ The main crate is organized in layers, top to bottom:
|
|||||||
|-------|-----------|----------------|
|
|-------|-----------|----------------|
|
||||||
| **Server** | `server/` | HTTP listener, TLS, CORS, compression, middleware, graceful shutdown |
|
| **Server** | `server/` | HTTP listener, TLS, CORS, compression, middleware, graceful shutdown |
|
||||||
| **Admin** | `admin/` | Admin API routing, 30+ handler modules, web console |
|
| **Admin** | `admin/` | Admin API routing, 30+ handler modules, web console |
|
||||||
| **App** | `app/` | Use-case orchestration: object_usecase, bucket_usecase, multipart_usecase |
|
| **App** | `app/` | Use-case orchestration: object (per-operation modules under `app/object/`, re-exported as `object_usecase`), bucket_usecase, multipart_usecase |
|
||||||
| **Storage** | `storage/` | S3 API translation, erasure-coded FS, SSE encryption, RPC, concurrency |
|
| **Storage** | `storage/` | S3 API translation, erasure-coded FS, SSE encryption, RPC, concurrency |
|
||||||
| **Auth** | `auth.rs` | S3 signature verification, credential validation |
|
| **Auth** | `auth.rs` | S3 signature verification, credential validation |
|
||||||
| **Config** | `config/` | CLI parsing, config struct, workload profiles |
|
| **Config** | `config/` | CLI parsing, config struct, workload profiles |
|
||||||
|
|||||||
@@ -23,8 +23,8 @@
|
|||||||
//! It was fixed in three layers on `main`, each with its own *unit* regression:
|
//! It was fixed in three layers on `main`, each with its own *unit* regression:
|
||||||
//! * rustfs#4594 — `GetObjectStreamingReader::poll_read` now returns
|
//! * rustfs#4594 — `GetObjectStreamingReader::poll_read` now returns
|
||||||
//! `UnexpectedEof` on a short body instead of a clean `Ok(())`
|
//! `UnexpectedEof` on a short body instead of a clean `Ok(())`
|
||||||
//! (`rustfs/src/app/object_usecase.rs`,
|
//! (`rustfs/src/app/object/get.rs`,
|
||||||
//! `app::object_usecase::tests::get_object_streaming_reader_errors_on_short_eof`).
|
//! `app::object::get::tests::get_object_streaming_reader_errors_on_short_eof`).
|
||||||
//! * rustfs#4560 — the lazy multipart codec reader degrades a later part to
|
//! * rustfs#4560 — the lazy multipart codec reader degrades a later part to
|
||||||
//! the legacy per-part decode in place, and surfaces reconstruction errors
|
//! the legacy per-part decode in place, and surfaces reconstruction errors
|
||||||
//! instead of silently truncating
|
//! instead of silently truncating
|
||||||
|
|||||||
@@ -107,7 +107,7 @@ inventory. Generic function-local names such as `CACHE`, `LOCK`, `INIT`, and
|
|||||||
| `AUTH_FS` | `rustfs/src/storage/access.rs` | Cache or constant / owner-local cache | Authorization tag-condition lookup keeps its filesystem helper private to the access owner. |
|
| `AUTH_FS` | `rustfs/src/storage/access.rs` | Cache or constant / owner-local cache | Authorization tag-condition lookup keeps its filesystem helper private to the access owner. |
|
||||||
| `DEADLOCK_DETECTOR` | `rustfs/src/storage/deadlock_detector.rs` | Process-global owner-local state | Deadlock detector lifecycle state stays private to the storage deadlock detector owner. |
|
| `DEADLOCK_DETECTOR` | `rustfs/src/storage/deadlock_detector.rs` | Process-global owner-local state | Deadlock detector lifecycle state stays private to the storage deadlock detector owner. |
|
||||||
| `CONCURRENCY_MANAGER`, `ACTIVE_GET_REQUESTS`, `ACTIVE_PUT_REQUESTS` | `rustfs/src/storage/concurrency/*` | Process-global owner-local scheduler state | Storage concurrency manager and request counters remain inside the storage concurrency owner boundary. |
|
| `CONCURRENCY_MANAGER`, `ACTIVE_GET_REQUESTS`, `ACTIVE_PUT_REQUESTS` | `rustfs/src/storage/concurrency/*` | Process-global owner-local scheduler state | Storage concurrency manager and request counters remain inside the storage concurrency owner boundary. |
|
||||||
| `GET_OBJECT_BUFFER_THRESHOLD_WARNED`, `GET_READER_STREAM_BUFFER_SIZE_OVERRIDE`, function-local `ENABLED`, `OBJECT_SEEK_SUPPORT_THRESHOLD`, `OBJECT_SEEK_SUPPORT_CONCURRENCY_THRESHOLDS` | `rustfs/src/app/object_usecase.rs` | Cache or constant / owner-local cache | Object GET/seek tuning caches and warning guards stay private to object usecase helpers. |
|
| `GET_OBJECT_BUFFER_THRESHOLD_WARNED`, `GET_READER_STREAM_BUFFER_SIZE_OVERRIDE`, function-local `ENABLED`, `OBJECT_SEEK_SUPPORT_THRESHOLD`, `OBJECT_SEEK_SUPPORT_CONCURRENCY_THRESHOLDS` | `rustfs/src/app/object/get.rs` | Cache or constant / owner-local cache | Object GET/seek tuning caches and warning guards stay private to object usecase helpers. |
|
||||||
| `SUPPORTED_HEADERS` | `rustfs/src/storage/options.rs` | Cache or constant / owner-local constant | Supported-header lookup state stays private to storage option parsing. |
|
| `SUPPORTED_HEADERS` | `rustfs/src/storage/options.rs` | Cache or constant / owner-local constant | Supported-header lookup state stays private to storage option parsing. |
|
||||||
| `AUDIT_TARGET_SPECS`, `NOTIFICATION_TARGET_SPECS` | `rustfs/src/admin/handlers/audit.rs`, `rustfs/src/admin/handlers/event.rs`, `rustfs/src/admin/handlers/plugins_instances.rs` | Cache or constant / owner-local constant | Admin target descriptor tables stay private to their handler owners. |
|
| `AUDIT_TARGET_SPECS`, `NOTIFICATION_TARGET_SPECS` | `rustfs/src/admin/handlers/audit.rs`, `rustfs/src/admin/handlers/event.rs`, `rustfs/src/admin/handlers/plugins_instances.rs` | Cache or constant / owner-local constant | Admin target descriptor tables stay private to their handler owners. |
|
||||||
| `SITE_REPLICATION_PEER_CLIENT` | `rustfs/src/admin/handlers/site_replication.rs` | Process-global owner-local cache | Site-replication peer client cache stays private to site-replication handlers. The state RMW transaction holds no process-local mutex — see `rustfs/src/admin/site_replication_state.rs`. |
|
| `SITE_REPLICATION_PEER_CLIENT` | `rustfs/src/admin/handlers/site_replication.rs` | Process-global owner-local cache | Site-replication peer client cache stays private to site-replication handlers. The state RMW transaction holds no process-local mutex — see `rustfs/src/admin/site_replication_state.rs`. |
|
||||||
|
|||||||
@@ -105,7 +105,7 @@ these as 部分兼容 at the client level.
|
|||||||
| CompleteMultipartUpload | 已实现 | `rustfs/src/storage/ecfs.rs` (`complete_multipart_upload`) |
|
| CompleteMultipartUpload | 已实现 | `rustfs/src/storage/ecfs.rs` (`complete_multipart_upload`) |
|
||||||
| AbortMultipartUpload | 已实现 | `rustfs/src/storage/ecfs.rs` (`abort_multipart_upload`) |
|
| AbortMultipartUpload | 已实现 | `rustfs/src/storage/ecfs.rs` (`abort_multipart_upload`) |
|
||||||
| ListParts | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_parts`) |
|
| ListParts | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_parts`) |
|
||||||
| PostObject (POST form upload) | 已实现 | Routed via the POST-object marker into the put-object path (`rustfs/src/app/object_usecase.rs`). See the "POST Object form upload checksum handling: planned" note in `s3-compatibility-matrix.md`. |
|
| PostObject (POST form upload) | 已实现 | Routed via the POST-object marker into the put-object path (`rustfs/src/app/object/put.rs`). See the "POST Object form upload checksum handling: planned" note in `s3-compatibility-matrix.md`. |
|
||||||
| GetObjectTorrent | 行为不一致 | `rustfs/src/storage/ecfs.rs` (`get_object_torrent`) — returns `404 NoSuchKey` by design (not `501 NotImplemented`) so clients degrade gracefully. |
|
| GetObjectTorrent | 行为不一致 | `rustfs/src/storage/ecfs.rs` (`get_object_torrent`) — returns `404 NoSuchKey` by design (not `501 NotImplemented`) so clients degrade gracefully. |
|
||||||
|
|
||||||
For the gate-level view of which of these are covered by executable s3tests,
|
For the gate-level view of which of these are covered by executable s3tests,
|
||||||
|
|||||||
@@ -199,7 +199,7 @@ upgrade.
|
|||||||
When the cluster-level generation capability is **not** negotiated on every
|
When the cluster-level generation capability is **not** negotiated on every
|
||||||
target disk, the behavior **falls back to current semantics** (existing lock +
|
target disk, the behavior **falls back to current semantics** (existing lock +
|
||||||
`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at
|
`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at
|
||||||
`rustfs/src/app/object_usecase.rs`; full fanout for #1314). Fail-closed is
|
`rustfs/src/app/object/get.rs`; full fanout for #1314). Fail-closed is
|
||||||
**only** an explicit administrator strict mode. Defaulting to fail-closed is
|
**only** an explicit administrator strict mode. Defaulting to fail-closed is
|
||||||
forbidden — it makes writes unavailable for the whole rolling-upgrade window.
|
forbidden — it makes writes unavailable for the whole rolling-upgrade window.
|
||||||
|
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ pub mod bucket_usecase;
|
|||||||
pub mod context;
|
pub mod context;
|
||||||
pub(crate) mod metadata_route;
|
pub(crate) mod metadata_route;
|
||||||
pub mod multipart_usecase;
|
pub mod multipart_usecase;
|
||||||
|
pub mod object;
|
||||||
pub(crate) mod object_data_cache;
|
pub(crate) mod object_data_cache;
|
||||||
pub(crate) mod object_traffic_health;
|
pub(crate) mod object_traffic_health;
|
||||||
pub mod object_usecase;
|
pub mod object_usecase;
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,481 @@
|
|||||||
|
// Copyright 2024 RustFS Team
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
//! HeadObject path.
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
impl DefaultObjectUsecase {
|
||||||
|
/// Serve a HEAD whose local lookup failed with not-found by proxying to
|
||||||
|
/// the bucket's replication targets (MinIO `proxyHeadToRepTarget`).
|
||||||
|
async fn proxy_head_object_to_replication_targets(
|
||||||
|
req: &S3Request<HeadObjectInput>,
|
||||||
|
bucket: &str,
|
||||||
|
key: &str,
|
||||||
|
opts: &ObjectOptions,
|
||||||
|
) -> Option<HeadObjectOutput> {
|
||||||
|
let targets = get_read_proxy_targets(bucket, key, opts).await;
|
||||||
|
if targets.is_empty() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let extra_headers = Self::proxy_read_passthrough_headers(&req.headers);
|
||||||
|
let range = req
|
||||||
|
.headers
|
||||||
|
.get(http::header::RANGE)
|
||||||
|
.and_then(|value| value.to_str().ok())
|
||||||
|
.map(str::to_owned);
|
||||||
|
let part_number = req.input.part_number;
|
||||||
|
|
||||||
|
for target in targets {
|
||||||
|
match target
|
||||||
|
.head_object_for_proxy(
|
||||||
|
&target.bucket,
|
||||||
|
key,
|
||||||
|
opts.version_id.clone(),
|
||||||
|
range.clone(),
|
||||||
|
part_number,
|
||||||
|
extra_headers.clone(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
Ok(remote) => {
|
||||||
|
// MinIO-aligned accounting: one total per proxy attempt,
|
||||||
|
// one failed when no target served it.
|
||||||
|
record_replication_proxy(bucket, "HeadObject", false).await;
|
||||||
|
return Some(Self::proxy_sdk_head_output_to_s3s(remote));
|
||||||
|
}
|
||||||
|
Err(err) if Self::proxy_sdk_error_is_not_found(&err) => {
|
||||||
|
debug!(bucket, key, arn = %target.arn, "read proxy: target does not have the object");
|
||||||
|
}
|
||||||
|
Err(err) => {
|
||||||
|
warn!(bucket, key, arn = %target.arn, error = %err, "read proxy: HEAD against replication target failed");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
record_replication_proxy(bucket, "HeadObject", true).await;
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Translate a proxied SDK HEAD response into the s3s output.
|
||||||
|
///
|
||||||
|
/// Known gaps: the SDK's HeadObjectOutput does not model 206/Content-Range
|
||||||
|
/// for a ranged HEAD (the SDK exposes no content_range member on HEAD),
|
||||||
|
/// and s3s' typed HeadObjectOutput has no tag_count field (the local path
|
||||||
|
/// injects x-amz-tagging-count as a raw header) — both are dropped for
|
||||||
|
/// proxied HEADs.
|
||||||
|
fn proxy_sdk_head_output_to_s3s(remote: aws_sdk_s3::operation::head_object::HeadObjectOutput) -> HeadObjectOutput {
|
||||||
|
HeadObjectOutput {
|
||||||
|
content_length: remote.content_length,
|
||||||
|
content_type: remote.content_type.as_deref().and_then(|v| ContentType::from_str(v).ok()),
|
||||||
|
content_encoding: remote.content_encoding,
|
||||||
|
content_disposition: remote.content_disposition,
|
||||||
|
content_language: remote.content_language,
|
||||||
|
cache_control: remote.cache_control,
|
||||||
|
accept_ranges: Some(ACCEPT_RANGES_BYTES.to_string()),
|
||||||
|
e_tag: remote.e_tag.as_deref().and_then(|v| ETag::from_str(v).ok()),
|
||||||
|
last_modified: remote
|
||||||
|
.last_modified
|
||||||
|
.and_then(|dt| OffsetDateTime::from_unix_timestamp_nanos(dt.as_nanos()).ok())
|
||||||
|
.map(Timestamp::from),
|
||||||
|
metadata: remote.metadata,
|
||||||
|
version_id: remote.version_id,
|
||||||
|
server_side_encryption: remote
|
||||||
|
.server_side_encryption
|
||||||
|
.map(|sse| ServerSideEncryption::from(sse.as_str().to_string())),
|
||||||
|
sse_customer_algorithm: remote.sse_customer_algorithm,
|
||||||
|
sse_customer_key_md5: remote.sse_customer_key_md5,
|
||||||
|
ssekms_key_id: remote.ssekms_key_id,
|
||||||
|
parts_count: remote.parts_count,
|
||||||
|
storage_class: remote.storage_class.map(|sc| StorageClass::from(sc.as_str().to_string())),
|
||||||
|
expiration: remote.expiration,
|
||||||
|
restore: remote.restore,
|
||||||
|
checksum_crc32: remote.checksum_crc32,
|
||||||
|
checksum_crc32c: remote.checksum_crc32_c,
|
||||||
|
checksum_crc64nvme: remote.checksum_crc64_nvme,
|
||||||
|
checksum_sha1: remote.checksum_sha1,
|
||||||
|
checksum_sha256: remote.checksum_sha256,
|
||||||
|
checksum_type: remote.checksum_type.map(|ct| ChecksumType::from(ct.as_str().to_string())),
|
||||||
|
..Default::default()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[instrument(level = "debug", skip(self, req))]
|
||||||
|
pub async fn execute_head_object(&self, req: S3Request<HeadObjectInput>) -> S3Result<S3Response<HeadObjectOutput>> {
|
||||||
|
if let Some(context) = &self.context {
|
||||||
|
let _ = context.object_store();
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut helper = OperationHelper::new(&req, EventName::ObjectAccessedHead, S3Operation::HeadObject).suppress_event();
|
||||||
|
// mc get 2
|
||||||
|
let HeadObjectInput {
|
||||||
|
bucket,
|
||||||
|
key,
|
||||||
|
version_id,
|
||||||
|
part_number,
|
||||||
|
range,
|
||||||
|
if_none_match,
|
||||||
|
if_match,
|
||||||
|
if_modified_since,
|
||||||
|
if_unmodified_since,
|
||||||
|
..
|
||||||
|
} = req.input.clone();
|
||||||
|
|
||||||
|
// Validate object key
|
||||||
|
validate_object_key(&key, "HEAD")?;
|
||||||
|
// Parse part number from Option<i32> to Option<usize> with validation
|
||||||
|
let part_number: Option<usize> = parse_part_number_i32_to_usize(part_number, "HEAD")?;
|
||||||
|
|
||||||
|
let rs = range.map(range_to_http_range_spec).transpose()?;
|
||||||
|
|
||||||
|
if rs.is_some() && part_number.is_some() {
|
||||||
|
return Err(s3_error!(InvalidArgument, "range and part_number invalid"));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Establish bucket existence before any bucket-metadata work (matches
|
||||||
|
// PUT/GET): nonexistent buckets fail here instead of paying the
|
||||||
|
// versioning lookup in get_opts first. Resolve the store through the
|
||||||
|
// request-bound server context (backlog#1052 S6), not the
|
||||||
|
// process-global handle.
|
||||||
|
let Some(store) = self.object_store() else {
|
||||||
|
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
|
||||||
|
};
|
||||||
|
validate_bucket_exists(&store, &bucket).await?;
|
||||||
|
|
||||||
|
let opts: ObjectOptions = get_opts(&bucket, &key, version_id, part_number, &req.headers)
|
||||||
|
.await
|
||||||
|
.map_err(ApiError::from)?;
|
||||||
|
|
||||||
|
// Modification Points: Explicitly handles get_object_info errors, distinguishing between object absence and other errors
|
||||||
|
let info = match store.get_object_info(&bucket, &key, &opts).await {
|
||||||
|
Ok(info) => info,
|
||||||
|
Err(err) => {
|
||||||
|
// If the error indicates the object or its version was not found, return 404 (NoSuchKey)
|
||||||
|
if is_err_object_not_found(&err) || is_err_version_not_found(&err) {
|
||||||
|
if is_dir_object(&key) {
|
||||||
|
let has_children = match probe_prefix_has_children(store, &bucket, &key, false).await {
|
||||||
|
Ok(has_children) => has_children,
|
||||||
|
Err(e) => {
|
||||||
|
error!(bucket, key, error = %e, "Failed to probe children for prefix");
|
||||||
|
false
|
||||||
|
}
|
||||||
|
};
|
||||||
|
let msg = head_prefix_not_found_message(&bucket, &key, has_children);
|
||||||
|
return Err(S3Error::with_message(S3ErrorCode::NoSuchKey, msg));
|
||||||
|
}
|
||||||
|
// Active-active replication lag window: an object missing
|
||||||
|
// locally may still be served by proxying the HEAD to a
|
||||||
|
// replication target (backlog#1675 P1-5).
|
||||||
|
if let Some(output) = Self::proxy_head_object_to_replication_targets(&req, &bucket, &key, &opts).await {
|
||||||
|
let response = wrap_response_with_cors(&bucket, &req.method, &req.headers, output).await;
|
||||||
|
let result = Ok(response);
|
||||||
|
let _ = helper
|
||||||
|
.version_id(req.input.version_id.clone().unwrap_or_default())
|
||||||
|
.complete(&result);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
return Err(S3Error::new(S3ErrorCode::NoSuchKey));
|
||||||
|
}
|
||||||
|
// Other errors, such as insufficient permissions, still return the original error
|
||||||
|
return Err(ApiError::from(err).into());
|
||||||
|
}
|
||||||
|
};
|
||||||
|
if info.delete_marker {
|
||||||
|
if opts.version_id.is_none() {
|
||||||
|
return Err(S3Error::new(S3ErrorCode::NoSuchKey));
|
||||||
|
}
|
||||||
|
return Err(S3Error::new(S3ErrorCode::MethodNotAllowed));
|
||||||
|
}
|
||||||
|
if let Some(match_etag) = if_none_match
|
||||||
|
&& let Some(strong_etag) = match_etag.into_etag()
|
||||||
|
&& info
|
||||||
|
.etag
|
||||||
|
.as_ref()
|
||||||
|
.is_some_and(|etag| ETag::Strong(etag.clone()) == strong_etag)
|
||||||
|
{
|
||||||
|
return Err(S3Error::new(S3ErrorCode::NotModified));
|
||||||
|
}
|
||||||
|
if let Some(modified_since) = if_modified_since {
|
||||||
|
// obj_time < givenTime + 1s
|
||||||
|
if info.mod_time.is_some_and(|mod_time| {
|
||||||
|
let give_time: OffsetDateTime = modified_since.into();
|
||||||
|
mod_time < give_time.add(time::Duration::seconds(1))
|
||||||
|
}) {
|
||||||
|
return Err(S3Error::new(S3ErrorCode::NotModified));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(match_etag) = if_match {
|
||||||
|
if let Some(strong_etag) = match_etag.into_etag()
|
||||||
|
&& info
|
||||||
|
.etag
|
||||||
|
.as_ref()
|
||||||
|
.is_some_and(|etag| ETag::Strong(etag.clone()) != strong_etag)
|
||||||
|
{
|
||||||
|
return Err(S3Error::new(S3ErrorCode::PreconditionFailed));
|
||||||
|
}
|
||||||
|
} else if let Some(unmodified_since) = if_unmodified_since
|
||||||
|
&& info.mod_time.is_some_and(|mod_time| {
|
||||||
|
let give_time: OffsetDateTime = unmodified_since.into();
|
||||||
|
mod_time > give_time.add(time::Duration::seconds(1))
|
||||||
|
})
|
||||||
|
{
|
||||||
|
return Err(S3Error::new(S3ErrorCode::PreconditionFailed));
|
||||||
|
}
|
||||||
|
// An authorized replication convergence check only needs etag/size/mtime
|
||||||
|
// to compare source and replica; it holds no customer key, so the SSE-C
|
||||||
|
// read validation is skipped for it (and only it).
|
||||||
|
let replication_check = replication_request_authorized(&req)
|
||||||
|
&& get_header(&req.headers, SUFFIX_SOURCE_REPLICATION_CHECK).as_deref() == Some("true");
|
||||||
|
if !replication_check {
|
||||||
|
validate_sse_headers_for_read(&info.user_defined, &req.headers)?;
|
||||||
|
|
||||||
|
// Validate SSE-C: if the object was encrypted with a customer-provided key,
|
||||||
|
// the caller must supply the matching key even for HEAD requests (per S3 spec).
|
||||||
|
validate_ssec_for_read(
|
||||||
|
&info.user_defined,
|
||||||
|
req.input.sse_customer_key.as_ref(),
|
||||||
|
req.input.sse_customer_key_md5.as_ref(),
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute x-amz-expiration header from lifecycle prediction (before info is partially moved)
|
||||||
|
let expiration_header = resolve_put_object_expiration(&bucket, &info).await;
|
||||||
|
// Clone ObjectInfo for event notification only when an event will
|
||||||
|
// actually be built — the clone is expensive for multipart objects.
|
||||||
|
let event_info = helper.wants_object_info().then(|| info.clone());
|
||||||
|
let content_type = {
|
||||||
|
if let Some(content_type) = &info.content_type {
|
||||||
|
match ContentType::from_str(content_type) {
|
||||||
|
Ok(res) => Some(res),
|
||||||
|
Err(err) => {
|
||||||
|
error!(content_type = %content_type, error = ?err, "Archive content-type parse failed");
|
||||||
|
//
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
};
|
||||||
|
let last_modified = info.mod_time.map(Timestamp::from);
|
||||||
|
|
||||||
|
let content_length = info.get_actual_size().map_err(|e| {
|
||||||
|
error!(error = %e, "Failed to resolve actual object size");
|
||||||
|
ApiError::from(e)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let metadata_map = info.user_defined.clone();
|
||||||
|
let server_side_encryption = metadata_map
|
||||||
|
.get("x-amz-server-side-encryption")
|
||||||
|
.map(|v| ServerSideEncryption::from(v.clone()));
|
||||||
|
let sse_customer_algorithm = metadata_map
|
||||||
|
.get("x-amz-server-side-encryption-customer-algorithm")
|
||||||
|
.map(|v| SSECustomerAlgorithm::from(v.clone()));
|
||||||
|
let sse_customer_key_md5 = metadata_map.get("x-amz-server-side-encryption-customer-key-md5").cloned();
|
||||||
|
let sse_kms_key_id = metadata_map.get("x-amz-server-side-encryption-aws-kms-key-id").cloned();
|
||||||
|
let storage_class = response_storage_class(&info, &metadata_map);
|
||||||
|
// checksum: classify once; additional algorithms (XXHash3/64/128, SHA-512, MD5)
|
||||||
|
// land in `extra` and are emitted as raw headers below (s3s has no typed field).
|
||||||
|
let ResponseChecksums {
|
||||||
|
crc32: checksum_crc32,
|
||||||
|
crc32c: checksum_crc32c,
|
||||||
|
sha1: checksum_sha1,
|
||||||
|
sha256: checksum_sha256,
|
||||||
|
crc64nvme: checksum_crc64nvme,
|
||||||
|
checksum_type,
|
||||||
|
extra: extra_checksum_headers,
|
||||||
|
} = if let Some(checksum_mode) = req.headers.get(AMZ_CHECKSUM_MODE)
|
||||||
|
&& checksum_mode.to_str().unwrap_or_default() == "ENABLED"
|
||||||
|
&& rs.is_none()
|
||||||
|
{
|
||||||
|
let (checksums, is_multipart) = info
|
||||||
|
.decrypt_checksums(opts.part_number.unwrap_or(0), &req.headers)
|
||||||
|
.map_err(ApiError::from)?;
|
||||||
|
classify_response_checksums(checksums, is_multipart)
|
||||||
|
} else {
|
||||||
|
ResponseChecksums::default()
|
||||||
|
};
|
||||||
|
// Extract standard HTTP headers from user_defined metadata
|
||||||
|
// Note: These headers are stored with lowercase keys by extract_metadata_from_mime
|
||||||
|
let cache_control = metadata_map.get("cache-control").cloned();
|
||||||
|
let content_disposition = metadata_map.get("content-disposition").cloned();
|
||||||
|
let content_language = metadata_map.get("content-language").cloned();
|
||||||
|
let website_redirect_location = metadata_map.get(AMZ_WEBSITE_REDIRECT_LOCATION).cloned();
|
||||||
|
let expires = info.expires.map(Timestamp::from);
|
||||||
|
|
||||||
|
// Calculate tag count from user_tags already in ObjectInfo
|
||||||
|
// This avoids an additional API call since user_tags is already populated by get_object_info
|
||||||
|
let tag_count = if !info.user_tags.is_empty() {
|
||||||
|
let tag_set = decode_tags(&info.user_tags);
|
||||||
|
tag_set.len()
|
||||||
|
} else {
|
||||||
|
0
|
||||||
|
};
|
||||||
|
let output = HeadObjectOutput {
|
||||||
|
content_length: Some(content_length),
|
||||||
|
content_type,
|
||||||
|
content_encoding: info.content_encoding.clone(),
|
||||||
|
cache_control,
|
||||||
|
content_disposition,
|
||||||
|
content_language,
|
||||||
|
accept_ranges: Some(ACCEPT_RANGES_BYTES.to_string()),
|
||||||
|
website_redirect_location,
|
||||||
|
expires,
|
||||||
|
last_modified,
|
||||||
|
e_tag: info.etag.map(|etag| to_s3s_etag(&etag)),
|
||||||
|
metadata: filter_object_metadata(&metadata_map),
|
||||||
|
version_id: info.version_id.map(|v| v.to_string()),
|
||||||
|
server_side_encryption,
|
||||||
|
sse_customer_algorithm,
|
||||||
|
sse_customer_key_md5,
|
||||||
|
ssekms_key_id: sse_kms_key_id,
|
||||||
|
checksum_crc32,
|
||||||
|
checksum_crc32c,
|
||||||
|
checksum_sha1,
|
||||||
|
checksum_sha256,
|
||||||
|
checksum_crc64nvme,
|
||||||
|
checksum_type,
|
||||||
|
storage_class,
|
||||||
|
// x-amz-restore from object metadata
|
||||||
|
restore: metadata_map.get(X_AMZ_RESTORE.as_str()).and_then(|v| {
|
||||||
|
let rs = parse_restore_obj_status(v).ok()?;
|
||||||
|
Some(rs.to_string2())
|
||||||
|
}),
|
||||||
|
// x-amz-expiration from lifecycle prediction
|
||||||
|
expiration: expiration_header,
|
||||||
|
// metadata: object_metadata,
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
|
||||||
|
let version_id = req.input.version_id.clone().unwrap_or_default();
|
||||||
|
if let Some(event_info) = event_info {
|
||||||
|
helper = helper.object(event_info);
|
||||||
|
}
|
||||||
|
helper = helper.version_id(version_id);
|
||||||
|
|
||||||
|
// NOTE ON CORS:
|
||||||
|
// Bucket-level CORS headers are intentionally applied only for object retrieval
|
||||||
|
// operations (GET/HEAD) via `wrap_response_with_cors`. Other S3 operations that
|
||||||
|
// interact with objects (PUT/POST/DELETE/LIST, etc.) rely on the system-level
|
||||||
|
// CORS layer instead. In case both are applicable, this bucket-level CORS logic
|
||||||
|
// takes precedence for these read operations.
|
||||||
|
let mut response = wrap_response_with_cors(&bucket, &req.method, &req.headers, output).await;
|
||||||
|
|
||||||
|
// Emit additional-checksum headers (XXHash3/64/128, SHA-512) that s3s cannot
|
||||||
|
// carry on the typed HeadObjectOutput (#1257).
|
||||||
|
inject_additional_checksum_headers(&mut response.headers, &extra_checksum_headers);
|
||||||
|
|
||||||
|
// Add x-amz-tagging-count header if object has tags
|
||||||
|
// Per S3 API spec, this header should be present in HEAD object response when tags exist
|
||||||
|
if tag_count > 0 {
|
||||||
|
let header_name = http::HeaderName::from_static(AMZ_TAG_COUNT);
|
||||||
|
if let Ok(header_value) = tag_count.to_string().parse::<HeaderValue>() {
|
||||||
|
response.headers.insert(header_name, header_value);
|
||||||
|
} else {
|
||||||
|
warn!("Failed to parse x-amz-tagging-count header; skipping");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(retain_date) = metadata_map
|
||||||
|
.get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER)
|
||||||
|
.or_else(|| metadata_map.get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE))
|
||||||
|
&& let Ok(header_name) = http::HeaderName::from_bytes(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.as_bytes())
|
||||||
|
&& let Ok(header_value) = HeaderValue::from_str(retain_date)
|
||||||
|
{
|
||||||
|
response.headers.insert(header_name, header_value);
|
||||||
|
}
|
||||||
|
if let Some(mode) = metadata_map
|
||||||
|
.get(AMZ_OBJECT_LOCK_MODE_LOWER)
|
||||||
|
.or_else(|| metadata_map.get(AMZ_OBJECT_LOCK_MODE))
|
||||||
|
&& let Ok(header_name) = http::HeaderName::from_bytes(AMZ_OBJECT_LOCK_MODE_LOWER.as_bytes())
|
||||||
|
&& let Ok(header_value) = HeaderValue::from_str(mode)
|
||||||
|
{
|
||||||
|
response.headers.insert(header_name, header_value);
|
||||||
|
}
|
||||||
|
if let Some(legal_hold) = metadata_map
|
||||||
|
.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER)
|
||||||
|
.or_else(|| metadata_map.get(AMZ_OBJECT_LOCK_LEGAL_HOLD))
|
||||||
|
&& let Ok(header_name) = http::HeaderName::from_bytes(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.as_bytes())
|
||||||
|
&& let Ok(header_value) = HeaderValue::from_str(legal_hold)
|
||||||
|
{
|
||||||
|
response.headers.insert(header_name, header_value);
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(amz_restore) = metadata_map.get(X_AMZ_RESTORE.as_str()) {
|
||||||
|
let Ok(restore_status) = parse_restore_obj_status(amz_restore) else {
|
||||||
|
return Err(S3Error::with_message(S3ErrorCode::Custom("ErrMeta".into()), "parse amz_restore failed."));
|
||||||
|
};
|
||||||
|
if let Ok(header_value) = HeaderValue::from_str(restore_status.to_string2().as_str()) {
|
||||||
|
response.headers.insert(X_AMZ_RESTORE, header_value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(amz_restore_request_date) = metadata_map.get(AMZ_RESTORE_REQUEST_DATE)
|
||||||
|
&& let Ok(header_name) = http::HeaderName::from_bytes(AMZ_RESTORE_REQUEST_DATE.as_bytes())
|
||||||
|
{
|
||||||
|
let Ok(amz_restore_request_date) = OffsetDateTime::parse(amz_restore_request_date, &Rfc3339) else {
|
||||||
|
return Err(S3Error::with_message(
|
||||||
|
S3ErrorCode::Custom("ErrMeta".into()),
|
||||||
|
"parse amz_restore_request_date failed.",
|
||||||
|
));
|
||||||
|
};
|
||||||
|
let Ok(amz_restore_request_date) = amz_restore_request_date.format(&RFC1123) else {
|
||||||
|
return Err(S3Error::with_message(
|
||||||
|
S3ErrorCode::Custom("ErrMeta".into()),
|
||||||
|
"format amz_restore_request_date failed.",
|
||||||
|
));
|
||||||
|
};
|
||||||
|
if let Ok(header_value) = HeaderValue::from_str(&amz_restore_request_date) {
|
||||||
|
response.headers.insert(header_name, header_value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(amz_restore_expiry_days) = metadata_map.get(AMZ_RESTORE_EXPIRY_DAYS)
|
||||||
|
&& let Ok(header_name) = http::HeaderName::from_bytes(AMZ_RESTORE_EXPIRY_DAYS.as_bytes())
|
||||||
|
&& let Ok(header_value) = HeaderValue::from_str(amz_restore_expiry_days)
|
||||||
|
{
|
||||||
|
response.headers.insert(header_name, header_value);
|
||||||
|
}
|
||||||
|
if info.replication_status != ReplicationStatusType::Empty
|
||||||
|
&& let Ok(header_name) = http::HeaderName::from_bytes(AMZ_BUCKET_REPLICATION_STATUS.to_ascii_lowercase().as_bytes())
|
||||||
|
&& let Ok(header_value) = HeaderValue::from_str(info.replication_status.as_str())
|
||||||
|
{
|
||||||
|
response.headers.insert(header_name, header_value);
|
||||||
|
}
|
||||||
|
|
||||||
|
let result = Ok(response);
|
||||||
|
let _ = helper.complete(&result);
|
||||||
|
|
||||||
|
result
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use http::Method;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn execute_head_object_rejects_range_with_part_number() {
|
||||||
|
let input = HeadObjectInput::builder()
|
||||||
|
.bucket("test-bucket".to_string())
|
||||||
|
.key("test-key".to_string())
|
||||||
|
.part_number(Some(1))
|
||||||
|
.range(Some(Range::Int { first: 0, last: Some(1) }))
|
||||||
|
.build()
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let req = build_request(input, Method::HEAD);
|
||||||
|
let usecase = DefaultObjectUsecase::without_context();
|
||||||
|
|
||||||
|
let err = usecase.execute_head_object(req).await.unwrap_err();
|
||||||
|
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,346 @@
|
|||||||
|
// Copyright 2024 RustFS Team
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
//! Object application use-case contracts.
|
||||||
|
|
||||||
|
// Performance metrics recording (with zero-copy-metrics integration)
|
||||||
|
use rustfs_io_metrics::buffered_write;
|
||||||
|
|
||||||
|
use crate::storage_api::table::get_bucket_metadata;
|
||||||
|
|
||||||
|
use super::storage_api::object_usecase::access::{
|
||||||
|
PostObjectRequestMarker, apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard, authorize_request,
|
||||||
|
has_bypass_governance_header, load_bucket_generation_from_store, recursive_force_delete_is_authorized,
|
||||||
|
replication_request_authorized, req_info_mut, req_info_ref,
|
||||||
|
};
|
||||||
|
#[cfg(test)]
|
||||||
|
use super::storage_api::object_usecase::bucket::quota::BucketQuota;
|
||||||
|
use super::storage_api::object_usecase::bucket::quota::checker::QuotaChecker;
|
||||||
|
#[cfg(test)]
|
||||||
|
use super::storage_api::object_usecase::bucket::replication::{ReplicationState, replication_statuses_map};
|
||||||
|
use super::storage_api::object_usecase::bucket::{
|
||||||
|
VersioningConfigExt as _,
|
||||||
|
lifecycle::{
|
||||||
|
bucket_lifecycle_audit::LcEventSrc,
|
||||||
|
bucket_lifecycle_ops::{enqueue_transition_immediate, post_restore_opts},
|
||||||
|
lifecycle::{self, TransitionOptions},
|
||||||
|
},
|
||||||
|
metadata_sys,
|
||||||
|
object_lock::{
|
||||||
|
objectlock::{get_object_legalhold_meta, get_object_retention_meta},
|
||||||
|
objectlock_sys::{check_object_lock_for_deletion, is_retention_active, replication_write_may_pass_worm_gate},
|
||||||
|
types::RetentionMode,
|
||||||
|
},
|
||||||
|
predict_lifecycle_expiration,
|
||||||
|
quota::{QuotaCheckResult, QuotaError, QuotaOperation},
|
||||||
|
replication::{
|
||||||
|
DeleteReplicationConfigSnapshot, REPLICATE_INCOMING_DELETE, ReplicationStatusType, commit_force_delete_intent,
|
||||||
|
delete_replication_state_from_config, delete_replication_version_id, deleted_object_has_pending_replication_delete,
|
||||||
|
force_delete_target_set, get_read_proxy_targets, has_active_delete_rule, load_delete_config_snapshot,
|
||||||
|
must_replicate_object, persist_force_delete_intent, record_replication_proxy, schedule_object_replication,
|
||||||
|
schedule_replication_delete, schedule_replication_deletes, set_deleted_object_replication_state,
|
||||||
|
should_schedule_delete_replication, should_use_existing_delete_replication_info,
|
||||||
|
},
|
||||||
|
tagging::decode_tags,
|
||||||
|
validate_restore_request,
|
||||||
|
versioning_sys::BucketVersioningSys,
|
||||||
|
};
|
||||||
|
use super::storage_api::object_usecase::compression::{MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible};
|
||||||
|
use super::storage_api::object_usecase::concurrency::{
|
||||||
|
self, ConcurrencyManager, DiskReadAdmission, GetObjectGuard, PutObjectAdmission, PutObjectGuard,
|
||||||
|
get_concurrency_aware_buffer_size, get_concurrency_manager, get_put_concurrency_aware_buffer_size,
|
||||||
|
};
|
||||||
|
#[cfg(test)]
|
||||||
|
use super::storage_api::object_usecase::contract::http::HTTPPreconditions;
|
||||||
|
use super::storage_api::object_usecase::contract::namespace::NamespaceLocking;
|
||||||
|
use super::storage_api::object_usecase::contract::object::{ObjectIO as _, ObjectOperations as _};
|
||||||
|
use super::storage_api::object_usecase::contract::range::HTTPRangeSpec;
|
||||||
|
use super::storage_api::object_usecase::data_usage::{
|
||||||
|
quota_object_size, record_bucket_delete_marker_memory, record_bucket_object_delete_memory,
|
||||||
|
record_bucket_object_version_write_memory, record_bucket_object_write_memory,
|
||||||
|
record_bucket_object_write_unknown_previous_memory,
|
||||||
|
};
|
||||||
|
use super::storage_api::object_usecase::deadlock_detector;
|
||||||
|
use super::storage_api::object_usecase::ecfs::FS;
|
||||||
|
use super::storage_api::object_usecase::error::{
|
||||||
|
Error as EcstoreError, StorageError, is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found,
|
||||||
|
};
|
||||||
|
use super::storage_api::object_usecase::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children};
|
||||||
|
use super::storage_api::object_usecase::helper::{OperationHelper, build_event_resp_elements, spawn_background_with_context};
|
||||||
|
use super::storage_api::object_usecase::io::{DynReader, HashReader, WritePlan, compression_metadata_value, wrap_reader};
|
||||||
|
#[cfg(test)]
|
||||||
|
use super::storage_api::object_usecase::object_cache::GetObjectBodySource;
|
||||||
|
#[cfg(test)]
|
||||||
|
use super::storage_api::object_usecase::object_cache::lookup_get_object_body_cache_hook;
|
||||||
|
use super::storage_api::object_usecase::object_cache::{GetObjectBodyCacheHookLookup, get_object_body_cache_plaintext_len};
|
||||||
|
use super::storage_api::object_usecase::object_utils::to_s3s_etag;
|
||||||
|
use super::storage_api::object_usecase::options::{
|
||||||
|
copy_dst_opts_with_replication_authorization, copy_src_opts, del_opts_with_versioning, extract_metadata,
|
||||||
|
extract_metadata_from_mime_with_object_name, filter_object_metadata, get_content_sha256_with_query, get_opts,
|
||||||
|
has_replication_retention_update, namespace_reserved_user_metadata, normalize_content_encoding_for_storage,
|
||||||
|
preserve_unclassified_user_metadata, put_opts_with_replication_authorization, validate_archive_content_encoding,
|
||||||
|
};
|
||||||
|
use super::storage_api::object_usecase::request_context::{self, spawn_traced, spawn_traced_join};
|
||||||
|
use super::storage_api::object_usecase::s3_api::multipart::parse_list_parts_params;
|
||||||
|
use super::storage_api::object_usecase::set_disk::{
|
||||||
|
get_lock_acquire_timeout, get_object_disk_read_timeout, is_valid_storage_class,
|
||||||
|
};
|
||||||
|
use super::storage_api::object_usecase::sse::{
|
||||||
|
DecryptionRequest, EncryptionRequest, SseKmsPrincipal, apply_bucket_default_lock_retention, authorize_sse_kms_object_read,
|
||||||
|
bucket_default_write_sse, build_ssec_read_headers, classify_sse_read_response, encryption_material_to_metadata,
|
||||||
|
extract_server_side_encryption_from_headers, extract_ssec_params_from_headers, extract_ssekms_context_from_headers,
|
||||||
|
get_buffer_size_opt_in, load_bucket_object_lock_config_state, map_get_object_reader_error, sse_encryption,
|
||||||
|
validate_bucket_object_lock_enabled_state,
|
||||||
|
};
|
||||||
|
use super::storage_api::object_usecase::storage_class as storageclass;
|
||||||
|
use super::storage_api::object_usecase::timeout_wrapper::{GetObjectTimeoutPolicy, RequestTimeoutWrapper};
|
||||||
|
use super::storage_api::object_usecase::{ECStore, OldCurrentSize};
|
||||||
|
use super::storage_api::object_usecase::{
|
||||||
|
RFC1123, check_preconditions, parse_object_lock_legal_hold, parse_object_lock_retention, parse_part_number_i32_to_usize,
|
||||||
|
remove_object_lock_metadata_for_copy, strip_managed_encryption_metadata, validate_bucket_exists, validate_object_key,
|
||||||
|
validate_sse_headers_for_read, validate_sse_headers_for_write, validate_ssec_for_read, wrap_response_with_cors,
|
||||||
|
};
|
||||||
|
use crate::app::runtime_sources::{
|
||||||
|
AppContext, current_app_context, current_notify_interface_for_context, current_object_data_cache_for_context,
|
||||||
|
current_object_store_handle_for_context,
|
||||||
|
};
|
||||||
|
use crate::config::RustFSBufferConfig;
|
||||||
|
use crate::delete_tail_activity::{DeleteTailActivityGuard, DeleteTailStage};
|
||||||
|
use crate::error::ApiError;
|
||||||
|
use crate::shared_types::convert_ecstore_object_info;
|
||||||
|
use crate::table_catalog;
|
||||||
|
use bytes::{BufMut as _, Bytes, BytesMut};
|
||||||
|
use futures::{Stream, StreamExt, TryStreamExt};
|
||||||
|
use http::{HeaderMap, HeaderValue, StatusCode};
|
||||||
|
use md5::{Digest as Md5Digest, Md5};
|
||||||
|
use metrics::{counter, histogram};
|
||||||
|
use pin_project_lite::pin_project;
|
||||||
|
use rustfs_audit::ObjectVersion as AuditObjectVersion;
|
||||||
|
use rustfs_concurrency::GetObjectQueueSnapshot;
|
||||||
|
use rustfs_config::MI_B;
|
||||||
|
use rustfs_filemeta::{NULL_VERSION_ID, RestoreStatusOps, parse_restore_obj_status};
|
||||||
|
use rustfs_io_core::{BytesPool, PooledBuffer};
|
||||||
|
use rustfs_io_metrics;
|
||||||
|
use rustfs_lock::NamespaceLockGuard;
|
||||||
|
use rustfs_notify::EventArgsBuilder;
|
||||||
|
use rustfs_object_capacity::capacity_manager::get_capacity_manager;
|
||||||
|
use rustfs_policy::policy::action::{Action, S3Action};
|
||||||
|
use rustfs_s3_ops::{S3Operation, delete_event_name_for_marker, put_event_name_for_post_object};
|
||||||
|
use rustfs_targets::{EventName, get_request_host, get_request_port, get_request_user_agent};
|
||||||
|
use rustfs_utils::CompressionAlgorithm;
|
||||||
|
#[cfg(test)]
|
||||||
|
use rustfs_utils::http::headers::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
|
||||||
|
#[cfg(test)]
|
||||||
|
use rustfs_utils::http::insert_header;
|
||||||
|
use rustfs_utils::http::{
|
||||||
|
AMZ_BUCKET_REPLICATION_STATUS, AMZ_CHECKSUM_MODE, AMZ_CHECKSUM_TYPE, AMZ_WEBSITE_REDIRECT_LOCATION, CONTENT_TYPE,
|
||||||
|
SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICA_STATUS, SUFFIX_REPLICA_TIMESTAMP,
|
||||||
|
SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_SOURCE_REPLICATION_CHECK,
|
||||||
|
SUFFIX_SOURCE_REPLICATION_REQUEST, get_header,
|
||||||
|
headers::{
|
||||||
|
AMZ_CONTENT_SHA256, AMZ_DECODED_CONTENT_LENGTH, AMZ_MINIO_SNOWBALL_IGNORE_DIRS, AMZ_MINIO_SNOWBALL_IGNORE_ERRORS,
|
||||||
|
AMZ_MINIO_SNOWBALL_PREFIX, AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE,
|
||||||
|
AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
|
||||||
|
AMZ_OBJECT_TAGGING, AMZ_RESTORE_EXPIRY_DAYS, AMZ_RESTORE_REQUEST_DATE, AMZ_RUSTFS_SNOWBALL_IGNORE_DIRS,
|
||||||
|
AMZ_RUSTFS_SNOWBALL_IGNORE_ERRORS, AMZ_RUSTFS_SNOWBALL_PREFIX, AMZ_SERVER_SIDE_ENCRYPTION,
|
||||||
|
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID, AMZ_SNOWBALL_EXTRACT,
|
||||||
|
AMZ_SNOWBALL_IGNORE_DIRS, AMZ_SNOWBALL_IGNORE_ERRORS, AMZ_SNOWBALL_PREFIX, AMZ_STORAGE_CLASS, AMZ_TAG_COUNT,
|
||||||
|
},
|
||||||
|
insert_str, project_ssec_transport_headers, remove_str,
|
||||||
|
};
|
||||||
|
use rustfs_utils::path::{encode_dir_object, is_dir_object, path_join_buf};
|
||||||
|
use rustfs_utils::retry::{DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, RetryTimer};
|
||||||
|
use rustfs_zip::{ArchiveLimits, CompressionFormat};
|
||||||
|
use s3s::StdError;
|
||||||
|
use s3s::dto::{
|
||||||
|
CacheControl, Checksum, ChecksumAlgorithm, ChecksumType, ContentDisposition, ContentEncoding, ContentLanguage, ContentType,
|
||||||
|
CopyObjectInput, CopyObjectOutput, CopyObjectResult, CopySource, DeleteObjectInput, DeleteObjectOutput, DeleteObjectsInput,
|
||||||
|
DeleteObjectsOutput, DeletedObject, ETag, GetObjectAttributesInput, GetObjectAttributesOutput, GetObjectAttributesParts,
|
||||||
|
GetObjectInput, GetObjectOutput, HeadObjectInput, HeadObjectOutput, MetadataDirective, ObjectAttributes, ObjectLockLegalHold,
|
||||||
|
ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ObjectPart, PutObjectInput,
|
||||||
|
PutObjectOutput, Range, RequestCharged, RestoreObjectInput, RestoreObjectOutput, RestoreStatus, SSECustomerAlgorithm,
|
||||||
|
SSECustomerKeyMD5, SSEKMSKeyId, SelectObjectContentInput, SelectObjectContentOutput, ServerSideEncryption,
|
||||||
|
ServerSideEncryptionConfiguration, StorageClass, StreamingBlob, TaggingDirective, TaggingHeader, Timestamp, TimestampFormat,
|
||||||
|
WebsiteRedirectLocation,
|
||||||
|
};
|
||||||
|
use s3s::header::{X_AMZ_RESTORE, X_AMZ_RESTORE_OUTPUT_PATH};
|
||||||
|
use s3s::stream::{ByteStream, DynByteStream, RemainingLength};
|
||||||
|
use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
|
||||||
|
|
||||||
|
mod copy;
|
||||||
|
mod delete;
|
||||||
|
mod extract;
|
||||||
|
mod get;
|
||||||
|
mod head;
|
||||||
|
mod put;
|
||||||
|
mod restore;
|
||||||
|
mod shared;
|
||||||
|
#[cfg(test)]
|
||||||
|
mod test_support;
|
||||||
|
|
||||||
|
pub(crate) use self::copy::*;
|
||||||
|
#[cfg(test)]
|
||||||
|
pub(crate) use self::delete::*;
|
||||||
|
pub(crate) use self::extract::*;
|
||||||
|
pub(crate) use self::get::*;
|
||||||
|
use self::put::*;
|
||||||
|
pub(crate) use self::shared::*;
|
||||||
|
#[cfg(test)]
|
||||||
|
use self::test_support::*;
|
||||||
|
|
||||||
|
use std::collections::HashMap;
|
||||||
|
use std::io;
|
||||||
|
use std::ops::Add;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::pin::Pin;
|
||||||
|
use std::task::{Context, Poll};
|
||||||
|
|
||||||
|
use std::str::FromStr;
|
||||||
|
#[cfg(test)]
|
||||||
|
use std::sync::atomic::AtomicUsize;
|
||||||
|
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||||
|
use std::sync::{Arc, Mutex, OnceLock};
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
|
||||||
|
use tokio::io::{AsyncRead, ReadBuf};
|
||||||
|
use tokio::sync::{OwnedSemaphorePermit, RwLock};
|
||||||
|
use tokio_tar::Archive;
|
||||||
|
#[cfg(test)]
|
||||||
|
use tokio_util::io::ReaderStream;
|
||||||
|
use tokio_util::io::{StreamReader, poll_read_buf};
|
||||||
|
use tracing::{debug, error, instrument, warn};
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
use super::storage_api::object_usecase::{
|
||||||
|
BUCKET_LIFECYCLE_LOCK_OBJECT, GetObjectReader, StorageDeletedObject, StorageObjectInfo as ObjectInfo,
|
||||||
|
StorageObjectLockDeleteOptions, StorageObjectOptions as ObjectOptions, StorageObjectToDelete as ObjectToDelete,
|
||||||
|
StoragePutObjReader as PutObjReader,
|
||||||
|
};
|
||||||
|
use crate::app::object_data_cache::{
|
||||||
|
ColdFillCoordinateOutcome, ColdFillDiskPermitOwner, ColdFillError, ColdFillProducer, GetObjectBodyCacheLookup,
|
||||||
|
GetObjectBodyCachePlan, GetObjectBodyCacheRequest, ObjectDataCacheAdapter, build_get_object_body_cache_plan,
|
||||||
|
build_get_object_body_cache_plan_for_revalidation, coordinate_cold_fill, current_cold_fill_disk_permit_owner,
|
||||||
|
fill_get_object_body_cache_from_buffered_body, fill_get_object_body_cache_from_materialized_body,
|
||||||
|
invalidate_object_data_cache_after_copy_success, invalidate_object_data_cache_after_delete_success,
|
||||||
|
invalidate_object_data_cache_after_put_success, invalidate_object_data_cache_before_mutation,
|
||||||
|
invalidate_object_data_cache_objects_after_delete_success, invalidate_object_data_cache_objects_before_mutation,
|
||||||
|
invalidate_object_data_cache_prefix_after_delete, invalidate_object_data_cache_prefix_before_mutation,
|
||||||
|
lookup_get_object_body_cache_hit, lookup_preplanned_get_object_body_cache_hook,
|
||||||
|
};
|
||||||
|
#[cfg(test)]
|
||||||
|
use crate::app::object_data_cache::{ColdFillRole, ColdFillWaitOutcome, scope_cold_fill_disk_permit_owner_for_test};
|
||||||
|
use crate::app::object_traffic_health::ObjectTrafficHealth;
|
||||||
|
|
||||||
|
#[derive(Clone, Default)]
|
||||||
|
pub struct DefaultObjectUsecase {
|
||||||
|
context: Option<Arc<AppContext>>,
|
||||||
|
#[cfg(test)]
|
||||||
|
get_object_timeout_policy: Option<GetObjectTimeoutPolicy>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl DefaultObjectUsecase {
|
||||||
|
#[cfg(test)]
|
||||||
|
pub fn without_context() -> Self {
|
||||||
|
Self {
|
||||||
|
context: None,
|
||||||
|
get_object_timeout_policy: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn from_global() -> Self {
|
||||||
|
Self {
|
||||||
|
context: current_app_context(),
|
||||||
|
#[cfg(test)]
|
||||||
|
get_object_timeout_policy: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build the use-case bound to an explicit application context
|
||||||
|
/// (backlog#1052 S6): the per-server request path passes its own context
|
||||||
|
/// so the use-case resolves that server's store; `None` falls back to the
|
||||||
|
/// ambient default.
|
||||||
|
pub fn with_context(context: Option<std::sync::Arc<crate::runtime_sources::AppContext>>) -> Self {
|
||||||
|
Self {
|
||||||
|
context,
|
||||||
|
#[cfg(test)]
|
||||||
|
get_object_timeout_policy: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
fn with_context_and_get_object_timeout_policy(
|
||||||
|
context: Option<std::sync::Arc<crate::runtime_sources::AppContext>>,
|
||||||
|
get_object_timeout_policy: GetObjectTimeoutPolicy,
|
||||||
|
) -> Self {
|
||||||
|
Self {
|
||||||
|
context,
|
||||||
|
get_object_timeout_policy: Some(get_object_timeout_policy),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bucket_metadata_sys(&self) -> Option<Arc<RwLock<metadata_sys::BucketMetadataSys>>> {
|
||||||
|
self.context.as_ref().and_then(|context| context.bucket_metadata().handle())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn object_store(&self) -> Option<Arc<ECStore>> {
|
||||||
|
current_object_store_handle_for_context(self.context.as_deref())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn object_data_cache(&self) -> Arc<ObjectDataCacheAdapter> {
|
||||||
|
current_object_data_cache_for_context(self.context.as_deref())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn object_traffic_health(&self) -> Option<Arc<ObjectTrafficHealth>> {
|
||||||
|
self.context
|
||||||
|
.as_ref()
|
||||||
|
.map(|context| context.object_traffic_health())
|
||||||
|
.or_else(|| current_app_context().map(|context| context.object_traffic_health()))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn base_buffer_size(&self) -> usize {
|
||||||
|
self.context
|
||||||
|
.clone()
|
||||||
|
.or_else(current_app_context)
|
||||||
|
.map(|context| context.buffer_config().get().base_config.default_unknown)
|
||||||
|
.unwrap_or_else(|| RustFSBufferConfig::default().base_config.default_unknown)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn check_bucket_quota(&self, bucket: &str, op: QuotaOperation, size: u64) -> S3Result<Option<QuotaCheckResult>> {
|
||||||
|
let Some(metadata_sys) = self.bucket_metadata_sys() else {
|
||||||
|
return Ok(None);
|
||||||
|
};
|
||||||
|
let quota_checker = QuotaChecker::new(metadata_sys);
|
||||||
|
map_quota_check_outcome(bucket, quota_checker.check_quota(bucket, op, size).await).map(Some)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[hotpath::measure(
|
||||||
|
label = "rustfs::app::object_usecase::DefaultObjectUsecase::execute_put_object",
|
||||||
|
impl_type = "DefaultObjectUsecase"
|
||||||
|
)]
|
||||||
|
#[hotpath::measure(
|
||||||
|
label = "rustfs::app::object_usecase::DefaultObjectUsecase::execute_get_object",
|
||||||
|
impl_type = "DefaultObjectUsecase"
|
||||||
|
)]
|
||||||
|
#[instrument(level = "debug", skip(self, req))]
|
||||||
|
pub async fn execute_select_object_content(
|
||||||
|
&self,
|
||||||
|
req: S3Request<SelectObjectContentInput>,
|
||||||
|
) -> S3Result<S3Response<SelectObjectContentOutput>> {
|
||||||
|
if let Some(context) = &self.context {
|
||||||
|
let _ = context.object_store();
|
||||||
|
}
|
||||||
|
|
||||||
|
crate::app::select_object::execute_select_object_content(req).await
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,355 @@
|
|||||||
|
// Copyright 2024 RustFS Team
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
//! RestoreObject path.
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
impl DefaultObjectUsecase {
|
||||||
|
#[instrument(level = "debug", skip(self, req))]
|
||||||
|
pub async fn execute_restore_object(&self, req: S3Request<RestoreObjectInput>) -> S3Result<S3Response<RestoreObjectOutput>> {
|
||||||
|
if let Some(context) = &self.context {
|
||||||
|
let _ = context.object_store();
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut helper = OperationHelper::new(&req, EventName::ObjectRestorePost, S3Operation::RestoreObject);
|
||||||
|
let RestoreObjectInput {
|
||||||
|
bucket,
|
||||||
|
key: object,
|
||||||
|
restore_request: rreq,
|
||||||
|
version_id,
|
||||||
|
..
|
||||||
|
} = req.input.clone();
|
||||||
|
|
||||||
|
validate_table_catalog_object_mutation(&bucket, &object).await?;
|
||||||
|
|
||||||
|
let rreq = rreq.ok_or_else(|| {
|
||||||
|
S3Error::with_message(S3ErrorCode::Custom("ErrValidRestoreObject".into()), "restore request is required")
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let Some(store) = self.object_store() else {
|
||||||
|
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
|
||||||
|
};
|
||||||
|
|
||||||
|
let version_id_str = version_id.clone().unwrap_or_default();
|
||||||
|
let mut opts = post_restore_opts(&version_id_str, &bucket, &object)
|
||||||
|
.await
|
||||||
|
.map_err(|_| S3Error::with_message(S3ErrorCode::Custom("ErrPostRestoreOpts".into()), "restore object failed."))?;
|
||||||
|
apply_bucket_generation_guard(&req, &bucket, &mut opts)?;
|
||||||
|
// `apply_bucket_generation_guard` deliberately tolerates a missing guard
|
||||||
|
// (only the S3 access layer installs one), so this must not hard-require
|
||||||
|
// it. Resolve the current generation instead, exactly as the copy path
|
||||||
|
// does. The fence is unaffected: the value is re-read from disk and
|
||||||
|
// compared below, before the restore is admitted.
|
||||||
|
let restore_bucket_incarnation_id = match opts.expected_bucket_incarnation_id {
|
||||||
|
Some(incarnation_id) => incarnation_id,
|
||||||
|
None => {
|
||||||
|
let incarnation_id = store.bucket_incarnation_id_from_disk(&bucket).await.map_err(ApiError::from)?;
|
||||||
|
opts.expected_bucket_incarnation_id = Some(incarnation_id);
|
||||||
|
incarnation_id
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
// SELECT-type restores skip both the ongoing check and the metadata
|
||||||
|
// write below, so the accept guard would protect nothing for them —
|
||||||
|
// they keep the plain (read-locked) accept path.
|
||||||
|
let is_select = rreq.type_.as_ref().is_some_and(|t| t.as_str() == "SELECT");
|
||||||
|
|
||||||
|
// Hold the restore-accept guard across the restore-status read, the
|
||||||
|
// ongoing/already-restored decision, and the metadata write below, so
|
||||||
|
// two concurrent (non-SELECT) POST ?restore cannot both observe
|
||||||
|
// ongoing=false and both start a copy-back (backlog#1304). Reads and
|
||||||
|
// writes inside this scope run with no_lock; the guard is dropped
|
||||||
|
// before the copy-back is spawned so it never blocks readers.
|
||||||
|
// Contention on the accept guard (e.g. a concurrent accept or an
|
||||||
|
// in-flight commit on the same object) is transient — answer 503
|
||||||
|
// SlowDown so SDK clients back off and retry instead of treating it
|
||||||
|
// as a hard failure.
|
||||||
|
let restore_bucket_lifecycle_guard = Some(acquire_copy_bucket_lifecycle_lock(store.as_ref(), &bucket).await?);
|
||||||
|
if store.bucket_incarnation_id_from_disk(&bucket).await.map_err(ApiError::from)? != restore_bucket_incarnation_id {
|
||||||
|
return Err(ApiError::from(StorageError::BucketNotFound(bucket.clone())).into());
|
||||||
|
}
|
||||||
|
let accept_guard = if is_select {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
let guard = store
|
||||||
|
.acquire_restore_accept_guard(&bucket, &object)
|
||||||
|
.await
|
||||||
|
.map_err(|_| S3Error::with_message(S3ErrorCode::SlowDown, "restore object failed."))?;
|
||||||
|
opts.no_lock = true;
|
||||||
|
Some(guard)
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut obj_info = store
|
||||||
|
.get_object_info(&bucket, &object, &opts)
|
||||||
|
.await
|
||||||
|
.map_err(|_| S3Error::with_message(S3ErrorCode::Custom("ErrInvalidObjectState".into()), "restore object failed."))?;
|
||||||
|
|
||||||
|
// Check if object is in a transitioned state
|
||||||
|
if obj_info.transitioned_object.status != lifecycle::TRANSITION_COMPLETE {
|
||||||
|
return Err(S3Error::with_message(
|
||||||
|
S3ErrorCode::Custom("ErrInvalidTransitionedState".into()),
|
||||||
|
"restore object failed.",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate restore request
|
||||||
|
if let Err(e) = validate_restore_request(&rreq, store.clone()) {
|
||||||
|
return Err(S3Error::with_message(
|
||||||
|
S3ErrorCode::Custom("ErrValidRestoreObject".into()),
|
||||||
|
format!("Restore object validation failed: {}", e),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if restore is already in progress. AWS answers this with
|
||||||
|
// 409 RestoreAlreadyInProgress; a Custom code would serialize as a
|
||||||
|
// retryable 500 and make SDK clients retry the conflict (backlog#1304).
|
||||||
|
if obj_info.restore_ongoing && !is_select {
|
||||||
|
return Err(S3Error::with_message(
|
||||||
|
S3ErrorCode::RestoreAlreadyInProgress,
|
||||||
|
"Object restore is already in progress.",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut already_restored = false;
|
||||||
|
if let Some(restore_expires) = obj_info.restore_expires
|
||||||
|
&& !obj_info.restore_ongoing
|
||||||
|
&& restore_expires.unix_timestamp() != 0
|
||||||
|
{
|
||||||
|
already_restored = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), *rreq.days.as_ref().unwrap_or(&1));
|
||||||
|
let mut metadata = (*obj_info.user_defined).clone();
|
||||||
|
let restore_operation_id = (!is_select && !already_restored).then(Uuid::new_v4);
|
||||||
|
|
||||||
|
let mut header = HeaderMap::new();
|
||||||
|
|
||||||
|
let event_object_info = obj_info.clone();
|
||||||
|
let obj_info_ = obj_info.clone();
|
||||||
|
if !is_select {
|
||||||
|
obj_info.metadata_only = true;
|
||||||
|
metadata.insert(AMZ_RESTORE_EXPIRY_DAYS.to_string(), rreq.days.unwrap_or(1).to_string());
|
||||||
|
let request_date = OffsetDateTime::now_utc().format(&Rfc3339).map_err(|e| {
|
||||||
|
S3Error::with_message(S3ErrorCode::InternalError, format!("format restore request date failed: {}", e))
|
||||||
|
})?;
|
||||||
|
metadata.insert(AMZ_RESTORE_REQUEST_DATE.to_string(), request_date);
|
||||||
|
if already_restored {
|
||||||
|
metadata.insert(
|
||||||
|
X_AMZ_RESTORE.as_str().to_string(),
|
||||||
|
RestoreStatus {
|
||||||
|
is_restore_in_progress: Some(false),
|
||||||
|
restore_expiry_date: Some(Timestamp::from(restore_expiry)),
|
||||||
|
}
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
metadata.insert(
|
||||||
|
X_AMZ_RESTORE.as_str().to_string(),
|
||||||
|
RestoreStatus {
|
||||||
|
is_restore_in_progress: Some(true),
|
||||||
|
restore_expiry_date: Some(Timestamp::from(OffsetDateTime::now_utc())),
|
||||||
|
}
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
if let Some(id) = restore_operation_id {
|
||||||
|
insert_str(&mut metadata, SUFFIX_RESTORE_OPERATION_ID, id.to_string());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
obj_info.user_defined = Arc::new(metadata);
|
||||||
|
|
||||||
|
// Fence the compare-and-set write: if the accept guard was lost
|
||||||
|
// (lock-service degradation), another node may have concurrently
|
||||||
|
// accepted this restore — back off instead of committing a second
|
||||||
|
// ongoing flag and double-starting the copy-back.
|
||||||
|
if accept_guard.as_ref().is_some_and(|g| g.is_lock_lost()) {
|
||||||
|
return Err(S3Error::with_message(S3ErrorCode::SlowDown, "restore object failed."));
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut restore_dst_opts = ObjectOptions {
|
||||||
|
version_id: obj_info_.version_id.map(|v| v.to_string()),
|
||||||
|
mod_time: obj_info_.mod_time,
|
||||||
|
no_lock: true,
|
||||||
|
expected_bucket_incarnation_id: Some(restore_bucket_incarnation_id),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
if let Some(guard) = restore_bucket_lifecycle_guard.as_ref() {
|
||||||
|
restore_dst_opts.add_bucket_lifecycle_lock_guard(guard);
|
||||||
|
}
|
||||||
|
if let Some(guard) = accept_guard.as_ref() {
|
||||||
|
guard.add_namespace_lock_fence(&mut restore_dst_opts);
|
||||||
|
}
|
||||||
|
store
|
||||||
|
.clone()
|
||||||
|
.copy_object(
|
||||||
|
&bucket,
|
||||||
|
&object,
|
||||||
|
&bucket,
|
||||||
|
&object,
|
||||||
|
&mut obj_info,
|
||||||
|
&ObjectOptions {
|
||||||
|
version_id: obj_info_.version_id.map(|v| v.to_string()),
|
||||||
|
// Inside the accept-guard critical section (see above).
|
||||||
|
no_lock: true,
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
&restore_dst_opts,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.map_err(|_| S3Error::with_message(S3ErrorCode::Custom("ErrCopyObject".into()), "restore object failed."))?;
|
||||||
|
rustfs_scanner::record_dirty_usage_bucket(&bucket);
|
||||||
|
|
||||||
|
if already_restored {
|
||||||
|
let output = RestoreObjectOutput {
|
||||||
|
request_charged: Some(RequestCharged::from_static(RequestCharged::REQUESTER)),
|
||||||
|
restore_output_path: None,
|
||||||
|
};
|
||||||
|
helper = helper
|
||||||
|
.object(event_object_info.clone())
|
||||||
|
.version_id(version_id_str.clone())
|
||||||
|
.suppress_event();
|
||||||
|
let result = Ok(S3Response::new(output));
|
||||||
|
let _ = helper.complete(&result);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The accept decision is committed; release the object write lock so
|
||||||
|
// the background copy-back and concurrent reads are never blocked on it.
|
||||||
|
drop(accept_guard);
|
||||||
|
drop(restore_bucket_lifecycle_guard);
|
||||||
|
|
||||||
|
// Handle output location for SELECT requests
|
||||||
|
if let Some(output_location) = &rreq.output_location
|
||||||
|
&& let Some(s3) = &output_location.s3
|
||||||
|
&& !s3.bucket_name.is_empty()
|
||||||
|
{
|
||||||
|
let restore_object = Uuid::new_v4().to_string();
|
||||||
|
if let Ok(header_value) = format!("{}{}{}", s3.bucket_name, s3.prefix, restore_object).parse() {
|
||||||
|
header.insert(X_AMZ_RESTORE_OUTPUT_PATH, header_value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Spawn restoration task in the background. Pin the copy-back to the
|
||||||
|
// version the accept resolved and flagged: with a versionless request
|
||||||
|
// on a versioned bucket, a PUT landing between the accept and the
|
||||||
|
// copy-back would otherwise re-resolve "latest" to the new version,
|
||||||
|
// fail (not transitioned), and strand the flagged version at
|
||||||
|
// ongoing=true forever (backlog#1304).
|
||||||
|
let store_clone = store.clone();
|
||||||
|
let bucket_clone = bucket.clone();
|
||||||
|
let object_clone = object.clone();
|
||||||
|
let rreq_clone = rreq.clone();
|
||||||
|
let version_id_clone = obj_info_
|
||||||
|
.version_id
|
||||||
|
.map(|v| v.to_string())
|
||||||
|
.or_else(|| (opts.versioned || opts.version_suspended).then(|| Uuid::nil().to_string()));
|
||||||
|
let versioned = opts.versioned;
|
||||||
|
let version_suspended = opts.version_suspended;
|
||||||
|
let mut restore_operation_metadata = HashMap::new();
|
||||||
|
if let Some(id) = restore_operation_id {
|
||||||
|
insert_str(&mut restore_operation_metadata, SUFFIX_RESTORE_OPERATION_ID, id.to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
spawn_traced(async move {
|
||||||
|
let opts = ObjectOptions {
|
||||||
|
transition: TransitionOptions {
|
||||||
|
restore_request: rreq_clone,
|
||||||
|
restore_expiry,
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
version_id: version_id_clone,
|
||||||
|
versioned,
|
||||||
|
version_suspended,
|
||||||
|
expected_bucket_incarnation_id: Some(restore_bucket_incarnation_id),
|
||||||
|
user_defined: restore_operation_metadata,
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
|
||||||
|
if let Err(err) = store_clone
|
||||||
|
.restore_transitioned_object(&bucket_clone, &object_clone, &opts)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
warn!(
|
||||||
|
"unable to restore transitioned bucket/object {}/{}: {}",
|
||||||
|
bucket_clone,
|
||||||
|
object_clone,
|
||||||
|
err.to_string()
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
rustfs_scanner::record_dirty_usage_bucket(&bucket_clone);
|
||||||
|
debug!(bucket = %bucket_clone, object = %object_clone, "Transitioned object restored");
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let output = RestoreObjectOutput {
|
||||||
|
request_charged: Some(RequestCharged::from_static(RequestCharged::REQUESTER)),
|
||||||
|
restore_output_path: None,
|
||||||
|
};
|
||||||
|
helper = helper.object(event_object_info).version_id(version_id_str);
|
||||||
|
let result = Ok(S3Response::with_headers(output, header));
|
||||||
|
let _ = helper.complete(&result);
|
||||||
|
result
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use http::Method;
|
||||||
|
use s3s::dto::RestoreRequest;
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn execute_restore_object_rejects_missing_restore_request() {
|
||||||
|
let input = RestoreObjectInput::builder()
|
||||||
|
.bucket("test-bucket".to_string())
|
||||||
|
.key("test-key".to_string())
|
||||||
|
.build()
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let req = build_request(input, Method::POST);
|
||||||
|
let usecase = DefaultObjectUsecase::without_context();
|
||||||
|
|
||||||
|
let err = usecase.execute_restore_object(req).await.unwrap_err();
|
||||||
|
match err.code() {
|
||||||
|
S3ErrorCode::Custom(code) => assert_eq!(code, "ErrValidRestoreObject"),
|
||||||
|
code => panic!("unexpected error code: {:?}", code),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn execute_restore_object_returns_internal_error_when_store_uninitialized() {
|
||||||
|
let restore_request = RestoreRequest {
|
||||||
|
days: Some(1),
|
||||||
|
description: None,
|
||||||
|
glacier_job_parameters: None,
|
||||||
|
output_location: None,
|
||||||
|
select_parameters: None,
|
||||||
|
tier: None,
|
||||||
|
type_: None,
|
||||||
|
};
|
||||||
|
let input = RestoreObjectInput::builder()
|
||||||
|
.bucket("test-bucket".to_string())
|
||||||
|
.key("test-key".to_string())
|
||||||
|
.restore_request(Some(restore_request))
|
||||||
|
.build()
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let req = build_request(input, Method::POST);
|
||||||
|
let usecase = DefaultObjectUsecase::without_context();
|
||||||
|
|
||||||
|
let err = usecase.execute_restore_object(req).await.unwrap_err();
|
||||||
|
assert_eq!(err.code(), &S3ErrorCode::InternalError);
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,98 @@
|
|||||||
|
// Copyright 2024 RustFS Team
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
//! Test-only scaffolding shared by the object use-case test modules.
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
use http::{Extensions, HeaderMap, Method, Uri};
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub(super) struct MockUploadStreamSha256Mismatch;
|
||||||
|
|
||||||
|
impl std::fmt::Display for MockUploadStreamSha256Mismatch {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
f.write_str("UploadStreamError: Sha256Mismatch")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::error::Error for MockUploadStreamSha256Mismatch {}
|
||||||
|
|
||||||
|
pub(super) fn build_request<T>(input: T, method: Method) -> S3Request<T> {
|
||||||
|
S3Request {
|
||||||
|
input,
|
||||||
|
method,
|
||||||
|
uri: Uri::from_static("/"),
|
||||||
|
headers: HeaderMap::new(),
|
||||||
|
extensions: Extensions::new(),
|
||||||
|
credentials: None,
|
||||||
|
region: None,
|
||||||
|
service: None,
|
||||||
|
trailing_headers: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) async fn real_cold_fill_test_context() -> (Arc<ECStore>, Arc<AppContext>) {
|
||||||
|
let store = crate::app::gating_test_env::shared_gating_ecstore().await;
|
||||||
|
if current_app_context().is_none() {
|
||||||
|
crate::app::runtime_sources::install_test_app_context(Arc::clone(&store)).await;
|
||||||
|
}
|
||||||
|
let ambient = current_app_context().expect("real cold-fill tests require an ambient AppContext");
|
||||||
|
let context = temp_env::with_vars(
|
||||||
|
[
|
||||||
|
(rustfs_config::ENV_OBJECT_DATA_CACHE_ENABLE, Some("true")),
|
||||||
|
(rustfs_config::ENV_OBJECT_DATA_CACHE_MODE, Some("fill_materialize_enabled")),
|
||||||
|
(rustfs_config::ENV_OBJECT_DATA_CACHE_MAX_BYTES, Some("8388608")),
|
||||||
|
(rustfs_config::ENV_OBJECT_DATA_CACHE_MAX_ENTRY_BYTES, Some("2097152")),
|
||||||
|
(rustfs_config::ENV_OBJECT_DATA_CACHE_MIN_FREE_MEMORY_PERCENT, Some("0")),
|
||||||
|
],
|
||||||
|
|| Arc::new(AppContext::new(Arc::clone(&store), ambient.iam(), ambient.kms())),
|
||||||
|
);
|
||||||
|
assert!(context.object_data_cache().materialize_fill_enabled());
|
||||||
|
(store, context)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) async fn put_real_cold_fill_object(store: &Arc<ECStore>, bucket: &str, object: &str, body: &[u8]) -> ObjectInfo {
|
||||||
|
let mut reader = PutObjReader::from_vec(body.to_vec());
|
||||||
|
store
|
||||||
|
.put_object(bucket, object, &mut reader, &ObjectOptions::default())
|
||||||
|
.await
|
||||||
|
.expect("real cold-fill test object must be written")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn real_cold_fill_plan(
|
||||||
|
adapter: &ObjectDataCacheAdapter,
|
||||||
|
bucket: &str,
|
||||||
|
object: &str,
|
||||||
|
info: &ObjectInfo,
|
||||||
|
) -> rustfs_object_data_cache::ObjectDataCacheGetPlan {
|
||||||
|
let length = info
|
||||||
|
.get_actual_size()
|
||||||
|
.expect("real cold-fill test metadata must expose plaintext size");
|
||||||
|
let GetObjectBodyCachePlan::Cacheable(plan) = build_get_object_body_cache_plan(
|
||||||
|
adapter,
|
||||||
|
GetObjectBodyCacheRequest {
|
||||||
|
bucket,
|
||||||
|
key: object,
|
||||||
|
info,
|
||||||
|
response_content_length: length,
|
||||||
|
has_range: false,
|
||||||
|
part_number: None,
|
||||||
|
encryption_applied: false,
|
||||||
|
},
|
||||||
|
) else {
|
||||||
|
panic!("real cold-fill test object must be cacheable");
|
||||||
|
};
|
||||||
|
plan
|
||||||
|
}
|
||||||
+5
-19691
File diff suppressed because it is too large
Load Diff
@@ -16415,12 +16415,22 @@ fn table_metadata_pointer_json_round_trips() {
|
|||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn object_mutation_entrypoints_call_reserved_prefix_guard() {
|
fn object_mutation_entrypoints_call_reserved_prefix_guard() {
|
||||||
let source = include_str!("../app/object_usecase.rs");
|
let source = [
|
||||||
let delete_object = source
|
include_str!("../app/object/mod.rs"),
|
||||||
|
include_str!("../app/object/shared.rs"),
|
||||||
|
include_str!("../app/object/extract.rs"),
|
||||||
|
include_str!("../app/object/put.rs"),
|
||||||
|
include_str!("../app/object/copy.rs"),
|
||||||
|
include_str!("../app/object/delete.rs"),
|
||||||
|
include_str!("../app/object/head.rs"),
|
||||||
|
include_str!("../app/object/restore.rs"),
|
||||||
|
]
|
||||||
|
.concat();
|
||||||
|
let delete_module = include_str!("../app/object/delete.rs");
|
||||||
|
let delete_object = delete_module
|
||||||
.split_once("pub async fn execute_delete_object")
|
.split_once("pub async fn execute_delete_object")
|
||||||
.and_then(|(_, remainder)| remainder.split_once("pub async fn execute_head_object"))
|
.map(|(_, remainder)| remainder.split_once("\nmod tests").map_or(remainder, |(entrypoints, _)| entrypoints))
|
||||||
.map(|(delete_object, _)| delete_object)
|
.expect("delete object entrypoints should remain in app/object/delete.rs");
|
||||||
.expect("delete object entrypoint should remain in the object usecase");
|
|
||||||
|
|
||||||
for expected in [
|
for expected in [
|
||||||
"validate_object_key(&key, request_method_name)?;\n validate_table_catalog_object_mutation(&bucket, &key).await?;",
|
"validate_object_key(&key, request_method_name)?;\n validate_table_catalog_object_mutation(&bucket, &key).await?;",
|
||||||
|
|||||||
@@ -2166,7 +2166,7 @@ fi
|
|||||||
|
|
||||||
(
|
(
|
||||||
cd "$ROOT_DIR"
|
cd "$ROOT_DIR"
|
||||||
rg -n --with-filename 'crate::app::(?:bucket_usecase|multipart_usecase|object_usecase)|Default(?:Bucket|Multipart|Object)Usecase::from_global\(\)' \
|
rg -n --with-filename 'crate::app::(?:bucket_usecase|multipart_usecase|object_usecase|object\b)|Default(?:Bucket|Multipart|Object)Usecase::from_global\(\)' \
|
||||||
rustfs/src/storage/ecfs.rs \
|
rustfs/src/storage/ecfs.rs \
|
||||||
--glob '*.rs' || true
|
--glob '*.rs' || true
|
||||||
) >"$RUSTFS_STORAGE_ECFS_USECASE_BYPASS_HITS_FILE"
|
) >"$RUSTFS_STORAGE_ECFS_USECASE_BYPASS_HITS_FILE"
|
||||||
@@ -2220,7 +2220,7 @@ fi
|
|||||||
|
|
||||||
(
|
(
|
||||||
cd "$ROOT_DIR"
|
cd "$ROOT_DIR"
|
||||||
rg -n --with-filename 'use crate::storage::\*;' rustfs/src/app --glob '*_usecase.rs' || true
|
rg -n --with-filename 'use crate::storage::\*;' rustfs/src/app --glob '*_usecase.rs' --glob 'object/*.rs' || true
|
||||||
) >"$RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE"
|
) >"$RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE"
|
||||||
|
|
||||||
if [[ -s "$RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE" ]]; then
|
if [[ -s "$RUSTFS_APP_USECASE_STORAGE_WILDCARD_HITS_FILE" ]]; then
|
||||||
@@ -2238,7 +2238,7 @@ fi
|
|||||||
|
|
||||||
(
|
(
|
||||||
cd "$ROOT_DIR"
|
cd "$ROOT_DIR"
|
||||||
rg -n --with-filename 'crate::storage::s3_api::|use crate::storage::s3_api|super::s3_api::|use super::s3_api' rustfs/src/app --glob '*_usecase.rs' || true
|
rg -n --with-filename 'crate::storage::s3_api::|use crate::storage::s3_api|super::s3_api::|use super::s3_api' rustfs/src/app --glob '*_usecase.rs' --glob 'object/*.rs' || true
|
||||||
) >"$RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE"
|
) >"$RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE"
|
||||||
|
|
||||||
if [[ -s "$RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE" ]]; then
|
if [[ -s "$RUSTFS_APP_USECASE_S3_API_BYPASS_HITS_FILE" ]]; then
|
||||||
@@ -2250,13 +2250,13 @@ fi
|
|||||||
{
|
{
|
||||||
rg -n --with-filename \
|
rg -n --with-filename \
|
||||||
'(use crate::storage::(access|helper|options|request_context|sse|timeout_wrapper|head_prefix|concurrency|ecfs)|crate::storage::sse::EncryptionKeyKind|use crate::storage::\{|use crate::storage::[A-Z])' \
|
'(use crate::storage::(access|helper|options|request_context|sse|timeout_wrapper|head_prefix|concurrency|ecfs)|crate::storage::sse::EncryptionKeyKind|use crate::storage::\{|use crate::storage::[A-Z])' \
|
||||||
rustfs/src/app/select_object.rs rustfs/src/app/*_usecase.rs || true
|
rustfs/src/app/select_object.rs rustfs/src/app/*_usecase.rs rustfs/src/app/object || true
|
||||||
rg -n --with-filename \
|
rg -n --with-filename \
|
||||||
'use super::(?:\{[^}]*\b(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|StorageError|DiskError|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b|(?:object_api_utils::to_s3s_etag|storageclass|StorageError|DiskError|DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b)' \
|
'use super::(?:\{[^}]*\b(?:DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|StorageError|DiskError|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b|(?:object_api_utils::to_s3s_etag|storageclass|StorageError|DiskError|DynReader|HashReader|WriteEncryption|WritePlan|DecryptReader|EncryptReader|HardLimitReader|boxed_reader|wrap_reader|compression_metadata_value|is_disk_compressible|MIN_DISK_COMPRESSIBLE_SIZE|get_lock_acquire_timeout|is_valid_storage_class|is_all_buckets_not_found|is_err_bucket_not_found|is_err_object_not_found|is_err_version_not_found)\b)' \
|
||||||
rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs || true
|
rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/object rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs || true
|
||||||
rg -n --with-filename \
|
rg -n --with-filename \
|
||||||
'use super::(?:\{[^}]*\b(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b|(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b)|super::(?:lifecycle|metadata_sys|object_lock|quota|replication|tagging|target|utils|versioning_sys|transition_api)::|super::super::(?:metadata_sys|lifecycle|target)::' \
|
'use super::(?:\{[^}]*\b(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b|(?:AppObjectLockConfigExt|AppReplicationConfigExt|AppVersioningConfigExt|predict_lifecycle_expiration|validate_restore_request|bucket_target_sys|lifecycle|metadata|metadata_sys|object_lock|policy_sys|quota|replication|tagging|target|utils|versioning_sys|transition_api|ObjectInfo|ObjectOptions)\b)|super::(?:lifecycle|metadata_sys|object_lock|quota|replication|tagging|target|utils|versioning_sys|transition_api)::|super::super::(?:metadata_sys|lifecycle|target)::' \
|
||||||
rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs rustfs/src/app/capacity_dirty_scope_test.rs rustfs/src/app/context.rs rustfs/src/app/context/handles.rs rustfs/src/app/context/interfaces.rs rustfs/src/app/context/runtime_sources.rs || true
|
rustfs/src/app/bucket_usecase.rs rustfs/src/app/object_usecase.rs rustfs/src/app/object rustfs/src/app/multipart_usecase.rs rustfs/src/app/lifecycle_transition_api_test.rs rustfs/src/app/capacity_dirty_scope_test.rs rustfs/src/app/context.rs rustfs/src/app/context/handles.rs rustfs/src/app/context/interfaces.rs rustfs/src/app/context/runtime_sources.rs || true
|
||||||
}
|
}
|
||||||
) >"$RUSTFS_APP_USECASE_STORAGE_API_BYPASS_HITS_FILE"
|
) >"$RUSTFS_APP_USECASE_STORAGE_API_BYPASS_HITS_FILE"
|
||||||
|
|
||||||
|
|||||||
@@ -1014,7 +1014,7 @@ if rg -n -U '(info|warn)!\(\s*target: "rustfs::heal::manager",[\s\S]{0,1000}"Hea
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if rg -n -U 'info!\([\s\S]{0,1000}"GetObject streaming body resumed from a reopened object read"' rustfs/src/app/object_usecase.rs >/dev/null; then
|
if rg -n -U 'info!\([\s\S]{0,1000}"GetObject streaming body resumed from a reopened object read"' rustfs/src/app/object >/dev/null; then
|
||||||
echo "❌ logging guardrail violation: successful per-object GetObject resume events must stay below INFO" >&2
|
echo "❌ logging guardrail violation: successful per-object GetObject resume events must stay below INFO" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
@@ -1056,7 +1056,7 @@ trace_hot_spans=(
|
|||||||
"crates/ecstore/src/core/sets.rs:list_objects_v2"
|
"crates/ecstore/src/core/sets.rs:list_objects_v2"
|
||||||
"crates/ecstore/src/set_disk/ops/list.rs:list_objects_v2"
|
"crates/ecstore/src/set_disk/ops/list.rs:list_objects_v2"
|
||||||
"rustfs/src/app/bucket_usecase.rs:execute_list_objects_v2"
|
"rustfs/src/app/bucket_usecase.rs:execute_list_objects_v2"
|
||||||
"rustfs/src/app/object_usecase.rs:execute_get_object"
|
"rustfs/src/app/object:execute_get_object"
|
||||||
)
|
)
|
||||||
|
|
||||||
for hot_span in "${trace_hot_spans[@]}"; do
|
for hot_span in "${trace_hot_spans[@]}"; do
|
||||||
|
|||||||
Reference in New Issue
Block a user