Compare commits

...

40 Commits

Author SHA1 Message Date
Hauser Zhi e296adef35 Merge branch 'main' into update_flake_lock_action 2026-11-22 14:44:52 +08:00
Hauser Zhi 496a34a6a7 chore(deps): drop hotpath shear ignore and refresh workspace deps (#9557)
* chore(shear): stop ignoring hotpath and clear shear findings

Remove hotpath from cargo-shear ignored so feature-only usage is reported,
drop unused deps found by shear --fix, delete empty access_owned stubs, and
suppress false-positive unlinked paths for #[path] test modules.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh workspace deps and bump faster-hex to 2.0

Run cargo update and cargo upgrade for compatible requirements, including
faster-hex 1.0.0 to 2.0.0. Existing hex_string/hex_decode call sites remain
compatible.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(zstd): import seekable directly for security tests

Use zstd_safe::seekable in the security test and document why the
seekable dev-dependency stays: it enables the feature and must remain
visible to cargo-shear as a real usage.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): avoid private-bounds on S3Router test helper

Move call_legacy to a free function so the pub(crate) OwnedOperation
bound is not attached to a public inherent impl.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): drop unused hotpath and demote rio md-5 to dev-deps

Remove hotpath from crates that only feature-forward it with no source
usage, keep rustfs-* hotpath feature links, make rio's hotpath optional
for its cfg-gated measure_block!, and move rio md-5 to dev-dependencies.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh lockfile and bump arrow/jiff requirements

Run cargo update, cargo upgrade, and cargo upgrade --incompatible.
Bump arrow/parquet to 60.2.0 and jiff to 1.0.2.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): run service_cohort tests on a larger stack

Linux libtest defaults to a 2 MiB test-thread stack; two-pool EC fixtures
overflow it and abort. Run each service_cohort case on a dedicated 32 MiB
thread, matching scoped_entry_fallback.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): upgrade russh-sftp to 4.0 and refresh lockfile

Run cargo update, cargo upgrade, and cargo upgrade --incompatible.
Bump russh-sftp 3 to 4 (opaque SftpHandle wire type), plus lapin,
hashbrown, and russh. Adapt the SFTP Handler and tests to SftpHandle.

Co-Authored-By: qunjiazhu <qunjiazhu@gmail.com>
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: qunjiazhu <qunjiazhu@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: RustFS <hello@rustfs.com>
2026-11-22 14:36:14 +08:00
overtrue 7c9ebadffe test(multipart): build the out-of-order complete envelope on the owned request
The out-of-order parts still go through the adapter's tag walk and still
fail as InvalidPartOrder.
2026-11-22 14:06:06 +08:00
overtrue e376e7d774 test(multipart): build the duplicate-part complete envelope on the owned request
The duplicate part numbers still go through the adapter's tag walk and still
must not fail as InvalidPartOrder.
2026-11-22 13:37:27 +08:00
overtrue d211df88d5 test(multipart): build the missing-parts-list complete envelope on the owned request
The empty CompleteMultipartUpload element still follows the adapter's Some branch
and checks InvalidRequest.
2026-11-22 13:08:17 +08:00
houseme af0a20527d chore(deps): update flake.lock
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/2074cc6' (2026-11-12)
  → 'github:NixOS/nixpkgs/93a9ed0' (2026-11-21)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/5b87ee0' (2026-11-14)
  → 'github:oxalica/rust-overlay/52db205' (2026-11-21)
2026-11-22 05:07:43 +00:00
overtrue 21caf8bf34 test(multipart): build the missing-parts complete envelope on the owned request
The fixture has no multipart upload, so it follows the adapter's None branch
and still checks InvalidPart.
2026-11-22 12:32:41 +08:00
overtrue d3eaaa21a6 test(multipart): build the range copy envelope on the owned request
The range path still maps the copy DTO the same way as the adapter and checks
EntityTooLarge, permit release, and the copied bytes.
2026-11-22 11:57:28 +08:00
overtrue ad457a2d1c test(multipart): build the cancelled copy envelope on the owned request
The cancellation path still maps the copy DTO the same way as the adapter
and still panics if the copy finishes before cancellation.
2026-11-22 11:25:09 +08:00
overtrue fa3ea5b4e5 test(multipart): build the deleted-source copy envelope on the owned request
The deleted-source rejection still maps the copy DTO the same way as the adapter
and checks NoSuchKey or MethodNotAllowed on the owned error.
2026-11-22 10:50:06 +08:00
overtrue f3396952b6 test(multipart): build the saturated copy envelope on the owned request
The occupied UploadPart pool still maps the copy DTO the same way as the adapter
and checks SlowDown on the owned error.
2026-11-22 10:23:10 +08:00
overtrue 8736b49d08 test(multipart): build the uninitialized copy envelope on the owned request
The uninitialized-store rejection still maps the copy DTO the same way as the adapter
and checks InternalError on the owned error.
2026-11-22 09:47:55 +08:00
overtrue eec468e488 test(multipart): build the admission upload envelope on the owned request. 2026-11-22 09:10:40 +08:00
overtrue f2af45ee68 test(multipart): build the invalid-storage-class create envelope on the owned request
The storage-class rejection still converts the create DTO separately
and checks InvalidStorageClass on the legacy error.
2026-11-22 08:33:58 +08:00
overtrue 25b55febc5 test(multipart): build the part-copy part-number envelope on the owned request
The part-number rejection still maps the copy DTO the same way as the adapter
and checks InvalidArgument with the same message.
2026-11-22 08:05:27 +08:00
overtrue c2e50b3b10 test(multipart): build the part-number upload envelope on the owned request
The part-number rejection still converts the upload DTO separately
and checks InvalidArgument with the same message.
2026-11-22 07:36:17 +08:00
overtrue 6fde592da1 test(multipart): build the oversize upload envelope on the owned request
The declared-size rejection still converts the upload DTO separately
and checks EntityTooLarge without polling the body.
2026-11-22 07:03:43 +08:00
overtrue 98259d7c46 test(multipart): build the missing-body upload envelope on the owned request
The missing-body rejection still converts the upload DTO separately
and checks IncompleteBody on the legacy error.
2026-11-22 06:35:22 +08:00
overtrue b5a35fc2c7 test(multipart): build the baseline create envelope on the owned request
The direct fixture still converts the create DTO separately and checks
the legacy error code and message.
2026-11-22 06:06:39 +08:00
overtrue 11a7572bba test(multipart): build the reserved-key complete envelope on the owned request
The reserved-key complete rejection still converts the DTO through the seam
and checks InvalidArgument on the owned error.
2026-11-22 05:37:37 +08:00
overtrue 4c4dc84671 test(multipart): build the reserved-key create envelope on the owned request
The reserved-key rejection still converts the create DTO separately
and checks InvalidArgument on the legacy error.
2026-11-22 05:06:23 +08:00
overtrue 52986eefd5 test(bucket): build the list-v1 envelope on the owned request
The XML-safe marker listing still converts ListObjectsInput separately
and keeps the same GET envelope with an empty head.
2026-11-22 04:25:57 +08:00
overtrue 7a5ffcfee7 test(multipart): build the complete listing envelopes on the owned request
The version and V1 listing checks build method-only envelopes with owned_envelope.
The DTOs still carry the bucket and prefix, and the V2 seam stays.
2026-11-22 03:55:16 +08:00
overtrue 334d615a67 test(bucket): build the encryption lookup envelopes on the owned request
The uninitialized encryption test builds its method-only envelopes with owned_envelope.
The InternalError and MalformedXML assertions stay.
2026-11-22 03:26:14 +08:00
overtrue 0c813e1a9e test(bucket): build the core uninitialized envelopes on the owned request
The no-req_info envelope is built with owned_envelope.
The arm that carries req_info still goes through from_s3s.
2026-11-22 02:50:05 +08:00
overtrue 0de0b85374 test(lifecycle): build the object-lock envelope on the owned request
The object-lock replication test builds its method-only envelope with from_parts.
The existing ignore attribute and the replication-count assertions stay.
2026-11-22 02:23:15 +08:00
overtrue 78f78ce5ee test(multipart): build the complete setup envelopes on the owned request
The versioning and encryption setup calls build method-only envelopes with owned_envelope.
The requests that carry a body still use build_request.
2026-11-22 01:54:13 +08:00
overtrue 82ad3f0007 test(object): build the attributes lookup envelope on the owned request
The uninitialized-store attributes test builds its method-only envelope with from_parts.
The InternalError assertion is unchanged.
2026-11-22 01:25:00 +08:00
overtrue 82f094f4d5 test(lifecycle): build the suspend-versioning envelope on the owned request
The suspended-bucket helper builds its method-only envelope with from_parts.
The restore call that carries a body still uses build_request.
2026-11-22 00:50:23 +08:00
overtrue e1df1d569b test(multipart): build the upload-part claim envelopes on the owned request
The two method-only upload-part claim tests build their envelopes with owned_envelope.
Every refusal assertion stays.
2026-11-22 00:21:35 +08:00
overtrue 19487a6e34 test(multipart): build the list-parts envelope on the owned request
The list-parts owner lookup builds its method-only envelope with owned_envelope.
The later requests that set a URI still go through from_s3s.
2026-11-21 23:50:20 +08:00
overtrue 6177fc280f test(bucket): build the website lookup envelopes on the owned request 2026-11-21 23:19:40 +08:00
overtrue 14db51bba7 test(bucket): build the logging lookup envelope on the owned request
The missing-bucket logging lookup builds its envelope with owned_envelope.
The NoSuchBucket assertion stays.
2026-11-21 22:48:38 +08:00
overtrue 353de54fcc test(bucket): build the missing-lifecycle envelope on the owned request 2026-11-21 22:14:58 +08:00
overtrue 6a4f93bbaf test(bucket): build the tagging envelope on the owned request 2026-11-21 21:40:20 +08:00
overtrue 74fab85ba0 test(bucket): build the list-buckets envelope on the owned request 2026-11-21 21:00:03 +08:00
overtrue 44db2be5f9 test(multipart): build the abort envelope on the owned request 2026-11-21 20:25:10 +08:00
overtrue 3e41233589 test(multipart): build list envelopes on the owned request 2026-11-21 19:50:10 +08:00
overtrue f36a771389 test(bucket): build policy envelopes on the owned request 2026-11-21 19:04:31 +08:00
overtrue 0447e68073 test(bucket): build method-only envelopes on the owned request 2026-11-21 18:31:08 +08:00
49 changed files with 1058 additions and 557 deletions
Generated
+184 -187
View File
File diff suppressed because it is too large Load Diff
+22 -19
View File
@@ -175,9 +175,9 @@ futures = "0.3.34"
futures-core = "0.3.34"
futures-lite = "2.6.1"
futures-util = "0.3.34"
pollster = "1.0.2"
pollster = "1.0.3"
pulsar = { default-features = false, version = "=6.9.0" }
lapin = { default-features = false, version = "4.12.3" }
lapin = { default-features = false, version = "4.13.0" }
hyper = { version = "1.12.0" }
h2 = { version = "0.4.20" }
hyper-rustls = { default-features = false, version = "0.27.10" }
@@ -216,7 +216,7 @@ csv-core = "0.1.13"
bytesize = "2.7.0"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
form_urlencoded = "1.2.2"
form_urlencoded = "1.2.3"
prost = "0.14.6"
quick-xml = "0.42.0"
rmp = { version = "0.8.15" }
@@ -258,7 +258,7 @@ proptest = "1"
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "1.0.1" }
jiff = { version = "1.0.2" }
time = { version = "0.3.55" }
# Database
@@ -278,15 +278,15 @@ atomic_enum = "0.3.0"
aws-config = { version = "1.13.3" }
aws-credential-types = { version = "1.3.1" }
aws-sdk-kms = { default-features = false, version = "1.133.0" }
aws-sdk-s3 = { default-features = false, version = "1.160.0" }
aws-sdk-sts = { default-features = false, version = "1.127.0" }
aws-sdk-s3 = { default-features = false, version = "1.161.0" }
aws-sdk-sts = { default-features = false, version = "1.128.0" }
aws-smithy-async = { version = "1.3.1" }
aws-smithy-http-client = { default-features = false, version = "1.5.3" }
aws-smithy-runtime-api = { version = "1.22.0" }
aws-smithy-types = { version = "1.9.4" }
base64-simd = "0.8.0"
brotli = "9.1.3"
clap = { version = "4.6.8" }
clap = { version = "4.6.9" }
const-str = { version = "1.1.0" }
convert_case = "0.12.0"
criterion = { version = "0.8" }
@@ -298,12 +298,12 @@ datafusion = { default-features = false, version = "55.3.0" }
derive_builder = "0.20.2"
ed25519-dalek = "3.0.0"
enumset = "1.1.15"
faster-hex = "1.0.0"
faster-hex = "2.0.0"
flate2 = "1.1.11"
glob = "0.3.4"
google-cloud-storage = "1.22.0"
google-cloud-auth = "1.19.0"
hashbrown = { version = "0.17.1" }
hashbrown = { version = "0.17.2" }
# Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
# every authenticator app expects). Already in the graph transitively.
data-encoding = "2.11.1"
@@ -337,7 +337,7 @@ percent-encoding = "2.3.2"
# CLI needs its own QR encoder. No default features: the image/render backends
# pull in an image stack this only needs SVG and text output from.
qrcode-rs = { version = "2.2.0", default-features = false, features = ["std", "svg"] }
pin-project-lite = "0.2.17"
pin-project-lite = "0.2.18"
pretty_assertions = "1.4.1"
rand = { version = "0.10.4" }
ratelimit = "2.0.0"
@@ -360,7 +360,7 @@ legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
smallvec = { version = "1.16.5" }
smallvec = { version = "1.16.6" }
compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.4.1" }
@@ -378,23 +378,23 @@ tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.34.0" }
tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
url = "2.5.9"
urlencoding = "2.1.3"
uuid = { version = "1.29.3" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
walkdir = "2.5.1"
winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.21" }
zip = "9.7.0"
zip = "9.8.1"
zstd = "0.14.4"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.8", default-features = false }
dial9 = { version = "0.5.9", default-features = false }
opentelemetry = { version = "0.33.1" }
opentelemetry-appender-tracing = { version = "0.33.1" }
opentelemetry-otlp = { version = "0.33.1" }
@@ -405,12 +405,12 @@ opentelemetry-stdout = { version = "0.33.1" }
pyroscope = { version = "2.1.5" }
# FTP and SFTP
libunftp = { version = "0.25.0" }
libunftp = { version = "0.25.1" }
unftp-core = "0.1.1"
suppaftp = { version = "13.0.0" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.65.2" }
russh-sftp = "3.0.1"
russh = { version = "0.65.4" }
russh-sftp = "4.0.0"
# WebDAV
dav-server = "0.12.0"
@@ -423,7 +423,10 @@ hotpath = { version = "0.32.1", default-features = false }
insta = { version = "1.49" }
[workspace.metadata.cargo-shear]
ignored = ["hotpath", "rustfs"]
# The `#[path]` test modules under `rustfs/src/server/http/tests/` are linked by
# rustc (including the `http.rs` + `http/` sibling-directory rule), but cargo-shear
# resolves those nested `#[path]` attributes against the wrong base directory.
ignored-paths = ["rustfs/src/server/http/tests/**/*.rs"]
[profile.dev]
# Full debuginfo roughly doubles compile+link time and produces multi-GB
+3 -4
View File
@@ -15,12 +15,11 @@ workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-core/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-core/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-core/hotpath-cpu"]
hotpath = ["rustfs-io-core/hotpath"]
hotpath-alloc = ["hotpath", "rustfs-io-core/hotpath-alloc"]
hotpath-cpu = ["hotpath", "rustfs-io-core/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
# Internal crates
rustfs-io-core = { workspace = true }
serde = { workspace = true, features = ["derive"] }
-1
View File
@@ -183,7 +183,6 @@ rmp.workspace = true
rmp-serde.workspace = true
tokio-util = { workspace = true, features = ["io", "compat", "rt"] }
tokio-stream = { workspace = true, features = ["sync"] }
hmac = { workspace = true }
sha2 = { workspace = true }
hex-simd = { workspace = true }
tempfile.workspace = true
+3 -4
View File
@@ -29,12 +29,11 @@ workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-metrics/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-metrics/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-metrics/hotpath-cpu"]
hotpath = ["rustfs-io-metrics/hotpath"]
hotpath-alloc = ["hotpath", "rustfs-io-metrics/hotpath-alloc"]
hotpath-cpu = ["hotpath", "rustfs-io-metrics/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["io-util", "fs", "sync", "rt-multi-thread"] }
+13 -7
View File
@@ -410,7 +410,10 @@ impl<S: StorageBackend + Send + Sync + 'static> SftpDriver<S> {
}
};
let handle = self.allocate_handle(HandleState::Dir(cursor))?;
Ok(Handle { id, handle })
Ok(Handle {
id,
handle: handle.into(),
})
}
/// READDIR body shared with the Handler trait wrapper. Removes the
@@ -666,8 +669,11 @@ mod tests {
assert_eq!(auth.continuation_token, call.continuation_token);
}
assert_eq!(authorized[2].continuation_token.as_ref().map(|value| value.as_str()), Some("next-page"));
let denied =
with_test_auth_override(|_, _, prefix| prefix != Some("private/"), driver.readdir_inner(2, handle.handle)).await;
let denied = with_test_auth_override(
|_, _, prefix| prefix != Some("private/"),
driver.readdir_inner(2, String::from_utf8(handle.handle.into_bytes()).expect("utf8 handle")),
)
.await;
assert!(denied.is_err(), "policy change must deny the next READDIR");
assert_eq!(backend.list_objects_calls().len(), 2, "denied READDIR cannot reach storage");
}
@@ -716,7 +722,7 @@ mod tests {
};
let handle_id = driver.allocate_handle(HandleState::Dir(cursor)).expect("allocate");
let readdir_fut = driver.readdir(1, handle_id.clone());
let readdir_fut = driver.readdir(1, handle_id.clone().into());
with_test_auth_override(|_, _, _| true, async {
tokio::select! {
@@ -758,7 +764,7 @@ mod tests {
// token, no skipped entries).
backend.clear_stall_list_objects_v2();
backend.queue_list_objects_v2_ok_empty();
let result = with_test_auth_override(|_, _, _| true, driver.readdir(2, handle_id)).await;
let result = with_test_auth_override(|_, _, _| true, driver.readdir(2, handle_id.into())).await;
// Empty page returns Eof per readdir's empty-Name-to-Eof translation.
let err = result.expect_err("re-issued READDIR against an empty listing must return Eof, not Ok");
assert!(
@@ -785,7 +791,7 @@ mod tests {
let handle_id = driver.allocate_handle(HandleState::Dir(cursor)).expect("allocate");
let (result, captured) =
capture_tracing_at(Level::ERROR, with_test_auth_override(|_, _, _| true, driver.readdir(7, handle_id))).await;
capture_tracing_at(Level::ERROR, with_test_auth_override(|_, _, _| true, driver.readdir(7, handle_id.into()))).await;
let err = result.expect_err("exhausted cursor must return Eof");
assert!(matches!(StatusCode::from(err), StatusCode::Eof));
assert!(
@@ -818,7 +824,7 @@ mod tests {
let handle_id = driver.allocate_handle(HandleState::Dir(cursor)).expect("allocate");
let (result, captured) =
capture_tracing_at(Level::ERROR, with_test_auth_override(|_, _, _| true, driver.readdir(8, handle_id))).await;
capture_tracing_at(Level::ERROR, with_test_auth_override(|_, _, _| true, driver.readdir(8, handle_id.into()))).await;
let err = result.expect_err("backend error must propagate as Err");
assert!(!matches!(StatusCode::from(err), StatusCode::Eof), "backend error must not be Eof");
assert!(
+30 -17
View File
@@ -33,7 +33,9 @@ use super::write::{build_write_tombstone, fstat_reported_size, rejects_excl_or_t
use crate::common::client::s3::StorageBackend;
use crate::common::gateway::{AuthorizationError, S3Action};
use crate::common::session::SessionContext;
use russh_sftp::protocol::{Attrs, Data, File, FileAttributes, Handle, Name, OpenFlags, Packet, Status, StatusCode, Version};
use russh_sftp::protocol::{
Attrs, Data, File, FileAttributes, Handle, Name, OpenFlags, Packet, SftpHandle, Status, StatusCode, Version,
};
use rustfs_credentials::Credentials;
use rustfs_gateway_types::dto::{AbortMultipartUploadInput, CopyObjectInput, ListObjectsV2Input};
use rustfs_gateway_types::{BucketName, ObjectKey, UploadIdClaim};
@@ -208,6 +210,12 @@ impl<S: StorageBackend + Send + Sync + 'static> SftpDriver<S> {
}
}
/// russh-sftp 4 hands the Handler opaque `SftpHandle` bytes; the table
/// keys remain the UTF-8 UUID strings this driver mints.
fn handle_table_key(handle: SftpHandle) -> Result<String, SftpError> {
String::from_utf8(handle.into_bytes()).map_err(|_| SftpError::code(StatusCode::Failure))
}
/// Generate a fresh UUID v4 handle, insert the given state into the
/// per-session handle table, and return the handle string. Enforces
/// self.handles_per_session before any UUID generation. Cap-exceeded
@@ -494,8 +502,9 @@ impl<S: StorageBackend + Send + Sync + 'static> russh_sftp::server::Handler for
/// Streaming returns (next_part_number - 1) * part_size + buffer
/// length. Failed returns the size recorded at the most recent
/// successful write. An unknown handle returns Failure.
#[tracing::instrument(level = "debug", skip(self), fields(id, handle = %handle), err(Debug))]
async fn fstat(&mut self, id: u32, handle: String) -> Result<Attrs, Self::Error> {
#[tracing::instrument(level = "debug", skip(self), fields(id, handle = ?handle), err(Debug))]
async fn fstat(&mut self, id: u32, handle: SftpHandle) -> Result<Attrs, Self::Error> {
let handle = Self::handle_table_key(handle)?;
self.session_diag.stamp();
let part_size = self.part_size;
let result = self.with_handle_ref(&handle, |state| match state {
@@ -541,8 +550,9 @@ impl<S: StorageBackend + Send + Sync + 'static> russh_sftp::server::Handler for
/// exhaustion, so it is normal control flow on this handler. The
/// instrument attribute therefore omits err(Debug) and non-Eof
/// failures are surfaced via the explicit error log below.
#[tracing::instrument(level = "debug", skip(self), fields(id, handle = %handle))]
async fn readdir(&mut self, id: u32, handle: String) -> Result<Name, Self::Error> {
#[tracing::instrument(level = "debug", skip(self), fields(id, handle = ?handle))]
async fn readdir(&mut self, id: u32, handle: SftpHandle) -> Result<Name, Self::Error> {
let handle = Self::handle_table_key(handle)?;
self.session_diag.stamp();
let result = self.readdir_inner(id, handle.clone()).await;
self.session_diag.stamp();
@@ -617,8 +627,9 @@ impl<S: StorageBackend + Send + Sync + 'static> russh_sftp::server::Handler for
/// read-past-end-of-file, so it is normal control flow on this
/// handler. The instrument attribute therefore omits err(Debug) and
/// non-Eof failures are surfaced via the explicit error log below.
#[tracing::instrument(level = "debug", skip(self), fields(id, handle = %handle, offset, len))]
async fn read(&mut self, id: u32, handle: String, offset: u64, len: u32) -> Result<Data, Self::Error> {
#[tracing::instrument(level = "debug", skip(self), fields(id, handle = ?handle, offset, len))]
async fn read(&mut self, id: u32, handle: SftpHandle, offset: u64, len: u32) -> Result<Data, Self::Error> {
let handle = Self::handle_table_key(handle)?;
self.session_diag.stamp();
let result = self.read_inner(id, handle.clone(), offset, len).await;
self.session_diag.stamp();
@@ -650,8 +661,9 @@ impl<S: StorageBackend + Send + Sync + 'static> russh_sftp::server::Handler for
///
/// A missing handle is treated as Ok to tolerate clients that
/// double-close on session teardown.
#[tracing::instrument(level = "info", skip(self), fields(id, handle = %handle), err(Debug))]
async fn close(&mut self, id: u32, handle: String) -> Result<Status, Self::Error> {
#[tracing::instrument(level = "info", skip(self), fields(id, handle = ?handle), err(Debug))]
async fn close(&mut self, id: u32, handle: SftpHandle) -> Result<Status, Self::Error> {
let handle = Self::handle_table_key(handle)?;
self.session_diag.stamp();
let removed = self.handles.remove(&handle);
let Some(HandleState::Write {
@@ -755,8 +767,9 @@ impl<S: StorageBackend + Send + Sync + 'static> russh_sftp::server::Handler for
/// to Streaming by issuing CreateMultipartUpload. A Failed handle
/// rejects every subsequent write with the same status that caused
/// the failure.
#[tracing::instrument(level = "debug", skip(self, data), fields(id, handle = %handle, offset, len = data.len()), err(Debug))]
async fn write(&mut self, id: u32, handle: String, offset: u64, data: Vec<u8>) -> Result<Status, Self::Error> {
#[tracing::instrument(level = "debug", skip(self, data), fields(id, handle = ?handle, offset, len = data.len()), err(Debug))]
async fn write(&mut self, id: u32, handle: SftpHandle, offset: u64, data: Vec<u8>) -> Result<Status, Self::Error> {
let handle = Self::handle_table_key(handle)?;
self.session_diag.stamp();
self.enforce_server_readonly()?;
@@ -983,8 +996,8 @@ impl<S: StorageBackend + Send + Sync + 'static> russh_sftp::server::Handler for
/// SSH_FXP_FSETSTAT, SFTP Internet Draft section 6.6. Same rationale
/// as setstat: S3 cannot honour POSIX attributes, and clients use
/// FSETSTAT during transfers to stamp the in-flight handle.
#[tracing::instrument(level = "debug", skip(self, _attrs), fields(id, handle = %_handle), err(Debug))]
async fn fsetstat(&mut self, id: u32, _handle: String, _attrs: FileAttributes) -> Result<Status, Self::Error> {
#[tracing::instrument(level = "debug", skip(self, _attrs), fields(id, handle = ?_handle), err(Debug))]
async fn fsetstat(&mut self, id: u32, _handle: SftpHandle, _attrs: FileAttributes) -> Result<Status, Self::Error> {
self.enforce_server_readonly()?;
Ok(ok_status(id))
}
@@ -1254,7 +1267,7 @@ mod tests {
let handle_id = driver
.allocate_handle(file_handle("bucket", "k", 1234, attrs.clone()))
.expect("allocate");
let out = driver.fstat(4, handle_id).await.expect("fstat on File must succeed");
let out = driver.fstat(4, handle_id.into()).await.expect("fstat on File must succeed");
assert_eq!(out.attrs.size, Some(1234));
assert_eq!(out.attrs.mtime, Some(1_700_000_000));
}
@@ -1267,7 +1280,7 @@ mod tests {
part_buffer: vec![0u8; 4096],
};
let handle_id = driver.allocate_handle(write_handle("bucket", "k", phase)).expect("allocate");
let out = driver.fstat(5, handle_id).await.expect("fstat on Write must succeed");
let out = driver.fstat(5, handle_id.into()).await.expect("fstat on Write must succeed");
assert_eq!(
out.attrs.size,
Some(4096),
@@ -1283,7 +1296,7 @@ mod tests {
.allocate_handle(file_handle("bucket", "k", 0, FileAttributes::default()))
.expect("allocate");
let status = driver
.fsetstat(6, handle_id, FileAttributes::default())
.fsetstat(6, handle_id.into(), FileAttributes::default())
.await
.expect("fsetstat must succeed on any attrs");
assert!(matches!(status.status_code, StatusCode::Ok));
@@ -1417,7 +1430,7 @@ mod tests {
let handle_id = driver
.allocate_handle(file_handle("bucket", "k", 0, FileAttributes::default()))
.expect("allocate");
let result = driver.fsetstat(10, handle_id, FileAttributes::default()).await;
let result = driver.fsetstat(10, handle_id.into(), FileAttributes::default()).await;
match result {
Err(err) => assert!(matches!(err.0, StatusCode::PermissionDenied)),
Ok(_) => panic!("fsetstat must error in read-only mode"),
+26 -19
View File
@@ -69,7 +69,10 @@ impl<S: StorageBackend + Send + Sync + 'static> SftpDriver<S> {
attrs,
read_cache,
})?;
Ok(Handle { id, handle })
Ok(Handle {
id,
handle: handle.into(),
})
}
/// Body of the SSH_FXP_READ handler. Returns up to len bytes starting
@@ -256,7 +259,7 @@ mod tests {
.allocate_handle(file_handle("bucket", "secret.txt", 4096, FileAttributes::default()))
.expect("allocate read handle");
driver.try_populate_read_cache(&handle, 0, vec![42; 4096]);
let error = with_test_auth_override(|_, _, _| true, driver.read(1, handle, 0, 1024))
let error = with_test_auth_override(|_, _, _| true, driver.read(1, handle.into(), 0, 1024))
.await
.expect_err("backend policy denial must prevent cached content disclosure");
assert!(matches!(StatusCode::from(error), StatusCode::PermissionDenied));
@@ -270,7 +273,7 @@ mod tests {
.allocate_handle(file_handle("bucket", "k", 100, FileAttributes::default()))
.expect("allocate");
let err = driver
.read(1, handle_id, 0, 0)
.read(1, handle_id.into(), 0, 0)
.await
.expect_err("len=0 must return BadMessage");
assert!(matches!(StatusCode::from(err), StatusCode::BadMessage));
@@ -284,7 +287,7 @@ mod tests {
.allocate_handle(file_handle("bucket", "k", 10, FileAttributes::default()))
.expect("allocate");
let err = driver
.read(2, handle_id, 10, 4)
.read(2, handle_id.into(), 10, 4)
.await
.expect_err("offset==size must return Eof");
assert!(matches!(StatusCode::from(err), StatusCode::Eof));
@@ -299,7 +302,7 @@ mod tests {
.allocate_handle(file_handle("bucket", "k", 5, FileAttributes::default()))
.expect("allocate");
let data = with_test_auth_override(|_, _, _| true, driver.read(3, handle_id, 0, 1024))
let data = with_test_auth_override(|_, _, _| true, driver.read(3, handle_id.into(), 0, 1024))
.await
.expect("read must succeed");
assert_eq!(data.data, b"hello".to_vec());
@@ -317,7 +320,7 @@ mod tests {
.allocate_handle(file_handle("bucket", "k", 10, FileAttributes::default()))
.expect("allocate");
let (result, captured) = capture_tracing_at(Level::ERROR, async { driver.read(11, handle_id, 10, 4).await }).await;
let (result, captured) = capture_tracing_at(Level::ERROR, async { driver.read(11, handle_id.into(), 10, 4).await }).await;
let err = result.expect_err("offset==size must return Eof");
assert!(matches!(StatusCode::from(err), StatusCode::Eof));
assert!(
@@ -342,8 +345,11 @@ mod tests {
.allocate_handle(file_handle("bucket", "k", 1024, FileAttributes::default()))
.expect("allocate");
let (result, captured) =
capture_tracing_at(Level::ERROR, with_test_auth_override(|_, _, _| true, driver.read(12, handle_id, 0, 256))).await;
let (result, captured) = capture_tracing_at(
Level::ERROR,
with_test_auth_override(|_, _, _| true, driver.read(12, handle_id.into(), 0, 256)),
)
.await;
let err = result.expect_err("backend error must propagate as Err");
assert!(!matches!(StatusCode::from(err), StatusCode::Eof), "backend error must not be Eof");
assert!(
@@ -379,7 +385,7 @@ mod tests {
let start = Instant::now();
let outcome = tokio::time::timeout(
Duration::from_secs(10),
with_test_auth_override(|_, _, _| true, driver.read(14, handle_id, 0, 4096)),
with_test_auth_override(|_, _, _| true, driver.read(14, handle_id.into(), 0, 4096)),
)
.await;
let elapsed = start.elapsed();
@@ -425,7 +431,7 @@ mod tests {
let mut assembled: Vec<u8> = Vec::with_capacity(object_size as usize);
let mut reads: u32 = 0;
while offset < object_size {
let data = with_test_auth_override(|_, _, _| true, driver.read(20 + reads, handle_id.clone(), offset, chunk))
let data = with_test_auth_override(|_, _, _| true, driver.read(20 + reads, handle_id.clone().into(), offset, chunk))
.await
.expect("read inside the cached window must succeed without a second backend call");
assert!(!data.data.is_empty(), "non-empty hit");
@@ -461,21 +467,21 @@ mod tests {
.expect("allocate");
// First read fetches from the backend and populates window 1.
let r1 = with_test_auth_override(|_, _, _| true, driver.read(30, handle_id.clone(), 0, 1024))
let r1 = with_test_auth_override(|_, _, _| true, driver.read(30, handle_id.clone().into(), 0, 1024))
.await
.expect("first read must succeed");
assert!(r1.data.iter().all(|b| *b == 0xAA), "first read must come from window 1");
// Second read inside the cached chunk is served from the
// buffer. No second backend call yet.
let r2 = with_test_auth_override(|_, _, _| true, driver.read(31, handle_id.clone(), 1024, 1024))
let r2 = with_test_auth_override(|_, _, _| true, driver.read(31, handle_id.clone().into(), 1024, 1024))
.await
.expect("mid-window read must succeed from cache");
assert!(r2.data.iter().all(|b| *b == 0xAA), "mid-window read still in window 1");
// Reading at offset == window falls outside the cached chunk
// and triggers the second backend call.
let r3 = with_test_auth_override(|_, _, _| true, driver.read(32, handle_id.clone(), window, 1024))
let r3 = with_test_auth_override(|_, _, _| true, driver.read(32, handle_id.clone().into(), window, 1024))
.await
.expect("read at offset=window must succeed via second backend call");
assert!(r3.data.iter().all(|b| *b == 0xBB), "read at window boundary must come from window 2");
@@ -483,7 +489,7 @@ mod tests {
// A read inside the second cached chunk is served from the
// buffer. The queue is empty by now, so any third backend
// call would fail.
let r4 = with_test_auth_override(|_, _, _| true, driver.read(33, handle_id, window + 1024, 1024))
let r4 = with_test_auth_override(|_, _, _| true, driver.read(33, handle_id.into(), window + 1024, 1024))
.await
.expect("mid-window-2 read must succeed from cache");
assert!(r4.data.iter().all(|b| *b == 0xBB), "mid-window-2 read still in window 2");
@@ -511,13 +517,13 @@ mod tests {
.expect("allocate");
// Populate window 1 with a full read.
let _ = with_test_auth_override(|_, _, _| true, driver.read(40, handle_id.clone(), 0, window as u32))
let _ = with_test_auth_override(|_, _, _| true, driver.read(40, handle_id.clone().into(), 0, window as u32))
.await
.expect("populate window 1");
// Ask for 256 bytes starting 64 bytes before window end. Only
// 64 bytes are in the window. The driver must return 64.
let edge = with_test_auth_override(|_, _, _| true, driver.read(41, handle_id, window - 64, 256))
let edge = with_test_auth_override(|_, _, _| true, driver.read(41, handle_id.into(), window - 64, 256))
.await
.expect("partial-hit read must succeed");
assert_eq!(edge.data.len(), 64, "partial hit must return only the in-window portion");
@@ -552,9 +558,10 @@ mod tests {
for i in 0..read_count {
let offset = (chunk_size as u64) * (i as u64);
let data = with_test_auth_override(|_, _, _| true, driver.read(50 + i, handle_id.clone(), offset, chunk_size as u32))
.await
.expect("each read must succeed via the backend");
let data =
with_test_auth_override(|_, _, _| true, driver.read(50 + i, handle_id.clone().into(), offset, chunk_size as u32))
.await
.expect("each read must succeed via the backend");
assert_eq!(data.data.len(), chunk_size, "read must return full requested length");
let expected_byte = (i + 1) as u8;
assert!(
+7 -4
View File
@@ -336,7 +336,10 @@ impl<S: StorageBackend + Send + Sync + 'static> SftpDriver<S> {
open_attrs,
phase: WritePhase::Buffering { part_buffer: Vec::new() },
})?;
Ok(Handle { id, handle })
Ok(Handle {
id,
handle: handle.into(),
})
}
/// Upload the buffered bytes to S3 with a single PutObject call.
@@ -2105,7 +2108,7 @@ mod tests {
)
.await
.expect("WRITE | CREATE | TRUNCATE on a missing file must allocate a handle");
assert!(!handle.handle.is_empty());
assert!(!handle.handle.as_bytes().is_empty());
assert!(backend.head_object_calls().is_empty(), "the non-EXCL accept path must not HEAD");
}
@@ -2130,7 +2133,7 @@ mod tests {
)
.await
.expect("WRITE | CREATE | TRUNCATE on an existing file must allocate a handle");
assert!(!handle.handle.is_empty());
assert!(!handle.handle.as_bytes().is_empty());
assert!(backend.head_object_calls().is_empty(), "the non-EXCL accept path must not HEAD");
}
@@ -2177,7 +2180,7 @@ mod tests {
)
.await
.expect("EXCL on a missing key must allow creation");
assert!(!handle.handle.is_empty());
assert!(!handle.handle.as_bytes().is_empty());
assert_eq!(backend.head_object_calls().len(), 1, "EXCL must HEAD exactly once");
}
-5
View File
@@ -38,8 +38,6 @@ default = []
# event. Requires `RUSTFLAGS="--cfg hyper_unstable_tracing"`.
diagnostic-internode-http2-ping = ["dep:hyper", "rustfs-io-metrics/diagnostic-internode-http2-ping"]
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"rustfs-common/hotpath",
"rustfs-config/hotpath",
"rustfs-io-metrics/hotpath",
@@ -48,7 +46,6 @@ hotpath = [
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-io-metrics/hotpath-alloc",
@@ -57,7 +54,6 @@ hotpath-alloc = [
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-io-metrics/hotpath-cpu",
@@ -66,7 +62,6 @@ hotpath-cpu = [
]
[dependencies]
hotpath.workspace = true
rustfs-common.workspace = true
rustfs-heal-contracts.workspace = true
rustfs-io-metrics.workspace = true
-5
View File
@@ -31,29 +31,24 @@ workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"rustfs-rio/hotpath",
"rustfs-utils/hotpath",
"rustfs-filemeta/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
aes-gcm = { workspace = true, features = ["rand_core"] }
bytes = { workspace = true, features = ["serde"] }
chacha20poly1305.workspace = true
+4 -2
View File
@@ -32,6 +32,7 @@ workspace = true
internode-transport = []
default = []
hotpath = [
"dep:hotpath",
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
@@ -59,7 +60,8 @@ hotpath-cpu = [
]
[dependencies]
hotpath.workspace = true
# Feature-gated in src (measure_block!); optional so default builds do not warn.
hotpath = { workspace = true, optional = true }
arc-swap.workspace = true
tokio = { workspace = true, features = ["io-util", "macros", "net", "rt-multi-thread", "sync", "time"] }
rand = { workspace = true, features = ["serde"] }
@@ -79,7 +81,6 @@ rustfs-io-metrics.workspace = true
rustfs-tls-runtime.workspace = true
rustfs-utils = { workspace = true, features = ["io", "hash", "compress"] }
serde_json = { workspace = true, features = ["raw_value"] }
md-5 = { workspace = true }
hmac.workspace = true
minlz.workspace = true
tracing.workspace = true
@@ -91,6 +92,7 @@ xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] }
hex-simd.workspace = true
[dev-dependencies]
md-5 = { workspace = true }
faster-hex.workspace = true
rcgen.workspace = true
tokio-rustls.workspace = true
@@ -15,6 +15,30 @@
use super::*;
use crate::data_usage_define::{DATA_USAGE_OBJ_NAME_PATH, read_config_with_revision};
// Two-pool EC fixtures keep many scan futures live at once and overflow
// Linux libtest's default 2 MiB test-thread stack. Run each case on a
// dedicated 32 MiB thread (same approach as scoped_entry_fallback).
fn run_cohort_test<F, Fut>(thread_name: &'static str, test_fn: F)
where
F: FnOnce() -> Fut + Send + 'static,
Fut: std::future::Future<Output = ()> + 'static,
{
let handle = std::thread::Builder::new()
.name(thread_name.to_string())
.stack_size(32 * 1024 * 1024)
.spawn(move || {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("service cohort test runtime should build");
runtime.block_on(test_fn());
})
.expect("service cohort test thread should spawn");
if let Err(payload) = handle.join() {
std::panic::resume_unwind(payload);
}
}
async fn create_cohort_bucket(store: &ECStore, bucket: &str) {
create_cohort_bucket_objects(store, bucket, 1).await;
}
@@ -104,9 +128,16 @@ async fn run_cohort_cycle(
(result, usage)
}
#[tokio::test]
#[test]
#[serial]
async fn service_cohort_production_dispatch_services_waiters_across_sources() {
fn service_cohort_production_dispatch_services_waiters_across_sources() {
run_cohort_test(
"cohort-prod-disp",
service_cohort_production_dispatch_services_waiters_across_sources_case,
);
}
async fn service_cohort_production_dispatch_services_waiters_across_sources_case() {
let (_dir, store) = setup_two_pool_scanner_store().await;
clear_dirty_usage_buckets_for_tests();
let hot = format!("a-hot-{}", Uuid::new_v4().simple());
@@ -197,9 +228,16 @@ async fn service_cohort_production_dispatch_services_waiters_across_sources() {
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[test]
#[serial]
async fn service_cohort_flat_bucket_budget_does_not_publish_unscanned_small_bucket() {
fn service_cohort_flat_bucket_budget_does_not_publish_unscanned_small_bucket() {
run_cohort_test(
"cohort-flat-bud",
service_cohort_flat_bucket_budget_does_not_publish_unscanned_small_bucket_case,
);
}
async fn service_cohort_flat_bucket_budget_does_not_publish_unscanned_small_bucket_case() {
let (_dir, store) = setup_two_pool_scanner_store().await;
clear_dirty_usage_buckets_for_tests();
let flat = format!("a-flat-{}", Uuid::new_v4().simple());
@@ -283,9 +321,16 @@ async fn service_cohort_flat_bucket_budget_does_not_publish_unscanned_small_buck
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[test]
#[serial]
async fn service_cohort_fresh_complete_aggregate_preserves_reordered_sources() {
fn service_cohort_fresh_complete_aggregate_preserves_reordered_sources() {
run_cohort_test(
"cohort-fresh-agg",
service_cohort_fresh_complete_aggregate_preserves_reordered_sources_case,
);
}
async fn service_cohort_fresh_complete_aggregate_preserves_reordered_sources_case() {
let (_dir, store) = setup_two_pool_scanner_store().await;
clear_dirty_usage_buckets_for_tests();
for bucket in ["cohort-first", "cohort-second"] {
@@ -329,9 +374,16 @@ async fn service_cohort_fresh_complete_aggregate_preserves_reordered_sources() {
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[test]
#[serial]
async fn service_cohort_cancelled_dispatch_does_not_consume_waiters_or_leak_permits() {
fn service_cohort_cancelled_dispatch_does_not_consume_waiters_or_leak_permits() {
run_cohort_test(
"cohort-cancelled",
service_cohort_cancelled_dispatch_does_not_consume_waiters_or_leak_permits_case,
);
}
async fn service_cohort_cancelled_dispatch_does_not_consume_waiters_or_leak_permits_case() {
let (_dir, store) = setup_two_pool_scanner_store().await;
clear_dirty_usage_buckets_for_tests();
let bucket = format!("cancel-{}", Uuid::new_v4().simple());
@@ -360,9 +412,16 @@ async fn service_cohort_cancelled_dispatch_does_not_consume_waiters_or_leak_perm
/// N=17, q=4: every bucket reaches a worker within ceil(17/4)=5 rounds while
/// one bucket stays hot and another fails forever, with leader-local state
/// discarded after each round as a restart would.
#[tokio::test]
#[test]
#[serial]
async fn cold_bucket_service_bound_survives_restart_with_hot_and_failing_bucket() {
fn cold_bucket_service_bound_survives_restart_with_hot_and_failing_bucket() {
run_cohort_test(
"cold-svc-bound",
cold_bucket_service_bound_survives_restart_with_hot_and_failing_bucket_case,
);
}
async fn cold_bucket_service_bound_survives_restart_with_hot_and_failing_bucket_case() {
const BUCKETS: usize = 17;
const QUANTA: u64 = 4;
const ROUND_BOUND: usize = 5;
@@ -434,9 +493,16 @@ fn quantum_budget(quantum: crate::scanner_budget::ScannerBucketQuantum) -> Arc<S
)
}
#[tokio::test]
#[test]
#[serial]
async fn source_turn_dispatch_rotates_real_sources_and_retains_completed_roots() {
fn source_turn_dispatch_rotates_real_sources_and_retains_completed_roots() {
run_cohort_test(
"src-turn-disp",
source_turn_dispatch_rotates_real_sources_and_retains_completed_roots_case,
);
}
async fn source_turn_dispatch_rotates_real_sources_and_retains_completed_roots_case() {
let mut evidence = None;
temp_env::async_with_vars([(ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, Some("1"))], async {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -503,9 +569,16 @@ async fn source_turn_dispatch_rotates_real_sources_and_retains_completed_roots()
);
}
#[tokio::test]
#[test]
#[serial]
async fn source_turn_zero_new_frontier_stops_requeuing_without_starving_other_sources() {
fn source_turn_zero_new_frontier_stops_requeuing_without_starving_other_sources() {
run_cohort_test(
"src-turn-zero",
source_turn_zero_new_frontier_stops_requeuing_without_starving_other_sources_case,
);
}
async fn source_turn_zero_new_frontier_stops_requeuing_without_starving_other_sources_case() {
temp_env::async_with_vars([(ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, Some("1"))], async {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
let (_dir, store) = setup_two_pool_scanner_store().await;
@@ -533,9 +606,13 @@ async fn source_turn_zero_new_frontier_stops_requeuing_without_starving_other_so
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
}
#[tokio::test]
#[test]
#[serial]
async fn fair_prefix_quantum_spent_bucket_returns_behind_waiting_buckets() {
fn fair_prefix_quantum_spent_bucket_returns_behind_waiting_buckets() {
run_cohort_test("fair-spent-qm", fair_prefix_quantum_spent_bucket_returns_behind_waiting_buckets_case);
}
async fn fair_prefix_quantum_spent_bucket_returns_behind_waiting_buckets_case() {
let (_dir, store) = setup_local_scanner_store(1, 4).await;
clear_dirty_usage_buckets_for_tests();
create_cohort_bucket_objects(&store, "a-flat", 6).await;
@@ -565,9 +642,13 @@ async fn fair_prefix_quantum_spent_bucket_returns_behind_waiting_buckets() {
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[test]
#[serial]
async fn fair_prefix_quantum_without_durable_motion_ends_the_bucket_turn() {
fn fair_prefix_quantum_without_durable_motion_ends_the_bucket_turn() {
run_cohort_test("fair-no-motion", fair_prefix_quantum_without_durable_motion_ends_the_bucket_turn_case);
}
async fn fair_prefix_quantum_without_durable_motion_ends_the_bucket_turn_case() {
let (_dir, store) = setup_local_scanner_store(1, 4).await;
clear_dirty_usage_buckets_for_tests();
create_cohort_bucket_objects(&store, "a-flat", 6).await;
@@ -590,9 +671,16 @@ async fn fair_prefix_quantum_without_durable_motion_ends_the_bucket_turn() {
/// A one-object share stops inside its first child, so the frontier never
/// moves. Requeuing that bucket would spin until the cycle deadline.
#[tokio::test]
#[test]
#[serial]
async fn fair_prefix_quantum_repeated_share_without_a_new_frontier_stops_requeuing() {
fn fair_prefix_quantum_repeated_share_without_a_new_frontier_stops_requeuing() {
run_cohort_test(
"fair-rep-share",
fair_prefix_quantum_repeated_share_without_a_new_frontier_stops_requeuing_case,
);
}
async fn fair_prefix_quantum_repeated_share_without_a_new_frontier_stops_requeuing_case() {
let (_dir, store) = setup_local_scanner_store(1, 4).await;
clear_dirty_usage_buckets_for_tests();
create_cohort_bucket_objects(&store, "a-flat", 3).await;
@@ -613,9 +701,16 @@ async fn fair_prefix_quantum_repeated_share_without_a_new_frontier_stops_requeui
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[test]
#[serial]
async fn cold_bucket_service_saved_order_keeps_coverage_and_rejects_older_owners() {
fn cold_bucket_service_saved_order_keeps_coverage_and_rejects_older_owners() {
run_cohort_test(
"cold-saved-ord",
cold_bucket_service_saved_order_keeps_coverage_and_rejects_older_owners_case,
);
}
async fn cold_bucket_service_saved_order_keeps_coverage_and_rejects_older_owners_case() {
let (_dir, store) = setup_local_scanner_store(1, 4).await;
clear_dirty_usage_buckets_for_tests();
create_cohort_bucket(&store, "cold-only").await;
+3 -8
View File
@@ -30,12 +30,11 @@ doctest = false
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-rio/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-rio/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-rio/hotpath-cpu"]
hotpath = ["rustfs-rio/hotpath"]
hotpath-alloc = ["hotpath", "rustfs-rio/hotpath-alloc"]
hotpath-cpu = ["hotpath", "rustfs-rio/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
async-compression = { workspace = true, features = [
"tokio",
"bzip2",
@@ -49,10 +48,6 @@ rustfs-rio.workspace = true
tokio = { workspace = true, features = ["io-util", "macros", "rt"] }
thiserror = { workspace = true }
[target.'cfg(target_os = "macos")'.dependencies]
# Keep XZ decoding self-contained in macOS binaries instead of linking Homebrew liblzma.
liblzma = { version = "0.4.9", features = ["static"] }
[dev-dependencies]
rustfs-tokio-tar = { workspace = true }
futures = { workspace = true }
+7 -5
View File
@@ -29,14 +29,16 @@ zdict_builder = ["zstd_fixed/zdict_builder"]
zstdmt = ["zstd_fixed/zstdmt"]
[dev-dependencies]
# Enables zstd-safe's `seekable` feature for tests/security.rs. The test imports
# `zstd_safe::seekable` directly so cargo-shear treats this as a used dependency.
zstd-safe = { version = "8.3.0", features = ["seekable"] }
toml = "1.2.1"
toml = "1.2.2"
semver = "1.0.28"
apache-avro.workspace = true
arrow-array = "60.1.0"
arrow-ipc = { version = "60.1.0", features = ["zstd"] }
arrow-schema = "60.1.0"
parquet = { version = "60.1.0", default-features = false, features = ["arrow", "zstd"] }
arrow-array = "60.2.0"
arrow-ipc = { version = "60.2.0", features = ["zstd"] }
arrow-schema = "60.2.0"
parquet = { version = "60.2.0", default-features = false, features = ["arrow", "zstd"] }
kafka-protocol = { version = "0.18.0", default-features = false, features = ["zstd"] }
bytes.workspace = true
+1 -1
View File
@@ -86,7 +86,7 @@ fn corrupt_legacy_frame_cannot_emit_bytes_when_writer_is_dropped() {
#[test]
fn seekable_offsets_past_eof_keep_vec_length_within_capacity() {
use zstd::zstd_safe::seekable::{Seekable, SeekableCStream};
use zstd_safe::seekable::{Seekable, SeekableCStream};
let source = b"seekable frame fixture with enough data for several independent frames";
let mut encoded = [0; 1024];
Generated
+6 -6
View File
@@ -2,11 +2,11 @@
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1794495034,
"narHash": "sha256-E0hYoioqUtM2XmPPvYYpJgdADF5AW9oOoPSEiM9dTHQ=",
"lastModified": 1795249095,
"narHash": "sha256-8JDgyVvWUZg5m9hNlL8Ge9aT647ng3Ukc8NPMiKNTPQ=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "2074cc672349f14240fb7ea90d509e0b643e9dd4",
"rev": "93a9ed0a866099a230ed5146a5bb7d16fd5c9b14",
"type": "github"
},
"original": {
@@ -29,11 +29,11 @@
]
},
"locked": {
"lastModified": 1794646145,
"narHash": "sha256-rpqDyUM/d5fACYB8W+wKxxO57ICWjjalzb2X25oFFkY=",
"lastModified": 1795253373,
"narHash": "sha256-Vg6Sg8gJCDG9qpVEdOZX+72Z50MLwdnwtfZ8FNBwiYI=",
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "5b87ee0513c29127a01ad06e9cb362ea5ce91e78",
"rev": "52db20539ec0ba063479d66092992d9efcf456ff",
"type": "github"
},
"original": {
+23 -24
View File
@@ -1021,31 +1021,30 @@ impl<T> S3Router<T> {
}
}
/// Dispatches one legacy admin request. Callers keep their assertions and do not name s3s.
/// Free function (not an inherent impl) so the `OwnedOperation` bound stays `pub(crate)`.
#[cfg(test)]
impl<T: OwnedOperation> S3Router<T> {
/// Dispatches one legacy admin request. Callers keep their assertions and do not name s3s.
pub(crate) async fn call_legacy(
&self,
method: Method,
uri: &str,
extensions: http::Extensions,
) -> s3s::S3Result<S3Response<Body>> {
S3Route::call(
self,
S3Request {
input: Body::empty(),
method,
uri: uri.parse().expect("test URI"),
headers: HeaderMap::new(),
extensions,
credentials: None,
region: None,
service: None,
trailing_headers: None,
},
)
.await
}
pub(crate) async fn call_legacy<T: OwnedOperation>(
router: &S3Router<T>,
method: Method,
uri: &str,
extensions: http::Extensions,
) -> s3s::S3Result<S3Response<Body>> {
S3Route::call(
router,
S3Request {
input: Body::empty(),
method,
uri: uri.parse().expect("test URI"),
headers: HeaderMap::new(),
extensions,
credentials: None,
region: None,
service: None,
trailing_headers: None,
},
)
.await
}
impl<T> Default for S3Router<T> {
+3 -5
View File
@@ -141,7 +141,7 @@ mod tests {
object_store_from_envelope, publish_test_app_context,
};
use crate::admin::operation::{AdminRequest, AdminResponse, Operation};
use crate::admin::router::S3Router;
use crate::admin::router::{S3Router, call_legacy};
use crate::app::{RequestEnvelope, RequestHead};
use hyper::{Method, StatusCode};
use matchit::Params;
@@ -323,15 +323,13 @@ mod tests {
.insert(Method::GET, "/context-probe", ContextDependentAdminRoute)
.expect("register test route");
let err = router
.call_legacy(Method::GET, "/context-probe", http::Extensions::new())
let err = call_legacy(&router, Method::GET, "/context-probe", http::Extensions::new())
.await
.expect_err("empty slot must fail closed");
assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable.into());
assert!(slot.install(distinct_context(&ambient)));
let response = router
.call_legacy(Method::GET, "/context-probe", http::Extensions::new())
let response = call_legacy(&router, Method::GET, "/context-probe", http::Extensions::new())
.await
.expect("installed slot must serve request");
assert_eq!(response.status, Some(StatusCode::NO_CONTENT));
+12 -13
View File
@@ -1320,23 +1320,22 @@ mod tests {
};
for (index, expected_key) in ["a&local", "z-local"].into_iter().enumerate() {
let request_marker = input.marker.clone().unwrap_or_default();
let list_req = S3Request {
input: input.clone(),
method: http::Method::GET,
uri: http::Uri::from_static("/"),
headers: HeaderMap::new(),
extensions: http::Extensions::new(),
credentials: None,
region: None,
service: None,
trailing_headers: None,
};
let envelope = crate::app::RequestEnvelope::from_s3s(&list_req);
let gateway_input =
rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::list_objects::input_from_s3s(
list_req.input,
input.clone(),
)
.expect("convert ListObjectsInput");
let envelope = crate::app::RequestEnvelope::from_parts(
crate::app::RequestHead {
method: http::Method::GET,
uri: http::Uri::from_static("/"),
headers: HeaderMap::new(),
credentials: None,
region: None,
service: None,
},
http::Extensions::new(),
);
let gateway_output = tokio::time::timeout(
Duration::from_secs(10),
DefaultBucketUsecase::from_global().execute_list_objects(envelope, gateway_input),
+2 -3
View File
@@ -509,7 +509,7 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::{build_request, build_request_with_req_info};
use crate::app::bucket_usecase::test_support::{build_request_with_req_info, owned_envelope};
use crate::app::storage_api::s3::dto::CreateBucketInput;
use http::Method;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
@@ -1005,11 +1005,10 @@ mod tests {
#[tokio::test]
async fn execute_list_buckets_returns_internal_error_when_store_uninitialized() {
let req = build_request((), Method::GET);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_list_buckets(RequestEnvelope::from_s3s(&req), gateway_dto::ListBucketsInput::default())
.execute_list_buckets(owned_envelope(Method::GET), gateway_dto::ListBucketsInput::default())
.await
.unwrap_err();
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
+5 -6
View File
@@ -366,7 +366,7 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::{build_request, build_request_with_req_info, s3_op_total};
use crate::app::bucket_usecase::test_support::{build_request_with_req_info, owned_envelope, s3_op_total};
use http::Method;
#[test]
@@ -399,11 +399,10 @@ mod tests {
/// The envelope of a request whose access phase installed `req_info`, or none.
fn envelope(req_info: Option<ReqInfo>) -> RequestEnvelope {
let req = match req_info {
Some(req_info) => build_request_with_req_info((), Method::PUT, req_info),
None => build_request((), Method::PUT),
};
RequestEnvelope::from_s3s(&req)
match req_info {
Some(req_info) => RequestEnvelope::from_s3s(&build_request_with_req_info((), Method::PUT, req_info)),
None => owned_envelope(Method::PUT),
}
}
#[test]
+5 -6
View File
@@ -203,7 +203,7 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::owned_envelope;
use crate::app::storage_api::s3::dto::{self, PutBucketEncryptionInput, ServerSideEncryption};
use http::Method;
use rustfs_gateway_types::persistence::{
@@ -292,20 +292,19 @@ mod tests {
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let prepared = crate::storage::persisted_from_s3s(&req.input.server_side_encryption_configuration);
let prepared = crate::storage::persisted_from_s3s(&input.server_side_encryption_configuration);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_bucket_encryption(RequestEnvelope::from_s3s(&req), &req.input.bucket, prepared.map_err(Into::into))
.execute_put_bucket_encryption(owned_envelope(Method::PUT), &input.bucket, prepared.map_err(Into::into))
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
let err = usecase
.execute_put_bucket_encryption(
RequestEnvelope::from_s3s(&req),
&req.input.bucket,
owned_envelope(Method::PUT),
&input.bucket,
Ok(PersistedBucketEncryptionConfiguration { rules: vec![] }),
)
.await
+3 -4
View File
@@ -405,7 +405,7 @@ mod tests {
// waiter again on the single-thread test runtime.
assert_eq!(admission.available_permits(), 1);
}
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::{build_request, owned_envelope};
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::bucket_usecase::helper::MutationCompletionProbe;
@@ -775,13 +775,12 @@ mod tests {
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_bucket_lifecycle_configuration(
RequestEnvelope::from_s3s(&req),
&req.input.bucket,
owned_envelope(Method::PUT),
&input.bucket,
PreparedLifecycleConfiguration::Missing,
)
.await
@@ -159,7 +159,7 @@ fn list_object_versions_output(
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::owned_envelope;
use rustfs_gateway_types::BucketName;
use rustfs_s3_types::S3ErrorCode;
@@ -319,7 +319,7 @@ mod tests {
};
let err = DefaultBucketUsecase::without_context()
.execute_list_object_versions(RequestEnvelope::from_s3s(&build_request((), http::Method::GET)), input)
.execute_list_object_versions(owned_envelope(http::Method::GET), input)
.await
.expect_err("a negative page size");
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
@@ -333,7 +333,7 @@ mod tests {
};
let err = DefaultBucketUsecase::without_context()
.execute_list_object_versions(RequestEnvelope::from_s3s(&build_request((), http::Method::GET)), input)
.execute_list_object_versions(owned_envelope(http::Method::GET), input)
.await
.expect_err("no store");
assert_eq!(err.code(), &S3ErrorCode::InternalError);
@@ -372,13 +372,12 @@ fn list_objects_next_marker(info: &ListObjectsV2Info, cursor: Option<&str>) -> O
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::owned_envelope;
use rustfs_gateway_types::BucketName;
#[test]
#[serial_test::serial]
fn list_apis_report_storage_read_quorum_as_unavailable() {
use crate::app::bucket_usecase::test_support::owned_envelope;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient, shared_gating_ecstore};
run_large_stack_test("list-quorum-surface", || async {
let usecase = DefaultBucketUsecase::with_context(Some(shared_gating_ambient().await));
@@ -485,7 +484,7 @@ mod tests {
}
fn envelope() -> RequestEnvelope {
RequestEnvelope::from_s3s(&build_request((), http::Method::GET))
owned_envelope(http::Method::GET)
}
#[test]
+2 -4
View File
@@ -56,7 +56,7 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::{build_request, owned_envelope};
use crate::app::bucket_usecase::{BucketOperations, DeleteBucketOptions, MakeBucketOptions};
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
@@ -75,10 +75,8 @@ mod tests {
let context = shared_gating_ambient().await;
let bucket = format!("logging-missing-{}", uuid::Uuid::new_v4().simple());
let usecase = DefaultBucketUsecase::with_context(Some(context));
let request = build_request((), Method::PUT);
let envelope = RequestEnvelope::from_s3s(&request);
let error = usecase
.execute_put_bucket_logging(envelope, &bucket, || {
.execute_put_bucket_logging(owned_envelope(Method::PUT), &bucket, || {
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid logging configuration"))
})
.await
+4 -7
View File
@@ -243,16 +243,15 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::owned_envelope;
use http::Method;
#[tokio::test]
async fn execute_delete_bucket_policy_returns_internal_error_when_store_uninitialized() {
let req = build_request((), Method::DELETE);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_delete_bucket_policy(RequestEnvelope::from_s3s(&req), "test-bucket")
.execute_delete_bucket_policy(owned_envelope(Method::DELETE), "test-bucket")
.await
.unwrap_err();
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
@@ -280,11 +279,10 @@ mod tests {
bucket: bucket("test-bucket"),
..Default::default()
};
let req = build_request((), Method::GET);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_get_bucket_policy_status(RequestEnvelope::from_s3s(&req), input)
.execute_get_bucket_policy_status(owned_envelope(Method::GET), input)
.await
.unwrap_err();
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
@@ -292,11 +290,10 @@ mod tests {
#[tokio::test]
async fn execute_put_bucket_policy_returns_internal_error_when_store_uninitialized() {
let req = build_request((), Method::PUT);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_bucket_policy(RequestEnvelope::from_s3s(&req), "test-bucket", "{}")
.execute_put_bucket_policy(owned_envelope(Method::PUT), "test-bucket", "{}")
.await
.unwrap_err();
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
@@ -112,7 +112,7 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::{build_request, owned_envelope};
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::bucket_usecase::helper::MutationCompletionProbe;
@@ -278,7 +278,7 @@ mod tests {
#[tokio::test]
async fn execute_delete_public_access_block_returns_internal_error_when_store_uninitialized() {
let envelope = RequestEnvelope::from_s3s(&build_request((), Method::DELETE));
let envelope = owned_envelope(Method::DELETE);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_delete_public_access_block(envelope, "test-bucket")
@@ -296,7 +296,7 @@ mod tests {
#[tokio::test]
async fn execute_put_public_access_block_returns_internal_error_when_store_uninitialized() {
let envelope = RequestEnvelope::from_s3s(&build_request((), Method::PUT));
let envelope = owned_envelope(Method::PUT);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_public_access_block(envelope, "test-bucket", Ok(PersistedPublicAccessBlockConfiguration::default()))
+3 -4
View File
@@ -138,7 +138,7 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::{build_request, owned_envelope};
use crate::app::storage_api::s3::dto::{self, PutBucketTaggingInput, Tag, Tagging};
use http::Method;
use rustfs_s3_types::S3ErrorCode;
@@ -249,11 +249,10 @@ mod tests {
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_bucket_tagging(RequestEnvelope::from_s3s(&req), &req.input.bucket, PersistedTagging::default())
.execute_put_bucket_tagging(owned_envelope(Method::PUT), &input.bucket, PersistedTagging::default())
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
@@ -267,7 +266,7 @@ mod tests {
(vec![tag; MAX_BUCKET_TAGS + 1], "Cannot have more than 50 tags per bucket"),
] {
let err = usecase
.execute_put_bucket_tagging(RequestEnvelope::from_s3s(&req), &req.input.bucket, PersistedTagging { tag_set })
.execute_put_bucket_tagging(owned_envelope(Method::PUT), &input.bucket, PersistedTagging { tag_set })
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidTag);
+11 -1
View File
@@ -47,7 +47,17 @@ pub(super) fn gateway_bucket(name: &str) -> rustfs_gateway_types::BucketName {
/// The envelope of a request that carries nothing but `method`.
pub(super) fn owned_envelope(method: Method) -> RequestEnvelope {
RequestEnvelope::from_s3s(&build_request((), method))
RequestEnvelope::from_parts(
crate::app::RequestHead {
method,
uri: Uri::from_static("/"),
headers: HeaderMap::new(),
credentials: None,
region: None,
service: None,
},
Extensions::new(),
)
}
pub(super) fn build_request_with_req_info<T>(input: T, method: Method, req_info: ReqInfo) -> S3Request<T> {
+4 -5
View File
@@ -81,7 +81,7 @@ impl DefaultBucketUsecase {
mod tests {
use super::super::{BUCKET_ACCELERATE_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG};
use super::*;
use crate::app::bucket_usecase::test_support::{build_request, build_request_with_req_info};
use crate::app::bucket_usecase::test_support::{build_request, build_request_with_req_info, owned_envelope};
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::object_usecase::helper::MutationCompletionProbe;
@@ -190,22 +190,21 @@ mod tests {
let ambient = shared_gating_ambient().await;
let usecase = DefaultBucketUsecase::with_context(Some(ambient));
let bucket = format!("core-config-missing-{}", uuid::Uuid::new_v4().simple());
let request = build_request((), http::Method::PUT);
let mut converted = [false; 3];
let accelerate = usecase
.execute_put_bucket_accelerate_configuration(RequestEnvelope::from_s3s(&request), &bucket, || {
.execute_put_bucket_accelerate_configuration(owned_envelope(http::Method::PUT), &bucket, || {
converted[0] = true;
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid acceleration"))
})
.await;
let payment = usecase
.execute_put_bucket_request_payment(RequestEnvelope::from_s3s(&request), &bucket, || {
.execute_put_bucket_request_payment(owned_envelope(http::Method::PUT), &bucket, || {
converted[1] = true;
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid request payment"))
})
.await;
let website = usecase
.execute_put_bucket_website(RequestEnvelope::from_s3s(&request), &bucket, || {
.execute_put_bucket_website(owned_envelope(http::Method::PUT), &bucket, || {
converted[2] = true;
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid website"))
})
@@ -52,8 +52,19 @@ async fn object_lock_handlers_schedule_replication() {
.expect("Failed to create object-lock bucket");
upload_test_object(&ecstore, bucket.as_str(), object, b"worm payload").await;
let req = build_request((), Method::PUT);
let envelope = || RequestEnvelope::from_s3s(&req);
let envelope = || {
RequestEnvelope::from_parts(
crate::app::RequestHead {
method: Method::PUT,
uri: Uri::from_static("/"),
headers: HeaderMap::new(),
credentials: None,
region: None,
service: None,
},
Extensions::new(),
)
};
// A bucket without a replication configuration can skip the decision.
MUST_REPLICATE_OBJECT_CALLS.store(0, Ordering::SeqCst);
@@ -22,7 +22,17 @@ const RESTORE_SUSPENDED_WAIT_TIMEOUT: Duration = Duration::from_secs(15);
async fn suspend_test_bucket(bucket: &str) {
DefaultBucketUsecase::from_global()
.execute_put_bucket_versioning(
crate::app::RequestEnvelope::from_s3s(&build_request((), Method::PUT)),
crate::app::RequestEnvelope::from_parts(
crate::app::RequestHead {
method: Method::PUT,
uri: Uri::from_static("/"),
headers: HeaderMap::new(),
credentials: None,
region: None,
service: None,
},
Extensions::new(),
),
bucket,
Ok(rustfs_gateway_types::persistence::PersistedVersioningConfiguration {
status: Some("Suspended".to_owned()),
+19 -6
View File
@@ -545,10 +545,17 @@ mod tests {
.key(key.to_owned())
.build()
.expect("valid create request DTO");
let error = make_usecase()
.execute_create_multipart_upload_from_s3s(build_request(create, Method::POST))
let error = match make_usecase()
.execute_create_multipart_upload(
owned_envelope(Method::POST),
super::create_values::CreateMultipartUploadFields::from_legacy(create),
None,
)
.await
.expect_err("reserved key must be rejected before accessing storage");
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("reserved key must be rejected before accessing storage"),
};
assert_eq!(error.code(), &crate::app::storage_api::s3::S3ErrorCode::InvalidArgument);
let complete = CompleteMultipartUploadInput::builder()
@@ -558,9 +565,15 @@ mod tests {
.multipart_upload(Some(CompletedMultipartUpload::default()))
.build()
.expect("valid complete request DTO");
let error = Box::pin(make_usecase().complete_for_s3s_request(build_request(complete, Method::POST)))
.await
.expect_err("uploads created before the fix must not publish reserved keys");
let error =
match rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload::input_from_s3s(
complete,
) {
Err(error) => crate::app::dto_values::seam_error(error),
Ok(input) => Box::pin(make_usecase().execute_complete_multipart_upload(owned_envelope(Method::POST), input))
.await
.expect_err("uploads created before the fix must not publish reserved keys"),
};
assert_eq!(error.code(), &rustfs_s3_types::S3ErrorCode::InvalidArgument);
}
}
@@ -24,8 +24,7 @@ use std::sync::atomic::{AtomicUsize, Ordering};
#[tokio::test]
async fn execute_abort_multipart_upload_returns_internal_error_when_store_uninitialized() {
let request = build_request((), Method::DELETE);
let envelope = RequestEnvelope::from_s3s(&request);
let envelope = owned_envelope(Method::DELETE);
let before = s3_op_total(S3Operation::AbortMultipartUpload);
let target = AbortMultipartUploadTarget::Legacy {
bucket: "bucket".to_owned(),
+233 -38
View File
@@ -943,21 +943,72 @@ mod tests {
#[tokio::test]
async fn execute_complete_multipart_upload_rejects_missing_parts_payload() {
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let input = CompleteMultipartUploadInput::builder()
.bucket("bucket".to_string())
.key("object".to_string())
.upload_id("upload-id".to_string())
.build()
.unwrap();
let req = build_request(input, Method::POST);
let err = Box::pin(make_usecase().complete_for_s3s_request(req)).await.unwrap_err();
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let err = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => crate::app::dto_values::seam_error(error),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
let result = match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(make_usecase().execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(make_usecase().execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
};
result.unwrap_err()
}
};
assert_eq!(err.code(), &S3ErrorCode::InvalidPart);
}
#[tokio::test]
async fn execute_complete_multipart_upload_rejects_missing_parts_list() {
use crate::app::storage_api::s3::xml::Deserialize as _;
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let mut deserializer = crate::app::storage_api::s3::xml::Deserializer::new(b"<CompleteMultipartUpload/>");
let multipart_upload =
@@ -974,11 +1025,57 @@ mod tests {
.multipart_upload(Some(multipart_upload))
.build()
.expect("complete multipart input should build");
let req = build_request(input, Method::POST);
let err = Box::pin(make_usecase().complete_for_s3s_request(req))
.await
.expect_err("missing parts list must be rejected");
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let err = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => crate::app::dto_values::seam_error(error),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
let result = match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(make_usecase().execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(make_usecase().execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
};
result.expect_err("missing parts list must be rejected")
}
};
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
}
@@ -1124,10 +1221,9 @@ mod tests {
crate::app::gating_test_env::run_large_stack_test("ssec-completion-checksums", || async {
let (_store, bucket) =
crate::app::gating_test_env::durable_quota_test_bucket("ssec-completion-checksums", 16_384).await;
let versioning_req = build_request((), Method::PUT);
crate::app::bucket_usecase::DefaultBucketUsecase::from_global()
.execute_put_bucket_versioning(
RequestEnvelope::from_s3s(&versioning_req),
super::super::test_support::owned_envelope(Method::PUT),
&bucket,
Ok(rustfs_gateway_types::persistence::PersistedVersioningConfiguration {
status: Some("Enabled".to_owned()),
@@ -1304,16 +1400,13 @@ mod tests {
let bucket_usecase = crate::app::bucket_usecase::DefaultBucketUsecase::from_global();
// Listings carry neither the SSE-C key nor a per-object decryption request.
let versions_req = build_request(
ListObjectVersionsInput {
bucket: bucket.to_string(),
prefix: Some(key.to_string()),
..Default::default()
},
Method::GET,
);
let versions_env = RequestEnvelope::from_s3s(&versions_req);
let versions_in = list_object_versions::input_from_s3s(versions_req.input).expect("convert ListObjectVersionsInput");
let versions_input = ListObjectVersionsInput {
bucket: bucket.to_string(),
prefix: Some(key.to_string()),
..Default::default()
};
let versions_env = super::super::test_support::owned_envelope(Method::GET);
let versions_in = list_object_versions::input_from_s3s(versions_input).expect("convert ListObjectVersionsInput");
let versions = list_object_versions::output_to_s3s(
bucket_usecase
.execute_list_object_versions(versions_env, versions_in)
@@ -1328,16 +1421,13 @@ mod tests {
assert_eq!(entries[0].e_tag.as_ref(), Some(etag));
assert_xml_etag(&versions, etag);
let v1_req = build_request(
ListObjectsInput {
bucket: bucket.to_string(),
prefix: Some(key.to_string()),
..Default::default()
},
Method::GET,
);
let v1_env = RequestEnvelope::from_s3s(&v1_req);
let v1_in = list_objects::input_from_s3s(v1_req.input).expect("convert ListObjectsInput");
let v1_input = ListObjectsInput {
bucket: bucket.to_string(),
prefix: Some(key.to_string()),
..Default::default()
};
let v1_env = super::super::test_support::owned_envelope(Method::GET);
let v1_in = list_objects::input_from_s3s(v1_input).expect("convert ListObjectsInput");
let v1 = list_objects::output_to_s3s(
bucket_usecase
.execute_list_objects(v1_env, v1_in)
@@ -1411,7 +1501,6 @@ mod tests {
.await
.expect("create versioned Object Lock bucket");
let key = "kms-encrypted";
let enc_req = build_request((), Method::PUT);
let enc_cfg = crate::app::storage_api::s3::dto::ServerSideEncryptionConfiguration {
rules: vec![crate::app::storage_api::s3::dto::ServerSideEncryptionRule {
apply_server_side_encryption_by_default: Some(
@@ -1425,7 +1514,7 @@ mod tests {
};
let prepared = crate::storage::persisted_from_s3s(&enc_cfg).map_err(Into::into);
crate::app::bucket_usecase::DefaultBucketUsecase::from_global()
.execute_put_bucket_encryption(RequestEnvelope::from_s3s(&enc_req), &bucket, prepared)
.execute_put_bucket_encryption(super::super::test_support::owned_envelope(Method::PUT), &bucket, prepared)
.await
.expect("configure bucket-default SSE-KMS");
let usecase = DefaultMultipartUsecase::from_global();
@@ -2151,6 +2240,8 @@ mod tests {
#[tokio::test]
async fn execute_complete_multipart_upload_allows_duplicate_part_numbers_by_using_last_occurrence() {
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let multipart_upload = CompletedMultipartUpload {
parts: Some(vec![
CompletedPart {
@@ -2170,14 +2261,67 @@ mod tests {
.multipart_upload(Some(multipart_upload))
.build()
.unwrap();
let req = build_request(input, Method::POST);
let err = Box::pin(make_usecase().complete_for_s3s_request(req)).await.unwrap_err();
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let err = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => crate::app::dto_values::seam_error(error),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
let result = match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(make_usecase().execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(make_usecase().execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
};
match result {
Err(error) => error,
Ok(_output) => panic!("duplicate part numbers must not succeed before the order check"),
}
}
};
assert_ne!(err.code(), &S3ErrorCode::InvalidPartOrder);
}
#[tokio::test]
async fn execute_complete_multipart_upload_rejects_out_of_order_parts() {
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::complete_multipart_upload;
let multipart_upload = CompletedMultipartUpload {
parts: Some(vec![
CompletedPart {
@@ -2197,9 +2341,60 @@ mod tests {
.multipart_upload(Some(multipart_upload))
.build()
.unwrap();
let req = build_request(input, Method::POST);
let err = Box::pin(make_usecase().complete_for_s3s_request(req)).await.unwrap_err();
let mut s3s_input = input;
let has_upload = s3s_input.multipart_upload.is_some();
s3s_input.multipart_upload.get_or_insert_with(Default::default);
let tags: Vec<Option<String>> = s3s_input
.multipart_upload
.iter_mut()
.flat_map(|upload| upload.parts.iter_mut().flatten())
.map(|part| {
part.part_number.get_or_insert(0);
part.e_tag.take().map(|tag| tag.value().to_owned())
})
.collect();
for checksum in [
&mut s3s_input.checksum_crc32,
&mut s3s_input.checksum_crc32c,
&mut s3s_input.checksum_crc64nvme,
&mut s3s_input.checksum_md5,
&mut s3s_input.checksum_sha1,
&mut s3s_input.checksum_sha256,
&mut s3s_input.checksum_sha512,
&mut s3s_input.checksum_xxhash128,
&mut s3s_input.checksum_xxhash3,
&mut s3s_input.checksum_xxhash64,
] {
*checksum = None;
}
let err = match complete_multipart_upload::input_from_s3s(s3s_input) {
Err(error) => crate::app::dto_values::seam_error(error),
Ok(mut owned) => {
for (part, tag) in owned.multipart_upload.parts.iter_mut().zip(tags) {
part.e_tag = tag.map(rustfs_gateway_types::ETag::from_stored);
}
let taken = std::mem::take(&mut owned.multipart_upload);
let parts = has_upload.then_some(taken);
let result = match parts {
Some(parts) => {
owned.multipart_upload = parts;
Box::pin(make_usecase().execute_complete_multipart_upload(owned_envelope(Method::POST), owned)).await
}
None => {
Box::pin(make_usecase().execute_complete_multipart_upload_parts(
owned_envelope(Method::POST),
owned,
None,
))
.await
}
};
match result {
Err(error) => error,
Ok(_output) => panic!("out-of-order parts must be rejected"),
}
}
};
assert_eq!(err.code(), &S3ErrorCode::InvalidPartOrder);
}
+16 -8
View File
@@ -545,10 +545,17 @@ mod tests {
.object_lock_retain_until_date(date.map(Timestamp::from))
.build()
.expect("direct fixture DTO");
let error = make_usecase()
.execute_create_multipart_upload_from_s3s(build_request(input, Method::POST))
let error = match make_usecase()
.execute_create_multipart_upload(
owned_envelope(Method::POST),
CreateMultipartUploadFields::from_legacy(input),
None,
)
.await
.expect_err("direct fixture does not create an upload");
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("direct fixture does not create an upload"),
};
assert_eq!(
(error.code().as_str(), error.message()),
(expected.0, Some(expected.1)),
@@ -590,12 +597,13 @@ mod tests {
.storage_class(Some(StorageClass::from("invalid".to_string())))
.build()
.unwrap();
let req = build_request(input, Method::POST);
let err = make_usecase()
.execute_create_multipart_upload_from_s3s(req)
let err = match make_usecase()
.execute_create_multipart_upload(owned_envelope(Method::POST), CreateMultipartUploadFields::from_legacy(input), None)
.await
.unwrap_err();
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("invalid storage class must be rejected"),
};
assert_eq!(err.code(), &S3ErrorCode::InvalidStorageClass);
}
+2 -2
View File
@@ -295,7 +295,7 @@ mod tests {
async fn list_uploads(input: gateway_dto::ListMultipartUploadsInput) -> rustfs_s3_types::S3Error {
make_usecase()
.execute_list_multipart_uploads(RequestEnvelope::from_s3s(&build_request((), Method::GET)), input)
.execute_list_multipart_uploads(owned_envelope(Method::GET), input)
.await
.expect_err("refused")
}
@@ -393,7 +393,7 @@ mod tests {
max_parts,
};
make_usecase()
.execute_list_parts(RequestEnvelope::from_s3s(&build_request((), Method::GET)), fields)
.execute_list_parts(owned_envelope(Method::GET), fields)
.await
.expect_err("refused")
}
@@ -14,7 +14,7 @@
//! ListParts execution against real multipart metadata and the native claim exchange.
use super::super::test_support::{Method, build_request};
use super::super::test_support::{Method, build_request, owned_envelope};
use super::*;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient, shared_gating_ecstore_and_disk_paths};
use crate::app::storage_api::multipart_usecase::access::{AuthorizationRequest, load_bucket_generation_from_store};
@@ -81,7 +81,7 @@ fn list_parts_resolves_real_owners_and_retains_storage_errors() {
let key = "fixture/one";
let foreign_key = "fixture/two";
let opts = ObjectOptions::default();
let envelope = || RequestEnvelope::from_s3s(&build_request((), Method::GET));
let envelope = || owned_envelope(Method::GET);
for name in [&bucket, &foreign_bucket] {
store
.make_bucket(name, &MakeBucketOptions::default())
@@ -18,6 +18,7 @@ use super::*;
use crate::app::storage_api::s3::S3Request;
pub(super) use crate::app::storage_api::s3::dto::CompletedMultipartUpload;
pub(super) use crate::app::storage_api::s3::dto::StorageClass;
use crate::app::{RequestEnvelope, RequestHead};
pub(super) use http::Extensions;
pub(super) use http::Method;
pub(super) use rustfs_filemeta::ObjectPartInfo;
@@ -50,6 +51,21 @@ pub(super) fn build_request<T>(input: T, method: Method) -> S3Request<T> {
}
}
/// The envelope of a request that carries nothing but `method`.
pub(super) fn owned_envelope(method: Method) -> RequestEnvelope {
RequestEnvelope::from_parts(
RequestHead {
method,
uri: Uri::from_static("/"),
headers: HeaderMap::new(),
credentials: None,
region: None,
service: None,
},
Extensions::new(),
)
}
pub(super) fn make_usecase() -> DefaultMultipartUsecase {
DefaultMultipartUsecase::without_context()
}
+27 -12
View File
@@ -741,9 +741,13 @@ mod tests {
.part_number(1)
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let err = make_usecase().execute_upload_part(req).await.unwrap_err();
let err = match make_usecase()
.execute_upload_part_with_audit(owned_envelope(Method::PUT), UploadPartWriteInput::from_legacy(input, None))
.await
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("missing body must be rejected"),
};
assert_eq!(err.code(), &LegacyErrorCode::IncompleteBody);
}
@@ -765,9 +769,13 @@ mod tests {
.content_length(Some(declared))
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let err = make_usecase().execute_upload_part(req).await.unwrap_err();
let err = match make_usecase()
.execute_upload_part_with_audit(owned_envelope(Method::PUT), UploadPartWriteInput::from_legacy(input, None))
.await
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("declared part must be rejected"),
};
if expect_too_large {
assert_eq!(err.code(), &LegacyErrorCode::EntityTooLarge, "declared {declared}");
assert_eq!(polls.load(std::sync::atomic::Ordering::SeqCst), 0, "body must not be polled");
@@ -787,9 +795,13 @@ mod tests {
.part_number(part_number)
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let err = make_usecase().execute_upload_part(req).await.unwrap_err();
let err = match make_usecase()
.execute_upload_part_with_audit(owned_envelope(Method::PUT), UploadPartWriteInput::from_legacy(input, None))
.await
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("invalid part number must be rejected"),
};
assert_eq!(err.code(), &LegacyErrorCode::InvalidArgument);
assert_eq!(err.message(), Some("partNumber must be between 1 and 10000"));
}
@@ -971,10 +983,13 @@ mod tests {
.build()
.expect("upload part input should build");
let err = usecase
.execute_upload_part(build_request(input, Method::PUT))
let err = match usecase
.execute_upload_part_with_audit(owned_envelope(Method::PUT), UploadPartWriteInput::from_legacy(input, None))
.await
.expect_err("full foreground write admission should reject before body ingest");
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("full foreground write admission should reject before body ingest"),
};
assert_eq!(err.code(), &LegacyErrorCode::SlowDown);
drop(held);
}
@@ -14,6 +14,7 @@
//! Upload target ownership and refusal order against real multipart state.
use super::super::super::test_support::owned_envelope;
use super::*;
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use rustfs_gateway_types::{BucketName, ObjectKey, UploadIdClaim};
@@ -101,7 +102,7 @@ async fn native_claim_checks_order_ownership_and_one_active_lookup() {
Body::from(Bytes::from_static(b"abc"))
}
});
let envelope = RequestEnvelope::from_s3s(&build_request((), Method::PUT));
let envelope = owned_envelope(Method::PUT);
LOOKUP
.scope((Cell::new(0), mode), async {
let result = usecase.execute_upload_part_with_audit(envelope, input).await;
@@ -211,7 +212,7 @@ fn native_copy_part_rejects_unproved_owner_without_changing_parts() {
copy_source_sse_customer_key: None,
copy_source_sse_customer_key_md5: None,
};
let envelope = RequestEnvelope::from_s3s(&build_request((), Method::PUT));
let envelope = owned_envelope(Method::PUT);
LOOKUP
.scope((Cell::new(0), mode), async {
let result = Box::pin(usecase.execute_upload_part_copy_with_audit(envelope, input)).await;
@@ -561,6 +561,9 @@ impl DefaultMultipartUsecase {
mod tests {
use super::super::test_support::*;
use super::*;
use crate::app::multipart_usecase::UploadPartTarget;
use crate::app::object_usecase::CopySourceCondition;
use crate::app::storage_api::s3::dto::ETag;
#[tokio::test]
async fn execute_upload_part_copy_returns_internal_error_when_store_uninitialized() {
@@ -576,9 +579,37 @@ mod tests {
.upload_id("upload-id".to_string())
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let err = Box::pin(make_usecase().execute_upload_part_copy(req)).await.unwrap_err();
let condition = |value| match value {
crate::app::storage_api::s3::dto::ETagCondition::Any => CopySourceCondition::Any,
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Strong(value)) => CopySourceCondition::Strong(value),
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Weak(_)) => CopySourceCondition::Weak,
};
let err = match Box::pin(make_usecase().execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: input.bucket,
key: input.key,
upload_id: input.upload_id,
},
part_number: input.part_number,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(input.copy_source),
copy_source_range: input.copy_source_range,
copy_source_if_match: input.copy_source_if_match.map(&condition),
copy_source_if_none_match: input.copy_source_if_none_match.map(&condition),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key,
sse_customer_key_md5: input.sse_customer_key_md5,
copy_source_sse_customer_algorithm: input.copy_source_sse_customer_algorithm,
copy_source_sse_customer_key: input.copy_source_sse_customer_key,
copy_source_sse_customer_key_md5: input.copy_source_sse_customer_key_md5,
},
))
.await
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("uninitialized store must be an internal error"),
};
assert_eq!(err.code(), &crate::app::storage_api::s3::S3ErrorCode::InternalError);
}
@@ -597,9 +628,37 @@ mod tests {
.upload_id("upload-id".to_string())
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let err = Box::pin(make_usecase().execute_upload_part_copy(req)).await.unwrap_err();
let condition = |value| match value {
crate::app::storage_api::s3::dto::ETagCondition::Any => CopySourceCondition::Any,
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Strong(value)) => CopySourceCondition::Strong(value),
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Weak(_)) => CopySourceCondition::Weak,
};
let err = match Box::pin(make_usecase().execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: input.bucket,
key: input.key,
upload_id: input.upload_id,
},
part_number: input.part_number,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(input.copy_source),
copy_source_range: input.copy_source_range,
copy_source_if_match: input.copy_source_if_match.map(&condition),
copy_source_if_none_match: input.copy_source_if_none_match.map(&condition),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key,
sse_customer_key_md5: input.sse_customer_key_md5,
copy_source_sse_customer_algorithm: input.copy_source_sse_customer_algorithm,
copy_source_sse_customer_key: input.copy_source_sse_customer_key,
copy_source_sse_customer_key_md5: input.copy_source_sse_customer_key_md5,
},
))
.await
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("invalid part number must be rejected"),
};
assert_eq!(err.code(), &crate::app::storage_api::s3::S3ErrorCode::InvalidArgument);
assert_eq!(err.message(), Some("partNumber must be between 1 and 10000"));
}
@@ -76,9 +76,38 @@ fn upload_part_copy_shares_foreground_admission_with_upload_part() {
.any(|label| label.key() == "size_bucket" && label.value() == source_size_bucket)
})
};
let error = Box::pin(usecase.execute_upload_part_copy(build_request(input.clone(), Method::PUT)))
.await
.expect_err("copy must not bypass the occupied UploadPart pool");
let saturated = input.clone();
let condition = |value| match value {
crate::app::storage_api::s3::dto::ETagCondition::Any => CopySourceCondition::Any,
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Strong(value)) => CopySourceCondition::Strong(value),
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Weak(_)) => CopySourceCondition::Weak,
};
let error = match Box::pin(usecase.execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: saturated.bucket,
key: saturated.key,
upload_id: saturated.upload_id,
},
part_number: saturated.part_number,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(saturated.copy_source),
copy_source_range: saturated.copy_source_range,
copy_source_if_match: saturated.copy_source_if_match.map(&condition),
copy_source_if_none_match: saturated.copy_source_if_none_match.map(&condition),
sse_customer_algorithm: saturated.sse_customer_algorithm,
sse_customer_key: saturated.sse_customer_key,
sse_customer_key_md5: saturated.sse_customer_key_md5,
copy_source_sse_customer_algorithm: saturated.copy_source_sse_customer_algorithm,
copy_source_sse_customer_key: saturated.copy_source_sse_customer_key,
copy_source_sse_customer_key_md5: saturated.copy_source_sse_customer_key_md5,
},
))
.await
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("copy must not bypass the occupied UploadPart pool"),
};
assert_eq!(error.code(), &S3ErrorCode::SlowDown);
assert!(!source_reader_observed(), "saturated copy must not construct a source body reader");
let parts = store
@@ -165,9 +194,37 @@ fn upload_part_copy_shares_foreground_admission_with_upload_part() {
.upload_id(deleted_upload.upload_id.clone())
.build()
.expect("deleted-source copy request");
let error = Box::pin(usecase.execute_upload_part_copy(build_request(input, Method::PUT)))
.await
.expect_err("deleted source must fail before conditions or admission");
let condition = |value| match value {
crate::app::storage_api::s3::dto::ETagCondition::Any => CopySourceCondition::Any,
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Strong(value)) => CopySourceCondition::Strong(value),
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Weak(_)) => CopySourceCondition::Weak,
};
let error = match Box::pin(usecase.execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: input.bucket,
key: input.key,
upload_id: input.upload_id,
},
part_number: input.part_number,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(input.copy_source),
copy_source_range: input.copy_source_range,
copy_source_if_match: input.copy_source_if_match.map(&condition),
copy_source_if_none_match: input.copy_source_if_none_match.map(&condition),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key,
sse_customer_key_md5: input.sse_customer_key_md5,
copy_source_sse_customer_algorithm: input.copy_source_sse_customer_algorithm,
copy_source_sse_customer_key: input.copy_source_sse_customer_key,
copy_source_sse_customer_key_md5: input.copy_source_sse_customer_key_md5,
},
))
.await
{
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("deleted source must fail before conditions or admission"),
};
assert_eq!(error.code(), &expected);
}
});
@@ -211,7 +268,32 @@ fn upload_part_copy_range_error_and_cancellation_release_foreground_permit() {
.upload_id(cancelled_upload.upload_id.clone())
.build()
.expect("cancelled copy request");
let mut cancelled_copy = Box::pin(usecase.execute_upload_part_copy(build_request(cancelled_input, Method::PUT)));
let condition = |value| match value {
crate::app::storage_api::s3::dto::ETagCondition::Any => CopySourceCondition::Any,
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Strong(value)) => CopySourceCondition::Strong(value),
crate::app::storage_api::s3::dto::ETagCondition::ETag(ETag::Weak(_)) => CopySourceCondition::Weak,
};
let mut cancelled_copy = Box::pin(usecase.execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: cancelled_input.bucket,
key: cancelled_input.key,
upload_id: cancelled_input.upload_id,
},
part_number: cancelled_input.part_number,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(cancelled_input.copy_source),
copy_source_range: cancelled_input.copy_source_range,
copy_source_if_match: cancelled_input.copy_source_if_match.map(&condition),
copy_source_if_none_match: cancelled_input.copy_source_if_none_match.map(&condition),
sse_customer_algorithm: cancelled_input.sse_customer_algorithm,
sse_customer_key: cancelled_input.sse_customer_key,
sse_customer_key_md5: cancelled_input.sse_customer_key_md5,
copy_source_sse_customer_algorithm: cancelled_input.copy_source_sse_customer_algorithm,
copy_source_sse_customer_key: cancelled_input.copy_source_sse_customer_key,
copy_source_sse_customer_key_md5: cancelled_input.copy_source_sse_customer_key_md5,
},
));
tokio::time::timeout(Duration::from_secs(10), async {
tokio::select! {
biased;
@@ -220,7 +302,10 @@ fn upload_part_copy_range_error_and_cancellation_release_foreground_permit() {
tokio::task::yield_now().await;
}
} => {}
result = &mut cancelled_copy => panic!("copy completed before cancellation: {result:?}"),
result = &mut cancelled_copy => match result {
Ok(_) => panic!("copy completed before cancellation"),
Err(error) => panic!("copy completed before cancellation: {error:?}"),
},
}
})
.await
@@ -260,17 +345,39 @@ fn upload_part_copy_range_error_and_cancellation_release_foreground_permit() {
.upload_id(upload.upload_id.clone())
.build()
.expect("range request");
let response = Box::pin(usecase.execute_upload_part_copy(build_request(input, Method::PUT))).await;
let response = Box::pin(usecase.execute_upload_part_copy_with_audit(
owned_envelope(Method::PUT),
UploadPartCopyWriteInput {
target: UploadPartTarget::Legacy {
bucket: input.bucket,
key: input.key,
upload_id: input.upload_id,
},
part_number: input.part_number,
copy_source: rustfs_gateway_types::compat::s3s_5761ddfe::copy_source::from_s3s(input.copy_source),
copy_source_range: input.copy_source_range,
copy_source_if_match: input.copy_source_if_match.map(&condition),
copy_source_if_none_match: input.copy_source_if_none_match.map(&condition),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key,
sse_customer_key_md5: input.sse_customer_key_md5,
copy_source_sse_customer_algorithm: input.copy_source_sse_customer_algorithm,
copy_source_sse_customer_key: input.copy_source_sse_customer_key,
copy_source_sse_customer_key_md5: input.copy_source_sse_customer_key_md5,
},
))
.await;
assert_eq!(
manager.put_object_admission_snapshot().active,
Some(0),
"copy releases its permit on every outcome"
);
if capped {
assert_eq!(
response.expect_err("range exceeds destination size limit").code(),
&S3ErrorCode::EntityTooLarge
);
let error = match response {
Err(error) => crate::app::storage_api::s3::S3Error::from(error),
Ok(_) => panic!("range exceeds destination size limit"),
};
assert_eq!(error.code(), &S3ErrorCode::EntityTooLarge);
let parts = store
.list_object_parts(&bucket, "object", &upload.upload_id, None, 1000, &ObjectOptions::default())
.await
@@ -281,16 +388,16 @@ fn upload_part_copy_range_error_and_cancellation_release_foreground_permit() {
.await
.expect("abort capped session");
} else {
let copied = response.expect("range copy after failed request released its source and permit");
let copied = match response {
Ok(output) => output,
Err(error) => panic!("range copy after failed request released its source and permit: {error:?}"),
};
let etag = copied
.output
.copy_part_result
.expect("range result")
.e_tag
.expect("range ETag")
.value()
.trim_matches('"')
.to_owned();
.as_deref()
.map(crate::storage::storage_api::to_s3s_xml_etag)
.expect("range ETag");
let etag = etag.value().trim_matches('"').to_owned();
Arc::clone(&store)
.complete_multipart_upload(
&bucket,
+11 -2
View File
@@ -258,8 +258,17 @@ mod tests {
sse_customer_key: None,
sse_customer_key_md5: None,
};
let request = build_request((), http::Method::GET);
let envelope = RequestEnvelope::from_s3s(&request);
let envelope = RequestEnvelope::from_parts(
crate::app::RequestHead {
method: http::Method::GET,
uri: http::Uri::from_static("/"),
headers: http::HeaderMap::new(),
credentials: None,
region: None,
service: None,
},
http::Extensions::new(),
);
let error = DefaultObjectUsecase::without_context()
.execute_get_object_attributes(envelope, fields)
.await
@@ -1,22 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Authorization shapes that only migration batch A2 (rustfs/backlog#2765) uses: bucket policy, policy status, versioning, notification and ACL.
//!
//! Responsible for: the owned authorization function of each such operation, generic over
//! `AuthorizationRequest`, so the legacy `S3Access` hook and `Served::authorize` call the same code.
//! Not responsible for: shapes more than one batch shares (those stay in `storage/access.rs`), the
//! policy evaluation itself (`authorize_request`), or the operation's use case.
//! Upstream: the legacy hook in `storage/access.rs` and the batch's `server/gateway_handlers/bucket_access.rs`.
//! Downstream: the shared shapes in `storage/access.rs`.
@@ -1,22 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Authorization shapes that only migration batch A1 (rustfs/backlog#2765) uses: bucket CORS, tagging, public access block and encryption.
//!
//! Responsible for: the owned authorization function of each such operation, generic over
//! `AuthorizationRequest`, so the legacy `S3Access` hook and `Served::authorize` call the same code.
//! Not responsible for: shapes more than one batch shares (those stay in `storage/access.rs`), the
//! policy evaluation itself (`authorize_request`), or the operation's use case.
//! Upstream: the legacy hook in `storage/access.rs` and the batch's `server/gateway_handlers/bucket_configs.rs`.
//! Downstream: the shared shapes in `storage/access.rs`.
-2
View File
@@ -15,8 +15,6 @@
//! The authorization shapes one migration batch owns (rustfs/backlog#2765), one file per batch so
//! parallel batches edit disjoint files. Shapes several batches share stay in `storage/access.rs`.
pub(crate) mod bucket_access;
pub(crate) mod bucket_configs;
pub(crate) mod bucket_core;
pub(crate) mod bucket_replication_lifecycle;
pub(crate) mod listing_multipart;