Compare commits

...

22 Commits

Author SHA1 Message Date
唐小鸭 cf1c45eb91 test(replication): pin directory-marker null version and replication (#7315) 2026-09-06 19:26:09 +00:00
唐小鸭 536c283716 test(ilm): realign source-marker assertions with handler order (#7321) 2026-09-07 02:10:13 +08:00
houseme a04ddc237e chore: remove Go heal outcome compatibility fixture (#7319)
* del go code

* chore: record ILM lifecycle validation on go-del (#7318)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 23:49:56 +08:00
houseme 7de6ac82e1 test(scanner): cover reset cleanup across store reopen (#7317)
Add a real ECStore reopen regression for scanner usage-state reset cleanup boundaries. The fixture seeds each partially completed cleanup state, recreates the store, then verifies the reset resumes without rewriting the bootstrap intent or deleting unrelated metadata.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 23:40:31 +08:00
唐小鸭 7993e11058 docs(s3): record object-key path-segment validation as intentional (#7314) 2026-09-06 23:27:10 +08:00
houseme 37ead2f69c test(ilm): align feature matrix lifecycle assertions (#7311)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 23:27:01 +08:00
houseme 1b1e590df7 test(scanner): verify quota state across reset and owner restart (#7303)
* test(scanner): verify quota state across reset and owner restart

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: restore ILM transition and lifecycle validation (#7312)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 23:26:01 +08:00
houseme 086ee8e48a fix(scanner): require root publication proof before dirty ack (#7297)
* fix(scanner): require root publication proof before dirty ack

Bind ACK expectations to validated scan candidates and confirm the actual primary-root revision and readback. Retain saved outcomes and dirty responsibility when stronger evidence is unavailable. Isolate CAS attempt confirmation and invalidate proof after scope mutations.

Revalidate observed candidate reuse before issuing a new publication proof, preserve the exact validated authoritative baseline work digest, and settle fixture commit tails before stable maintenance scans. Keep scoped ACK production disabled.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: restore ILM transition and lifecycle validation (#7316)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 23:25:40 +08:00
Zhengchao An 22bff27aee fix(storage): derive multipart identity from stored parts (#7305) 2026-09-06 22:04:15 +08:00
houseme 8a20498705 fix(heal): retain completed reports during clock rollback (#7299)
Treat a negative wall-clock age as zero without bypassing count or byte eviction. Cover canonical and alias queries, terminal outcomes, exact TTL expiry, and capacity limits during rollback.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 21:28:22 +08:00
houseme c3e6d90c6f docs(heal): clarify start retries and execution budgets (#7300)
Distinguish control requests, cumulative task execution and object retries. Explain ambiguous responses and bounded envelope replay without promising HTTP idempotency or changing runtime policy.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 21:27:37 +08:00
Zhengchao An d2af8f073e test(s3): exclude invalid lifecycle filter vector (#7295)
Co-authored-by: houseme <housemecn@gmail.com>
2026-09-06 21:21:12 +08:00
Zhengchao An 2baba1bda7 fix(scanner): preserve verified maintenance digest (#7293)
Co-authored-by: houseme <housemecn@gmail.com>
2026-09-06 21:20:58 +08:00
Zhengchao An bf2ca9113f test(scanner): settle fixture writes before activity baseline (#7290) 2026-09-06 21:20:46 +08:00
Zhengchao An f474ea30d4 fix(admin): preserve decommission readiness error (#7285) 2026-09-06 21:20:32 +08:00
cxymds 002ac9544c feat(ilm): add immutable legacy recovery exports (#7283)
* feat(ilm): add immutable legacy recovery exports

* feat(ilm): add legacy recovery disposition records (#7292)

* fix(ilm): stabilize legacy recovery decode errors

* fix(admin): route recovery auth errors through gateway

* fix(ilm): resolve recovery disposition clippy errors

* fix(ilm): remove redundant recovery test clones
2026-09-06 21:20:16 +08:00
Zhengchao An d74d970e24 fix(ecstore): retain namespace ownership during multipart commit (#7282) 2026-09-06 21:20:02 +08:00
Zhengchao An cc15eae479 fix(lifecycle): allow multiple filter predicates without And wrapper (#7298) 2026-09-06 21:09:11 +08:00
dependabot[bot] c130d00d4b chore(deps): bump golang.org/x/net from 0.54.0 to 0.55.0 in /scripts/compat/heal-outcome in the go_modules group across 1 directory (#7296)
chore(deps): bump golang.org/x/net

Bumps the go_modules group with 1 update in the /scripts/compat/heal-outcome directory: [golang.org/x/net](https://github.com/golang/net).


Updates `golang.org/x/net` from 0.54.0 to 0.55.0
- [Commits](https://github.com/golang/net/compare/v0.54.0...v0.55.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.55.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-06 19:13:21 +08:00
dependabot[bot] e49d9cdea2 chore(deps): bump the go_modules group across 1 directory with 2 updates (#7288)
Bumps the go_modules group with 1 update in the /scripts/compat/heal-outcome directory: [golang.org/x/crypto](https://github.com/golang/crypto).


Updates `golang.org/x/crypto` from 0.37.0 to 0.52.0
- [Commits](https://github.com/golang/crypto/compare/v0.37.0...v0.52.0)

Updates `golang.org/x/net` from 0.39.0 to 0.54.0
- [Commits](https://github.com/golang/net/compare/v0.39.0...v0.54.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.52.0
  dependency-type: indirect
  dependency-group: go_modules
- dependency-name: golang.org/x/net
  dependency-version: 0.54.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-06 18:56:54 +08:00
houseme f5778d8b97 chore(deps): refresh concurrency and protocol dependencies (#7289)
chore(deps): update Crossbeam, Redis, DER, and ipnet

Refresh the shared concurrency, Redis client, DER decoding and IP network dependencies while retaining existing feature selections and the hotpath pin.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-06 18:51:12 +08:00
Zhengchao An 9da42a899a test(connect): refresh offline enrollment e2e chain (#7294) 2026-09-06 18:30:04 +08:00
61 changed files with 7149 additions and 570 deletions
Generated
+34 -34
View File
@@ -2527,18 +2527,18 @@ checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b"
[[package]]
name = "crossbeam-channel"
version = "0.5.16"
version = "0.5.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d85363c37faeca707aef026efa9f3b34d077bce547e48f770770625c6013679e"
checksum = "98b0cc327b5bc766e7fda9c9260cc0fa81b43a8e240440422dff70788e3f9ef1"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-deque"
version = "0.8.7"
version = "0.8.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb"
checksum = "622f3fc73690be383c7214310406f28a90e6edeadc3cea882f9d71e495b9711a"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
@@ -2546,27 +2546,27 @@ dependencies = [
[[package]]
name = "crossbeam-epoch"
version = "0.9.20"
version = "0.9.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f"
checksum = "dc74980687109a3b14c72fd458107bf0baa1da1a1a805e178d15501ba9b86d9d"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-queue"
version = "0.3.13"
version = "0.3.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "803d13fb3b09d88be9f4dbc29062c66b19bf7170867ceb746d2a8689bf6c7a26"
checksum = "03e8bd762f7479489c70ed6c768ddca99d7296857de437a68dcb2a94365b3fae"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.22"
version = "0.8.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17"
checksum = "a31eee39dddec8330830986fcd7625edb5a24ec90ea038215273bbc3adb08ac6"
[[package]]
name = "crunchy"
@@ -3673,9 +3673,9 @@ dependencies = [
[[package]]
name = "der"
version = "0.8.1"
version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a69dedd701da44b0536442edf09c81a64b0ab97a7a4a5e3d1971f00027cbc63d"
checksum = "a878c850e9e421b20262e9b41f9c860e4785fa07541c266b62ff9d1ef998a80a"
dependencies = [
"const-oid 0.10.2",
"pem-rfc7468 1.0.0",
@@ -3946,7 +3946,7 @@ dependencies = [
"libc",
"option-ext",
"redox_users 0.5.2",
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -4090,7 +4090,7 @@ version = "0.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0681a4fc24c767085329728d8dfba959af91228aa4610cca4f8ce317ba46ae0"
dependencies = [
"der 0.8.1",
"der 0.8.2",
"digest 0.11.3",
"elliptic-curve 0.14.1",
"rfc6979 0.6.0",
@@ -4295,7 +4295,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -5693,7 +5693,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "20fd6de4ccfcc187e38bc21cfa543cb5a302cb86a8b114eb7f0bf0dc9f8ac00f"
dependencies = [
"io-lifetimes 3.0.1",
"windows-sys 0.60.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -5734,9 +5734,9 @@ dependencies = [
[[package]]
name = "ipnet"
version = "2.12.1"
version = "2.12.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78"
checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
dependencies = [
"serde",
]
@@ -5758,7 +5758,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
dependencies = [
"hermit-abi",
"libc",
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -6955,7 +6955,7 @@ version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -7933,7 +7933,7 @@ version = "0.8.0-rc.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "986d2e952779af96ea048f160fd9194e1751b4faea78bcf3ceb456efe008088e"
dependencies = [
"der 0.8.1",
"der 0.8.2",
"spki 0.8.0",
]
@@ -7976,7 +7976,7 @@ dependencies = [
"aes 0.9.3",
"aes-gcm",
"cbc 0.2.1",
"der 0.8.1",
"der 0.8.2",
"pbkdf2 0.13.0",
"rand_core 0.10.1",
"scrypt 0.12.0",
@@ -8000,7 +8000,7 @@ version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "451913da69c775a56034ea8d9003d27ee8948e12443eae7c038ba100a4f21cb7"
dependencies = [
"der 0.8.1",
"der 0.8.2",
"pkcs5 0.8.1",
"rand_core 0.10.1",
"spki 0.8.0",
@@ -8706,7 +8706,7 @@ dependencies = [
"once_cell",
"socket2",
"tracing",
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -8942,9 +8942,9 @@ dependencies = [
[[package]]
name = "redis"
version = "1.6.0"
version = "1.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e37a4ca5c6ca42aa3e6df2fd32b987a65d32a4c2159a6f3fe0fd1df306a2658f"
checksum = "2acbc41a996f7652b2ddd9dfd98cc4ff602cfd742ae35382f07f608405ab50ed"
dependencies = [
"arc-swap",
"arcstr",
@@ -9326,7 +9326,7 @@ dependencies = [
"curve25519-dalek 5.0.0",
"data-encoding",
"delegate",
"der 0.8.1",
"der 0.8.2",
"digest 0.11.3",
"ecdsa 0.17.0",
"ed25519-dalek 3.0.0",
@@ -11066,7 +11066,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -11149,7 +11149,7 @@ dependencies = [
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -11420,7 +11420,7 @@ checksum = "d56d437c2f19203ce5f7122e507831de96f3d2d4d3be5af44a0b0a09d8a80e4d"
dependencies = [
"base16ct 1.0.0",
"ctutils",
"der 0.8.1",
"der 0.8.2",
"hybrid-array",
"subtle",
"zeroize",
@@ -11994,7 +11994,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d9efca8738c78ee9484207732f728b1ef517bbb1833d6fc0879ca898a522f6f"
dependencies = [
"base64ct",
"der 0.8.1",
"der 0.8.2",
]
[[package]]
@@ -12406,10 +12406,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.4.3",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -13528,7 +13528,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.61.2",
"windows-sys 0.59.0",
]
[[package]]
+5 -5
View File
@@ -256,10 +256,10 @@ clap = { version = "4.6.6" }
const-str = { version = "1.1.0" }
convert_case = "0.12.0"
criterion = { version = "0.8" }
crossbeam-queue = "0.3.13"
crossbeam-channel = "0.5.16"
crossbeam-deque = "0.8.7"
crossbeam-utils = "0.8.22"
crossbeam-queue = "0.3.14"
crossbeam-channel = "0.5.17"
crossbeam-deque = "0.8.8"
crossbeam-utils = "0.8.23"
datafusion = { default-features = false, version = "55.0.0" }
derive_builder = "0.20.2"
enumset = "1.1.14"
@@ -306,7 +306,7 @@ rustfs-erasure-codec = { version = "8.0.2" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.6.0" }
redis = { version = "1.7.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
@@ -21,7 +21,7 @@
use super::common::{BoxError, OdmSourceSpec, OdmTestEnv, SeedObject};
use crate::fake_s3_target::{BucketMode, Operation};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use aws_sdk_s3::types::{BucketVersioningStatus, ObjectAttributes, VersioningConfiguration};
use bytes::Bytes;
use std::time::Duration;
@@ -103,11 +103,19 @@ async fn get_miss_pulls_inline_and_serves_locally_afterwards() -> TestResult {
#[tokio::test]
async fn get_large_object_streams_through_and_backfills_in_background() -> TestResult {
const PART_SIZE: usize = 5 * 1024 * 1024;
let bucket = "odm-get-large";
let env = configured_env(bucket, |spec| spec.policy.inline_max_bytes = 4096).await?;
let env = configured_env(bucket, |spec| {
spec.policy.inline_max_bytes = 4096;
spec.policy.multipart_part_size_bytes = u64::try_from(PART_SIZE).expect("part size fits in u64");
})
.await?;
let key = "large/archive.bin";
let body = payload(512 * 1024);
env.seed_source(SOURCE_BUCKET, &[SeedObject::new(key, body.clone())]);
let body = payload(PART_SIZE + 4096);
let etag = env
.seed_source(SOURCE_BUCKET, &[SeedObject::new(key, body.clone())])
.remove(0);
assert_eq!(etag.len(), 32, "the source fixture has a plain MD5 ETag");
let response = env.raw_get(bucket, key).await?;
assert_eq!(response.status, 200, "{}", String::from_utf8_lossy(&response.body));
@@ -125,6 +133,68 @@ async fn get_large_object_streams_through_and_backfills_in_background() -> TestR
vec![None, None],
"one passthrough GET plus one background pull, both unranged"
);
let source_requests = env.source.requests().len();
let second_part = env.client.get_object().bucket(bucket).key(key).part_number(2).send().await?;
assert_eq!(second_part.content_length(), Some(4096), "the completed second part is the tail");
assert_eq!(
second_part.content_range(),
Some(format!("bytes {PART_SIZE}-{}/{}", body.len() - 1, body.len()).as_str()),
"partNumber reads the stored multipart boundary"
);
assert_eq!(
second_part.body.collect().await?.into_bytes(),
body.slice(PART_SIZE..),
"the local second part contains the exact source tail"
);
let third_part = env
.client
.get_object()
.bucket(bucket)
.key(key)
.part_number(3)
.send()
.await
.expect_err("the completed object has exactly two parts");
assert_eq!(third_part.code(), Some("InvalidPart"));
let mut part_marker = None;
for (part_number, part_size) in [(1, PART_SIZE), (2, 4096)] {
let attributes = env
.client
.get_object_attributes()
.bucket(bucket)
.key(key)
.object_attributes(ObjectAttributes::ObjectParts)
.object_attributes(ObjectAttributes::Etag)
.max_parts(1)
.set_part_number_marker(part_marker.clone())
.send()
.await?;
assert_eq!(
attributes.e_tag().map(|value| value.trim_matches('"')),
Some(etag.as_str()),
"multipart write-back preserves the source MD5 ETag"
);
let parts = attributes
.object_parts()
.expect("RustFS must expose the stored multipart layout");
assert_eq!(parts.total_parts_count(), Some(2));
assert_eq!(parts.max_parts(), Some(1));
assert_eq!(parts.is_truncated(), Some(part_number == 1));
assert_eq!(parts.parts().len(), 1, "RustFS returns one stored part per requested page");
assert_eq!(parts.parts()[0].part_number(), Some(part_number));
assert_eq!(parts.parts()[0].size(), Some(i64::try_from(part_size).expect("part size fits in i64")));
part_marker = parts.next_part_number_marker().map(str::to_owned);
if part_number == 1 {
assert_eq!(part_marker.as_deref(), Some("1"), "the next request continues after the first part");
}
}
assert_eq!(
env.source.requests().len(),
source_requests,
"local part reads must not consult the source"
);
Ok(())
}
@@ -23,16 +23,20 @@
use super::common::{
ALLOW_LOOPBACK_SOURCE_ENV, AdminResponse, BackfillOp, BackfillRequest, BoxError, ODM_MODULE_SWITCH_ENV, ODM_SERVER_ENV,
OdmEnvOptions, OdmSourceSpec, OdmTestEnv, SeedObject, start_configured_env, start_configured_env_with,
OdmEnvOptions, OdmSourceSpec, OdmTestEnv, SeedObject, start_configured_env, start_configured_env_with, start_source_rustfs,
};
use crate::common::{RustFSTestEnvironment, replication_fast_env, signed_request};
use crate::fake_s3_target::{BucketMode, FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, Operation};
use crate::object_lock::common::put_object_lock_configuration;
use crate::replication_extension_test::{
ReplicationTargetOptions, enable_bucket_versioning, set_replication_target_with_options,
};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::types::{
BucketVersioningStatus, Event, FilterRule, FilterRuleName, NotificationConfiguration, NotificationConfigurationFilter,
ObjectLockRetentionMode, QueueConfiguration, S3KeyFilter, ServerSideEncryption, ServerSideEncryptionByDefault,
ServerSideEncryptionConfiguration, ServerSideEncryptionRule, Tag, Tagging, VersioningConfiguration,
ObjectAttributes, ObjectLockRetentionMode, QueueConfiguration, S3KeyFilter, ServerSideEncryption,
ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, Tag, Tagging,
VersioningConfiguration,
};
use bytes::Bytes;
use local_ip_address::local_ip;
@@ -580,6 +584,130 @@ async fn test_odm_pulled_object_replicates_and_target_as_source_is_rejected() ->
"a bucket may not migrate from its own replication target: {}",
rejected.body
);
Box::pin(assert_odm_multipart_replicates_to_rustfs(&env, bucket)).await?;
Ok(())
}
async fn assert_odm_multipart_replicates_to_rustfs(env: &OdmTestEnv, bucket: &str) -> TestResult {
const PART_SIZE: usize = 5 * 1024 * 1024;
let replica = start_source_rustfs().await?;
let replica_bucket = "odm-real-replica";
replica.create_test_bucket(replica_bucket).await?;
enable_bucket_versioning(&replica, replica_bucket).await?;
let arn = set_replication_target_with_options(
&env.rustfs,
bucket,
ReplicationTargetOptions {
endpoint: &replica.address,
access_key: &replica.access_key,
secret_key: &replica.secret_key,
target_bucket: replica_bucket,
secure: false,
skip_tls_verify: false,
ca_cert_pem: None,
},
)
.await?;
put_bucket_replication(&env.rustfs, bucket, &arn).await?;
let mut spec = env.fake_source_spec(SOURCE_BUCKET);
// Below the 16 MiB inline default the pull is one tee'd PUT with a single
// part; force the passthrough + background multipart write-back instead.
spec.policy.inline_max_bytes = 4096;
spec.policy.multipart_part_size_bytes = PART_SIZE as u64;
spec.policy.preserve_etag = true;
env.configure_and_wait(bucket, &spec).await?;
let key = "replicated/preserved-md5-multipart.bin";
let body = payload(PART_SIZE + 4096);
let source_put = env
.source_client()
.put_object()
.bucket(SOURCE_BUCKET)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from(body.clone()))
.send()
.await?;
let etag = source_put.e_tag().ok_or("source PUT omitted its ETag")?.trim_matches('"');
assert_eq!(etag.len(), 32, "the source must retain a single-PUT MD5 ETag");
assert!(etag.bytes().all(|byte| byte.is_ascii_hexdigit()));
let pulled = env.raw_get(bucket, key).await?;
assert_eq!(pulled.status, 200, "{}", String::from_utf8_lossy(&pulled.body));
assert_eq!(pulled.body, body);
assert!(env.wait_local_listed(bucket, key, SETTLE).await?, "the multipart pull must persist");
let deadline = Instant::now() + SETTLE;
let source_head = loop {
let head = env.client.head_object().bucket(bucket).key(key).send().await?;
match head.replication_status().map(|status| status.as_str()) {
Some("COMPLETED") => break head,
Some("FAILED") => return Err("the ODM multipart copy failed replication to RustFS".into()),
_ => {
assert!(Instant::now() < deadline, "the ODM multipart copy never completed replication to RustFS");
tokio::time::sleep(Duration::from_millis(200)).await;
}
}
};
let version = source_head
.version_id()
.ok_or("the versioned ODM copy omitted its version id")?;
assert_ne!(version, "null");
let replica_client = replica.create_s3_client();
for (client, object_bucket) in [(&env.client, bucket), (&replica_client, replica_bucket)] {
let attributes = client
.get_object_attributes()
.bucket(object_bucket)
.key(key)
.version_id(version)
.object_attributes(ObjectAttributes::Etag)
.object_attributes(ObjectAttributes::ObjectParts)
.send()
.await?;
assert_eq!(attributes.e_tag().map(|value| value.trim_matches('"')), Some(etag));
let parts = attributes
.object_parts()
.ok_or("the local copy and replica must both expose two parts")?;
assert_eq!(parts.total_parts_count(), Some(2));
assert_eq!(
parts
.parts()
.iter()
.map(|part| (part.part_number(), part.size()))
.collect::<Vec<_>>(),
[(Some(1), Some(PART_SIZE as i64)), (Some(2), Some(4096))]
);
}
// REPLICA status surfaces on HEAD, like the other inbound-replica checks.
let replica_head = replica_client
.head_object()
.bucket(replica_bucket)
.key(key)
.version_id(version)
.send()
.await?;
assert_eq!(replica_head.replication_status().map(|status| status.as_str()), Some("REPLICA"));
let replica_get = replica_client
.get_object()
.bucket(replica_bucket)
.key(key)
.version_id(version)
.send()
.await?;
assert_eq!(replica_get.version_id(), Some(version));
assert_eq!(replica_get.body.collect().await?.into_bytes(), body);
let boundary = replica_client
.get_object()
.bucket(replica_bucket)
.key(key)
.version_id(version)
.range(format!("bytes={}-{}", PART_SIZE - 32, PART_SIZE + 31))
.send()
.await?;
assert_eq!(boundary.body.collect().await?.into_bytes(), body.slice(PART_SIZE - 32..PART_SIZE + 32));
assert_eq!(
env.source.count_requests(Operation::GetObject, key),
2,
"one passthrough GET plus one background pull; replication and local reads must not fetch the migration source again"
);
Ok(())
}
@@ -3837,6 +3837,77 @@ async fn test_bucket_replication_converges_delete_marker_and_version_purge() ->
Ok(())
}
/// Regression for rustfs/backlog#2340 (not Wasabi specific): a directory
/// marker (`prefix/` with a body) in a versioned bucket is stored as the null
/// version, like MinIO (`putOpts`: "for directory objects skip creating new
/// versions"), and must still replicate to completion instead of staying
/// `PENDING`.
#[tokio::test]
async fn test_bucket_replication_replicates_directory_marker_in_versioned_bucket() -> TestResult {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
let mut source_env_vars = replication_fast_env();
source_env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_env.start_rustfs_server_with_env(vec![], &source_env_vars).await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env.start_rustfs_server_without_cleanup(vec![]).await?;
let source_bucket = "replication-dir-marker-src";
let target_bucket = "replication-dir-marker-dst";
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
target_client.create_bucket().bucket(target_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication(&source_env, source_bucket, &target_arn).await?;
let marker_key = "dir/trailing/";
let body = b"directory marker body";
let put = source_client
.put_object()
.bucket(source_bucket)
.key(marker_key)
.body(ByteStream::from_static(body))
.send()
.await?;
assert!(
put.version_id()
.is_none_or(|id| id == "null" || id == uuid::Uuid::nil().to_string()),
"a directory marker is the null version even in a versioned bucket: {:?}",
put.version_id()
);
wait_for_source_replication_status(&source_client, source_bucket, marker_key, "COMPLETED", false).await?;
let replica = target_client
.get_object()
.bucket(target_bucket)
.key(marker_key)
.send()
.await?;
assert_eq!(replica.body.collect().await?.into_bytes().as_ref(), body);
let listed = target_client
.list_object_versions()
.bucket(target_bucket)
.prefix(marker_key)
.send()
.await?;
let marker_versions: Vec<_> = listed.versions().iter().filter(|v| v.key() == Some(marker_key)).collect();
assert_eq!(marker_versions.len(), 1, "the marker must land exactly once: {marker_versions:?}");
assert_eq!(
marker_versions[0].version_id(),
Some("null"),
"the replica keeps the null version identity"
);
Ok(())
}
#[tokio::test]
async fn test_bucket_replication_disabled_delete_marker_does_not_propagate() -> TestResult {
init_logging();
@@ -31,17 +31,19 @@
//! Adding a target behavior the fleet has shown: add the mode to the fake
//! target, add a row here, and record any cell that is red before the fix.
use crate::common::{RustFSTestEnvironment, init_logging, replication_fast_env};
use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY};
use crate::common::{init_logging, replication_fast_env};
use crate::fake_s3_target::{BucketMode, FAKE_ACCESS_KEY, FAKE_SECRET_KEY};
use crate::fake_s3_target::{FakeS3Target, Operation as FakeTargetOperation, RequestRecord};
use crate::on_demand_migration::common::fake_source_client;
use crate::on_demand_migration::common::{OdmEnvOptions, OdmTestEnv, fake_source_client};
use crate::replication_extension_test::{
LOOPBACK_REPLICATION_TARGET_ENV, ReplicationTargetOptions, enable_bucket_versioning, put_bucket_replication,
set_replication_target_with_options,
};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::{ByteStream, DateTime};
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart, ObjectLockLegalHoldStatus, ObjectLockMode};
use aws_sdk_s3::types::{
Checksum, CompletedMultipartUpload, CompletedPart, ObjectAttributes, ObjectLockLegalHoldStatus, ObjectLockMode,
};
use bytes::Bytes;
use std::error::Error;
use std::time::{SystemTime, UNIX_EPOCH};
@@ -110,16 +112,19 @@ enum ObjectShape {
/// Two-part multipart upload with a GOVERNANCE retention period; the
/// lock headers travel on CreateMultipartUpload, which has no body.
LockedMultipart,
/// ODM stores two local parts while preserving a single-PUT source's MD5 ETag.
OdmPreservedMd5Multipart,
}
impl ObjectShape {
const ALL: [ObjectShape; 6] = [
const ALL: [ObjectShape; 7] = [
ObjectShape::Empty,
ObjectShape::Plain,
ObjectShape::Retention,
ObjectShape::LegalHold,
ObjectShape::Multipart,
ObjectShape::LockedMultipart,
ObjectShape::OdmPreservedMd5Multipart,
];
fn key(self) -> &'static str {
@@ -130,6 +135,7 @@ impl ObjectShape {
ObjectShape::LegalHold => "matrix/legal-hold.bin",
ObjectShape::Multipart => "matrix/multipart.bin",
ObjectShape::LockedMultipart => "matrix/locked-multipart.bin",
ObjectShape::OdmPreservedMd5Multipart => "matrix/odm-preserved-md5.bin",
}
}
@@ -139,7 +145,8 @@ impl ObjectShape {
/// Upload the shape to the source and return the bytes the target must
/// end up holding.
async fn put(self, client: &Client, bucket: &str) -> Result<Bytes, Box<dyn Error + Send + Sync>> {
async fn put(self, env: &OdmTestEnv, bucket: &str) -> Result<Bytes, Box<dyn Error + Send + Sync>> {
let client = &env.client;
let key = self.key();
match self {
ObjectShape::Empty => {
@@ -190,6 +197,7 @@ impl ObjectShape {
}
ObjectShape::Multipart => multipart_put(client, bucket, key, 0x44, false).await,
ObjectShape::LockedMultipart => multipart_put(client, bucket, key, 0x55, true).await,
ObjectShape::OdmPreservedMd5Multipart => odm_preserved_md5_multipart(env, bucket, key).await,
}
}
}
@@ -270,11 +278,15 @@ async fn run_row(mode: TargetMode) -> TestResult {
target.create_bucket_with_object_lock(target_bucket.clone());
mode.apply(&target);
let mut source_env = RustFSTestEnvironment::new().await?;
let mut env_vars = replication_fast_env();
env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
env_vars.extend_from_slice(&[("NO_PROXY", "127.0.0.1,localhost"), ("HTTP_PROXY", ""), ("HTTPS_PROXY", "")]);
source_env.start_rustfs_server_with_env(vec![], &env_vars).await?;
let env = OdmTestEnv::start_with(OdmEnvOptions {
env: env_vars,
..OdmEnvOptions::default()
})
.await?;
let source_env = &env.rustfs;
let source_bucket = format!("matrix-{}-src", mode.slug());
let source_client = source_env.create_s3_client();
@@ -284,9 +296,9 @@ async fn run_row(mode: TargetMode) -> TestResult {
.object_lock_enabled_for_bucket(true)
.send()
.await?;
enable_bucket_versioning(&source_env, &source_bucket).await?;
enable_bucket_versioning(source_env, &source_bucket).await?;
let target_arn = set_replication_target_with_options(
&source_env,
source_env,
&source_bucket,
ReplicationTargetOptions {
endpoint: &target.address(),
@@ -299,14 +311,21 @@ async fn run_row(mode: TargetMode) -> TestResult {
},
)
.await?;
put_bucket_replication(&source_env, &source_bucket, &target_arn).await?;
put_bucket_replication(source_env, &source_bucket, &target_arn).await?;
let target_client = fake_source_client(&target);
let mut failures = Vec::new();
for shape in ObjectShape::ALL {
let cell = format!("{}/{:?}", mode.slug(), shape);
let expected_body = shape.put(&source_client, &source_bucket).await?;
let expected_body = shape.put(&env, &source_bucket).await?;
let status = wait_for_terminal_replication_status(&source_client, &source_bucket, shape.key()).await?;
if shape == ObjectShape::OdmPreservedMd5Multipart {
assert_eq!(
env.source.count_requests(FakeTargetOperation::GetObject, shape.key()),
2,
"one passthrough GET plus one background pull; replication must read the persisted local parts"
);
}
let journal = target.requests();
let outcome = match expectation(mode, shape) {
Expectation::Completed => {
@@ -379,6 +398,36 @@ async fn check_completed_cell(
if uploads.is_empty() {
return Err("no upload reached the target although the source reports COMPLETED".into());
}
if shape == ObjectShape::OdmPreservedMd5Multipart {
let key_requests: Vec<_> = journal
.iter()
.filter(|record| record.key.as_deref() == Some(shape.key()))
.collect();
for operation in [
FakeTargetOperation::CreateMultipartUpload,
FakeTargetOperation::CompleteMultipartUpload,
] {
if !key_requests.iter().any(|record| record.operation == operation) {
return Err(format!("preserved-MD5 multipart object did not use {operation:?}").into());
}
}
if key_requests
.iter()
.any(|record| record.operation == FakeTargetOperation::PutObject)
{
return Err("preserved-MD5 multipart object used a single PutObject".into());
}
let mut part_numbers: Vec<_> = key_requests
.iter()
.filter(|record| record.operation == FakeTargetOperation::UploadPart)
.map(|record| record.part_number)
.collect();
part_numbers.sort_unstable();
part_numbers.dedup();
if part_numbers != [Some(1), Some(2)] {
return Err(format!("preserved-MD5 multipart object uploaded unexpected parts: {part_numbers:?}").into());
}
}
if let Some(framed) = uploads.iter().find(|record| record.transport.aws_chunked) {
return Err(format!("{cell}: an upload went out aws-chunked (rustfs#6853 framing): {framed:?}").into());
}
@@ -455,6 +504,71 @@ async fn wait_for_terminal_replication_status(
}
}
async fn odm_preserved_md5_multipart(env: &OdmTestEnv, bucket: &str, key: &str) -> Result<Bytes, Box<dyn Error + Send + Sync>> {
const PART_SIZE: usize = 5 * 1024 * 1024;
let origin_bucket = format!("{bucket}-origin");
env.source.create_bucket_with_mode(&origin_bucket, BucketMode::Unversioned);
let mut spec = env.fake_source_spec(&origin_bucket);
// Below the 16 MiB inline default the pull is one tee'd PUT with a single
// part; force the passthrough + background multipart write-back instead.
spec.policy.inline_max_bytes = 4096;
spec.policy.multipart_part_size_bytes = PART_SIZE as u64;
spec.policy.preserve_etag = true;
env.configure_and_wait(bucket, &spec).await?;
// A normal source PUT produces the MD5 ETag; only ODM chooses the local parts.
let body = payload(PART_SIZE + 4096, 0x66);
let source_put = env
.source_client()
.put_object()
.bucket(&origin_bucket)
.key(key)
.body(ByteStream::from(body.clone()))
.send()
.await?;
let source_etag = source_put.e_tag().ok_or("source PUT omitted its ETag")?.trim_matches('"');
assert_eq!(source_etag.len(), 32, "source fixture must have a single-PUT MD5 ETag");
assert!(source_etag.bytes().all(|byte| byte.is_ascii_hexdigit()));
let pulled = env.raw_get(bucket, key).await?;
assert_eq!(pulled.status, 200, "{}", String::from_utf8_lossy(&pulled.body));
assert_eq!(pulled.body, body);
assert!(
env.wait_local_listed(bucket, key, Duration::from_secs(30)).await?,
"ODM must persist the object"
);
let attributes = env
.client
.get_object_attributes()
.bucket(bucket)
.key(key)
.object_attributes(ObjectAttributes::Etag)
.object_attributes(ObjectAttributes::ObjectParts)
.object_attributes(ObjectAttributes::Checksum)
.send()
.await?;
assert_eq!(attributes.e_tag().map(|etag| etag.trim_matches('"')), Some(source_etag));
let parts = attributes
.object_parts()
.ok_or("the ODM copy must expose its two local parts")?;
assert_eq!(parts.total_parts_count(), Some(2));
assert_eq!(
parts
.parts()
.iter()
.map(|part| (part.part_number(), part.size()))
.collect::<Vec<_>>(),
[(Some(1), Some(PART_SIZE as i64)), (Some(2), Some(4096))]
);
assert!(
attributes
.checksum()
.is_none_or(|checksum| checksum == &Checksum::builder().build()),
"multipart routing must work without an object checksum record"
);
Ok(body)
}
async fn multipart_put(
client: &Client,
bucket: &str,
+22 -3
View File
@@ -76,6 +76,22 @@ pub mod bucket {
};
}
pub mod recovery_disposition {
pub use crate::bucket::lifecycle::recovery_disposition::{
CreatedIlmRecoveryDisposition, IlmRecoveryDisposition, IlmRecoveryDispositionAction, IlmRecoveryDispositionError,
IlmRecoveryDispositionIdentity, IlmRecoveryDispositionOwnerLease, IlmRecoveryDispositionReasonCode,
IlmRecoveryDispositionState, ObservedIlmRecoveryDisposition, create_recovery_disposition_if_absent,
load_recovery_disposition, recovery_disposition_id, save_recovery_disposition_if_current,
};
}
pub mod recovery_export {
pub use crate::bucket::lifecycle::recovery_export::{
IlmRecoveryExportCreated, IlmRecoveryExportObservation, create_recovery_export,
inspect_recovery_export_observation, load_recovery_export,
};
}
pub mod transition_transaction {
pub use crate::bucket::lifecycle::transition_transaction::{
TransitionOperatorDeleteResult, TransitionOperatorError, TransitionOperatorProbe, TransitionOperatorStatus,
@@ -293,7 +309,7 @@ pub mod cache {
pub mod capacity {
pub use crate::core::pools::{
DecommissionUnresolvedEntry, PoolDecommissionInfo, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
path2_bucket_object, path2_bucket_object_with_base_path,
is_pool_activation_fleet_proof_error, path2_bucket_object, path2_bucket_object_with_base_path,
};
pub use crate::store::utils::is_reserved_or_invalid_bucket;
}
@@ -446,9 +462,12 @@ pub mod notification {
#[cfg(any(test, feature = "test-util"))]
pub use crate::services::notification_sys::rotate_cross_pool_fence_fleet_proof_for_test;
pub use crate::services::notification_sys::{
ClusterTierDailyStats, CrossPoolFenceFleetProofToken, LegacyTransitionStateReconcileFleetProofToken, NotificationPeerErr,
NotificationSys, ScannerPublicationLeaseGrant, acquire_cross_pool_fence_fleet_proof,
ClusterTierDailyStats, CrossPoolFenceFleetProofToken, IlmRecoveryExportFleetProofToken,
LegacyTransitionStateReconcileFleetProofToken, NotificationPeerErr, NotificationSys, ScannerPublicationLeaseGrant,
acquire_cross_pool_fence_fleet_proof, acquire_ilm_recovery_export_fleet_proof,
acquire_legacy_transition_state_reconcile_fleet_proof, cross_pool_fence_fleet_proof_matches, get_global_notification_sys,
ilm_recovery_export_fleet_proof_matches, ilm_recovery_export_local_process_epoch,
ilm_recovery_export_member_epochs_sha256, ilm_recovery_export_topology_generation,
legacy_transition_state_reconcile_fleet_proof_matches, new_global_notification_sys,
scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe,
};
@@ -41,6 +41,21 @@ where
com::read_config(api, file).await
}
pub(crate) async fn read_config_limited_preserve_empty<S>(api: Arc<S>, file: &str, max_bytes: usize) -> Result<Vec<u8>>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
com::read_config_limited_preserve_empty(api, file, max_bytes).await
}
pub(crate) async fn read_config_with_metadata<S>(api: Arc<S>, file: &str, opts: &ObjectOptions) -> Result<(Vec<u8>, ObjectInfo)>
where
S: ObjectIO<
@@ -56,6 +71,26 @@ where
com::read_config_with_metadata(api, file, opts).await
}
pub(crate) async fn read_config_limited_preserve_empty_with_metadata<S>(
api: Arc<S>,
file: &str,
opts: &ObjectOptions,
max_bytes: usize,
) -> Result<(Vec<u8>, ObjectInfo)>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
com::read_config_limited_preserve_empty_with_metadata_opts(api, file, opts, max_bytes).await
}
pub(crate) async fn save_config<S>(api: Arc<S>, file: &str, data: Vec<u8>) -> Result<()>
where
S: ObjectIO<
@@ -22,7 +22,7 @@ use super::{
bucket_lifecycle_ops::{
ManualTransitionQueueSnapshot, ManualTransitionRunReport, decode_manual_transition_continuation_token,
},
manual_transition_job, recovery_control, tier_delete_journal, transition_transaction,
manual_transition_job, recovery_control, recovery_disposition, recovery_export, tier_delete_journal, transition_transaction,
};
use crate::error::{Error, Result};
use crate::services::tier::tier_probe_intent;
@@ -42,6 +42,8 @@ pub(crate) enum DurableIlmRecordKind {
ManualTransitionTask,
ManualTransitionWorkerResult,
RecoveryControl,
RecoveryExport,
RecoveryDisposition,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -112,8 +114,20 @@ pub(crate) const RECOVERY_CONTROL_NAMESPACE: DurableIlmNamespace = DurableIlmNam
max_record_size: recovery_control::MAX_ILM_RECOVERY_CONTROL_SIZE,
kind: DurableIlmRecordKind::RecoveryControl,
};
pub(crate) const RECOVERY_EXPORT_NAMESPACE: DurableIlmNamespace = DurableIlmNamespace {
name: "recovery-export",
prefix: recovery_export::ILM_RECOVERY_EXPORT_PREFIX,
max_record_size: recovery_export::MAX_ILM_RECOVERY_EXPORT_SIZE,
kind: DurableIlmRecordKind::RecoveryExport,
};
pub(crate) const RECOVERY_DISPOSITION_NAMESPACE: DurableIlmNamespace = DurableIlmNamespace {
name: "recovery-disposition",
prefix: recovery_disposition::ILM_RECOVERY_DISPOSITION_PREFIX,
max_record_size: recovery_disposition::MAX_ILM_RECOVERY_DISPOSITION_SIZE,
kind: DurableIlmRecordKind::RecoveryDisposition,
};
pub(crate) const DURABLE_ILM_NAMESPACES: [DurableIlmNamespace; 10] = [
pub(crate) const DURABLE_ILM_NAMESPACES: [DurableIlmNamespace; 12] = [
TIER_DELETE_JOURNAL_NAMESPACE,
TIER_DELETE_JOURNAL_V6_NAMESPACE,
TIER_DELETE_DISPATCH_MANIFEST_NAMESPACE,
@@ -124,6 +138,8 @@ pub(crate) const DURABLE_ILM_NAMESPACES: [DurableIlmNamespace; 10] = [
MANUAL_TRANSITION_TASK_NAMESPACE,
MANUAL_TRANSITION_WORKER_RESULT_NAMESPACE,
RECOVERY_CONTROL_NAMESPACE,
RECOVERY_EXPORT_NAMESPACE,
RECOVERY_DISPOSITION_NAMESPACE,
];
#[derive(Debug, Clone, PartialEq, Eq)]
@@ -261,6 +277,28 @@ pub(crate) enum DurableIlmRecordCheckpoint {
#[serde(default, skip_serializing_if = "Option::is_none")]
owner_fence_sha256: Option<String>,
},
RecoveryExport {
content_sha256: String,
source_generation_sha256: String,
topology_generation: String,
member_epochs_sha256: String,
creator_sha256: String,
retain_until_unix_nanos: i64,
},
RecoveryDisposition {
content_sha256: String,
identity_sha256: String,
copy_manifest_sha256: String,
copy_manifest_count: usize,
created_at_unix_nanos: i64,
revision: u64,
state: recovery_disposition::IlmRecoveryDispositionState,
owner_fence_sha256: Option<String>,
owner_lease_acquired_at_unix_nanos: Option<i64>,
owner_lease_expires_at_unix_nanos: Option<i64>,
confirmed_absent_sha256: Vec<String>,
retain_until_unix_nanos: i64,
},
}
impl DurableIlmRecordCheckpoint {
@@ -275,7 +313,9 @@ impl DurableIlmRecordCheckpoint {
| Self::ManualTransitionScope { content_sha256, .. }
| Self::ManualTransitionTask { content_sha256 }
| Self::ManualTransitionWorkerResult { content_sha256 }
| Self::RecoveryControl { content_sha256, .. } => content_sha256,
| Self::RecoveryControl { content_sha256, .. }
| Self::RecoveryExport { content_sha256, .. }
| Self::RecoveryDisposition { content_sha256, .. } => content_sha256,
}
}
@@ -316,6 +356,9 @@ impl DurableIlmRecordCheckpoint {
{
return Err(Error::other("durable ILM tier delete journal checkpoint is invalid"));
}
if !recovery_disposition_checkpoint_is_valid(checkpoint) {
return Err(Error::other("durable ILM recovery disposition checkpoint is invalid"));
}
}
if self == next {
if let Self::ManualTransitionJob {
@@ -594,6 +637,83 @@ impl DurableIlmRecordCheckpoint {
&& previous_attempts == next_attempts;
adjacent && (claim || source_refresh || completion)
}
(
Self::RecoveryDisposition {
identity_sha256: previous_identity,
copy_manifest_sha256: previous_manifest,
copy_manifest_count: previous_manifest_count,
created_at_unix_nanos: previous_created_at,
revision: previous_revision,
state: previous_state,
owner_fence_sha256: previous_owner,
owner_lease_acquired_at_unix_nanos: previous_owner_acquired,
owner_lease_expires_at_unix_nanos: previous_owner_expires,
confirmed_absent_sha256: previous_confirmed,
retain_until_unix_nanos: previous_retain_until,
..
},
Self::RecoveryDisposition {
identity_sha256: next_identity,
copy_manifest_sha256: next_manifest,
copy_manifest_count: next_manifest_count,
created_at_unix_nanos: next_created_at,
revision: next_revision,
state: next_state,
owner_fence_sha256: next_owner,
owner_lease_acquired_at_unix_nanos: next_owner_acquired,
owner_lease_expires_at_unix_nanos: next_owner_expires,
confirmed_absent_sha256: next_confirmed,
retain_until_unix_nanos: next_retain_until,
..
},
) => {
use recovery_disposition::IlmRecoveryDispositionState::{Applying, Completed, Prepared};
let immutable_identity_matches = previous_identity == next_identity
&& previous_manifest == next_manifest
&& previous_manifest_count == next_manifest_count
&& previous_created_at == next_created_at
&& previous_retain_until == next_retain_until;
let adjacent = previous_revision.checked_add(1) == Some(*next_revision);
let progress_is_monotonic = sorted_sha256_set_is_subset(previous_confirmed, next_confirmed);
let legal_edge = match (previous_state, next_state) {
(Prepared, Prepared) => {
let claim = previous_owner.is_none() && next_owner.is_some();
let takeover = previous_owner.is_some()
&& previous_owner != next_owner
&& previous_owner_expires
.zip(*next_owner_acquired)
.is_some_and(|(expires, acquired)| acquired >= expires);
previous_confirmed == next_confirmed && (claim || takeover)
}
(Prepared, Applying) => {
previous_confirmed == next_confirmed
&& previous_owner.is_some()
&& previous_owner == next_owner
&& previous_owner_acquired == next_owner_acquired
&& previous_owner_expires == next_owner_expires
}
(Applying, Applying) => {
let progress = previous_owner == next_owner
&& previous_owner_acquired == next_owner_acquired
&& previous_owner_expires == next_owner_expires
&& previous_confirmed.len().checked_add(1) == Some(next_confirmed.len());
let takeover = previous_owner.is_some()
&& previous_owner != next_owner
&& previous_confirmed == next_confirmed
&& previous_owner_expires
.zip(*next_owner_acquired)
.is_some_and(|(expires, acquired)| acquired >= expires);
progress || takeover
}
(Applying, Completed) => {
previous_owner.is_some() && next_owner.is_none() && previous_confirmed == next_confirmed
}
_ => false,
};
immutable_identity_matches && adjacent && progress_is_monotonic && legal_edge
}
_ => false,
};
@@ -611,6 +731,11 @@ impl DurableIlmRecordCheckpoint {
/// after the exact terminal ETag and terminal receipt were committed, to
/// purge older object versions exposed by that deletion.
pub(crate) fn is_predecessor_of_terminal(&self, terminal: &Self) -> bool {
for checkpoint in [self, terminal] {
if !recovery_disposition_checkpoint_is_valid(checkpoint) {
return false;
}
}
if let Self::TierProbeIntent { state, .. } = terminal
&& !matches!(
state,
@@ -627,6 +752,11 @@ impl DurableIlmRecordCheckpoint {
{
return false;
}
if let Self::RecoveryDisposition { state, .. } = terminal
&& state != &recovery_disposition::IlmRecoveryDispositionState::Completed
{
return false;
}
if self == terminal || self.validate_successor(terminal).is_ok() {
return true;
}
@@ -752,11 +882,121 @@ impl DurableIlmRecordCheckpoint {
&& terminal_revision > previous_revision
&& terminal_attempts >= previous_attempts
}
(
Self::RecoveryDisposition {
identity_sha256: previous_identity,
copy_manifest_sha256: previous_manifest,
copy_manifest_count: previous_manifest_count,
created_at_unix_nanos: previous_created_at,
revision: previous_revision,
state: previous_state,
owner_fence_sha256: previous_owner,
confirmed_absent_sha256: previous_confirmed,
retain_until_unix_nanos: previous_retain_until,
..
},
Self::RecoveryDisposition {
identity_sha256: terminal_identity,
copy_manifest_sha256: terminal_manifest,
copy_manifest_count: terminal_manifest_count,
created_at_unix_nanos: terminal_created_at,
revision: terminal_revision,
state: recovery_disposition::IlmRecoveryDispositionState::Completed,
confirmed_absent_sha256: terminal_confirmed,
retain_until_unix_nanos: terminal_retain_until,
..
},
) => {
matches!(
previous_state,
recovery_disposition::IlmRecoveryDispositionState::Prepared
| recovery_disposition::IlmRecoveryDispositionState::Applying
) && previous_identity == terminal_identity
&& previous_manifest == terminal_manifest
&& previous_manifest_count == terminal_manifest_count
&& previous_created_at == terminal_created_at
&& previous_retain_until == terminal_retain_until
&& terminal_revision.checked_sub(*previous_revision).is_some_and(|distance| {
let minimum_distance = match previous_state {
recovery_disposition::IlmRecoveryDispositionState::Prepared if previous_owner.is_some() => 3,
recovery_disposition::IlmRecoveryDispositionState::Prepared => 4,
recovery_disposition::IlmRecoveryDispositionState::Applying
if previous_confirmed.len() == *previous_manifest_count =>
{
1
}
recovery_disposition::IlmRecoveryDispositionState::Applying => 2,
recovery_disposition::IlmRecoveryDispositionState::Completed => u64::MAX,
};
distance >= minimum_distance
})
&& sorted_sha256_set_is_subset(previous_confirmed, terminal_confirmed)
}
_ => false,
}
}
}
fn recovery_disposition_checkpoint_is_valid(checkpoint: &DurableIlmRecordCheckpoint) -> bool {
use recovery_disposition::IlmRecoveryDispositionState::{Applying, Completed, Prepared};
let DurableIlmRecordCheckpoint::RecoveryDisposition {
content_sha256,
identity_sha256,
copy_manifest_sha256,
copy_manifest_count,
created_at_unix_nanos,
revision,
state,
owner_fence_sha256,
owner_lease_acquired_at_unix_nanos,
owner_lease_expires_at_unix_nanos,
confirmed_absent_sha256,
retain_until_unix_nanos,
} = checkpoint
else {
return true;
};
let owner_fence_sha256 = owner_fence_sha256.as_deref();
is_canonical_sha256(content_sha256)
&& is_canonical_sha256(identity_sha256)
&& is_canonical_sha256(copy_manifest_sha256)
&& *copy_manifest_count > 0
&& *created_at_unix_nanos > 0
&& *revision > 0
&& *retain_until_unix_nanos > 0
&& owner_fence_sha256.is_none_or(is_canonical_sha256)
&& match (
owner_fence_sha256,
*owner_lease_acquired_at_unix_nanos,
*owner_lease_expires_at_unix_nanos,
) {
(None, None, None) => true,
(Some(_), Some(acquired), Some(expires)) => acquired > 0 && expires > acquired,
_ => false,
}
&& confirmed_absent_sha256.len() <= *copy_manifest_count
&& confirmed_absent_sha256.iter().all(|digest| is_canonical_sha256(digest))
&& confirmed_absent_sha256.windows(2).all(|pair| pair[0] < pair[1])
&& match *state {
Prepared => confirmed_absent_sha256.is_empty(),
Applying => owner_fence_sha256.is_some(),
Completed => owner_fence_sha256.is_none() && confirmed_absent_sha256.len() == *copy_manifest_count,
}
}
fn is_canonical_sha256(value: &str) -> bool {
is_sha256_checksum(value)
&& !value
.bytes()
.any(|byte| byte.is_ascii_hexdigit() && byte.is_ascii_uppercase())
}
fn sorted_sha256_set_is_subset(subset: &[String], superset: &[String]) -> bool {
subset.iter().all(|candidate| superset.binary_search(candidate).is_ok())
}
fn tier_delete_dispatch_parent_progress_delta(
previous_sequence: u64,
previous_completed_journals: u64,
@@ -1348,6 +1588,55 @@ pub(crate) fn validate_durable_ilm_record(path: &str, data: &[u8]) -> Result<Val
},
)
}
DurableIlmRecordKind::RecoveryExport => {
let (protocol, export_id) = recovery_export::recovery_export_id_from_record_object_name(path)?;
let export = recovery_export::IlmRecoveryExport::decode(&export_id, data)?;
let canonical = recovery_export::recovery_export_record_object_name(protocol, &export_id)?;
if canonical != path || export.protocol != protocol {
return Err(Error::other("ILM recovery export path is not canonical"));
}
let source_generation_sha256 = checkpoint_hash(&export.source_generation)?;
(
"export_id",
export_id,
DurableIlmRecordCheckpoint::RecoveryExport {
content_sha256,
source_generation_sha256,
topology_generation: export.topology_generation,
member_epochs_sha256: export.member_epochs_sha256,
creator_sha256: export.creator_sha256,
retain_until_unix_nanos: export.retain_until_unix_nanos,
},
)
}
DurableIlmRecordKind::RecoveryDisposition => {
// The disposition module owns strict schema, checksum, canonical
// path, immutable-manifest, and state-specific validation. Keep
// this boundary limited to decommission identity/checkpoint
// projection so the two readers cannot accept different records.
let disposition = recovery_disposition::decode_recovery_disposition_checkpoint(path, data)?;
if disposition.content_sha256 != content_sha256 {
return Err(Error::other("ILM recovery disposition checkpoint content digest is invalid"));
}
(
"disposition_id",
disposition.disposition_id,
DurableIlmRecordCheckpoint::RecoveryDisposition {
content_sha256: disposition.content_sha256,
identity_sha256: disposition.identity_sha256,
copy_manifest_sha256: disposition.copy_manifest_sha256,
copy_manifest_count: disposition.copy_manifest_count,
created_at_unix_nanos: disposition.created_at_unix_nanos,
revision: disposition.revision,
state: disposition.state,
owner_fence_sha256: disposition.owner_fence_sha256,
owner_lease_acquired_at_unix_nanos: disposition.owner_lease_acquired_at_unix_nanos,
owner_lease_expires_at_unix_nanos: disposition.owner_lease_expires_at_unix_nanos,
confirmed_absent_sha256: disposition.confirmed_absent_sha256,
retain_until_unix_nanos: disposition.retain_until_unix_nanos,
},
)
}
DurableIlmRecordKind::ManualTransitionJob => {
let job_id = manual_transition_job::manual_transition_job_id_from_record_object_name(path)
.map_err(|err| Error::other(err.to_string()))?;
@@ -1503,6 +1792,203 @@ mod tests {
}
}
fn recovery_disposition_checkpoint(
revision: u64,
state: recovery_disposition::IlmRecoveryDispositionState,
owner_fence: Option<&str>,
confirmed_absent_sha256: Vec<String>,
) -> DurableIlmRecordCheckpoint {
let (owner_lease_acquired_at_unix_nanos, owner_lease_expires_at_unix_nanos) = match owner_fence {
Some("f") => (Some(10), Some(20)),
Some(_) => (Some(1), Some(10)),
None => (None, None),
};
DurableIlmRecordCheckpoint::RecoveryDisposition {
content_sha256: format!("{revision:064x}"),
identity_sha256: "a".repeat(64),
copy_manifest_sha256: "d".repeat(64),
copy_manifest_count: 2,
created_at_unix_nanos: 1_700_000_000_000_000_000,
revision,
state,
owner_fence_sha256: owner_fence.map(|digest| digest.repeat(64)),
owner_lease_acquired_at_unix_nanos,
owner_lease_expires_at_unix_nanos,
confirmed_absent_sha256,
retain_until_unix_nanos: 1_820_000_000_000_000_000,
}
}
#[test]
fn recovery_disposition_namespace_is_registered_without_shadowing_its_root() {
let disposition_id = "a".repeat(64);
let path = format!(
"{}/tier_delete_journal/{}/{}/{}.json",
recovery_disposition::ILM_RECOVERY_DISPOSITION_PREFIX,
&disposition_id[..2],
&disposition_id[2..4],
disposition_id
);
let namespace = classify_durable_ilm_record(&path)
.expect("recovery disposition path should classify")
.expect("recovery disposition should be durable");
assert_eq!(namespace, &RECOVERY_DISPOSITION_NAMESPACE);
assert!(classify_durable_ilm_record(recovery_disposition::ILM_RECOVERY_DISPOSITION_PREFIX).is_err());
}
#[test]
fn recovery_disposition_checkpoint_accepts_only_monotonic_progress() {
use recovery_disposition::IlmRecoveryDispositionState::{Applying, Completed, Prepared};
let first_copy = "b".repeat(64);
let second_copy = "c".repeat(64);
let prepared = recovery_disposition_checkpoint(1, Prepared, None, Vec::new());
let claimed = recovery_disposition_checkpoint(2, Prepared, Some("e"), Vec::new());
let applying = recovery_disposition_checkpoint(3, Applying, Some("e"), Vec::new());
let first_absent = recovery_disposition_checkpoint(4, Applying, Some("e"), vec![first_copy.clone()]);
let taken_over = recovery_disposition_checkpoint(5, Applying, Some("f"), vec![first_copy.clone()]);
let all_absent = recovery_disposition_checkpoint(6, Applying, Some("f"), vec![first_copy.clone(), second_copy.clone()]);
let completed = recovery_disposition_checkpoint(7, Completed, None, vec![first_copy.clone(), second_copy.clone()]);
prepared
.validate_successor(&claimed)
.expect("Prepared should record an owner claim without absence progress");
claimed
.validate_successor(&applying)
.expect("Prepared should advance to Applying without folding in deletion progress");
applying
.validate_successor(&first_absent)
.expect("Applying should append newly confirmed absent copies");
first_absent
.validate_successor(&taken_over)
.expect("Applying should record a fenced owner takeover without losing progress");
taken_over
.validate_successor(&all_absent)
.expect("Applying should preserve every earlier confirmation while making progress");
all_absent
.validate_successor(&completed)
.expect("a fully confirmed manifest should advance to Completed");
assert!(
prepared.validate_successor(&completed).is_err(),
"adjacent receipt updates must not skip Applying"
);
assert!(
first_absent
.validate_successor(&recovery_disposition_checkpoint(5, Applying, Some("e"), Vec::new()))
.is_err(),
"confirmed-absent progress must not move backwards"
);
assert!(
applying
.validate_successor(&recovery_disposition_checkpoint(4, Completed, None, vec![first_copy.clone()]))
.is_err(),
"Completed must cover the complete immutable copy manifest"
);
assert!(
completed
.validate_successor(&recovery_disposition_checkpoint(7, Applying, Some("e"), vec![second_copy]))
.is_err(),
"Completed is terminal"
);
assert!(
first_absent
.validate_successor(&recovery_disposition_checkpoint(5, Applying, Some("e"), vec![first_copy.clone()]))
.is_err(),
"a same-state revision bump must change the owner fence or absence progress"
);
assert!(
applying
.validate_successor(&recovery_disposition_checkpoint(4, Applying, None, vec![first_copy]))
.is_err(),
"Applying must retain a fenced owner"
);
let mut noncanonical_identity = claimed.clone();
if let DurableIlmRecordCheckpoint::RecoveryDisposition { identity_sha256, .. } = &mut noncanonical_identity {
*identity_sha256 = "A".repeat(64);
}
assert!(prepared.validate_successor(&noncanonical_identity).is_err());
let mut changed_created_at = claimed;
if let DurableIlmRecordCheckpoint::RecoveryDisposition {
created_at_unix_nanos, ..
} = &mut changed_created_at
{
*created_at_unix_nanos += 1;
}
assert!(prepared.validate_successor(&changed_created_at).is_err());
let mut early_takeover = taken_over;
if let DurableIlmRecordCheckpoint::RecoveryDisposition {
owner_lease_acquired_at_unix_nanos,
..
} = &mut early_takeover
{
*owner_lease_acquired_at_unix_nanos = Some(9);
}
assert!(first_absent.validate_successor(&early_takeover).is_err());
assert!(
applying
.validate_successor(&recovery_disposition_checkpoint(
4,
Applying,
Some("e"),
vec!["c".repeat(64), "b".repeat(64)],
))
.is_err(),
"confirmed-absent entries must be a canonical sorted set"
);
}
#[test]
fn recovery_disposition_terminal_predecessor_requires_exact_identity_and_full_manifest() {
use recovery_disposition::IlmRecoveryDispositionState::{Applying, Completed, Prepared};
let first_copy = "b".repeat(64);
let second_copy = "c".repeat(64);
let prepared = recovery_disposition_checkpoint(1, Prepared, None, Vec::new());
let applying = recovery_disposition_checkpoint(3, Applying, Some("e"), vec![first_copy.clone()]);
let completed = recovery_disposition_checkpoint(5, Completed, None, vec![first_copy.clone(), second_copy]);
assert!(prepared.is_predecessor_of_terminal(&completed));
assert!(applying.is_predecessor_of_terminal(&completed));
assert!(
!prepared.is_predecessor_of_terminal(&recovery_disposition_checkpoint(2, Applying, Some("e"), Vec::new())),
"a nonterminal disposition must not authorize terminal cleanup"
);
assert!(
!prepared.is_predecessor_of_terminal(&recovery_disposition_checkpoint(
4,
Completed,
None,
vec![first_copy.clone(), "c".repeat(64)],
)),
"terminal proof must leave enough revisions for claim, apply, progress, and completion"
);
assert!(
!applying.is_predecessor_of_terminal(&recovery_disposition_checkpoint(
4,
Completed,
None,
vec![first_copy.clone(), "c".repeat(64)],
)),
"an incomplete Applying checkpoint cannot complete without a progress generation"
);
let mut other_identity = completed;
if let DurableIlmRecordCheckpoint::RecoveryDisposition { identity_sha256, .. } = &mut other_identity {
*identity_sha256 = "e".repeat(64);
}
assert!(!prepared.is_predecessor_of_terminal(&other_identity));
let incomplete_terminal = recovery_disposition_checkpoint(4, Completed, None, vec![first_copy]);
assert!(
!prepared.is_predecessor_of_terminal(&incomplete_terminal),
"a partial confirmed-absent set must not become terminal proof"
);
}
fn tier_probe_intent_fixture() -> tier_probe_intent::TierProbeIntent {
let probe_id = Uuid::parse_str("36e2220e-9ad2-495b-b3bc-c4d2caf70a31").expect("fixture uuid should parse");
tier_probe_intent::TierProbeIntent {
@@ -25,6 +25,8 @@ mod object_handlers_common;
mod object_lock_boundary;
pub use self::core as lifecycle;
pub mod recovery_control;
pub mod recovery_disposition;
pub mod recovery_export;
mod replication_sink;
pub mod rule;
mod runtime_boundary;
@@ -168,7 +168,7 @@ impl IlmRecoverySourceGeneration {
Ok(generation)
}
fn validate(&self) -> Result<()> {
pub(crate) fn validate(&self) -> Result<()> {
if self.source_schema.trim().is_empty() {
return Err(IlmRecoveryControlError::Corrupt("source schema is empty"));
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,840 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::{collections::HashSet, sync::Arc};
use rustfs_utils::crypto::{hex_sha256, is_sha256_checksum};
use serde::{Deserialize, Serialize};
use super::config_boundary;
use super::recovery_control::{
IlmRecoveryClassification, IlmRecoveryControl, IlmRecoveryProtocol, IlmRecoverySourceCopy, IlmRecoverySourceGeneration,
MAX_ILM_RECOVERY_CONTROL_SIZE, ObservedIlmRecoveryControl, ObservedIlmRecoverySource, recovery_control_record_object_name,
};
use super::tier_delete_journal::{
TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA, TIER_DELETE_JOURNAL_V2_RECOVERY_SCHEMA, validate_legacy_tier_delete_recovery_source,
};
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result};
use crate::object_api::{ObjectOptions, WriteCompletion};
use crate::services::notification_sys::{
acquire_ilm_recovery_export_fleet_proof, ilm_recovery_export_fleet_proof_matches, ilm_recovery_export_member_epochs_sha256,
ilm_recovery_export_topology_generation,
};
use crate::storage_api_contracts::{list::ListOperations as _, namespace::NamespaceLocking as _, object::HTTPPreconditions};
use crate::store::ECStore;
pub const ILM_RECOVERY_EXPORT_SCHEMA: &str = "rustfs-ilm-recovery-export-v1";
pub const ILM_RECOVERY_EXPORT_PREFIX: &str = "ilm/recovery-exports";
pub const MAX_ILM_RECOVERY_EXPORT_SIZE: usize = 128 * 1024;
const MAX_ILM_RECOVERY_EXPORTS: usize = 10_000;
const MAX_ILM_RECOVERY_EXPORT_BYTES: u64 = 1024 * 1024 * 1024;
const MAX_ACTOR_EXPORTS_PER_MINUTE: usize = 10;
const MAX_CLUSTER_EXPORTS_PER_MINUTE: usize = 100;
const EXPORT_RETENTION_NANOS: i64 = 90 * 24 * 60 * 60 * 1_000_000_000;
const EXPORT_ADMISSION_LOCK: &str = "ilm/recovery-admission/export.lock";
const MAX_LEGACY_TIER_DELETE_SOURCE_SIZE: usize = 64 * 1024;
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct IlmRecoveryExportObservation {
pub control_id: String,
pub protocol: IlmRecoveryProtocol,
pub control_etag: String,
pub control_revision: u64,
pub classification: IlmRecoveryClassification,
pub canonical_source_path: String,
pub source_generation: IlmRecoverySourceGeneration,
pub topology_generation: String,
pub member_epochs_sha256: String,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct IlmRecoveryExport {
pub export_id: String,
pub control_id: String,
pub protocol: IlmRecoveryProtocol,
pub control_etag: String,
pub control_revision: u64,
pub classification: IlmRecoveryClassification,
pub canonical_source_path: String,
pub source_generation: IlmRecoverySourceGeneration,
pub topology_generation: String,
pub member_epochs_sha256: String,
pub creator_sha256: String,
pub created_at_unix_nanos: i64,
pub retain_until_unix_nanos: i64,
pub source_bytes_base64: String,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct PersistedIlmRecoveryExport {
schema: String,
content_sha256: String,
export: IlmRecoveryExport,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct IlmRecoveryExportCreated {
pub export_id: String,
pub content_sha256: String,
pub encoded: Vec<u8>,
pub replayed: bool,
}
impl IlmRecoveryExport {
fn validate(&self) -> Result<()> {
self.source_generation.validate().map_err(Error::other)?;
validate_sha256(&self.export_id, "ILM recovery export ID is invalid")?;
validate_sha256(&self.control_id, "ILM recovery export control ID is invalid")?;
validate_sha256(&self.topology_generation, "ILM recovery export topology generation is invalid")?;
validate_sha256(&self.member_epochs_sha256, "ILM recovery export member epoch digest is invalid")?;
validate_sha256(&self.creator_sha256, "ILM recovery export creator digest is invalid")?;
if self.protocol != IlmRecoveryProtocol::TierDeleteJournal
|| self.classification != IlmRecoveryClassification::RetainedAmbiguous
|| !is_legacy_export_schema(&self.source_generation.source_schema)
{
return Err(Error::other("ILM recovery export source is not an exportable legacy journal"));
}
if self.control_etag.trim().is_empty() || self.control_revision == 0 {
return Err(Error::other("ILM recovery export control generation is invalid"));
}
if self.canonical_source_path.is_empty()
|| self.canonical_source_path.starts_with('/')
|| self.canonical_source_path.ends_with('/')
|| self.canonical_source_path.split('/').any(str::is_empty)
{
return Err(Error::other("ILM recovery export source path is invalid"));
}
if self.created_at_unix_nanos <= 0
|| self.retain_until_unix_nanos < self.created_at_unix_nanos.saturating_add(EXPORT_RETENTION_NANOS)
{
return Err(Error::other("ILM recovery export retention is invalid"));
}
let source = base64_simd::STANDARD
.decode_to_vec(self.source_bytes_base64.as_bytes())
.map_err(|_| Error::other("ILM recovery export source encoding is invalid"))?;
validate_legacy_tier_delete_recovery_source(&self.canonical_source_path, &self.source_generation.source_schema, &source)?;
let encoded_len = u64::try_from(source.len()).map_err(|_| Error::other("ILM recovery export source length overflow"))?;
if source.is_empty()
|| source.len() > MAX_LEGACY_TIER_DELETE_SOURCE_SIZE
|| hex_sha256(&source, ToOwned::to_owned) != self.source_generation.content_sha256
|| self.source_generation.copies.iter().any(|copy| {
copy.canonical_path != self.canonical_source_path
|| copy.etag != self.source_generation.source_etag
|| copy.content_sha256 != self.source_generation.content_sha256
|| copy.encoded_len != encoded_len
})
{
return Err(Error::other("ILM recovery export source bytes do not match the observed generation"));
}
if recovery_export_id(&self.control_id, &self.source_generation)? != self.export_id {
return Err(Error::other("ILM recovery export ID does not match its source generation"));
}
Ok(())
}
pub fn encode(&self) -> Result<Vec<u8>> {
self.validate()?;
let export_bytes = serde_json::to_vec(self).map_err(Error::other)?;
let persisted = PersistedIlmRecoveryExport {
schema: ILM_RECOVERY_EXPORT_SCHEMA.to_string(),
content_sha256: hex_sha256(&export_bytes, ToOwned::to_owned),
export: self.clone(),
};
let encoded = serde_json::to_vec(&persisted).map_err(Error::other)?;
if encoded.len() > MAX_ILM_RECOVERY_EXPORT_SIZE {
return Err(Error::other("encoded ILM recovery export exceeds maximum size"));
}
Ok(encoded)
}
pub fn decode(expected_export_id: &str, data: &[u8]) -> Result<Self> {
validate_sha256(expected_export_id, "ILM recovery export ID is invalid")?;
if data.len() > MAX_ILM_RECOVERY_EXPORT_SIZE {
return Err(Error::other("encoded ILM recovery export exceeds maximum size"));
}
let persisted: PersistedIlmRecoveryExport = serde_json::from_slice(data).map_err(Error::other)?;
if persisted.schema != ILM_RECOVERY_EXPORT_SCHEMA {
return Err(Error::other("ILM recovery export schema is unsupported"));
}
validate_sha256(&persisted.content_sha256, "ILM recovery export checksum is invalid")?;
let export_bytes = serde_json::to_vec(&persisted.export).map_err(Error::other)?;
if hex_sha256(&export_bytes, ToOwned::to_owned) != persisted.content_sha256 {
return Err(Error::other("ILM recovery export checksum mismatch"));
}
persisted.export.validate()?;
if persisted.export.export_id != expected_export_id {
return Err(Error::other("ILM recovery export ID does not match record key"));
}
Ok(persisted.export)
}
}
pub fn recovery_export_record_object_name(protocol: IlmRecoveryProtocol, export_id: &str) -> Result<String> {
validate_sha256(export_id, "ILM recovery export ID is invalid")?;
Ok(format!(
"{}/{}/{}/{}/{}.json",
ILM_RECOVERY_EXPORT_PREFIX,
protocol.as_str(),
&export_id[..2],
&export_id[2..4],
export_id
))
}
pub fn recovery_export_id_from_record_object_name(object: &str) -> Result<(IlmRecoveryProtocol, String)> {
let suffix = object
.strip_prefix(ILM_RECOVERY_EXPORT_PREFIX)
.and_then(|suffix| suffix.strip_prefix('/'))
.ok_or_else(|| Error::other("ILM recovery export path has wrong prefix"))?;
let mut parts = suffix.split('/');
let protocol = match parts.next() {
Some("tier_delete_journal") => IlmRecoveryProtocol::TierDeleteJournal,
_ => return Err(Error::other("ILM recovery export protocol is invalid")),
};
let shard_a = parts
.next()
.ok_or_else(|| Error::other("ILM recovery export path is incomplete"))?;
let shard_b = parts
.next()
.ok_or_else(|| Error::other("ILM recovery export path is incomplete"))?;
let export_id = parts
.next()
.and_then(|name| name.strip_suffix(".json"))
.ok_or_else(|| Error::other("ILM recovery export suffix is invalid"))?;
if parts.next().is_some() {
return Err(Error::other("ILM recovery export path is not canonical"));
}
validate_sha256(export_id, "ILM recovery export ID is invalid")?;
if shard_a != &export_id[..2] || shard_b != &export_id[2..4] {
return Err(Error::other("ILM recovery export shard does not match export ID"));
}
Ok((protocol, export_id.to_string()))
}
pub async fn inspect_recovery_export_observation(api: Arc<ECStore>, control_id: &str) -> Result<IlmRecoveryExportObservation> {
let proof = acquire_ilm_recovery_export_fleet_proof()
.await
.ok_or_else(|| Error::other("ILM recovery export fleet proof is unavailable"))?;
let observed_control = load_exportable_control(api.clone(), control_id).await?;
let observed_source = observe_export_source(
api,
&observed_control.control.identity.canonical_source_path,
&observed_control.control.observed_source_generation.source_schema,
)
.await?;
if !observed_source.is_consistent()
|| observed_source.generation != observed_control.control.observed_source_generation
|| !ilm_recovery_export_fleet_proof_matches(&proof).await
{
return Err(Error::other("ILM recovery export observation changed or is incomplete"));
}
Ok(IlmRecoveryExportObservation {
control_id: control_id.to_string(),
protocol: observed_control.control.identity.protocol,
control_etag: observed_control.etag,
control_revision: observed_control.control.revision,
classification: observed_control.control.classification,
canonical_source_path: observed_control.control.identity.canonical_source_path,
source_generation: observed_source.generation,
topology_generation: ilm_recovery_export_topology_generation(&proof),
member_epochs_sha256: ilm_recovery_export_member_epochs_sha256(&proof),
})
}
pub async fn create_recovery_export(
api: Arc<ECStore>,
observation: &IlmRecoveryExportObservation,
creator_sha256: &str,
) -> Result<IlmRecoveryExportCreated> {
validate_sha256(creator_sha256, "ILM recovery export creator digest is invalid")?;
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, EXPORT_ADMISSION_LOCK).await?;
let admission_guard = lock.get_write_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
let proof = acquire_ilm_recovery_export_fleet_proof()
.await
.ok_or_else(|| Error::other("ILM recovery export fleet proof is unavailable"))?;
if ilm_recovery_export_topology_generation(&proof) != observation.topology_generation
|| ilm_recovery_export_member_epochs_sha256(&proof) != observation.member_epochs_sha256
{
return Err(Error::PreconditionFailed);
}
let control_object = recovery_control_record_object_name(IlmRecoveryProtocol::TierDeleteJournal, &observation.control_id)
.map_err(Error::other)?;
let control_lock = api.new_ns_lock(RUSTFS_META_BUCKET, &control_object).await?;
let control_guard = control_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let source_lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &observation.canonical_source_path)
.await?;
let source_guard = source_lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
let locks_current = || !admission_guard.is_lock_lost() && !control_guard.is_lock_lost() && !source_guard.is_lock_lost();
let (current, current_source_bytes) = current_observation_under_proof_no_lock(api.clone(), observation, &proof).await?;
if &current != observation || !locks_current() {
return Err(Error::PreconditionFailed);
}
let current_source_base64 = base64_simd::STANDARD.encode_to_string(current_source_bytes);
let candidate_export_id = recovery_export_id(&current.control_id, &current.source_generation)?;
let object = recovery_export_record_object_name(current.protocol, &candidate_export_id)?;
match load_recovery_export_decoded(api.clone(), &candidate_export_id).await {
Ok((existing, export)) if export_matches_observation(&export, observation) => {
if !locks_current() || !ilm_recovery_export_fleet_proof_matches(&proof).await {
return Err(Error::PreconditionFailed);
}
api.record_durable_ilm_decommission_progress(&object, &existing.encoded)
.await?;
if !locks_current() {
return Err(Error::PreconditionFailed);
}
return Ok(existing.with_replayed());
}
Ok(_) => return Err(Error::PreconditionFailed),
Err(Error::ConfigNotFound) => {}
Err(err) => return Err(err),
}
let inventory = collect_export_inventory(api.clone()).await?;
if !locks_current() || !ilm_recovery_export_fleet_proof_matches(&proof).await {
return Err(Error::PreconditionFailed);
}
let created_at_unix_nanos = now_unix_nanos()?;
let export = build_export_from_source(&current, creator_sha256, created_at_unix_nanos, &current_source_base64)?;
let encoded = export.encode()?;
inventory.check(creator_sha256, encoded.len(), created_at_unix_nanos)?;
let mut write_options = ObjectOptions {
max_parity: true,
write_completion: WriteCompletion::TailDrained,
http_preconditions: Some(HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
}),
..Default::default()
};
write_options.add_namespace_lock_guard(&admission_guard);
write_options.add_namespace_lock_guard(&control_guard);
write_options.add_namespace_lock_guard(&source_guard);
if !locks_current() {
return Err(Error::PreconditionFailed);
}
let write_result = config_boundary::save_config_with_opts(api.clone(), &object, encoded.clone(), &write_options).await;
let stored = match load_recovery_export(api.clone(), &export.export_id).await {
Ok(stored) if stored.encoded == encoded => stored,
Ok(_) => return Err(Error::PreconditionFailed),
Err(read_err) => return Err(write_result.err().unwrap_or(read_err)),
};
if !locks_current() || !ilm_recovery_export_fleet_proof_matches(&proof).await {
return Err(Error::PreconditionFailed);
}
api.record_durable_ilm_decommission_progress(&object, &encoded).await?;
if !locks_current() {
return Err(Error::PreconditionFailed);
}
Ok(stored)
}
pub async fn load_recovery_export(api: Arc<ECStore>, export_id: &str) -> Result<IlmRecoveryExportCreated> {
let (created, _) = load_recovery_export_decoded(api, export_id).await?;
Ok(created)
}
async fn load_recovery_export_decoded(
api: Arc<ECStore>,
export_id: &str,
) -> Result<(IlmRecoveryExportCreated, IlmRecoveryExport)> {
let object = recovery_export_record_object_name(IlmRecoveryProtocol::TierDeleteJournal, export_id)?;
let encoded = config_boundary::read_config_limited_preserve_empty(api, &object, MAX_ILM_RECOVERY_EXPORT_SIZE).await?;
let export = IlmRecoveryExport::decode(export_id, &encoded)?;
let content_sha256 = hex_sha256(&encoded, ToOwned::to_owned);
Ok((
IlmRecoveryExportCreated {
export_id: export.export_id.clone(),
content_sha256,
encoded,
replayed: false,
},
export,
))
}
impl IlmRecoveryExportCreated {
fn with_replayed(mut self) -> Self {
self.replayed = true;
self
}
}
async fn load_exportable_control(api: Arc<ECStore>, control_id: &str) -> Result<ObservedIlmRecoveryControl> {
load_exportable_control_with_options(api, control_id, &ObjectOptions::default()).await
}
async fn load_exportable_control_no_lock(api: Arc<ECStore>, control_id: &str) -> Result<ObservedIlmRecoveryControl> {
load_exportable_control_with_options(
api,
control_id,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
}
async fn load_exportable_control_with_options(
api: Arc<ECStore>,
control_id: &str,
options: &ObjectOptions,
) -> Result<ObservedIlmRecoveryControl> {
let object = recovery_control_record_object_name(IlmRecoveryProtocol::TierDeleteJournal, control_id).map_err(Error::other)?;
let (data, metadata) =
config_boundary::read_config_limited_preserve_empty_with_metadata(api, &object, options, MAX_ILM_RECOVERY_CONTROL_SIZE)
.await?;
let etag = metadata
.etag
.filter(|etag| !etag.trim().is_empty())
.ok_or_else(|| Error::other("ILM recovery control is missing an ETag"))?;
let control = IlmRecoveryControl::decode(control_id, &data).map_err(Error::other)?;
if control.identity.protocol != IlmRecoveryProtocol::TierDeleteJournal
|| control.classification != IlmRecoveryClassification::RetainedAmbiguous
|| !is_legacy_export_schema(&control.observed_source_generation.source_schema)
{
return Err(Error::other("ILM recovery control is not exportable"));
}
Ok(ObservedIlmRecoveryControl { control, etag })
}
async fn current_observation_under_proof_no_lock(
api: Arc<ECStore>,
expected: &IlmRecoveryExportObservation,
proof: &crate::services::notification_sys::IlmRecoveryExportFleetProofToken,
) -> Result<(IlmRecoveryExportObservation, Vec<u8>)> {
let observed_control = load_exportable_control_no_lock(api.clone(), &expected.control_id).await?;
let observed_source = observe_export_source_no_lock(
api,
&observed_control.control.identity.canonical_source_path,
&observed_control.control.observed_source_generation.source_schema,
)
.await?;
let source_bytes = observed_source
.canonical_data
.clone()
.ok_or_else(|| Error::other("ILM recovery export source copies diverge"))?;
if !observed_source.is_consistent()
|| observed_source.generation != observed_control.control.observed_source_generation
|| !ilm_recovery_export_fleet_proof_matches(proof).await
{
return Err(Error::PreconditionFailed);
}
Ok((
IlmRecoveryExportObservation {
control_id: expected.control_id.clone(),
protocol: observed_control.control.identity.protocol,
control_etag: observed_control.etag,
control_revision: observed_control.control.revision,
classification: observed_control.control.classification,
canonical_source_path: observed_control.control.identity.canonical_source_path,
source_generation: observed_source.generation,
topology_generation: ilm_recovery_export_topology_generation(proof),
member_epochs_sha256: ilm_recovery_export_member_epochs_sha256(proof),
},
source_bytes,
))
}
async fn observe_export_source(
api: Arc<ECStore>,
canonical_path: &str,
source_schema: &str,
) -> Result<ObservedIlmRecoverySource> {
if canonical_path.is_empty()
|| canonical_path.starts_with('/')
|| canonical_path.ends_with('/')
|| canonical_path.split('/').any(str::is_empty)
|| !is_legacy_export_schema(source_schema)
{
return Err(Error::other("ILM recovery export source identity is invalid"));
}
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, canonical_path).await?;
let _guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
observe_export_source_no_lock(api, canonical_path, source_schema).await
}
async fn observe_export_source_no_lock(
api: Arc<ECStore>,
canonical_path: &str,
source_schema: &str,
) -> Result<ObservedIlmRecoverySource> {
let mut copies = Vec::new();
let mut canonical: Option<(String, String, Vec<u8>)> = None;
let mut consistent = true;
for set in api.all_set_disks() {
let authority = format!("pool-{}/set-{}", set.pool_index, set.set_index);
let result = config_boundary::read_config_limited_preserve_empty_with_metadata(
set,
canonical_path,
&ObjectOptions {
no_lock: true,
..Default::default()
},
MAX_LEGACY_TIER_DELETE_SOURCE_SIZE,
)
.await;
match result {
Ok((data, metadata)) => {
if data.is_empty() || data.len() > MAX_LEGACY_TIER_DELETE_SOURCE_SIZE {
return Err(Error::other("ILM recovery export source exceeds its protocol size limit"));
}
validate_legacy_tier_delete_recovery_source(canonical_path, source_schema, &data)?;
let etag = metadata
.etag
.filter(|etag| !etag.trim().is_empty())
.ok_or_else(|| Error::other("ILM recovery export source copy is missing an ETag"))?;
let content_sha256 = hex_sha256(&data, ToOwned::to_owned);
let encoded_len =
u64::try_from(data.len()).map_err(|_| Error::other("ILM recovery export source length does not fit u64"))?;
copies.push(IlmRecoverySourceCopy {
authority,
canonical_path: canonical_path.to_string(),
etag: etag.clone(),
encoded_len,
content_sha256: content_sha256.clone(),
});
match canonical.as_ref() {
Some((first_etag, first_digest, first_data)) => {
consistent &= first_etag == &etag && first_digest == &content_sha256 && first_data == &data;
}
None => canonical = Some((etag, content_sha256, data)),
}
}
Err(err) if export_source_is_missing(&err) => {}
Err(err) => return Err(err),
}
}
let Some((source_etag, content_sha256, source_bytes)) = canonical else {
return Err(Error::ConfigNotFound);
};
let generation =
IlmRecoverySourceGeneration::new(source_schema, source_etag, content_sha256, copies).map_err(Error::other)?;
Ok(ObservedIlmRecoverySource {
generation,
canonical_data: consistent.then_some(source_bytes),
})
}
fn export_source_is_missing(err: &Error) -> bool {
matches!(
err,
Error::ConfigNotFound | Error::FileNotFound | Error::ObjectNotFound(_, _) | Error::VersionNotFound(_, _, _)
)
}
fn build_export_from_source(
observation: &IlmRecoveryExportObservation,
creator_sha256: &str,
created_at_unix_nanos: i64,
source_bytes_base64: &str,
) -> Result<IlmRecoveryExport> {
let retain_until_unix_nanos = created_at_unix_nanos
.checked_add(EXPORT_RETENTION_NANOS)
.ok_or_else(|| Error::other("ILM recovery export retention timestamp overflow"))?;
let export = IlmRecoveryExport {
export_id: recovery_export_id(&observation.control_id, &observation.source_generation)?,
control_id: observation.control_id.clone(),
protocol: observation.protocol,
control_etag: observation.control_etag.clone(),
control_revision: observation.control_revision,
classification: observation.classification,
canonical_source_path: observation.canonical_source_path.clone(),
source_generation: observation.source_generation.clone(),
topology_generation: observation.topology_generation.clone(),
member_epochs_sha256: observation.member_epochs_sha256.clone(),
creator_sha256: creator_sha256.to_string(),
created_at_unix_nanos,
retain_until_unix_nanos,
source_bytes_base64: source_bytes_base64.to_string(),
};
export.validate()?;
Ok(export)
}
pub(crate) fn recovery_export_id(control_id: &str, generation: &IlmRecoverySourceGeneration) -> Result<String> {
validate_sha256(control_id, "ILM recovery export control ID is invalid")?;
validate_sha256(&generation.content_sha256, "ILM recovery export source checksum is invalid")?;
validate_sha256(&generation.copy_set_sha256, "ILM recovery export copy-set checksum is invalid")?;
let mut data = Vec::new();
for part in [control_id, &generation.content_sha256, &generation.copy_set_sha256] {
data.extend_from_slice(&(part.len() as u64).to_be_bytes());
data.extend_from_slice(part.as_bytes());
}
Ok(hex_sha256(&data, ToOwned::to_owned))
}
fn export_matches_observation(export: &IlmRecoveryExport, observation: &IlmRecoveryExportObservation) -> bool {
export.control_id == observation.control_id
&& export.protocol == observation.protocol
&& export.classification == observation.classification
&& export.canonical_source_path == observation.canonical_source_path
&& export.source_generation == observation.source_generation
}
#[derive(Debug, Default)]
struct IlmRecoveryExportInventory {
count: usize,
bytes: u64,
creations: Vec<(i64, String)>,
}
impl IlmRecoveryExportInventory {
fn check(&self, creator_sha256: &str, candidate_len: usize, now: i64) -> Result<()> {
let recent_after = now.saturating_sub(60 * 1_000_000_000);
let cluster_recent = self
.creations
.iter()
.filter(|(created_at, _)| *created_at > recent_after)
.count();
let actor_recent = self
.creations
.iter()
.filter(|(created_at, creator)| *created_at > recent_after && creator == creator_sha256)
.count();
check_export_admission(self.count, self.bytes, actor_recent, cluster_recent, candidate_len)
}
}
async fn collect_export_inventory(api: Arc<ECStore>) -> Result<IlmRecoveryExportInventory> {
let mut marker = None;
let mut seen_markers = HashSet::new();
let mut inventory = IlmRecoveryExportInventory::default();
loop {
let page = api
.clone()
.list_objects_v2(
RUSTFS_META_BUCKET,
&format!("{ILM_RECOVERY_EXPORT_PREFIX}/"),
marker.clone(),
None,
1_000,
false,
None,
false,
)
.await?;
for object in page.objects {
let (_, export_id) = recovery_export_id_from_record_object_name(&object.name)?;
let (stored, export) = load_recovery_export_decoded(api.clone(), &export_id).await?;
inventory.count = inventory
.count
.checked_add(1)
.ok_or_else(|| Error::other("ILM recovery export count overflow"))?;
inventory.bytes = inventory
.bytes
.checked_add(u64::try_from(stored.encoded.len()).map_err(|_| Error::other("ILM recovery export size overflow"))?)
.ok_or_else(|| Error::other("ILM recovery export byte total overflow"))?;
inventory
.creations
.push((export.created_at_unix_nanos, export.creator_sha256));
}
if !page.is_truncated {
break;
}
let next = page
.next_continuation_token
.ok_or_else(|| Error::other("ILM recovery export inventory omitted its continuation marker"))?;
marker = Some(record_export_inventory_marker(&mut seen_markers, next)?);
}
Ok(inventory)
}
fn record_export_inventory_marker(seen_markers: &mut HashSet<String>, next: String) -> Result<String> {
if !seen_markers.insert(next.clone()) {
return Err(Error::other("ILM recovery export inventory repeated its continuation marker"));
}
Ok(next)
}
fn check_export_admission(
count: usize,
bytes: u64,
actor_recent: usize,
cluster_recent: usize,
candidate_len: usize,
) -> Result<()> {
let candidate_len = u64::try_from(candidate_len).map_err(|_| Error::other("ILM recovery export size does not fit u64"))?;
if count >= MAX_ILM_RECOVERY_EXPORTS
|| bytes
.checked_add(candidate_len)
.is_none_or(|total| total > MAX_ILM_RECOVERY_EXPORT_BYTES)
|| actor_recent >= MAX_ACTOR_EXPORTS_PER_MINUTE
|| cluster_recent >= MAX_CLUSTER_EXPORTS_PER_MINUTE
{
return Err(Error::SlowDown);
}
Ok(())
}
fn is_legacy_export_schema(schema: &str) -> bool {
matches!(schema, TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA | TIER_DELETE_JOURNAL_V2_RECOVERY_SCHEMA)
}
fn validate_sha256(value: &str, message: &'static str) -> Result<()> {
if !is_sha256_checksum(value)
|| value
.bytes()
.any(|byte| byte.is_ascii_hexdigit() && byte.is_ascii_uppercase())
{
return Err(Error::other(message));
}
Ok(())
}
fn now_unix_nanos() -> Result<i64> {
i64::try_from(time::OffsetDateTime::now_utc().unix_timestamp_nanos())
.map_err(|_| Error::other("ILM recovery export timestamp does not fit i64"))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bucket::lifecycle::recovery_control::IlmRecoverySourceCopy;
const PINNED_V1_EXPORT: &[u8] = br#"{"schema":"rustfs-ilm-recovery-export-v1","content_sha256":"3dfb3ec3892256e909de1211c1a963ca7008963ff32b3a869f7161a7b9b44028","export":{"export_id":"2b78e7a825bfc2edbf7f773d0b6ed3bf93e360ff1702d73a449109c11bfaa105","control_id":"0fcd568a5cb9bdb4677b69354b11ee415af8f784519cff3da49a26f84eaee7f2","protocol":"tier_delete_journal","control_etag":"control-etag","control_revision":1,"classification":"retained_ambiguous","canonical_source_path":"ilm/tier-delete-journal/872072554f66ab326f10ce7adbae11422b7a4b0663aa7112d6061a8f6ed41b94.json","source_generation":{"source_schema":"rustfs-tier-delete-journal-v1","source_etag":"etag-a","content_sha256":"0e0b010ebdeeb7b41473fe8575e989d6bb1303c0ca551dd984e9400f0ae306bd","copy_set_sha256":"5a7406115b6c3923ffe79dcd1f43ccae7beed786e557163f019dd10ec409a653","copies":[{"authority":"pool-0/set-0","canonical_path":"ilm/tier-delete-journal/872072554f66ab326f10ce7adbae11422b7a4b0663aa7112d6061a8f6ed41b94.json","etag":"etag-a","encoded_len":81,"content_sha256":"0e0b010ebdeeb7b41473fe8575e989d6bb1303c0ca551dd984e9400f0ae306bd"}]},"topology_generation":"e6e2b826e31fca5c36125c48f130dcb6f961e698ff8a8776a1f290cf0892e8e6","member_epochs_sha256":"612dd8a861161819a4ad8f6f3e2a0567602877c043a2353ca933a13c78dc0ed4","creator_sha256":"50c9c4aeb40b5b206b6d98f516f8b8c0efd29ce2e56a76b345fb9240c225a1b7","created_at_unix_nanos":1000000000,"retain_until_unix_nanos":7776001000000000,"source_bytes_base64":"eyJ2ZXJzaW9uIjoxLCJvYmpfbmFtZSI6ImxlZ2FjeS9yZW1vdGUiLCJ2ZXJzaW9uX2lkIjoib3BhcXVlIiwidGllcl9uYW1lIjoiV0FSTSJ9"}}"#;
fn legacy_source() -> Vec<u8> {
br#"{"version":1,"obj_name":"legacy/remote","version_id":"opaque","tier_name":"WARM"}"#.to_vec()
}
fn observation() -> IlmRecoveryExportObservation {
let source = legacy_source();
let source_path = super::super::tier_delete_journal::tier_delete_journal_object_name(
&super::super::tier_delete_journal::decode_tier_delete_journal_entry(&source).expect("legacy fixture should decode"),
);
let source_sha256 = hex_sha256(&source, ToOwned::to_owned);
let generation = IlmRecoverySourceGeneration::new(
TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA,
"etag-a",
source_sha256.clone(),
vec![IlmRecoverySourceCopy {
authority: "pool-0/set-0".to_string(),
canonical_path: source_path.clone(),
etag: "etag-a".to_string(),
encoded_len: source.len() as u64,
content_sha256: source_sha256,
}],
)
.expect("generation should be valid");
IlmRecoveryExportObservation {
control_id: hex_sha256(b"control", ToOwned::to_owned),
protocol: IlmRecoveryProtocol::TierDeleteJournal,
control_etag: "control-etag".to_string(),
control_revision: 1,
classification: IlmRecoveryClassification::RetainedAmbiguous,
canonical_source_path: source_path,
source_generation: generation,
topology_generation: hex_sha256(b"topology", ToOwned::to_owned),
member_epochs_sha256: hex_sha256(b"epochs", ToOwned::to_owned),
}
}
#[test]
fn recovery_export_round_trip_is_strict_and_deterministic() {
let observed = observation();
let creator = hex_sha256(b"actor", ToOwned::to_owned);
let export = build_export_from_source(
&observed,
&creator,
1_000_000_000,
&base64_simd::STANDARD.encode_to_string(legacy_source()),
)
.expect("export should be valid");
assert_eq!(
export.export_id,
recovery_export_id(&observed.control_id, &observed.source_generation).unwrap()
);
let encoded = export.encode().expect("export should encode");
assert_eq!(encoded, PINNED_V1_EXPORT, "v1 export wire format must remain pinned");
assert_eq!(IlmRecoveryExport::decode(&export.export_id, &encoded).unwrap(), export);
assert_eq!(
IlmRecoveryExport::decode("2b78e7a825bfc2edbf7f773d0b6ed3bf93e360ff1702d73a449109c11bfaa105", PINNED_V1_EXPORT)
.unwrap(),
export,
);
let path = recovery_export_record_object_name(export.protocol, &export.export_id).unwrap();
let durable = super::super::durable_namespace::validate_durable_ilm_record(&path, &encoded)
.expect("export should be registered as a durable ILM record");
assert_eq!(durable.namespace, "recovery-export");
assert_eq!(durable.id_kind, "export_id");
assert_eq!(durable.id, export.export_id);
let mut wrong_source = export.clone();
wrong_source.source_bytes_base64 = base64_simd::STANDARD.encode_to_string(b"changed");
assert!(wrong_source.encode().is_err());
let mut persisted: serde_json::Value = serde_json::from_slice(&encoded).unwrap();
persisted["unknown"] = serde_json::json!(true);
assert!(IlmRecoveryExport::decode(&export.export_id, &serde_json::to_vec(&persisted).unwrap()).is_err());
}
#[test]
fn export_inventory_rejects_non_adjacent_continuation_cycles() {
let mut seen = HashSet::new();
assert_eq!(record_export_inventory_marker(&mut seen, "a".to_string()).unwrap(), "a");
assert_eq!(record_export_inventory_marker(&mut seen, "b".to_string()).unwrap(), "b");
record_export_inventory_marker(&mut seen, "a".to_string())
.expect_err("a non-adjacent continuation marker cycle must fail closed");
}
#[test]
fn recovery_export_path_rejects_noncanonical_shards() {
let id = hex_sha256(b"export", ToOwned::to_owned);
let path = recovery_export_record_object_name(IlmRecoveryProtocol::TierDeleteJournal, &id).unwrap();
assert_eq!(recovery_export_id_from_record_object_name(&path).unwrap().1, id);
let wrong_shard = path.replacen(&format!("/{}/", &id[..2]), "/zz/", 1);
assert!(recovery_export_id_from_record_object_name(&wrong_shard).is_err());
}
#[test]
fn canonical_replay_survives_fleet_rotation_but_not_source_change() {
let observed = observation();
let creator = hex_sha256(b"actor", ToOwned::to_owned);
let export = build_export_from_source(
&observed,
&creator,
1_000_000_000,
&base64_simd::STANDARD.encode_to_string(legacy_source()),
)
.unwrap();
let mut rotated = observed;
rotated.control_etag = "new-control-etag".to_string();
rotated.control_revision += 1;
rotated.topology_generation = hex_sha256(b"new-topology", ToOwned::to_owned);
rotated.member_epochs_sha256 = hex_sha256(b"new-members", ToOwned::to_owned);
assert!(export_matches_observation(&export, &rotated));
rotated.source_generation.content_sha256 = hex_sha256(b"changed", ToOwned::to_owned);
assert!(!export_matches_observation(&export, &rotated));
}
#[test]
fn export_admission_enforces_exact_count_byte_and_rate_boundaries() {
assert!(check_export_admission(9_999, MAX_ILM_RECOVERY_EXPORT_BYTES - 1, 9, 99, 1).is_ok());
assert!(check_export_admission(10_000, 0, 0, 0, 1).is_err());
assert!(check_export_admission(0, MAX_ILM_RECOVERY_EXPORT_BYTES, 0, 0, 1).is_err());
assert!(check_export_admission(0, 0, 10, 0, 1).is_err());
assert!(check_export_admission(0, 0, 0, 100, 1).is_err());
}
}
@@ -82,8 +82,8 @@ const TIER_DELETE_DISPATCH_MEMBER_DELETE_CONCURRENCY: usize = 32;
const TIER_DELETE_DISPATCH_PREPARE_CONCURRENCY: usize = 16;
const TIER_DELETE_DISPATCH_CAS_CONCURRENCY: usize = 32;
const TIER_DELETE_JOURNAL_VERSION: u8 = 2;
const TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA: &str = "rustfs-tier-delete-journal-v1";
const TIER_DELETE_JOURNAL_V2_RECOVERY_SCHEMA: &str = "rustfs-tier-delete-journal-v2";
pub(crate) const TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA: &str = "rustfs-tier-delete-journal-v1";
pub(crate) const TIER_DELETE_JOURNAL_V2_RECOVERY_SCHEMA: &str = "rustfs-tier-delete-journal-v2";
const TIER_DELETE_JOURNAL_UNKNOWN_RECOVERY_SCHEMA: &str = "rustfs-tier-delete-journal-unknown";
const TIER_DELETE_JOURNAL_V1_RECOVERY_CLASS: &str = "tier_delete_journal_v1";
const TIER_DELETE_JOURNAL_V2_RECOVERY_CLASS: &str = "tier_delete_journal_v2";
@@ -884,6 +884,23 @@ struct PersistedTierDeleteJournalEntry {
}
impl PersistedTierDeleteJournalEntry {
fn validate_legacy_recovery_shape(&self) -> Result<()> {
let has_later_version_fields = self.version_id_exact.is_some()
|| self.version_state.is_some()
|| self.state.is_some()
|| self.source.is_some()
|| self.dispatch.is_some();
match self.version {
1 if self.backend_identity.is_none() && !has_later_version_fields => Ok(()),
TIER_DELETE_JOURNAL_VERSION if self.backend_identity.is_some() && !has_later_version_fields => Ok(()),
1 => Err(Error::other("tier delete journal v1 entry contains fields from a later version")),
TIER_DELETE_JOURNAL_VERSION => Err(Error::other(
"tier delete journal v2 entry is missing its identity or contains fields from a later version",
)),
_ => Err(Error::other("tier delete journal is not an exportable legacy version")),
}
}
fn from_jentry(je: &Jentry) -> Result<Self> {
validate_version_state(je.version_state, &je.version_id, je.version_id_exact)?;
let legacy_unknown = je.version_state == rustfs_filemeta::TransitionVersionState::Unknown;
@@ -5531,6 +5548,12 @@ fn canonical_legacy_tier_delete_journal_identity(object_name: &str) -> Option<&s
.then_some(identity)
}
pub(crate) fn validate_legacy_tier_delete_recovery_path(object_name: &str) -> Result<()> {
canonical_legacy_tier_delete_journal_identity(object_name)
.map(|_| ())
.ok_or_else(|| Error::other("legacy tier delete journal path is not canonical"))
}
fn legacy_tier_delete_recovery_descriptor(entry: &Jentry) -> Option<(&'static str, &'static str)> {
match entry.persisted_version {
1 => Some((TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA, TIER_DELETE_JOURNAL_V1_RECOVERY_CLASS)),
@@ -5539,6 +5562,21 @@ fn legacy_tier_delete_recovery_descriptor(entry: &Jentry) -> Option<(&'static st
}
}
pub(crate) fn validate_legacy_tier_delete_recovery_source(object_name: &str, source_schema: &str, data: &[u8]) -> Result<()> {
validate_legacy_tier_delete_recovery_path(object_name)?;
let persisted: PersistedTierDeleteJournalEntry =
serde_json::from_slice(data).map_err(|err| Error::other_with_context("decode tier delete journal failed", err))?;
persisted.validate_legacy_recovery_shape()?;
let entry = persisted.into_jentry()?;
let Some((decoded_schema, _)) = legacy_tier_delete_recovery_descriptor(&entry) else {
return Err(Error::other("tier delete journal is not an exportable legacy version"));
};
if decoded_schema != source_schema || tier_delete_journal_object_name(&entry) != object_name {
return Err(Error::other("legacy tier delete journal identity does not match its recovery source"));
}
Ok(())
}
fn legacy_tier_delete_control_matches(
control: &IlmRecoveryControl,
identity: &IlmRecoveryControlIdentity,
@@ -6140,17 +6178,18 @@ where
#[cfg(test)]
mod tests {
use super::{
TIER_DELETE_DISPATCH_MANIFEST_VERSION, TIER_DELETE_DISPATCH_PARENT_RECORD_TYPE, TIER_DELETE_DISPATCH_PARENT_VERSION,
TIER_DELETE_JOURNAL_EXACT_VERSION, TIER_DELETE_JOURNAL_LEGACY_PREFIX, TIER_DELETE_JOURNAL_SOLE_OWNER_VERSION,
TIER_DELETE_JOURNAL_STATE_VERSION, TIER_DELETE_JOURNAL_TRANSACTION_VERSION, TIER_DELETE_JOURNAL_V6_PREFIX,
TierDeleteDispatchChunkBinding, TierDeleteDispatchManifest, TierDeleteDispatchManifestState, TierDeleteDispatchParent,
TierDeleteDispatchParentState, TierDeleteDispatchRecord, await_tier_delete_journal_recovery,
PersistedTierDeleteJournalEntry, TIER_DELETE_DISPATCH_MANIFEST_VERSION, TIER_DELETE_DISPATCH_PARENT_RECORD_TYPE,
TIER_DELETE_DISPATCH_PARENT_VERSION, TIER_DELETE_JOURNAL_EXACT_VERSION, TIER_DELETE_JOURNAL_LEGACY_PREFIX,
TIER_DELETE_JOURNAL_SOLE_OWNER_VERSION, TIER_DELETE_JOURNAL_STATE_VERSION, TIER_DELETE_JOURNAL_TRANSACTION_VERSION,
TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA, TIER_DELETE_JOURNAL_V2_RECOVERY_SCHEMA, TIER_DELETE_JOURNAL_V6_PREFIX,
TIER_DELETE_JOURNAL_VERSION, TierDeleteDispatchChunkBinding, TierDeleteDispatchManifest, TierDeleteDispatchManifestState,
TierDeleteDispatchParent, TierDeleteDispatchParentState, TierDeleteDispatchRecord, await_tier_delete_journal_recovery,
decode_tier_delete_dispatch_record, decode_tier_delete_journal_entry, encode_tier_delete_dispatch_manifest,
encode_tier_delete_dispatch_parent, encode_tier_delete_journal_entry, object_info_references_tier_delete,
record_tier_delete_journal_backend_identity, same_tier_delete_authorization_identity, same_tier_delete_journal_identity,
tier_delete_dispatch_child_matches_parent, tier_delete_dispatch_chunk_manifest_object_name,
tier_delete_dispatch_journal_set_digest, tier_delete_dispatch_manifest_object_name, tier_delete_journal_object_name,
tier_delete_source_matches_dispatch_scope,
tier_delete_source_matches_dispatch_scope, validate_legacy_tier_delete_recovery_source,
};
use crate::bucket::lifecycle::tier_sweeper::{
Jentry, TierDeleteDispatchBinding, TierDeleteJournalState, TierDeleteSourceIdentity,
@@ -6609,6 +6648,72 @@ mod tests {
}
}
#[test]
fn legacy_recovery_export_rejects_fields_from_later_journal_versions() {
let later = bound_v6_journal_entry(TierDeleteJournalState::Prepared);
let v1 = PersistedTierDeleteJournalEntry {
version: 1,
obj_name: "remote/object".to_string(),
version_id: "opaque".to_string(),
tier_name: "WARM".to_string(),
backend_identity: None,
version_id_exact: None,
version_state: None,
state: None,
source: None,
dispatch: None,
};
let mut v2 = v1.clone();
v2.version = TIER_DELETE_JOURNAL_VERSION;
v2.backend_identity = Some([7; 32]);
let assert_rejected = |persisted: PersistedTierDeleteJournalEntry, schema: &str| {
let normalized = persisted
.clone()
.into_jentry()
.expect("the generic compatibility decoder should demonstrate the discarded field");
let object_name = tier_delete_journal_object_name(&normalized);
let encoded = serde_json::to_vec(&persisted).expect("mixed-version journal fixture should encode");
let err = validate_legacy_tier_delete_recovery_source(&object_name, schema, &encoded)
.expect_err("legacy recovery export must reject fields from later versions");
assert!(err.to_string().contains("later version"));
};
let mut invalid_v1 = Vec::new();
let mut with_backend = v1.clone();
with_backend.backend_identity = Some([7; 32]);
invalid_v1.push(with_backend);
for persisted in [&v1, &v2] {
let schema = if persisted.version == 1 {
TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA
} else {
TIER_DELETE_JOURNAL_V2_RECOVERY_SCHEMA
};
let mut invalid = Vec::new();
let mut with_exact = persisted.clone();
with_exact.version_id_exact = Some(false);
invalid.push(with_exact);
let mut with_version_state = persisted.clone();
with_version_state.version_state = Some(rustfs_filemeta::TransitionVersionState::Unknown);
invalid.push(with_version_state);
let mut with_state = persisted.clone();
with_state.state = Some(TierDeleteJournalState::Committed);
invalid.push(with_state);
let mut with_source = persisted.clone();
with_source.source = later.source.clone();
invalid.push(with_source);
let mut with_dispatch = persisted.clone();
with_dispatch.dispatch = later.dispatch.clone();
invalid.push(with_dispatch);
for record in invalid {
assert_rejected(record, schema);
}
}
for record in invalid_v1 {
assert_rejected(record, TIER_DELETE_JOURNAL_V1_RECOVERY_SCHEMA);
}
}
#[test]
fn tier_delete_journal_path_is_stable_and_sanitized() {
let je = journal_entry();
@@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(test)]
pub(crate) use rustfs_filemeta::ObjectPartInfo;
pub use rustfs_replication::{MrfOpKind, MrfReplicateEntry};
pub(crate) use rustfs_replication::{
REPLICATE_EXISTING, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction,
@@ -12,6 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(test)]
pub(crate) use rustfs_replication::ReplicationMultipartPlanError;
pub use rustfs_replication::{
MustReplicateOptions, ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, delete_replication_state_from_config,
delete_replication_version_id, should_schedule_delete_replication, should_use_existing_delete_replication_info,
@@ -4656,6 +4656,58 @@ where
result
}
#[derive(Debug)]
struct MultipartReplicationReadPlan {
part_number: i32,
part_size: i64,
range: Option<HTTPRangeSpec>,
next_offset: i64,
}
fn multipart_replication_read_plan(
object_info: &ObjectInfo,
obj_opts: &ObjectOptions,
mut input: ReplicationMultipartPartInput,
stored_size: usize,
is_last: bool,
) -> std::io::Result<MultipartReplicationReadPlan> {
let empty_last_part = is_last && input.part_size == 0 && stored_size == 0;
// Raw reads address stored bytes. Only untransformed legacy parts may
// substitute their stored size for a missing logical size.
if obj_opts.raw_data_movement_read || (input.part_size == 0 && !object_info.is_compressed() && !object_info.is_encrypted()) {
input.part_size = i64::try_from(stored_size).map_err(|_| {
std::io::Error::new(std::io::ErrorKind::InvalidData, "multipart replication stored part size exceeds i64")
})?;
}
if empty_last_part {
if input.offset < 0 {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"empty multipart replication part has a negative offset",
));
}
let part_number = i32::try_from(input.part_number)
.map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, "multipart replication part number exceeds i32"))?;
return Ok(MultipartReplicationReadPlan {
part_number,
part_size: 0,
range: None,
next_offset: input.offset,
});
}
let plan = replication_multipart_part_plan(input).map_err(std::io::Error::other)?;
Ok(MultipartReplicationReadPlan {
part_number: plan.part_number,
part_size: plan.part_size,
range: Some(HTTPRangeSpec {
is_suffix_length: false,
start: plan.range.start,
end: plan.range.end,
}),
next_offset: plan.next_offset,
})
}
async fn replicate_multipart_parts_and_complete<S: ReplicationObjectIO>(
ctx: MultipartReplicationContext<'_, S>,
upload_id: &str,
@@ -4676,35 +4728,31 @@ async fn replicate_multipart_parts_and_complete<S: ReplicationObjectIO>(
let mut header_size = replication_put_object_header_size(&put_opts);
let mut offset: i64 = 0;
for part_info in object_info.parts.iter() {
// Ciphertext passthrough (raw read) ranges over the stored part
// bytes; decrypted reads range over the logical plaintext parts.
let part_size = if obj_opts.raw_data_movement_read {
part_info.size as i64
} else {
part_info.actual_size
};
let part_plan = replication_multipart_part_plan(ReplicationMultipartPartInput {
offset,
part_number: part_info.number,
part_size,
})
.map_err(|err| std::io::Error::other(err.to_string()))?;
let range_spec = HTTPRangeSpec {
is_suffix_length: false,
start: part_plan.range.start,
end: part_plan.range.end,
};
for (index, part_info) in object_info.parts.iter().enumerate() {
let part_plan = multipart_replication_read_plan(
object_info,
obj_opts,
ReplicationMultipartPartInput {
offset,
part_number: part_info.number,
part_size: part_info.actual_size,
},
part_info.size,
index + 1 == object_info.parts.len(),
)?;
offset = part_plan.next_offset;
let part_reader = storage
.get_object_reader(src_bucket, object, Some(range_spec), HeaderMap::new(), obj_opts)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
let part_stream = wrap_with_bandwidth_monitor_with_header(part_reader.stream, src_bucket, arn, header_size);
let byte_stream = if let Some(range_spec) = part_plan.range {
let part_reader = storage
.get_object_reader(src_bucket, object, Some(range_spec), HeaderMap::new(), obj_opts)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
let part_stream = wrap_with_bandwidth_monitor_with_header(part_reader.stream, src_bucket, arn, header_size);
async_read_to_bytestream(part_stream)
} else {
ByteStream::from_static(b"")
};
header_size = 0;
let byte_stream = async_read_to_bytestream(part_stream);
let object_part = cli
.put_object_part(
@@ -4760,6 +4808,173 @@ async fn replicate_multipart_parts_and_complete<S: ReplicationObjectIO>(
#[cfg(test)]
mod tests {
use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
use super::super::replication_object_decision_boundary::ReplicationMultipartPlanError;
#[test]
fn multipart_read_plan_preserves_legacy_plain_part_ranges() {
const MIB: usize = 1024 * 1024;
let object_info = ObjectInfo {
etag: Some("0123456789abcdef0123456789abcdef".to_string()),
size: 6 * 1024 * 1024,
..Default::default()
};
let mut offset = 0;
for (part_number, stored_size, start, end) in [
(1, 5 * MIB, 0, 5 * 1024 * 1024 - 1),
(2, MIB, 5 * 1024 * 1024, 6 * 1024 * 1024 - 1),
] {
let plan = multipart_replication_read_plan(
&object_info,
&ObjectOptions::default(),
ReplicationMultipartPartInput {
offset,
part_number,
part_size: 0,
},
stored_size,
part_number == 2,
)
.expect("legacy plain parts must use their stored sizes");
assert_eq!(plan.part_number, i32::try_from(part_number).expect("part number fits"));
assert_eq!(plan.part_size, i64::try_from(stored_size).expect("stored size fits"));
let range = plan.range.expect("a nonempty part must read a range");
assert!(!range.is_suffix_length);
assert_eq!((range.start, range.end), (start, end));
assert_eq!(plan.next_offset, end + 1);
offset = plan.next_offset;
}
assert_eq!(offset, object_info.size);
}
#[test]
fn multipart_read_plan_distinguishes_transformed_and_raw_sizes() {
for metadata in [
HashMap::from([("x-rustfs-internal-compression".to_string(), "klauspost/compress/s2".to_string())]),
HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())]),
] {
let object_info = ObjectInfo {
user_defined: Arc::new(metadata),
..Default::default()
};
assert!(object_info.is_compressed() || object_info.is_encrypted());
for raw in [false, true] {
for actual_size in [-1, 0, 5] {
let result = multipart_replication_read_plan(
&object_info,
&ObjectOptions {
raw_data_movement_read: raw,
..Default::default()
},
ReplicationMultipartPartInput {
offset: 7,
part_number: 2,
part_size: actual_size,
},
9,
true,
);
if !raw && actual_size <= 0 {
let err = result.expect_err("transformed reads cannot substitute physical bytes for unknown plaintext");
assert!(matches!(
err.get_ref().and_then(|err| err.downcast_ref::<ReplicationMultipartPlanError>()),
Some(ReplicationMultipartPlanError::InvalidPartSize { part_size })
if *part_size == actual_size
));
} else {
let plan = result.expect("the selected representation has a known positive size");
let expected_size = if raw { 9 } else { 5 };
assert_eq!(plan.part_number, 2);
assert_eq!(plan.part_size, expected_size);
let range = plan.range.expect("a nonempty part must read a range");
assert_eq!((range.start, range.end), (7, 7 + expected_size - 1));
assert_eq!(plan.next_offset, 7 + expected_size);
}
}
}
}
}
#[test]
fn multipart_read_plan_retains_an_empty_last_part_without_advancing() {
for offset in [5 * 1024 * 1024, i64::MAX] {
for raw in [false, true] {
let plan = multipart_replication_read_plan(
&ObjectInfo::default(),
&ObjectOptions {
raw_data_movement_read: raw,
..Default::default()
},
ReplicationMultipartPartInput {
offset,
part_number: 2,
part_size: 0,
},
0,
true,
)
.expect("an empty final part needs no range read");
assert_eq!(plan.part_number, 2);
assert_eq!(plan.part_size, 0);
assert!(plan.range.is_none());
assert_eq!(plan.next_offset, offset);
}
}
}
#[test]
fn multipart_read_plan_rejects_invalid_empty_parts_and_ranges() {
for (offset, part_number, actual_size, stored_size, is_last) in [
(0, 1, 0, 0, false),
(0, 2, -1, 0, true),
(0, 2, -1, 9, true),
(-1, 2, 0, 0, true),
(0, usize::try_from(i32::MAX).expect("i32 fits usize") + 1, 0, 0, true),
(i64::MAX, 2, 1, 1, true),
(i64::MAX, 2, 2, 2, true),
] {
let err = multipart_replication_read_plan(
&ObjectInfo::default(),
&ObjectOptions::default(),
ReplicationMultipartPartInput {
offset,
part_number,
part_size: actual_size,
},
stored_size,
is_last,
)
.expect_err("invalid part metadata must not become a successful transport plan");
assert!(
err.kind() == std::io::ErrorKind::InvalidData
|| err.get_ref().is_some_and(|err| { err.is::<ReplicationMultipartPlanError>() }),
"the failure must preserve a typed metadata or planner error: {err}"
);
}
}
#[cfg(target_pointer_width = "64")]
#[test]
fn multipart_read_plan_rejects_physical_size_overflow() {
for raw in [false, true] {
let err = multipart_replication_read_plan(
&ObjectInfo::default(),
&ObjectOptions {
raw_data_movement_read: raw,
..Default::default()
},
ReplicationMultipartPartInput {
offset: 0,
part_number: 1,
part_size: 0,
},
usize::MAX,
true,
)
.expect_err("a physical size outside the range API must be rejected before casting");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert_eq!(err.to_string(), "multipart replication stored part size exceeds i64");
}
}
#[test]
fn same_state_terminal_retry_uses_validate_only() {
@@ -6339,4 +6554,326 @@ mod tests {
"one target's report must not silence another's"
);
}
mod multipart_transport_tests {
use super::super::super::replication_filemeta_boundary::ObjectPartInfo;
use super::super::super::replication_storage_boundary::ObjectIO as _;
use super::*;
use bytes::Bytes;
use http_body_util::{BodyExt, Full};
use std::convert::Infallible;
#[derive(Debug)]
struct Source {
body: Bytes,
info: ObjectInfo,
ranges: StdMutex<Vec<(i64, i64)>>,
full_reads: std::sync::atomic::AtomicUsize,
}
#[async_trait::async_trait]
impl super::super::super::replication_storage_boundary::ObjectIO for Source {
type Error = Error;
type RangeSpec = HTTPRangeSpec;
type HeaderMap = HeaderMap;
type ObjectOptions = ObjectOptions;
type ObjectInfo = ObjectInfo;
type GetObjectReader = GetObjectReader;
type PutObjectReader = super::super::super::replication_storage_boundary::PutObjReader;
async fn get_object_reader(
&self,
_bucket: &str,
_object: &str,
range: Option<HTTPRangeSpec>,
_headers: HeaderMap,
opts: &ObjectOptions,
) -> Result<GetObjectReader> {
assert_eq!(
opts.version_id,
self.info.version_id.map(|id| id.to_string()),
"every read retains the selected source version"
);
if range.is_none() {
self.full_reads.fetch_add(1, Ordering::Relaxed);
return Ok(GetObjectReader {
stream: Box::new(std::io::Cursor::new(self.body.clone())),
object_info: self.info.clone(),
buffered_body: None,
body_source: Default::default(),
});
}
let range = range.expect("multipart transport must request an explicit nonempty range");
assert!(!range.is_suffix_length);
assert!(range.start <= range.end, "empty parts must not issue an inverted range");
self.ranges.lock().expect("range journal lock").push((range.start, range.end));
let start = usize::try_from(range.start).expect("nonnegative start");
let end = usize::try_from(range.end).expect("nonnegative end");
let body = self.body.slice(start..=end);
Ok(GetObjectReader {
stream: Box::new(std::io::Cursor::new(body)),
object_info: self.info.clone(),
buffered_body: None,
body_source: Default::default(),
})
}
async fn put_object(
&self,
_bucket: &str,
_object: &str,
_data: &mut Self::PutObjectReader,
_opts: &ObjectOptions,
) -> Result<ObjectInfo> {
panic!("replication must not overwrite its source")
}
}
#[derive(Debug)]
struct RequestRecord {
method: http::Method,
query: HashMap<String, String>,
headers: HeaderMap,
body: Bytes,
}
#[tokio::test]
async fn multipart_transport_preserves_legacy_zero_actual_sizes() {
run_transport(4096, None).await;
}
#[tokio::test]
async fn multipart_transport_uploads_an_empty_last_part_without_reading_a_range() {
run_transport(0, None).await;
}
#[tokio::test]
async fn multipart_transport_preserves_transformed_unknown_nonempty_parts() {
for unknown_part in [(0, 0), (1, 0), (0, -1), (1, -1)] {
run_transport(4096, Some(unknown_part)).await;
}
}
#[tokio::test]
async fn multipart_transport_preserves_transformed_empty_tail() {
run_transport(0, Some((1, 0))).await;
}
async fn run_transport(tail_size: usize, unknown_part: Option<(usize, i64)>) {
const FIRST_SIZE: usize = 5 * 1024 * 1024;
let body = Bytes::from([vec![0x35; FIRST_SIZE], vec![0xa7; tail_size]].concat());
let etag = faster_hex::hex_string(rustfs_utils::hash::HashAlgorithm::Md5.hash_encode(&body).as_ref());
let source = Arc::new(Source {
info: ObjectInfo {
size: i64::try_from(body.len() + if unknown_part.is_some() { 16 } else { 0 }).expect("stored size"),
actual_size: i64::try_from(body.len()).expect("body size"),
etag: Some(etag.clone()),
version_id: Some(Uuid::new_v4()),
user_defined: Arc::new(if unknown_part.is_some() {
HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())])
} else {
HashMap::new()
}),
parts: Arc::new(vec![
ObjectPartInfo {
number: 1,
size: FIRST_SIZE + if unknown_part.is_some() { 8 } else { 0 },
actual_size: if let Some((0, size)) = unknown_part {
size
} else if unknown_part.is_some() || tail_size == 0 {
i64::try_from(FIRST_SIZE).expect("first part size")
} else {
0
},
..Default::default()
},
ObjectPartInfo {
number: 2,
size: tail_size + if unknown_part.is_some() { 8 } else { 0 },
actual_size: if let Some((1, size)) = unknown_part {
size
} else if unknown_part.is_some() {
i64::try_from(tail_size).expect("tail logical size")
} else {
0
},
..Default::default()
},
]),
..Default::default()
},
body: body.clone(),
ranges: StdMutex::new(Vec::new()),
full_reads: std::sync::atomic::AtomicUsize::new(0),
});
let journal = Arc::new(StdMutex::new(Vec::<RequestRecord>::new()));
let listener = tokio::net::TcpListener::bind(("127.0.0.1", 0))
.await
.expect("bind multipart target");
let endpoint = format!("http://{}", listener.local_addr().expect("multipart target address"));
let server_journal = journal.clone();
let server = tokio::spawn(async move {
let mut connections = JoinSet::new();
loop {
let (stream, _) = listener.accept().await.expect("accept multipart request");
let journal = server_journal.clone();
connections.spawn(async move {
let service = hyper::service::service_fn(move |request: hyper::Request<hyper::body::Incoming>| {
let journal = journal.clone();
async move {
let (request, body) = request.into_parts();
let query: HashMap<String, String> = url::form_urlencoded::parse(
request.uri.query().unwrap_or_default().as_bytes(),
).into_owned().collect();
let body = body.collect().await.expect("read complete multipart request body").to_bytes();
let response = if request.method == http::Method::POST && query.contains_key("uploads") {
"<InitiateMultipartUploadResult><Bucket>target-bucket</Bucket><Key>object</Key><UploadId>upload-1</UploadId></InitiateMultipartUploadResult>"
} else if request.method == http::Method::PUT {
""
} else if request.method == http::Method::POST && query.contains_key("uploadId") {
"<CompleteMultipartUploadResult><Location>http://localhost/object</Location><Bucket>target-bucket</Bucket><Key>object</Key><ETag>&quot;target-2&quot;</ETag></CompleteMultipartUploadResult>"
} else if request.method == http::Method::DELETE && query.contains_key("uploadId") {
""
} else {
panic!("unexpected multipart request: {} {}", request.method, request.uri)
};
let response_etag = if request.method == http::Method::PUT && !query.contains_key("partNumber") {
format!("\"{}\"", faster_hex::hex_string(rustfs_utils::hash::HashAlgorithm::Md5.hash_encode(&body).as_ref()))
} else {
"\"uploaded-part\"".to_string()
};
journal.lock().expect("request journal lock").push(RequestRecord {
method: request.method, query, headers: request.headers, body,
});
Ok::<_, Infallible>(hyper::Response::builder()
.header("content-type", "application/xml")
.header("etag", response_etag)
.body(Full::new(Bytes::from_static(response.as_bytes())))
.expect("multipart response"))
}
});
hyper::server::conn::http1::Builder::new()
.serve_connection(hyper_util::rt::TokioIo::new(stream), service)
.await.expect("serve multipart connection");
});
}
});
let mut target = test_target_client(endpoint);
let config = target
.client
.config()
.to_builder()
.request_checksum_calculation(aws_sdk_s3::config::RequestChecksumCalculation::WhenRequired)
.force_path_style(true)
.build();
Arc::get_mut(&mut target).expect("unshared test target").client = Arc::new(aws_sdk_s3::Client::from_conf(config));
let (put_opts, is_multipart) = replication_put_object_options("STANDARD", &source.info).expect("replication options");
let opts = ObjectOptions {
version_id: source.info.version_id.map(|id| id.to_string()),
..Default::default()
};
let reader = source
.get_object_reader("source", "object", None, HeaderMap::new(), &opts)
.await
.expect("open the existing full-object stream");
let result = tokio::time::timeout(
std::time::Duration::from_secs(30),
replicate_all_payload_to_target(
ReplicateAllPayloadContext {
storage: &source,
tgt_client: &target,
bucket: "source",
object: "object",
object_info: &source.info,
obj_opts: &opts,
arn: &target.arn,
transfer_size: i64::try_from(body.len()).expect("plaintext size"),
is_multipart,
put_opts,
},
reader,
),
)
.await;
server.abort();
assert!(server.await.expect_err("fixture server is stopped").is_cancelled());
if let Some(error) = result.expect("replication must finish") {
panic!("legacy parts must replicate successfully: {error}");
}
assert_eq!(
source.full_reads.load(Ordering::Relaxed),
1,
"reuse the initial full stream without an extra read"
);
if unknown_part.is_some() {
let requests = journal.lock().expect("request journal lock");
assert_eq!(requests.len(), 1, "unknown transformed boundaries retain one streaming PUT");
let request = &requests[0];
assert_eq!(request.method, http::Method::PUT);
let source_version = source.info.version_id.map(|id| id.to_string()).expect("versioned fixture");
assert_eq!(
request.query,
HashMap::from([
("x-id".to_string(), "PutObject".to_string()),
("versionId".to_string(), source_version.clone()),
]),
"single PUT carries only the SDK operation query and the source versionId the target must reuse"
);
assert_eq!(request.body, body, "single PUT includes every byte of both source parts");
assert_eq!(
request.headers.get("content-length").expect("body length"),
body.len().to_string().as_str()
);
assert_eq!(
rustfs_utils::http::get_header(&request.headers, rustfs_utils::http::SUFFIX_SOURCE_ETAG).as_deref(),
Some(etag.as_str())
);
assert_eq!(
rustfs_utils::http::get_header(&request.headers, rustfs_utils::http::SUFFIX_SOURCE_VERSION_ID)
.map(|value| value.into_owned()),
Some(source_version),
"single PUT preserves the selected source version"
);
assert!(
source.ranges.lock().expect("range journal lock").is_empty(),
"unknown logical boundaries must not issue guessed ranges"
);
return;
}
let requests = journal.lock().expect("request journal lock");
assert_eq!(requests.len(), 4, "initiate, two upload parts, and complete without retries");
assert!(requests[0].query.contains_key("uploads"));
for (index, expected) in [(1, body.slice(..FIRST_SIZE)), (2, body.slice(FIRST_SIZE..))] {
assert_eq!(requests[index].method, http::Method::PUT);
assert_eq!(requests[index].query.get("partNumber"), Some(&index.to_string()));
assert_eq!(requests[index].body, expected, "upload part contains the exact source range");
assert_eq!(
requests[index].headers.get("content-length").expect("part content length"),
expected.len().to_string().as_str()
);
}
let complete = &requests[3];
assert_eq!(complete.method, http::Method::POST);
assert_eq!(
rustfs_utils::http::get_header(&complete.headers, rustfs_utils::http::SUFFIX_SOURCE_ETAG).as_deref(),
Some(etag.as_str())
);
let complete_xml = std::str::from_utf8(&complete.body).expect("complete XML");
assert_eq!(
complete_xml.matches("<Part>").count(),
2,
"the empty final part must remain in the completion list"
);
assert!(complete_xml.contains("<PartNumber>1</PartNumber>"));
assert!(complete_xml.contains("<PartNumber>2</PartNumber>"));
let mut expected_ranges = vec![(0, i64::try_from(FIRST_SIZE - 1).expect("first end"))];
if tail_size > 0 {
expected_ranges.push((
i64::try_from(FIRST_SIZE).expect("tail start"),
i64::try_from(body.len() - 1).expect("tail end"),
));
}
assert_eq!(*source.ranges.lock().expect("range journal lock"), expected_ranges);
}
}
}
@@ -248,7 +248,16 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
meta.insert(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "aws:kms".to_string());
}
let mut is_multipart = object_info.is_multipart();
// Older transformed objects can have physical parts without logical part
// lengths. Keep their existing whole-object transport: physical sizes are
// not plaintext boundaries for a multipart replication read.
let legacy_single_put = object_info.etag.as_deref().is_none_or(|etag| etag.len() == 32);
let base_is_multipart = object_info.is_multipart()
&& !(legacy_single_put
&& object_info.parts.len() > 1
&& (object_info.is_compressed() || object_info.is_encrypted())
&& object_info.parts.iter().any(|part| part.actual_size <= 0));
let mut is_multipart = base_is_multipart;
if let Some(checksum_data) = &object_info.checksum
&& !checksum_data.is_empty()
@@ -259,8 +268,8 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
} else if object_info.is_encrypted() {
// Encrypted checksums cannot be exposed as plaintext headers, and
// decrypt_checksums reports is_multipart=false for them (a value
// the response path relies on). Keep the object's own multipart
// flag so encrypted objects stay on the multipart route.
// the response path relies on). Keep the transport selected from
// the object's layout and readable part boundaries.
} else {
let (checksum_meta, checksum_record_is_multipart) = object_info.decrypt_checksums(0, &HeaderMap::new())?;
// The checksum record describes how the *checksum* is composed,
@@ -268,9 +277,9 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
// MULTIPART flag even on a multipart upload, so trusting it here
// routed a 768-part object through a single PutObject and the
// target rejected the 6 GiB body with EntityTooLarge
// (rustfs#6825). The object's own shape is the authority: the
// (rustfs#6825). The usable part layout is the authority: the
// record may only add multipart-ness, never take it away.
is_multipart = object_info.is_multipart() || checksum_record_is_multipart;
is_multipart = base_is_multipart || checksum_record_is_multipart;
for (key, value) in checksum_meta.iter() {
if key != AMZ_CHECKSUM_TYPE {
@@ -278,7 +287,7 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
}
}
if !object_info.is_multipart()
if !base_is_multipart
&& checksum_meta
.get(AMZ_CHECKSUM_TYPE)
.is_some_and(|value| value == AMZ_CHECKSUM_TYPE_FULL_OBJECT)
@@ -516,6 +525,7 @@ fn is_standard_header(key: &str) -> bool {
#[cfg(test)]
mod tests {
use super::super::replication_filemeta_boundary::ObjectPartInfo;
use super::*;
use aws_smithy_types::DateTime;
use rustfs_replication::content_matches_by_etag;
@@ -550,6 +560,109 @@ mod tests {
checksum.to_bytes(&combined)
}
fn replication_route_metadata() -> [(&'static str, Arc<HashMap<String, String>>); 4] {
let mut compressed = HashMap::new();
rustfs_utils::http::insert_str(&mut compressed, rustfs_utils::http::SUFFIX_COMPRESSION, "zstd".to_string());
[
("plain", Arc::new(HashMap::new())),
("compressed", Arc::new(compressed)),
(
"encrypted",
Arc::new(HashMap::from([(AMZ_SERVER_SIDE_ENCRYPTION.to_string(), "AES256".to_string())])),
),
(
"ssec",
Arc::new(HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())])),
),
]
}
fn replication_route_object(
etag: Option<&str>,
actual_sizes: [i64; 3],
metadata: Arc<HashMap<String, String>>,
) -> ObjectInfo {
ObjectInfo {
etag: etag.map(str::to_string),
size: 48,
actual_size: 12,
user_defined: metadata,
parts: Arc::new(
actual_sizes
.into_iter()
.enumerate()
.map(|(index, actual_size)| ObjectPartInfo {
number: index + 1,
size: 16,
actual_size,
..Default::default()
})
.collect(),
),
..Default::default()
}
}
#[test]
fn legacy_transformed_single_put_parts_keep_the_previous_replication_route() {
let [_, (_, compressed), (_, encrypted), (_, ssec)] = replication_route_metadata();
let cases = [
(
"compressed middle zero",
compressed.clone(),
Some("0123456789abcdef0123456789abcdef"),
[4, 0, 4],
),
("compressed tail unknown", compressed, None, [4, 4, -1]),
(
"encrypted middle unknown",
encrypted,
Some("gggggggggggggggggggggggggggggggg"),
[4, -1, 4],
),
("ssec tail zero", ssec.clone(), None, [4, 4, 0]),
("ssec middle unknown", ssec, Some("gggggggggggggggggggggggggggggggg"), [4, -1, 4]),
];
for (name, metadata, etag, actual_sizes) in cases {
for checksum in [None, Some(full_object_multipart_checksum_record())] {
let mut object_info = replication_route_object(etag, actual_sizes, metadata.clone());
object_info.checksum = checksum;
assert!(object_info.is_multipart(), "{name}: physical parts remain visible to metadata APIs");
assert!(object_info.is_compressed() || object_info.is_encrypted());
let (options, is_multipart) =
replication_put_object_options("STANDARD", &object_info).expect("legacy transformed put options");
assert!(
!is_multipart,
"{name}: unknown logical part sizes must preserve the old whole-object route"
);
assert_eq!(options.internal.source_etag, etag.unwrap_or_default());
if metadata.contains_key(SSEC_ALGORITHM_HEADER) {
assert_eq!(
get_header_map(&options.user_metadata, SUFFIX_REPLICATION_SSEC_CRC).is_some(),
object_info.checksum.is_some(),
"SSE-C checksums retain their raw passthrough transport"
);
}
}
}
}
#[test]
fn positive_part_sizes_and_legacy_multipart_etags_keep_the_replication_route() {
for (name, metadata) in replication_route_metadata() {
for (etag, actual_sizes) in [
("0123456789abcdef0123456789abcdef", [4, 4, 4]),
("0123456789abcdef0123456789abcdef-3", [4, 0, -1]),
] {
let mut object_info = replication_route_object(Some(etag), actual_sizes, metadata.clone());
object_info.checksum = Some(full_object_multipart_checksum_record());
let (_, is_multipart) = replication_put_object_options("STANDARD", &object_info).expect("multipart put options");
assert!(is_multipart, "{name}/{etag}: usable sizes and old multipart ETags must retain MPU");
}
}
}
#[test]
fn multipart_object_with_full_object_checksum_keeps_the_multipart_route() {
// rustfs#6825: a 768-part upload was replicated with a single
@@ -582,6 +695,36 @@ mod tests {
);
}
#[test]
fn stored_multipart_parts_keep_the_replication_route_without_a_multipart_etag() {
for etag in [Some("0123456789abcdef0123456789abcdef"), None] {
for checksum in [None, Some(full_object_multipart_checksum_record())] {
let object_info = ObjectInfo {
etag: etag.map(str::to_string),
checksum,
parts: Arc::new(
(1..=2)
.map(|number| ObjectPartInfo {
number,
..Default::default()
})
.collect(),
),
..Default::default()
};
let (options, is_multipart) =
replication_put_object_options("STANDARD", &object_info).expect("build put options");
assert!(
is_multipart,
"stored parts must retain multipart routing: etag={etag:?}, checksum={:?}",
object_info.checksum
);
assert_eq!(options.internal.source_etag, etag.unwrap_or_default());
}
}
}
#[test]
fn checksum_record_never_changes_the_transport_a_single_part_object_needs() {
// The mirror of the rustfs#6825 guard: an object stored as one PUT
@@ -592,6 +735,10 @@ mod tests {
let object_info = ObjectInfo {
etag: Some("0123456789abcdef0123456789abcdef".to_string()),
checksum: Some(checksum.to_bytes(&[])),
parts: Arc::new(vec![ObjectPartInfo {
number: 1,
..Default::default()
}]),
..Default::default()
};
@@ -628,6 +775,19 @@ mod tests {
let (_, is_multipart) = replication_put_object_options("STANDARD", &object_info).expect("build put options");
assert!(is_multipart, "a composite-checksum multipart object must stay on the multipart transport");
for (name, metadata) in replication_route_metadata() {
let mut legacy = replication_route_object(Some("0123456789abcdef0123456789abcdef"), [4, 0, 4], metadata);
legacy.checksum = Some(checksum.to_bytes(&combined));
let (_, record_is_multipart) = legacy.decrypt_checksums(0, &HeaderMap::new()).expect("decode checksum");
let (_, is_multipart) = replication_put_object_options("STANDARD", &legacy).expect("legacy checksum put options");
if legacy.is_encrypted() {
assert!(!is_multipart, "{name}: encrypted checksum records must not change the old transport");
} else {
assert!(record_is_multipart, "the composite checksum must carry its own multipart signal");
assert!(is_multipart, "{name}: a composite record can still promote the legacy route to MPU");
}
}
}
#[test]
@@ -1688,6 +1688,15 @@ impl PeerRestClient {
Ok((self.topology_member.clone(), supported_version, epoch))
}
pub async fn probe_ilm_recovery_export(&self, topology_fingerprint: String) -> Result<(String, Uuid)> {
let probe = rustfs_protos::ilm_recovery_export_capability_probe(Uuid::new_v4().as_bytes());
let result = self
.heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
.await?;
let epoch = decode_remote_version_state_capability(&self.topology_member, &result)?;
Ok((self.topology_member.clone(), epoch))
}
pub async fn load_bucket_metadata(&self, bucket: &str, scanner_maintenance_change: bool) -> Result<()> {
let result = tokio::time::timeout(BUCKET_METADATA_RELOAD_TIMEOUT, async {
let result = self.load_bucket_metadata_once(bucket, scanner_maintenance_change).await;
+20 -8
View File
@@ -449,14 +449,6 @@ where
Ok(data)
}
pub(crate) async fn read_config_limited<S>(api: Arc<S>, file: &str, max_bytes: usize) -> Result<Vec<u8>>
where
S: EcstoreObjectIO,
{
let (data, _obj) = read_config_with_metadata_inner(api, file, &ObjectOptions::default(), false, Some(max_bytes)).await?;
Ok(data)
}
pub(crate) async fn read_config_limited_preserve_empty<S>(api: Arc<S>, file: &str, max_bytes: usize) -> Result<Vec<u8>>
where
S: EcstoreObjectIO,
@@ -465,6 +457,14 @@ where
Ok(data)
}
pub(crate) async fn read_config_limited<S>(api: Arc<S>, file: &str, max_bytes: usize) -> Result<Vec<u8>>
where
S: EcstoreObjectIO,
{
let (data, _obj) = read_config_with_metadata_inner(api, file, &ObjectOptions::default(), false, Some(max_bytes)).await?;
Ok(data)
}
pub(crate) async fn read_config_limited_preserve_empty_with_metadata<S>(
api: Arc<S>,
file: &str,
@@ -476,6 +476,18 @@ where
read_config_with_metadata_inner(api, file, &ObjectOptions::default(), true, Some(max_bytes)).await
}
pub(crate) async fn read_config_limited_preserve_empty_with_metadata_opts<S>(
api: Arc<S>,
file: &str,
opts: &ObjectOptions,
max_bytes: usize,
) -> Result<(Vec<u8>, ObjectInfo)>
where
S: EcstoreObjectIO,
{
read_config_with_metadata_inner(api, file, opts, true, Some(max_bytes)).await
}
/// Read an existing config object without treating an empty payload as absent.
/// Callers that validate their own payload format need to distinguish corruption
/// from `ConfigNotFound`.
+1 -1
View File
@@ -3385,7 +3385,7 @@ pub(crate) async fn acquire_pool_activation_fleet_proof(
.ok_or_else(|| Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED))
}
pub(crate) fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
pub fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
// Save-stage helpers add context by formatting the original error, so the
// marker may be nested in the display string. Restrict matching to the
// `Error::other` I/O shape used by this activation path.
+2 -2
View File
@@ -2673,7 +2673,7 @@ mod tests {
]),
..Default::default()
};
assert!(!object_info.is_multipart());
assert!(object_info.is_multipart());
assert!(should_use_multipart_data_movement(&object_info, false));
let single_nonstandard_part = ObjectInfo {
@@ -3050,7 +3050,7 @@ mod tests {
..Default::default()
};
assert!(!object_info.is_multipart());
assert!(object_info.is_multipart());
assert!(object_info.parts.iter().any(|part| part.checksums.is_some()));
let opts = data_movement_put_object_opts(&object_info, 0);
assert!(!rustfs_utils::http::contains_key_str(&opts.user_defined, SUFFIX_PART_CHECKSUMS));
+170
View File
@@ -2278,6 +2278,121 @@ mod tests {
assert_eq!(read, fixture.plaintext, "SSE-C + compression full GET must reassemble all parts");
}
#[tokio::test]
async fn multipart_empty_tail_full_reads_preserve_plaintext() {
let key = [0x6Eu8; 32];
let part_sizes = [5 * 1024 * 1024, 0];
let encrypted = build_legacy_ssec_multipart_fixture(key, &part_sizes).await;
for (kind, mut fixture, headers) in [
(
"encrypted",
CompressedMultipartFixture {
object_info: encrypted.object_info,
stored: encrypted.ciphertext,
plaintext: encrypted.plaintext,
},
ssec_headers_from_key(key),
),
("compressed", compressed_multipart_fixture(&part_sizes).await, HeaderMap::new()),
(
"compressed and encrypted",
compressed_encrypted_multipart_fixture(key, &part_sizes).await,
ssec_headers_from_key(key),
),
] {
fixture.object_info.etag = Some(faster_hex::hex_string(Md5::digest(&fixture.plaintext).as_ref()));
assert_eq!(fixture.object_info.etag.as_ref().expect("source ETag").len(), 32);
assert_eq!(fixture.object_info.parts.len(), 2);
let tail = &fixture.object_info.parts[1];
assert_eq!(tail.actual_size, 0, "{kind}: final part has no plaintext");
if kind == "compressed" {
assert_eq!(tail.size, 0, "unpadded compression emits no bytes for an empty part");
} else {
assert!(tail.size > 0, "{kind}: the empty part still has a stored frame");
}
let stored_size = i64::try_from(fixture.stored.len()).expect("fixture size fits i64");
let (mut reader, offset, length) = GetObjectReader::new(
Box::new(Cursor::new(fixture.stored)),
None,
&fixture.object_info,
&ObjectOptions::default(),
&headers,
)
.await
.expect("full transformed read must include the empty tail");
assert_eq!((offset, length), (0, stored_size), "{kind}: full read includes all stored parts");
let mut body = Vec::new();
reader
.stream
.read_to_end(&mut body)
.await
.expect("read through the complete decoder EOF");
assert_eq!(body, fixture.plaintext, "{kind}: no plaintext is added or lost by the empty tail");
}
}
#[tokio::test]
async fn multipart_empty_tail_full_read_authenticates_v2_final_frame() {
let key = [0x6Eu8; 32];
let plaintext = legacy_fixture_part_plaintext(1, 5 * 1024 * 1024);
let mut ciphertext = Vec::new();
let mut parts = Vec::new();
for (number, body) in [(1, plaintext.as_slice()), (2, b"".as_slice())] {
let start = ciphertext.len();
rustfs_rio::EncryptReader::new_multipart_v2(Cursor::new(body), key, LEGACY_FIXTURE_BASE_NONCE, number)
.read_to_end(&mut ciphertext)
.await
.expect("encrypt a v2 fixture part with an authenticated final frame");
parts.push(ObjectPartInfo {
number,
size: ciphertext.len() - start,
actual_size: i64::try_from(body.len()).expect("fixture plaintext size fits"),
..Default::default()
});
}
let tail_start = parts[0].size;
assert_eq!(parts[1].actual_size, 0);
assert!(parts[1].size > 8, "the empty final frame carries more than an END marker");
let object_info = ObjectInfo {
bucket: "bucket".to_string(),
name: "v2-empty-tail".to_string(),
size: i64::try_from(ciphertext.len()).expect("fixture ciphertext size fits"),
etag: Some(faster_hex::hex_string(Md5::digest(&plaintext).as_ref())),
parts: Arc::new(parts),
user_defined: Arc::new(legacy_ssec_multipart_metadata(key, plaintext.len())),
..Default::default()
};
for corrupt_tail in [false, true] {
let mut stored = ciphertext.clone();
if corrupt_tail {
// The v2 header is authenticated associated data, including
// the header of a final frame containing zero plaintext.
stored[tail_start + 5] ^= 1;
}
let (mut reader, offset, length) = GetObjectReader::new(
Box::new(Cursor::new(stored)),
None,
&object_info,
&ObjectOptions::default(),
&ssec_headers_from_key(key),
)
.await
.expect("construct the full reader before consuming the final frame");
assert_eq!((offset, length), (0, object_info.size));
let result = tokio::io::copy(&mut reader.stream, &mut tokio::io::sink()).await;
if corrupt_tail {
let err = result.expect_err("EOF must authenticate the empty final frame after all plaintext is returned");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert_eq!(err.to_string(), "v2 encrypted frame failed authentication");
} else {
assert_eq!(
result.expect("valid empty final frame must reach EOF"),
u64::try_from(plaintext.len()).expect("plaintext length fits")
);
}
}
}
#[tokio::test]
async fn compressed_encrypted_multipart_range_crosses_part_boundary() {
let key_bytes = [0x6Eu8; 32];
@@ -3656,6 +3771,61 @@ mod tests {
.await;
}
#[tokio::test]
async fn multipart_full_read_preserves_legacy_zero_and_negative_part_sizes() {
let key = [0x77; 32];
let part_sizes = [5 * 1024 * 1024, 1024 * 1024];
let encrypted = build_legacy_ssec_multipart_fixture(key, &part_sizes).await;
// The encrypted case supplies the fixture key explicitly. This covers
// full decrypted reads, not managed-key acquisition.
for (kind, fixture, headers) in [
("compressed", compressed_multipart_fixture(&part_sizes).await, HeaderMap::new()),
(
"encrypted with supplied key",
CompressedMultipartFixture {
object_info: encrypted.object_info,
stored: encrypted.ciphertext,
plaintext: encrypted.plaintext,
},
ssec_headers_from_key(key),
),
] {
let source_etag = faster_hex::hex_string(Md5::digest(&fixture.plaintext).as_ref());
assert_eq!(source_etag.len(), 32);
assert_eq!(fixture.plaintext.len(), 6 * 1024 * 1024);
for part_index in 0..part_sizes.len() {
assert!(fixture.object_info.parts[part_index].actual_size > 0, "the selected part is nonempty");
for actual_size in [0, -1] {
let mut object_info = fixture.object_info.clone();
object_info.etag = Some(source_etag.clone());
Arc::make_mut(&mut object_info.parts)[part_index].actual_size = actual_size;
let (mut reader, offset, length) = GetObjectReader::new(
Box::new(Cursor::new(fixture.stored.clone())),
None,
&object_info,
&ObjectOptions::default(),
&headers,
)
.await
.expect("the authoritative total size must keep full legacy reads available");
assert_eq!(offset, 0);
assert_eq!(length, i64::try_from(fixture.stored.len()).expect("stored size fits"));
let mut body = Vec::new();
reader
.stream
.read_to_end(&mut body)
.await
.expect("full read must reach EOF despite an unspecified per-part logical size");
assert_eq!(
body, fixture.plaintext,
"{kind}: part {part_index} with actual_size={actual_size} must not lose readable data"
);
assert_eq!(reader.object_info.etag.as_deref(), Some(source_etag.as_str()));
}
}
}
}
/// The physical part sizes must add up to `oi.size` for a seek to be safe;
/// inconsistent metadata must fall back to the previous full-object read
/// instead of scheduling an erasure read past the object end.
+30 -1
View File
@@ -1597,7 +1597,7 @@ impl ObjectInfo {
}
pub fn is_multipart(&self) -> bool {
self.etag.as_ref().is_some_and(|v| v.len() != 32)
self.parts.len() > 1 || self.etag.as_ref().is_some_and(|v| v.len() != 32)
}
pub fn is_encrypted(&self) -> bool {
@@ -2235,6 +2235,35 @@ mod tests {
}
use rustfs_filemeta::{FileInfo, FileMeta, MetaCacheEntry, TRANSITION_COMPLETE};
#[test]
fn multipart_identity_uses_stored_parts_and_preserves_the_etag_fallback() {
let plain_etag = "0123456789abcdef0123456789abcdef";
let multipart_etag = "0123456789abcdef0123456789abcdef-1";
for (case, part_count, etag, expected) in [
("preserved source ETag", 2, Some(plain_etag), true),
("missing ETag", 2, None, true),
("ordinary PUT", 1, Some(plain_etag), false),
("ordinary PUT without ETag", 1, None, false),
("single-part MPU", 1, Some(multipart_etag), true),
("legacy MPU without parts", 0, Some(multipart_etag), true),
] {
let object = ObjectInfo {
etag: etag.map(str::to_string),
parts: Arc::new(
(1..=part_count)
.map(|number| ObjectPartInfo {
number,
..Default::default()
})
.collect(),
),
..Default::default()
};
assert_eq!(object.is_multipart(), expected, "{case}");
}
}
fn inline_fast_path_object(size: i64, versioned: bool) -> ObjectInfo {
ObjectInfo {
size,
+305 -19
View File
@@ -33,11 +33,11 @@ use rustfs_madmin::net::NetInfo;
use rustfs_madmin::{ItemState, ServerProperties, StorageInfo};
use rustfs_utils::XHost;
use sha2::{Digest, Sha256};
use std::collections::{BTreeMap, HashMap, hash_map::DefaultHasher};
use std::collections::{BTreeMap, BTreeSet, HashMap, hash_map::DefaultHasher};
use std::future::Future;
use std::hash::{Hash, Hasher};
use std::sync::{
Arc, Mutex, OnceLock,
Arc, LazyLock, Mutex, OnceLock,
atomic::{AtomicBool, AtomicUsize, Ordering},
};
use std::time::{Duration, Instant, SystemTime};
@@ -311,12 +311,20 @@ pub struct LegacyTransitionStateReconcileFleetProofToken {
_permit: FleetCapabilityProofPermit,
}
/// Effect-window authority for one immutable ILM recovery export.
pub struct IlmRecoveryExportFleetProofToken {
token: FleetCapabilityProofToken,
_permit: FleetCapabilityProofPermit,
}
static REMOTE_VERSION_STATE_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
static CROSS_POOL_FENCE_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
static TIER_DELETE_JOURNAL_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
static DECOMMISSION_TARGET_FENCE_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
static LEGACY_TRANSITION_STATE_RECONCILE_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
static ILM_RECOVERY_EXPORT_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
static REMOTE_VERSION_STATE_PROBE_TOPOLOGY: OnceLock<String> = OnceLock::new();
static ILM_RECOVERY_EXPORT_LOCAL_PROCESS_EPOCH: LazyLock<Uuid> = LazyLock::new(Uuid::new_v4);
fn cross_pool_fence_fleet_proof_slot() -> &'static std::sync::RwLock<FleetCapabilityProofState> {
CROSS_POOL_FENCE_FLEET_PROOF.get_or_init(|| std::sync::RwLock::new(FleetCapabilityProofState::default()))
@@ -338,6 +346,10 @@ fn legacy_transition_state_reconcile_fleet_proof_slot() -> &'static std::sync::R
LEGACY_TRANSITION_STATE_RECONCILE_FLEET_PROOF.get_or_init(|| std::sync::RwLock::new(FleetCapabilityProofState::default()))
}
fn ilm_recovery_export_fleet_proof_slot() -> &'static std::sync::RwLock<FleetCapabilityProofState> {
ILM_RECOVERY_EXPORT_FLEET_PROOF.get_or_init(|| std::sync::RwLock::new(FleetCapabilityProofState::default()))
}
fn revoke_fleet_capability_proof_state(state: &mut FleetCapabilityProofState) {
if let Some(proof) = state.proof.take() {
proof.generation.revoke();
@@ -573,6 +585,117 @@ pub async fn legacy_transition_state_reconcile_fleet_proof_matches(
.await
}
pub async fn acquire_ilm_recovery_export_fleet_proof() -> Option<IlmRecoveryExportFleetProofToken> {
let expected_topology = REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get()?;
let proof = {
let state = ilm_recovery_export_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
acquire_ilm_recovery_export_fleet_proof_from(&state, expected_topology, Instant::now())?
};
let observed = observe_ilm_recovery_export_fleet(expected_topology).await?;
let state = ilm_recovery_export_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
ilm_recovery_export_fleet_proof_matches_observation_at(&state, &proof, expected_topology, &observed, Instant::now())
.then_some(proof)
}
fn acquire_ilm_recovery_export_fleet_proof_from(
state: &FleetCapabilityProofState,
expected_topology: &str,
now: Instant,
) -> Option<IlmRecoveryExportFleetProofToken> {
let token = acquire_fleet_capability_proof_from(state, expected_topology, now)?;
let permit = state.proof.as_ref()?.generation.try_acquire()?;
Some(IlmRecoveryExportFleetProofToken { token, _permit: permit })
}
pub async fn ilm_recovery_export_fleet_proof_matches(proof: &IlmRecoveryExportFleetProofToken) -> bool {
let Some(expected_topology) = REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get() else {
return false;
};
{
let state = ilm_recovery_export_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if !ilm_recovery_export_fleet_proof_matches_at(&state, proof, expected_topology, Instant::now()) {
return false;
}
}
let Some(observed) = observe_ilm_recovery_export_fleet(expected_topology).await else {
return false;
};
let state = ilm_recovery_export_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
ilm_recovery_export_fleet_proof_matches_observation_at(&state, proof, expected_topology, &observed, Instant::now())
}
pub fn ilm_recovery_export_topology_generation(proof: &IlmRecoveryExportFleetProofToken) -> String {
let mut hasher = Sha256::new();
hasher.update(b"rustfs-ilm-recovery-export-topology-v1\0");
hasher.update(proof.token.topology_fingerprint.as_bytes());
rustfs_utils::crypto::hex(hasher.finalize().as_slice())
}
pub fn ilm_recovery_export_member_epochs_sha256(proof: &IlmRecoveryExportFleetProofToken) -> String {
let encoded = serde_json::to_vec(proof.token.peer_epochs.as_ref()).expect("member epoch map is JSON encodable");
let mut hasher = Sha256::new();
hasher.update(b"rustfs-ilm-recovery-export-members-v1\0");
hasher.update(encoded);
rustfs_utils::crypto::hex(hasher.finalize().as_slice())
}
pub fn ilm_recovery_export_local_process_epoch() -> Uuid {
*ILM_RECOVERY_EXPORT_LOCAL_PROCESS_EPOCH
}
fn ilm_recovery_export_fleet_proof_matches_at(
state: &FleetCapabilityProofState,
proof: &IlmRecoveryExportFleetProofToken,
expected_topology: &str,
now: Instant,
) -> bool {
proof._permit.generation.is_accepting()
&& fleet_capability_proof_matches_at(state, &proof.token, expected_topology, now)
&& state
.proof
.as_ref()
.is_some_and(|current| Arc::ptr_eq(&current.generation, &proof._permit.generation))
}
fn ilm_recovery_export_fleet_proof_matches_observation_at(
state: &FleetCapabilityProofState,
proof: &IlmRecoveryExportFleetProofToken,
expected_topology: &str,
observed: &BTreeMap<String, Uuid>,
now: Instant,
) -> bool {
ilm_recovery_export_fleet_proof_matches_at(state, proof, expected_topology, now)
&& proof.token.peer_epochs.as_ref() == observed
}
async fn observe_ilm_recovery_export_fleet(expected_topology: &str) -> Option<BTreeMap<String, Uuid>> {
#[cfg(test)]
{
let state = ilm_recovery_export_fleet_proof_slot()
.read()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if fleet_capability_proof_valid_at(state.proof.as_ref(), expected_topology, Instant::now()) {
return state.proof.as_ref().map(|proof| proof.peer_epochs.as_ref().clone());
}
}
let notification_sys = get_global_notification_sys()?;
timeout(
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
notification_sys.probe_ilm_recovery_export_fleet(expected_topology),
)
.await
.ok()?
.ok()
}
async fn legacy_transition_state_reconcile_fleet_proof_matches_with_observer<F, Fut>(
slot: &std::sync::RwLock<FleetCapabilityProofState>,
proof: &LegacyTransitionStateReconcileFleetProofToken,
@@ -661,7 +784,7 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
state.proof.clone()
} else {
Some(FleetCapabilityProof::new(
topology,
topology.clone(),
Arc::new(BTreeMap::new()),
now + Duration::from_secs(60 * 60),
))
@@ -694,6 +817,21 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
decommission_state.topology_conflict = false;
decommission_state.draining_generation = None;
decommission_state.proof = proof.as_ref().map(FleetCapabilityProof::with_fresh_generation);
drop(decommission_state);
let mut export_state = ilm_recovery_export_fleet_proof_slot()
.write()
.unwrap_or_else(std::sync::PoisonError::into_inner);
if !fleet_capability_proof_valid_at(export_state.proof.as_ref(), &topology, now) {
debug_assert!(
export_state
.proof
.as_ref()
.is_none_or(|current| current.generation.is_drained())
);
export_state.topology_conflict = false;
export_state.draining_generation = None;
export_state.proof = proof.as_ref().map(FleetCapabilityProof::with_fresh_generation);
}
}
#[cfg(test)]
@@ -950,6 +1088,7 @@ pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
tier_delete_journal_fleet_proof_slot(),
decommission_target_fence_fleet_proof_slot(),
legacy_transition_state_reconcile_fleet_proof_slot(),
ilm_recovery_export_fleet_proof_slot(),
] {
mark_fleet_capability_topology_conflict(slot);
}
@@ -959,29 +1098,42 @@ pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
tokio::spawn(async move {
loop {
let result = match get_global_notification_sys() {
Some(notification_sys) => {
match timeout(
let notification_sys = get_global_notification_sys();
let remote_version_state_probe = async {
match notification_sys.as_ref() {
Some(notification_sys) => timeout(
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
notification_sys.probe_remote_version_state_fleet(&topology_fingerprint),
)
.await
{
Ok(result) => result,
Err(_) => Err(Error::other("remote version state fleet capability probe timed out")),
}
.unwrap_or_else(|_| Err(Error::other("remote version state fleet capability probe timed out"))),
None => Err(Error::other("remote version state fleet capability notification system is unavailable")),
}
None => Err(Error::other("remote version state fleet capability notification system is unavailable")),
};
let fence_probe = match get_global_notification_sys() {
Some(notification_sys) => timeout(
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
notification_sys.probe_cross_pool_fence_fleet(&topology_fingerprint),
)
.await
.unwrap_or_else(|_| Err(Error::other("cross-pool fence fleet capability probe timed out"))),
None => Err(Error::other("cross-pool fence fleet capability notification system is unavailable")),
let cross_pool_fence_probe = async {
match notification_sys.as_ref() {
Some(notification_sys) => timeout(
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
notification_sys.probe_cross_pool_fence_fleet(&topology_fingerprint),
)
.await
.unwrap_or_else(|_| Err(Error::other("cross-pool fence fleet capability probe timed out"))),
None => Err(Error::other("cross-pool fence fleet capability notification system is unavailable")),
}
};
let recovery_export_probe = async {
match notification_sys.as_ref() {
Some(notification_sys) => timeout(
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
notification_sys.probe_ilm_recovery_export_fleet(&topology_fingerprint),
)
.await
.unwrap_or_else(|_| Err(Error::other("ILM recovery export fleet capability probe timed out"))),
None => Err(Error::other("ILM recovery export fleet capability notification system is unavailable")),
}
};
let (result, fence_probe, recovery_export_result) =
tokio::join!(remote_version_state_probe, cross_pool_fence_probe, recovery_export_probe);
let (fence_result, journal_result, decommission_target_fence_result, reconcile_result) = match fence_probe {
Ok((peer_epochs, minimum_version)) => cross_pool_fence_policy_results(peer_epochs, minimum_version),
Err(err) => {
@@ -1004,6 +1156,7 @@ pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
revoke_fleet_capability_proof(tier_delete_journal_fleet_proof_slot());
revoke_fleet_capability_proof(decommission_target_fence_fleet_proof_slot());
revoke_fleet_capability_proof(legacy_transition_state_reconcile_fleet_proof_slot());
revoke_fleet_capability_proof(ilm_recovery_export_fleet_proof_slot());
} else if let Some(err) = publish_fleet_capability_probe_result(
remote_version_state_fleet_proof_slot(),
&topology_fingerprint,
@@ -1030,6 +1183,24 @@ pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
"notification capability probe"
);
}
if !topology_conflict
&& let Some(err) = publish_fleet_capability_probe_result(
ilm_recovery_export_fleet_proof_slot(),
&topology_fingerprint,
recovery_export_result,
Instant::now(),
)
{
debug!(
event = EVENT_NOTIFICATION_CAPABILITY_PROBE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_NOTIFICATION,
capability = "ilm_recovery_export_v1",
state = "failed_closed",
error = %err,
"notification capability probe"
);
}
if !topology_conflict
&& let Some(err) = publish_fleet_capability_probe_result(
tier_delete_journal_fleet_proof_slot(),
@@ -1174,6 +1345,46 @@ impl NotificationSys {
}
Ok((peer_epochs, minimum_version))
}
async fn probe_ilm_recovery_export_fleet(&self, topology_fingerprint: &str) -> Result<BTreeMap<String, Uuid>> {
if self.peer_clients.len() != self.peer_topology_hosts.len() {
return Err(Error::other("ILM recovery export capability fleet membership is incomplete"));
}
let local_member = runtime_sources::local_node_name().await;
if local_member.trim().is_empty() {
return Err(Error::other("ILM recovery export local member identity is unavailable"));
}
let mut peer_epochs = BTreeMap::new();
insert_remote_version_state_peer(&mut peer_epochs, local_member.clone(), ilm_recovery_export_local_process_epoch())?;
let probes = self.peer_clients.iter().map(|client| async {
let client = client
.as_ref()
.ok_or_else(|| Error::other("ILM recovery export capability peer is unreachable"))?;
client.probe_ilm_recovery_export(topology_fingerprint.to_string()).await
});
for result in join_all(probes).await {
let (peer, epoch) = result?;
insert_remote_version_state_peer(&mut peer_epochs, peer, epoch)?;
}
validate_ilm_recovery_export_members(&self.peer_topology_hosts, &local_member, &peer_epochs)?;
Ok(peer_epochs)
}
}
fn validate_ilm_recovery_export_members(
expected_remote_members: &[String],
local_member: &str,
observed: &BTreeMap<String, Uuid>,
) -> Result<()> {
let expected = expected_remote_members
.iter()
.cloned()
.chain(std::iter::once(local_member.to_string()))
.collect::<BTreeSet<_>>();
if expected.len() != expected_remote_members.len().saturating_add(1) || observed.keys().ne(expected.iter()) {
return Err(Error::other("ILM recovery export capability fleet membership does not match topology"));
}
Ok(())
}
/// Rolling tier activity summed over every cluster member that answered, with
@@ -3507,6 +3718,81 @@ mod tests {
assert!(captured != restarted.token());
}
#[test]
fn ilm_recovery_export_member_digest_is_order_independent_and_epoch_bound() {
let now = Instant::now();
let local_epoch = ilm_recovery_export_local_process_epoch();
assert!(!local_epoch.is_nil());
assert_eq!(local_epoch, ilm_recovery_export_local_process_epoch());
let remote_epoch = Uuid::new_v4();
let slot = std::sync::RwLock::new(FleetCapabilityProofState::default());
let peers = BTreeMap::from([("node-b".to_string(), remote_epoch), ("node-a".to_string(), local_epoch)]);
assert!(publish_fleet_capability_probe_result(&slot, "topology-a", Ok(peers), now).is_none());
let proof = {
let state = slot.read().expect("export proof slot should not poison");
acquire_ilm_recovery_export_fleet_proof_from(&state, "topology-a", now).expect("complete fleet should admit export")
};
let digest = ilm_recovery_export_member_epochs_sha256(&proof);
let changed_slot = std::sync::RwLock::new(FleetCapabilityProofState::default());
let changed = BTreeMap::from([("node-a".to_string(), local_epoch), ("node-b".to_string(), Uuid::new_v4())]);
assert!(publish_fleet_capability_probe_result(&changed_slot, "topology-a", Ok(changed), now).is_none());
let changed_proof = {
let state = changed_slot.read().expect("export proof slot should not poison");
acquire_ilm_recovery_export_fleet_proof_from(&state, "topology-a", now).expect("complete fleet should admit export")
};
assert_ne!(digest, ilm_recovery_export_member_epochs_sha256(&changed_proof));
}
#[test]
fn ilm_recovery_export_members_must_match_the_exact_topology() {
let expected_remote = vec!["node-b".to_string()];
let local = "node-a";
let complete = BTreeMap::from([
(local.to_string(), Uuid::new_v4()),
(expected_remote[0].clone(), Uuid::new_v4()),
]);
assert!(validate_ilm_recovery_export_members(&expected_remote, local, &complete).is_ok());
let unexpected = BTreeMap::from([(local.to_string(), Uuid::new_v4()), ("node-c".to_string(), Uuid::new_v4())]);
assert!(validate_ilm_recovery_export_members(&expected_remote, local, &unexpected).is_err());
assert!(
validate_ilm_recovery_export_members(&[local.to_string()], local, &complete).is_err(),
"the configured remote set cannot repeat the local member"
);
}
#[test]
fn ilm_recovery_export_restart_revokes_authority_until_permit_drains() {
let slot = std::sync::RwLock::new(FleetCapabilityProofState::default());
let now = Instant::now();
let original = BTreeMap::from([("node-a".to_string(), Uuid::new_v4())]);
assert!(publish_fleet_capability_probe_result(&slot, "topology-a", Ok(original), now).is_none());
let admitted = {
let state = slot.read().expect("export proof slot should not poison");
acquire_ilm_recovery_export_fleet_proof_from(&state, "topology-a", now).expect("fresh fleet should admit export")
};
let restarted = BTreeMap::from([("node-a".to_string(), Uuid::new_v4())]);
let draining = publish_fleet_capability_probe_result(&slot, "topology-a", Ok(restarted.clone()), now)
.expect("restart must wait for the admitted export effect window");
assert!(draining.to_string().contains("previous generation to drain"));
{
let state = slot.read().expect("export proof slot should not poison");
assert!(!ilm_recovery_export_fleet_proof_matches_at(&state, &admitted, "topology-a", now));
assert!(
acquire_ilm_recovery_export_fleet_proof_from(&state, "topology-a", now).is_none(),
"successor authority must wait for the old effect window to drain"
);
}
drop(admitted);
assert!(
publish_fleet_capability_probe_result(&slot, "topology-a", Ok(restarted), now + Duration::from_millis(1)).is_none()
);
let state = slot.read().expect("export proof slot should not poison");
assert!(acquire_ilm_recovery_export_fleet_proof_from(&state, "topology-a", now).is_some());
}
#[test]
fn tier_delete_journal_generation_is_stable_across_members_and_process_restarts() {
let topology = "topology-a";
+302 -2
View File
@@ -3301,13 +3301,18 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
// (rustfs/backlog#1009): CompleteMultipartUpload keeps its pre-commit
// `get_object_info` lookup, so the backfill has no consumer here yet.
Self::assign_rename_data_indexes(&mut parts_metadatas);
let mut rename_result = SetDisks::rename_data_owned(
// Disk deadlines can expire before physical publication or failure undo drains.
let mut rename_result = SetDisks::rename_data_owned_with_fence(
&commit_disks,
(RUSTFS_META_MULTIPART_BUCKET, &commit_upload_id_path),
parts_metadatas,
(&commit_bucket, &commit_object),
write_quorum,
commit_allows_early_ack,
crate::set_disk::core::io_primitives::RenameDataFenceOptions::new(write_quorum, None)
.with_namespace_commit_guard(
(!crate::bucket::utils::is_meta_bucketname(&commit_bucket))
.then(|| commit_set.ctx.begin_namespace_commit()),
),
)
.await;
if let Ok(rename_commit) = rename_result.as_mut() {
@@ -6758,6 +6763,301 @@ mod tests {
.await;
}
#[tokio::test]
#[serial(capacity_dirty_scope)]
async fn complete_multipart_advances_namespace_generation_after_commit() {
let (dirs, disks, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-namespace-commit";
let object = "completed-object";
let body = vec![0x65; 4096];
make_bucket_on_all(&disks, bucket).await;
let before = set_disks.ctx.namespace_commit_generation();
let (upload_id, parts) =
stage_upload_with_create_opts(&set_disks, bucket, object, &body, &ObjectOptions::default()).await;
assert_eq!(set_disks.ctx.namespace_commit_generation(), before, "staging is not publication");
assert!(!set_disks.ctx.namespace_commits_pending());
tokio::time::timeout(
Duration::from_secs(10),
set_disks
.clone()
.complete_multipart_upload(bucket, object, &upload_id, parts, &ObjectOptions::default()),
)
.await
.expect("completion must finish")
.expect("the four real shards must commit");
let mut reader = tokio::time::timeout(
Duration::from_secs(5),
set_disks.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()),
)
.await
.expect("GET after completion must finish")
.expect("a successful completion must be immediately readable");
let mut actual = Vec::new();
tokio::time::timeout(Duration::from_secs(5), reader.stream.read_to_end(&mut actual))
.await
.expect("the completed body stream must finish")
.expect("read completed body");
assert_eq!(actual, body);
let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
for dir in &dirs {
assert!(!dir.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path).exists());
}
assert!(matches!(
set_disks.check_upload_id_exists(bucket, object, &upload_id, false).await,
Err(StorageError::InvalidUploadID(..))
));
assert!(!set_disks.ctx.namespace_commits_pending());
assert_eq!(
set_disks.ctx.namespace_commit_generation(),
before + 2,
"one completed MPU must invalidate snapshots at namespace admission and physical retirement"
);
}
#[cfg(not(windows))]
async fn assert_complete_multipart_physical_namespace_owner(undo: bool) {
use crate::disk::os::{self, prepared_publication_test_hooks as hooks};
use crate::set_disk::core::io_primitives::rename_fault_injection;
use futures::FutureExt;
temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("60"))], async {
let (dirs, disks, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-physical-namespace";
let object = if undo { "undo-tail" } else { "publication-tail" };
let old_body = vec![0x41; 1024];
let new_body = vec![0x62; 4096];
make_bucket_on_all(&disks, bucket).await;
let old = set_disks
.put_object(
bucket,
object,
&mut PutObjReader::from_vec(old_body.clone()),
&ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("seed a real readable old version");
let old_etag = old.etag.expect("the committed old object must have an ETag");
let before = set_disks.ctx.namespace_commit_generation();
assert!(!set_disks.ctx.namespace_commits_pending());
let (upload_id, parts) =
stage_upload_with_create_opts(&set_disks, bucket, object, &new_body, &ObjectOptions::default()).await;
let new_etag = get_complete_multipart_md5(&parts);
let upload_path = SetDisks::get_upload_id_dir(bucket, object, &upload_id);
for dir in &dirs {
assert!(dir.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path).exists());
}
assert_eq!(set_disks.ctx.namespace_commit_generation(), before);
let _fault = undo.then(|| rename_fault_injection::fail_rename_on(object, &[2, 3]));
let stage = if undo {
hooks::Stage::Rename
} else {
hooks::Stage::PreparedRename
};
let (entered_tx, mut entered_rx) = tokio::sync::mpsc::unbounded_channel();
let mut hooks = Vec::new();
let mut releases = Vec::new();
let mut mutation_paths = Vec::new();
for (index, disk) in disks.iter().enumerate() {
let disk::Disk::Local(local) = disk.as_ref() else {
panic!("physical MPU fixture requires local disks");
};
let destination = local
.get_disk()
.get_object_path_for_io(bucket, object)
.expect("leased IO path");
let entered_tx = entered_tx.clone();
let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();
hooks.push(hooks::install_at(stage, &destination.join(STORAGE_FORMAT_FILE), move || {
let _ = entered_tx.send(index);
// Dropping senders releases every syscall on assertion failure, too.
let _ = release_rx.recv();
}));
releases.push(release_tx);
// Canonical rename serializes the object directory; backup restore
// serializes its xl.meta destination. Drain the actual executor key.
mutation_paths.push(if undo {
destination.join(STORAGE_FORMAT_FILE)
} else {
destination
});
}
drop(entered_tx);
let complete_set = set_disks.clone();
let complete_upload = upload_id.clone();
let mut complete = tokio::spawn(async move {
complete_set
.complete_multipart_upload(bucket, object, &complete_upload, parts, &ObjectOptions::default())
.await
});
let mut complete_joined = false;
let mut observed_counts = None;
let observations = std::panic::AssertUnwindSafe(async {
let expected_publishers = if undo { 2 } else { 4 };
let entered = tokio::time::timeout(Duration::from_secs(10), async {
let mut entered = HashSet::new();
while entered.len() < expected_publishers {
tokio::select! {
index = entered_rx.recv() => {
assert!(entered.insert(index.expect("physical publisher must signal entry")));
}
result = &mut complete => {
complete_joined = true;
panic!("completion returned before physical entry: {result:?}");
}
}
}
entered
})
.await
.expect("all expected physical metadata operations must enter");
let pending_at_entry = set_disks.ctx.namespace_commits_pending();
let generation_at_entry = set_disks.ctx.namespace_commit_generation();
for &index in &entered {
let metadata = tokio::time::timeout(
Duration::from_secs(5),
disks[index].read_version("", bucket, object, "", &ReadOptions::default()),
)
.await
.expect("metadata observation must finish while publication is paused")
.expect("metadata before the paused physical action must be readable");
assert_eq!(
metadata.metadata.get("etag"),
Some(if undo { &new_etag } else { &old_etag }),
"undo must follow actual publication; prepared rename must precede publication"
);
}
// The entry signals run inside the real blocking closures, after each
// wrapper installed its normal deadline. No quota/external guard disables it.
tokio::time::pause();
tokio::time::advance(Duration::from_secs(61)).await;
tokio::time::resume();
let joined = tokio::time::timeout(Duration::from_secs(5), &mut complete).await;
complete_joined = joined.is_ok();
let result = joined
.expect("ordinary MPU disk/undo deadlines must still return before physical drain")
.expect("completion task must not panic");
assert!(result.is_err(), "a timed-out or two-shard commit cannot acknowledge success");
let pending_after_timeout = set_disks.ctx.namespace_commits_pending();
let generation_after_timeout = set_disks.ctx.namespace_commit_generation();
observed_counts = Some((pending_at_entry, generation_at_entry, pending_after_timeout, generation_after_timeout));
for &index in &entered {
assert!(
os::acquire_rename_data_mutation_lease(&disks[index].path(), bucket, &mutation_paths[index])
.now_or_never()
.is_none(),
"timed-out physical work must still own its object serialization"
);
}
for dir in &dirs {
assert!(
dir.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&upload_path).exists(),
"failed completion must not clean the upload staging"
);
}
})
.catch_unwind()
.await;
// Release even after a failed observation, then finish dispatch before
// draining every physical key. No per-disk assertion may skip a later drain.
drop(releases);
drop(hooks);
let coordinator_drained = complete_joined
|| tokio::time::timeout(Duration::from_secs(10), &mut complete).await.is_ok();
if !coordinator_drained {
complete.abort();
let _ = tokio::time::timeout(Duration::from_secs(5), &mut complete).await;
}
let drains = futures::future::join_all(disks.iter().zip(&mutation_paths).map(|(disk, destination)| async move {
tokio::time::timeout(
Duration::from_secs(5),
os::acquire_rename_data_mutation_lease(&disk.path(), bucket, destination),
)
.await
.map(drop)
}))
.await;
let owner_drained = tokio::time::timeout(Duration::from_secs(5), async {
while set_disks.ctx.namespace_commits_pending() {
tokio::task::yield_now().await;
}
})
.await;
let physical_drained = drains.iter().all(|drain| drain.is_ok());
if !coordinator_drained || !physical_drained || owner_drained.is_err() {
// A bounded cleanup failure cannot justify deleting roots that a
// detached executor might still use. Keep them for diagnosis.
let retained = dirs.into_iter().map(TempDir::keep).collect::<Vec<_>>();
eprintln!("MPU cleanup incomplete: coordinator={coordinator_drained}, physical={physical_drained}, retained={retained:?}");
if let Err(panic) = observations {
std::panic::resume_unwind(panic);
}
panic!("MPU cleanup did not drain: coordinator={coordinator_drained}, physical={physical_drained}, retained={retained:?}");
}
if let Err(panic) = observations {
std::panic::resume_unwind(panic);
}
// Preserve the original drain checks after collecting every result.
for drained in drains {
drained.expect("released physical MPU work must drain");
}
owner_drained.expect("physical retirement must finish its namespace counter decrement");
for disk in &disks {
let metadata = disk
.read_version("", bucket, object, "", &ReadOptions::default())
.await
.expect("all disks must expose the expected final metadata");
assert_eq!(metadata.metadata.get("etag"), Some(if undo { &old_etag } else { &new_etag }));
}
let (pending_at_entry, generation_at_entry, pending_after_timeout, generation_after_timeout) =
observed_counts.expect("successful observations must record namespace counters");
let mut reader = tokio::time::timeout(
Duration::from_secs(5),
set_disks.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default()),
)
.await
.expect("GET after physical drain must finish")
.expect("the real final object must be readable");
let mut actual = Vec::new();
tokio::time::timeout(Duration::from_secs(5), reader.stream.read_to_end(&mut actual))
.await
.expect("the final object stream must finish")
.expect("read final object bytes");
assert_eq!(actual, if undo { old_body } else { new_body });
assert!(
pending_at_entry && pending_after_timeout,
"physical MPU work outlived namespace accounting: undo={undo}"
);
assert_eq!(generation_at_entry, before + 1);
assert_eq!(
generation_after_timeout, generation_at_entry,
"the blocked physical owner cannot retire early"
);
assert_eq!(set_disks.ctx.namespace_commit_generation(), before + 2);
assert!(!set_disks.ctx.namespace_commits_pending());
})
.await;
}
#[cfg(not(windows))]
#[tokio::test]
#[serial(capacity_dirty_scope)]
async fn complete_multipart_timeout_keeps_namespace_owner_until_physical_publication() {
assert_complete_multipart_physical_namespace_owner(false).await;
}
#[cfg(not(windows))]
#[tokio::test]
#[serial(capacity_dirty_scope)]
async fn complete_multipart_failed_quorum_keeps_namespace_owner_until_physical_undo() {
assert_complete_multipart_physical_namespace_owner(true).await;
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn complete_multipart_releases_disk_snapshot_before_cleanup() {
+42 -1
View File
@@ -830,6 +830,10 @@ mod tests {
IlmRecoveryProtocol, MAX_RECOVERY_ATTEMPTS, list_recovery_controls, load_recovery_control,
observe_recovery_source, save_recovery_control_if_absent,
},
recovery_export::{
create_recovery_export, inspect_recovery_export_observation, load_recovery_export,
recovery_export_record_object_name,
},
tier_delete_journal::{
DecommissionCheckpointTargetFailureHook, TIER_DELETE_DISPATCH_MANIFEST_PREFIX, TIER_DELETE_JOURNAL_PREFIX,
TierDeleteChunkTestBarrier, TierDeleteChunkTestStage, TierDeleteDispatchBatchLimitGuard,
@@ -4369,7 +4373,7 @@ mod tests {
}
retry_source_info.parts = Arc::new(retry_source_parts);
assert_eq!(retry_source_info.etag.as_deref(), Some(retry_object_etag.as_str()));
assert!(!retry_source_info.is_multipart());
assert!(retry_source_info.is_multipart());
assert!(retry_source_info.parts.iter().all(|part| part.checksums.is_some()));
assert_eq!(retry_source_info.checksum.as_deref(), Some(retry_object_checksum_bytes.as_ref()));
assert!(
@@ -16870,6 +16874,43 @@ mod tests {
}
}
let creator_sha256 = rustfs_utils::crypto::hex_sha256(b"legacy-export-actor", ToOwned::to_owned);
let mut created_exports = Vec::new();
for exportable in first_controls
.iter()
.filter(|control| control.classification == IlmRecoveryClassification::RetainedAmbiguous)
{
let observation = inspect_recovery_export_observation(store.clone(), &exportable.control_id)
.await
.expect("fresh legacy recovery observation should be exportable");
let created = create_recovery_export(store.clone(), &observation, &creator_sha256)
.await
.expect("legacy recovery export should be created exactly once");
assert!(!created.replayed);
let loaded = load_recovery_export(store.clone(), &created.export_id)
.await
.expect("created legacy recovery export should load");
assert_eq!(loaded.encoded, created.encoded, "export readback must preserve the exact committed bytes");
let replayed = create_recovery_export(store.clone(), &observation, &creator_sha256)
.await
.expect("the same observed generation should replay its immutable export");
assert!(replayed.replayed);
assert_eq!(replayed.encoded, created.encoded);
created_exports.push(created);
}
assert_eq!(created_exports.len(), 2, "both v1 and v2 legacy journals must have an export path");
let corrupt_export_id = &created_exports[0].export_id;
let export_path = recovery_export_record_object_name(IlmRecoveryProtocol::TierDeleteJournal, corrupt_export_id)
.expect("export path should build");
com::save_config(store.clone(), &export_path, Vec::new())
.await
.expect("zero-byte corruption fixture should persist");
let corrupt_export = load_recovery_export(store.clone(), corrupt_export_id)
.await
.expect_err("an existing zero-byte export must fail closed");
assert!(!matches!(corrupt_export, Error::ConfigNotFound));
com::save_config(
store.clone(),
&journal_paths[0],
+2 -5
View File
@@ -731,11 +731,8 @@ pub(super) fn prune_completed_heal_statuses(completed_heals: &mut HashMap<String
}
pub(super) fn prune_completed_heal_statuses_at(completed_heals: &mut HashMap<String, Arc<CompletedHealStatus>>, now: SystemTime) {
completed_heals.retain(|_, completed| {
now.duration_since(completed.completed_at)
.map(|age| age <= KEEP_HEAL_TASK_STATUS_DURATION)
.unwrap_or(false)
});
completed_heals
.retain(|_, completed| now.duration_since(completed.completed_at).unwrap_or_default() <= KEEP_HEAL_TASK_STATUS_DURATION);
let entry_bytes = |key: &String, value: &Arc<CompletedHealStatus>| {
key.capacity()
.saturating_add(size_of::<(String, Arc<CompletedHealStatus>)>())
+89 -22
View File
@@ -189,37 +189,104 @@ fn completed_retention_count_ttl_and_alias_eviction_are_bounded() {
);
entries.insert("future".to_string(), Arc::new(completed_retention_fixture(now + Duration::from_nanos(1))));
prune_completed_heal_statuses_at(&mut entries, now);
assert_eq!(entries.len(), 1);
assert_eq!(entries.len(), 2);
assert!(entries.contains_key("ttl-boundary"));
assert!(entries.contains_key("future"), "clock rollback must not expire a new completion");
prune_completed_heal_statuses_at(&mut entries, now + Duration::from_nanos(1));
assert_eq!(entries.len(), 1);
assert!(entries.contains_key("future"));
prune_completed_heal_statuses_at(&mut entries, now + Duration::from_nanos(1) + KEEP_HEAL_TASK_STATUS_DURATION);
assert!(entries.contains_key("future"), "the exact TTL boundary remains retained");
prune_completed_heal_statuses_at(&mut entries, now + Duration::from_nanos(2) + KEEP_HEAL_TASK_STATUS_DURATION);
assert!(entries.is_empty());
}
#[tokio::test]
async fn completed_retention_clock_rollback_preserves_terminal_alias_queries() {
let completed_at = SystemTime::now() + Duration::from_secs(3600);
for status in [
HealTaskStatus::Completed,
HealTaskStatus::Failed {
error: "fixture failure".to_string(),
},
HealTaskStatus::Cancelled,
] {
let manager = HealManager::new(Arc::new(MockStorage), None);
let mut snapshot = completed_retention_fixture(completed_at);
snapshot.status = status.clone();
let expected_progress = snapshot.progress.clone();
let snapshot = Arc::new(snapshot);
{
let mut completed = manager.completed_heals.lock().await;
completed.insert("canonical".to_string(), Arc::clone(&snapshot));
completed.insert("alias".to_string(), Arc::clone(&snapshot));
}
for token in ["canonical", "alias"] {
let report = manager
.get_task_report_since(token, Some(3))
.await
.expect("a clock rollback must retain terminal queries");
assert_eq!(report.status, status);
assert_eq!(report.progress, expected_progress);
assert_eq!(report.result_items.len(), 1);
assert_eq!((report.min_seq, report.next_seq), (3, 5));
assert!(!report.result_items_truncated);
}
let mut completed = manager.completed_heals.lock().await;
prune_completed_heal_statuses_at(&mut completed, completed_at + KEEP_HEAL_TASK_STATUS_DURATION);
assert_eq!(completed.len(), 2, "both tokens remain at the exact TTL boundary");
prune_completed_heal_statuses_at(&mut completed, completed_at + KEEP_HEAL_TASK_STATUS_DURATION + Duration::from_nanos(1));
assert!(completed.is_empty(), "both tokens expire after the TTL");
}
}
#[test]
fn completed_retention_clock_rollback_keeps_count_and_alias_eviction_bounded() {
let now = SystemTime::UNIX_EPOCH + Duration::from_secs(3600);
let oldest = Arc::new(completed_retention_fixture(now + Duration::from_secs(1)));
let mut entries = HashMap::from([
("oldest".to_string(), Arc::clone(&oldest)),
("oldest-alias".to_string(), oldest),
]);
for index in 2..=MAX_COMPLETED_HEAL_TOKENS {
entries.insert(
format!("task-{index}"),
Arc::new(completed_retention_fixture(now + Duration::from_secs(2))),
);
}
prune_completed_heal_statuses_at(&mut entries, now);
assert_eq!(entries.len(), MAX_COMPLETED_HEAL_TOKENS - 1);
assert!(!entries.contains_key("oldest"));
assert!(!entries.contains_key("oldest-alias"));
}
#[test]
fn completed_retention_total_byte_cap_and_cap_plus_one() {
let now = SystemTime::now();
let key = "large".to_string();
let mut entry = completed_retention_fixture(now);
let base_bytes = entry.retained_bytes() + key.capacity() + size_of::<(String, Arc<CompletedHealStatus>)>();
entry.retained_bytes.take();
entry.status = HealTaskStatus::Failed {
error: "x".repeat(MAX_COMPLETED_HEAL_BYTES - base_bytes),
};
assert_eq!(
entry.retained_bytes() + key.capacity() + size_of::<(String, Arc<CompletedHealStatus>)>(),
MAX_COMPLETED_HEAL_BYTES
);
let mut entries = HashMap::from([(key, Arc::new(entry))]);
prune_completed_heal_statuses_at(&mut entries, now);
assert_eq!(entries.len(), 1, "exact byte cap remains retained");
let mut over = Arc::try_unwrap(entries.remove("large").expect("entry retained")).expect("entry not shared");
over.retained_bytes.take();
if let HealTaskStatus::Failed { error } = &mut over.status {
*error = "x".repeat(error.len() + 1);
for completed_at in [now, now + Duration::from_secs(1)] {
let key = "large".to_string();
let mut entry = completed_retention_fixture(completed_at);
let base_bytes = entry.retained_bytes() + key.capacity() + size_of::<(String, Arc<CompletedHealStatus>)>();
entry.retained_bytes.take();
entry.status = HealTaskStatus::Failed {
error: "x".repeat(MAX_COMPLETED_HEAL_BYTES - base_bytes),
};
assert_eq!(
entry.retained_bytes() + key.capacity() + size_of::<(String, Arc<CompletedHealStatus>)>(),
MAX_COMPLETED_HEAL_BYTES
);
let mut entries = HashMap::from([(key, Arc::new(entry))]);
prune_completed_heal_statuses_at(&mut entries, now);
assert_eq!(entries.len(), 1, "exact byte cap remains retained");
let mut over = Arc::try_unwrap(entries.remove("large").expect("entry retained")).expect("entry not shared");
over.retained_bytes.take();
if let HealTaskStatus::Failed { error } = &mut over.status {
*error = "x".repeat(error.len() + 1);
}
entries.insert("large".to_string(), Arc::new(over));
prune_completed_heal_statuses_at(&mut entries, now);
assert!(entries.is_empty(), "oversized metadata cannot escape total byte bound");
}
entries.insert("large".to_string(), Arc::new(over));
prune_completed_heal_statuses_at(&mut entries, now);
assert!(entries.is_empty(), "oversized metadata cannot escape total byte bound");
}
#[tokio::test]
+1 -2
View File
@@ -66,9 +66,8 @@ const ERR_LIFECYCLE_EXPIRED_OBJECT_DELETE_MARKER_WITH_TAGS: &str =
const ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION: &str = "Rule must have at least one of Expiration, Transition, NoncurrentVersionExpiration, NoncurrentVersionTransition, or DelMarkerExpiration";
const ERR_LIFECYCLE_PREFIX_FILTER_CONFLICT: &str = "Legacy Prefix and Filter cannot both be present in a lifecycle rule. Use Filter.Prefix instead of the top-level Prefix element.";
const ERR_LIFECYCLE_INVALID_NEWER_NONCURRENT_VERSIONS: &str = "'NewerNoncurrentVersions' must be a non-negative integer";
const ERR_LIFECYCLE_FILTER_TOO_MANY_PREDICATES: &str =
"Filter must have at most one of Prefix, Tag, ObjectSizeGreaterThan, ObjectSizeLessThan or And; combine predicates with And";
const ERR_LIFECYCLE_FILTER_AND_TOO_FEW_PREDICATES: &str = "Filter And must contain at least two predicates";
const ERR_LIFECYCLE_FILTER_TOO_MANY_PREDICATES: &str = "Filter has too many predicates";
const ERR_LIFECYCLE_FILTER_DUPLICATE_TAG_KEY: &str = "Filter must not repeat a tag key";
const ERR_LIFECYCLE_FILTER_INVALID_TAG: &str = "Tag key must be 1-128 characters and tag value must be at most 256 characters";
const ERR_LIFECYCLE_FILTER_NEGATIVE_SIZE: &str = "ObjectSizeGreaterThan and ObjectSizeLessThan must not be negative";
+33 -6
View File
@@ -175,6 +175,7 @@ pub const BACKGROUND_HEAL_STATUS_PROTOCOL_VERSION: u32 = 2;
pub const HEAL_CONTROL_CAPABILITY_PROBE_PREFIX: &[u8] = b"rustfs-heal-control-capability-v3\0";
pub const REMOTE_VERSION_STATE_CAPABILITY_PROBE_PREFIX: &[u8] = b"rustfs-tier-remote-version-state-capability-v1\0";
pub const CROSS_POOL_FENCE_CAPABILITY_PROBE_PREFIX: &[u8] = b"rustfs-cross-pool-fence-capability-v1\0";
pub const ILM_RECOVERY_EXPORT_CAPABILITY_PROBE_PREFIX: &[u8] = b"rustfs-ilm-recovery-export-capability-v1\0";
pub const TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE: usize = 64 * 1024;
pub const TIER_MUTATION_RPC_MAX_COMMIT_PAYLOAD_SIZE: usize = 1024;
pub const TIER_MUTATION_RPC_MAX_ABORT_PAYLOAD_SIZE: usize = TIER_MUTATION_RPC_MAX_PREPARE_PAYLOAD_SIZE;
@@ -219,6 +220,18 @@ pub fn is_cross_pool_fence_capability_probe(command: &[u8]) -> bool {
&& command.starts_with(CROSS_POOL_FENCE_CAPABILITY_PROBE_PREFIX)
}
pub fn ilm_recovery_export_capability_probe(nonce: &[u8; 16]) -> Vec<u8> {
let mut probe = Vec::with_capacity(ILM_RECOVERY_EXPORT_CAPABILITY_PROBE_PREFIX.len() + nonce.len());
probe.extend_from_slice(ILM_RECOVERY_EXPORT_CAPABILITY_PROBE_PREFIX);
probe.extend_from_slice(nonce);
probe
}
pub fn is_ilm_recovery_export_capability_probe(command: &[u8]) -> bool {
command.len() == ILM_RECOVERY_EXPORT_CAPABILITY_PROBE_PREFIX.len() + 16
&& command.starts_with(ILM_RECOVERY_EXPORT_CAPABILITY_PROBE_PREFIX)
}
pub fn encode_remote_version_state_capability(
topology_member: &str,
process_epoch: &[u8; 16],
@@ -2127,12 +2140,13 @@ mod scanner_activity_tests {
mod heal_control_tests {
use super::{
CROSS_POOL_FENCE_CAPABILITY_PROBE_PREFIX, HEAL_CONTROL_CAPABILITY_PROBE_PREFIX, HEAL_CONTROL_PROTOCOL_VERSION,
REMOTE_VERSION_STATE_CAPABILITY_PROBE_PREFIX, canonical_heal_control_capability_ack, canonical_heal_control_request_body,
canonical_heal_control_response_body, decode_remote_version_state_capability, encode_cross_pool_fence_capability,
encode_remote_version_state_capability, heal_control_capability_probe, heal_control_coordinator_epoch,
heal_control_execution_timeout, heal_control_execution_timeout_for, internode_rpc_timeout,
is_cross_pool_fence_capability_probe, is_heal_control_capability_probe, is_remote_version_state_capability_probe,
normalize_internode_rpc_timeout, remote_version_state_capability_probe,
ILM_RECOVERY_EXPORT_CAPABILITY_PROBE_PREFIX, REMOTE_VERSION_STATE_CAPABILITY_PROBE_PREFIX,
canonical_heal_control_capability_ack, canonical_heal_control_request_body, canonical_heal_control_response_body,
decode_remote_version_state_capability, encode_cross_pool_fence_capability, encode_remote_version_state_capability,
heal_control_capability_probe, heal_control_coordinator_epoch, heal_control_execution_timeout,
heal_control_execution_timeout_for, ilm_recovery_export_capability_probe, internode_rpc_timeout,
is_cross_pool_fence_capability_probe, is_heal_control_capability_probe, is_ilm_recovery_export_capability_probe,
is_remote_version_state_capability_probe, normalize_internode_rpc_timeout, remote_version_state_capability_probe,
};
use crate::heal_control;
use std::time::Duration;
@@ -2196,6 +2210,19 @@ mod heal_control_tests {
assert!(!is_remote_version_state_capability_probe(REMOTE_VERSION_STATE_CAPABILITY_PROBE_PREFIX));
}
#[test]
fn ilm_recovery_export_capability_probe_requires_exact_prefix_and_nonce() {
let probe = ilm_recovery_export_capability_probe(&[7; 16]);
assert!(is_ilm_recovery_export_capability_probe(&probe));
assert!(!is_ilm_recovery_export_capability_probe(ILM_RECOVERY_EXPORT_CAPABILITY_PROBE_PREFIX));
let mut wrong_prefix = probe.clone();
wrong_prefix[0] ^= 1;
assert!(!is_ilm_recovery_export_capability_probe(&wrong_prefix));
let mut extra = probe;
extra.push(0);
assert!(!is_ilm_recovery_export_capability_probe(&extra));
}
#[test]
fn remote_version_state_capability_binds_member_and_process_epoch() {
let encoded =
+34 -11
View File
@@ -1910,10 +1910,10 @@ where
.as_ref()
.map(|(notification_system, grants)| (Arc::clone(notification_system), grants.clone()));
let remote_lease_release_safe = Arc::new(AtomicBool::new(true));
let mut usage_persist_outcome = match publication_defer_reason {
let mut usage_publication_result = match publication_defer_reason {
Some(reason) => {
drop(receiver);
DataUsagePersistOutcome::Deferred(reason)
DataUsagePublicationResult::from(DataUsagePersistOutcome::Deferred(reason))
}
None => {
// ScannerIO emits its complete or observational update only after
@@ -1928,6 +1928,11 @@ where
.as_ref()
.map(|(_, grants)| grants.iter().map(|grant| grant.lease.token).collect())
.unwrap_or_default();
let ack_expectation = scan_result
.as_ref()
.ok()
.filter(|result| result.has_dirty_usage_to_acknowledge())
.and_then(ScannerCycleResult::publication_expectation);
let mut usage_persist_task = AbortOnDropHandle::new(tokio::spawn(async move {
store_data_usage_in_backend_with_outcome_for_epoch_and_baseline_and_route_probe_for_publication_epoch_and_lease_fence(
ctx_clone,
@@ -1940,6 +1945,7 @@ where
remote_lease_deadline,
remote_lease_fence,
)
.with_ack_expectation(ack_expectation)
.with_remote_lease_tokens(remote_lease_tokens)
.with_lease_release_flag(remote_lease_release_safe_for_task),
move || {
@@ -1973,7 +1979,7 @@ where
error = %err,
"Scanner data usage persistence task failed"
);
DataUsagePersistOutcome::Failed
DataUsagePublicationResult::from(DataUsagePersistOutcome::Failed)
}
DataUsagePersistTaskResult::Cancelled => {
debug!(
@@ -1985,7 +1991,7 @@ where
state = "usage_persist_task_cancelled",
"Scanner data usage persistence task cancelled"
);
DataUsagePersistOutcome::Failed
DataUsagePublicationResult::from(DataUsagePersistOutcome::Failed)
}
DataUsagePersistTaskResult::TimedOut => {
error!(
@@ -1998,11 +2004,12 @@ where
state = "usage_persist_task_timed_out",
"Scanner data usage persistence task timed out"
);
DataUsagePersistOutcome::Failed
DataUsagePublicationResult::from(DataUsagePersistOutcome::Failed)
}
}
}
};
let mut usage_persist_outcome = usage_publication_result.outcome();
let lease_expired = remote_publication_leases
.as_ref()
.is_some_and(|(_, grants)| grants.iter().any(|grant| !grant.lease.is_valid()));
@@ -2202,8 +2209,9 @@ where
};
}
usage_publication_result.restrict_outcome(usage_persist_outcome);
let (completion_outcome, scanner_pending_maintenance_work, remote_dirty_usage_acknowledgements) =
finalize_scanner_cycle_result(scan_cycle_result, usage_persist_outcome);
finalize_scanner_cycle_result(scan_cycle_result, usage_publication_result);
let remote_dirty_usage_pending = if remote_dirty_usage_acknowledgements.is_empty() {
false
} else if let Some(notification_system) = storeapi.scanner_notification_system() {
@@ -3437,21 +3445,35 @@ fn scanner_cycle_completion_outcome(
fn finalize_scanner_cycle_result(
scan_cycle_result: crate::scanner_io::ScannerCycleResult,
usage_persist_outcome: DataUsagePersistOutcome,
publication: DataUsagePublicationResult,
) -> (ScannerCycleOutcome, bool, Vec<ScannerDirtyUsageAcknowledgement>) {
let (usage_persist_outcome, proof) = publication.into_parts();
let completion_outcome = scanner_cycle_completion_outcome_for_result(&scan_cycle_result, usage_persist_outcome);
let pending_maintenance_work = scan_cycle_result.has_pending_maintenance_work();
let durable_complete_snapshot = scan_cycle_result.status == ScannerCycleStatus::Complete
&& matches!(
usage_persist_outcome,
DataUsagePersistOutcome::Saved | DataUsagePersistOutcome::AlreadyDurable
);
)
&& scan_cycle_result.publication_expectation().as_ref().is_some_and(|expected| {
proof
.as_ref()
.is_some_and(|proof| proof.verified_version_for(expected).is_some())
});
let pending_maintenance_work = scan_cycle_result.has_pending_maintenance_work()
|| (scan_cycle_result.has_dirty_usage_to_acknowledge() && !durable_complete_snapshot);
let remote_dirty_usage_acknowledgements = if durable_complete_snapshot {
scan_cycle_result.acknowledge_durable_usage()
match proof {
Some(proof) => scan_cycle_result.acknowledge_durable_usage(&proof),
None => Vec::new(),
}
} else {
Vec::new()
};
(completion_outcome, pending_maintenance_work, remote_dirty_usage_acknowledgements)
(
completion_outcome,
pending_maintenance_work || crate::scanner_io::dirty_usage_buckets_pending(),
remote_dirty_usage_acknowledgements,
)
}
fn scanner_cycle_completion_outcome_for_result(
@@ -3551,6 +3573,7 @@ use activity::*;
use backlog::*;
use cycle_state::*;
use leadership::*;
pub(crate) use usage_store::RootPublicationProof;
use usage_store::*;
pub use activity::scanner_topology_digest;
+121 -15
View File
@@ -36,7 +36,10 @@ use tokio::time::{Duration, advance};
const TEST_DEFAULT_SCANNER_CYCLE_SECS: u64 = 24 * 60 * 60;
mod quota_reset_preservation;
mod recovery_control;
mod scoped_ack_publication;
async fn setup_scanner_cycle_store() -> (tempfile::TempDir, Arc<ECStore>) {
setup_scanner_cycle_store_with_usage_baseline(true).await
@@ -5288,6 +5291,103 @@ async fn scanner_usage_state_reset_resumes_every_cleanup_boundary_without_rewrit
}
}
#[tokio::test]
#[serial]
async fn scanner_usage_state_reset_resumes_real_store_cleanup_boundaries_after_reopen() {
let primary_path = DATA_USAGE_OBJ_NAME_PATH.as_str();
let cleanup_paths = [
format!("{primary_path}.bkp"),
LEGACY_DATA_USAGE_OBJ_NAME_PATH.as_str().to_string(),
format!("{}.bkp", LEGACY_DATA_USAGE_OBJ_NAME_PATH.as_str()),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str().to_string(),
];
for completed in 0..=cleanup_paths.len() {
let (_temp_dir, store) = setup_scanner_cycle_store().await;
let cycle = CurrentCycle {
current: 12,
next: 42,
cycle_completed: vec![Utc::now()],
started: Utc::now(),
};
save_config(
store.clone(),
DATA_USAGE_BLOOM_NAME_PATH.as_str(),
encode_scanner_cycle_state(&cycle, 3).expect("cycle state should encode"),
)
.await
.expect("cycle state should persist");
let marker = scanner_usage_bootstrap_marker(std::time::SystemTime::UNIX_EPOCH, Some(3));
save_config(
store.clone(),
primary_path,
serde_json::to_vec(&marker).expect("usage reset marker should encode"),
)
.await
.expect("usage reset marker should persist");
for path in cleanup_paths.iter().skip(completed) {
let mut usage = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH), 0);
usage.scanner_epoch = Some(1);
usage.scanner_cycle = Some(12);
save_config(store.clone(), path, serde_json::to_vec(&usage).expect("cleanup slot should encode"))
.await
.expect("cleanup slot should persist");
}
for path in ["buckets/quota-reservations/ledger", "buckets/example/incarnation"] {
save_config(store.clone(), path, b"retain".to_vec())
.await
.expect("unrelated state should persist before reopen");
}
let restarted = restart_scanner_cycle_store_from(&store).await;
let intent_before = read_config_with_revision(restarted.clone(), primary_path)
.await
.expect("reopened reset intent should be readable");
let result = reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), restarted.clone())
.await
.expect("reopened usage reset should complete");
assert_eq!(result.leader_epoch, 3, "boundary {completed}");
assert_eq!(result.next_cycle, 42, "boundary {completed}");
assert_eq!(result.reset_paths.len(), cleanup_paths.len() + 1 - completed, "boundary {completed}");
assert_eq!(
read_config_with_revision(restarted.clone(), primary_path)
.await
.expect("completed reset intent should remain readable"),
intent_before,
"boundary {completed}: resumed cleanup must not rewrite the reset intent"
);
let (floor, state) = persisted_usage_floor_for_startup(restarted.clone(), false)
.await
.expect("completed reset marker should remain resumable");
assert_eq!(floor.leader_epoch, 3, "boundary {completed}");
assert_eq!(state, PersistedUsageFloorStartup::BootstrapPending, "boundary {completed}");
assert!(
persisted_usage_floor(restarted.clone()).await.is_err(),
"boundary {completed}: bootstrap marker must not become an authoritative floor"
);
for path in &cleanup_paths {
assert!(
matches!(read_config(restarted.clone(), path).await, Err(EcstoreError::ConfigNotFound)),
"boundary {completed}: reset should remove stale usage slot {path}"
);
}
for path in ["buckets/quota-reservations/ledger", "buckets/example/incarnation"] {
assert_eq!(
read_config(restarted.clone(), path)
.await
.expect("unrelated state should survive reopened reset"),
b"retain",
"boundary {completed}: reset must preserve non-scanner-state config"
);
}
}
}
#[tokio::test]
async fn scanner_usage_state_reset_stops_usage_fence_after_owner_loss() {
let store = Arc::new(MemoryConfigStore::default());
@@ -6184,7 +6284,7 @@ async fn coordinator_classifies_an_expired_publication_lease() {
.await;
assert_eq!(
outcome,
outcome.outcome(),
DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::PublicationLeaseDeadlineExceeded)
);
assert!(store.put_counts.lock().await.is_empty(), "expired lease must prevent a PUT");
@@ -7325,7 +7425,7 @@ fn scanner_cycle_cache_floor_stays_pending_during_deferred_usage_publication() {
#[test]
#[serial]
fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
fn finalizing_a_saved_enum_without_proof_keeps_dirty_pending() {
crate::scanner_io::clear_dirty_usage_bucket("photos");
crate::scanner_io::record_dirty_usage_bucket("photos");
let dirty_snapshot = crate::scanner_io::dirty_usage_buckets_for_tests();
@@ -7337,17 +7437,19 @@ fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
};
let unsaved = crate::scanner_io::ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(dirty_snapshot.clone()))
.with_remote_dirty_usage_acknowledgements(vec![remote_acknowledgement.clone()]);
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(unsaved, DataUsagePersistOutcome::NoUpdate);
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(unsaved, DataUsagePersistOutcome::NoUpdate.into());
assert_eq!(outcome, ScannerCycleOutcome::Failed);
assert!(acknowledgements.is_empty());
assert!(crate::scanner_io::dirty_usage_buckets_pending());
let saved = crate::scanner_io::ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(dirty_snapshot))
.with_remote_dirty_usage_acknowledgements(vec![remote_acknowledgement.clone()]);
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(saved, DataUsagePersistOutcome::Saved);
.with_remote_dirty_usage_acknowledgements(vec![remote_acknowledgement]);
let (outcome, pending, acknowledgements) = finalize_scanner_cycle_result(saved, DataUsagePersistOutcome::Saved.into());
assert_eq!(outcome, ScannerCycleOutcome::Completed);
assert_eq!(acknowledgements, vec![remote_acknowledgement]);
assert!(!crate::scanner_io::dirty_usage_buckets_pending());
assert!(acknowledgements.is_empty());
assert!(pending);
assert!(crate::scanner_io::dirty_usage_buckets_pending());
crate::scanner_io::clear_dirty_usage_bucket("photos");
}
#[test]
@@ -7359,7 +7461,7 @@ fn finalizing_a_deferred_usage_save_keeps_dirty_work_pending() {
let deferred = crate::scanner_io::ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(dirty_snapshot));
let (outcome, _, acknowledgements) =
finalize_scanner_cycle_result(deferred, DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::DataMovement));
finalize_scanner_cycle_result(deferred, DataUsagePersistOutcome::Deferred(ScannerCycleDeferReason::DataMovement).into());
assert_eq!(outcome, ScannerCycleOutcome::Deferred(ScannerCycleDeferReason::DataMovement));
assert!(acknowledgements.is_empty());
@@ -7379,7 +7481,7 @@ fn finalizing_post_scan_observation_advances_partially_without_dirty_ack() {
)
.with_observational_snapshot_published(true);
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(observed, DataUsagePersistOutcome::Saved);
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(observed, DataUsagePersistOutcome::Saved.into());
assert_eq!(outcome, ScannerCycleOutcome::Partial);
assert!(acknowledgements.is_empty());
@@ -7415,17 +7517,20 @@ async fn scanner_cycle_keeps_remote_pending_acknowledgement() {
#[test]
#[serial]
fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() {
fn finalizing_an_already_durable_enum_without_proof_keeps_dirty_pending() {
crate::scanner_io::clear_dirty_usage_bucket("photos");
crate::scanner_io::record_dirty_usage_bucket("photos");
let dirty_snapshot = crate::scanner_io::dirty_usage_buckets_for_tests();
let durable = crate::scanner_io::ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(dirty_snapshot));
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(durable, DataUsagePersistOutcome::AlreadyDurable);
let (outcome, pending, acknowledgements) =
finalize_scanner_cycle_result(durable, DataUsagePersistOutcome::AlreadyDurable.into());
assert_eq!(outcome, ScannerCycleOutcome::Completed);
assert!(acknowledgements.is_empty());
assert!(!crate::scanner_io::dirty_usage_buckets_pending());
assert!(pending);
assert!(crate::scanner_io::dirty_usage_buckets_pending());
crate::scanner_io::clear_dirty_usage_bucket("photos");
}
#[test]
@@ -7436,7 +7541,8 @@ fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() {
let dirty_snapshot = crate::scanner_io::dirty_usage_buckets_for_tests();
let durable = crate::scanner_io::ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(dirty_snapshot));
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(durable, DataUsagePersistOutcome::PriorCycleDurable);
let (outcome, _, acknowledgements) =
finalize_scanner_cycle_result(durable, DataUsagePersistOutcome::PriorCycleDurable.into());
assert_eq!(outcome, ScannerCycleOutcome::Completed);
assert!(acknowledgements.is_empty());
@@ -7452,7 +7558,7 @@ fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() {
let dirty_snapshot = crate::scanner_io::dirty_usage_buckets_for_tests();
let superseded = crate::scanner_io::ScannerCycleResult::new(ScannerCycleStatus::Superseded, Some(dirty_snapshot));
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(superseded, DataUsagePersistOutcome::Saved);
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(superseded, DataUsagePersistOutcome::Saved.into());
assert_eq!(outcome, ScannerCycleOutcome::Superseded);
assert!(acknowledgements.is_empty());
@@ -8881,7 +8987,7 @@ fn post_lease_activity_proof_rejects_a_put_tail_that_finished_before_lease_acqui
]);
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(
result,
DataUsagePersistOutcome::Deferred(reason.expect("changed namespace should defer publication")),
DataUsagePersistOutcome::Deferred(reason.expect("changed namespace should defer publication")).into(),
);
assert_eq!(
outcome,
@@ -0,0 +1,148 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
use crate::storage_api::owner::ObjectOperations as _;
const BUCKET: &str = "quota-reset-preservation";
const OPERATION: &str = "00000000-0000-0000-0000-000000000002";
async fn reservation_fixture() -> (tempfile::TempDir, Arc<ECStore>, Uuid, String, Vec<u8>) {
let (directory, store) = setup_scanner_cycle_store().await;
store
.make_bucket(BUCKET, &crate::storage_api::scan::MakeBucketOptions::default())
.await
.expect("create the reservation fixture bucket through its owner");
let incarnation = store
.bucket_incarnation_id_from_disk(BUCKET)
.await
.expect("durable bucket incarnation");
assert!(!incarnation.is_nil());
let path = format!("config/quota-ledger/{BUCKET}.json");
let bytes = serde_json::to_vec(&serde_json::json!({
"version": 1,
"bucket_incarnation": incarnation,
"quota_revision_unix_nanos": 1,
"accounted_usage": 100,
"reservations": {
OPERATION: {
"object": "pending-object",
"old_size": 0,
"new_size": 64,
"created_at": 1,
"pool_index": 0,
"set_index": 0,
"commit_started": true
}
}
}))
.expect("encode the committed reservation fixture");
save_config(store.clone(), &path, bytes.clone())
.await
.expect("persist reservation bytes through the real storage owner");
(directory, store, incarnation, path, bytes)
}
async fn assert_reservation_retained(store: &Arc<ECStore>, path: &str, expected: &[u8], incarnation: Uuid) {
let bytes = read_config(store.clone(), path)
.await
.expect("read the actual reservation ledger");
assert_eq!(bytes, expected, "scanner reset must not rewrite the reservation ledger");
let ledger: serde_json::Value = serde_json::from_slice(&bytes).expect("persisted ledger JSON");
assert_eq!(ledger["version"], 1);
assert_eq!(ledger["bucket_incarnation"], incarnation.to_string());
assert_eq!(ledger["accounted_usage"], 100);
let reservations = ledger["reservations"].as_object().expect("reservation map");
assert_eq!(reservations.len(), 1);
let pending = &reservations[OPERATION];
assert_eq!(pending["old_size"], 0);
assert_eq!(pending["new_size"], 64);
assert_eq!(pending["commit_started"], true);
assert_eq!(
store
.bucket_incarnation_id_from_disk(BUCKET)
.await
.expect("owner incarnation after restart"),
incarnation
);
}
#[tokio::test]
#[serial]
async fn quota_reset_preservation_survives_storage_owner_reconstruction() {
let (_directory, store, incarnation, path, bytes) = reservation_fixture().await;
let reset = reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), store.clone())
.await
.expect("reset scanner usage through the fenced production entry");
assert_eq!(reset.usage_state, "bootstrap-pending");
let restarted = restart_scanner_cycle_store_from(&store).await;
assert!(
!Arc::ptr_eq(&store, &restarted),
"the assertion must read through a newly constructed ECStore"
);
assert_reservation_retained(&restarted, &path, &bytes, incarnation).await;
let usage = read_config(restarted.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("read reset usage through the reconstructed owner");
let usage: DataUsageInfo = serde_json::from_slice(&usage).expect("bootstrap usage JSON");
assert!(data_usage_info_is_bootstrap_pending(&usage));
assert!(!data_usage_info_has_persisted_baseline_identity(&usage));
}
#[tokio::test]
#[serial]
async fn quota_reset_preservation_unknown_protocol_rejects_put_after_restart() {
for quota_shape in ["zero", "null", "missing"] {
let (_directory, store, incarnation, path, bytes) = reservation_fixture().await;
let mut quota = serde_json::json!({
"quota_type": "Hard",
"reservation_protocol": 2,
"reservation_quota": 1024
});
match quota_shape {
"zero" => quota["quota"] = serde_json::json!(0),
"null" => quota["quota"] = serde_json::Value::Null,
"missing" => {}
_ => unreachable!("fixed quota shapes"),
}
let unknown_quota = serde_json::to_vec(&quota).expect("unknown but syntactically valid quota protocol");
store
.update_bucket_metadata_config(BUCKET, rustfs_config::QUOTA_CONFIG_FILE, unknown_quota)
.await
.expect("persist a future protocol using the real metadata owner");
assert_eq!(
store
.bucket_incarnation_id_from_disk(BUCKET)
.await
.expect("same metadata owner incarnation"),
incarnation
);
reset_scanner_usage_state_for_full_rebuild(CancellationToken::new(), store.clone())
.await
.expect("scanner reset must not change quota metadata");
let restarted = restart_scanner_cycle_store_from(&store).await;
assert!(!Arc::ptr_eq(&store, &restarted));
assert_reservation_retained(&restarted, &path, &bytes, incarnation).await;
let mut reader = PutObjReader::from_vec(b"must-not-commit".to_vec());
let result = restarted.pools[0].disk_set[0]
.put_object(BUCKET, "rejected-object", &mut reader, &ObjectOptions::default())
.await;
let error = match result {
Err(error) => error,
Ok(_) => panic!("unknown reservation protocol with quota={quota_shape} must not admit a PUT"),
};
assert!(
matches!(error, EcstoreError::PartMissingOrCorrupt),
"unexpected protocol rejection: {error}"
);
let missing = restarted.pools[0].disk_set[0]
.get_object_info(BUCKET, "rejected-object", &ObjectOptions::default())
.await
.expect_err("the rejected PUT must not create an object");
assert!(
matches!(missing, EcstoreError::FileNotFound | EcstoreError::ObjectNotFound(_, _)),
"object absence must not be confused with another storage failure: {missing}"
);
assert_reservation_retained(&restarted, &path, &bytes, incarnation).await;
}
}
@@ -0,0 +1,490 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::super::usage_store::DataUsagePublicationResult;
use super::*;
use crate::scanner_io::ScannerBucketScanScope;
use rustfs_utils::path::path_join_buf;
use sha2::Digest;
use std::time::SystemTime;
const PROOF_BUCKET: &str = "publication-proof-bucket";
const PROOF_EPOCH: u64 = 7;
const PROOF_CYCLE: u64 = 11;
async fn settle_namespace_commits(store: &ECStore) {
tokio::time::timeout(Duration::from_secs(30), async {
while store.scanner_data_usage_publication_blocked().await {
tokio::time::sleep(Duration::from_millis(1)).await;
}
})
.await
.expect("fixture namespace commits must settle before collecting complete coverage");
}
async fn complete_candidate(store: &Arc<ECStore>, cycle: u64) -> (crate::scanner_io::ScannerCycleResult, DataUsageInfo) {
settle_namespace_commits(store).await;
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new_with_progress_tracking(
&ctx,
ScannerCycleBudgetConfig {
max_objects: Some(8),
..Default::default()
},
);
let (updates, mut receiver) = mpsc::channel(1);
let result = crate::scanner_io::nsscanner_with_storage_status_scoped(
store.as_ref(),
crate::scanner_io::ScannerCycleRequest {
ctx,
budget,
updates,
want_cycle: cycle,
leader_epoch: PROOF_EPOCH,
scan_mode: HealScanMode::Normal,
scan_scope: ScannerBucketScanScope::default(),
persisted_usage_baseline: None,
observed_usage_candidate: None,
requires_full_scan: true,
service_cohort: None,
resolved_scope_observer: None,
},
)
.await
.expect("real scanner must produce the fixture candidate");
assert_eq!(result.status, ScannerCycleStatus::Complete);
let candidate = receiver.recv().await.expect("complete scanner snapshot");
assert!(candidate.usage_snapshot_complete);
assert_eq!(candidate.usage_snapshot_converged, Some(true));
assert_eq!(candidate.scanner_cycle, Some(cycle));
assert_eq!(candidate.scanner_epoch, Some(PROOF_EPOCH));
(result, candidate)
}
async fn candidate_store() -> (tempfile::TempDir, Arc<ECStore>) {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let (directory, store) = setup_scanner_cycle_store_with_usage_baseline(false).await;
store
.make_bucket(PROOF_BUCKET, &crate::storage_api::scan::MakeBucketOptions::default())
.await
.expect("create proof fixture bucket through the owner");
let mut reader = PutObjReader::from_vec(b"proof".to_vec());
store.pools[0].disk_set[0]
.put_object(PROOF_BUCKET, "initial", &mut reader, &ObjectOptions::default())
.await
.expect("persist fixture object through the owner");
crate::scanner_io::record_dirty_usage_bucket(PROOF_BUCKET);
settle_namespace_commits(&store).await;
(directory, store)
}
async fn read_root(store: &Arc<ECStore>) -> (Option<Vec<u8>>, DataUsageCacheRevision) {
read_config_with_revision(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("read actual v2 root bytes and revision")
}
async fn publish_candidate(
store: &Arc<ECStore>,
scan: &crate::scanner_io::ScannerCycleResult,
candidate: DataUsageInfo,
baseline: Option<DataUsagePersistBaseline>,
) -> DataUsagePublicationResult {
let expectation = scan.publication_expectation();
assert!(expectation.is_some(), "only a real complete scan may supply the expectation");
let (sender, receiver) = mpsc::channel(1);
sender.send(candidate).await.expect("enqueue the real scan candidate");
drop(sender);
store_data_usage_in_backend_with_outcome_for_epoch_and_baseline_and_route_probe_for_publication_epoch_and_lease_fence(
CancellationToken::new(),
store.clone(),
receiver,
Some(PROOF_EPOCH),
baseline,
ScannerPublicationFence::new(scan.publication_epoch(), None, None).with_ack_expectation(expectation),
|| async { None },
)
.await
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_companion_only_does_not_authorize_root_ack() {
for companion in [
format!("{}.bkp", DATA_USAGE_OBJ_NAME_PATH.as_str()),
LEGACY_DATA_USAGE_OBJ_NAME_PATH.to_string(),
format!("{}.bkp", LEGACY_DATA_USAGE_OBJ_NAME_PATH.as_str()),
] {
let (_directory, store) = candidate_store().await;
let (scan, candidate) = complete_candidate(&store, PROOF_CYCLE).await;
let bytes = serde_json::to_vec(&candidate).expect("actual candidate JSON");
save_config(store.clone(), &companion, bytes.clone())
.await
.expect("persist the companion on real disks");
let baseline = read_data_usage_persist_baseline(store.clone())
.await
.expect("companion fallback baseline");
assert_eq!(baseline.data.as_deref(), Some(bytes.as_slice()));
assert_eq!(baseline.revision, DataUsageCacheRevision::Missing);
assert_eq!(read_root(&store).await.0, None);
let dirty = crate::scanner_io::dirty_usage_buckets_for_tests();
let publication = publish_candidate(&store, &scan, candidate, Some(baseline)).await;
assert_eq!(publication.outcome(), DataUsagePersistOutcome::AlreadyDurable);
let (_, pending, acknowledgements) = finalize_scanner_cycle_result(scan, publication);
assert!(pending, "unacknowledged durable companion work must remain pending");
assert!(acknowledgements.is_empty());
assert_eq!(
crate::scanner_io::dirty_usage_buckets_for_tests(),
dirty,
"a companion is not the v2 root target"
);
assert_eq!(read_root(&store).await, (None, DataUsageCacheRevision::Missing));
assert_eq!(read_config(store.clone(), &companion).await.expect("companion retained"), bytes);
}
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_actual_root_readback_accepts_semantic_json_equivalence() {
let (_directory, store) = candidate_store().await;
let (scan, candidate) = complete_candidate(&store, PROOF_CYCLE).await;
let canonical = serde_json::to_vec(&candidate).expect("candidate encoding");
let mut value = serde_json::to_value(&candidate).expect("candidate value");
value
.as_object_mut()
.expect("usage object")
.insert("fixture_unknown_field".into(), serde_json::json!({"retained": true}));
let different_bytes = serde_json::to_vec_pretty(&value).expect("noncanonical primary JSON");
assert_ne!(different_bytes, canonical);
assert_eq!(
serde_json::from_slice::<DataUsageInfo>(&different_bytes).expect("semantic primary"),
candidate
);
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), different_bytes.clone())
.await
.expect("persist actual primary representation");
let before = read_root(&store).await;
assert!(matches!(&before.1, DataUsageCacheRevision::Etag(etag) if !etag.is_empty()));
let baseline = read_data_usage_persist_baseline(store.clone())
.await
.expect("real primary revision");
assert!(crate::scanner_io::dirty_usage_buckets_pending());
let publication = publish_candidate(&store, &scan, candidate.clone(), Some(baseline)).await;
assert_eq!(publication.outcome(), DataUsagePersistOutcome::AlreadyDurable);
let (_, proof) = publication.into_parts();
let proof = proof.expect("actual primary readback must produce its own root proof");
let expected = scan.publication_expectation().expect("real scan expectation");
let (etag, raw_digest) = proof.verified_version_for(&expected).expect("proof must bind this candidate");
let DataUsageCacheRevision::Etag(expected_etag) = &before.1 else { panic!("actual root ETag") };
assert_eq!(etag, expected_etag);
let expected_digest: [u8; 32] = sha2::Sha256::digest(&different_bytes).into();
assert_eq!(
*raw_digest, expected_digest,
"proof must record actual bytes, not reserialized candidate bytes"
);
// Obtain another proof through the same real readback path rather than
// fabricating a publication result from the inspected proof above.
let publication = publish_candidate(&store, &scan, candidate, None).await;
let (outcome, _, acknowledgements) = finalize_scanner_cycle_result(scan, publication);
assert_eq!(outcome, ScannerCycleOutcome::Completed);
assert!(acknowledgements.is_empty(), "the single-node fixture has no remote targets");
assert!(
!crate::scanner_io::dirty_usage_buckets_pending(),
"actual root bytes plus a real revision authorize this scan"
);
assert_eq!(read_root(&store).await, before, "readback must not rewrite unknown fields or whitespace");
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_successful_root_cas_authorizes_its_scan() {
let (_directory, store) = candidate_store().await;
let (scan, candidate) = complete_candidate(&store, PROOF_CYCLE).await;
let baseline = read_data_usage_persist_baseline(store.clone())
.await
.expect("initial root revision");
assert_eq!(baseline.revision, DataUsageCacheRevision::Missing);
assert!(crate::scanner_io::dirty_usage_buckets_pending());
let publication = publish_candidate(&store, &scan, candidate.clone(), Some(baseline)).await;
assert_eq!(publication.outcome(), DataUsagePersistOutcome::Saved);
let (bytes, revision) = read_root(&store).await;
assert!(matches!(revision, DataUsageCacheRevision::Etag(etag) if !etag.is_empty()));
assert_eq!(
serde_json::from_slice::<DataUsageInfo>(&bytes.expect("actual saved root")).expect("root JSON"),
candidate
);
let (outcome, pending, acknowledgements) = finalize_scanner_cycle_result(scan, publication);
assert_eq!(outcome, ScannerCycleOutcome::Completed);
assert!(!pending);
assert!(acknowledgements.is_empty(), "the single-node fixture has no remote targets");
assert!(
!crate::scanner_io::dirty_usage_buckets_pending(),
"the real root CAS must authorize its matching scan"
);
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_observed_candidate_reuse_requires_a_new_root_proof() {
let (_directory, store) = candidate_store().await;
let bootstrap = scanner_usage_bootstrap_marker(SystemTime::now(), Some(PROOF_EPOCH));
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&bootstrap).expect("bootstrap root encoding"),
)
.await
.expect("persist authoritative bootstrap root");
let (prior_scan, mut observed_candidate) = complete_candidate(&store, PROOF_CYCLE).await;
// Seed a complete but unconverged observation from real scanner coverage;
// the production writer attaches its authoritative baseline identity.
observed_candidate.usage_snapshot_converged = Some(false);
let observation = publish_candidate(&store, &prior_scan, observed_candidate, None).await;
let (outcome, proof) = observation.into_parts();
assert_eq!(outcome, DataUsagePersistOutcome::Saved);
assert!(proof.is_none(), "an observational write cannot authorize a root ACK");
let (root_before, revision_before) = read_root(&store).await;
let observed = read_config(store.clone(), DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str())
.await
.expect("read real persisted observation");
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
let (updates, mut receiver) = mpsc::channel(1);
let (observer, selected) = tokio::sync::oneshot::channel();
let scan = crate::scanner_io::nsscanner_with_storage_status_scoped(
store.as_ref(),
crate::scanner_io::ScannerCycleRequest {
ctx,
budget,
updates,
want_cycle: PROOF_CYCLE + 1,
leader_epoch: PROOF_EPOCH,
scan_mode: HealScanMode::Normal,
scan_scope: ScannerBucketScanScope::default(),
persisted_usage_baseline: root_before.clone().map(Bytes::from),
observed_usage_candidate: Some(Bytes::from(observed)),
requires_full_scan: false,
service_cohort: None,
resolved_scope_observer: Some(observer),
},
)
.await
.expect("observation-backed scope must run through the real scanner");
let scope = selected.await.expect("production resolver decision");
assert_eq!(scope.selected_buckets_for_tests(), Some(&HashSet::from([PROOF_BUCKET.to_string()])));
assert_eq!(scan.status, ScannerCycleStatus::Complete);
let expectation = scan.publication_expectation().expect("reused coverage must be revalidated");
assert!(
!expectation.same_candidate(&prior_scan.publication_expectation().expect("prior real candidate")),
"the observation cannot transfer the previous scan's expectation"
);
assert_eq!(read_root(&store).await, (root_before, revision_before));
assert!(crate::scanner_io::dirty_usage_buckets_pending());
let candidate = receiver.recv().await.expect("new validated root candidate");
assert_eq!(candidate.scanner_cycle, Some(PROOF_CYCLE + 1));
assert_eq!(candidate.usage_snapshot_converged, Some(true));
let publication = publish_candidate(&store, &scan, candidate, None).await;
assert_eq!(publication.outcome(), DataUsagePersistOutcome::Saved);
let (outcome, pending, acknowledgements) = finalize_scanner_cycle_result(scan, publication);
assert_eq!(outcome, ScannerCycleOutcome::Completed);
assert!(!pending);
assert!(acknowledgements.is_empty());
assert!(!crate::scanner_io::dirty_usage_buckets_pending());
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_stale_root_cas_keeps_dirty_after_bucket_save() {
let (_directory, store) = candidate_store().await;
let (scan, candidate) = complete_candidate(&store, PROOF_CYCLE).await;
let mut bucket_cache = DataUsageCache::default();
bucket_cache
.load(store.pools[0].disk_set[0].clone(), &path_join_buf(&[PROOF_BUCKET, DATA_USAGE_CACHE_NAME]))
.await
.expect("real bucket checkpoint must be persisted before root publication");
assert!(bucket_cache.info.snapshot_complete);
assert_eq!(
bucket_cache
.checked_flatten(PROOF_BUCKET)
.expect("persisted bucket root")
.objects,
1
);
let stale_baseline = read_data_usage_persist_baseline(store.clone())
.await
.expect("missing root revision");
assert_eq!(stale_baseline.revision, DataUsageCacheRevision::Missing);
let mut competing = candidate.clone();
competing.scanner_epoch = Some(PROOF_EPOCH + 1);
competing.scanner_cycle = Some(PROOF_CYCLE + 1);
for state in &mut competing.usage_snapshot_set_states {
state.scanner_epoch = Some(PROOF_EPOCH + 1);
state.scanner_cycle = Some(PROOF_CYCLE + 1);
}
let competing_bytes = serde_json::to_vec(&competing).expect("competing root");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), competing_bytes.clone())
.await
.expect("another publisher wins the actual root slot");
let before = read_root(&store).await;
let dirty = crate::scanner_io::dirty_usage_buckets_for_tests();
let publication = publish_candidate(&store, &scan, candidate, Some(stale_baseline)).await;
assert_eq!(
publication.outcome(),
DataUsagePersistOutcome::Current,
"the old missing revision loses CAS and reconciles the newer root"
);
let (_, _, acknowledgements) = finalize_scanner_cycle_result(scan, publication);
assert!(acknowledgements.is_empty());
assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty);
assert_eq!(
read_root(&store).await,
before,
"bucket durability must not authorize replacing the winning root"
);
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_cannot_transfer_proof_between_real_scan_results() {
let (_directory, store) = candidate_store().await;
let (first_scan, first_candidate) = complete_candidate(&store, PROOF_CYCLE).await;
let (second_scan, second_candidate) = complete_candidate(&store, PROOF_CYCLE).await;
assert_eq!(first_candidate.scanner_epoch, second_candidate.scanner_epoch);
assert_eq!(first_candidate.scanner_cycle, second_candidate.scanner_cycle);
assert_eq!(first_candidate.objects_total_count, second_candidate.objects_total_count);
let baseline = read_data_usage_persist_baseline(store.clone())
.await
.expect("initial root revision");
let dirty = crate::scanner_io::dirty_usage_buckets_for_tests();
let publication = publish_candidate(&store, &first_scan, first_candidate, Some(baseline)).await;
assert_eq!(publication.outcome(), DataUsagePersistOutcome::Saved);
assert!(read_root(&store).await.0.is_some(), "the first scan really published its root");
let (_, pending, acknowledgements) = finalize_scanner_cycle_result(second_scan, publication);
assert!(pending, "another scan's publication must not finish this scan's dirty maintenance work");
assert!(acknowledgements.is_empty());
assert_eq!(
crate::scanner_io::dirty_usage_buckets_for_tests(),
dirty,
"same counters and cycle cannot transfer another scan's proof"
);
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_stale_baseline_cannot_prove_a_replaced_root() {
let (_directory, store) = candidate_store().await;
let (first_scan, first_candidate) = complete_candidate(&store, PROOF_CYCLE).await;
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&first_candidate).expect("first candidate"),
)
.await
.expect("persist the first candidate on real disks");
let stale_baseline = read_data_usage_persist_baseline(store.clone())
.await
.expect("capture the genuine first root revision");
let dirty = crate::scanner_io::dirty_usage_buckets_for_tests();
let mut reader = PutObjReader::from_vec(b"second".to_vec());
store.pools[0].disk_set[0]
.put_object(PROOF_BUCKET, "second", &mut reader, &ObjectOptions::default())
.await
.expect("commit a real namespace change");
assert_eq!(
crate::scanner_io::dirty_usage_buckets_for_tests(),
dirty,
"direct storage writes leave this fixture's scanner hint generation unchanged"
);
let (_, replacement) = complete_candidate(&store, PROOF_CYCLE).await;
assert_eq!(first_candidate.scanner_epoch, replacement.scanner_epoch);
assert_eq!(first_candidate.scanner_cycle, replacement.scanner_cycle);
assert_eq!((first_candidate.objects_total_count, replacement.objects_total_count), (1, 2));
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&replacement).expect("replacement candidate"),
)
.await
.expect("publish the replacement root");
let current = read_root(&store).await;
assert_ne!(current.1, stale_baseline.revision);
// The supplied baseline still equals candidate A, but the actual target
// now contains B. Compatibility's AlreadyDurable outcome is not proof.
let publication = publish_candidate(&store, &first_scan, first_candidate, Some(stale_baseline)).await;
assert_eq!(publication.outcome(), DataUsagePersistOutcome::AlreadyDurable);
let (_, pending, acknowledgements) = finalize_scanner_cycle_result(first_scan, publication);
assert!(pending);
assert!(acknowledgements.is_empty());
assert_eq!(crate::scanner_io::dirty_usage_buckets_for_tests(), dirty);
assert_eq!(read_root(&store).await, current);
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[serial]
async fn scoped_ack_publication_rejects_builder_mutation_after_real_root_publish() {
for mutation in ["remote_ack_target", "publication_epoch", "remote_lease_targets"] {
let (_directory, store) = candidate_store().await;
let (scan, candidate) = complete_candidate(&store, PROOF_CYCLE).await;
let baseline = read_data_usage_persist_baseline(store.clone())
.await
.expect("initial root revision");
let dirty = crate::scanner_io::dirty_usage_buckets_for_tests();
let changed_generation = dirty
.get(PROOF_BUCKET)
.expect("the real scan has dirty work")
.checked_add(1)
.expect("bounded fixture generation");
let changed_epoch = scan
.publication_epoch()
.expect("real scan publication epoch")
.checked_add(1)
.expect("bounded fixture epoch");
let publication = publish_candidate(&store, &scan, candidate.clone(), Some(baseline)).await;
assert_eq!(publication.outcome(), DataUsagePersistOutcome::Saved, "{mutation}");
let root_before = read_root(&store).await;
assert_eq!(
serde_json::from_slice::<DataUsageInfo>(root_before.0.as_deref().expect("actual saved root"))
.expect("persisted root JSON"),
candidate,
"{mutation}: the original candidate really reached root storage"
);
let changed = match mutation {
"remote_ack_target" => scan.with_remote_dirty_usage_acknowledgements(vec![ScannerDirtyUsageAcknowledgement {
host: "proof-peer:9000".to_string(),
instance_id: crate::scanner_activity_epoch().to_string(),
generation: changed_generation,
}]),
"publication_epoch" => scan.with_publication_epoch(Some(changed_epoch)),
"remote_lease_targets" => scan.with_remote_publication_lease_targets(vec![(
"proof-peer:9000".to_string(),
crate::scanner_activity_epoch().to_string(),
changed_generation,
)]),
_ => unreachable!("fixed mutation cases"),
};
let (_, pending, acknowledgements) = finalize_scanner_cycle_result(changed, publication);
assert!(
acknowledgements.is_empty(),
"{mutation}: the old root proof must not authorize changed ACK work"
);
assert!(pending, "{mutation}: changed maintenance work must remain pending");
assert_eq!(
crate::scanner_io::dirty_usage_buckets_for_tests(),
dirty,
"{mutation}: the changed scan must not clear local dirty work"
);
assert_eq!(read_root(&store).await, root_before, "{mutation}: the durable original root is retained");
}
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
+192 -15
View File
@@ -34,6 +34,139 @@ pub(super) enum DataUsagePersistOutcome {
Failed,
}
#[derive(Debug)]
pub(crate) struct RootPublicationProof {
candidate: crate::scanner_io::ScannerPublicationExpectation,
root_version: (String, [u8; 32]),
}
impl RootPublicationProof {
pub(crate) fn verified_version_for(
&self,
expected: &crate::scanner_io::ScannerPublicationExpectation,
) -> Option<&(String, [u8; 32])> {
self.candidate.same_candidate(expected).then_some(&self.root_version)
}
}
#[derive(Debug)]
pub(super) struct DataUsagePublicationResult {
outcome: DataUsagePersistOutcome,
proof: Option<RootPublicationProof>,
}
impl From<DataUsagePersistOutcome> for DataUsagePublicationResult {
fn from(outcome: DataUsagePersistOutcome) -> Self {
Self { outcome, proof: None }
}
}
impl DataUsagePublicationResult {
pub(super) fn outcome(&self) -> DataUsagePersistOutcome {
self.outcome
}
pub(super) fn restrict_outcome(&mut self, outcome: DataUsagePersistOutcome) {
if outcome != self.outcome {
self.proof = None;
}
self.outcome = outcome;
}
pub(super) fn into_parts(self) -> (DataUsagePersistOutcome, Option<RootPublicationProof>) {
(self.outcome, self.proof)
}
}
fn root_ack_write_is_confirmed<T, E>(
result: &std::result::Result<T, E>,
state: Option<ScannerPublicationCommitState>,
written_etag: Option<&str>,
) -> bool {
result.is_ok() && state == Some(ScannerPublicationCommitState::Committed) && written_etag.is_some_and(|etag| !etag.is_empty())
}
#[cfg(test)]
mod root_publication_confirmation_tests {
use super::*;
#[test]
fn root_publication_confirmation_requires_committed_state_and_write_revision() {
let saved = Ok::<(), ()>(());
for state in [
None,
Some(ScannerPublicationCommitState::Admitted),
Some(ScannerPublicationCommitState::InFlight),
Some(ScannerPublicationCommitState::AbortedBeforeCommit),
Some(ScannerPublicationCommitState::Indeterminate),
] {
assert!(!root_ack_write_is_confirmed(&saved, state, Some("revision")));
}
for etag in [None, Some("")] {
assert!(!root_ack_write_is_confirmed(&saved, Some(ScannerPublicationCommitState::Committed), etag));
}
assert!(root_ack_write_is_confirmed(
&saved,
Some(ScannerPublicationCommitState::Committed),
Some("revision")
));
}
#[test]
fn root_publication_confirmation_does_not_carry_state_across_cas_attempts() {
let attempts = [
(Err(()), Some(ScannerPublicationCommitState::Committed), Some("first")),
(Ok(()), Some(ScannerPublicationCommitState::AbortedBeforeCommit), Some("second")),
(Ok(()), None, Some("legacy")),
(Ok(()), Some(ScannerPublicationCommitState::Committed), Some("confirmed")),
];
let confirmations = attempts
.iter()
.map(|(result, state, etag)| root_ack_write_is_confirmed(result, *state, *etag))
.collect::<Vec<_>>();
assert_eq!(confirmations, [false, false, false, true]);
}
}
async fn read_root_publication_proof<S: ScannerObjectIO + ScannerConfigObjectDelete>(
store: Arc<S>,
ctx: &CancellationToken,
deadline: tokio::time::Instant,
epoch: u64,
expected: &crate::scanner_io::ScannerPublicationExpectation,
candidate: &DataUsageInfo,
written_etag: Option<&str>,
) -> Option<RootPublicationProof> {
let read = async {
let _admission = scanner_publication_admission_for_epoch(store.clone(), epoch).await?;
let (bytes, revision) = read_config_with_revision(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.ok()?;
let bytes = bytes?;
let DataUsageCacheRevision::Etag(etag) = revision else {
return None;
};
if etag.is_empty() || written_etag.is_some_and(|written| written != etag) {
return None;
}
let persisted: DataUsageInfo = serde_json::from_slice(&bytes).ok()?;
if &persisted != candidate {
return None;
}
let root_digest = Sha256::digest(&bytes).into();
if ctx.is_cancelled() || tokio::time::Instant::now() >= deadline {
return None;
}
Some(RootPublicationProof {
candidate: expected.clone(),
root_version: (etag, root_digest),
})
};
tokio::select! {
biased;
_ = ctx.cancelled() => None,
result = tokio::time::timeout_at(deadline, read) => result.ok().flatten(),
}
}
fn remote_lease_expired(deadline: Option<std::time::Instant>) -> bool {
deadline.is_some_and(|deadline| std::time::Instant::now() >= deadline)
}
@@ -166,6 +299,7 @@ pub(super) struct ScannerPublicationFence {
pub(super) scanner_publication_lease_fence: Option<String>,
pub(super) remote_lease_tokens: Vec<Uuid>,
pub(super) lease_release_safe: Arc<AtomicBool>,
pub(super) ack_expectation: Option<crate::scanner_io::ScannerPublicationExpectation>,
}
impl ScannerPublicationFence {
@@ -180,6 +314,7 @@ impl ScannerPublicationFence {
scanner_publication_lease_fence,
remote_lease_tokens: Vec::new(),
lease_release_safe: Arc::new(AtomicBool::new(true)),
ack_expectation: None,
}
}
@@ -192,21 +327,26 @@ impl ScannerPublicationFence {
self.lease_release_safe = lease_release_safe;
self
}
pub(super) fn with_ack_expectation(mut self, expected: Option<crate::scanner_io::ScannerPublicationExpectation>) -> Self {
self.ack_expectation = expected;
self
}
}
#[derive(Debug)]
pub(super) enum DataUsagePersistTaskResult {
Completed(DataUsagePersistOutcome),
pub(super) enum DataUsagePersistTaskResult<T = DataUsagePersistOutcome> {
Completed(T),
Cancelled,
TimedOut,
JoinFailed(tokio::task::JoinError),
}
pub(super) async fn wait_for_data_usage_persist_task(
pub(super) async fn wait_for_data_usage_persist_task<T>(
ctx: &CancellationToken,
task: &mut AbortOnDropHandle<DataUsagePersistOutcome>,
task: &mut AbortOnDropHandle<T>,
timeout: Duration,
) -> DataUsagePersistTaskResult {
) -> DataUsagePersistTaskResult<T> {
tokio::select! {
biased;
result = &mut *task => match result {
@@ -320,6 +460,7 @@ where
route_probe,
)
.await
.outcome()
}
pub(super) async fn store_data_usage_in_backend_with_outcome_for_epoch_and_baseline_and_route_probe_for_publication_epoch_and_lease_fence<
@@ -333,7 +474,7 @@ pub(super) async fn store_data_usage_in_backend_with_outcome_for_epoch_and_basel
initial_baseline: Option<DataUsagePersistBaseline>,
publication_fence: ScannerPublicationFence,
route_probe: F,
) -> DataUsagePersistOutcome
) -> DataUsagePublicationResult
where
F: Fn() -> Fut + Send + Sync,
Fut: Future<Output = Option<ScannerCycleDeferReason>> + Send,
@@ -344,11 +485,15 @@ where
scanner_publication_lease_fence,
remote_lease_tokens,
lease_release_safe,
ack_expectation,
} = publication_fence;
let ack_deadline = scanner_publication_scope_deadline(data_usage_persist_timeout(), remote_lease_deadline);
let mut outcome = DataUsagePersistOutcome::NoUpdate;
let mut proof = None;
let mut next_baseline = initial_baseline;
'updates: while let Some(mut data_usage_info) = receiver.recv().await {
proof = None;
let _activity_guard = ScannerActivityGuard::new();
if ctx.is_cancelled() {
break;
@@ -523,10 +668,14 @@ where
continue;
}
};
let sha256hex = (!data.is_empty()).then(|| hex_simd::encode_to_string(Sha256::digest(&data), hex_simd::AsciiCase::Lower));
let data_digest: [u8; 32] = Sha256::digest(&data).into();
let sha256hex = (!data.is_empty()).then(|| hex_simd::encode_to_string(data_digest, hex_simd::AsciiCase::Lower));
let data = Bytes::from(data);
let backup_due = !observational && data_usage_backup_due(&data_usage_info);
let mut cas_retry = 0usize;
let mut ack_epoch = None;
let mut write_confirmed = false;
let mut written_etag = None;
let save_outcome = loop {
if ctx.is_cancelled() {
break 'updates;
@@ -557,6 +706,7 @@ where
} else {
None
};
ack_epoch = Some(publication_epoch_for_save);
let (existing_data, revision) = match baseline {
Some(baseline) => (baseline.data, baseline.revision),
None => match read_config_with_revision(storeapi.clone(), target_path).await {
@@ -645,7 +795,7 @@ where
}
let done_save = Metrics::time(Metric::SaveUsage);
let save_result = {
let (save_result, commit_state) = {
let publication_scope = storeapi
.scanner_data_usage_publication_commit_scope_with_release_flag(
publication_epoch_for_save,
@@ -681,24 +831,33 @@ where
.await;
drop(legacy_publication_admission);
if let Some(scope) = publication_scope {
match scope.wait_for_completion().await {
ScannerPublicationCommitState::Committed | ScannerPublicationCommitState::AbortedBeforeCommit => {
save_result
}
let state = scope.wait_for_completion().await;
let result = match state {
ScannerPublicationCommitState::Committed => save_result,
ScannerPublicationCommitState::AbortedBeforeCommit => save_result,
ScannerPublicationCommitState::Indeterminate
| ScannerPublicationCommitState::Admitted
| ScannerPublicationCommitState::InFlight => Err(EcstoreError::other(
"scanner publication commit scope did not reach a safe terminal state",
)),
}
};
(result, Some(state))
} else {
save_result
(save_result, None)
}
};
done_save();
let attempt_confirmed = root_ack_write_is_confirmed(
&save_result,
commit_state,
save_result.as_ref().ok().and_then(|info| info.etag.as_deref()),
);
match save_result {
Ok(object_info) => {
write_confirmed = attempt_confirmed;
written_etag = object_info.etag.as_ref().filter(|etag| !etag.is_empty()).cloned();
if !observational {
next_baseline = object_info
.etag
@@ -909,9 +1068,27 @@ where
break 'updates;
}
}
if !observational
&& data_usage_info.usage_snapshot_converged == Some(true)
&& matches!(outcome, DataUsagePersistOutcome::Saved | DataUsagePersistOutcome::AlreadyDurable)
&& (outcome == DataUsagePersistOutcome::AlreadyDurable || write_confirmed)
&& let (Some(expected), Some(epoch)) = (ack_expectation.as_ref(), ack_epoch)
&& expected.matches_encoded_candidate(&data_digest)
{
proof = read_root_publication_proof(
storeapi.clone(),
&ctx,
ack_deadline,
epoch,
expected,
&data_usage_info,
written_etag.as_deref(),
)
.await;
}
}
outcome
DataUsagePublicationResult { outcome, proof }
}
async fn cleanup_observed_data_usage_snapshot_for_epoch_and_lease(
+44 -3
View File
@@ -114,6 +114,11 @@ pub(crate) struct ScannerBucketScanScope {
}
impl ScannerBucketScanScope {
#[cfg(test)]
pub(crate) fn selected_buckets_for_tests(&self) -> Option<&HashSet<String>> {
self.selected_buckets.as_deref()
}
fn is_default(&self) -> bool {
self.selected_buckets.is_none() && self.baseline_scan_plan_digest.is_none()
}
@@ -218,8 +223,8 @@ fn complete_scanner_cache_snapshot_plan_digest(
fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<'_>) -> Option<DataUsageScanPlanDigest> {
let authoritative = serde_json::from_slice::<DataUsageInfo>(proof.authoritative_data?).ok()?;
if complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, true).is_some() {
return Some(proof.scan_plan_digest);
if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, true) {
return Some(validated_digest);
}
// A complete but superseded observation may reuse its per-set cache only
@@ -896,6 +901,7 @@ pub(crate) struct ScannerCycleResult {
failed_dirty_usage: bool,
pending_maintenance_work: bool,
required_cycle_floor: Option<u64>,
publication_expectation: Option<ScannerPublicationExpectation>,
}
impl ScannerCycleResult {
@@ -911,10 +917,12 @@ impl ScannerCycleResult {
failed_dirty_usage: false,
pending_maintenance_work: false,
required_cycle_floor: None,
publication_expectation: None,
}
}
pub(crate) fn with_publication_epoch(mut self, publication_epoch: Option<u64>) -> Self {
self.publication_expectation = None;
self.publication_epoch = publication_epoch;
self
}
@@ -924,6 +932,7 @@ impl ScannerCycleResult {
}
fn with_activity_digest(mut self, activity_digest: [u8; 32]) -> Self {
self.publication_expectation = None;
self.activity_digest = Some(activity_digest);
self
}
@@ -933,6 +942,7 @@ impl ScannerCycleResult {
}
pub(crate) fn with_observational_snapshot_published(mut self, published: bool) -> Self {
self.publication_expectation = None;
self.observational_snapshot_published = published;
self
}
@@ -942,16 +952,19 @@ impl ScannerCycleResult {
}
fn with_failed_dirty_usage(mut self, failed_dirty_usage: bool) -> Self {
self.publication_expectation = None;
self.failed_dirty_usage = failed_dirty_usage;
self
}
fn with_pending_maintenance_work(mut self, pending_maintenance_work: bool) -> Self {
self.publication_expectation = None;
self.pending_maintenance_work = pending_maintenance_work;
self
}
fn with_required_cycle_floor(mut self, required_cycle_floor: Option<u64>) -> Self {
self.publication_expectation = None;
self.required_cycle_floor = required_cycle_floor;
self
}
@@ -960,11 +973,13 @@ impl ScannerCycleResult {
mut self,
acknowledgements: Vec<crate::scanner::ScannerDirtyUsageAcknowledgement>,
) -> Self {
self.publication_expectation = None;
self.remote_dirty_usage_acknowledgements = acknowledgements;
self
}
pub(crate) fn with_remote_publication_lease_targets(mut self, targets: Vec<(String, String, u64)>) -> Self {
self.publication_expectation = None;
self.remote_publication_lease_targets = targets;
self
}
@@ -973,7 +988,32 @@ impl ScannerCycleResult {
&self.remote_publication_lease_targets
}
pub(crate) fn acknowledge_durable_usage(self) -> Vec<crate::scanner::ScannerDirtyUsageAcknowledgement> {
pub(crate) fn publication_expectation(&self) -> Option<ScannerPublicationExpectation> {
self.publication_expectation.clone()
}
fn with_publication_expectation(mut self, expectation: Option<ScannerPublicationExpectation>) -> Self {
// Seal only after all coverage and acknowledgement inputs are final.
self.publication_expectation = expectation;
self
}
pub(crate) fn acknowledge_durable_usage(
self,
proof: &crate::scanner::RootPublicationProof,
) -> Vec<crate::scanner::ScannerDirtyUsageAcknowledgement> {
if self.status != ScannerCycleStatus::Complete
|| self
.publication_expectation
.as_ref()
.is_none_or(|expected| proof.verified_version_for(expected).is_none())
{
return Vec::new();
}
self.clear_verified_usage()
}
fn clear_verified_usage(self) -> Vec<crate::scanner::ScannerDirtyUsageAcknowledgement> {
if let Some(snapshot) = self.dirty_usage_clear {
clear_dirty_usage_buckets(&snapshot);
}
@@ -1013,6 +1053,7 @@ mod publish_gate_tests;
#[cfg(test)]
mod tests;
pub(crate) use cache::ScannerPublicationExpectation;
use cache::*;
use dirty_usage::*;
use guards::*;
+98 -1
View File
@@ -308,6 +308,86 @@ impl<'a> ValidatedScannerSnapshot<'a> {
}
}
#[derive(Clone, Debug)]
pub(crate) struct ScannerPublicationExpectation {
candidate: Arc<([u8; 32], DataUsageScanPlanDigest)>,
}
impl ScannerPublicationExpectation {
pub(crate) fn matches_encoded_candidate(&self, digest: &[u8; 32]) -> bool {
&self.candidate.0 == digest
}
pub(crate) fn same_candidate(&self, other: &Self) -> bool {
Arc::ptr_eq(&self.candidate, &other.candidate) && self.candidate.1 == other.candidate.1
}
}
pub(super) struct ValidatedUsageCandidate {
data: DataUsageInfo,
#[cfg(test)]
last_update: SystemTime,
coverage_digest: DataUsageScanPlanDigest,
}
pub(super) fn empty_namespace_usage_candidate(
all_buckets: &[BucketInfo],
sources: &HashSet<DataUsageCacheSource>,
buckets_by_source: &HashMap<DataUsageCacheSource, Vec<BucketInfo>>,
identity: ScannerSnapshotIdentity,
) -> Option<ValidatedUsageCandidate> {
if !all_buckets.is_empty()
|| sources.is_empty()
|| sources.len() != buckets_by_source.len()
|| sources
.iter()
.any(|source| buckets_by_source.get(source).is_none_or(|buckets| !buckets.is_empty()))
{
return None;
}
let last_update = SystemTime::now();
Some(ValidatedUsageCandidate {
data: DataUsageInfo {
last_update: Some(last_update),
scanner_cycle: Some(identity.cycle),
scanner_epoch: Some(identity.leader_epoch),
usage_snapshot_complete: true,
..Default::default()
},
#[cfg(test)]
last_update,
coverage_digest: identity.coverage_digest,
})
}
impl ValidatedUsageCandidate {
pub(super) fn prepare(mut self, status: ScannerCycleStatus) -> (DataUsageInfo, Option<ScannerPublicationExpectation>) {
self.data.usage_snapshot_converged = Some(status == ScannerCycleStatus::Complete);
let expectation = if status == ScannerCycleStatus::Complete {
struct DigestWriter(Sha256);
impl std::io::Write for DigestWriter {
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
self.0.update(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
let mut writer = DigestWriter(Sha256::new());
serde_json::to_writer(&mut writer, &self.data)
.ok()
.map(|()| ScannerPublicationExpectation {
candidate: Arc::new((writer.0.finalize().into(), self.coverage_digest)),
})
} else {
None
};
(self.data, expectation)
}
}
#[cfg(test)]
pub(super) fn completed_data_usage_info(
results: &[DataUsageCache],
scope: &ScannerSnapshotScope<'_>,
@@ -316,6 +396,18 @@ pub(super) fn completed_data_usage_info(
budget_elapsed: bool,
cancelled: bool,
) -> Option<(DataUsageInfo, SystemTime)> {
completed_usage_candidate(results, scope, tier_registry_names, bucket_plan_complete, budget_elapsed, cancelled)
.map(|candidate| (candidate.data, candidate.last_update))
}
pub(super) fn completed_usage_candidate(
results: &[DataUsageCache],
scope: &ScannerSnapshotScope<'_>,
tier_registry_names: &[String],
bucket_plan_complete: bool,
budget_elapsed: bool,
cancelled: bool,
) -> Option<ValidatedUsageCandidate> {
if !bucket_plan_complete {
return None;
}
@@ -393,7 +485,12 @@ pub(super) fn completed_data_usage_info(
usage_snapshot_set_states,
..Default::default()
};
Some((data_usage_info, merged_last_update))
Some(ValidatedUsageCandidate {
data: data_usage_info,
#[cfg(test)]
last_update: merged_last_update,
coverage_digest: scope.identity.coverage_digest,
})
}
fn tier_accounting_proof_is_publishable(
+23 -9
View File
@@ -338,12 +338,21 @@ where
dirty_usage_status,
activity_status,
);
let empty_usage = DataUsageInfo {
last_update: Some(SystemTime::now()),
scanner_cycle: Some(want_cycle),
usage_snapshot_complete: true,
..Default::default()
let Some(candidate) = empty_namespace_usage_candidate(
&all_buckets,
&expected_sources,
&buckets_by_source,
ScannerSnapshotIdentity {
cycle: want_cycle,
leader_epoch,
plan_digest: scan_plan_digest,
coverage_digest: bucket_coverage_digest,
tier_registry_generation: Some(tier_registry_generation),
},
) else {
return Ok(ScannerCycleResult::new(ScannerCycleStatus::Incomplete, None).with_publication_epoch(publication_epoch));
};
let (empty_usage, publication_expectation) = candidate.prepare(status);
let observational_snapshot_published = if should_publish_observational_snapshot(status) {
publish_observational_snapshot(&updates, empty_usage).await?
} else {
@@ -366,7 +375,8 @@ where
.with_activity_digest(activity_digest)
.with_observational_snapshot_published(observational_snapshot_published)
.with_remote_publication_lease_targets(remote_publication_lease_targets)
.with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements));
.with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements)
.with_publication_expectation(publication_expectation));
}
let total_results = expected_sources.len();
@@ -595,7 +605,7 @@ where
let (activity_status, remote_publication_lease_targets) =
scanner_cycle_activity_status(store, distributed, &activity_before).await;
let all_bucket_names = all_buckets.iter().map(|bucket| bucket.name.clone()).collect::<Vec<_>>();
let completed_usage = completed_data_usage_info(
let completed_usage = completed_usage_candidate(
&results,
&ScannerSnapshotScope {
sources: &expected_sources,
@@ -636,7 +646,10 @@ where
dirty_usage_status,
activity_status,
);
let observational_snapshot_published = if let Some((data_usage_info, _)) = completed_usage {
let mut publication_expectation = None;
let observational_snapshot_published = if let Some(candidate) = completed_usage {
let (data_usage_info, expectation) = candidate.prepare(cycle_status);
publication_expectation = expectation;
if should_publish_observational_snapshot(cycle_status) {
publish_observational_snapshot(&updates, data_usage_info).await?
} else {
@@ -674,5 +687,6 @@ where
.with_remote_dirty_usage_acknowledgements(remote_dirty_usage_acknowledgements)
.with_failed_dirty_usage(!failed_buckets.is_empty())
.with_pending_maintenance_work(pending_maintenance_work)
.with_required_cycle_floor(required_cycle_floor))
.with_required_cycle_floor(required_cycle_floor)
.with_publication_expectation(publication_expectation))
}
+13 -3
View File
@@ -453,7 +453,16 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure(
.put_object(bucket, "initial", &mut reader, &ScannerObjectOptions::default())
.await
.expect("initial object should persist");
let lock = store.pools[0].disk_set[0]
.new_ns_lock(bucket, "initial")
.await
.expect("fixture namespace lock should be created");
let _settled = lock
.get_write_lock(Duration::from_secs(30))
.await
.expect("fixture rename tail should finish before the usage scan");
}
wait_for_namespace_commit_tails(&store).await;
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
let (updates, receiver) = mpsc::channel(1);
@@ -503,6 +512,7 @@ async fn scoped_scan_same_cycle_maintenance_rewalks_after_root_delivery_failure(
.put_object("cold-bucket", "new", &mut reader, &ScannerObjectOptions::default())
.await
.expect("new cold object should persist");
wait_for_namespace_commit_tails(&store).await;
record_dirty_usage_bucket("hot-bucket");
if scan_mode == HealScanMode::Normal && !requires_full_scan {
record_dirty_usage_bucket("cold-bucket");
@@ -994,8 +1004,8 @@ fn dirty_usage_snapshot_clears_a_stably_absent_bucket_after_durable_save() {
assert!(dirty_usage_buckets().contains_key("temporarily-omitted"));
assert_eq!(dirty_usage_snapshot_status(&snapshot), DirtyUsageSnapshotStatus::Current);
let acknowledgements = ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(snapshot.buckets.as_ref().clone()))
.acknowledge_durable_usage();
let acknowledgements =
ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(snapshot.buckets.as_ref().clone())).clear_verified_usage();
assert!(acknowledgements.is_empty());
assert!(!dirty_usage_buckets().contains_key("temporarily-omitted"));
clear_dirty_usage_buckets_for_tests();
@@ -1101,7 +1111,7 @@ fn dirty_usage_is_acknowledged_only_after_durable_usage_confirmation() {
assert!(dirty_usage_buckets().contains_key("photos"));
let confirmed = ScannerCycleResult::new(ScannerCycleStatus::Complete, Some(snapshot.buckets.as_ref().clone()));
let acknowledgements = confirmed.acknowledge_durable_usage();
let acknowledgements = confirmed.clear_verified_usage();
assert!(acknowledgements.is_empty());
assert!(!dirty_usage_buckets().contains_key("photos"));
clear_dirty_usage_buckets_for_tests();
@@ -55,6 +55,15 @@ Standard S3 areas that must not be described as complete:
`excluded_tests.txt` holds tests that must not block the compatibility gate: vendor-specific or non-portable behavior, and intentionally unsupported product behavior such as ACL authorization.
## Intentional Deviations From AWS S3
Object keys are stored as file-system paths under each drive (`{drive}/{bucket}/{object}/xl.meta`), the same layout MinIO uses. The rules below exist to keep that layout unambiguous and are not compatibility gaps to close; clients that need the AWS behavior must adapt on their side.
| Behavior | RustFS | AWS S3 | Why |
|---|---|---|---|
| Object key with a `.` or `..` path segment, or an empty segment (`//`), such as `a//b/./c/../d` | `400 InvalidArgument` (`check_object_args` in `crates/ecstore/src/bucket/utils.rs`, mirroring MinIO `IsValidObjectPrefix`) | Accepted as an opaque key | A `..` segment would resolve to a parent directory and `.`/`//` segments would alias other keys on disk; encoding them would change the MinIO-compatible on-disk format. |
| Directory marker (key ending in `/`, with or without a body) in a versioned bucket | Stored as the null version: `PutObject`/`HeadObject` report version id `00000000-0000-0000-0000-000000000000`, `ListObjectVersions` reports `null`, and a later PUT of the same key overwrites in place (`put_opts` in `rustfs/src/storage/options.rs`, mirroring MinIO `putOpts`: "for directory objects skip creating new versions") | A real version id per PUT, with a version history | The marker only exists to make an empty prefix listable; keeping a history for it would leave hidden versions behind every prefix delete. Replication still copies the marker as its null version (`test_bucket_replication_replicates_directory_marker_in_versioned_bucket` in `crates/e2e_test/src/replication_extension_test.rs`). |
## Update Rule
When a feature starts passing, move its test entries from `unimplemented_tests.txt` to `implemented_tests.txt` and update the row here in the same PR. Do not change README wording beyond the supported coverage. Handler-level status (missing, stubbed, or diverging endpoints) is tracked in [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md).
+2
View File
@@ -255,6 +255,8 @@ Source-backed responses report only what the source can vouch for: its ETag, its
- The source ETag is always recorded in internal metadata regardless of the policy, so an audit or a later comparison can still see it.
- `Last-Modified` of a pulled object is the local write time, not the source's. The source timestamp is preserved in metadata.
A preserved source ETag does not describe the local part layout. Part reads and object attributes use the stored parts, and replication uses multipart transport when their logical boundaries are available. Legacy compressed or encrypted objects without a multipart ETag can lack those boundaries; they retain their existing streaming PUT path and its 5 GiB limit. Their replication target may therefore store a different part layout.
## Metadata mapping
Copied to the local object:
@@ -299,6 +299,24 @@ Heal knobs are environment-only and read by `HealConfig::default` (`crates/heal/
| `RUSTFS_HEAL_MRF_REPLAY_BATCH` | `256` (`DEFAULT_HEAL_MRF_REPLAY_BATCH`) | Intents per replay push round. |
| `RUSTFS_HEAL_DANGLING_DELETE_GRACE_SECS` | `3600` (`DEFAULT_HEAL_DANGLING_DELETE_GRACE_SECS`, `crates/ecstore/src/set_disk/core/io_primitives.rs`) | A recently modified object is never deleted as dangling inside this window; `0` disables the grace window. |
### Admin heal start, retries, and budgets
Admin heal has three separate budgets. Increasing one does not extend the others:
| Budget | Existing behavior |
|---|---|
| Control request | A start/query/cancel envelope has a bounded lifetime derived from the internode RPC timeout, with room for the transport response. It bounds control execution, not the admitted repair task. The HTTP caller can also stop waiting independently. |
| Task execution | `RUSTFS_HEAL_TASK_TIMEOUT_SECS` supplies the default of 300 seconds when the execution has no explicit timeout. Elapsed execution time is deducted before a recoverable scheduler retry, which keeps the request identity and remaining budget. Object/listing backoff and pressure pacing inside an execution consume that budget; time queued or in the scheduler's between-attempt backoff is not a new absolute wall-clock deadline. Zero is not an unlimited execution budget. |
| Object/listing retry | A recursive bucket/prefix traversal retries recoverable failures at most three times after the initial attempt, with 2/4/8-second delays plus the existing task-derived jitter. Object retries also obey the bounded delayed window and its 30-second age, checked at safe boundaries; listing retries keep their cursor. These limits do not extend the task execution budget or force an in-flight storage operation to finish within the retry age. |
For a large admin heal, select a finite configured execution budget appropriate to the expected work and contention, and inspect progress while it runs. Investigate stalled work and object failures before increasing this budget; a larger timeout must not hide lock or quorum problems. A longer HTTP or internode timeout does not keep a repair task alive past its execution budget. A successful start response returns the canonical token for accepted or merged work; it does not prove that repair has completed. Query that token without `forceStart`; terminal timeout/cancellation reports retain completed progress while the task report remains available.
Capability preflight runs before constructing and admitting a start request. However, the public `cluster heal coordination unavailable` error can also follow a transport failure or an invalid coordinator response after a request was sent. An unknown or lost HTTP response is therefore not proof that no task was admitted.
The coordinator can replay the result of the exact original RPC envelope within its existing replay lifetime and coordinator epoch. Reusing an ID with changed parameters or nonce is rejected. This bounded, process-local receipt cache is not a general HTTP idempotency key or a restart-surviving admission receipt. Each new HTTP start constructs a new request/envelope, and a fresh `forceStart` intentionally requests a distinct start: do not automatically resend it after an ambiguous response. A fresh non-forced request follows the configured overlap policy, rather than recovering the original receipt. The v3 API rejects `forceStart` combined with `clientToken` or `forceStop`.
Implementation references: `rustfs/src/admin/handlers/heal.rs` (`submit_cluster_heal_start`, `new_heal_control_metadata`), `rustfs/src/storage/rpc/node_service.rs` (`execute_heal_control_envelope_with_manager`), `crates/protos/src/lib.rs` (`heal_control_execution_timeout`), and `crates/heal/src/heal/task.rs` (`retry_request_with_remaining_timeout`, `remaining_timeout`, `bucket_object_retry_delay`). These contracts do not replace the separate real response-loss, long-task, or start-latency validation lanes.
### Running admin heal pacing
The manager passes its existing workload provider and a configuration snapshot into each admin execution. Bucket/prefix listing and object boundaries resample foreground pressure; erasure-set page workers also resample after earlier work releases page capacity. `High`, `Urgent`, and `force_start` do not exempt ordinary admin execution from this runtime pacing. The existing start-time bypass and overlap-control meanings are unchanged.
File diff suppressed because it is too large Load Diff
+49 -4
View File
@@ -63,6 +63,7 @@ const EVENT_ADMIN_REQUEST_STATE: &str = "admin_request_state";
const EVENT_ADMIN_REQUEST_REJECTED: &str = "admin_request_rejected";
const EVENT_ADMIN_REQUEST_FAILED: &str = "admin_request_failed";
const EVENT_ADMIN_RESPONSE_EMITTED: &str = "admin_response_emitted";
const POOL_ACTIVATION_FLEET_PROOF_REQUIRED: &str = "pool activation requires a live fleet capability proof";
fn admin_request_id(headers: &HeaderMap) -> Option<&str> {
headers
@@ -321,6 +322,17 @@ fn contextualize_admin_pool_api_error(
}
}
fn decommission_start_api_error(err: crate::storage_api::error::StorageError) -> ApiError {
if crate::storage_api::capacity::is_pool_activation_fleet_proof_error(&err) {
return ApiError {
code: S3ErrorCode::InternalError,
message: POOL_ACTIVATION_FLEET_PROOF_REQUIRED.to_string(),
source: Some(Box::new(err)),
};
}
ApiError::from(err)
}
fn decommission_admin_not_initialized_error_with_audit(operation: &str, audit: PoolAuditContext<'_>) -> S3Error {
error!(
event = EVENT_ADMIN_REQUEST_FAILED,
@@ -790,7 +802,24 @@ impl Operation for StartDecommission {
store
.decommission(ctx.clone(), pools_indices.clone())
.await
.map_err(ApiError::from)
.map_err(|err| {
error!(
event = EVENT_ADMIN_REQUEST_FAILED,
component = LOG_COMPONENT_ADMIN_API,
subsystem = LOG_SUBSYSTEM_POOL_ADMIN,
operation = "start_decommission",
action = "start_decommission",
result = "failed",
reason = "storage_decommission_failed",
request_id = %request_id,
actor = %actor,
remote_addr = %remote_addr,
pool_indices = ?pools_indices,
error = %err,
"admin request failed"
);
decommission_start_api_error(err)
})
.map_err(|err| contextualize_admin_pool_api_error(err, "start decommission", &pool_context))?;
}
}
@@ -1018,9 +1047,10 @@ impl Operation for ClearDecommission {
#[cfg(test)]
mod pools_handler_tests {
use super::{
AdminPoolStatus, Body, CancelDecommission, ClearDecommission, HeaderMap, ListPools, Method, Operation, Params,
PoolAuditContext, S3ErrorCode, S3Request, StartDecommission, StatusDecommission, StatusPool, Uri,
contextualize_admin_pool_api_error, decommission_admin_not_initialized_error_with_audit, decommission_peer_target,
AdminPoolStatus, Body, CancelDecommission, ClearDecommission, HeaderMap, ListPools, Method, Operation,
POOL_ACTIVATION_FLEET_PROOF_REQUIRED, Params, PoolAuditContext, S3ErrorCode, S3Request, StartDecommission,
StatusDecommission, StatusPool, Uri, contextualize_admin_pool_api_error,
decommission_admin_not_initialized_error_with_audit, decommission_peer_target, decommission_start_api_error,
has_duplicate_indices, parse_mutation_pool_query, parse_pool_idx_by_id, parse_status_pool_query,
pool_admin_missing_credentials_error, pool_admin_missing_credentials_error_with_request,
pool_admin_pool_index_error_with_audit, pool_admin_pool_not_found_error_with_audit,
@@ -1209,6 +1239,21 @@ mod pools_handler_tests {
);
}
#[test]
fn test_decommission_start_api_error_preserves_fleet_proof_retry_marker() {
let err = crate::storage_api::error::StorageError::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED);
let err = decommission_start_api_error(err);
assert_eq!(err.code, s3s::S3ErrorCode::InternalError);
assert_eq!(err.message, POOL_ACTIVATION_FLEET_PROOF_REQUIRED);
assert!(err.source.is_some());
let unrelated = decommission_start_api_error(crate::storage_api::error::StorageError::other("disk read failed"));
assert_eq!(unrelated.code, s3s::S3ErrorCode::InternalError);
assert_eq!(unrelated.message, "We encountered an internal error, please try again.");
}
#[test]
fn test_contextualize_admin_pool_api_error_preserves_source() {
let err = contextualize_admin_pool_api_error(
+14
View File
@@ -507,6 +507,18 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
LIST_TIER,
RouteRiskLevel::Sensitive,
),
admin(
HttpMethod::Post,
"/rustfs/admin/v3/ilm/recovery/records/{control_id}",
SET_TIER,
RouteRiskLevel::High,
),
admin(
HttpMethod::Get,
"/rustfs/admin/v3/ilm/recovery/exports/{export_id}",
SET_TIER,
RouteRiskLevel::High,
),
admin(HttpMethod::Post, "/rustfs/admin/v3/ilm/transition/run", SET_TIER, RouteRiskLevel::High),
admin(
HttpMethod::Get,
@@ -2173,6 +2185,8 @@ mod tests {
fn route_policy_uses_tier_actions_for_transition_routes() {
assert_action(HttpMethod::Get, "/rustfs/admin/v3/ilm/recovery/records", LIST_TIER);
assert_action(HttpMethod::Get, "/rustfs/admin/v3/ilm/recovery/records/{control_id}", LIST_TIER);
assert_action(HttpMethod::Post, "/rustfs/admin/v3/ilm/recovery/records/{control_id}", SET_TIER);
assert_action(HttpMethod::Get, "/rustfs/admin/v3/ilm/recovery/exports/{export_id}", SET_TIER);
assert_action(HttpMethod::Post, "/rustfs/admin/v3/ilm/transition/run", SET_TIER);
assert_action(HttpMethod::Get, "/rustfs/admin/v3/ilm/transition/jobs/{job_id}", SET_TIER);
assert_action(HttpMethod::Delete, "/rustfs/admin/v3/ilm/transition/jobs/{job_id}", SET_TIER);
@@ -215,6 +215,16 @@ fn expected_admin_route_matrix() -> Vec<RouteMatrixEntry> {
"/v3/ilm/recovery/records/{control_id}",
"/v3/ilm/recovery/records/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
),
admin_route_sample(
Method::POST,
"/v3/ilm/recovery/records/{control_id}",
"/v3/ilm/recovery/records/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
),
admin_route_sample(
Method::GET,
"/v3/ilm/recovery/exports/{export_id}",
"/v3/ilm/recovery/exports/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
),
admin_route(Method::POST, "/v3/ilm/transition/run"),
admin_route_sample(
Method::GET,
@@ -942,6 +952,16 @@ fn test_register_routes_cover_representative_admin_paths() {
Method::GET,
&admin_path("/v3/ilm/recovery/records/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"),
);
assert_route(
&router,
Method::POST,
&admin_path("/v3/ilm/recovery/records/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"),
);
assert_route(
&router,
Method::GET,
&admin_path("/v3/ilm/recovery/exports/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"),
);
assert_route(&router, Method::POST, &admin_path("/v3/ilm/transition/run"));
assert_route(
&router,
+4 -1
View File
@@ -233,7 +233,10 @@ pub(crate) mod lifecycle {
super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::ManualTransitionRunOptions;
pub(crate) type ManualTransitionRunReport = super::ecstore_bucket::lifecycle::bucket_lifecycle_ops::ManualTransitionRunReport;
pub(crate) use super::ecstore_bucket::lifecycle::recovery_control::{
IlmRecoveryClassification, IlmRecoveryProtocol, inspect_recovery_control, list_recovery_controls,
IlmRecoveryClassification, IlmRecoveryControlView, IlmRecoveryProtocol, inspect_recovery_control, list_recovery_controls,
};
pub(crate) use super::ecstore_bucket::lifecycle::recovery_export::{
IlmRecoveryExportObservation, create_recovery_export, inspect_recovery_export_observation, load_recovery_export,
};
pub(crate) use super::ecstore_bucket::lifecycle::transition_transaction::{
TransitionOperatorDeleteResult, TransitionOperatorError, delete_transition_candidate_for_operator,
+5 -6
View File
@@ -4070,14 +4070,13 @@ mod tests {
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: Some(s3s::dto::LifecycleRuleFilter {
prefix: Some("logs/".to_string()),
tag: Some(s3s::dto::Tag {
key: Some("env".to_string()),
value: Some("prod".to_string()),
and: Some(s3s::dto::LifecycleRuleAndOperator {
prefix: Some("logs/".to_string()),
..Default::default()
}),
..Default::default()
}),
id: Some("two-predicates".to_string()),
id: Some("one-member-and".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
@@ -4087,7 +4086,7 @@ mod tests {
&ObjectLockConfiguration::default(),
)
.await
.expect_err("a Filter with two predicates is a schema violation");
.expect_err("a Filter And with one predicate is a schema violation");
assert_eq!(*lifecycle_validation_error(&malformed).code(), S3ErrorCode::MalformedXML);
let invalid_value = validate_lifecycle_config(
+40 -12
View File
@@ -187,6 +187,30 @@ static NODE_CAPABILITY_SERVER_EPOCH: LazyLock<Uuid> = LazyLock::new(Uuid::new_v4
// operations do not add another peer RPC.
const CROSS_POOL_FENCE_SUPPORTED_VERSION: u32 = 4;
fn encode_heal_capability_response(
topology_member: &str,
remote_version_state_probe: bool,
recovery_export_probe: bool,
) -> Result<Vec<u8>, Status> {
if recovery_export_probe {
rustfs_protos::encode_remote_version_state_capability(
topology_member,
crate::storage::storage_api::ilm_recovery_export_local_process_epoch().as_bytes(),
)
.map_err(|_| Status::internal("ILM recovery export capability length cannot be represented"))
} else if remote_version_state_probe {
rustfs_protos::encode_remote_version_state_capability(topology_member, NODE_CAPABILITY_SERVER_EPOCH.as_bytes())
.map_err(|_| Status::internal("remote version state capability length cannot be represented"))
} else {
rustfs_protos::encode_cross_pool_fence_capability(
CROSS_POOL_FENCE_SUPPORTED_VERSION,
topology_member,
NODE_CAPABILITY_SERVER_EPOCH.as_bytes(),
)
.map_err(|_| Status::internal("cross-pool fence capability length cannot be represented"))
}
}
fn admit_heal_control_replay(
replay_cache: &mut HashMap<String, Arc<HealControlReplayEntry>>,
request_id: &str,
@@ -1003,7 +1027,8 @@ impl heal_control_service_server::HealControlService for HealControlRpcService {
}
let remote_version_state_probe = rustfs_protos::is_remote_version_state_capability_probe(&request.get_ref().command);
let cross_pool_fence_probe = rustfs_protos::is_cross_pool_fence_capability_probe(&request.get_ref().command);
if remote_version_state_probe || cross_pool_fence_probe {
let recovery_export_probe = rustfs_protos::is_ilm_recovery_export_capability_probe(&request.get_ref().command);
if remote_version_state_probe || cross_pool_fence_probe || recovery_export_probe {
let topology_member = self
.endpoint_pools()
.await
@@ -1013,17 +1038,7 @@ impl heal_control_service_server::HealControlService for HealControlRpcService {
if topology_member.is_empty() {
return Err(Status::failed_precondition("local topology member identity is unavailable"));
}
let result = if remote_version_state_probe {
rustfs_protos::encode_remote_version_state_capability(&topology_member, NODE_CAPABILITY_SERVER_EPOCH.as_bytes())
.map_err(|_| Status::internal("remote version state capability length cannot be represented"))?
} else {
rustfs_protos::encode_cross_pool_fence_capability(
CROSS_POOL_FENCE_SUPPORTED_VERSION,
&topology_member,
NODE_CAPABILITY_SERVER_EPOCH.as_bytes(),
)
.map_err(|_| Status::internal("cross-pool fence capability length cannot be represented"))?
};
let result = encode_heal_capability_response(&topology_member, remote_version_state_probe, recovery_export_probe)?;
let canonical_response = rustfs_protos::canonical_heal_control_response_body(
request.get_ref().version,
&request.get_ref().topology_fingerprint,
@@ -4040,6 +4055,19 @@ mod tests {
.expect_err("proof from one challenge must not be reusable");
}
#[test]
fn ilm_recovery_export_probe_uses_the_shared_local_process_epoch() {
let result = super::encode_heal_capability_response("node-a:9000", false, true)
.expect("ILM recovery export capability should encode");
let (member, epoch) =
rustfs_protos::decode_remote_version_state_capability(&result).expect("ILM recovery export capability should decode");
assert_eq!(member, "node-a:9000");
assert_eq!(
Uuid::from_slice(epoch).expect("capability epoch should be a UUID"),
crate::storage::storage_api::ilm_recovery_export_local_process_epoch(),
);
}
#[tokio::test]
async fn cross_pool_fence_probe_authenticates_supported_v4_state() {
let _ = rustfs_credentials::set_global_rpc_secret("cross-pool-fence-node-service-test-secret".to_string());
+7 -3
View File
@@ -417,7 +417,7 @@ pub(crate) mod ecstore_bucket {
pub(crate) mod ecstore_capacity {
pub(crate) use rustfs_ecstore::api::capacity::{
DecommissionUnresolvedEntry, PoolDecommissionInfo, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
is_reserved_or_invalid_bucket,
is_pool_activation_fleet_proof_error, is_reserved_or_invalid_bucket,
};
}
@@ -513,8 +513,8 @@ pub(crate) mod ecstore_notification {
pub(crate) use rustfs_ecstore::api::notification::rotate_cross_pool_fence_fleet_proof_for_test;
pub(crate) use rustfs_ecstore::api::notification::{
ClusterTierDailyStats, CrossPoolFenceFleetProofToken, NotificationSys, acquire_cross_pool_fence_fleet_proof,
cross_pool_fence_fleet_proof_matches, get_global_notification_sys, new_global_notification_sys,
start_remote_version_state_fleet_probe,
cross_pool_fence_fleet_proof_matches, get_global_notification_sys, ilm_recovery_export_local_process_epoch,
new_global_notification_sys, start_remote_version_state_fleet_probe,
};
}
@@ -1182,6 +1182,10 @@ pub(crate) fn start_remote_version_state_fleet_probe(topology_fingerprint: Strin
ecstore_notification::start_remote_version_state_fleet_probe(topology_fingerprint);
}
pub(crate) fn ilm_recovery_export_local_process_epoch() -> uuid::Uuid {
ecstore_notification::ilm_recovery_export_local_process_epoch()
}
pub(crate) async fn read_config(api: Arc<ECStore>, file: &str) -> Result<Vec<u8>> {
ecstore_config::com::read_config(api, file).await
}
+2
View File
@@ -18,6 +18,8 @@
use rustfs_storage_api as storage_contracts;
pub(crate) mod capacity {
pub(crate) use crate::storage::storage_api::ecstore_capacity::is_pool_activation_fleet_proof_error;
pub(crate) mod service {
pub(crate) use crate::storage::storage_api::{all_local_disk, disk_drive_path, disk_endpoint};
}
@@ -1,8 +1,8 @@
{
"bytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Qub2ZmbGluZS5lbnJvbGxtZW50Q2hhbGxlbmdlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsImNoYWxsZW5nZUlkIjoiMDE4ZjdlNmQtOWQ2YS03ZDkzLThmNjQtOGIyMGIzMzg0NzEyIiwib3JnYW5pemF0aW9uTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE4ZjdlNmQtOWQ2YS03ZDkzLThmNjQtOGIyMGIzMzg0NzEzIiwiY2x1c3Rlck5hbWUiOiJvcmdhbml6YXRpb25zLzAxOGY3ZTZkLTlkNmEtN2Q5My04ZjY0LThiMjBiMzM4NDcxMy9jbHVzdGVycy8wMThmN2U2ZC05ZDZhLTdkOTMtOGY2NC04YjIwYjMzODQ3MTQiLCJub25jZSI6InBLU2twS1NrcEtTa3BLU2twS1NrcEtTa3BLU2twS1NrcEtTa3BLU2twS1EiLCJpc3N1ZWRBdCI6IjIwOTktMDEtMDFUMDA6MDA6MDBaIiwiZXhwaXJlc0F0IjoiMjA5OS0wMS0wOFQwMDowMDowMFoiLCJjb25uZWN0S2V5SWQiOiIwNjA3NWY0ODRkY2U1YzA3NDVjYTc2NWE4ODk5MzA5YmQ3MTc2MTI5NDdhMzc3MGEwZDI3YjhjY2I4YmY5MDUyIiwidHJ1c3RDaGFpbiI6W3siYnl0ZXMiOiJleUptYjNKdFlYUldaWEp6YVc5dUlqb2ljblZ6ZEdaekxtTnZibTVsWTNRdWIyWm1iR2x1WlM1MGNuVnpkRXhwYm1zdk1TSXNJbkJ5YjNSdlkyOXNWbVZ5YzJsdmJpSTZJbll4SWl3aWMyVnlhV0ZzSWpvaVpUQTBZVEF3TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNREVpTENKeWIyeGxJam9pYVc1MFpYSnRaV1JwWVhSbElpd2lhWE56ZFdWeVMyVjVTV1FpT2lJNVlUVmpNemcxWVdKa09UZGpOMlppT0RSak5UUXdZalZtTldVME1tUTNPVGt4T1RNd1lUUTBOMk01TURJMk56azVaVGhtWkRSak5ESXpNR0kxWXpOaklpd2ljM1ZpYW1WamRFdGxlVWxrSWpvaVlUazFOVGxpWkRSbE1EazBPV0ZqWXpFeU1tRm1abU0xTldVMlpHSTRaVGt4Wmpoa01EazNaVFUxWldNNE9UbGtZVFl4T1RGbE5EWTVZelE0T0RKaE1pSXNJbk4xWW1wbFkzUlFkV0pzYVdOTFpYa2lPaUpDVG13elZFeDFTazV0Tld4NFRGUmFNRGxUWVdwWVRHMUZSM0ZqZHpGaVQyZzROVWhEYmpWdVVUazJWVlZUYUZVNFZHeEhVVWsxYmxaSldWUnJRWFpLYjFWRVMwWmhVVlp0WVZGVk9WRkhaemsyVDJ0RU1EUWlMQ0p1YjNSQ1pXWnZjbVVpT2lJeU1ESXdMVEF4TFRBeFZEQXdPakF3T2pBd1dpSXNJbTV2ZEVGbWRHVnlJam9pTWpBNU9TMHdNUzB3TVZRd01Eb3dNRG93TUZvaWZRPT0iLCJzaWduYXR1cmUiOnsiYWxnb3JpdGhtIjoiRVMyNTYiLCJrZXlJZCI6IjlhNWMzODVhYmQ5N2M3ZmI4NGM1NDBiNWY1ZTQyZDc5OTE5MzBhNDQ3YzkwMjY3OTllOGZkNGM0MjMwYjVjM2MiLCJ2YWx1ZSI6IklUSThKVGFURjRoRHJ0bF9tSk0zYmZ0eUlhMlJaSjFNNVpPVk1xaEE2cVJSSzFNcEcxQlF0emRRVzB6bHo4eG1RZUJNWl84clViTUlFcnFBel8tSWZRIn19LHsiYnl0ZXMiOiJleUptYjNKdFlYUldaWEp6YVc5dUlqb2ljblZ6ZEdaekxtTnZibTVsWTNRdWIyWm1iR2x1WlM1MGNuVnpkRXhwYm1zdk1TSXNJbkJ5YjNSdlkyOXNWbVZ5YzJsdmJpSTZJbll4SWl3aWMyVnlhV0ZzSWpvaVpUQTBZVEF3TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNRElpTENKeWIyeGxJam9pYzJsbmJtbHVaeUlzSW1semMzVmxja3RsZVVsa0lqb2lZVGsxTlRsaVpEUmxNRGswT1dGall6RXlNbUZtWm1NMU5XVTJaR0k0WlRreFpqaGtNRGszWlRVMVpXTTRPVGxrWVRZeE9URmxORFk1WXpRNE9ESmhNaUlzSW5OMVltcGxZM1JMWlhsSlpDSTZJakEyTURjMVpqUTROR1JqWlRWak1EYzBOV05oTnpZMVlUZzRPVGt6TURsaVpEY3hOell4TWprME4yRXpOemN3WVRCa01qZGlPR05qWWpoaVpqa3dOVElpTENKemRXSnFaV04wVUhWaWJHbGpTMlY1SWpvaVFrRk9lWFpITFdkR1YwdFRaM0pMWTNkUVpGUmhNRWN3TTBKdlVtRkNSbWhvZG10cWRFUjZlVk5tYm5weFYxcHROa2xLV0U1RWRVZEdlVXBFZVZWSmNYZzJSa1p5TlZSV2NsOXVhbUZCYzFWb2FEbE1VMlJ2SWl3aWJtOTBRbVZtYjNKbElqb2lNakF5TUMwd01TMHdNVlF3TURvd01Eb3dNRm9pTENKdWIzUkJablJsY2lJNklqSXdPVGt0TURFdE1ERlVNREE2TURBNk1EQmFJbjA9Iiwic2lnbmF0dXJlIjp7ImFsZ29yaXRobSI6IkVTMjU2Iiwia2V5SWQiOiJhOTU1OWJkNGUwOTQ5YWNjMTIyYWZmYzU1ZTZkYjhlOTFmOGQwOTdlNTVlYzg5OWRhNjE5MWU0NjljNDg4MmEyIiwidmFsdWUiOiI4MTVrVGNMY1hkZXZORFRia1dBck1KSnpORTVmaEdxbTNUb3hTM3Exdl9ZQkRVcmFieldmTFhjMEc5TjZiVXZFeEd3WEJUMVFpVzJvUDBCM3JKUS0ydyJ9fV19",
"signature": {
"algorithm": "ES256",
"keyId": "06075f484dce5c0745ca765a8899309bd717612947a3770a0d27b8ccb8bf9052",
"value": "8i2nvYhNsZbL1UPF82Otr4pcecYfKhdRV1Q7m3Geqcx-TdvYk486Q_cZHSQ7OaqnIhTZwPgMYFYKUuEhwAaYEQ"
}
"bytes": "eyJjaGFsbGVuZ2VJZCI6IjAxOGY3ZTZkLTlkNmEtN2Q5My04ZjY0LThiMjBiMzM4NDcxMiIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMThmN2U2ZC05ZDZhLTdkOTMtOGY2NC04YjIwYjMzODQ3MTMvY2x1c3RlcnMvMDE4ZjdlNmQtOWQ2YS03ZDkzLThmNjQtOGIyMGIzMzg0NzE0IiwiY29ubmVjdEtleUlkIjoiZGY4MmNlNjA4MGIwZWZkMDhkNGFlNjNhMjI0NGZmNmNiZWE5YTU0NzUyNjhjZGFkZTMzNzEyMmVjYzY3ZjJhNiIsImV4cGlyZXNBdCI6IjIwOTktMDEtMDhUMDA6MDA6MDBaIiwiZm9ybWF0VmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0Lm9mZmxpbmUuZW5yb2xsbWVudENoYWxsZW5nZS8xIiwiaXNzdWVkQXQiOiIyMDk5LTAxLTAxVDAwOjAwOjAwWiIsIm5vbmNlIjoicEtTa3BLU2twS1NrcEtTa3BLU2twS1NrcEtTa3BLU2twS1NrcEtTa3BLUSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOGY3ZTZkLTlkNmEtN2Q5My04ZjY0LThiMjBiMzM4NDcxMyIsInByb3RvY29sVmVyc2lvbiI6InYxIiwidHJ1c3RDaGFpbiI6W3siYnl0ZXMiOiJleUptYjNKdFlYUldaWEp6YVc5dUlqb2ljblZ6ZEdaekxtTnZibTVsWTNRdWIyWm1iR2x1WlM1MGNuVnpkRXhwYm1zdk1TSXNJbkJ5YjNSdlkyOXNWbVZ5YzJsdmJpSTZJbll4SWl3aWMyVnlhV0ZzSWpvaVpUQTBZVEF3TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNREVpTENKeWIyeGxJam9pYVc1MFpYSnRaV1JwWVhSbElpd2lhWE56ZFdWeVMyVjVTV1FpT2lKbU9EVTNZV1UwWldRMlpUTTBaVE16TnpZeFlXVmhNalZqWVdGbFpUTm1aVFUwWVRFMU9UWXdabUk1TW1SalpEWXpZVE0zTldGaU1USXhaR1ZpTW1FNUlpd2ljM1ZpYW1WamRFdGxlVWxrSWpvaU5XRXdOMkUzTWpOak56azFOamd6TUdGbE1tVTJZbVV3WldZM01XSXpPVGt4TVRZeVpqVmxZVFF5WkRKa05UUmlPRFk1WTJZeU1UWTNNelJrTnpoa015SXNJbk4xWW1wbFkzUlFkV0pzYVdOTFpYa2lPaUpDUmxWUVVuaEJSRGc1TFZoM09UbFJZWE5sV0RsdVNXWnpZVWczWlRRNWRtYzVTV3RUV1hCc2VVazBhMFV5UTFReGQwVjFWVXB3ZW1OV2VUbERkME5xZWtGZk1IUmpRV0pRWDI5YVlYSklOMDF1UVRKMVQxa2lMQ0p1YjNSQ1pXWnZjbVVpT2lJeU1EazRMVEV5TFRBeFZEQXdPakF3T2pBd1dpSXNJbTV2ZEVGbWRHVnlJam9pTWpBNU9TMHhNUzB6TUZRd01Eb3dNRG93TUZvaWZRPT0iLCJzaWduYXR1cmUiOnsiYWxnb3JpdGhtIjoiRVMyNTYiLCJrZXlJZCI6ImY4NTdhZTRlZDZlMzRlMzM3NjFhZWEyNWNhYWVlM2ZlNTRhMTU5NjBmYjkyZGNkNjNhMzc1YWIxMjFkZWIyYTkiLCJ2YWx1ZSI6IlhzbjVaRkI4OHU1bHg1TDU5UDJCWlNLMWNrTVc5ekpiNGFYUmNfdjB5cFIzd1pxc3BlTzJCNzZyV1V0RnhDRE9Vek9GdHFDZlFMRi1McWNsLWE5dFhRIn19LHsiYnl0ZXMiOiJleUptYjNKdFlYUldaWEp6YVc5dUlqb2ljblZ6ZEdaekxtTnZibTVsWTNRdWIyWm1iR2x1WlM1MGNuVnpkRXhwYm1zdk1TSXNJbkJ5YjNSdlkyOXNWbVZ5YzJsdmJpSTZJbll4SWl3aWMyVnlhV0ZzSWpvaVpUQTBZVEF3TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNRElpTENKeWIyeGxJam9pYzJsbmJtbHVaeUlzSW1semMzVmxja3RsZVVsa0lqb2lOV0V3TjJFM01qTmpOemsxTmpnek1HRmxNbVUyWW1Vd1pXWTNNV0l6T1RreE1UWXlaalZsWVRReVpESmtOVFJpT0RZNVkyWXlNVFkzTXpSa056aGtNeUlzSW5OMVltcGxZM1JMWlhsSlpDSTZJbVJtT0RKalpUWXdPREJpTUdWbVpEQTRaRFJoWlRZellUSXlORFJtWmpaalltVmhPV0UxTkRjMU1qWTRZMlJoWkdVek16Y3hNakpsWTJNMk4yWXlZVFlpTENKemRXSnFaV04wVUhWaWJHbGpTMlY1SWpvaVFrWnJZWFF6U0hKMlVERjBia3hyU2xSU1FteExTek5TY0hBeFJYZHpTREpLWDBOS04wWnBOWGhvY21adU1EVnhkbmN3UlZoQmVIQlBhbmh2Y2xoNVdIbHVTeTFhVGpjd2IyMWZjekJ0VUdSdFMydHVaMUJCSWl3aWJtOTBRbVZtYjNKbElqb2lNakE1T0MweE1pMHhOVlF3TURvd01Eb3dNRm9pTENKdWIzUkJablJsY2lJNklqSXdPVGt0TURFdE1UVlVNREE2TURBNk1EQmFJbjA9Iiwic2lnbmF0dXJlIjp7ImFsZ29yaXRobSI6IkVTMjU2Iiwia2V5SWQiOiI1YTA3YTcyM2M3OTU2ODMwYWUyZTZiZTBlZjcxYjM5OTExNjJmNWVhNDJkMmQ1NGI4NjljZjIxNjczNGQ3OGQzIiwidmFsdWUiOiJSUGJHWEt4RVRRRDZKVklEVmcwNkRRaENCQ0NfYjJkay1oZTdVUjZOeXJ0bVlQRUEzb3RhM0hXd25YdUpnWG84TlVWbnF2Q2ZldkNQNGFFelVSbEdaQSJ9fV19",
"signature": {
"algorithm": "ES256",
"keyId": "df82ce6080b0efd08d4ae63a2244ff6cbea9a5475268cdade337122ecc67f2a6",
"value": "m83S9i09JTlxGnhAdnmawYxkM75u_v8F2-5ygK374fs-d0vWh6L_j_yMDS3B29CrViPVjo3FQKaGutw4MDI_LQ"
}
}
+2 -2
View File
@@ -1,4 +1,4 @@
{
"keyId": "9a5c385abd97c7fb84c540b5f5e42d7991930a447c9026799e8fd4c4230b5c3c",
"publicKey": "BIPyiA1W2NDuy3ftHLtlY2tO2WNjkjGINZ_9QQvyyN9syzbMb91QYG2SN0AqZPylFsTL-loF4M1tVySZtXJpKhM"
"keyId": "f857ae4ed6e34e33761aea25caaee3fe54a15960fb92dcd63a375ab121deb2a9",
"publicKey": "BG_wO5SSQc4drdQ1GeaWDgqFtBppoFwygQOqK84VlMoWPE91OlW_AdxT9sCwx-7ni0DG_30lqW4igrmJzvccFEo"
}
@@ -1 +1 @@
[{"bytes":"eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Qub2ZmbGluZS50cnVzdExpbmsvMSIsInByb3RvY29sVmVyc2lvbiI6InYxIiwic2VyaWFsIjoiZTA0YTAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDEiLCJyb2xlIjoiaW50ZXJtZWRpYXRlIiwiaXNzdWVyS2V5SWQiOiI5YTVjMzg1YWJkOTdjN2ZiODRjNTQwYjVmNWU0MmQ3OTkxOTMwYTQ0N2M5MDI2Nzk5ZThmZDRjNDIzMGI1YzNjIiwic3ViamVjdEtleUlkIjoiYTk1NTliZDRlMDk0OWFjYzEyMmFmZmM1NWU2ZGI4ZTkxZjhkMDk3ZTU1ZWM4OTlkYTYxOTFlNDY5YzQ4ODJhMiIsInN1YmplY3RQdWJsaWNLZXkiOiJCTmwzVEx1Sk5tNWx4TFRaMDlTYWpYTG1FR3FjdzFiT2g4NUhDbjVuUTk2VVVTaFU4VGxHUUk1blZJWVRrQXZKb1VES0ZhUVZtYVFVOVFHZzk2T2tEMDQiLCJub3RCZWZvcmUiOiIyMDIwLTAxLTAxVDAwOjAwOjAwWiIsIm5vdEFmdGVyIjoiMjA5OS0wMS0wMVQwMDowMDowMFoifQ==","signature":{"algorithm":"ES256","keyId":"9a5c385abd97c7fb84c540b5f5e42d7991930a447c9026799e8fd4c4230b5c3c","value":"ITI8JTaTF4hDrtl_mJM3bftyIa2RZJ1M5ZOVMqhA6qRRK1MpG1BQtzdQW0zlz8xmQeBMZ_8rUbMIErqAz_-IfQ"}},{"bytes":"eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Qub2ZmbGluZS50cnVzdExpbmsvMSIsInByb3RvY29sVmVyc2lvbiI6InYxIiwic2VyaWFsIjoiZTA0YTAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDIiLCJyb2xlIjoic2lnbmluZyIsImlzc3VlcktleUlkIjoiYTk1NTliZDRlMDk0OWFjYzEyMmFmZmM1NWU2ZGI4ZTkxZjhkMDk3ZTU1ZWM4OTlkYTYxOTFlNDY5YzQ4ODJhMiIsInN1YmplY3RLZXlJZCI6IjA2MDc1ZjQ4NGRjZTVjMDc0NWNhNzY1YTg4OTkzMDliZDcxNzYxMjk0N2EzNzcwYTBkMjdiOGNjYjhiZjkwNTIiLCJzdWJqZWN0UHVibGljS2V5IjoiQkFOeXZHLWdGV0tTZ3JLY3dQZFRhMEcwM0JvUmFCRmhodmtqdER6eVNmbnpxV1ptNklKWE5EdUdGeUpEeVVJcXg2RkZyNVRWcl9uamFBc1VoaDlMU2RvIiwibm90QmVmb3JlIjoiMjAyMC0wMS0wMVQwMDowMDowMFoiLCJub3RBZnRlciI6IjIwOTktMDEtMDFUMDA6MDA6MDBaIn0=","signature":{"algorithm":"ES256","keyId":"a9559bd4e0949acc122affc55e6db8e91f8d097e55ec899da6191e469c4882a2","value":"815kTcLcXdevNDTbkWArMJJzNE5fhGqm3ToxS3q1v_YBDUrabzWfLXc0G9N6bUvExGwXBT1QiW2oP0B3rJQ-2w"}}]
[{"bytes":"eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Qub2ZmbGluZS50cnVzdExpbmsvMSIsInByb3RvY29sVmVyc2lvbiI6InYxIiwic2VyaWFsIjoiZTA0YTAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDEiLCJyb2xlIjoiaW50ZXJtZWRpYXRlIiwiaXNzdWVyS2V5SWQiOiJmODU3YWU0ZWQ2ZTM0ZTMzNzYxYWVhMjVjYWFlZTNmZTU0YTE1OTYwZmI5MmRjZDYzYTM3NWFiMTIxZGViMmE5Iiwic3ViamVjdEtleUlkIjoiNWEwN2E3MjNjNzk1NjgzMGFlMmU2YmUwZWY3MWIzOTkxMTYyZjVlYTQyZDJkNTRiODY5Y2YyMTY3MzRkNzhkMyIsInN1YmplY3RQdWJsaWNLZXkiOiJCRlVQUnhBRDg5LVh3OTlRYXNlWDluSWZzYUg3ZTQ5dmc5SWtTWXBseUk0a0UyQ1Qxd0V1VUpwemNWeTlDd0NqekFfMHRjQWJQX29aYXJIN01uQTJ1T1kiLCJub3RCZWZvcmUiOiIyMDk4LTEyLTAxVDAwOjAwOjAwWiIsIm5vdEFmdGVyIjoiMjA5OS0xMS0zMFQwMDowMDowMFoifQ==","signature":{"algorithm":"ES256","keyId":"f857ae4ed6e34e33761aea25caaee3fe54a15960fb92dcd63a375ab121deb2a9","value":"Xsn5ZFB88u5lx5L59P2BZSK1ckMW9zJb4aXRc_v0ypR3wZqspeO2B76rWUtFxCDOUzOFtqCfQLF-Lqcl-a9tXQ"}},{"bytes":"eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Qub2ZmbGluZS50cnVzdExpbmsvMSIsInByb3RvY29sVmVyc2lvbiI6InYxIiwic2VyaWFsIjoiZTA0YTAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDIiLCJyb2xlIjoic2lnbmluZyIsImlzc3VlcktleUlkIjoiNWEwN2E3MjNjNzk1NjgzMGFlMmU2YmUwZWY3MWIzOTkxMTYyZjVlYTQyZDJkNTRiODY5Y2YyMTY3MzRkNzhkMyIsInN1YmplY3RLZXlJZCI6ImRmODJjZTYwODBiMGVmZDA4ZDRhZTYzYTIyNDRmZjZjYmVhOWE1NDc1MjY4Y2RhZGUzMzcxMjJlY2M2N2YyYTYiLCJzdWJqZWN0UHVibGljS2V5IjoiQkZrYXQzSHJ2UDF0bkxrSlRSQmxLSzNScHAxRXdzSDJKX0NKN0ZpNXhocmZuMDVxdncwRVhBeHBPanhvclh5WHluSy1aTjcwb21fczBtUGRtS2tuZ1BBIiwibm90QmVmb3JlIjoiMjA5OC0xMi0xNVQwMDowMDowMFoiLCJub3RBZnRlciI6IjIwOTktMDEtMTVUMDA6MDA6MDBaIn0=","signature":{"algorithm":"ES256","keyId":"5a07a723c7956830ae2e6be0ef71b3991162f5ea42d2d54b869cf216734d78d3","value":"RPbGXKxETQD6JVIDVg06DQhCBCC_b2dk-he7UR6NyrtmYPEA3ota3HWwnXuJgXo8NUVnqvCfevCP4aEzURlGZA"}}]
-33
View File
@@ -1,33 +0,0 @@
# Legacy Heal Outcome Compatibility
This fixture executes the real `madmin-go` HTTP decoder and a pinned `mc` binary against synthetic v3 responses. Rust owner, admin adapter, and SDK tests validate the same JSON cases in `crates/madmin/tests/fixtures/heal-outcome-v3.json`. It does not run a storage repair or prove distributed recovery.
Pinned primary sources:
- `mc` release `RELEASE.2025-08-13T08-35-41Z`, commit `7394ce0dd2a80935aded936b09fa12cbb3cb8096`: [polling implementation](https://github.com/minio/mc/blob/7394ce0dd2a80935aded936b09fa12cbb3cb8096/cmd/admin-heal-ui.go#L414).
- Its `madmin-go/v3` dependency is `v3.0.107-0.20250415152934-4b504b82db63`: [decoder and response type](https://github.com/minio/madmin-go/blob/4b504b82db633e978a57d49443b2be75824244c3/heal-commands.go#L101).
The old decoder ignores additional JSON fields. The old poller returns success for `finished` without examining `detail`; only `stopped` returns a terminal error. Consequently `completed_with_errors` retains its canonical outcome and complete traversal coverage, but uses legacy summary `stopped`. `completed` describes execution only: unknown storage receipts remain `unknown`, not `repaired`.
Run from the repository root with an isolated tool cache and binary directory:
```sh
(
set -eu
compat_dir=$(mktemp -d)
trap 'rm -rf "$compat_dir"' EXIT
export GOPATH="$compat_dir/gopath" GOMODCACHE="$compat_dir/mod" GOCACHE="$compat_dir/cache" GOBIN="$compat_dir/bin"
export CGO_ENABLED=0 GOTOOLCHAIN=local GOMAXPROCS=2
go install github.com/minio/mc@v0.0.0-20250813083541-7394ce0dd2a8
cd scripts/compat/heal-outcome
MC_BINARY="$compat_dir/bin/mc" NO_PROXY=127.0.0.1,localhost go test -mod=readonly -p 2 -count=1 -v ./...
)
```
The subshell keeps the calling shell unchanged. `MC_BINARY` is mandatory and its Go build metadata must identify the pinned commit. Each subprocess gets a temporary mc configuration directory and synthetic credentials; it never edits the user's mc configuration. Loopback socket permission is required. The Go tests do not skip unavailable prerequisites.
Six cases cover completed traversal, unknown repair proof, completed traversal with failures, cancellation, deadline, and untraversable listing. Two receiver cases carry a remote `finished` summary that contradicts an aborted or completed-with-errors outcome. The admin test applies the heal owner's wire validator to each `remoteResponse` and must produce the corresponding public `response`; the old CLI must then exit with an error. Unknown extension fields remain intact. Unknown or missing execution fields cannot validate a successful summary.
New counters do not replace or reinterpret legacy progress. Outcome is a cumulative snapshot, not a page delta; `sinceSeq` only pages legacy result items. Result cursors and truncation markers remain separate from execution and traversal coverage.
Two existing CLI limitations remain explicit: this mc does not terminate on `notFound`, and `-f` polling sends `forceStart` together with `clientToken`, a combination the RustFS v3 request contract rejects. The fixture uses the standard non-force polling flow. Neither limitation is hidden by emitting a new summary string or reporting a missing task as completed.
-137
View File
@@ -1,137 +0,0 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
package compat_test
import (
"context"
"debug/buildinfo"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
madmin "github.com/minio/madmin-go/v3"
)
type fixture struct {
Name string `json:"name"`
CLIExit int `json:"cliExit"`
Response json.RawMessage `json:"response"`
}
func fixtures(t *testing.T) []fixture {
t.Helper()
data, err := os.ReadFile("../../../crates/madmin/tests/fixtures/heal-outcome-v3.json")
if err != nil {
t.Fatal(err)
}
var cases []fixture
if err := json.Unmarshal(data, &cases); err != nil {
t.Fatal(err)
}
if len(cases) != 8 {
t.Fatalf("expected eight owner/receiver-validated fixtures, got %d", len(cases))
}
return cases
}
func fixtureServer(t *testing.T, response []byte, polls *atomic.Int32) *httptest.Server {
t.Helper()
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost || !strings.HasPrefix(r.URL.Path, "/minio/admin/v3/heal/") {
t.Errorf("unexpected client request %s %s", r.Method, r.URL.Path)
http.Error(w, "unexpected request", http.StatusBadRequest)
return
}
w.Header().Set("Content-Type", "application/json")
if r.URL.Query().Get("clientToken") == "" {
fmt.Fprint(w, `{"clientToken":"fixture-token","clientAddress":"","startTime":"2026-01-01T00:00:00Z"}`)
return
}
polls.Add(1)
w.Write(response)
}))
t.Cleanup(server.Close)
return server
}
func TestLegacyMadminDecoder(t *testing.T) {
for _, f := range fixtures(t) {
t.Run(f.Name, func(t *testing.T) {
var polls atomic.Int32
server := fixtureServer(t, f.Response, &polls)
client, err := madmin.New(strings.TrimPrefix(server.URL, "http://"), "fixture-access", "fixture-secret", false)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, status, err := client.Heal(ctx, "bucket", "", madmin.HealOpts{}, "fixture-token", false, false)
if err != nil {
t.Fatal(err)
}
var expected struct {
Summary string `json:"summary"`
Detail string `json:"detail"`
}
if err := json.Unmarshal(f.Response, &expected); err != nil {
t.Fatal(err)
}
if status.Summary != expected.Summary || status.FailureDetail != expected.Detail || polls.Load() != 1 {
t.Fatalf("decoder changed legacy fields: %+v, polls=%d", status, polls.Load())
}
})
}
}
func TestLegacyMCPoll(t *testing.T) {
binary := os.Getenv("MC_BINARY")
if binary == "" {
t.Fatal("MC_BINARY must point to the pinned mc release; this check cannot be skipped")
}
info, err := buildinfo.ReadFile(binary)
if err != nil {
t.Fatal(err)
}
if info.Main.Path != "github.com/minio/mc" || !strings.Contains(info.Main.Version, "7394ce0dd2a8") {
t.Fatalf("expected mc RELEASE.2025-08-13T08-35-41Z (7394ce0dd2a8), got %+v", info.Main)
}
for _, f := range fixtures(t) {
t.Run(f.Name, func(t *testing.T) {
var polls atomic.Int32
server := fixtureServer(t, f.Response, &polls)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, binary, "--config-dir", filepath.Join(t.TempDir(), "mc"), "--json", "admin", "heal", "--recursive", "w23/bucket")
cmd.Env = append(os.Environ(), "MC_HOST_w23="+strings.Replace(server.URL, "http://", "http://fixture-access:fixture-secret@", 1), "MC_NO_COLOR=1")
output, err := cmd.CombinedOutput()
if ctx.Err() != nil {
t.Fatalf("legacy poll did not terminate: %s", output)
}
exit := 0
if err != nil {
var ok bool
var status *exec.ExitError
status, ok = err.(*exec.ExitError)
if !ok {
t.Fatal(err)
}
exit = status.ExitCode()
}
if exit != f.CLIExit || polls.Load() != 1 {
t.Fatalf("exit=%d expected=%d polls=%d output=%s", exit, f.CLIExit, polls.Load(), output)
}
if f.CLIExit != 0 && !strings.Contains(string(output), "Heal had an error") {
t.Fatalf("failure was not the expected legacy terminal result: %s", output)
}
})
}
}
-34
View File
@@ -1,34 +0,0 @@
module rustfs.local/heal-outcome-compat
go 1.24.0
require github.com/minio/madmin-go/v3 v3.0.107-0.20250415152934-4b504b82db63
require (
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/go-ini/ini v1.67.0 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/minio/minio-go/v7 v7.0.90 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/philhofer/fwd v1.1.3-0.20240916144458-20a13a1f6b7c // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.63.0 // indirect
github.com/prometheus/procfs v0.16.0 // indirect
github.com/prometheus/prom2json v1.4.2 // indirect
github.com/prometheus/prometheus v0.303.0 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/secure-io/sio-go v0.3.1 // indirect
github.com/shirou/gopsutil/v3 v3.24.5 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/tinylib/msgp v1.2.5 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
golang.org/x/crypto v0.37.0 // indirect
golang.org/x/net v0.39.0 // indirect
golang.org/x/sys v0.32.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
)
-64
View File
@@ -1,64 +0,0 @@
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo=
github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/minio/madmin-go/v3 v3.0.107-0.20250415152934-4b504b82db63 h1:ktN/FrMuM9sjvjIbPZYRKeHEzBDOXQdpYUDiNO0CutE=
github.com/minio/madmin-go/v3 v3.0.107-0.20250415152934-4b504b82db63/go.mod h1:U0bL6ip4yKFwvo0keonUcWFQp0Hd462tOLLeVyPzWmE=
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.0.90 h1:TmSj1083wtAD0kEYTx7a5pFsv3iRYMsOJ6A4crjA1lE=
github.com/minio/minio-go/v7 v7.0.90/go.mod h1:uvMUcGrpgeSAAI6+sD3818508nUyMULw94j2Nxku/Go=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/philhofer/fwd v1.1.3-0.20240916144458-20a13a1f6b7c h1:dAMKvw0MlJT1GshSTtih8C2gDs04w8dReiOGXrGLNoY=
github.com/philhofer/fwd v1.1.3-0.20240916144458-20a13a1f6b7c/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.63.0 h1:YR/EIY1o3mEFP/kZCD7iDMnLPlGyuU2Gb3HIcXnA98k=
github.com/prometheus/common v0.63.0/go.mod h1:VVFF/fBIoToEnWRVkYoXEkq3R3paCoxG9PXP74SnV18=
github.com/prometheus/procfs v0.16.0 h1:xh6oHhKwnOJKMYiYBDWmkHqQPyiY40sny36Cmx2bbsM=
github.com/prometheus/procfs v0.16.0/go.mod h1:8veyXUu3nGP7oaCxhX6yeaM5u4stL2FeMXnCqhDthZg=
github.com/prometheus/prom2json v1.4.2 h1:PxCTM+Whqi/eykO1MKsEL0p/zMpxp9ybpsmdFamw6po=
github.com/prometheus/prom2json v1.4.2/go.mod h1:zuvPm7u3epZSbXPWHny6G+o8ETgu6eAK3oPr6yFkRWE=
github.com/prometheus/prometheus v0.303.0 h1:wsNNsbd4EycMCphYnTmNY9JASBVbp7NWwJna857cGpA=
github.com/prometheus/prometheus v0.303.0/go.mod h1:8PMRi+Fk1WzopMDeb0/6hbNs9nV6zgySkU/zds5Lu3o=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/secure-io/sio-go v0.3.1 h1:dNvY9awjabXTYGsTF1PiCySl9Ltofk9GA3VdWlo7rRc=
github.com/secure-io/sio-go v0.3.1/go.mod h1:+xbkjDzPjwh4Axd07pRKSNriS9SCiYksWnZqdnfpQxs=
github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/iZ9pI=
github.com/shirou/gopsutil/v3 v3.24.5/go.mod h1:bsoOS1aStSs9ErQ1WWfxllSeS1K5D+U30r2NfcubMVk=
github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=
github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
github.com/tinylib/msgp v1.2.5 h1:WeQg1whrXRFiZusidTQqzETkRpGjFjcIhW6uqWH09po=
github.com/tinylib/msgp v1.2.5/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0=
github.com/tklauser/go-sysconf v0.3.15 h1:VE89k0criAymJ/Os65CSn1IXaol+1wrsFHEB8Ol49K4=
github.com/tklauser/go-sysconf v0.3.15/go.mod h1:Dmjwr6tYFIseJw7a3dRLJfsHAMXZ3nEnL/aZY+0IuI4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE=
golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.39.0 h1:ZCu7HMWDxpXpaiKdhzIfaltL9Lp31x/3fCP11bc6/fY=
golang.org/x/net v0.39.0/go.mod h1:X7NRbYVEA+ewNkCNyJ513WmMdQ3BineSwVtN2zD/d+E=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20=
golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
+3
View File
@@ -170,11 +170,14 @@ test_head_bucket_usage
# --- Batch 2 candidates: expiration with tag / size filters + noncurrent counts.
# These need the same debug/scanner lane as batch 1 plus tag/size-filter and
# NewerNoncurrentVersions coverage (see ilm-9). Queued for ilm-10 batch 2.
# tags1 sends Prefix and Tag as sibling Filter predicates instead of And;
# AWS also rejects that malformed request, so it is not a compatibility gate.
test_lifecycle_deletemarker_expiration_with_days_tag
test_lifecycle_expiration_newer_noncurrent
test_lifecycle_expiration_noncur_tags1
test_lifecycle_expiration_size_gt
test_lifecycle_expiration_size_lt
test_lifecycle_expiration_tags1
test_lifecycle_expiration_tags2
test_lifecycle_expiration_versioned_tags2
#
-1
View File
@@ -539,7 +539,6 @@ test_encryption_sse_c_unaligned_multipart_upload
test_get_nonpublicpolicy_acl_bucket_policy_status
test_get_publicpolicy_acl_bucket_policy_status
test_lifecycle_expiration_date
test_lifecycle_expiration_tags1
test_lifecycle_expiration_versioning_enabled
test_list_multipart_upload
test_non_multipart_get_part