mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-21 15:33:58 +00:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d20c40702e | |||
| faa76508dd | |||
| 95ba225098 |
@@ -22,6 +22,8 @@ Identify the tested commit and any local changes. When testing a prebuilt binary
|
||||
List relevant checks not run and the remaining risk. Use the validation tier in AGENTS.md; do not run broader checks solely to fill this section. For documentation-only changes, list the applicable documentation checks. Use N/A only when verification is not applicable.
|
||||
-->
|
||||
|
||||
**auto-testing regression coverage** <!-- For each issue-backed bug fix, give the added or extended case ID or PR link, the existing case ID/path and assertion that catches the bug, or "Skipped" with a brief reason. Omit this line for other changes, including documentation-only fixes. -->
|
||||
|
||||
## Impact
|
||||
<!--
|
||||
Describe user-facing, compatibility, API, deployment, configuration, or documentation impact. Use N/A when there is no expected impact.
|
||||
|
||||
@@ -80,6 +80,18 @@ Do not commit one-shot plans, trackers, migration ledgers, benchmark snapshots,
|
||||
|
||||
Select checks from the final task-owned diff. Scoped `AGENTS.md` files may add a concrete path-specific check, but must not replace this tiering with a generic full-workspace gate.
|
||||
|
||||
### Issue Fixes and External Regression Coverage
|
||||
|
||||
After fixing a bug tracked by an issue and running the applicable local checks, ask the developer once per fixed issue whether to cover it in [rustfs/auto-testing](https://github.com/rustfs/auto-testing). Ask before opening or updating the fix PR, or before reporting completion if there is no PR. Reuse a choice already made for that fix. Investigation alone and documentation-only changes do not trigger this prompt.
|
||||
|
||||
When presenting the three choices, describe the failing scenario and the assertion that would catch it.
|
||||
|
||||
- **Add or extend a case.** Prefer extending an existing scenario when it can exercise the regression. Record the resulting case ID or auto-testing PR link.
|
||||
- **Already covered.** Cite the existing auto-testing case ID or path and the assertion that would catch this bug.
|
||||
- **Skip.** Record the developer's one-line reason and continue with the fix.
|
||||
|
||||
Record the choice in the PR's Verification section or in the completion report for work without a PR. All three choices retain the applicable RustFS verification requirements below.
|
||||
|
||||
### Documentation and Instructions
|
||||
|
||||
For prose, comments, agent instructions, and skill metadata that cannot affect runtime/build output:
|
||||
|
||||
@@ -129,12 +129,6 @@ when restoring `/rustfs/console` for a custom OEM build.
|
||||
- `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (canonical)
|
||||
- `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (canonical)
|
||||
- `RUSTFS_SCANNER_WALKERS_PER_DISK` (canonical)
|
||||
- `RUSTFS_SCANNER_READ_ONLY_HANDOFF` (canonical, default `false`)
|
||||
|
||||
Read-only restart continuity:
|
||||
|
||||
- `RUSTFS_SCANNER_READ_ONLY_HANDOFF=true` enables reuse of unpublished scanner progress across verified read-only process restarts. Unset, false or invalid values disable both identity restoration and receipt creation. Existing receipts are still durably consumed before recovery starts, even when the feature is disabled.
|
||||
- Deploy receipt-aware readers to every possible writer before enabling this flag. Before rolling back to an older binary, disable it on every node, start each node with all its drives present, wait for readiness, and shut down cleanly. Do not start the older binary until every node completes this preparation. Direct rollback after enabling receipt writes is unsupported: older startup/recovery paths can write without durably invalidating the receipt. See [the receipt and rollback contract](../../docs/architecture/scanner-usage-publication.md#read-only-restart-continuity).
|
||||
|
||||
Scanner cycle budget controls:
|
||||
|
||||
|
||||
@@ -21,11 +21,6 @@ pub const APP_NAME: &str = "RustFS";
|
||||
/// Cargo package version for this workspace member.
|
||||
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
|
||||
|
||||
/// Opt in to scanner checkpoint continuity across verified read-only restarts.
|
||||
/// Defaults to false. Enable only after all possible writers support receipt
|
||||
/// consumption; disable and restart before rolling back to an older writer.
|
||||
pub const ENV_SCANNER_READ_ONLY_HANDOFF: &str = "RUSTFS_SCANNER_READ_ONLY_HANDOFF";
|
||||
|
||||
/// Default configuration logger level
|
||||
/// Default value: error
|
||||
/// Environment variable: RUSTFS_OBS_LOGGER_LEVEL
|
||||
|
||||
@@ -171,70 +171,6 @@ impl DataUsageBucketObservation {
|
||||
}
|
||||
|
||||
impl DataUsageInfo {
|
||||
/// Hide values that cannot be bound to the current storage-owned identity.
|
||||
///
|
||||
/// `incarnations` contains canonical non-nil identities read from storage;
|
||||
/// a missing entry means identity was unavailable. Neither legacy values
|
||||
/// without identity nor metadata read failures establish confirmed zero.
|
||||
/// Changed responses are display-only and cannot authorize quota or writes.
|
||||
/// If retained counters overflow, all values remain unknown.
|
||||
/// Returns the number of canonical values removed.
|
||||
pub fn retain_current_bucket_usage_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
|
||||
let original_count = self.buckets_usage.len();
|
||||
let original_observation_count = self.usage_snapshot_bucket_observations.len();
|
||||
let matches_current = |bucket: &String, usage: &BucketUsageInfo| {
|
||||
incarnations.get(bucket).is_some_and(|current| {
|
||||
!current.is_empty()
|
||||
&& current != "00000000-0000-0000-0000-000000000000"
|
||||
&& usage.bucket_incarnation.as_ref() == Some(current)
|
||||
})
|
||||
};
|
||||
self.buckets_usage.retain(|bucket, usage| matches_current(bucket, usage));
|
||||
self.usage_snapshot_bucket_observations.retain(|bucket, observation| {
|
||||
matches_current(bucket, &observation.usage)
|
||||
&& observation
|
||||
.sources
|
||||
.iter()
|
||||
.all(|source| incarnations.get(bucket) == Some(&source.bucket_incarnation))
|
||||
});
|
||||
if original_count == self.buckets_usage.len()
|
||||
&& original_observation_count == self.usage_snapshot_bucket_observations.len()
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
let totals = self.buckets_usage.values().try_fold([0_u64; 4], |total, usage| {
|
||||
Some([
|
||||
total[0].checked_add(usage.size)?,
|
||||
total[1].checked_add(usage.objects_count)?,
|
||||
total[2].checked_add(usage.versions_count)?,
|
||||
total[3].checked_add(usage.delete_markers_count)?,
|
||||
])
|
||||
});
|
||||
let [size, objects, versions, markers] = totals.unwrap_or_else(|| {
|
||||
self.buckets_usage.clear();
|
||||
self.usage_snapshot_bucket_observations.clear();
|
||||
[0; 4]
|
||||
});
|
||||
self.bucket_sizes = self
|
||||
.buckets_usage
|
||||
.iter()
|
||||
.map(|(bucket, usage)| (bucket.clone(), usage.size))
|
||||
.collect();
|
||||
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
|
||||
self.objects_total_size = size;
|
||||
self.objects_total_count = objects;
|
||||
self.versions_total_count = versions;
|
||||
self.delete_markers_total_count = markers;
|
||||
self.replication_info.clear();
|
||||
self.tier_stats = None;
|
||||
self.unknown_tier_stats = None;
|
||||
self.usage_snapshot_complete = false;
|
||||
self.usage_snapshot_partial = true;
|
||||
self.usage_snapshot_converged = Some(false);
|
||||
self.usage_snapshot_admin_projection = true;
|
||||
original_count.saturating_sub(self.buckets_usage.len())
|
||||
}
|
||||
|
||||
/// Overlay validated companions on an owned admin response only.
|
||||
///
|
||||
/// `incarnations` must come from current storage metadata, not the companion.
|
||||
|
||||
@@ -321,110 +321,3 @@ fn provenance_and_companions_change_candidate_bytes_and_round_trip_as_maps() {
|
||||
let restored: DataUsageInfo = rmp_serde::from_slice(&named).expect("read named map with additive fields");
|
||||
assert_eq!(restored, canonical);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_identity_filter_preserves_an_unchanged_complete_snapshot() {
|
||||
let mut candidate = snapshot();
|
||||
let original = candidate.clone();
|
||||
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&incarnations()), 0);
|
||||
assert_eq!(candidate, original);
|
||||
assert!(candidate.is_complete_bucket_usage_snapshot());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_identity_filter_hides_unbound_values_without_confirming_zero() {
|
||||
for identity in [
|
||||
None,
|
||||
Some(""),
|
||||
Some("corrupt"),
|
||||
Some("00000000-0000-0000-0000-000000000000"),
|
||||
Some("22222222-2222-4222-8222-222222222222"),
|
||||
] {
|
||||
let mut candidate = snapshot();
|
||||
candidate.usage_snapshot_bucket_observations.clear();
|
||||
candidate
|
||||
.buckets_usage
|
||||
.get_mut("bucket")
|
||||
.expect("canonical")
|
||||
.bucket_incarnation = identity.map(str::to_string);
|
||||
assert_eq!(
|
||||
candidate.retain_current_bucket_usage_for_admin(&incarnations()),
|
||||
1,
|
||||
"identity {identity:?}"
|
||||
);
|
||||
assert!(!candidate.buckets_usage.contains_key("bucket"));
|
||||
assert!(!candidate.bucket_sizes.contains_key("bucket"));
|
||||
assert_eq!(candidate.usage_snapshot_converged, Some(false));
|
||||
assert!(candidate.usage_snapshot_admin_projection);
|
||||
assert!(!candidate.is_complete_bucket_usage_snapshot());
|
||||
assert!(!candidate.is_valid_partial_snapshot());
|
||||
}
|
||||
for current in ["", "00000000-0000-0000-0000-000000000000"] {
|
||||
let mut candidate = snapshot();
|
||||
candidate
|
||||
.buckets_usage
|
||||
.get_mut("bucket")
|
||||
.expect("canonical")
|
||||
.bucket_incarnation = Some(current.to_string());
|
||||
assert_eq!(
|
||||
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("bucket".to_string(), current.to_string())])),
|
||||
1
|
||||
);
|
||||
assert!(candidate.usage_snapshot_admin_projection);
|
||||
}
|
||||
let mut candidate = snapshot();
|
||||
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&HashMap::new()), 1);
|
||||
assert!(candidate.usage_snapshot_bucket_observations.is_empty());
|
||||
assert_eq!(candidate.usage_snapshot_converged, Some(false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_identity_filter_keeps_healthy_buckets_and_recalculates_partial_totals() {
|
||||
let mut candidate = snapshot();
|
||||
candidate.buckets_usage.insert("cold".to_string(), usage(7));
|
||||
candidate.buckets_count = 2;
|
||||
candidate.bucket_sizes.insert("cold".to_string(), 7);
|
||||
candidate.calculate_totals();
|
||||
candidate.replication_info.insert("target".to_string(), Default::default());
|
||||
candidate.tier_stats = Some(Default::default());
|
||||
candidate.unknown_tier_stats = Some(Default::default());
|
||||
assert_eq!(
|
||||
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("cold".to_string(), INCARNATION.to_string())])),
|
||||
1
|
||||
);
|
||||
assert_eq!(candidate.buckets_usage.len(), 1);
|
||||
assert_eq!(candidate.buckets_usage["cold"].size, 7);
|
||||
assert_eq!(candidate.bucket_sizes, HashMap::from([("cold".to_string(), 7)]));
|
||||
assert_eq!(
|
||||
(
|
||||
candidate.buckets_count,
|
||||
candidate.objects_total_size,
|
||||
candidate.objects_total_count,
|
||||
candidate.versions_total_count
|
||||
),
|
||||
(1, 7, 7, 7)
|
||||
);
|
||||
assert!(candidate.replication_info.is_empty());
|
||||
assert!(candidate.tier_stats.is_none());
|
||||
assert!(candidate.unknown_tier_stats.is_none());
|
||||
assert!(candidate.usage_snapshot_admin_projection);
|
||||
assert_eq!(candidate.usage_snapshot_converged, Some(false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn current_identity_filter_overflow_leaves_totals_unknown() {
|
||||
let mut candidate = snapshot();
|
||||
candidate.buckets_usage.insert("large".to_string(), usage(u64::MAX));
|
||||
candidate.buckets_usage.insert("small".to_string(), usage(1));
|
||||
candidate.buckets_count = 3;
|
||||
let current = HashMap::from([
|
||||
("large".to_string(), INCARNATION.to_string()),
|
||||
("small".to_string(), INCARNATION.to_string()),
|
||||
]);
|
||||
assert_eq!(candidate.retain_current_bucket_usage_for_admin(¤t), 3);
|
||||
assert!(candidate.buckets_usage.is_empty());
|
||||
assert!(candidate.bucket_sizes.is_empty());
|
||||
assert_eq!(candidate.usage_snapshot_converged, Some(false));
|
||||
assert!(candidate.usage_snapshot_admin_projection);
|
||||
assert!(!candidate.is_complete_bucket_usage_snapshot());
|
||||
}
|
||||
|
||||
@@ -1758,129 +1758,6 @@ mod upgrade_write_readiness_tests {
|
||||
}
|
||||
}
|
||||
|
||||
async fn stop_scanner_handoff_server(env: &mut RustFSTestEnvironment) -> TestResult {
|
||||
let process = env.process.as_mut().ok_or("handoff server is not running")?;
|
||||
signal_process(process.id(), "TERM")?;
|
||||
tokio::time::timeout(Duration::from_secs(90), async {
|
||||
loop {
|
||||
if let Some(status) = process.try_wait()? {
|
||||
assert!(status.success(), "handoff shutdown failed: {status}");
|
||||
return Ok::<(), BoxError>(());
|
||||
}
|
||||
sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
})
|
||||
.await??;
|
||||
env.process.take();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn scanner_handoff_identity(env: &RustFSTestEnvironment) -> Result<String, BoxError> {
|
||||
let receipt: Value = serde_json::from_slice(&std::fs::read(Path::new(&env.temp_dir).join(".rustfs.sys/unclean-shutdown"))?)?;
|
||||
assert_eq!(receipt["schema"], 1);
|
||||
assert_eq!(receipt["owner"], env.address);
|
||||
Ok(receipt["checkpoint_instance_id"]
|
||||
.as_str()
|
||||
.ok_or("handoff receipt omitted checkpoint identity")?
|
||||
.to_owned())
|
||||
}
|
||||
|
||||
async fn assert_scanner_handoff_consumed(marker: &Path) -> TestResult {
|
||||
let marker = tokio::fs::read_to_string(marker).await?;
|
||||
// Heal may replace the startup marker with its ordinary timestamp before
|
||||
// readiness. Neither form carries the prior checkpoint identity.
|
||||
assert!(
|
||||
marker == "running" || marker.parse::<u64>().is_ok(),
|
||||
"readiness must expose an ordinary unclean marker, never a handoff receipt"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scanner_read_only_handoff_requires_explicit_opt_in() -> TestResult {
|
||||
const HANDOFF: &str = "RUSTFS_SCANNER_READ_ONLY_HANDOFF";
|
||||
assert!(std::env::var_os(HANDOFF).is_none(), "this scenario tests the unset default");
|
||||
let mut env = RustFSTestEnvironment::new().await?;
|
||||
let marker = Path::new(&env.temp_dir).join(".rustfs.sys/unclean-shutdown");
|
||||
env.start_rustfs_server_without_cleanup(vec![]).await?;
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
assert!(!marker.exists(), "the default must not create a receipt");
|
||||
|
||||
env.start_rustfs_server_without_cleanup_with_env(&[(HANDOFF, "true")]).await?;
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
let previous = scanner_handoff_identity(&env)?;
|
||||
env.start_rustfs_server_without_cleanup_with_env(&[(HANDOFF, "true")]).await?;
|
||||
assert_scanner_handoff_consumed(&marker).await?;
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
assert_eq!(scanner_handoff_identity(&env)?, previous, "opted-in read-only boots retain identity");
|
||||
|
||||
for disabled in ["invalid", "false"] {
|
||||
let before_disabled = scanner_handoff_identity(&env)?;
|
||||
env.start_rustfs_server_without_cleanup_with_env(&[(HANDOFF, disabled)])
|
||||
.await?;
|
||||
assert_scanner_handoff_consumed(&marker).await?;
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
assert!(!marker.exists(), "disabled shutdown must not recreate the receipt");
|
||||
env.start_rustfs_server_without_cleanup_with_env(&[(HANDOFF, "true")]).await?;
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
assert_ne!(scanner_handoff_identity(&env)?, before_disabled, "opt-out breaks the prior lineage");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// Exercise the receipt protocol in the existing pinned-release rollback lane.
|
||||
async fn assert_scanner_handoff_prepares_legacy_rollback(previous_binary: &Path) -> TestResult {
|
||||
const HANDOFF: &str = "RUSTFS_SCANNER_READ_ONLY_HANDOFF";
|
||||
let mut env = RustFSTestEnvironment::new().await?;
|
||||
let marker = Path::new(&env.temp_dir).join(".rustfs.sys/unclean-shutdown");
|
||||
env.start_rustfs_server_without_cleanup_with_env(&[(HANDOFF, "true")]).await?;
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
let previous = scanner_handoff_identity(&env)?;
|
||||
|
||||
// Complete the current reader's rollback preparation before any old writer
|
||||
// opens storage, including an old writer with all background services off.
|
||||
env.start_rustfs_server_without_cleanup_with_env(&[(HANDOFF, "false")])
|
||||
.await?;
|
||||
assert_scanner_handoff_consumed(&marker).await?;
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
assert!(!marker.exists());
|
||||
env.start_rustfs_server_from_binary_without_cleanup(
|
||||
previous_binary,
|
||||
vec![],
|
||||
&[
|
||||
("RUSTFS_SCANNER_ENABLED", "false"),
|
||||
("RUSTFS_HEAL_ENABLED", "false"),
|
||||
("RUSTFS_HEAL_AUTO_HEAL_ENABLE", "false"),
|
||||
("RUSTFS_HEAL_MRF_ENABLE", "false"),
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
let client = env.create_s3_client();
|
||||
env.create_test_bucket("handoff-rollback").await?;
|
||||
client
|
||||
.put_object()
|
||||
.bucket("handoff-rollback")
|
||||
.key("legacy-write")
|
||||
.body(ByteStream::from_static(b"legacy writer survived rollback preparation"))
|
||||
.send()
|
||||
.await?;
|
||||
env.stop_server();
|
||||
assert!(!marker.exists(), "the legacy crash cannot resurrect a consumed receipt");
|
||||
|
||||
env.start_rustfs_server_without_cleanup_with_env(&[(HANDOFF, "true")]).await?;
|
||||
assert_eq!(
|
||||
read_object(&client, "handoff-rollback", "legacy-write", None).await?.1,
|
||||
b"legacy writer survived rollback preparation"
|
||||
);
|
||||
stop_scanner_handoff_server(&mut env).await?;
|
||||
assert_ne!(
|
||||
scanner_handoff_identity(&env)?,
|
||||
previous,
|
||||
"a legacy write must not retain the old lineage"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Pins the published old writer's limitation and the supported recovery
|
||||
/// procedure. This is not a promise that mixed-version ODM is supported.
|
||||
/// Replace the loss assertion when ODM gains independent persistence;
|
||||
@@ -3585,7 +3462,6 @@ async fn rollback_to_previous_release_reads_current_bucket_metadata() -> TestRes
|
||||
init_logging();
|
||||
let previous_binary = source_binary()?;
|
||||
let evidence_context = g09_evidence_context(&previous_binary).await?;
|
||||
assert_scanner_handoff_prepares_legacy_rollback(&previous_binary).await?;
|
||||
|
||||
let replication_target = FakeS3Target::start().await?;
|
||||
replication_target.create_bucket(ROLLBACK_REPLICA_BUCKET);
|
||||
|
||||
@@ -454,8 +454,8 @@ pub mod data_usage {
|
||||
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
|
||||
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
|
||||
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
|
||||
load_admin_data_usage_with_memory_overlay, load_bucket_usage_from_backend_cached, load_compression_total_from_memory,
|
||||
load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
|
||||
load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
|
||||
load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
|
||||
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
|
||||
record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
|
||||
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
|
||||
@@ -503,7 +503,7 @@ pub mod disk {
|
||||
PoolMetaNativeInspection, PoolMetaNativeObject, PoolMetaNativeRecoveryRequest, PoolMetaNativeRecoveryResponse,
|
||||
QUOTA_MUTATION_FENCE_METADATA_SUFFIX, RUSTFS_META_BUCKET, RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_TMP_BUCKET,
|
||||
ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, SnapshotLeaseToken,
|
||||
UNCLEAN_SHUTDOWN_MARKER_PATH, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, is_quota_mutation_fence_path, new_disk,
|
||||
UpdateMetadataOpts, VolumeInfo, WalkDirOptions, is_quota_mutation_fence_path, new_disk,
|
||||
validate_batch_read_version_item_count, validate_format_control_path,
|
||||
};
|
||||
pub use crate::disk::{
|
||||
@@ -769,8 +769,8 @@ pub mod storage {
|
||||
pub use crate::store::external_rpc_disk;
|
||||
pub use crate::store::{
|
||||
BootstrapLocalTarget, ECStore, SCANNER_PUBLICATION_LEASE_TTL_MS, ScannerDataMovementPauseStatus, all_local_disk,
|
||||
all_local_disk_path, find_local_disk_by_ref, find_local_disk_by_ref_for_rpc, find_local_disk_by_ref_for_scanner_rpc,
|
||||
init_local_disks, init_local_disks_with_instance_ctx, init_lock_clients, prewarm_local_disk_id_map,
|
||||
all_local_disk_path, find_local_disk_by_ref, find_local_disk_by_ref_for_rpc, init_local_disks,
|
||||
init_local_disks_with_instance_ctx, init_lock_clients, prewarm_local_disk_id_map,
|
||||
prewarm_local_disk_id_map_with_instance_ctx,
|
||||
};
|
||||
pub use crate::store::{HealObjectAbsenceProof, HealObjectStorageResult};
|
||||
|
||||
@@ -5082,14 +5082,7 @@ mod tests {
|
||||
let cert = generate_simple_self_signed(vec!["localhost".to_string()]).expect("certificate should generate");
|
||||
let (port, server) = spawn_https_server(&cert, 2);
|
||||
let endpoint = format!("https://localhost:{port}");
|
||||
let strict_client = s3_client_for_test(port, None);
|
||||
// A failed trust check must consume just one of the two fixture connections.
|
||||
let strict_config = strict_client
|
||||
.config()
|
||||
.to_builder()
|
||||
.retry_config(aws_sdk_s3::config::retry::RetryConfig::disabled())
|
||||
.build();
|
||||
let strict = target_client_for_test("arn:strict", endpoint.clone(), S3Client::from_conf(strict_config));
|
||||
let strict = target_client_for_test("arn:strict", endpoint.clone(), s3_client_for_test(port, None));
|
||||
let insecure = target_client_for_test(
|
||||
"arn:insecure",
|
||||
endpoint,
|
||||
|
||||
@@ -21,7 +21,7 @@ use crate::cluster::rpc::{set_tonic_canonical_body_digest, set_tonic_mutation_bo
|
||||
use crate::error::{Error, Result};
|
||||
use crate::storage_api_contracts::internode::{
|
||||
SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION, SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION, SCANNER_ACTIVITY_PROTOCOL_VERSION,
|
||||
SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_ACTIVITY_V7_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES,
|
||||
SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
|
||||
};
|
||||
use crate::{
|
||||
@@ -192,7 +192,6 @@ fn validate_signal_service_protocol(sig: u64, sub_sys: &str, protocol_version: u
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct ScannerPeerActivity {
|
||||
pub instance_id: String,
|
||||
pub checkpoint_instance_id: Option<String>,
|
||||
pub namespace_generation: u64,
|
||||
pub maintenance_generation: u64,
|
||||
pub protocol_version: u32,
|
||||
@@ -355,9 +354,6 @@ fn decode_scanner_activity_with_verifier(
|
||||
challenge: &[u8; 16],
|
||||
verify_proof: impl FnOnce(&[u8], &[u8]) -> Result<()>,
|
||||
) -> Result<ScannerPeerActivity> {
|
||||
if response.protocol_version != SCANNER_ACTIVITY_PROTOCOL_VERSION && response.checkpoint_instance_id.is_some() {
|
||||
return Err(Error::other("legacy scanner activity peer returned unauthenticated checkpoint identity"));
|
||||
}
|
||||
let instance_id = &response.instance_id;
|
||||
if !scanner_instance_id_is_valid(instance_id) {
|
||||
return Err(Error::other("peer returned an invalid scanner activity instance ID"));
|
||||
@@ -427,7 +423,7 @@ fn decode_scanner_activity_with_verifier(
|
||||
None,
|
||||
)
|
||||
}
|
||||
SCANNER_ACTIVITY_V7_PROTOCOL_VERSION | SCANNER_ACTIVITY_PROTOCOL_VERSION => {
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION => {
|
||||
if response.dirty_usage_pending && response.dirty_usage_generation == 0 {
|
||||
return Err(Error::other("scanner activity peer returned pending dirty usage without a generation"));
|
||||
}
|
||||
@@ -440,20 +436,8 @@ fn decode_scanner_activity_with_verifier(
|
||||
if movement_generation == u64::MAX {
|
||||
return Err(Error::other("scanner activity peer exhausted its movement generation"));
|
||||
}
|
||||
let canonical = if response.protocol_version == SCANNER_ACTIVITY_PROTOCOL_VERSION {
|
||||
let instance = response
|
||||
.checkpoint_instance_id
|
||||
.as_deref()
|
||||
.filter(|instance| scanner_instance_id_is_valid(instance))
|
||||
.ok_or_else(|| Error::other("scanner activity peer omitted its checkpoint identity"))?;
|
||||
if instance.bytes().all(|byte| byte == b'0') {
|
||||
return Err(Error::other("scanner activity peer returned an empty checkpoint identity"));
|
||||
}
|
||||
rustfs_protos::canonical_scanner_activity_v8_response_body(challenge, &response)
|
||||
} else {
|
||||
rustfs_protos::canonical_scanner_activity_v7_response_body(challenge, &response)
|
||||
}
|
||||
.map_err(|_| Error::other("scanner activity peer response is too large to authenticate"))?;
|
||||
let canonical = rustfs_protos::canonical_scanner_activity_v7_response_body(challenge, &response)
|
||||
.map_err(|_| Error::other("scanner activity peer response is too large to authenticate"))?;
|
||||
verify_proof(&canonical, &response.response_proof)?;
|
||||
(
|
||||
Some(
|
||||
@@ -474,7 +458,6 @@ fn decode_scanner_activity_with_verifier(
|
||||
};
|
||||
Ok(ScannerPeerActivity {
|
||||
instance_id: response.instance_id,
|
||||
checkpoint_instance_id: response.checkpoint_instance_id,
|
||||
namespace_generation: response.namespace_generation,
|
||||
maintenance_generation: response.maintenance_generation,
|
||||
protocol_version: response.protocol_version,
|
||||
@@ -2469,15 +2452,11 @@ impl PeerRestClient {
|
||||
.scanner_activity_request_with_protocol(String::new(), 0, SCANNER_ACTIVITY_PROTOCOL_VERSION)
|
||||
.await;
|
||||
if result.as_ref().err().is_some_and(scanner_activity_protocol_unsupported) {
|
||||
let previous = self
|
||||
.scanner_activity_request_with_protocol(String::new(), 0, SCANNER_ACTIVITY_V7_PROTOCOL_VERSION)
|
||||
.await;
|
||||
if previous.as_ref().err().is_some_and(scanner_activity_protocol_unsupported) {
|
||||
self.scanner_activity_request_with_protocol(String::new(), 0, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION)
|
||||
.await
|
||||
} else {
|
||||
previous
|
||||
}
|
||||
// A v6 peer cannot parse the v7 marker. Its authenticated
|
||||
// response is still decoded as untrusted terminal state, so the
|
||||
// scanner will defer publication until every peer is v7.
|
||||
self.scanner_activity_request_with_protocol(String::new(), 0, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION)
|
||||
.await
|
||||
} else {
|
||||
result
|
||||
}
|
||||
@@ -2744,15 +2723,8 @@ impl PeerRestClient {
|
||||
.scanner_activity_request_with_protocol(instance_id.clone(), generation, SCANNER_ACTIVITY_PROTOCOL_VERSION)
|
||||
.await;
|
||||
if result.as_ref().err().is_some_and(scanner_activity_protocol_unsupported) {
|
||||
let previous = self
|
||||
.scanner_activity_request_with_protocol(instance_id.clone(), generation, SCANNER_ACTIVITY_V7_PROTOCOL_VERSION)
|
||||
.await;
|
||||
if previous.as_ref().err().is_some_and(scanner_activity_protocol_unsupported) {
|
||||
self.scanner_activity_request_with_protocol(instance_id, generation, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION)
|
||||
.await
|
||||
} else {
|
||||
previous
|
||||
}
|
||||
self.scanner_activity_request_with_protocol(instance_id, generation, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION)
|
||||
.await
|
||||
} else {
|
||||
result
|
||||
}
|
||||
@@ -3708,7 +3680,6 @@ mod tests {
|
||||
#[test]
|
||||
fn scanner_scoped_dirty_usage_ack_reconciliation_requires_same_clean_instance() {
|
||||
let activity = |instance_id: &str, generation, pending| ScannerPeerActivity {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: instance_id.to_string(),
|
||||
namespace_generation: 1,
|
||||
maintenance_generation: 1,
|
||||
@@ -3990,44 +3961,9 @@ mod tests {
|
||||
.await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scanner_checkpoint_identity_requires_v8_proof_and_preserves_rpc_epoch() {
|
||||
let response = ScannerActivityResponse {
|
||||
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
|
||||
instance_id: "1".repeat(32),
|
||||
checkpoint_instance_id: Some("2".repeat(32)),
|
||||
topology_digest: vec![7; 32].into(),
|
||||
movement_generation: Some(0),
|
||||
publication_blocked: Some(false),
|
||||
response_proof: b"proof".to_vec().into(),
|
||||
..Default::default()
|
||||
};
|
||||
let decoded = decode_test_scanner_activity(response.clone()).expect("authenticated v8");
|
||||
assert_eq!(decoded.instance_id, "1".repeat(32));
|
||||
assert_eq!(decoded.checkpoint_instance_id, Some("2".repeat(32)));
|
||||
for checkpoint_instance_id in [None, Some(String::new()), Some("0".repeat(32)), Some("A".repeat(32))] {
|
||||
let malformed = ScannerActivityResponse {
|
||||
checkpoint_instance_id,
|
||||
..response.clone()
|
||||
};
|
||||
assert!(decode_test_scanner_activity(malformed).is_err(), "invalid continuity must fail closed");
|
||||
}
|
||||
let v7 = ScannerActivityResponse {
|
||||
protocol_version: SCANNER_ACTIVITY_V7_PROTOCOL_VERSION,
|
||||
..response
|
||||
};
|
||||
assert!(decode_test_scanner_activity(v7.clone()).is_err(), "legacy proof does not bind continuity");
|
||||
let v7 = ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
..v7
|
||||
};
|
||||
assert!(decode_test_scanner_activity(v7).is_ok(), "v7 without continuity remains supported");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scanner_activity_requires_restart_safe_peer_identity() {
|
||||
let legacy = decode_test_scanner_activity(ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4044,7 +3980,6 @@ mod tests {
|
||||
assert_eq!(
|
||||
legacy,
|
||||
ScannerPeerActivity {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4059,7 +3994,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let previous = decode_test_scanner_activity(ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4076,7 +4010,6 @@ mod tests {
|
||||
assert_eq!(
|
||||
previous,
|
||||
ScannerPeerActivity {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4091,7 +4024,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let v6 = decode_test_scanner_activity(ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4110,7 +4042,6 @@ mod tests {
|
||||
assert_eq!(v6.dirty_usage_generation, Some(11));
|
||||
|
||||
let malformed_topology = ScannerActivityResponse {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4131,7 +4062,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let missing_instance = ScannerActivityResponse {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: String::new(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4152,7 +4082,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let malformed_instance = ScannerActivityResponse {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: "ABCDEF0123456789ABCDEF0123456789".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4173,7 +4102,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let activity = decode_test_scanner_activity(ScannerActivityResponse {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4190,7 +4118,6 @@ mod tests {
|
||||
assert_eq!(
|
||||
activity,
|
||||
ScannerPeerActivity {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4205,7 +4132,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let missing_movement_generation = ScannerActivityResponse {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4226,7 +4152,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let pending_without_generation = ScannerActivityResponse {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4247,7 +4172,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let previous_with_dirty_usage = ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4268,7 +4192,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let legacy_with_topology = ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4289,7 +4212,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let unsupported_protocol = ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -4310,7 +4232,6 @@ mod tests {
|
||||
);
|
||||
|
||||
let missing_proof = ScannerActivityResponse {
|
||||
checkpoint_instance_id: Some("1123456789abcdef0123456789abcdef".to_owned()),
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
|
||||
@@ -1463,39 +1463,22 @@ async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataU
|
||||
let observed = load_observed_data_usage_snapshot(store.clone()).await.ok().flatten();
|
||||
let (selected, selected_is_current_format) =
|
||||
select_admin_data_usage_snapshot(authoritative, source.is_authoritative(), observed);
|
||||
Ok(normalize_loaded_data_usage(selected, selected_is_current_format).await.0)
|
||||
}
|
||||
|
||||
async fn project_admin_data_usage_for_current_buckets(
|
||||
store: Arc<ECStore>,
|
||||
mut response: DataUsageInfo,
|
||||
with_memory_overlay: bool,
|
||||
) -> DataUsageInfo {
|
||||
let mut response = normalize_loaded_data_usage(selected, selected_is_current_format).await.0;
|
||||
use futures::StreamExt as _;
|
||||
if !(response.is_complete_bucket_usage_snapshot() || response.is_valid_partial_snapshot()) {
|
||||
return DataUsageInfo::default();
|
||||
}
|
||||
|
||||
let overlay_authoritative = with_memory_overlay && !runtime_sources::setup_is_dist_erasure().await;
|
||||
let deadline = tokio::time::Instant::now() + ADMIN_USAGE_INCARNATION_TIMEOUT;
|
||||
let mut incarnations = HashMap::new();
|
||||
{
|
||||
// Identity reads happen after releasing snapshot and refresh locks,
|
||||
// including on cache hits. Metadata transaction reads linearize each
|
||||
// bucket against creation/deletion without relying on peer notices.
|
||||
// The common deadline cancels unfinished work and bounds added latency.
|
||||
let mut buckets = response
|
||||
.buckets_usage
|
||||
// Display-only identity reads must not serialize every bucket behind a
|
||||
// slow metadata lock while holding the shared admin refresh permit.
|
||||
// The common deadline bounds their total added latency; dropping this
|
||||
// stream cancels all unfinished reads without spawning detached work.
|
||||
// Own the names and store across awaits so this future remains Send
|
||||
// when composed into spawned metrics collectors.
|
||||
let buckets = response
|
||||
.usage_snapshot_bucket_observations
|
||||
.keys()
|
||||
.chain(response.usage_snapshot_bucket_observations.keys())
|
||||
.cloned()
|
||||
.collect::<HashSet<_>>();
|
||||
if overlay_authoritative {
|
||||
// Copy only names while holding the memory lock, then release it
|
||||
// before storage I/O. A later new entry has no identity proof and
|
||||
// will remain unknown in this response.
|
||||
buckets.extend(memory_cache().read().await.keys().cloned());
|
||||
}
|
||||
.collect::<Vec<_>>();
|
||||
let mut identities = futures::stream::iter(buckets)
|
||||
.map(move |bucket| {
|
||||
let store = store.clone();
|
||||
@@ -1509,6 +1492,7 @@ async fn project_admin_data_usage_for_current_buckets(
|
||||
let Ok(Some((bucket, identity))) = tokio::time::timeout_at(deadline, identities.next()).await else {
|
||||
break;
|
||||
};
|
||||
// Unavailable, timed-out or recreated buckets retain canonical values.
|
||||
if let Ok(incarnation) = identity
|
||||
&& !incarnation.is_nil()
|
||||
{
|
||||
@@ -1517,17 +1501,7 @@ async fn project_admin_data_usage_for_current_buckets(
|
||||
}
|
||||
}
|
||||
response.project_bucket_observations_for_admin(&incarnations);
|
||||
if apply_bucket_usage_memory_overlay_if_authoritative(&mut response, overlay_authoritative).await {
|
||||
response.usage_snapshot_complete = false;
|
||||
response.usage_snapshot_partial = true;
|
||||
response.usage_snapshot_converged = Some(false);
|
||||
response.usage_snapshot_admin_projection = true;
|
||||
response.replication_info.clear();
|
||||
response.tier_stats = None;
|
||||
response.unknown_tier_stats = None;
|
||||
}
|
||||
response.retain_current_bucket_usage_for_admin(&incarnations);
|
||||
response
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
fn discard_incomplete_bucket_usage(data_usage_info: &mut DataUsageInfo) {
|
||||
@@ -1716,7 +1690,11 @@ async fn project_data_usage_from_backend_cached<T>(
|
||||
}
|
||||
}
|
||||
|
||||
async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
|
||||
/// Load the freshest structurally complete snapshot for authenticated admin
|
||||
/// observability. A scan raced by namespace activity may be selected here, but
|
||||
/// never by [`load_data_usage_from_backend_cached`], which remains the
|
||||
/// converged source for quota admission.
|
||||
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
|
||||
let ttl = Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
|
||||
|
||||
loop {
|
||||
@@ -1765,38 +1743,6 @@ async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<Da
|
||||
}
|
||||
}
|
||||
|
||||
/// Load cached scanner statistics for authenticated admin observability.
|
||||
///
|
||||
/// Each response verifies bucket identities against current storage metadata,
|
||||
/// including cache hits. Missing, failed, or timed-out identity reads leave that
|
||||
/// bucket's usage absent (unknown). Legacy values without an incarnation stay
|
||||
/// unknown until a scanner snapshot certifies their identity. Per-bucket reads
|
||||
/// linearize independently; this is not an atomic namespace snapshot.
|
||||
/// Identity reads have a shared deadline and do not outlive cancellation. The
|
||||
/// unchanged cached baseline remains independent of quota and persistence.
|
||||
///
|
||||
/// # Errors
|
||||
/// Returns the cached snapshot loading error when no baseline can be loaded.
|
||||
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
|
||||
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
|
||||
Ok(project_admin_data_usage_for_current_buckets(store, response, false).await)
|
||||
}
|
||||
|
||||
/// Load the admin view with the existing authoritative single-process overlay.
|
||||
///
|
||||
/// Only a complete or valid partial scanner snapshot can receive memory values.
|
||||
/// The final identity check excludes statistics from retired or unverified buckets.
|
||||
/// Distributed nodes continue to use scanner statistics without local totals.
|
||||
/// Cancellation, unknown values, and per-bucket read semantics are the same as
|
||||
/// [`load_admin_data_usage_from_backend_cached`]. No overlay is persisted.
|
||||
///
|
||||
/// # Errors
|
||||
/// Returns the cached snapshot loading error when no baseline can be loaded.
|
||||
pub async fn load_admin_data_usage_with_memory_overlay(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
|
||||
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
|
||||
Ok(project_admin_data_usage_for_current_buckets(store, response, true).await)
|
||||
}
|
||||
|
||||
/// Invalidate the process-local persisted usage snapshot cache after a durable
|
||||
/// scanner write.
|
||||
pub async fn invalidate_data_usage_snapshot_cache() {
|
||||
@@ -2859,17 +2805,16 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
|
||||
*cache = next_cache;
|
||||
}
|
||||
|
||||
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) -> bool {
|
||||
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) {
|
||||
if !authoritative {
|
||||
return false;
|
||||
return;
|
||||
}
|
||||
|
||||
let cache = memory_cache().read().await;
|
||||
if cache.is_empty() {
|
||||
return false;
|
||||
return;
|
||||
}
|
||||
|
||||
let was_complete = data_usage_info.usage_snapshot_complete;
|
||||
let persisted_update = data_usage_info.last_update;
|
||||
let mut changed = false;
|
||||
let mut added_bucket = false;
|
||||
@@ -2906,7 +2851,6 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu
|
||||
data_usage_info.buckets_count = data_usage_info.buckets_usage.len() as u64;
|
||||
data_usage_info.calculate_totals();
|
||||
}
|
||||
changed || was_complete != data_usage_info.usage_snapshot_complete
|
||||
}
|
||||
|
||||
pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageInfo) {
|
||||
@@ -3800,23 +3744,6 @@ mod tests {
|
||||
*cache_updating().write().await = false;
|
||||
}
|
||||
|
||||
async fn bind_admin_usage_test_identities(store: &Arc<ECStore>, snapshot: &mut DataUsageInfo) {
|
||||
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
for (bucket, usage) in &mut snapshot.buckets_usage {
|
||||
store
|
||||
.make_bucket(bucket, &crate::storage_api_contracts::bucket::MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create a real bucket for current admin identity validation");
|
||||
usage.bucket_incarnation = Some(
|
||||
store
|
||||
.bucket_incarnation_id_from_disk(bucket)
|
||||
.await
|
||||
.expect("read the storage-owned fixture identity")
|
||||
.to_string(),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn data_usage_info_for_test(bucket: &str, objects_count: u64, size: u64, last_update: SystemTime) -> DataUsageInfo {
|
||||
let mut info = DataUsageInfo {
|
||||
last_update: Some(last_update),
|
||||
@@ -4352,11 +4279,7 @@ mod tests {
|
||||
use crate::config::com::save_config;
|
||||
|
||||
let (_dirs, store) = isolated_store_over_temp_disks().await;
|
||||
let (mut authoritative, mut observed) = usage_snapshots_with_missing_observed_set();
|
||||
bind_admin_usage_test_identities(&store, &mut authoritative).await;
|
||||
for (bucket, usage) in &mut observed.buckets_usage {
|
||||
usage.bucket_incarnation = authoritative.buckets_usage[bucket].bucket_incarnation.clone();
|
||||
}
|
||||
let (authoritative, observed) = usage_snapshots_with_missing_observed_set();
|
||||
let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage");
|
||||
let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation");
|
||||
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
|
||||
@@ -4405,11 +4328,6 @@ mod tests {
|
||||
reconciled.scanner_epoch = Some(4);
|
||||
reconciled.scanner_cycle = Some(12);
|
||||
reconciled.usage_snapshot_converged = Some(true);
|
||||
reconciled
|
||||
.buckets_usage
|
||||
.get_mut("large-bucket")
|
||||
.expect("reconciled bucket")
|
||||
.bucket_incarnation = authoritative.buckets_usage["large-bucket"].bucket_incarnation.clone();
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_OBJ_NAME_PATH.as_str(),
|
||||
@@ -4931,12 +4849,6 @@ mod tests {
|
||||
backup.scanner_epoch = Some(4);
|
||||
backup.scanner_cycle = Some(10);
|
||||
backup.usage_snapshot_converged = Some(true);
|
||||
bind_admin_usage_test_identities(&store, &mut current).await;
|
||||
backup
|
||||
.buckets_usage
|
||||
.get_mut("existing")
|
||||
.expect("backup bucket")
|
||||
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
|
||||
for (path, info) in [
|
||||
(DATA_USAGE_OBJ_NAME_PATH.as_str(), ¤t),
|
||||
(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), &backup),
|
||||
@@ -4959,11 +4871,6 @@ mod tests {
|
||||
after_delete.scanner_epoch = current.scanner_epoch;
|
||||
after_delete.scanner_cycle = current.scanner_cycle;
|
||||
after_delete.usage_snapshot_converged = Some(true);
|
||||
after_delete
|
||||
.buckets_usage
|
||||
.get_mut("existing")
|
||||
.expect("post-delete bucket")
|
||||
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
|
||||
let mut stale_observation =
|
||||
data_usage_info_for_test("retired-incarnation", 99, 999, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
|
||||
stale_observation.scanner_epoch = Some(4);
|
||||
@@ -5025,7 +4932,6 @@ mod tests {
|
||||
authoritative.scanner_epoch = Some(7);
|
||||
authoritative.scanner_cycle = Some(10);
|
||||
authoritative.usage_snapshot_converged = Some(true);
|
||||
bind_admin_usage_test_identities(&store, &mut authoritative).await;
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_OBJ_NAME_PATH.as_str(),
|
||||
@@ -5040,10 +4946,6 @@ mod tests {
|
||||
info.scanner_cycle = Some(cycle);
|
||||
info.usage_snapshot_converged = Some(false);
|
||||
info.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
|
||||
info.buckets_usage
|
||||
.get_mut("bucket")
|
||||
.expect("observed bucket")
|
||||
.bucket_incarnation = authoritative.buckets_usage["bucket"].bucket_incarnation.clone();
|
||||
info
|
||||
};
|
||||
let first = observation(7, 11, 200, 11, 1_100);
|
||||
|
||||
@@ -221,7 +221,7 @@ async fn companion_admin_cache_isolated_from_quota_memory_and_persistence() {
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_buckets() {
|
||||
async fn blocked_companion_identity_keeps_canonical_without_delaying_healthy_buckets() {
|
||||
use crate::bucket::metadata_sys::{
|
||||
acquire_metadata_transaction_guard, init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks,
|
||||
};
|
||||
@@ -280,11 +280,8 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
|
||||
.await
|
||||
.expect("optional projection must return before the blocked metadata lock's long timeout")
|
||||
.expect("canonical admin response stays available");
|
||||
assert!(
|
||||
!response.buckets_usage.contains_key(blocked),
|
||||
"an unavailable current identity cannot bind the canonical value"
|
||||
);
|
||||
assert!(!response.bucket_sizes.contains_key(blocked));
|
||||
assert_eq!(response.buckets_usage[blocked].size, 100);
|
||||
assert_eq!(response.buckets_usage[blocked].usage_value_state, Some(BucketUsageValueState::Carried));
|
||||
assert_eq!(response.buckets_usage[healthy].size, 250);
|
||||
assert_eq!(response.buckets_usage[healthy].usage_value_state, Some(BucketUsageValueState::Observed));
|
||||
assert!(response.usage_snapshot_admin_projection);
|
||||
@@ -296,12 +293,13 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
|
||||
.is_ok()
|
||||
);
|
||||
drop(blocked_guard);
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
let recovered = load_admin_data_usage_from_backend_cached(store.clone())
|
||||
.await
|
||||
.expect("refresh after lock recovery");
|
||||
assert_eq!(
|
||||
recovered.buckets_usage[blocked].size, 250,
|
||||
"unavailable identity can recover on a cache hit"
|
||||
"unavailable identity can recover on the next refresh"
|
||||
);
|
||||
assert_eq!(recovered.buckets_usage[healthy].size, 250);
|
||||
let quota = load_data_usage_from_backend_cached(store)
|
||||
@@ -310,331 +308,3 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
|
||||
assert_eq!(quota.buckets_usage[blocked].size, 100);
|
||||
assert_eq!(quota.buckets_usage[healthy].size, 100);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn cached_canonical_admin_usage_rechecks_current_identity_without_peer_invalidation() {
|
||||
use crate::bucket::metadata::{load_bucket_incarnation, save_bucket_incarnation};
|
||||
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
|
||||
use crate::config::com::save_config;
|
||||
use crate::storage_api_contracts::bucket::MakeBucketOptions;
|
||||
|
||||
let (_dirs, store) = isolated_store_over_temp_disks().await;
|
||||
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
let bucket = "usage-recreated-cache";
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create identity fixture");
|
||||
let old_identity = store
|
||||
.bucket_incarnation_id_from_disk(bucket)
|
||||
.await
|
||||
.expect("original identity");
|
||||
let mut canonical = companion_snapshot(bucket, old_identity);
|
||||
canonical.usage_snapshot_bucket_observations.clear();
|
||||
canonical.usage_snapshot_converged = Some(true);
|
||||
canonical.usage_snapshot_pending_buckets.clear();
|
||||
canonical.usage_snapshot_pending_bucket_count = 0;
|
||||
canonical
|
||||
.buckets_usage
|
||||
.get_mut(bucket)
|
||||
.expect("canonical bucket")
|
||||
.objects_count = 107;
|
||||
canonical.calculate_totals();
|
||||
let original_bytes = serde_json::to_vec(&canonical).expect("encode canonical old bucket");
|
||||
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), original_bytes.clone())
|
||||
.await
|
||||
.expect("persist canonical root");
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
let first = load_admin_data_usage_from_backend_cached(store.clone())
|
||||
.await
|
||||
.expect("prime raw admin cache");
|
||||
assert_eq!(first.buckets_usage[bucket].objects_count, 107);
|
||||
assert!(first.is_complete_bucket_usage_snapshot());
|
||||
let cache_generation = admin_data_usage_snapshot_generation();
|
||||
let new_identity = uuid::Uuid::new_v4();
|
||||
// Simulate the storage-owned identity changing on another node while this
|
||||
// process has not received its invalidation notice. Do not clear the cache.
|
||||
save_bucket_incarnation(store.clone(), bucket, new_identity)
|
||||
.await
|
||||
.expect("publish changed disk identity");
|
||||
assert_eq!(
|
||||
load_bucket_incarnation(store.clone(), bucket)
|
||||
.await
|
||||
.expect("read new sidecar"),
|
||||
Some(new_identity)
|
||||
);
|
||||
let second = load_admin_data_usage_from_backend_cached(store.clone())
|
||||
.await
|
||||
.expect("read raw cache with fresh identity");
|
||||
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
|
||||
assert!(
|
||||
!second.buckets_usage.contains_key(bucket),
|
||||
"same name cannot retain the predecessor's 107"
|
||||
);
|
||||
assert!(!second.bucket_sizes.contains_key(bucket));
|
||||
assert_eq!(second.usage_snapshot_converged, Some(false));
|
||||
assert!(second.usage_snapshot_admin_projection);
|
||||
assert!(reject_admin_usage_projection(&second).is_err());
|
||||
assert!(certified_bucket_usage(&second, bucket, new_identity).is_none());
|
||||
assert!(parse_usage_snapshot(&serde_json::to_vec(&second).expect("encode display response")).is_err());
|
||||
let quota = load_data_usage_from_backend_cached(store.clone())
|
||||
.await
|
||||
.expect("independent quota baseline");
|
||||
assert_eq!(quota.buckets_usage[bucket].objects_count, 107);
|
||||
assert_eq!(
|
||||
read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
|
||||
.await
|
||||
.expect("unchanged persisted root"),
|
||||
original_bytes
|
||||
);
|
||||
// Rebinding a cached raw baseline is transient: recovery does not need to
|
||||
// reload or permanently blank a healthy bucket's statistics.
|
||||
save_bucket_incarnation(store.clone(), bucket, old_identity)
|
||||
.await
|
||||
.expect("restore fixture identity");
|
||||
let recovered = load_admin_data_usage_from_backend_cached(store)
|
||||
.await
|
||||
.expect("recover on another cache hit");
|
||||
assert_eq!(recovered.buckets_usage[bucket].objects_count, 107);
|
||||
assert!(recovered.is_complete_bucket_usage_snapshot());
|
||||
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn admin_memory_overlay_requires_a_measured_scanner_baseline() {
|
||||
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
|
||||
use crate::config::com::save_config;
|
||||
use crate::storage_api_contracts::bucket::MakeBucketOptions;
|
||||
|
||||
let (_dirs, store) = isolated_store_over_temp_disks().await;
|
||||
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
assert!(!runtime_sources::setup_is_dist_erasure().await);
|
||||
let bucket = "usage-unmeasured-overlay";
|
||||
store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap();
|
||||
let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.unwrap();
|
||||
let mut baseline = data_usage_info_for_test(bucket, 1, 100, SystemTime::UNIX_EPOCH);
|
||||
let usage = baseline.buckets_usage.get_mut(bucket).unwrap();
|
||||
usage.bucket_incarnation = Some(incarnation.to_string());
|
||||
let memory_usage = BucketUsageInfo {
|
||||
size: 150,
|
||||
..usage.clone()
|
||||
};
|
||||
clear_usage_memory_cache_for_test().await;
|
||||
replace_bucket_usage_memory_from_authoritative(bucket, memory_usage, SystemTime::now()).await;
|
||||
|
||||
// A missing snapshot and a mismatched bucket count both leave usage unknown.
|
||||
for (buckets_count, expected_size) in [(None, None), (Some(2), None), (Some(1), Some(150))] {
|
||||
if let Some(count) = buckets_count {
|
||||
baseline.buckets_count = count;
|
||||
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), serde_json::to_vec(&baseline).unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
let response = load_admin_data_usage_with_memory_overlay(store.clone()).await.unwrap();
|
||||
assert_eq!(
|
||||
response.buckets_usage.get(bucket).map(|usage| usage.size),
|
||||
expected_size,
|
||||
"baseline count {buckets_count:?}"
|
||||
);
|
||||
assert_eq!(response.bucket_sizes.get(bucket).copied(), expected_size);
|
||||
assert_eq!(response.objects_total_size, expected_size.unwrap_or_default());
|
||||
assert_eq!(response.usage_snapshot_admin_projection, expected_size.is_some());
|
||||
assert!(!response.is_complete_bucket_usage_snapshot());
|
||||
assert!(!response.is_valid_partial_snapshot());
|
||||
}
|
||||
clear_usage_memory_cache_for_test().await;
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn admin_identity_filter_does_not_validate_an_unmeasured_partial_snapshot() {
|
||||
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
|
||||
use crate::config::com::save_config;
|
||||
|
||||
let (_dirs, store) = isolated_store_over_temp_disks().await;
|
||||
let current = "usage-partial-current";
|
||||
let stale = "usage-partial-stale";
|
||||
let mut baseline = data_usage_info_for_test(current, 5, 100, SystemTime::UNIX_EPOCH);
|
||||
baseline.buckets_usage.insert(
|
||||
stale.to_owned(),
|
||||
BucketUsageInfo {
|
||||
size: 200,
|
||||
objects_count: 10,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
baseline.bucket_sizes.insert(stale.to_owned(), 200);
|
||||
baseline.buckets_count = 2;
|
||||
baseline.calculate_totals();
|
||||
baseline.usage_snapshot_complete = false;
|
||||
baseline.usage_snapshot_partial = true;
|
||||
baseline.usage_snapshot_converged = Some(false);
|
||||
baseline.scanner_epoch = Some(2);
|
||||
baseline.scanner_cycle = Some(9);
|
||||
baseline.usage_snapshot_set_states = vec![DataUsageSnapshotSetState {
|
||||
scanner_epoch: baseline.scanner_epoch,
|
||||
scanner_cycle: baseline.scanner_cycle,
|
||||
scan_plan_digest: Some([1; 32]),
|
||||
..Default::default()
|
||||
}];
|
||||
bind_admin_usage_test_identities(&store, &mut baseline).await;
|
||||
let current_incarnation = baseline.buckets_usage[current].bucket_incarnation.clone();
|
||||
assert_ne!(current_incarnation, baseline.buckets_usage[stale].bucket_incarnation);
|
||||
baseline
|
||||
.buckets_usage
|
||||
.get_mut(stale)
|
||||
.expect("stale bucket fixture")
|
||||
.bucket_incarnation = current_incarnation;
|
||||
|
||||
for (case, snapshot, expected_size) in [
|
||||
(
|
||||
"missing set provenance",
|
||||
DataUsageInfo {
|
||||
usage_snapshot_set_states: Vec::new(),
|
||||
..baseline.clone()
|
||||
},
|
||||
None,
|
||||
),
|
||||
(
|
||||
"mismatched bucket count",
|
||||
DataUsageInfo {
|
||||
buckets_count: 3,
|
||||
..baseline.clone()
|
||||
},
|
||||
None,
|
||||
),
|
||||
("valid partial snapshot", baseline, Some(100)),
|
||||
] {
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_OBJ_NAME_PATH.as_str(),
|
||||
serde_json::to_vec(&snapshot).expect("encode partial snapshot"),
|
||||
)
|
||||
.await
|
||||
.expect("persist partial snapshot");
|
||||
invalidate_admin_data_usage_snapshot_cache().await;
|
||||
|
||||
let response = load_admin_data_usage_from_backend_cached(store.clone())
|
||||
.await
|
||||
.expect("load admin usage through the identity filter");
|
||||
assert_eq!(response.buckets_usage.get(current).map(|usage| usage.size), expected_size, "{case}");
|
||||
assert_eq!(response.bucket_sizes.get(current).copied(), expected_size, "{case}");
|
||||
assert!(!response.buckets_usage.contains_key(stale), "{case}");
|
||||
assert!(!response.bucket_sizes.contains_key(stale), "{case}");
|
||||
assert_eq!(response.objects_total_size, expected_size.unwrap_or_default(), "{case}");
|
||||
assert_eq!(response.usage_snapshot_admin_projection, expected_size.is_some(), "{case}");
|
||||
}
|
||||
invalidate_admin_data_usage_snapshot_cache().await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn admin_memory_overlay_cannot_reintroduce_a_predecessor_identity() {
|
||||
use crate::bucket::metadata::save_bucket_incarnation;
|
||||
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
|
||||
use crate::config::com::save_config;
|
||||
use crate::storage_api_contracts::bucket::MakeBucketOptions;
|
||||
|
||||
let (_dirs, store) = isolated_store_over_temp_disks().await;
|
||||
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
let bucket = "usage-recreated-overlay";
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create overlay identity fixture");
|
||||
let old_identity = store
|
||||
.bucket_incarnation_id_from_disk(bucket)
|
||||
.await
|
||||
.expect("original identity");
|
||||
let mut canonical = companion_snapshot(bucket, old_identity);
|
||||
canonical.usage_snapshot_bucket_observations.clear();
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_OBJ_NAME_PATH.as_str(),
|
||||
serde_json::to_vec(&canonical).expect("encode canonical root"),
|
||||
)
|
||||
.await
|
||||
.expect("persist overlay fixture");
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
let cached_usage = canonical.buckets_usage[bucket].clone();
|
||||
replace_bucket_usage_memory_from_authoritative(bucket, cached_usage, SystemTime::now()).await;
|
||||
let new_identity = uuid::Uuid::new_v4();
|
||||
save_bucket_incarnation(store.clone(), bucket, new_identity)
|
||||
.await
|
||||
.expect("publish a replacement identity");
|
||||
let response = load_admin_data_usage_with_memory_overlay(store.clone())
|
||||
.await
|
||||
.expect("project after memory overlay");
|
||||
assert!(!response.buckets_usage.contains_key(bucket));
|
||||
assert_eq!(response.usage_snapshot_converged, Some(false));
|
||||
assert!(response.usage_snapshot_admin_projection);
|
||||
clear_usage_memory_cache_for_test().await;
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn admin_overlay_flags_only_change_keeps_identity_bound_cold_usage() {
|
||||
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
|
||||
use crate::config::com::save_config;
|
||||
use crate::storage_api_contracts::bucket::MakeBucketOptions;
|
||||
|
||||
let (_dirs, store) = isolated_store_over_temp_disks().await;
|
||||
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
assert!(
|
||||
!runtime_sources::setup_is_dist_erasure().await,
|
||||
"exercise the production single-process overlay"
|
||||
);
|
||||
let cold = "usage-overlay-cold";
|
||||
let unverified = "usage-overlay-unverified";
|
||||
for bucket in [cold, unverified] {
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create overlay fixture bucket");
|
||||
}
|
||||
let identity = store
|
||||
.bucket_incarnation_id_from_disk(cold)
|
||||
.await
|
||||
.expect("cold bucket identity");
|
||||
let mut canonical = companion_snapshot(cold, identity);
|
||||
canonical.usage_snapshot_bucket_observations.clear();
|
||||
canonical.usage_snapshot_converged = Some(true);
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_OBJ_NAME_PATH.as_str(),
|
||||
serde_json::to_vec(&canonical).expect("encode cold canonical value"),
|
||||
)
|
||||
.await
|
||||
.expect("persist cold baseline");
|
||||
clear_usage_memory_cache_for_test().await;
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
memory_cache().write().await.insert(
|
||||
unverified.to_string(),
|
||||
cached_bucket_usage_from_backend(BucketUsageInfo::default(), SystemTime::UNIX_EPOCH, false),
|
||||
);
|
||||
record_bucket_object_write_memory(unverified, None, 42).await;
|
||||
{
|
||||
let cache = memory_cache().read().await;
|
||||
assert!(cache[unverified].dirty);
|
||||
assert!(!cache[unverified].authoritative);
|
||||
}
|
||||
let response = load_admin_data_usage_with_memory_overlay(store)
|
||||
.await
|
||||
.expect("identity-filtered flags-only overlay");
|
||||
assert_eq!(response.buckets_usage[cold].size, 100);
|
||||
assert_eq!(response.objects_total_size, 100);
|
||||
assert_eq!(response.bucket_sizes[cold], 100);
|
||||
assert!(!response.buckets_usage.contains_key(unverified));
|
||||
assert!(response.usage_snapshot_admin_projection);
|
||||
assert_eq!(response.usage_snapshot_converged, Some(false));
|
||||
assert!(!response.is_complete_bucket_usage_snapshot());
|
||||
assert!(!response.is_valid_partial_snapshot());
|
||||
clear_usage_memory_cache_for_test().await;
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
}
|
||||
|
||||
@@ -1033,26 +1033,16 @@ mod tests {
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn server_info_reads_complete_observation_without_promoting_authority() {
|
||||
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
|
||||
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
|
||||
use crate::config::com::{read_config, save_config};
|
||||
use crate::data_usage::{
|
||||
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
|
||||
load_data_usage_from_backend_cached,
|
||||
};
|
||||
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
|
||||
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
|
||||
use std::time::{Duration, SystemTime};
|
||||
|
||||
let (_dirs, store) = isolated_store_over_temp_disks().await;
|
||||
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
store
|
||||
.make_bucket("bucket", &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create a real observed bucket");
|
||||
let incarnation = store
|
||||
.bucket_incarnation_id_from_disk("bucket")
|
||||
.await
|
||||
.expect("read current storage-owned incarnation");
|
||||
let primary = DataUsageInfo {
|
||||
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
|
||||
scanner_epoch: Some(1),
|
||||
@@ -1076,7 +1066,6 @@ mod tests {
|
||||
versions_count: 4,
|
||||
delete_markers_count: 1,
|
||||
size: 6,
|
||||
bucket_incarnation: Some(incarnation.to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)]
|
||||
@@ -1128,46 +1117,6 @@ mod tests {
|
||||
invalidate_data_usage_snapshot_cache().await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identity_filtered_usage_remains_unknown_in_server_info() {
|
||||
let mut info = rustfs_data_usage::DataUsageInfo {
|
||||
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
|
||||
buckets_count: 1,
|
||||
objects_total_count: 107,
|
||||
objects_total_size: 1024,
|
||||
buckets_usage: [(
|
||||
"bucket".to_string(),
|
||||
rustfs_data_usage::BucketUsageInfo {
|
||||
objects_count: 107,
|
||||
size: 1024,
|
||||
bucket_incarnation: Some("11111111-1111-4111-8111-111111111111".to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)]
|
||||
.into_iter()
|
||||
.collect(),
|
||||
usage_snapshot_complete: true,
|
||||
..Default::default()
|
||||
};
|
||||
let current =
|
||||
std::collections::HashMap::from([("bucket".to_string(), "22222222-2222-4222-8222-222222222222".to_string())]);
|
||||
assert_eq!(info.retain_current_bucket_usage_for_admin(¤t), 1);
|
||||
assert!(info.buckets_usage.is_empty());
|
||||
assert!(info.usage_snapshot_admin_projection);
|
||||
assert_eq!(info.usage_snapshot_converged, Some(false));
|
||||
let mut buckets = rustfs_madmin::Buckets::default();
|
||||
let mut objects = rustfs_madmin::Objects::default();
|
||||
let mut versions = rustfs_madmin::Versions::default();
|
||||
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
|
||||
let mut usage = rustfs_madmin::Usage::default();
|
||||
apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
|
||||
assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
|
||||
assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
|
||||
assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
|
||||
assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
|
||||
assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn incomplete_data_usage_is_unavailable_in_server_info() {
|
||||
let mut buckets = rustfs_madmin::Buckets::default();
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
use crate::disk::cleanup_runtime;
|
||||
use crate::disk::io_admission::{
|
||||
AdmissionReader, BoxedChunkReader, DiskIoAdmission, DiskIoClass, DiskIoContext, current_disk_io_context,
|
||||
current_resource_unit, with_detached_disk_io_context, with_disk_io_context, with_disk_io_permit, with_optional_resource_unit,
|
||||
current_resource_unit, with_disk_io_context, with_disk_io_permit, with_optional_resource_unit,
|
||||
};
|
||||
use crate::disk::{
|
||||
CheckPartsResp, DataDirDeleteStatus, DeleteOptions, DiskAPI, DiskError, DiskInfo, DiskInfoOptions, DiskLocation, Endpoint,
|
||||
@@ -479,13 +479,10 @@ where
|
||||
if external_guard.is_none() {
|
||||
return operation().await;
|
||||
}
|
||||
tokio::spawn(with_optional_resource_unit(
|
||||
current_resource_unit(),
|
||||
with_detached_disk_io_context(async move {
|
||||
let _external_guard = external_guard;
|
||||
operation().await
|
||||
}),
|
||||
))
|
||||
tokio::spawn(with_optional_resource_unit(current_resource_unit(), async move {
|
||||
let _external_guard = external_guard;
|
||||
operation().await
|
||||
}))
|
||||
.await
|
||||
.map_err(|_| Error::other("owned mutation task failed"))?
|
||||
}
|
||||
@@ -1831,7 +1828,6 @@ impl LocalDiskWrapper {
|
||||
current_disk_io_context().unwrap_or_else(|| DiskIoContext {
|
||||
class: DiskIoClass::Foreground,
|
||||
cancellation: CancellationToken::new(),
|
||||
admission_deadline: None,
|
||||
admission: None,
|
||||
permit: None,
|
||||
resource_unit: current_resource_unit(),
|
||||
@@ -1874,10 +1870,10 @@ impl LocalDiskWrapper {
|
||||
|
||||
async fn acquire_disk_io_permit(&self) -> Result<Option<crate::disk::io_admission::DiskIoPermit>> {
|
||||
if let Some(context) = current_disk_io_context() {
|
||||
if let Some(permit) = context.permit.as_ref()
|
||||
if let Some(permit) = context.permit
|
||||
&& permit.is_for(&self.io_admission)
|
||||
{
|
||||
return Ok(Some(permit.clone()));
|
||||
return Ok(Some(permit));
|
||||
}
|
||||
if context
|
||||
.admission
|
||||
@@ -1887,8 +1883,9 @@ impl LocalDiskWrapper {
|
||||
return Ok(None);
|
||||
}
|
||||
if context.class == DiskIoClass::Background {
|
||||
return context
|
||||
.background_permit(&self.io_admission)
|
||||
return self
|
||||
.io_admission
|
||||
.background(&context.cancellation)
|
||||
.await
|
||||
.map(Some)
|
||||
.map_err(DiskError::Io);
|
||||
@@ -4412,29 +4409,6 @@ mod tests {
|
||||
assert!(matches!(wrapper.read_all(volume, &body_path).await, Err(DiskError::FileNotFound)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn owned_mutation_retains_background_class_without_waiter_cancellation() {
|
||||
let cancellation = CancellationToken::new();
|
||||
let caller = cancellation.clone();
|
||||
let guard: Arc<dyn Send + Sync> = Arc::new(());
|
||||
crate::disk::with_background_disk_io(
|
||||
cancellation,
|
||||
run_owned_mutation(Some(guard), move || async move {
|
||||
caller.cancel();
|
||||
let context = current_disk_io_context().expect("publication owner retains background class");
|
||||
assert_eq!(context.class, DiskIoClass::Background);
|
||||
assert!(!context.cancellation.is_cancelled(), "waiter cancellation must not interrupt publication");
|
||||
assert!(
|
||||
context.admission_deadline.is_some(),
|
||||
"queued publication must retain a bounded admission lifetime"
|
||||
);
|
||||
Ok(())
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.expect("owned publication completes");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn owned_mutation_keeps_publication_guard_after_waiter_cancellation() {
|
||||
use rustfs_concurrency::adaptive::AdaptiveAdmission;
|
||||
|
||||
@@ -161,24 +161,11 @@ pub(crate) enum DiskIoClass {
|
||||
pub(crate) struct DiskIoContext {
|
||||
pub(crate) class: DiskIoClass,
|
||||
pub(crate) cancellation: CancellationToken,
|
||||
pub(crate) admission_deadline: Option<Instant>,
|
||||
pub(crate) admission: Option<Arc<DiskIoAdmission>>,
|
||||
pub(crate) permit: Option<DiskIoPermit>,
|
||||
pub(crate) resource_unit: Option<ResourceIoUnit>,
|
||||
}
|
||||
|
||||
impl DiskIoContext {
|
||||
pub(crate) async fn background_permit(&self, admission: &Arc<DiskIoAdmission>) -> io::Result<DiskIoPermit> {
|
||||
let acquire = admission.background_with_resource(&self.cancellation, self.resource_unit.clone());
|
||||
match self.admission_deadline {
|
||||
Some(deadline) => tokio::time::timeout_at(deadline, acquire)
|
||||
.await
|
||||
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "owned background disk admission timed out"))?,
|
||||
None => acquire.await,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Mark disk I/O issued by `future` as background work for local-disk admission.
|
||||
#[doc(hidden)]
|
||||
pub async fn with_background_disk_io<F: Future>(cancellation: CancellationToken, future: F) -> F::Output {
|
||||
@@ -186,7 +173,6 @@ pub async fn with_background_disk_io<F: Future>(cancellation: CancellationToken,
|
||||
DiskIoContext {
|
||||
class: DiskIoClass::Background,
|
||||
cancellation,
|
||||
admission_deadline: None,
|
||||
admission: None,
|
||||
permit: None,
|
||||
resource_unit: current_resource_unit(),
|
||||
@@ -259,7 +245,6 @@ pub(crate) async fn with_disk_io_permit<F: Future>(permit: Option<DiskIoPermit>,
|
||||
let mut context = current_disk_io_context().unwrap_or_else(|| DiskIoContext {
|
||||
class: DiskIoClass::Foreground,
|
||||
cancellation: CancellationToken::new(),
|
||||
admission_deadline: None,
|
||||
admission: None,
|
||||
permit: None,
|
||||
resource_unit: None,
|
||||
@@ -294,34 +279,6 @@ pub(crate) fn with_optional_disk_io_context<F: Future>(
|
||||
}
|
||||
}
|
||||
|
||||
/// An owned commit outlives its waiter. Retain an outer I/O class and physical
|
||||
/// ownership, but give admission its own cancellation lifetime. Leave per-disk
|
||||
/// scopes to the caller's existing ownership handoff: inheriting their class
|
||||
/// without a lease could wait behind the parent awaiting this commit.
|
||||
/// The drive timeout bounds only queued admission; an admitted mutation still
|
||||
/// drains to completion. Nested owners retain the original admission deadline.
|
||||
pub(crate) fn with_detached_disk_io_context<F: Future>(future: F) -> impl Future<Output = F::Output> {
|
||||
let context = current_disk_io_context()
|
||||
.filter(|context| context.admission.is_none() && context.permit.is_none())
|
||||
.map(|mut context| {
|
||||
context.cancellation = CancellationToken::new();
|
||||
let timeout = super::disk_store::get_max_timeout_duration();
|
||||
if context.class == DiskIoClass::Background
|
||||
&& !timeout.is_zero()
|
||||
&& let Some(deadline) = Instant::now().checked_add(timeout)
|
||||
{
|
||||
context.admission_deadline = Some(context.admission_deadline.map_or(deadline, |old| old.min(deadline)));
|
||||
}
|
||||
context
|
||||
});
|
||||
async move {
|
||||
match context {
|
||||
Some(context) => with_disk_io_context(context, future).await,
|
||||
None => future.await,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn current_disk_io_context() -> Option<DiskIoContext> {
|
||||
DISK_IO_CONTEXT
|
||||
.try_with(|context| {
|
||||
@@ -340,17 +297,17 @@ pub(crate) async fn acquire_context_disk_io_permit() -> io::Result<Option<DiskIo
|
||||
let Some(context) = current_disk_io_context() else {
|
||||
return Ok(None);
|
||||
};
|
||||
let Some(admission) = context.admission.as_ref() else {
|
||||
let Some(admission) = context.admission else {
|
||||
return Ok(None);
|
||||
};
|
||||
if let Some(permit) = context.permit.as_ref()
|
||||
&& permit.is_for(admission)
|
||||
if let Some(permit) = context.permit
|
||||
&& permit.is_for(&admission)
|
||||
{
|
||||
return Ok(Some(permit.clone()));
|
||||
return Ok(Some(permit));
|
||||
}
|
||||
match context.class {
|
||||
DiskIoClass::Foreground => Ok(Some(admission.foreground())),
|
||||
DiskIoClass::Background => Ok(Some(context.background_permit(admission).await?)),
|
||||
DiskIoClass::Background => Ok(Some(admission.background(&context.cancellation).await?)),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -424,7 +381,6 @@ impl DiskIoAdmission {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn background(self: &Arc<Self>, cancellation: &CancellationToken) -> std::io::Result<DiskIoPermit> {
|
||||
self.background_with_resource(cancellation, current_resource_unit()).await
|
||||
}
|
||||
@@ -707,8 +663,11 @@ impl<R> AdmissionReader<R> {
|
||||
let mut wait = self.wait.lock().unwrap_or_else(|error| error.into_inner());
|
||||
if wait.is_none() {
|
||||
let admission = self.admission.clone();
|
||||
let context = self.context.clone();
|
||||
*wait = Some(Box::pin(async move { context.background_permit(&admission).await }));
|
||||
let cancellation = self.context.cancellation.clone();
|
||||
let resource_unit = self.context.resource_unit.clone();
|
||||
*wait = Some(Box::pin(
|
||||
async move { admission.background_with_resource(&cancellation, resource_unit).await },
|
||||
));
|
||||
}
|
||||
let Some(wait_future) = wait.as_mut() else {
|
||||
return Poll::Ready(Err(io::Error::other("background disk I/O admission waiter is missing")));
|
||||
@@ -950,7 +909,6 @@ mod tests {
|
||||
let existing = DiskIoContext {
|
||||
class: DiskIoClass::Foreground,
|
||||
cancellation: cancellation.clone(),
|
||||
admission_deadline: None,
|
||||
admission: None,
|
||||
permit: None,
|
||||
resource_unit: None,
|
||||
@@ -996,94 +954,6 @@ mod tests {
|
||||
assert_eq!(gate.snapshot().active, 0);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn detached_commit_keeps_background_admission_after_waiter_cancellation() {
|
||||
let admission = DiskIoAdmission::new("detached-commit".to_string());
|
||||
let foreground = admission.foreground();
|
||||
let cancellation = CancellationToken::new();
|
||||
let worker_admission = admission.clone();
|
||||
with_background_disk_io(cancellation.clone(), async {
|
||||
let worker = tokio::spawn(with_detached_disk_io_context(async move {
|
||||
let context = current_disk_io_context().expect("owned commit retains its class");
|
||||
assert_eq!(context.class, DiskIoClass::Background);
|
||||
context.background_permit(&worker_admission).await
|
||||
}));
|
||||
wait_for_background_waiters(&admission, 1).await;
|
||||
cancellation.cancel();
|
||||
tokio::task::yield_now().await;
|
||||
assert!(!worker.is_finished(), "caller cancellation must not abandon a queued commit");
|
||||
drop(foreground);
|
||||
let permit = tokio::time::timeout(Duration::from_secs(1), worker)
|
||||
.await
|
||||
.expect("commit admission should finish after foreground work")
|
||||
.expect("owned worker joins")
|
||||
.expect("owned commit survives waiter cancellation");
|
||||
assert_eq!(admission.background_admissions_for_tests(), 1);
|
||||
drop(permit);
|
||||
})
|
||||
.await;
|
||||
assert!(!admission.background_active_for_tests());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn detached_commit_does_not_inherit_a_parent_disk_lease() {
|
||||
let admission = DiskIoAdmission::new("parent-lease".to_string());
|
||||
let cancellation = CancellationToken::new();
|
||||
let permit = admission.background(&cancellation).await.expect("parent disk admission");
|
||||
with_background_disk_io(
|
||||
cancellation,
|
||||
with_disk_io_permit(Some(permit), async {
|
||||
tokio::spawn(with_detached_disk_io_context(async {
|
||||
assert!(current_disk_io_context().is_none(), "do not queue behind the parent's disk lease");
|
||||
}))
|
||||
.await
|
||||
.expect("resource-only commit joins while parent keeps its disk lease");
|
||||
assert!(admission.background_active_for_tests());
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
assert!(!admission.background_active_for_tests());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn detached_commit_bounds_queued_admission_without_extending_an_existing_deadline() {
|
||||
let admission = DiskIoAdmission::new("stalled-background-disk");
|
||||
let held = admission
|
||||
.background(&CancellationToken::new())
|
||||
.await
|
||||
.expect("existing native I/O");
|
||||
let deadline = Instant::now() + Duration::from_secs(2);
|
||||
let worker_admission = admission.clone();
|
||||
let error = with_background_disk_io(CancellationToken::new(), async {
|
||||
let mut context = current_disk_io_context().expect("background scope");
|
||||
context.admission_deadline = Some(deadline);
|
||||
with_disk_io_context(context, async {
|
||||
let worker = tokio::spawn(with_detached_disk_io_context(async move {
|
||||
let context = current_disk_io_context().expect("owned scope");
|
||||
assert_eq!(
|
||||
context.admission_deadline,
|
||||
Some(deadline),
|
||||
"nested ownership must not reset admission time"
|
||||
);
|
||||
context.background_permit(&worker_admission).await
|
||||
}));
|
||||
wait_for_background_waiters(&admission, 1).await;
|
||||
tokio::time::advance(Duration::from_secs(2)).await;
|
||||
worker
|
||||
.await
|
||||
.expect("owned admission worker")
|
||||
.expect_err("stalled disk must not retain queued commit")
|
||||
})
|
||||
.await
|
||||
})
|
||||
.await;
|
||||
assert_eq!(error.kind(), io::ErrorKind::TimedOut);
|
||||
assert_eq!(admission.background_waiters.load(Ordering::Acquire), 0);
|
||||
assert!(admission.background_active_for_tests(), "timeout must not interrupt admitted native I/O");
|
||||
drop(held);
|
||||
assert!(!admission.background_active_for_tests());
|
||||
}
|
||||
|
||||
struct BlockingWorker(tokio::task::JoinHandle<()>);
|
||||
|
||||
async fn wait_for_background_waiters(admission: &DiskIoAdmission, expected: usize) {
|
||||
@@ -1202,7 +1072,6 @@ mod tests {
|
||||
DiskIoContext {
|
||||
class: DiskIoClass::Foreground,
|
||||
cancellation: CancellationToken::new(),
|
||||
admission_deadline: None,
|
||||
admission: None,
|
||||
permit: None,
|
||||
resource_unit: None,
|
||||
|
||||
@@ -79,21 +79,12 @@ impl BackgroundMemoryBudget {
|
||||
pub(crate) fn reserve(self: &Arc<Self>, bytes: usize) -> io::Result<Reservation> {
|
||||
let bytes = u64::try_from(bytes).map_err(|_| io::Error::from(io::ErrorKind::OutOfMemory))?;
|
||||
let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let expired = Instant::now() >= state.valid_until;
|
||||
let exhausted = || {
|
||||
io::Error::new(
|
||||
io::ErrorKind::WouldBlock,
|
||||
BackgroundMemoryExhausted {
|
||||
requested: bytes,
|
||||
used: state.used,
|
||||
limit: state.limit,
|
||||
expired,
|
||||
},
|
||||
)
|
||||
};
|
||||
let used = state.used.checked_add(bytes).ok_or_else(exhausted)?;
|
||||
if bytes != 0 && (expired || used > state.limit) {
|
||||
return Err(exhausted());
|
||||
let used = state
|
||||
.used
|
||||
.checked_add(bytes)
|
||||
.ok_or_else(|| io::Error::new(io::ErrorKind::WouldBlock, BackgroundMemoryExhausted))?;
|
||||
if bytes != 0 && (Instant::now() >= state.valid_until || used > state.limit) {
|
||||
return Err(io::Error::new(io::ErrorKind::WouldBlock, BackgroundMemoryExhausted));
|
||||
}
|
||||
state.used = used;
|
||||
Ok(Reservation {
|
||||
@@ -104,20 +95,11 @@ impl BackgroundMemoryBudget {
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct BackgroundMemoryExhausted {
|
||||
requested: u64,
|
||||
used: u64,
|
||||
limit: u64,
|
||||
expired: bool,
|
||||
}
|
||||
struct BackgroundMemoryExhausted;
|
||||
|
||||
impl std::fmt::Display for BackgroundMemoryExhausted {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
write!(
|
||||
formatter,
|
||||
"background buffer memory budget exhausted (requested={}, used={}, limit={}, expired={})",
|
||||
self.requested, self.used, self.limit, self.expired,
|
||||
)
|
||||
formatter.write_str("background buffer memory budget exhausted")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -339,31 +321,6 @@ impl io::Write for BackgroundEncodeBuffer {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn rejected_reservation_reports_capacity_and_expiration_without_charging() {
|
||||
let owner = BackgroundMemoryBudget::new();
|
||||
owner.set_sampled_limit(64, Duration::from_secs(3));
|
||||
let held = owner.reserve(32).expect("initial reservation fits");
|
||||
let error = owner.reserve(33).err().expect("one byte over remaining capacity");
|
||||
assert!(is_memory_pressure(&error));
|
||||
assert_eq!(
|
||||
error.to_string(),
|
||||
"background buffer memory budget exhausted (requested=33, used=32, limit=64, expired=false)"
|
||||
);
|
||||
assert_eq!(owner.snapshot().used, 32);
|
||||
|
||||
tokio::time::advance(Duration::from_secs(3)).await;
|
||||
let error = owner.reserve(1).err().expect("expired lease rejects a fitting request");
|
||||
assert!(is_memory_pressure(&error));
|
||||
assert_eq!(
|
||||
error.to_string(),
|
||||
"background buffer memory budget exhausted (requested=1, used=32, limit=64, expired=true)"
|
||||
);
|
||||
assert_eq!(owner.snapshot().used, 32);
|
||||
drop(held);
|
||||
assert_eq!(owner.snapshot().used, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn encoded_buffer_growth_retains_both_allocations_and_the_last_slice() {
|
||||
use std::io::Write;
|
||||
|
||||
@@ -140,28 +140,6 @@ pub(crate) fn admit(binding: &PhysicalDiskBinding) -> io::Result<()> {
|
||||
.unwrap_or(Ok(()))
|
||||
}
|
||||
|
||||
/// Wait only at the namespace RPC entry, before storage locks or native I/O.
|
||||
/// The caller owns the deadline; dropping this future releases its queue turn.
|
||||
pub(crate) async fn admit_scanner_preflight(binding: &PhysicalDiskBinding) -> io::Result<()> {
|
||||
let node = REMOTE_RESOURCES
|
||||
.try_with(|context| context.admitted.borrow().is_none().then(|| context.node.clone()))
|
||||
.ok()
|
||||
.flatten();
|
||||
let Some(node) = node else {
|
||||
return admit(binding);
|
||||
};
|
||||
let deferred = || io::Error::from(io::ErrorKind::WouldBlock);
|
||||
let plan = binding.plan().ok_or_else(deferred)?;
|
||||
let permit = plan.acquire(&node).await.ok_or_else(deferred)?;
|
||||
REMOTE_RESOURCES.with(|context| {
|
||||
let mut admitted = context.admitted.borrow_mut();
|
||||
if admitted.is_none() {
|
||||
*admitted = Some(permit);
|
||||
}
|
||||
});
|
||||
admit(binding)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -184,76 +162,6 @@ mod tests {
|
||||
node
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn scanner_preflight_cancellation_and_timeout_release_queued_capacity() {
|
||||
let node = node();
|
||||
let gate = AdaptiveAdmission::new(NonZeroU16::MIN, 1);
|
||||
let binding = PhysicalDiskBinding::default();
|
||||
binding.set_test_capacity(vec![gate.clone()]).expect("device plan");
|
||||
for deadline in [false, true] {
|
||||
let held = gate.try_acquire().expect("concurrent disk operation");
|
||||
with_remote_background_io(Some(RemoteBackgroundIo::new(node.clone())), async {
|
||||
let mut wait = Box::pin(admit_scanner_preflight(&binding));
|
||||
assert!(futures_util::poll!(&mut wait).is_pending());
|
||||
assert_eq!(node.snapshot().active, 0, "queued disk work cannot reserve node capacity");
|
||||
if deadline {
|
||||
assert!(tokio::time::timeout(std::time::Duration::from_secs(1), wait).await.is_err());
|
||||
} else {
|
||||
drop(wait);
|
||||
}
|
||||
drop(held);
|
||||
admit(&binding).expect("cancelled queue turn cannot block the immediate path");
|
||||
})
|
||||
.await;
|
||||
assert_eq!([node.snapshot().active, gate.snapshot().active], [0, 0]);
|
||||
assert_eq!([node.completed(), gate.completed()], [Some(0), Some(0)]);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scanner_preflight_rejects_a_plan_replaced_while_waiting() {
|
||||
let node = node();
|
||||
let gate = AdaptiveAdmission::new(NonZeroU16::MIN, 1);
|
||||
let binding = PhysicalDiskBinding::default();
|
||||
binding.set_test_capacity(vec![gate.clone()]).expect("old plan");
|
||||
let held = gate.try_acquire().expect("busy disk");
|
||||
with_remote_background_io(Some(RemoteBackgroundIo::new(node.clone())), async {
|
||||
let wait = admit_scanner_preflight(&binding);
|
||||
tokio::pin!(wait);
|
||||
assert!(futures_util::poll!(&mut wait).is_pending());
|
||||
binding
|
||||
.set_test_capacity(vec![AdaptiveAdmission::new(NonZeroU16::MIN, 1)])
|
||||
.expect("new plan");
|
||||
drop(held);
|
||||
assert_eq!(wait.await.expect_err("old plan is invalid").kind(), io::ErrorKind::WouldBlock);
|
||||
assert!(resources().is_none());
|
||||
})
|
||||
.await;
|
||||
assert_eq!([node.snapshot().active, gate.snapshot().active], [0, 0]);
|
||||
assert_eq!(node.completed(), Some(0));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scanner_preflight_never_waits_for_additional_devices_under_existing_admission() {
|
||||
let node = node();
|
||||
let first = PhysicalDiskBinding::default();
|
||||
let gate = AdaptiveAdmission::new(NonZeroU16::MIN, 1);
|
||||
first.set_test_capacity(vec![gate.clone()]).expect("first plan");
|
||||
let other = PhysicalDiskBinding::default();
|
||||
other
|
||||
.set_test_capacity(vec![AdaptiveAdmission::new(NonZeroU16::MIN, 0)])
|
||||
.expect("closed plan");
|
||||
with_remote_background_io(Some(RemoteBackgroundIo::new(node.clone())), async {
|
||||
admit(&first).expect("existing request admission");
|
||||
let other = admit_scanner_preflight(&other);
|
||||
tokio::pin!(other);
|
||||
assert!(matches!(futures_util::poll!(other), std::task::Poll::Ready(Err(error)) if error.kind() == io::ErrorKind::WouldBlock));
|
||||
assert_eq!([node.snapshot().active, gate.snapshot().active], [1, 1]);
|
||||
})
|
||||
.await;
|
||||
assert_eq!([node.snapshot().active, gate.snapshot().active], [0, 0]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn receiver_reuses_one_receipt_across_phases_and_releases_only_native_subsets() {
|
||||
let node = node();
|
||||
|
||||
@@ -3692,17 +3692,10 @@ pub(crate) fn has_delete_version_test_failpoints() -> bool {
|
||||
#[cfg(not(test))]
|
||||
pub(crate) fn has_delete_version_test_failpoints() -> bool {
|
||||
#[cfg(all(feature = "test-util", not(windows)))]
|
||||
let prepared_hooks = os::prepared_publication_test_hooks::has_any_hooks();
|
||||
#[cfg(not(all(feature = "test-util", not(windows))))]
|
||||
let prepared_hooks = false;
|
||||
#[cfg(feature = "e2e-test-hooks")]
|
||||
{
|
||||
prepared_hooks || std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").is_some()
|
||||
}
|
||||
#[cfg(not(feature = "e2e-test-hooks"))]
|
||||
{
|
||||
prepared_hooks
|
||||
if os::prepared_publication_test_hooks::has_any_hooks() {
|
||||
return true;
|
||||
}
|
||||
cfg!(feature = "e2e-test-hooks") && std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").is_some()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -12488,13 +12481,9 @@ impl LocalDisk {
|
||||
// Files written here (format.json, ...) stay where they land — no
|
||||
// rename follows — so the new directory entry must be fsynced too.
|
||||
// System-critical volumes are pinned to strict by effective_durability;
|
||||
// A consumed shutdown receipt must never survive a later writer crash,
|
||||
// including when data-shard syncing is disabled.
|
||||
let sync = if volume == RUSTFS_META_BUCKET && path == super::UNCLEAN_SHUTDOWN_MARKER_PATH {
|
||||
SyncMode::FileAndDir
|
||||
} else {
|
||||
effective_durability(volume).metadata_sync_mode()
|
||||
};
|
||||
// only the legacy full-off switch (historical semantics) skips this.
|
||||
let durability = effective_durability(volume);
|
||||
let sync = durability.metadata_sync_mode();
|
||||
self.write_all_private(volume, path, data, sync, &volume_dir).await?;
|
||||
|
||||
Ok(())
|
||||
@@ -22563,33 +22552,6 @@ mod test {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[allow(clippy::await_holding_lock)]
|
||||
async fn shutdown_receipt_is_durable_with_legacy_sync_disabled() {
|
||||
let _mode = durability_mode_override::set(DurabilityMode::LegacyOff);
|
||||
let dir = tempfile::tempdir().expect("shutdown marker disk");
|
||||
let endpoint = Endpoint::try_from(dir.path().to_str().expect("disk path")).expect("local endpoint");
|
||||
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk");
|
||||
ensure_test_volume(&disk, RUSTFS_META_BUCKET).await;
|
||||
disk.write_all(
|
||||
RUSTFS_META_BUCKET,
|
||||
crate::disk::UNCLEAN_SHUTDOWN_MARKER_PATH,
|
||||
Bytes::from_static(b"running"),
|
||||
)
|
||||
.await
|
||||
.expect("consume shutdown receipt");
|
||||
let parent = disk
|
||||
.get_object_path(RUSTFS_META_BUCKET, crate::disk::UNCLEAN_SHUTDOWN_MARKER_PATH)
|
||||
.expect("marker path")
|
||||
.parent()
|
||||
.expect("marker parent")
|
||||
.to_path_buf();
|
||||
assert!(
|
||||
os::fsync_dir_recorder::was_fsynced(&parent),
|
||||
"a stale clean receipt must not reappear after a crash"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[allow(clippy::await_holding_lock)]
|
||||
async fn test_write_all_public_still_fsyncs_parent_dir() {
|
||||
|
||||
@@ -122,27 +122,17 @@ fn validate_certificate_shape(
|
||||
}
|
||||
|
||||
impl LocalDisk {
|
||||
// Separate inspection, abort and recovery layouts, and construct the
|
||||
// selected boxed future outside caller poll frames.
|
||||
// Give readonly inspection its own future layout while retaining one
|
||||
// selected Box per request and constructing it outside caller poll frames.
|
||||
#[inline(never)]
|
||||
pub(super) fn recover_multipart_source_native(
|
||||
&self,
|
||||
request: MultipartSourceRequest,
|
||||
) -> impl std::future::Future<Output = Result<MultipartSourceResponse>> + Send + '_ {
|
||||
match &request.operation {
|
||||
MultipartSourceOperation::Inspect { .. } => {
|
||||
futures::future::Either::Left(Box::pin(self.inspect_multipart_source_native(request)))
|
||||
}
|
||||
MultipartSourceOperation::UpgradeLegacy { .. }
|
||||
| MultipartSourceOperation::PrepareAbort { .. }
|
||||
| MultipartSourceOperation::CommitAbort { .. }
|
||||
| MultipartSourceOperation::DeleteAborted { .. }
|
||||
| MultipartSourceOperation::ConfirmCleanupTarget { .. } => futures::future::Either::Right(
|
||||
futures::future::Either::Left(Box::pin(self.abort_multipart_source_native(request))),
|
||||
),
|
||||
_ => futures::future::Either::Right(futures::future::Either::Right(Box::pin(
|
||||
self.recover_multipart_source_native_inner(request),
|
||||
))),
|
||||
if matches!(&request.operation, MultipartSourceOperation::Inspect { .. }) {
|
||||
futures::future::Either::Left(Box::pin(self.inspect_multipart_source_native(request)))
|
||||
} else {
|
||||
futures::future::Either::Right(Box::pin(self.recover_multipart_source_native_inner(request)))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -190,24 +180,20 @@ impl LocalDisk {
|
||||
}))
|
||||
}
|
||||
|
||||
async fn abort_multipart_source_native(&self, request: MultipartSourceRequest) -> Result<MultipartSourceResponse> {
|
||||
request.validate()?;
|
||||
let store = self.open_multipart_source_store(&request.path, None).await?;
|
||||
self.recover_multipart_abort_owned(store, request.operation).await
|
||||
}
|
||||
|
||||
async fn recover_multipart_source_native_inner(&self, request: MultipartSourceRequest) -> Result<MultipartSourceResponse> {
|
||||
request.validate()?;
|
||||
// Source RW ownership precedes source namespace, target metadata, and
|
||||
// target publication ownership. Every filesystem tail retains it.
|
||||
let store = self.open_multipart_source_store(&request.path, None).await?;
|
||||
match request.operation {
|
||||
MultipartSourceOperation::Inspect { .. }
|
||||
| MultipartSourceOperation::UpgradeLegacy { .. }
|
||||
operation @ (MultipartSourceOperation::UpgradeLegacy { .. }
|
||||
| MultipartSourceOperation::PrepareAbort { .. }
|
||||
| MultipartSourceOperation::CommitAbort { .. }
|
||||
| MultipartSourceOperation::DeleteAborted { .. }
|
||||
| MultipartSourceOperation::ConfirmCleanupTarget { .. } => Err(DiskError::UnsupportedDisk),
|
||||
| MultipartSourceOperation::ConfirmCleanupTarget { .. }) => {
|
||||
self.recover_multipart_abort_owned(store, operation).await
|
||||
}
|
||||
MultipartSourceOperation::Inspect { .. } => Err(DiskError::UnsupportedDisk),
|
||||
MultipartSourceOperation::Claim { expected, round } => {
|
||||
let loaded = store.load().await?;
|
||||
require_expected(&loaded, &expected)?;
|
||||
|
||||
@@ -316,18 +316,11 @@ pub(super) fn put_active_multipart_source_head(source: impl AsRef<Path>, raw: By
|
||||
}
|
||||
|
||||
fn stamp_matches_active_disk(source: &Path, expected_head_stamp: FileStamp) -> bool {
|
||||
let stamp_matches = FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp);
|
||||
#[cfg(test)]
|
||||
{
|
||||
stamp_matches
|
||||
&& !source.join(STATE).exists()
|
||||
&& !source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|
||||
&& !source.join(".mpu-completions").exists()
|
||||
}
|
||||
#[cfg(not(test))]
|
||||
{
|
||||
stamp_matches
|
||||
}
|
||||
FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp)
|
||||
&& !(cfg!(test)
|
||||
&& (source.join(STATE).exists()
|
||||
|| source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|
||||
|| source.join(".mpu-completions").exists()))
|
||||
}
|
||||
|
||||
pub(super) fn get_active_multipart_source_head(source: &Path) -> Option<(Bytes, FileInfo)> {
|
||||
@@ -607,13 +600,11 @@ impl SourceStore {
|
||||
let disk_id = self.disk_id;
|
||||
let physical_slot = self.physical_slot;
|
||||
let disk_ids = self.disk_ids.clone();
|
||||
// Keep the loaded state out of each blocking-queue and oneshot poll frame.
|
||||
os::run_blocking_namespace_operation(self.lease.clone(), move || {
|
||||
Ok(load_source(&source, &base, &root, &path, disk_id, physical_slot, &disk_ids).map(Box::new))
|
||||
Ok(load_source(&source, &base, &root, &path, disk_id, physical_slot, &disk_ids))
|
||||
})
|
||||
.await
|
||||
.map_err(DiskError::from)?
|
||||
.map(|loaded| *loaded)
|
||||
}
|
||||
pub async fn persist(&self, state: &SourceState) -> Result<()> {
|
||||
self.persist_with_source_sync(state, false).await.map(|_| ())
|
||||
|
||||
@@ -45,7 +45,7 @@ pub use local::pool_meta_native::{PoolMetaNativeCasCondition, PoolMetaNativeFile
|
||||
pub(crate) use io_admission::{DiskIoClass, test_resource_units};
|
||||
pub(crate) use io_admission::{
|
||||
PhysicalHealBinding, acquire_context_disk_io_permit, current_disk_io_context, current_resource_unit, has_resource_io_scope,
|
||||
with_detached_disk_io_context, with_optional_disk_io_context, with_optional_resource_unit,
|
||||
with_optional_disk_io_context, with_optional_resource_unit,
|
||||
};
|
||||
|
||||
pub const RUSTFS_META_BUCKET: &str = ".rustfs.sys";
|
||||
@@ -77,8 +77,6 @@ pub fn validate_format_control_path(path: &str) -> Result<()> {
|
||||
/// Per-disk marker present while an erasure-set heal is rebuilding this disk.
|
||||
/// `LocalDisk::disk_info` reports `healing = true` while the file exists.
|
||||
pub const HEALING_MARKER_PATH: &str = "healing.bin";
|
||||
/// Boot-time crash fence and clean scanner continuity receipt.
|
||||
pub const UNCLEAN_SHUTDOWN_MARKER_PATH: &str = "unclean-shutdown";
|
||||
pub const STORAGE_FORMAT_FILE: &str = "xl.meta";
|
||||
pub const STORAGE_FORMAT_FILE_BACKUP: &str = "xl.meta.bkp";
|
||||
/// Original upload authority after it stops admitting active part mutations.
|
||||
|
||||
@@ -417,15 +417,11 @@ pub(crate) mod prepared_publication_test_hooks {
|
||||
}
|
||||
|
||||
pub(crate) fn has_any_hooks() -> bool {
|
||||
let before_publication = !BEFORE_PUBLICATION.lock().is_empty();
|
||||
#[cfg(test)]
|
||||
{
|
||||
before_publication || !RENAME_DESTINATIONS.lock().is_empty() || !HARDLINK_DESTINATIONS.lock().is_empty()
|
||||
}
|
||||
#[cfg(not(test))]
|
||||
{
|
||||
before_publication
|
||||
if !RENAME_DESTINATIONS.lock().is_empty() || !HARDLINK_DESTINATIONS.lock().is_empty() {
|
||||
return true;
|
||||
}
|
||||
!BEFORE_PUBLICATION.lock().is_empty()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -1045,13 +1045,6 @@ impl InstanceContext {
|
||||
self.local_disk_map.clone()
|
||||
}
|
||||
|
||||
/// Snapshot the boot-time disk registry before a store is published.
|
||||
/// Missing local drives remain explicit so startup receipts cannot treat
|
||||
/// an available subset as the complete instance inventory.
|
||||
pub async fn local_disk_inventory(&self) -> Vec<Option<DiskStore>> {
|
||||
self.local_disk_map.read().await.values().cloned().collect()
|
||||
}
|
||||
|
||||
/// Handle to this instance's local disk-id→endpoint map.
|
||||
pub(crate) fn local_disk_id_map(&self) -> Arc<RwLock<HashMap<Uuid, String>>> {
|
||||
self.local_disk_id_map.clone()
|
||||
|
||||
@@ -5958,7 +5958,6 @@ mod tests {
|
||||
#[test]
|
||||
fn scanner_dirty_usage_acknowledgement_preserves_newer_pending_work() {
|
||||
let activity = |dirty_usage_pending| ScannerPeerActivity {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 1,
|
||||
maintenance_generation: 1,
|
||||
|
||||
@@ -4638,11 +4638,10 @@ impl SetDisks {
|
||||
write_quorum,
|
||||
};
|
||||
let resource_unit = resource_unit.clone();
|
||||
let producer = crate::disk::with_optional_resource_unit(resource_unit, async move {
|
||||
tasks.spawn(crate::disk::with_optional_resource_unit(resource_unit, async move {
|
||||
let outcome = Self::run_unary_rename_slot(context).await;
|
||||
let _ = outcome_tx.send(outcome).await;
|
||||
});
|
||||
tasks.spawn(crate::disk::with_detached_disk_io_context(producer));
|
||||
}));
|
||||
}
|
||||
for group in fanout_plan.batch_groups {
|
||||
let outcome_tx = outcome_tx.clone();
|
||||
@@ -4653,7 +4652,7 @@ impl SetDisks {
|
||||
let group_disks = disks.clone();
|
||||
let group_tokens = fence_tokens.clone();
|
||||
let resource_unit = resource_unit.clone();
|
||||
let producer = crate::disk::with_optional_resource_unit(resource_unit, async move {
|
||||
tasks.spawn(crate::disk::with_optional_resource_unit(resource_unit, async move {
|
||||
let prepared = join_all(group.slots.into_iter().map(|slot| {
|
||||
let context = RenameSlotContext {
|
||||
slot,
|
||||
@@ -4816,8 +4815,7 @@ impl SetDisks {
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
tasks.spawn(crate::disk::with_detached_disk_io_context(producer));
|
||||
}));
|
||||
}
|
||||
drop(outcome_tx);
|
||||
RenameOutcomeProducers {
|
||||
@@ -4872,7 +4870,7 @@ impl SetDisks {
|
||||
|
||||
let coordinator_failure_receipt = rollback_receipt.clone();
|
||||
let early_ack_spawn_to_quorum_wait_started = rustfs_io_metrics::put_stage_timer();
|
||||
let tail_drain = tokio::spawn(crate::disk::with_detached_disk_io_context({
|
||||
let tail_drain = tokio::spawn({
|
||||
let fanout_src_bucket = src_bucket.clone();
|
||||
let fanout_src_object = src_object.clone();
|
||||
let fanout_dst_bucket = dst_bucket.clone();
|
||||
@@ -5126,7 +5124,7 @@ impl SetDisks {
|
||||
all_online_local_succeeded,
|
||||
})
|
||||
}
|
||||
}));
|
||||
});
|
||||
|
||||
rustfs_io_metrics::record_put_object_stage_duration_from(
|
||||
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_EARLY_ACK_SPAWN_TO_QUORUM_WAIT,
|
||||
@@ -5307,7 +5305,6 @@ impl SetDisks {
|
||||
rollback_fault_injection::after_fanout(&fanout_dst_object);
|
||||
(results, fanout_file_infos)
|
||||
};
|
||||
let fanout_future = crate::disk::with_detached_disk_io_context(fanout_future);
|
||||
let fanout = if protected_multipart {
|
||||
tokio::spawn(crate::disk::with_optional_resource_unit(
|
||||
crate::disk::current_resource_unit(),
|
||||
|
||||
@@ -6601,7 +6601,7 @@ impl SetDisks {
|
||||
let diagnostic_context = crate::cluster::rpc::runtime_sources::ReadVersionDiagnosticContext::capture_entry(
|
||||
rustfs_io_metrics::internode_metrics::ReadVersionEntrySource::PutCommitOwner,
|
||||
);
|
||||
let result = tokio::spawn(crate::disk::with_detached_disk_io_context(async move {
|
||||
let result = tokio::spawn(async move {
|
||||
let commit = Box::pin(commit(commit_submit_to_closure_enter_started, Some(child_token)));
|
||||
#[cfg(feature = "diagnostic-read-version-caller")]
|
||||
return crate::cluster::rpc::runtime_sources::with_read_version_diagnostic_context(
|
||||
@@ -6612,7 +6612,7 @@ impl SetDisks {
|
||||
.await;
|
||||
#[cfg(not(feature = "diagnostic-read-version-caller"))]
|
||||
commit.await
|
||||
}))
|
||||
})
|
||||
.await
|
||||
.map_err(|err| Error::other(format!("put_object commit task failed: {err}")))?;
|
||||
rustfs_io_metrics::record_put_object_stage_duration_from(
|
||||
@@ -25096,64 +25096,6 @@ mod put_object_tmp_cleanup_tests {
|
||||
wait_for_tmp_workspace_to_drain(&dirs, "cancelled full-tail commit should release staging ownership").await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn background_put_retains_disk_admission_across_owned_commit() {
|
||||
use crate::disk::with_background_disk_io;
|
||||
|
||||
let (_dirs, disks, set) = hermetic_set_disks(4).await;
|
||||
let object = "background-cache-commit";
|
||||
let barriers: Vec<_> = (0..disks.len())
|
||||
.map(|index| rename_fanout_barrier::arm(object, index, rename_fanout_barrier::PHASE_RENAME))
|
||||
.collect();
|
||||
let writer = Arc::clone(&set);
|
||||
let put = tokio::spawn(async move {
|
||||
with_background_disk_io(CancellationToken::new(), async {
|
||||
let mut reader = PutObjReader::from_vec(b"scanner checkpoint".to_vec());
|
||||
writer
|
||||
.put_object(
|
||||
RUSTFS_META_BUCKET,
|
||||
object,
|
||||
&mut reader,
|
||||
&ObjectOptions {
|
||||
write_completion: WriteCompletion::TailDrained,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
})
|
||||
.await
|
||||
});
|
||||
for barrier in &barriers {
|
||||
tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused())
|
||||
.await
|
||||
.expect("every disk reaches the commit boundary");
|
||||
}
|
||||
let local_disks: Vec<_> = disks
|
||||
.iter()
|
||||
.map(|disk| match disk.as_ref() {
|
||||
disk::Disk::Local(local) => local,
|
||||
disk::Disk::Remote(_) => panic!("hermetic fixture must use local disks"),
|
||||
})
|
||||
.collect();
|
||||
let before: Vec<_> = local_disks
|
||||
.iter()
|
||||
.map(|disk| disk.background_io_admissions_for_tests())
|
||||
.collect();
|
||||
for barrier in barriers {
|
||||
barrier.release();
|
||||
}
|
||||
put.await.expect("owned put task").expect("background checkpoint commits");
|
||||
for ((disk, local), before) in disks.iter().zip(local_disks).zip(before) {
|
||||
assert!(
|
||||
local.background_io_admissions_for_tests() > before,
|
||||
"the detached rename must retain background admission on every disk"
|
||||
);
|
||||
disk.read_version("", RUSTFS_META_BUCKET, object, "", &ReadOptions::default())
|
||||
.await
|
||||
.expect("committed checkpoint remains readable");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial(capacity_dirty_scope)]
|
||||
async fn borrowed_write_context_rejects_wrong_namespace_and_cache_flag() {
|
||||
|
||||
@@ -33,10 +33,9 @@ pub(crate) mod internode {
|
||||
PUT_FILE_AUTH_TRAILER_MAC_LEN, PUT_FILE_AUTH_TRAILER_MAGIC, PUT_FILE_AUTH_V1, PUT_FILE_CAPABILITY_CHALLENGE_QUERY,
|
||||
PUT_FILE_CAPABILITY_QUERY, PUT_FILE_CAPABILITY_VERSION, PUT_FILE_NONCE_QUERY, PUT_FILE_SERVER_EPOCH_QUERY,
|
||||
PutFileCapabilityResponse, SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION, SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION,
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_ACTIVITY_V7_PROTOCOL_VERSION,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES, SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE, WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_STREAM_COMPLETION_QUERY,
|
||||
WALK_DIR_STREAM_COMPLETION_V1,
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
|
||||
WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_STREAM_COMPLETION_QUERY, WALK_DIR_STREAM_COMPLETION_V1,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -469,8 +469,8 @@ pub use peer::external_rpc_disk;
|
||||
use peer::init_local_peer;
|
||||
pub use peer::{
|
||||
BootstrapLocalTarget, all_local_disk, all_local_disk_path, find_local_disk_by_ref, find_local_disk_by_ref_for_rpc,
|
||||
find_local_disk_by_ref_for_scanner_rpc, get_disk_infos, init_local_disks, init_local_disks_with_instance_ctx,
|
||||
init_lock_clients, prewarm_local_disk_id_map, prewarm_local_disk_id_map_with_instance_ctx,
|
||||
get_disk_infos, init_local_disks, init_local_disks_with_instance_ctx, init_lock_clients, prewarm_local_disk_id_map,
|
||||
prewarm_local_disk_id_map_with_instance_ctx,
|
||||
};
|
||||
|
||||
pub struct ECStore {
|
||||
|
||||
@@ -261,7 +261,6 @@ impl ECStore {
|
||||
}
|
||||
|
||||
/// Execute on this instance's active local disk through the physical owner.
|
||||
/// A verified scanner publication uses the receiver's background disk admission.
|
||||
pub async fn rename_local_data(
|
||||
&self,
|
||||
disk_ref: &str,
|
||||
@@ -279,7 +278,7 @@ impl ECStore {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let rename = rename_local_data_with_ctx(
|
||||
rename_local_data_with_ctx(
|
||||
&self.ctx,
|
||||
disk_ref,
|
||||
source,
|
||||
@@ -290,12 +289,8 @@ impl ECStore {
|
||||
external_guard,
|
||||
namespace_owner: None,
|
||||
},
|
||||
);
|
||||
if scanner_token.is_some() {
|
||||
crate::disk::with_background_disk_io(tokio_util::sync::CancellationToken::new(), rename).await
|
||||
} else {
|
||||
rename.await
|
||||
}
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Recovery stays bound to this admitted instance even after RPC cancellation.
|
||||
@@ -343,24 +338,10 @@ async fn write_local_format_control_with_ctx(
|
||||
}
|
||||
|
||||
async fn local_disk_candidate(ctx: &Arc<InstanceContext>, disk_ref: &str) -> DiskResult<(DiskStore, Option<Uuid>)> {
|
||||
local_disk_candidate_with_admission(ctx, disk_ref, RpcDiskAdmission::Immediate).await
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
enum RpcDiskAdmission {
|
||||
Immediate,
|
||||
ScannerPreflight,
|
||||
}
|
||||
|
||||
async fn local_disk_candidate_with_admission(
|
||||
ctx: &Arc<InstanceContext>,
|
||||
disk_ref: &str,
|
||||
admission: RpcDiskAdmission,
|
||||
) -> DiskResult<(DiskStore, Option<Uuid>)> {
|
||||
let map = ctx.local_disk_map();
|
||||
let path_disk = map.read().await.get(disk_ref).and_then(Option::as_ref).cloned();
|
||||
if let Some(disk) = path_disk {
|
||||
admit_rpc_disk(&disk, admission).await?;
|
||||
admit_rpc_disk(&disk)?;
|
||||
return Ok((disk, None));
|
||||
}
|
||||
let disk_id = Uuid::parse_str(disk_ref).map_err(|_| DiskError::DiskNotFound)?;
|
||||
@@ -370,7 +351,7 @@ async fn local_disk_candidate_with_admission(
|
||||
if let Some(disk) = cached_disk
|
||||
&& matches!(disk.as_ref(), Disk::Local(_))
|
||||
{
|
||||
admit_rpc_disk(&disk, admission).await?;
|
||||
admit_rpc_disk(&disk)?;
|
||||
if disk.get_disk_id().await? == Some(disk_id) {
|
||||
return Ok((disk, None));
|
||||
}
|
||||
@@ -389,24 +370,13 @@ async fn local_disk_candidate_with_admission(
|
||||
Err(DiskError::DiskNotFound)
|
||||
}
|
||||
|
||||
async fn local_scanner_disk_candidate(ctx: &Arc<InstanceContext>, disk_ref: &str) -> DiskResult<(DiskStore, Option<Uuid>)> {
|
||||
let (disk, id) = local_disk_candidate_with_admission(ctx, disk_ref, RpcDiskAdmission::ScannerPreflight).await?;
|
||||
// A queued preflight must not resume on a replaced or quarantined attachment.
|
||||
admit_local_disk(ctx, &disk, id, false, None).await?;
|
||||
Ok((disk, id))
|
||||
}
|
||||
|
||||
async fn admit_rpc_disk(disk: &DiskStore, admission: RpcDiskAdmission) -> DiskResult<()> {
|
||||
fn admit_rpc_disk(disk: &DiskStore) -> DiskResult<()> {
|
||||
use crate::disk::remote_io_admission;
|
||||
if remote_io_admission::active() {
|
||||
let binding = disk
|
||||
.physical_admission()
|
||||
.ok_or_else(|| DiskError::Io(std::io::Error::from(std::io::ErrorKind::WouldBlock)))?;
|
||||
match admission {
|
||||
RpcDiskAdmission::Immediate => remote_io_admission::admit(&binding),
|
||||
RpcDiskAdmission::ScannerPreflight => remote_io_admission::admit_scanner_preflight(&binding).await,
|
||||
}
|
||||
.map_err(DiskError::Io)?;
|
||||
remote_io_admission::admit(&binding).map_err(DiskError::Io)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -597,17 +567,6 @@ pub async fn find_local_disk_by_ref_for_rpc(disk_ref: &str) -> DiskResult<Option
|
||||
Ok(Some(disk))
|
||||
}
|
||||
|
||||
/// Resolve an authenticated namespace scan before its storage locks, waiting for
|
||||
/// initial background capacity. The caller must bound the wait with a deadline.
|
||||
/// Cancellation returns queued capacity; unknown or replaced targets are rejected.
|
||||
pub async fn find_local_disk_by_ref_for_scanner_rpc(disk_ref: &str) -> DiskResult<Option<DiskStore>> {
|
||||
if !crate::disk::remote_io_admission::active() {
|
||||
return Ok(find_local_disk_by_ref(disk_ref).await);
|
||||
}
|
||||
let (disk, _) = local_scanner_disk_candidate(&crate::runtime::global::current_ctx(), disk_ref).await?;
|
||||
Ok(Some(disk))
|
||||
}
|
||||
|
||||
/// Resolve the target from the verified listener's instance and acquire any background admission.
|
||||
#[cfg(feature = "internode-transport")]
|
||||
pub async fn external_rpc_disk(ctx: &Arc<InstanceContext>, disk_ref: &str) -> DiskResult<DiskStore> {
|
||||
@@ -1082,123 +1041,6 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn namespace_target_lookup_waits_for_busy_preflight_capacity() {
|
||||
use crate::disk::{RemoteBackgroundIo, with_remote_background_io};
|
||||
use rustfs_concurrency::adaptive::{
|
||||
AdaptiveAdmission, FeedbackController, FeedbackObservation, NodeAdmission, ResourcePressure,
|
||||
};
|
||||
use std::num::NonZeroU16;
|
||||
|
||||
let ctx = Arc::new(InstanceContext::new());
|
||||
let root = tempfile::tempdir().expect("scanner target directory");
|
||||
let disk = target_disk(&ctx, root.path(), Uuid::new_v4()).await;
|
||||
let gate = AdaptiveAdmission::new(NonZeroU16::MIN, 1);
|
||||
disk.physical_admission()
|
||||
.expect("local binding")
|
||||
.set_test_capacity(vec![gate.clone()])
|
||||
.expect("physical device");
|
||||
let node = NodeAdmission::new(NonZeroU16::MIN);
|
||||
node.update(
|
||||
&mut FeedbackController::new(),
|
||||
Some(FeedbackObservation {
|
||||
generation: 1,
|
||||
max_concurrency: 1,
|
||||
pressure: ResourcePressure::Unknown,
|
||||
completed_units: None,
|
||||
}),
|
||||
);
|
||||
let held = gate.try_acquire().expect("concurrent scanner operation");
|
||||
let path = disk.endpoint().to_string();
|
||||
with_remote_background_io(Some(RemoteBackgroundIo::new(node.clone())), async {
|
||||
let lookup = local_scanner_disk_candidate(&ctx, &path);
|
||||
tokio::pin!(lookup);
|
||||
assert!(
|
||||
futures_util::poll!(lookup.as_mut()).is_pending(),
|
||||
"a namespace preflight must wait for busy capacity before storage locks, not fail the bucket"
|
||||
);
|
||||
assert_eq!(node.snapshot().active, 0, "waiting for the disk must not reserve node capacity");
|
||||
drop(held);
|
||||
let (selected, _) = tokio::time::timeout(std::time::Duration::from_secs(5), lookup)
|
||||
.await
|
||||
.expect("capacity release wakes namespace preflight")
|
||||
.expect("namespace target admission");
|
||||
assert!(Arc::ptr_eq(&selected, &disk));
|
||||
assert_eq!([node.snapshot().active, gate.snapshot().active], [1, 1]);
|
||||
})
|
||||
.await;
|
||||
assert_eq!([node.snapshot().active, gate.snapshot().active], [0, 0]);
|
||||
assert_eq!([node.completed(), gate.completed()], [Some(0), Some(0)]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn namespace_target_wait_rejects_quarantined_or_replaced_disk() {
|
||||
use crate::disk::{RemoteBackgroundIo, with_remote_background_io};
|
||||
use rustfs_concurrency::adaptive::{
|
||||
AdaptiveAdmission, FeedbackController, FeedbackObservation, NodeAdmission, ResourcePressure,
|
||||
};
|
||||
use std::num::NonZeroU16;
|
||||
|
||||
for replace in [false, true] {
|
||||
let ctx = Arc::new(InstanceContext::new());
|
||||
let root = tempfile::tempdir().expect("target directory");
|
||||
let id = Uuid::new_v4();
|
||||
let disk = target_disk(&ctx, root.path(), id).await;
|
||||
assert!(record_local_disk_id_if_active(&ctx, &disk, id).await);
|
||||
let replacement = if replace {
|
||||
let replacement = target_disk(&ctx, root.path(), id).await;
|
||||
ctx.local_disk_map()
|
||||
.write()
|
||||
.await
|
||||
.insert(disk.endpoint().to_string(), Some(disk.clone()));
|
||||
Some(replacement)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let gate = AdaptiveAdmission::new(NonZeroU16::MIN, 1);
|
||||
disk.physical_admission()
|
||||
.expect("local binding")
|
||||
.set_test_capacity(vec![gate.clone()])
|
||||
.expect("physical device");
|
||||
let node = NodeAdmission::new(NonZeroU16::MIN);
|
||||
node.update(
|
||||
&mut FeedbackController::new(),
|
||||
Some(FeedbackObservation {
|
||||
generation: 1,
|
||||
max_concurrency: 1,
|
||||
pressure: ResourcePressure::Unknown,
|
||||
completed_units: None,
|
||||
}),
|
||||
);
|
||||
let held = gate.try_acquire().expect("busy disk");
|
||||
// Both lookup forms must revalidate the registry after queueing.
|
||||
let disk_ref = if replace {
|
||||
id.to_string()
|
||||
} else {
|
||||
disk.endpoint().to_string()
|
||||
};
|
||||
with_remote_background_io(Some(RemoteBackgroundIo::new(node.clone())), async {
|
||||
let lookup = local_scanner_disk_candidate(&ctx, &disk_ref);
|
||||
tokio::pin!(lookup);
|
||||
assert!(futures_util::poll!(&mut lookup).is_pending());
|
||||
if let Some(replacement) = replacement {
|
||||
assert!(!Arc::ptr_eq(&replacement, &disk));
|
||||
ctx.local_disk_map()
|
||||
.write()
|
||||
.await
|
||||
.insert(disk.endpoint().to_string(), Some(replacement));
|
||||
} else {
|
||||
ctx.local_disk_map().write().await.insert(disk.endpoint().to_string(), None);
|
||||
}
|
||||
drop(held);
|
||||
assert!(matches!(lookup.await, Err(DiskError::DiskNotFound)));
|
||||
})
|
||||
.await;
|
||||
assert_eq!([node.snapshot().active, gate.snapshot().active], [0, 0]);
|
||||
assert_eq!([node.completed(), gate.completed()], [Some(0), Some(0)]);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn target_uuid_cache_cannot_admit_a_different_format_at_the_same_path() {
|
||||
let ctx = Arc::new(InstanceContext::new());
|
||||
@@ -1542,10 +1384,6 @@ mod tests {
|
||||
.await
|
||||
.expect("bounded physical entry");
|
||||
let at_entry = (ctx.namespace_commits_pending(), ctx.namespace_commit_generation());
|
||||
let Disk::Local(wrapper) = disk.as_ref() else {
|
||||
panic!("fixture must use a local disk");
|
||||
};
|
||||
assert!(!wrapper.background_io_active_for_tests(), "an ordinary rename stays foreground");
|
||||
assert!(!rename.is_finished(), "caller must still await the paused physical rename");
|
||||
rename.abort();
|
||||
let cancelled = tokio::time::timeout(Duration::from_secs(5), &mut rename)
|
||||
@@ -1664,18 +1502,7 @@ mod tests {
|
||||
})
|
||||
.await
|
||||
.expect("bounded physical entry");
|
||||
let Disk::Local(wrapper) = disk.as_ref() else {
|
||||
panic!("fixture must use a local disk");
|
||||
};
|
||||
assert!(
|
||||
wrapper.background_io_active_for_tests(),
|
||||
"a verified scanner publication must retain background admission on the receiving disk"
|
||||
);
|
||||
drop(rename);
|
||||
assert!(
|
||||
wrapper.background_io_active_for_tests(),
|
||||
"native publication keeps admission after RPC cancellation"
|
||||
);
|
||||
assert!(
|
||||
!store.scanner_data_usage_publication_blocked().await,
|
||||
"a pending namespace commit is not a local publication barrier"
|
||||
|
||||
@@ -165,26 +165,22 @@ impl HealManager {
|
||||
let mut recovery_disks = Vec::new();
|
||||
for disk in &local_disks {
|
||||
let endpoint = disk.endpoint();
|
||||
if let Some(previous) = super::super::shutdown_marker::previous_shutdown_was_unclean(&endpoint.to_string()) {
|
||||
unclean |= previous;
|
||||
} else {
|
||||
match disk
|
||||
.read_all(super::super::RUSTFS_META_BUCKET, super::super::UNCLEAN_SHUTDOWN_MARKER_PATH)
|
||||
.await
|
||||
{
|
||||
Ok(_) => unclean = true,
|
||||
Err(DiskError::FileNotFound) | Err(DiskError::VolumeNotFound) => {}
|
||||
Err(err) => {
|
||||
debug!(
|
||||
target: "rustfs::heal::manager",
|
||||
event = EVENT_HEAL_UNCLEAN_SHUTDOWN,
|
||||
component = LOG_COMPONENT_HEAL,
|
||||
subsystem = LOG_SUBSYSTEM_MANAGER,
|
||||
endpoint = %endpoint,
|
||||
error = ?err,
|
||||
"Unclean-shutdown marker check failed"
|
||||
);
|
||||
}
|
||||
match disk
|
||||
.read_all(super::super::RUSTFS_META_BUCKET, super::super::UNCLEAN_SHUTDOWN_MARKER_PATH)
|
||||
.await
|
||||
{
|
||||
Ok(_) => unclean = true,
|
||||
Err(DiskError::FileNotFound) | Err(DiskError::VolumeNotFound) => {}
|
||||
Err(err) => {
|
||||
debug!(
|
||||
target: "rustfs::heal::manager",
|
||||
event = EVENT_HEAL_UNCLEAN_SHUTDOWN,
|
||||
component = LOG_COMPONENT_HEAL,
|
||||
subsystem = LOG_SUBSYSTEM_MANAGER,
|
||||
endpoint = %endpoint,
|
||||
error = ?err,
|
||||
"Unclean-shutdown marker check failed"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -23,7 +23,6 @@ pub mod progress;
|
||||
mod replacement_execution;
|
||||
pub(crate) mod replacement_readiness;
|
||||
pub mod resume;
|
||||
mod shutdown_marker;
|
||||
pub mod storage;
|
||||
pub(crate) mod storage_api;
|
||||
pub mod task;
|
||||
@@ -39,7 +38,6 @@ use storage_api::owner::{
|
||||
pub use erasure_healer::ErasureSetHealer;
|
||||
pub use manager::{HealAdmissionTelemetry, HealManager, HealOperationsSnapshot, HealPriorityCounts, HealSourceCounts};
|
||||
pub use resume::{CheckpointManager, ResumeCheckpoint, ResumeManager, ResumeState, ResumeUtils};
|
||||
pub use shutdown_marker::{consume_read_only_scanner_handoff, save_read_only_scanner_handoff};
|
||||
pub use task::{HealOptions, HealPriority, HealRequest, HealTask, HealType};
|
||||
|
||||
pub(crate) const DATA_USAGE_CACHE_NAME: &str = ECSTORE_DATA_USAGE_CACHE_NAME;
|
||||
@@ -51,7 +49,7 @@ pub(crate) const POOL_META_NAME: &str = ECSTORE_POOL_META_NAME;
|
||||
/// [`clear_unclean_shutdown_markers`] on graceful shutdown. Finding it at
|
||||
/// startup means the previous run crashed or lost power, so the heal manager
|
||||
/// proactively re-verifies all local erasure sets.
|
||||
pub(crate) const UNCLEAN_SHUTDOWN_MARKER_PATH: &str = storage_api::ECSTORE_UNCLEAN_SHUTDOWN_MARKER_PATH;
|
||||
pub(crate) const UNCLEAN_SHUTDOWN_MARKER_PATH: &str = "unclean-shutdown";
|
||||
|
||||
/// Remove the unclean-shutdown markers from all local disks. Call at the end of
|
||||
/// a graceful shutdown, after the data plane has stopped accepting writes.
|
||||
|
||||
@@ -1,214 +0,0 @@
|
||||
// Copyright 2026 RustFS Team
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! A clean, read-only handoff reuses the existing shutdown marker. Receipt-aware
|
||||
//! readers must consume it before legacy rollback; legacy startup alone cannot
|
||||
//! guarantee that checkpoint continuity expires before a write.
|
||||
|
||||
use super::{DiskError, DiskStore, EcstoreDiskAPI, RUSTFS_META_BUCKET, UNCLEAN_SHUTDOWN_MARKER_PATH, local_disk_map_read};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{LazyLock, Mutex};
|
||||
use uuid::Uuid;
|
||||
|
||||
static PREVIOUS_SHUTDOWN: LazyLock<Mutex<HashMap<String, bool>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct ReadOnlyHandoff {
|
||||
schema: u32,
|
||||
owner: String,
|
||||
disk_id: Uuid,
|
||||
inventory: [u8; 32],
|
||||
checkpoint_instance_id: String,
|
||||
}
|
||||
|
||||
fn valid_instance(instance: &str) -> bool {
|
||||
instance.len() == 32
|
||||
&& instance
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
|
||||
&& instance.bytes().any(|byte| byte != b'0')
|
||||
}
|
||||
|
||||
fn decode(bytes: &[u8], owner: &str, disk_id: Option<Uuid>, inventory: Option<[u8; 32]>) -> Option<ReadOnlyHandoff> {
|
||||
if bytes.len() > 2048 || owner.is_empty() || disk_id.is_none_or(|id| id.is_nil()) {
|
||||
return None;
|
||||
}
|
||||
let record: ReadOnlyHandoff = serde_json::from_slice(bytes).ok()?;
|
||||
(record.schema == 1
|
||||
&& record.owner == owner
|
||||
&& Some(record.disk_id) == disk_id
|
||||
&& Some(record.inventory) == inventory
|
||||
&& valid_instance(&record.checkpoint_instance_id))
|
||||
.then_some(record)
|
||||
}
|
||||
|
||||
async fn local_inventory(disks: &[Option<DiskStore>]) -> Vec<(DiskStore, Option<Uuid>)> {
|
||||
let mut inventory = Vec::with_capacity(disks.len());
|
||||
for disk in disks.iter().flatten() {
|
||||
let id = disk.get_disk_id().await.ok().flatten();
|
||||
inventory.push((disk.clone(), id));
|
||||
}
|
||||
inventory
|
||||
}
|
||||
|
||||
fn inventory_digest(disks: &[(DiskStore, Option<Uuid>)]) -> Option<[u8; 32]> {
|
||||
if disks.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let mut members = disks
|
||||
.iter()
|
||||
.map(|(disk, id)| (disk.endpoint().to_string(), *id))
|
||||
.collect::<Vec<_>>();
|
||||
members.sort_unstable_by(|left, right| left.0.cmp(&right.0));
|
||||
if members.windows(2).any(|pair| pair[0].0 == pair[1].0) {
|
||||
return None;
|
||||
}
|
||||
let mut hash = Sha256::new();
|
||||
hash.update(b"scanner-read-only-local-disks-v1");
|
||||
for (endpoint, id) in members {
|
||||
let id = id.filter(|id| !id.is_nil())?;
|
||||
hash.update(u64::try_from(endpoint.len()).ok()?.to_be_bytes());
|
||||
hash.update(endpoint.as_bytes());
|
||||
hash.update(id.as_bytes());
|
||||
}
|
||||
Some(hash.finalize().into())
|
||||
}
|
||||
|
||||
pub(super) fn previous_shutdown_was_unclean(endpoint: &str) -> Option<bool> {
|
||||
PREVIOUS_SHUTDOWN
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.get(endpoint)
|
||||
.copied()
|
||||
}
|
||||
|
||||
/// Consume every local receipt before starting mutation producers or Scanner.
|
||||
/// Continuity requires matching receipts from the complete local disk inventory.
|
||||
/// The fresh unclean marker must be durable before returning a reusable identity.
|
||||
///
|
||||
/// # Errors
|
||||
/// Returns a storage error if a valid handoff cannot be consumed. Starting a
|
||||
/// writer in that case could leave an older receipt authorizing stale coverage.
|
||||
pub async fn consume_read_only_scanner_handoff(owner: &str, registry: &[Option<DiskStore>]) -> crate::Result<Option<String>> {
|
||||
let disks = local_inventory(registry).await;
|
||||
let inventory = if registry.iter().any(Option::is_none) {
|
||||
None
|
||||
} else {
|
||||
inventory_digest(&disks)
|
||||
};
|
||||
let mut candidate = None;
|
||||
let mut complete = inventory.is_some() && !owner.is_empty();
|
||||
let mut previous = HashMap::new();
|
||||
for (disk, disk_id) in disks {
|
||||
let (record, unclean, absent) = match disk.read_all(RUSTFS_META_BUCKET, UNCLEAN_SHUTDOWN_MARKER_PATH).await {
|
||||
Ok(bytes) => {
|
||||
let record = decode(&bytes, owner, disk_id, inventory);
|
||||
let unclean = record.is_none();
|
||||
(record, unclean, false)
|
||||
}
|
||||
Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => (None, false, true),
|
||||
Err(_) => (None, true, false),
|
||||
};
|
||||
previous.insert(disk.endpoint().to_string(), unclean);
|
||||
if let Some(record) = &record {
|
||||
if candidate
|
||||
.as_ref()
|
||||
.is_some_and(|instance| instance != &record.checkpoint_instance_id)
|
||||
{
|
||||
complete = false;
|
||||
}
|
||||
candidate = Some(record.checkpoint_instance_id.clone());
|
||||
} else {
|
||||
complete = false;
|
||||
}
|
||||
// This system-volume write uses the disk's metadata durability path.
|
||||
// A crash after consumption leaves no clean receipt to reuse.
|
||||
if let Err(error) = disk
|
||||
.write_all(RUSTFS_META_BUCKET, UNCLEAN_SHUTDOWN_MARKER_PATH, bytes::Bytes::from_static(b"running"))
|
||||
.await
|
||||
{
|
||||
if !absent {
|
||||
return Err(error.into());
|
||||
}
|
||||
complete = false;
|
||||
}
|
||||
}
|
||||
*PREVIOUS_SHUTDOWN.lock().unwrap_or_else(std::sync::PoisonError::into_inner) = previous;
|
||||
Ok(complete.then_some(candidate).flatten())
|
||||
}
|
||||
|
||||
/// Publish a handoff only after frontends, background producers and detached
|
||||
/// mutations have drained, and only if the caller observed no namespace or
|
||||
/// maintenance mutation during this process. No dirty hint is cleared here.
|
||||
///
|
||||
/// # Errors
|
||||
/// Fails closed on a missing disk identity, invalid owner, or persistence error.
|
||||
/// A partially written inventory cannot authorize continuity on the next start.
|
||||
pub async fn save_read_only_scanner_handoff(owner: &str, instance: &str) -> crate::Result<()> {
|
||||
if owner.is_empty() || !valid_instance(instance) {
|
||||
return Err(crate::Error::InvalidCheckpoint("invalid scanner handoff identity".to_owned()));
|
||||
}
|
||||
let registry = local_disk_map_read().await.values().cloned().collect::<Vec<_>>();
|
||||
if registry.iter().any(Option::is_none) {
|
||||
return Err(crate::Error::InvalidCheckpoint(
|
||||
"scanner handoff has an unavailable local disk".to_owned(),
|
||||
));
|
||||
}
|
||||
let disks = local_inventory(®istry).await;
|
||||
let inventory = inventory_digest(&disks)
|
||||
.ok_or_else(|| crate::Error::InvalidCheckpoint("scanner handoff has no complete disk inventory".to_owned()))?;
|
||||
for (disk, disk_id) in disks {
|
||||
let disk_id = disk_id
|
||||
.filter(|id| !id.is_nil())
|
||||
.ok_or_else(|| crate::Error::InvalidCheckpoint("scanner handoff has no disk identity".to_owned()))?;
|
||||
let record = ReadOnlyHandoff {
|
||||
schema: 1,
|
||||
owner: owner.to_owned(),
|
||||
disk_id,
|
||||
inventory,
|
||||
checkpoint_instance_id: instance.to_owned(),
|
||||
};
|
||||
let bytes = serde_json::to_vec(&record).map_err(|error| crate::Error::Serialization(error.to_string()))?;
|
||||
disk.write_all(RUSTFS_META_BUCKET, UNCLEAN_SHUTDOWN_MARKER_PATH, bytes.into())
|
||||
.await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn read_only_handoff_requires_exact_owner_disk_and_supported_state() {
|
||||
let id = Uuid::new_v4();
|
||||
let record = ReadOnlyHandoff {
|
||||
schema: 1,
|
||||
owner: "node-a".to_owned(),
|
||||
disk_id: id,
|
||||
inventory: [7; 32],
|
||||
checkpoint_instance_id: "1".repeat(32),
|
||||
};
|
||||
let bytes = serde_json::to_vec(&record).expect("valid handoff");
|
||||
assert!(decode(&bytes, "node-a", Some(id), Some([7; 32])).is_some());
|
||||
assert!(decode(&bytes, "node-a", Some(id), Some([8; 32])).is_none());
|
||||
for (owner, disk) in [("node-b", Some(id)), ("node-a", Some(Uuid::new_v4())), ("node-a", None)] {
|
||||
assert!(decode(&bytes, owner, disk, Some([7; 32])).is_none());
|
||||
}
|
||||
for bytes in [b"running".as_slice(), b"1730000000", b"{}", b"{\"schema\":1}"] {
|
||||
assert!(decode(bytes, "node-a", Some(id), Some([7; 32])).is_none());
|
||||
}
|
||||
for field in ["schema", "checkpoint_instance_id", "unknown"] {
|
||||
let mut value = serde_json::to_value(&record).expect("handoff JSON");
|
||||
value[field] = if field == "schema" {
|
||||
serde_json::json!(2)
|
||||
} else {
|
||||
serde_json::json!("0".repeat(32))
|
||||
};
|
||||
assert!(decode(&serde_json::to_vec(&value).expect("invalid handoff"), "node-a", Some(id), Some([7; 32])).is_none());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -25,7 +25,6 @@ pub(crate) use rustfs_ecstore::api::disk::{
|
||||
DiskStore as EcstoreDiskStore, HEAL_REPLACEMENT_RECOVERY_DIR as ECSTORE_HEAL_REPLACEMENT_RECOVERY_DIR,
|
||||
HEALING_MARKER_PATH as ECSTORE_HEALING_MARKER_PATH, RUSTFS_META_BUCKET as ECSTORE_RUSTFS_META_BUCKET,
|
||||
ReplacementExecutionLease as EcstoreReplacementExecutionLease,
|
||||
UNCLEAN_SHUTDOWN_MARKER_PATH as ECSTORE_UNCLEAN_SHUTDOWN_MARKER_PATH,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::disk::{
|
||||
|
||||
@@ -3351,9 +3351,6 @@ pub struct AllocatorMemoryObservation {
|
||||
pub reserved_bytes: Option<u64>,
|
||||
pub committed_bytes: Option<u64>,
|
||||
pub page_committed_bytes: Option<u64>,
|
||||
/// Allocator-reported block bytes, including size-class rounding and delayed statistics updates.
|
||||
/// This is not live requested bytes or resident memory.
|
||||
pub allocated_bytes: Option<u64>,
|
||||
pub malloc_requested_bytes: Option<u64>,
|
||||
pub malloc_requested_peak_bytes: Option<u64>,
|
||||
pub malloc_requested_total_bytes: Option<u64>,
|
||||
@@ -3388,11 +3385,6 @@ pub fn record_allocator_memory_observation(backend: &'static str, observation: A
|
||||
.set(if observation.reserved_bytes.is_some() { 1.0 } else { 0.0 });
|
||||
gauge!("rustfs_memory_allocator_committed_bytes_available", "backend" => backend)
|
||||
.set(if observation.committed_bytes.is_some() { 1.0 } else { 0.0 });
|
||||
gauge!("rustfs_memory_allocator_allocated_bytes_available", "backend" => backend)
|
||||
.set(if observation.allocated_bytes.is_some() { 1.0 } else { 0.0 });
|
||||
if let Some(allocated_bytes) = observation.allocated_bytes {
|
||||
gauge!("rustfs_memory_allocator_allocated_bytes", "backend" => backend).set(allocated_bytes as f64);
|
||||
}
|
||||
if let Some(reserved_bytes) = observation.reserved_bytes {
|
||||
gauge!("rustfs_memory_allocator_reserved_bytes", "backend" => backend).set(reserved_bytes as f64);
|
||||
}
|
||||
@@ -3867,41 +3859,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allocator_block_bytes_preserve_decreases_missing_and_zero() {
|
||||
let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner());
|
||||
let sequence = [
|
||||
(None, None, 0.0),
|
||||
(Some(128), Some(128.0), 1.0),
|
||||
(Some(64), Some(64.0), 1.0),
|
||||
(None, Some(64.0), 0.0),
|
||||
(Some(0), Some(0.0), 1.0),
|
||||
];
|
||||
for (index, (_, expected, available)) in sequence.into_iter().enumerate() {
|
||||
// Debugging snapshots consume values; replay each prefix before its only snapshot.
|
||||
let recorder = DebuggingRecorder::new();
|
||||
let snapshotter = recorder.snapshotter();
|
||||
metrics::with_local_recorder(&recorder, || {
|
||||
set_metrics_enabled(true);
|
||||
for (current, ..) in sequence.iter().take(index + 1) {
|
||||
record_allocator_memory_observation(
|
||||
"mimalloc",
|
||||
AllocatorMemoryObservation {
|
||||
allocated_bytes: *current,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
}
|
||||
set_metrics_enabled(false);
|
||||
});
|
||||
let rows = snapshotter.snapshot().into_vec();
|
||||
assert_eq!(gauge_value(&rows, "rustfs_memory_allocator_allocated_bytes_available"), Some(available));
|
||||
assert_eq!(gauge_value(&rows, "rustfs_memory_allocator_allocated_bytes"), expected);
|
||||
assert_eq!(gauge_value(&rows, "rustfs_memory_allocator_malloc_requested_bytes_available"), Some(0.0));
|
||||
assert_eq!(gauge_value(&rows, "rustfs_memory_allocator_malloc_requested_bytes"), None);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allocator_current_availability_tracks_missing_zero_and_recovery() {
|
||||
let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner());
|
||||
|
||||
@@ -1,300 +0,0 @@
|
||||
// Copyright 2026 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::{INTERNODE_CONTROL_RPC_CONCURRENCY_LIMIT, dial_channel, proto_gen::node_service};
|
||||
use prost::Message;
|
||||
use std::{
|
||||
collections::HashSet,
|
||||
io,
|
||||
sync::{
|
||||
Arc,
|
||||
atomic::{AtomicUsize, Ordering},
|
||||
},
|
||||
time::Duration,
|
||||
};
|
||||
use tokio::{
|
||||
io::{AsyncReadExt, AsyncWriteExt},
|
||||
net::{TcpListener, TcpStream},
|
||||
sync::{mpsc, oneshot},
|
||||
task::JoinSet,
|
||||
};
|
||||
|
||||
const BATCHES: usize = 80;
|
||||
const HEADERS: u8 = 1;
|
||||
const RST_STREAM: u8 = 3;
|
||||
const SETTINGS: u8 = 4;
|
||||
const PING: u8 = 6;
|
||||
const GOAWAY: u8 = 7;
|
||||
const WINDOW_UPDATE: u8 = 8;
|
||||
const END_HEADERS: u8 = 4;
|
||||
const END_STREAM: u8 = 1;
|
||||
|
||||
struct Frame {
|
||||
kind: u8,
|
||||
flags: u8,
|
||||
stream: u32,
|
||||
payload: Vec<u8>,
|
||||
}
|
||||
|
||||
async fn read_frame(socket: &mut TcpStream) -> io::Result<Frame> {
|
||||
let mut header = [0; 9];
|
||||
socket.read_exact(&mut header).await?;
|
||||
let length = usize::from(header[0]) * 65536 + usize::from(header[1]) * 256 + usize::from(header[2]);
|
||||
if length > 16384 {
|
||||
return Err(io::Error::other("test peer received an oversized frame"));
|
||||
}
|
||||
let mut payload = vec![0; length];
|
||||
socket.read_exact(&mut payload).await?;
|
||||
Ok(Frame {
|
||||
kind: header[3],
|
||||
flags: header[4],
|
||||
stream: u32::from_be_bytes(header[5..9].try_into().expect("four stream ID bytes")) & 0x7fff_ffff,
|
||||
payload,
|
||||
})
|
||||
}
|
||||
|
||||
async fn write_frame(socket: &mut TcpStream, kind: u8, flags: u8, stream: u32, payload: &[u8]) -> io::Result<()> {
|
||||
let length = u32::try_from(payload.len()).expect("bounded test frame length").to_be_bytes();
|
||||
assert_eq!(length[0], 0, "HTTP/2 frame length fits 24 bits");
|
||||
let mut header = [0; 9];
|
||||
header[..3].copy_from_slice(&length[1..]);
|
||||
header[3] = kind;
|
||||
header[4] = flags;
|
||||
header[5..].copy_from_slice(&stream.to_be_bytes());
|
||||
socket.write_all(&header).await?;
|
||||
socket.write_all(payload).await
|
||||
}
|
||||
|
||||
async fn write_probe_response(socket: &mut TcpStream, stream: u32) -> io::Result<()> {
|
||||
// HPACK's static :status=200 and literal content-type=application/grpc.
|
||||
write_frame(socket, HEADERS, END_HEADERS, stream, b"\x88\x0f\x10\x10application/grpc").await?;
|
||||
let response = node_service::ReadVersionResponse {
|
||||
success: true,
|
||||
..Default::default()
|
||||
}
|
||||
.encode_to_vec();
|
||||
let mut body = vec![0];
|
||||
body.extend_from_slice(&u32::try_from(response.len()).expect("small probe response").to_be_bytes());
|
||||
body.extend(response);
|
||||
write_frame(socket, 0, 0, stream, &body).await?;
|
||||
// A literal grpc-status=0 trailer ends the live probe RPC.
|
||||
write_frame(socket, HEADERS, END_HEADERS | END_STREAM, stream, b"\x00\x0bgrpc-status\x010").await
|
||||
}
|
||||
|
||||
async fn accept_peer(listener: TcpListener) -> io::Result<TcpStream> {
|
||||
let (mut socket, _) = listener.accept().await?;
|
||||
socket.set_nodelay(true)?;
|
||||
let mut preface = [0; 24];
|
||||
socket.read_exact(&mut preface).await?;
|
||||
assert_eq!(&preface, b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n");
|
||||
write_frame(&mut socket, SETTINGS, 0, 0, &[]).await?;
|
||||
Ok(socket)
|
||||
}
|
||||
|
||||
async fn delayed_response_peer(listener: TcpListener, ready: mpsc::Sender<()>, error_resets: Arc<AtomicUsize>) -> io::Result<()> {
|
||||
let mut socket = accept_peer(listener).await?;
|
||||
let mut cancelled = Vec::with_capacity(INTERNODE_CONTROL_RPC_CONCURRENCY_LIMIT);
|
||||
let mut resets = HashSet::new();
|
||||
let mut probe = None;
|
||||
loop {
|
||||
let frame = read_frame(&mut socket).await?;
|
||||
match frame.kind {
|
||||
SETTINGS | PING if frame.flags & 1 == 0 => {
|
||||
let payload = if frame.kind == PING { frame.payload.as_slice() } else { &[] };
|
||||
write_frame(&mut socket, frame.kind, 1, 0, payload).await?;
|
||||
}
|
||||
0 if !frame.payload.is_empty() => {
|
||||
let credit = u32::try_from(frame.payload.len()).expect("bounded DATA frame").to_be_bytes();
|
||||
write_frame(&mut socket, WINDOW_UPDATE, 0, 0, &credit).await?;
|
||||
}
|
||||
HEADERS if cancelled.len() < INTERNODE_CONTROL_RPC_CONCURRENCY_LIMIT => {
|
||||
cancelled.push(frame.stream);
|
||||
if cancelled.len() == INTERNODE_CONTROL_RPC_CONCURRENCY_LIMIT {
|
||||
ready.send(()).await.map_err(|_| io::Error::other("test driver stopped"))?;
|
||||
}
|
||||
}
|
||||
HEADERS => {
|
||||
assert!(probe.replace(frame.stream).is_none(), "one live probe per batch");
|
||||
}
|
||||
RST_STREAM => {
|
||||
let reason = u32::from_be_bytes(frame.payload.as_slice().try_into().expect("RST_STREAM error code"));
|
||||
if reason == 5 {
|
||||
error_resets.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
if cancelled.contains(&frame.stream) {
|
||||
resets.insert(frame.stream);
|
||||
}
|
||||
}
|
||||
GOAWAY => return Err(io::Error::other(format!("client GOAWAY: {:?}", frame.payload))),
|
||||
_ => {}
|
||||
}
|
||||
if let Some(stream) = probe
|
||||
&& resets.len() == INTERNODE_CONTROL_RPC_CONCURRENCY_LIMIT
|
||||
{
|
||||
// A raw peer controls bytes already in flight at cancellation. A normal
|
||||
// server would consume RST_STREAM and suppress these delayed responses.
|
||||
for cancelled_stream in &cancelled {
|
||||
write_frame(&mut socket, HEADERS, END_HEADERS | END_STREAM, *cancelled_stream, b"\x88").await?;
|
||||
}
|
||||
write_probe_response(&mut socket, stream).await?;
|
||||
cancelled.clear();
|
||||
resets.clear();
|
||||
probe = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn assert_cancelled_responses_are_ignored(bulk: bool) {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind isolated peer");
|
||||
let address = format!("http://{}", listener.local_addr().expect("peer address"));
|
||||
let (ready_tx, mut ready_rx) = mpsc::channel(1);
|
||||
let error_resets = Arc::new(AtomicUsize::new(0));
|
||||
let peer = tokio::spawn(delayed_response_peer(listener, ready_tx, error_resets.clone()));
|
||||
let cache_key = if bulk {
|
||||
super::bulk_cache_key(&address, 0)
|
||||
} else {
|
||||
address.clone()
|
||||
};
|
||||
let result = tokio::time::timeout(Duration::from_secs(30), async {
|
||||
let (channel, _) = dial_channel(&address, &cache_key, Duration::from_secs(10))
|
||||
.await
|
||||
.expect("dial real internode channel");
|
||||
let mut client = node_service::node_service_client::NodeServiceClient::new(channel);
|
||||
for batch in 0..BATCHES {
|
||||
let mut requests = JoinSet::new();
|
||||
for _ in 0..INTERNODE_CONTROL_RPC_CONCURRENCY_LIMIT {
|
||||
let mut pending_client = client.clone();
|
||||
requests.spawn(async move { pending_client.read_version(node_service::ReadVersionRequest::default()).await });
|
||||
}
|
||||
ready_rx.recv().await.expect("peer observed every pending request");
|
||||
requests.abort_all();
|
||||
while let Some(result) = requests.join_next().await {
|
||||
assert!(result.expect_err("pending RPC task was cancelled").is_cancelled());
|
||||
}
|
||||
let response = client
|
||||
.read_version(node_service::ReadVersionRequest::default())
|
||||
.await
|
||||
.map_err(|error| format!("live probe after cancellation batch {batch}: {error}"))?;
|
||||
assert!(response.into_inner().success);
|
||||
}
|
||||
Ok::<_, String>(())
|
||||
})
|
||||
.await;
|
||||
peer.abort();
|
||||
let peer_result = peer.await;
|
||||
assert!(matches!(result, Ok(Ok(()))), "shared channel failed: {result:?}; peer: {peer_result:?}");
|
||||
assert_eq!(
|
||||
error_resets.load(Ordering::Relaxed),
|
||||
0,
|
||||
"late cancelled responses must not spend the error budget"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn cancelled_metadata_responses_keep_the_shared_channel_usable() {
|
||||
assert_cancelled_responses_are_ignored(false).await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn cancelled_metadata_responses_keep_the_bulk_channel_usable() {
|
||||
temp_env::async_with_vars(
|
||||
[(rustfs_config::ENV_INTERNODE_BULK_CONCURRENCY_LIMIT, Some("64"))],
|
||||
assert_cancelled_responses_are_ignored(true),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
async fn unsolicited_response_peer(listener: TcpListener, attack: oneshot::Receiver<()>) -> io::Result<(usize, Vec<u8>)> {
|
||||
let mut socket = accept_peer(listener).await?;
|
||||
let mut completed = Vec::with_capacity(1025);
|
||||
while completed.len() < 1025 {
|
||||
let frame = read_frame(&mut socket).await?;
|
||||
match frame.kind {
|
||||
HEADERS => {
|
||||
write_probe_response(&mut socket, frame.stream).await?;
|
||||
completed.push(frame.stream);
|
||||
}
|
||||
SETTINGS if frame.flags & 1 == 0 => write_frame(&mut socket, SETTINGS, 1, 0, &[]).await?,
|
||||
PING if frame.flags & 1 == 0 => write_frame(&mut socket, PING, 1, 0, &frame.payload).await?,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
attack
|
||||
.await
|
||||
.map_err(|_| io::Error::other("test driver stopped before attack"))?;
|
||||
// These streams completed normally, so duplicate responses are protocol errors,
|
||||
// not late responses to a cancellation that the grace period should absorb.
|
||||
let mut resets = 0;
|
||||
for stream in completed {
|
||||
write_frame(&mut socket, HEADERS, END_HEADERS | END_STREAM, stream, b"\x88").await?;
|
||||
// Drain each reset before provoking the next one: GOAWAY may discard
|
||||
// queued frames, so a burst's wire count is not the internal error count.
|
||||
loop {
|
||||
let frame = read_frame(&mut socket).await?;
|
||||
match frame.kind {
|
||||
RST_STREAM => {
|
||||
assert_eq!(frame.stream, stream);
|
||||
assert_eq!(frame.payload, 5_u32.to_be_bytes());
|
||||
resets += 1;
|
||||
break;
|
||||
}
|
||||
GOAWAY => return Ok((resets, frame.payload)),
|
||||
PING if frame.flags & 1 == 0 => write_frame(&mut socket, PING, 1, 0, &frame.payload).await?,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(io::Error::other("peer accepted more than 1024 protocol errors"))
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn cancelled_stream_retention_preserves_the_protocol_error_limit() {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind isolated peer");
|
||||
let address = format!("http://{}", listener.local_addr().expect("peer address"));
|
||||
let (attack_tx, attack_rx) = oneshot::channel();
|
||||
let mut peer = tokio::spawn(unsolicited_response_peer(listener, attack_rx));
|
||||
let result = tokio::time::timeout(Duration::from_secs(30), async {
|
||||
let (channel, _) = dial_channel(&address, &address, Duration::from_secs(10))
|
||||
.await
|
||||
.expect("dial real internode channel");
|
||||
let mut client = node_service::node_service_client::NodeServiceClient::new(channel);
|
||||
for _ in 0..1025 {
|
||||
assert!(
|
||||
client
|
||||
.read_version(node_service::ReadVersionRequest::default())
|
||||
.await
|
||||
.expect("initial live probe")
|
||||
.into_inner()
|
||||
.success
|
||||
);
|
||||
}
|
||||
attack_tx
|
||||
.send(())
|
||||
.expect("release duplicate responses after the RPC completes");
|
||||
(&mut peer).await
|
||||
})
|
||||
.await;
|
||||
if result.is_err() {
|
||||
peer.abort();
|
||||
let _ = peer.await;
|
||||
}
|
||||
let (resets, payload) = result
|
||||
.expect("error budget is bounded")
|
||||
.expect("peer task")
|
||||
.expect("peer exchange");
|
||||
assert_eq!(resets, 1024, "the existing local protocol-error reset limit remains enabled");
|
||||
assert!(payload.len() >= 8);
|
||||
assert_eq!(u32::from_be_bytes(payload[4..8].try_into().expect("GOAWAY reason")), 11);
|
||||
assert_eq!(&payload[8..], b"too_many_internal_resets");
|
||||
}
|
||||
@@ -1588,10 +1588,6 @@ pub struct ScannerActivityResponse {
|
||||
pub movement_generation: ::core::option::Option<u64>,
|
||||
#[prost(bool, optional, tag = "11")]
|
||||
pub publication_blocked: ::core::option::Option<bool>,
|
||||
/// v8 binds checkpoint reuse to a verified clean, read-only handoff.
|
||||
/// The process instance above remains fresh for RPC and ACK fencing.
|
||||
#[prost(string, optional, tag = "12")]
|
||||
pub checkpoint_instance_id: ::core::option::Option<::prost::alloc::string::String>,
|
||||
}
|
||||
#[allow(clippy::all)]
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
|
||||
@@ -12,8 +12,6 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#[cfg(test)]
|
||||
mod channel_reset_tests;
|
||||
pub mod compat_manifest;
|
||||
// SAFETY: `generated` is prost/tonic-generated protocol code. The allowance is
|
||||
// scoped to that module so generated internals do not relax lints elsewhere.
|
||||
@@ -709,21 +707,6 @@ pub fn canonical_scanner_activity_v7_response_body(
|
||||
pub mod scoped_dirty_usage;
|
||||
pub mod usage_watch;
|
||||
|
||||
/// Binds checkpoint continuity independently of the fresh RPC process ID.
|
||||
pub fn canonical_scanner_activity_v8_response_body(
|
||||
challenge: &[u8],
|
||||
response: &proto_gen::node_service::ScannerActivityResponse,
|
||||
) -> Result<Vec<u8>, std::num::TryFromIntError> {
|
||||
let mut body = canonical_scanner_activity_v7_response_body(challenge, response)?;
|
||||
body.extend_from_slice(b"rustfs-scanner-checkpoint-continuity-v1\0");
|
||||
body.push(u8::from(response.checkpoint_instance_id.is_some()));
|
||||
if let Some(instance) = &response.checkpoint_instance_id {
|
||||
body.extend_from_slice(&u64::try_from(instance.len())?.to_be_bytes());
|
||||
body.extend_from_slice(instance.as_bytes());
|
||||
}
|
||||
Ok(body)
|
||||
}
|
||||
|
||||
pub fn canonical_scanner_dirty_usage_snapshot_request_body(
|
||||
request: &proto_gen::node_service::ScannerDirtyUsageSnapshotRequest,
|
||||
) -> Result<Vec<u8>, std::num::TryFromIntError> {
|
||||
@@ -2515,9 +2498,9 @@ mod scanner_activity_tests {
|
||||
use super::{
|
||||
canonical_scanner_activity_request_body, canonical_scanner_activity_response_body,
|
||||
canonical_scanner_activity_v4_response_body, canonical_scanner_activity_v7_response_body,
|
||||
canonical_scanner_activity_v8_response_body, canonical_scanner_dirty_usage_snapshot_request_body,
|
||||
canonical_scanner_dirty_usage_snapshot_response_body, canonical_scanner_publication_lease_release_request_body,
|
||||
canonical_scanner_publication_lease_request_body, canonical_scanner_publication_lease_response_body,
|
||||
canonical_scanner_dirty_usage_snapshot_request_body, canonical_scanner_dirty_usage_snapshot_response_body,
|
||||
canonical_scanner_publication_lease_release_request_body, canonical_scanner_publication_lease_request_body,
|
||||
canonical_scanner_publication_lease_response_body,
|
||||
proto_gen::node_service::{
|
||||
ScannerActivityRequest, ScannerActivityResponse, ScannerDirtyUsageBucket, ScannerDirtyUsageSnapshotRequest,
|
||||
ScannerDirtyUsageSnapshotResponse, ScannerPublicationLeaseRequest, ScannerPublicationLeaseResponse,
|
||||
@@ -2669,7 +2652,6 @@ mod scanner_activity_tests {
|
||||
#[test]
|
||||
fn canonical_scanner_activity_response_binds_challenge_and_every_status_field() {
|
||||
let response = ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -2752,36 +2734,9 @@ mod scanner_activity_tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scanner_activity_v8_authenticates_checkpoint_continuity() {
|
||||
let response = ScannerActivityResponse {
|
||||
protocol_version: 8,
|
||||
instance_id: "1".repeat(32),
|
||||
checkpoint_instance_id: Some("2".repeat(32)),
|
||||
..Default::default()
|
||||
};
|
||||
let baseline = canonical_scanner_activity_v8_response_body(&[1; 16], &response).expect("v8 body");
|
||||
for checkpoint_instance_id in [None, Some("3".repeat(32))] {
|
||||
let changed = ScannerActivityResponse {
|
||||
checkpoint_instance_id,
|
||||
..response.clone()
|
||||
};
|
||||
assert_ne!(
|
||||
baseline,
|
||||
canonical_scanner_activity_v8_response_body(&[1; 16], &changed).expect("changed body")
|
||||
);
|
||||
assert_eq!(
|
||||
canonical_scanner_activity_v7_response_body(&[1; 16], &response).expect("v7 body"),
|
||||
canonical_scanner_activity_v7_response_body(&[1; 16], &changed).expect("v7 compatibility"),
|
||||
"v7 does not authenticate continuity and its decoder must reject this field"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scanner_activity_v4_response_canonicalization_ignores_v5_fields() {
|
||||
let response = ScannerActivityResponse {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: "0123456789abcdef0123456789abcdef".to_string(),
|
||||
namespace_generation: 7,
|
||||
maintenance_generation: 3,
|
||||
@@ -3778,9 +3733,9 @@ fn complete_pending_rebuild_on_cache_hit(dial_addr: &str, cache_key: &str) -> bo
|
||||
pub const INTERNODE_CONTROL_RPC_CONCURRENCY_LIMIT: usize = 64;
|
||||
|
||||
// Quorum reads cancel surplus metadata RPCs in bursts. Keep bounded reset state
|
||||
// for their in-flight replies through the HTTP/2 grace period without spending
|
||||
// the connection's protocol-error budget (h2 remembers only 50 resets by default).
|
||||
const INTERNODE_RPC_MAX_CONCURRENT_RESET_STREAMS: usize = 8192;
|
||||
// for their in-flight replies without spending the connection's protocol-error
|
||||
// budget (h2 remembers only 50 resets by default).
|
||||
const INTERNODE_RPC_MAX_CONCURRENT_RESET_STREAMS: usize = 4096;
|
||||
|
||||
/// Whether control/bulk channel isolation is enabled (env-gated, default off for safe rollout).
|
||||
fn channel_isolation_enabled() -> bool {
|
||||
|
||||
@@ -1011,9 +1011,6 @@ message ScannerActivityResponse {
|
||||
// response shape while newer readers fail closed when they are absent.
|
||||
optional uint64 movement_generation = 10;
|
||||
optional bool publication_blocked = 11;
|
||||
// v8 binds checkpoint reuse to a verified clean, read-only handoff.
|
||||
// The process instance above remains fresh for RPC and ACK fencing.
|
||||
optional string checkpoint_instance_id = 12;
|
||||
}
|
||||
|
||||
message ScannerDirtyUsageBucket {
|
||||
|
||||
@@ -58,7 +58,7 @@ impl From<Region> for legacy_s3s::region::Region {
|
||||
fn from(region: Region) -> Self {
|
||||
match Self::new(region.into_boxed_str()) {
|
||||
Ok(region) => region,
|
||||
Err(invalid) => unreachable!("rustfs_s3_types::Region and the legacy region type share one grammar: {invalid}"),
|
||||
Err(invalid) => unreachable!("rustfs_s3_types::Region and s3s::region::Region share one grammar: {invalid}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -67,7 +67,7 @@ impl From<legacy_s3s::region::Region> for Region {
|
||||
fn from(region: legacy_s3s::region::Region) -> Self {
|
||||
match Self::new(region.into_boxed_str()) {
|
||||
Ok(region) => region,
|
||||
Err(invalid) => unreachable!("the legacy region type and rustfs_s3_types::Region share one grammar: {invalid}"),
|
||||
Err(invalid) => unreachable!("s3s::region::Region and rustfs_s3_types::Region share one grammar: {invalid}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -133,16 +133,6 @@ impl From<S3Error> for legacy_s3s::S3Error {
|
||||
}
|
||||
}
|
||||
|
||||
impl S3Error {
|
||||
/// The pinned legacy serializer's XML response. Call sites stay on this type.
|
||||
/// DELETE BY T4.2 with the rest of this file.
|
||||
pub fn to_http_response(self) -> Result<http::Response<crate::Body>, Self> {
|
||||
let response = legacy_s3s::S3Error::from(self).to_http_response().map_err(Self::from)?;
|
||||
let (parts, body) = response.into_parts();
|
||||
Ok(http::Response::from_parts(parts, crate::Body::from(body)))
|
||||
}
|
||||
}
|
||||
|
||||
impl From<legacy_s3s::S3Error> for S3Error {
|
||||
fn from(error: legacy_s3s::S3Error) -> Self {
|
||||
let code: S3ErrorCode = error.code().clone().into();
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
pub mod init_time;
|
||||
pub mod last_minute;
|
||||
pub mod metrics;
|
||||
pub mod scan_observation;
|
||||
pub mod startup;
|
||||
pub mod usage_candidate;
|
||||
|
||||
|
||||
@@ -13,9 +13,6 @@
|
||||
// limitations under the License.
|
||||
|
||||
use crate::last_minute::{AccElem, LastMinuteLatency};
|
||||
use crate::scan_observation::{
|
||||
ScannerMetadataReadGuard, ScannerMetadataReadOutcome, ScannerScanObservation, ScannerScanObservationSnapshot,
|
||||
};
|
||||
use crate::usage_candidate::{
|
||||
ScannerUsageCandidateCount, ScannerUsageCandidateKind, ScannerUsageCandidateOutcome, USAGE_CANDIDATE_COUNTER_SLOTS,
|
||||
};
|
||||
@@ -980,7 +977,6 @@ impl ScannerUsagePublicationObservation {
|
||||
|
||||
pub struct Metrics {
|
||||
startup: crate::startup::ScannerStartupObservation,
|
||||
scan_observation: ScannerScanObservation,
|
||||
operations: Vec<AtomicU64>,
|
||||
latency: Vec<LockedLastMinuteLatency>,
|
||||
actions: Vec<AtomicU64>,
|
||||
@@ -1635,9 +1631,6 @@ pub struct ScannerMetricsReport {
|
||||
pub lifecycle_transition: ScannerLifecycleTransitionSnapshot,
|
||||
#[serde(default)]
|
||||
pub usage_freshness: ScannerUsageFreshnessSnapshot,
|
||||
/// Node-local, process-lifetime work observations. Older nodes have no evidence.
|
||||
#[serde(default)]
|
||||
pub scan_observation: ScannerScanObservationSnapshot,
|
||||
/// Process-lifetime candidate-construction results. Missing on older reports
|
||||
/// means unobserved, not accepted or healthy. Publication is tracked separately.
|
||||
#[serde(default)]
|
||||
@@ -2132,38 +2125,9 @@ impl Metrics {
|
||||
&self.startup
|
||||
}
|
||||
|
||||
pub fn scan_observation(&self) -> &ScannerScanObservation {
|
||||
&self.scan_observation
|
||||
}
|
||||
|
||||
/// Start after scanner node admission, before polling the disk read future.
|
||||
/// Disk-permit and RPC waits remain included; decoding is excluded. Dropping
|
||||
/// an unfinished guard observes cancellation, not physical I/O completion.
|
||||
pub fn start_scanner_metadata_read(&self) -> ScannerMetadataReadGuard<'_> {
|
||||
self.scan_observation.start_metadata_read();
|
||||
ScannerMetadataReadGuard {
|
||||
metrics: self,
|
||||
started: std::time::Instant::now(),
|
||||
outcome: ScannerMetadataReadOutcome::Cancelled,
|
||||
bytes: 0,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn finish_scanner_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
|
||||
let bytes = if outcome == ScannerMetadataReadOutcome::Success {
|
||||
bytes
|
||||
} else {
|
||||
0
|
||||
};
|
||||
self.scan_observation.finish_metadata_read(outcome, bytes, elapsed);
|
||||
self.operations[Metric::ReadMetadata as usize].fetch_add(1, Ordering::Relaxed);
|
||||
self.latency[Metric::ReadMetadata as usize].add_size(elapsed, bytes);
|
||||
}
|
||||
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
startup: crate::startup::ScannerStartupObservation::default(),
|
||||
scan_observation: ScannerScanObservation::default(),
|
||||
operations: (0..Metric::Last as usize).map(|_| AtomicU64::new(0)).collect(),
|
||||
// Each slot gets its own fresh LockedLastMinuteLatency so that
|
||||
// different metrics never accidentally share state.
|
||||
@@ -2769,8 +2733,6 @@ impl Metrics {
|
||||
/// walked the whole namespace, so an operator can tell why scoped
|
||||
/// convergence is not happening.
|
||||
pub fn record_scanner_scope_fallback(&self, reason: impl Into<String>) {
|
||||
let reason = reason.into();
|
||||
self.scan_observation.record_scope_fallback(&reason);
|
||||
self.scanner_scope_fallback_total.fetch_add(1, Ordering::Relaxed);
|
||||
self.scanner_scope_fallback_last_unix_secs
|
||||
.store(unix_now_secs(), Ordering::Relaxed);
|
||||
@@ -2778,7 +2740,7 @@ impl Metrics {
|
||||
Ok(guard) => guard,
|
||||
Err(poisoned) => poisoned.into_inner(),
|
||||
};
|
||||
*last_reason = reason;
|
||||
*last_reason = reason.into();
|
||||
}
|
||||
|
||||
pub fn record_scanner_usage_durable_success(&self) {
|
||||
@@ -3913,7 +3875,6 @@ impl Metrics {
|
||||
failed: self.scanner_transition_failed.load(Ordering::Relaxed),
|
||||
};
|
||||
m.usage_candidates = self.usage_candidate_counts();
|
||||
m.scan_observation = self.scan_observation.snapshot();
|
||||
let usage_save_result = ScannerUsageSaveResult::from_code(self.scanner_usage_last_save_result.load(Ordering::Relaxed));
|
||||
m.usage_freshness = ScannerUsageFreshnessSnapshot {
|
||||
dirty_pending_buckets: self.scanner_dirty_usage_pending_buckets.load(Ordering::Relaxed),
|
||||
|
||||
@@ -1,675 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Node-local, process-lifetime observations of scanner work. Counters are
|
||||
//! independently loaded, so concurrent reports need not balance exactly.
|
||||
//! These observations never grant cache reuse or publication authority.
|
||||
|
||||
use crate::metrics::Metrics;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
sync::atomic::{AtomicU64, Ordering},
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
|
||||
pub const PREFIX_REUSE_BLOCKERS: [&str; 8] = [
|
||||
"missing_producer_identity",
|
||||
"missing_durable_journal_replay",
|
||||
"restart_gap",
|
||||
"generation_gap",
|
||||
"overflow",
|
||||
"stale_ack_generation",
|
||||
"missing_cold_zero_walk_oracle",
|
||||
"distributed_without_peer_invalidation",
|
||||
];
|
||||
|
||||
const SCOPE_FALLBACK_REASONS: [&str; 6] = [
|
||||
"dirty_snapshot_incomplete",
|
||||
"dirty_generation_exhausted",
|
||||
"dirty_bucket_overflow",
|
||||
"no_listed_dirty_buckets",
|
||||
"baseline_unavailable",
|
||||
"peer_scope_unavailable",
|
||||
];
|
||||
|
||||
/// A storage read result, independent of later metadata decoding or ILM work.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerMetadataReadOutcome {
|
||||
Success,
|
||||
NotFound,
|
||||
Error,
|
||||
Cancelled,
|
||||
}
|
||||
|
||||
/// The existing path that actually copied a cached subtree instead of walking it.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerCacheReuseKind {
|
||||
CleanPrefix,
|
||||
CoveredNamespace,
|
||||
SampledNamespace,
|
||||
ClassifiedNamespace,
|
||||
}
|
||||
|
||||
impl ScannerCacheReuseKind {
|
||||
const ALL: [Self; 4] = [
|
||||
Self::CleanPrefix,
|
||||
Self::CoveredNamespace,
|
||||
Self::SampledNamespace,
|
||||
Self::ClassifiedNamespace,
|
||||
];
|
||||
|
||||
fn label(self) -> &'static str {
|
||||
match self {
|
||||
Self::CleanPrefix => "clean_prefix",
|
||||
Self::CoveredNamespace => "covered_namespace",
|
||||
Self::SampledNamespace => "sampled_namespace",
|
||||
Self::ClassifiedNamespace => "classified_namespace",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Completed bucket-level reuse work, independent of durable publication.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerBucketReuseKind {
|
||||
ScopedColdBucket,
|
||||
CurrentRoot,
|
||||
}
|
||||
|
||||
impl ScannerBucketReuseKind {
|
||||
const ALL: [Self; 2] = [Self::ScopedColdBucket, Self::CurrentRoot];
|
||||
|
||||
fn label(self) -> &'static str {
|
||||
match self {
|
||||
Self::ScopedColdBucket => "scoped_cold_bucket",
|
||||
Self::CurrentRoot => "current_root",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A production folder gate, not a diagnostic preflight.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerCacheReuseGate {
|
||||
CleanPrefix,
|
||||
CoveredNamespace,
|
||||
NamespaceSampling,
|
||||
}
|
||||
|
||||
impl ScannerCacheReuseGate {
|
||||
const ALL: [Self; 3] = [Self::CleanPrefix, Self::CoveredNamespace, Self::NamespaceSampling];
|
||||
|
||||
fn label(self) -> &'static str {
|
||||
match self {
|
||||
Self::CleanPrefix => "clean_prefix",
|
||||
Self::CoveredNamespace => "covered_namespace",
|
||||
Self::NamespaceSampling => "namespace_sampling",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The first existing condition reached by a folder reuse gate.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerCacheReuseDecision {
|
||||
MissingScope,
|
||||
NoResumeSweep,
|
||||
DeepScan,
|
||||
ForwardSweep,
|
||||
ActiveReplication,
|
||||
NotRoot,
|
||||
CompactedParent,
|
||||
MissingCachedChild,
|
||||
InvalidPrefix,
|
||||
DirtyPrefix,
|
||||
ErasureHealthRequired,
|
||||
MissingCoverage,
|
||||
CachedFailures,
|
||||
UncompactedChild,
|
||||
SelectedForWalk,
|
||||
Eligible,
|
||||
}
|
||||
|
||||
impl ScannerCacheReuseDecision {
|
||||
const ALL: [Self; 16] = [
|
||||
Self::MissingScope,
|
||||
Self::NoResumeSweep,
|
||||
Self::DeepScan,
|
||||
Self::ForwardSweep,
|
||||
Self::ActiveReplication,
|
||||
Self::NotRoot,
|
||||
Self::CompactedParent,
|
||||
Self::MissingCachedChild,
|
||||
Self::InvalidPrefix,
|
||||
Self::DirtyPrefix,
|
||||
Self::ErasureHealthRequired,
|
||||
Self::MissingCoverage,
|
||||
Self::CachedFailures,
|
||||
Self::UncompactedChild,
|
||||
Self::SelectedForWalk,
|
||||
Self::Eligible,
|
||||
];
|
||||
|
||||
fn label(self) -> &'static str {
|
||||
match self {
|
||||
Self::MissingScope => "missing_scope",
|
||||
Self::NoResumeSweep => "no_resume_sweep",
|
||||
Self::DeepScan => "deep_scan",
|
||||
Self::ForwardSweep => "forward_sweep",
|
||||
Self::ActiveReplication => "active_replication",
|
||||
Self::NotRoot => "not_root",
|
||||
Self::CompactedParent => "compacted_parent",
|
||||
Self::MissingCachedChild => "missing_cached_child",
|
||||
Self::InvalidPrefix => "invalid_prefix",
|
||||
Self::DirtyPrefix => "dirty_prefix",
|
||||
Self::ErasureHealthRequired => "erasure_health_required",
|
||||
Self::MissingCoverage => "missing_coverage",
|
||||
Self::CachedFailures => "cached_failures",
|
||||
Self::UncompactedChild => "uncompacted_child",
|
||||
Self::SelectedForWalk => "selected_for_walk",
|
||||
Self::Eligible => "eligible",
|
||||
}
|
||||
}
|
||||
|
||||
fn not_evaluated(self) -> bool {
|
||||
matches!(
|
||||
self,
|
||||
Self::MissingScope | Self::NoResumeSweep | Self::DeepScan | Self::ForwardSweep | Self::ActiveReplication
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// Scope admission and end-of-cycle evidence are separate observations. An
|
||||
/// activated completion proof does not mean this cycle used prefix reuse.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerPrefixReuseStage {
|
||||
ScopeResolution,
|
||||
CycleCompletion,
|
||||
}
|
||||
|
||||
/// The first existing baseline validation check that rejected a snapshot.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerBaselineRejection {
|
||||
Missing,
|
||||
Decode,
|
||||
Incomplete,
|
||||
Partial,
|
||||
Marks,
|
||||
Epoch,
|
||||
SourceCount,
|
||||
PlanMissing,
|
||||
PlanMismatch,
|
||||
SourceIndex,
|
||||
SourceMembership,
|
||||
SourceDuplicate,
|
||||
SetIncomplete,
|
||||
SetTombstone,
|
||||
SetEpoch,
|
||||
SetCycle,
|
||||
SetPlan,
|
||||
SourceCoverage,
|
||||
AuthoritativeIdentity,
|
||||
NotNewer,
|
||||
}
|
||||
|
||||
impl ScannerBaselineRejection {
|
||||
const ALL: [Self; 20] = [
|
||||
Self::Missing,
|
||||
Self::Decode,
|
||||
Self::Incomplete,
|
||||
Self::Partial,
|
||||
Self::Marks,
|
||||
Self::Epoch,
|
||||
Self::SourceCount,
|
||||
Self::PlanMissing,
|
||||
Self::PlanMismatch,
|
||||
Self::SourceIndex,
|
||||
Self::SourceMembership,
|
||||
Self::SourceDuplicate,
|
||||
Self::SetIncomplete,
|
||||
Self::SetTombstone,
|
||||
Self::SetEpoch,
|
||||
Self::SetCycle,
|
||||
Self::SetPlan,
|
||||
Self::SourceCoverage,
|
||||
Self::AuthoritativeIdentity,
|
||||
Self::NotNewer,
|
||||
];
|
||||
|
||||
fn label(self) -> &'static str {
|
||||
match self {
|
||||
Self::Missing => "missing",
|
||||
Self::Decode => "decode",
|
||||
Self::Incomplete => "incomplete",
|
||||
Self::Partial => "partial",
|
||||
Self::Marks => "marks",
|
||||
Self::Epoch => "epoch",
|
||||
Self::SourceCount => "source_count",
|
||||
Self::PlanMissing => "plan_missing",
|
||||
Self::PlanMismatch => "plan_mismatch",
|
||||
Self::SourceIndex => "source_index",
|
||||
Self::SourceMembership => "source_membership",
|
||||
Self::SourceDuplicate => "source_duplicate",
|
||||
Self::SetIncomplete => "set_incomplete",
|
||||
Self::SetTombstone => "set_tombstone",
|
||||
Self::SetEpoch => "set_epoch",
|
||||
Self::SetCycle => "set_cycle",
|
||||
Self::SetPlan => "set_plan",
|
||||
Self::SourceCoverage => "source_coverage",
|
||||
Self::AuthoritativeIdentity => "authoritative_identity",
|
||||
Self::NotNewer => "not_newer",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct ScannerMetadataReadSnapshot {
|
||||
pub attempts: u64,
|
||||
pub succeeded: u64,
|
||||
pub not_found: u64,
|
||||
pub errors: u64,
|
||||
pub cancelled: u64,
|
||||
/// Bytes returned by successful storage reads, including corrupt metadata.
|
||||
pub bytes: u64,
|
||||
/// Elapsed read-future time, including failed and cancelled reads.
|
||||
pub elapsed_nanos: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct ScannerCacheReuseSnapshot {
|
||||
/// Completed subtree reuse decisions; not cache entries or objects skipped.
|
||||
pub count: u64,
|
||||
/// Time spent copying and attaching the reused subtree, excluding listing.
|
||||
pub elapsed_nanos: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct ScannerBucketReuseSnapshot {
|
||||
/// Cold bucket copies or current roots successfully handed to the collector.
|
||||
pub count: u64,
|
||||
/// Cold copy/attachment time or current-root channel-send time; excludes validation.
|
||||
pub elapsed_nanos: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct ScannerCacheReuseDecisionSnapshot {
|
||||
/// Policy prerequisites that stopped the gate before cache proof checks.
|
||||
pub not_evaluated: BTreeMap<String, u64>,
|
||||
/// First failed cache/work condition; later conditions are not evaluated.
|
||||
pub blocked: BTreeMap<String, u64>,
|
||||
/// Admitted candidates, which may still need EC classification before copying.
|
||||
pub eligible: u64,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct ScannerPrefixReuseSnapshot {
|
||||
pub evaluated: u64,
|
||||
pub activated: u64,
|
||||
/// All failed checks are counted; one evaluation can have several blockers.
|
||||
pub blockers: BTreeMap<String, u64>,
|
||||
}
|
||||
|
||||
/// Missing on an older node means unobserved, not zero work or healthy reuse.
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
#[serde(default)]
|
||||
pub struct ScannerScanObservationSnapshot {
|
||||
pub metadata_reads: ScannerMetadataReadSnapshot,
|
||||
pub cache_reuse: BTreeMap<String, ScannerCacheReuseSnapshot>,
|
||||
pub bucket_reuse: BTreeMap<String, ScannerBucketReuseSnapshot>,
|
||||
pub reuse_decisions: BTreeMap<String, ScannerCacheReuseDecisionSnapshot>,
|
||||
/// Clean root children still walked because EC scans also check health.
|
||||
pub erasure_clean_prefix_walks: u64,
|
||||
pub prefix_scope_resolution: ScannerPrefixReuseSnapshot,
|
||||
pub prefix_cycle_completion: ScannerPrefixReuseSnapshot,
|
||||
pub scope_fallbacks: BTreeMap<String, u64>,
|
||||
/// Rejections only for cycles with a baseline_unavailable scope fallback.
|
||||
pub authoritative_baseline_rejections: BTreeMap<String, u64>,
|
||||
pub observed_baseline_rejections: BTreeMap<String, u64>,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct ScannerScanObservation {
|
||||
metadata: [AtomicU64; 7],
|
||||
cache_count: [AtomicU64; 4],
|
||||
cache_nanos: [AtomicU64; 4],
|
||||
bucket_count: [AtomicU64; 2],
|
||||
bucket_nanos: [AtomicU64; 2],
|
||||
reuse_decisions: [[AtomicU64; ScannerCacheReuseDecision::ALL.len()]; 3],
|
||||
erasure_clean_prefix_walks: AtomicU64,
|
||||
prefix_evaluated: [AtomicU64; 2],
|
||||
prefix_activated: [AtomicU64; 2],
|
||||
prefix_blockers: [[AtomicU64; PREFIX_REUSE_BLOCKERS.len()]; 2],
|
||||
scope_fallbacks: [AtomicU64; SCOPE_FALLBACK_REASONS.len()],
|
||||
baseline_rejections: [[AtomicU64; ScannerBaselineRejection::ALL.len()]; 2],
|
||||
}
|
||||
|
||||
fn duration_nanos(duration: Duration) -> u64 {
|
||||
u64::try_from(duration.as_nanos()).unwrap_or(u64::MAX)
|
||||
}
|
||||
|
||||
fn labelled_counts(labels: impl IntoIterator<Item = &'static str>, counters: &[AtomicU64]) -> BTreeMap<String, u64> {
|
||||
labels
|
||||
.into_iter()
|
||||
.zip(counters)
|
||||
.map(|(label, count)| (label.to_owned(), count.load(Ordering::Relaxed)))
|
||||
.collect()
|
||||
}
|
||||
|
||||
impl ScannerScanObservation {
|
||||
pub(crate) fn start_metadata_read(&self) {
|
||||
self.metadata[0].fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub(crate) fn finish_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
|
||||
self.metadata[1 + outcome as usize].fetch_add(1, Ordering::Relaxed);
|
||||
if outcome == ScannerMetadataReadOutcome::Success {
|
||||
self.metadata[5].fetch_add(bytes, Ordering::Relaxed);
|
||||
}
|
||||
self.metadata[6].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn record_cache_reuse(&self, kind: ScannerCacheReuseKind, elapsed: Duration) {
|
||||
self.cache_count[kind as usize].fetch_add(1, Ordering::Relaxed);
|
||||
self.cache_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Records completed reuse work only; it grants no publication authority.
|
||||
pub fn record_bucket_reuse(&self, kind: ScannerBucketReuseKind, count: u64, elapsed: Duration) {
|
||||
self.bucket_count[kind as usize].fetch_add(count, Ordering::Relaxed);
|
||||
self.bucket_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Records one first-condition decision without inspecting later prerequisites.
|
||||
pub fn record_cache_reuse_decision(&self, gate: ScannerCacheReuseGate, decision: ScannerCacheReuseDecision) {
|
||||
self.reuse_decisions[gate as usize][decision as usize].fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn record_erasure_clean_prefix_walk(&self) {
|
||||
self.erasure_clean_prefix_walks.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn record_prefix_reuse(&self, stage: ScannerPrefixReuseStage, activated: bool, blockers: &[Option<&str>; 8]) {
|
||||
let stage = stage as usize;
|
||||
self.prefix_evaluated[stage].fetch_add(1, Ordering::Relaxed);
|
||||
if activated {
|
||||
self.prefix_activated[stage].fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
for blocker in blockers.iter().flatten() {
|
||||
if let Some(index) = PREFIX_REUSE_BLOCKERS.iter().position(|label| label == blocker) {
|
||||
self.prefix_blockers[stage][index].fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn record_scope_fallback(&self, reason: &str) {
|
||||
if let Some(index) = SCOPE_FALLBACK_REASONS.iter().position(|label| *label == reason) {
|
||||
self.scope_fallbacks[index].fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn record_baseline_rejection(&self, authoritative: ScannerBaselineRejection, observed: Option<ScannerBaselineRejection>) {
|
||||
self.baseline_rejections[0][authoritative as usize].fetch_add(1, Ordering::Relaxed);
|
||||
if let Some(observed) = observed {
|
||||
self.baseline_rejections[1][observed as usize].fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn snapshot(&self) -> ScannerScanObservationSnapshot {
|
||||
let prefix = |stage: ScannerPrefixReuseStage| ScannerPrefixReuseSnapshot {
|
||||
evaluated: self.prefix_evaluated[stage as usize].load(Ordering::Relaxed),
|
||||
activated: self.prefix_activated[stage as usize].load(Ordering::Relaxed),
|
||||
blockers: labelled_counts(PREFIX_REUSE_BLOCKERS, &self.prefix_blockers[stage as usize]),
|
||||
};
|
||||
ScannerScanObservationSnapshot {
|
||||
metadata_reads: ScannerMetadataReadSnapshot {
|
||||
attempts: self.metadata[0].load(Ordering::Relaxed),
|
||||
succeeded: self.metadata[1].load(Ordering::Relaxed),
|
||||
not_found: self.metadata[2].load(Ordering::Relaxed),
|
||||
errors: self.metadata[3].load(Ordering::Relaxed),
|
||||
cancelled: self.metadata[4].load(Ordering::Relaxed),
|
||||
bytes: self.metadata[5].load(Ordering::Relaxed),
|
||||
elapsed_nanos: self.metadata[6].load(Ordering::Relaxed),
|
||||
},
|
||||
cache_reuse: ScannerCacheReuseKind::ALL
|
||||
.into_iter()
|
||||
.map(|kind| {
|
||||
(
|
||||
kind.label().to_owned(),
|
||||
ScannerCacheReuseSnapshot {
|
||||
count: self.cache_count[kind as usize].load(Ordering::Relaxed),
|
||||
elapsed_nanos: self.cache_nanos[kind as usize].load(Ordering::Relaxed),
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
bucket_reuse: ScannerBucketReuseKind::ALL
|
||||
.into_iter()
|
||||
.map(|kind| {
|
||||
(
|
||||
kind.label().to_owned(),
|
||||
ScannerBucketReuseSnapshot {
|
||||
count: self.bucket_count[kind as usize].load(Ordering::Relaxed),
|
||||
elapsed_nanos: self.bucket_nanos[kind as usize].load(Ordering::Relaxed),
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
reuse_decisions: ScannerCacheReuseGate::ALL
|
||||
.into_iter()
|
||||
.map(|gate| {
|
||||
let counts = &self.reuse_decisions[gate as usize];
|
||||
let decisions = |not_evaluated| {
|
||||
ScannerCacheReuseDecision::ALL
|
||||
.into_iter()
|
||||
.filter(|decision| {
|
||||
*decision != ScannerCacheReuseDecision::Eligible && decision.not_evaluated() == not_evaluated
|
||||
})
|
||||
.map(|decision| (decision.label().to_owned(), counts[decision as usize].load(Ordering::Relaxed)))
|
||||
.collect()
|
||||
};
|
||||
(
|
||||
gate.label().to_owned(),
|
||||
ScannerCacheReuseDecisionSnapshot {
|
||||
not_evaluated: decisions(true),
|
||||
blocked: decisions(false),
|
||||
eligible: counts[ScannerCacheReuseDecision::Eligible as usize].load(Ordering::Relaxed),
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
erasure_clean_prefix_walks: self.erasure_clean_prefix_walks.load(Ordering::Relaxed),
|
||||
prefix_scope_resolution: prefix(ScannerPrefixReuseStage::ScopeResolution),
|
||||
prefix_cycle_completion: prefix(ScannerPrefixReuseStage::CycleCompletion),
|
||||
scope_fallbacks: labelled_counts(SCOPE_FALLBACK_REASONS, &self.scope_fallbacks),
|
||||
authoritative_baseline_rejections: labelled_counts(
|
||||
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
|
||||
&self.baseline_rejections[0],
|
||||
),
|
||||
observed_baseline_rejections: labelled_counts(
|
||||
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
|
||||
&self.baseline_rejections[1],
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Dropping an unfinished read records cancellation. The guard observes the
|
||||
/// read future only; it cannot cancel it or change its storage result.
|
||||
#[must_use]
|
||||
pub struct ScannerMetadataReadGuard<'a> {
|
||||
pub(crate) metrics: &'a Metrics,
|
||||
pub(crate) started: Instant,
|
||||
pub(crate) outcome: ScannerMetadataReadOutcome,
|
||||
pub(crate) bytes: u64,
|
||||
}
|
||||
|
||||
impl ScannerMetadataReadGuard<'_> {
|
||||
pub fn finish(mut self, outcome: ScannerMetadataReadOutcome, bytes: u64) {
|
||||
self.outcome = outcome;
|
||||
self.bytes = bytes;
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ScannerMetadataReadGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
self.metrics
|
||||
.finish_scanner_metadata_read(self.outcome, self.bytes, self.started.elapsed());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::Arc;
|
||||
|
||||
#[tokio::test]
|
||||
async fn metadata_read_reports_storage_results_and_existing_read_metric() {
|
||||
let metrics = Metrics::new();
|
||||
metrics
|
||||
.start_scanner_metadata_read()
|
||||
.finish(ScannerMetadataReadOutcome::Success, 127);
|
||||
metrics
|
||||
.start_scanner_metadata_read()
|
||||
.finish(ScannerMetadataReadOutcome::Success, 0);
|
||||
metrics
|
||||
.start_scanner_metadata_read()
|
||||
.finish(ScannerMetadataReadOutcome::NotFound, 0);
|
||||
metrics
|
||||
.start_scanner_metadata_read()
|
||||
.finish(ScannerMetadataReadOutcome::Error, 0);
|
||||
let report = metrics.report().await;
|
||||
let reads = &report.scan_observation.metadata_reads;
|
||||
assert_eq!(
|
||||
(reads.attempts, reads.succeeded, reads.not_found, reads.errors, reads.cancelled),
|
||||
(4, 2, 1, 1, 0)
|
||||
);
|
||||
assert_eq!(reads.bytes, 127);
|
||||
assert_eq!(report.life_time_ops["read_metadata"], 4);
|
||||
assert_eq!(report.last_minute.actions["read_metadata"].count, 4);
|
||||
assert_eq!(report.last_minute.actions["read_metadata"].bytes, 127);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn metadata_read_cancellation_balances_an_actually_polled_future_once() {
|
||||
let metrics = Arc::new(Metrics::new());
|
||||
let worker_metrics = Arc::clone(&metrics);
|
||||
let (started, admitted) = tokio::sync::oneshot::channel();
|
||||
let worker = tokio::spawn(async move {
|
||||
let read = worker_metrics.start_scanner_metadata_read();
|
||||
started.send(()).expect("observe started read");
|
||||
std::future::pending::<()>().await;
|
||||
read.finish(ScannerMetadataReadOutcome::Success, 99);
|
||||
});
|
||||
tokio::time::timeout(Duration::from_secs(5), admitted)
|
||||
.await
|
||||
.expect("read starts within deadline")
|
||||
.expect("read started");
|
||||
let in_flight = metrics.scan_observation().snapshot().metadata_reads;
|
||||
assert_eq!((in_flight.attempts, in_flight.cancelled, in_flight.succeeded), (1, 0, 0));
|
||||
worker.abort();
|
||||
assert!(worker.await.expect_err("read task was aborted").is_cancelled());
|
||||
let report = metrics.report().await;
|
||||
let reads = &report.scan_observation.metadata_reads;
|
||||
assert_eq!((reads.attempts, reads.cancelled, reads.succeeded, reads.bytes), (1, 1, 0, 0));
|
||||
assert_eq!(report.life_time_ops["read_metadata"], 1);
|
||||
assert_eq!(report.last_minute.actions["read_metadata"].count, 1);
|
||||
assert_eq!(
|
||||
Duration::from_nanos(reads.elapsed_nanos).as_secs(),
|
||||
report.last_minute.actions["read_metadata"].acc_time
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn observation_report_keeps_stages_reasons_and_copy_costs_separate() {
|
||||
let metrics = Metrics::new();
|
||||
let observation = metrics.scan_observation();
|
||||
observation.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, Duration::from_nanos(321));
|
||||
observation.record_bucket_reuse(ScannerBucketReuseKind::ScopedColdBucket, 2, Duration::from_nanos(654));
|
||||
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, Duration::from_nanos(987));
|
||||
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::MissingScope);
|
||||
observation
|
||||
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::ErasureHealthRequired);
|
||||
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, ScannerCacheReuseDecision::Eligible);
|
||||
observation.record_erasure_clean_prefix_walk();
|
||||
observation.record_prefix_reuse(
|
||||
ScannerPrefixReuseStage::ScopeResolution,
|
||||
false,
|
||||
&[
|
||||
Some("missing_producer_identity"),
|
||||
Some("missing_durable_journal_replay"),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
],
|
||||
);
|
||||
observation.record_prefix_reuse(ScannerPrefixReuseStage::CycleCompletion, true, &[None; 8]);
|
||||
metrics.record_scanner_scope_fallback("baseline_unavailable");
|
||||
observation.record_baseline_rejection(ScannerBaselineRejection::Epoch, Some(ScannerBaselineRejection::Missing));
|
||||
let report = metrics.report().await;
|
||||
let snapshot = &report.scan_observation;
|
||||
assert_eq!(snapshot.cache_reuse["clean_prefix"].count, 1);
|
||||
assert_eq!(snapshot.cache_reuse["clean_prefix"].elapsed_nanos, 321);
|
||||
assert_eq!(snapshot.cache_reuse["sampled_namespace"].count, 0);
|
||||
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].count, 2);
|
||||
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].elapsed_nanos, 654);
|
||||
assert_eq!(snapshot.bucket_reuse["current_root"].count, 1);
|
||||
assert_eq!(snapshot.bucket_reuse["current_root"].elapsed_nanos, 987);
|
||||
assert_eq!(snapshot.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"], 1);
|
||||
assert_eq!(snapshot.reuse_decisions["clean_prefix"].blocked["erasure_health_required"], 1);
|
||||
assert_eq!(snapshot.reuse_decisions["covered_namespace"].eligible, 1);
|
||||
|
||||
let mut prior_observations = serde_json::to_value(snapshot).expect("encode observations");
|
||||
let fields = prior_observations.as_object_mut().expect("observation object");
|
||||
fields.remove("bucket_reuse");
|
||||
fields.remove("reuse_decisions");
|
||||
let prior: ScannerScanObservationSnapshot =
|
||||
serde_json::from_value(prior_observations).expect("read earlier observations");
|
||||
assert!(prior.bucket_reuse.is_empty());
|
||||
assert!(prior.reuse_decisions.is_empty());
|
||||
assert_eq!(prior.cache_reuse["clean_prefix"].count, 1);
|
||||
|
||||
assert_eq!(snapshot.erasure_clean_prefix_walks, 1);
|
||||
assert_eq!(
|
||||
(snapshot.prefix_scope_resolution.evaluated, snapshot.prefix_scope_resolution.activated),
|
||||
(1, 0)
|
||||
);
|
||||
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_producer_identity"], 1);
|
||||
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_durable_journal_replay"], 1);
|
||||
assert_eq!(
|
||||
(snapshot.prefix_cycle_completion.evaluated, snapshot.prefix_cycle_completion.activated),
|
||||
(1, 1)
|
||||
);
|
||||
assert_eq!(
|
||||
snapshot.scope_fallbacks["baseline_unavailable"],
|
||||
report.usage_freshness.scope_fallback_total
|
||||
);
|
||||
assert_eq!(snapshot.authoritative_baseline_rejections["epoch"], 1);
|
||||
assert_eq!(snapshot.observed_baseline_rejections["missing"], 1);
|
||||
|
||||
let mut legacy = serde_json::to_value(&report).expect("encode report");
|
||||
legacy
|
||||
.as_object_mut()
|
||||
.expect("report is an object")
|
||||
.remove("scan_observation");
|
||||
let decoded: crate::metrics::ScannerMetricsReport = serde_json::from_value(legacy).expect("read legacy report");
|
||||
assert!(decoded.scan_observation.cache_reuse.is_empty(), "legacy counters are unobserved");
|
||||
assert_eq!(decoded.scan_observation.metadata_reads.attempts, 0);
|
||||
}
|
||||
}
|
||||
@@ -206,32 +206,6 @@ pub(crate) struct DataUsageCacheRevisions {
|
||||
backup: Option<DataUsageCacheRevision>,
|
||||
}
|
||||
|
||||
/// In-memory provenance of the decoded cache and the revisions observed by its loader.
|
||||
#[derive(Clone, Debug)]
|
||||
pub(crate) struct DataUsageCacheBaselineRevisions {
|
||||
revisions: DataUsageCacheRevisions,
|
||||
loaded_from_main: bool,
|
||||
}
|
||||
|
||||
impl DataUsageCacheBaselineRevisions {
|
||||
pub(crate) fn revisions(&self) -> &DataUsageCacheRevisions {
|
||||
&self.revisions
|
||||
}
|
||||
|
||||
/// A backup may catch up to an unchanged primary without changing the decoded baseline.
|
||||
/// Both reads must have decoded that primary; an ETag on a corrupt primary is insufficient.
|
||||
pub(crate) fn permits_unchanged_baseline_since(&self, initial: &Self) -> bool {
|
||||
if self.revisions == initial.revisions {
|
||||
return true;
|
||||
}
|
||||
self.loaded_from_main
|
||||
&& initial.loaded_from_main
|
||||
&& matches!(&initial.revisions.main, DataUsageCacheRevision::Etag(etag) if !etag.is_empty())
|
||||
&& self.revisions.main == initial.revisions.main
|
||||
&& self.revisions.backup.as_ref() == Some(&self.revisions.main)
|
||||
}
|
||||
}
|
||||
|
||||
pub static DATA_USAGE_BUCKET: LazyLock<String> =
|
||||
LazyLock::new(|| format!("{RUSTFS_META_BUCKET}{SLASH_SEPARATOR}{BUCKET_META_PREFIX}"));
|
||||
|
||||
@@ -959,8 +933,8 @@ pub struct DataUsageCacheInfo {
|
||||
#[serde(default)]
|
||||
pub scan_plan_digest: Option<DataUsageScanPlanDigest>,
|
||||
/// Full activity and inventory scope of a set scan; only a complete
|
||||
/// snapshot proves coverage. A completed bucket stores its checkpoint
|
||||
/// coverage plan here until the authoritative aggregate includes it.
|
||||
/// snapshot proves coverage. Bucket caches bind this scope into their
|
||||
/// opaque scan plan digest instead.
|
||||
#[serde(default)]
|
||||
pub scan_coverage_digest: Option<DataUsageScanPlanDigest>,
|
||||
#[serde(default)]
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
use super::*;
|
||||
use bytes::Bytes;
|
||||
use std::sync::Arc;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Debug)]
|
||||
struct CachePublicationAdmissionUnavailable;
|
||||
@@ -71,7 +70,6 @@ struct DataUsageCacheLoadResult {
|
||||
cache: DataUsageCache,
|
||||
main_revision: DataUsageCacheRevision,
|
||||
backup_revision: Option<DataUsageCacheRevision>,
|
||||
loaded_from_main: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
@@ -108,18 +106,6 @@ impl DataUsageCache {
|
||||
store: Arc<S>,
|
||||
name: &str,
|
||||
) -> StorageResult<DataUsageCacheRevisions> {
|
||||
self.load_with_baseline_revisions(store, name)
|
||||
.await
|
||||
.map(|baseline| baseline.revisions)
|
||||
}
|
||||
|
||||
/// Loads cache revisions with provenance for unchanged-primary backup reconciliation.
|
||||
/// Missing or rebuilt caches and a decoded backup do not establish a decoded primary.
|
||||
pub(crate) async fn load_with_baseline_revisions<S: ScannerObjectIO>(
|
||||
&mut self,
|
||||
store: Arc<S>,
|
||||
name: &str,
|
||||
) -> StorageResult<DataUsageCacheBaselineRevisions> {
|
||||
let backup_name = format!("{name}.bkp");
|
||||
let backup_path = path_join_buf(&[BUCKET_META_PREFIX, &backup_name]);
|
||||
let loaded = Self::load_cache(store.clone(), name).await?;
|
||||
@@ -147,10 +133,7 @@ impl DataUsageCache {
|
||||
let main = loaded.main_revision;
|
||||
*self = loaded.cache;
|
||||
|
||||
Ok(DataUsageCacheBaselineRevisions {
|
||||
revisions: DataUsageCacheRevisions { main, backup },
|
||||
loaded_from_main: loaded.loaded_from_main,
|
||||
})
|
||||
Ok(DataUsageCacheRevisions { main, backup })
|
||||
}
|
||||
|
||||
pub(crate) async fn read_revisions<S: ScannerObjectIO>(store: Arc<S>, name: &str) -> StorageResult<DataUsageCacheRevisions> {
|
||||
@@ -173,13 +156,10 @@ impl DataUsageCache {
|
||||
cache,
|
||||
revision: Some(main_revision),
|
||||
} => {
|
||||
// Legacy-location fallback carries Missing for the primary path.
|
||||
let loaded_from_main = matches!(&main_revision, DataUsageCacheRevision::Etag(_));
|
||||
return Ok(DataUsageCacheLoadResult {
|
||||
cache: *cache,
|
||||
main_revision,
|
||||
backup_revision: None,
|
||||
loaded_from_main,
|
||||
});
|
||||
}
|
||||
DataUsageCacheLoadAttempt::Loaded { revision: None, .. } => {
|
||||
@@ -217,7 +197,6 @@ impl DataUsageCache {
|
||||
cache: *cache,
|
||||
main_revision,
|
||||
backup_revision: Some(backup_revision),
|
||||
loaded_from_main: false,
|
||||
});
|
||||
}
|
||||
DataUsageCacheLoadState::Loaded
|
||||
@@ -241,7 +220,6 @@ impl DataUsageCache {
|
||||
main_revision: main_revision
|
||||
.ok_or_else(|| Error::other(format!("scanner cache missing state has no revision: {name}")))?,
|
||||
backup_revision,
|
||||
loaded_from_main: false,
|
||||
});
|
||||
}
|
||||
(DataUsageCacheLoadState::Corrupt, DataUsageCacheLoadState::Missing)
|
||||
@@ -261,7 +239,6 @@ impl DataUsageCache {
|
||||
main_revision: main_revision
|
||||
.ok_or_else(|| Error::other(format!("scanner cache corrupt state has no revision: {name}")))?,
|
||||
backup_revision,
|
||||
loaded_from_main: false,
|
||||
});
|
||||
}
|
||||
_ => {}
|
||||
@@ -648,22 +625,14 @@ impl DataUsageCache {
|
||||
self.save_inner(store, name, Some(revisions), None).await
|
||||
}
|
||||
|
||||
/// All epoch-fenced cache writes use background disk admission. Incremental
|
||||
/// checkpoints pass their scan token; finalization uses a fresh token so a
|
||||
/// spent scan budget can still persist progress within the save timeout.
|
||||
pub(crate) async fn save_with_revisions_for_epoch<S: ScannerObjectIO + ScannerConfigObjectDelete>(
|
||||
&self,
|
||||
store: Arc<S>,
|
||||
name: &str,
|
||||
revisions: &DataUsageCacheRevisions,
|
||||
expected_epoch: u64,
|
||||
cancellation: CancellationToken,
|
||||
) -> StorageResult<()> {
|
||||
crate::storage_api::ecstore_with_background_disk_io(
|
||||
cancellation,
|
||||
self.save_inner(store, name, Some(revisions), Some(expected_epoch)),
|
||||
)
|
||||
.await
|
||||
self.save_inner(store, name, Some(revisions), Some(expected_epoch)).await
|
||||
}
|
||||
|
||||
async fn save_inner<S: ScannerObjectIO + ScannerConfigObjectDelete>(
|
||||
|
||||
@@ -772,102 +772,6 @@ async fn corrupt_primary_cache_loads_valid_backup() {
|
||||
assert_eq!(store.puts.load(Ordering::SeqCst), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn baseline_revisions_allow_only_a_decoded_unchanged_primary_backup_catchup() {
|
||||
let etag = |value: &str| DataUsageCacheRevision::Etag(value.to_owned());
|
||||
let initial = DataUsageCacheBaselineRevisions {
|
||||
revisions: DataUsageCacheRevisions {
|
||||
main: etag("primary"),
|
||||
backup: Some(etag("old-backup")),
|
||||
},
|
||||
loaded_from_main: true,
|
||||
};
|
||||
assert!(initial.permits_unchanged_baseline_since(&initial));
|
||||
let mut caught_up = initial.clone();
|
||||
caught_up.revisions.backup = Some(etag("primary"));
|
||||
assert!(caught_up.permits_unchanged_baseline_since(&initial));
|
||||
assert!(
|
||||
!initial.permits_unchanged_baseline_since(&caught_up),
|
||||
"backup rollback changes the baseline"
|
||||
);
|
||||
for backup in [None, Some(DataUsageCacheRevision::Missing)] {
|
||||
let mut missing_backup = initial.clone();
|
||||
missing_backup.revisions.backup = backup;
|
||||
assert!(caught_up.permits_unchanged_baseline_since(&missing_backup));
|
||||
assert!(!missing_backup.permits_unchanged_baseline_since(&initial));
|
||||
}
|
||||
for (initial_from_main, current_from_main) in [(false, true), (true, false), (false, false)] {
|
||||
let mut baseline = initial.clone();
|
||||
baseline.loaded_from_main = initial_from_main;
|
||||
let mut current = caught_up.clone();
|
||||
current.loaded_from_main = current_from_main;
|
||||
assert!(
|
||||
!current.permits_unchanged_baseline_since(&baseline),
|
||||
"backup fallback grants no primary proof"
|
||||
);
|
||||
current.revisions = baseline.revisions.clone();
|
||||
assert!(
|
||||
current.permits_unchanged_baseline_since(&baseline),
|
||||
"exact revisions retain existing admission"
|
||||
);
|
||||
}
|
||||
let mut changed_main = caught_up.clone();
|
||||
changed_main.revisions.main = etag("new-primary");
|
||||
changed_main.revisions.backup = Some(etag("new-primary"));
|
||||
assert!(!changed_main.permits_unchanged_baseline_since(&initial));
|
||||
let mut changed_backup = caught_up;
|
||||
changed_backup.revisions.backup = Some(etag("unrelated-backup"));
|
||||
assert!(!changed_backup.permits_unchanged_baseline_since(&initial));
|
||||
for main in [DataUsageCacheRevision::Missing, etag("")] {
|
||||
let mut baseline = initial.clone();
|
||||
baseline.revisions.main = main.clone();
|
||||
let mut current = baseline.clone();
|
||||
current.revisions.backup = Some(main);
|
||||
assert!(
|
||||
!current.permits_unchanged_baseline_since(&baseline),
|
||||
"missing or empty primary ETag is not proof"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn baseline_revisions_record_the_actual_decoded_cache_source() {
|
||||
let mut expected = DataUsageCache::default();
|
||||
expected.info.name = "decoded".to_owned();
|
||||
let valid = expected.marshal_msg().expect("encode valid cache");
|
||||
for (main, backup, from_main, name) in [
|
||||
(valid.clone(), Some(valid.clone()), true, "decoded"),
|
||||
(b"corrupt".to_vec(), Some(valid.clone()), false, "decoded"),
|
||||
(b"corrupt".to_vec(), None, false, ""),
|
||||
(Vec::new(), None, false, ""),
|
||||
] {
|
||||
let store = Arc::new(CacheReadStore::new(CacheReadBody::Bytes(main), backup));
|
||||
let mut loaded = DataUsageCache::default();
|
||||
let baseline = loaded
|
||||
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("load actual cache source");
|
||||
assert_eq!(baseline.loaded_from_main, from_main);
|
||||
assert_eq!(loaded.info.name, name);
|
||||
let mut raw_loaded = DataUsageCache::default();
|
||||
let raw = raw_loaded
|
||||
.load_with_revisions(store, DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("load compatible raw revisions");
|
||||
assert_eq!(baseline.revisions(), &raw, "raw loader retains its revision contract");
|
||||
}
|
||||
let mut legacy_store = CacheReadStore::new(CacheReadBody::Bytes(Vec::new()), None);
|
||||
legacy_store.legacy = Some(valid);
|
||||
let mut legacy_cache = DataUsageCache::default();
|
||||
let legacy = legacy_cache
|
||||
.load_with_baseline_revisions(Arc::new(legacy_store), "legacy-cache")
|
||||
.await
|
||||
.expect("load cache from the legacy location");
|
||||
assert_eq!(legacy_cache.info.name, "decoded");
|
||||
assert!(!legacy.loaded_from_main);
|
||||
assert_eq!(legacy.revisions.main, DataUsageCacheRevision::Missing);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn corrupt_cache_without_valid_backup_rebuilds_with_cas_revisions() {
|
||||
for backup in [None, Some(b"also-not-msgpack".to_vec())] {
|
||||
|
||||
@@ -5,7 +5,6 @@ use super::*;
|
||||
use std::hint::black_box;
|
||||
use std::sync::atomic::AtomicU64;
|
||||
use std::time::Instant as WallInstant;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
const MAX_WIRE_BYTES: u64 = 32 * 1024 * 1024;
|
||||
const CACHE_NAME: &str = "bucket/cache-cost.bin";
|
||||
@@ -231,7 +230,7 @@ async fn profile_case(objects: usize, scenario: &str, samples: usize) {
|
||||
.await
|
||||
.expect("initial revisions");
|
||||
baseline
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &initial, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &initial, 0)
|
||||
.await
|
||||
.expect("baseline save");
|
||||
|
||||
@@ -288,7 +287,7 @@ async fn profile_case(objects: usize, scenario: &str, samples: usize) {
|
||||
store.reset_counts();
|
||||
let started = WallInstant::now();
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("measured save");
|
||||
save_ns.push(elapsed_ns(started));
|
||||
@@ -310,7 +309,7 @@ async fn profile_case(objects: usize, scenario: &str, samples: usize) {
|
||||
conflicting.info.next_cycle += 1;
|
||||
assert!(matches!(
|
||||
conflicting
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &initial, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &initial, 0)
|
||||
.await,
|
||||
Err(Error::PreconditionFailed)
|
||||
));
|
||||
|
||||
@@ -114,10 +114,8 @@ pub use scanner_io::{
|
||||
probe_scoped_dirty_usage, record_dirty_usage_bucket, record_dirty_usage_bucket_from_producer,
|
||||
record_dirty_usage_bucket_from_producers, record_dirty_usage_object, record_dirty_usage_object_from_producer,
|
||||
record_scanner_maintenance_change, record_status_class_mutation, replay_durable_dirty_usage_producer_record,
|
||||
restore_scanner_checkpoint_instance_id, scanner_activity_epoch, scanner_checkpoint_instance_id,
|
||||
scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state,
|
||||
scanner_maintenance_generation, scanner_process_was_read_only, set_scanner_dirty_usage_clear_observer,
|
||||
set_scanner_dirty_usage_mutation_observer,
|
||||
scanner_activity_epoch, scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state,
|
||||
scanner_maintenance_generation, set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer,
|
||||
};
|
||||
pub use segment_invalidation::SegmentInvalidationProducerIdentity;
|
||||
pub use sleeper::{
|
||||
@@ -127,9 +125,9 @@ pub use sleeper::{
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
pub use storage_api::ScannerReplicationConfig as ReplicationConfig;
|
||||
pub use storage_api::scan::{
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_ACTIVITY_V7_PROTOCOL_VERSION,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES, SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE, SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES,
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
|
||||
SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES,
|
||||
};
|
||||
pub use workload_admission::{set_scanner_node_admission, set_scanner_workload_admission_snapshot_provider};
|
||||
|
||||
|
||||
@@ -595,13 +595,7 @@ pub(crate) async fn write_budget_drain_fixture_frames<W: AsyncWrite + Unpin>(
|
||||
cache.info.scan_plan_digest = Some(request.scan_plan_digest);
|
||||
cache.info.last_update = Some(std::time::SystemTime::now());
|
||||
cache
|
||||
.save_with_revisions_for_epoch(
|
||||
store.clone(),
|
||||
&cache_name,
|
||||
&revisions,
|
||||
identity.publication_epoch,
|
||||
CancellationToken::new(),
|
||||
)
|
||||
.save_with_revisions_for_epoch(store.clone(), &cache_name, &revisions, identity.publication_epoch)
|
||||
.await
|
||||
.expect("durable current-request root CAS");
|
||||
let mut persisted = DataUsageCache::default();
|
||||
@@ -854,7 +848,6 @@ async fn classify_remote_usage_watch_outcome(
|
||||
// The folder completes under its coverage plan; finished cache
|
||||
// reuse and the response remain bound to the requested reuse plan.
|
||||
cache.info.scan_plan_digest = Some(scan_plan_digest);
|
||||
cache.info.scan_coverage_digest = checkpoint_plan_digest;
|
||||
(cache, None)
|
||||
}
|
||||
}
|
||||
@@ -989,27 +982,6 @@ async fn scan_and_persist_local_bucket_with_store(
|
||||
let tier_registry_generation = crate::runtime_tier_registry_for_cycle(next_cycle, leader_epoch)
|
||||
.await
|
||||
.generation;
|
||||
// Only a leader handoff needs this authoritative floor; ordinary cycles
|
||||
// avoid an extra config read. An unavailable/legacy floor forces a walk.
|
||||
let completed_checkpoint_cycle_floor = if cache.info.snapshot_complete
|
||||
&& cache.info.leader_epoch < leader_epoch
|
||||
&& cache.info.scan_plan_digest == Some(scan_plan_digest)
|
||||
&& checkpoint_plan_digest.is_some()
|
||||
&& cache.info.scan_coverage_digest == checkpoint_plan_digest
|
||||
&& cache.info.next_cycle < next_cycle
|
||||
{
|
||||
let baseline = read_config(store.clone(), crate::DATA_USAGE_OBJ_NAME_PATH.as_str())
|
||||
.await
|
||||
.ok();
|
||||
let sources = store
|
||||
.all_set_disks()
|
||||
.iter()
|
||||
.map(|set| DataUsageCacheSource::new(set.pool_index, set.set_index))
|
||||
.collect();
|
||||
crate::scanner_io::completed_checkpoint_cycle_floor(baseline.as_deref(), &sources, next_cycle, leader_epoch)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let current_identity = crate::scanner_io::scanner_bucket_checkpoint_identity(
|
||||
&set,
|
||||
&bucket,
|
||||
@@ -1040,7 +1012,6 @@ async fn scan_and_persist_local_bucket_with_store(
|
||||
tier_registry_generation: Some(tier_registry_generation),
|
||||
checkpoint_identity: current_identity,
|
||||
checkpoint_plan_digest,
|
||||
completed_checkpoint_cycle_floor,
|
||||
},
|
||||
);
|
||||
match scan_state {
|
||||
@@ -1293,7 +1264,7 @@ async fn scan_and_persist_local_bucket_with_store(
|
||||
// the scan. A movement transition that starts and ends during the scan
|
||||
// therefore cannot admit the stale cache under the new epoch.
|
||||
let save_result = cache
|
||||
.save_with_revisions_for_epoch(set.clone(), &cache_name, &revisions, expected_publication_epoch, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(set.clone(), &cache_name, &revisions, expected_publication_epoch)
|
||||
.await;
|
||||
done_save();
|
||||
if let Err(err) = save_result {
|
||||
|
||||
@@ -591,7 +591,6 @@ async fn remote_worker_rechecks_real_producers_before_persisting_complete_usage(
|
||||
tier_registry_generation: Some(tier_generation),
|
||||
checkpoint_identity: Some(identity),
|
||||
checkpoint_plan_digest: Some(plan),
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
},
|
||||
);
|
||||
crate::scanner_io::bind_usage_watch_to_checkpoint(&mut cache, Some(&proof), plan);
|
||||
|
||||
@@ -248,13 +248,13 @@ const CLEAN_IDLE_BACKOFF_FACTOR: u32 = 2;
|
||||
/// otherwise idle instance whose clean-idle backoff had doubled a 60 s
|
||||
/// interval), which defeats the fast path it is meant to protect.
|
||||
///
|
||||
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] protects
|
||||
/// against a persistently hot bucket driving an unbroken full-scan loop, so
|
||||
/// retries can start small: 5 s, 10 s, 20 s … Superseded retries cap this
|
||||
/// schedule at the ordinary cycle interval and preserve an operator-selected
|
||||
/// nominal cadence. Preflight deferrals retain the generic
|
||||
/// [`SCANNER_RETRY_MAX_INTERVAL`] cap so a temporarily unavailable peer cannot
|
||||
/// drive a tight retry loop.
|
||||
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] is
|
||||
/// what protects against a persistently hot bucket driving an unbroken
|
||||
/// full-scan loop, so it can start small: 5 s, 10 s, 20 s … capped by
|
||||
/// [`SCANNER_RETRY_MAX_INTERVAL`]. A one-off race recovers in seconds; a
|
||||
/// genuinely hot bucket still reaches minute-scale backoff within a handful of
|
||||
/// cycles. Preflight deferrals use the same bounded schedule so a temporarily
|
||||
/// unavailable peer cannot drive a tight retry loop.
|
||||
const SCANNER_RETRY_BASE_INTERVAL: Duration = Duration::from_secs(5);
|
||||
const SCANNER_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30 * 60);
|
||||
/// Probe cadence while data movement pauses the scanner. Recovery retries use
|
||||
@@ -3528,7 +3528,8 @@ where
|
||||
));
|
||||
let mut wait_plan =
|
||||
scanner_cycle_wait_plan(&runtime_config, clean_idle_backoff, backoff_enabled, randomized_cycle_delay_for);
|
||||
let superseded_retry_interval = scanner_superseded_retry_interval(superseded_backoff, &runtime_config);
|
||||
let superseded_retry_interval =
|
||||
scanner_superseded_retry_interval(superseded_backoff, &runtime_config, maintenance_features);
|
||||
let deferred_retry_interval = deferred_backoff.retry_interval(runtime_config.cycle_interval);
|
||||
let scope_fallback_retry_interval = scanner_scope_fallback_retry_interval(scope_fallback_backoff, &runtime_config);
|
||||
let mut convergence_retry_interval = superseded_retry_interval
|
||||
@@ -4450,10 +4451,9 @@ pub use activity::scanner_topology_digest;
|
||||
pub(crate) use activity::{
|
||||
LOCAL_SCANNER_ACTIVITY_NODE, ScannerActivityGuardrailVeto, ScannerActivitySnapshot, ScannerDirtyUsageAcknowledgement,
|
||||
ScannerDirtyUsageAcknowledgementKind, probe_scanner_activity, scanner_activity_allows_usage_publication,
|
||||
scanner_activity_bucket_reuse_digest, scanner_activity_coverage_digest, scanner_activity_dirty_usage_state_for_host,
|
||||
scanner_activity_guardrail_veto, scanner_activity_publication_lease_targets, scanner_activity_snapshot_digest,
|
||||
scanner_activity_structural_digest, scanner_activity_usage_watch_structure_digest, scanner_dirty_usage_acknowledgements,
|
||||
scanner_peer_dirty_usage_moved,
|
||||
scanner_activity_coverage_digest, scanner_activity_dirty_usage_state_for_host, scanner_activity_guardrail_veto,
|
||||
scanner_activity_publication_lease_targets, scanner_activity_snapshot_digest, scanner_activity_structural_digest,
|
||||
scanner_activity_usage_watch_structure_digest, scanner_dirty_usage_acknowledgements, scanner_peer_dirty_usage_moved,
|
||||
};
|
||||
pub(crate) use activity::{ScannerCycleOutcome, scanner_cycle_outcome_with_pending_maintenance};
|
||||
pub use backlog::{
|
||||
|
||||
@@ -192,16 +192,19 @@ impl ScannerRetryBackoff {
|
||||
pub(super) fn scanner_superseded_retry_interval(
|
||||
backoff: ScannerRetryBackoff,
|
||||
runtime_config: &ScannerRuntimeConfig,
|
||||
maintenance_features: ScannerMaintenanceFeatures,
|
||||
) -> Option<Duration> {
|
||||
let cycle_interval = runtime_config.cycle_interval.max(Duration::from_secs(1));
|
||||
let retry_interval = backoff.retry_interval(cycle_interval)?;
|
||||
// Preserve an operator-selected nominal cadence during convergence.
|
||||
let mut retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?;
|
||||
// Preserve an operator-selected duty-cycle floor during convergence.
|
||||
if runtime_config.cycle_interval_source != ScannerRuntimeConfigSource::Default {
|
||||
return Some(cycle_interval);
|
||||
retry_interval = retry_interval.max(runtime_config.cycle_interval);
|
||||
}
|
||||
// A complete but non-authoritative walk must not postpone convergence
|
||||
// beyond the ordinary cadence, even without lifecycle or replication.
|
||||
Some(retry_interval.min(cycle_interval))
|
||||
// A complete but non-authoritative walk must not postpone lifecycle or
|
||||
// replication evaluation beyond its ordinary maintenance cadence.
|
||||
if maintenance_features.needs_regular_cycle() {
|
||||
retry_interval = retry_interval.min(runtime_config.cycle_interval.max(Duration::from_secs(1)));
|
||||
}
|
||||
Some(retry_interval)
|
||||
}
|
||||
|
||||
/// Pacing for the cycle after a full-namespace scope fallback while dirty
|
||||
@@ -532,7 +535,6 @@ pub(crate) const LOCAL_SCANNER_ACTIVITY_NODE: &str = "<local>";
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) struct ScannerNodeActivity {
|
||||
pub(super) instance_id: String,
|
||||
pub(super) checkpoint_instance_id: Option<String>,
|
||||
pub(super) namespace_generation: u64,
|
||||
pub(super) maintenance_generation: u64,
|
||||
pub(super) protocol_version: u32,
|
||||
@@ -549,7 +551,6 @@ impl ScannerNodeActivity {
|
||||
pub(crate) fn e2e_observation(&self) -> serde_json::Value {
|
||||
serde_json::json!({
|
||||
"instance_id": self.instance_id,
|
||||
"checkpoint_instance_id": self.checkpoint_instance_id,
|
||||
"namespace_generation": self.namespace_generation,
|
||||
"maintenance_generation": self.maintenance_generation,
|
||||
"protocol_version": self.protocol_version,
|
||||
@@ -574,7 +575,6 @@ pub(crate) fn scanner_node_activity_for_tests(
|
||||
) -> ScannerNodeActivity {
|
||||
ScannerNodeActivity {
|
||||
instance_id: instance_id.to_string(),
|
||||
checkpoint_instance_id: None,
|
||||
namespace_generation,
|
||||
maintenance_generation: 0,
|
||||
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
|
||||
@@ -1095,32 +1095,11 @@ where
|
||||
}
|
||||
|
||||
pub(crate) fn scanner_activity_snapshot_digest(snapshot: &ScannerActivitySnapshot) -> [u8; 32] {
|
||||
hash_scanner_activity_members(snapshot.iter().map(|(host, activity)| (host.as_str(), activity)), false)
|
||||
}
|
||||
|
||||
/// Completed bucket reuse retains all mutation and maintenance fences while
|
||||
/// allowing a verified read-only handoff to replace only the RPC process epoch.
|
||||
pub(crate) fn scanner_activity_bucket_reuse_digest(snapshot: &ScannerActivitySnapshot, local_node: &str) -> [u8; 32] {
|
||||
hash_scanner_activity_members(scanner_activity_members(snapshot, local_node).into_iter(), true)
|
||||
}
|
||||
|
||||
fn hash_scanner_activity_members<'a>(
|
||||
members: impl ExactSizeIterator<Item = (&'a str, &'a ScannerNodeActivity)>,
|
||||
checkpoint_continuity: bool,
|
||||
) -> [u8; 32] {
|
||||
let mut hasher = Sha256::new();
|
||||
if checkpoint_continuity {
|
||||
hasher.update(b"scanner-bucket-reuse-v1");
|
||||
}
|
||||
hasher.update(u64::try_from(members.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
for (host, activity) in members {
|
||||
hasher.update(u64::try_from(snapshot.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
for (host, activity) in snapshot {
|
||||
let host = host.as_bytes();
|
||||
let instance_id = if checkpoint_continuity {
|
||||
activity.checkpoint_instance_id.as_deref().unwrap_or(&activity.instance_id)
|
||||
} else {
|
||||
&activity.instance_id
|
||||
}
|
||||
.as_bytes();
|
||||
let instance_id = activity.instance_id.as_bytes();
|
||||
hasher.update(u64::try_from(host.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
hasher.update(host);
|
||||
hasher.update(u64::try_from(instance_id.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
@@ -1139,11 +1118,11 @@ fn hash_scanner_activity_members<'a>(
|
||||
|
||||
/// The reuse identity: hash the activity inputs that make an existing
|
||||
/// scanner result unsafe to reuse without walking again. Besides the
|
||||
/// coverage identity below it binds each peer's checkpoint continuity and
|
||||
/// `maintenance_generation`. A read-only, clean shutdown may preserve that
|
||||
/// continuity; a writer or crash starts fresh because its last dirty-journal
|
||||
/// window may be missing. A changed lifecycle or replication rule also needs
|
||||
/// a maintenance pass. With
|
||||
/// coverage identity below it binds each peer's `instance_id` and
|
||||
/// `maintenance_generation`: a restarted node may have lost the dirty hints
|
||||
/// of its last journal window and a changed lifecycle or replication rule
|
||||
/// needs a maintenance pass, so both invalidate the authoritative baseline
|
||||
/// and the set caches and force the next cycle to walk every bucket. With
|
||||
/// the bucket list it is the plan under which set caches and the
|
||||
/// scoped-scan baseline are reused (`scanner_bucket_plan_digest`).
|
||||
///
|
||||
@@ -1152,17 +1131,12 @@ fn hash_scanner_activity_members<'a>(
|
||||
/// `publication_blocked` is omitted from every identity: a movement pause
|
||||
/// vetoes the cycle it moves in (`scanner_activity_guardrail_veto`) and
|
||||
/// describes nothing about the heap.
|
||||
pub(crate) fn scanner_activity_structural_digest(snapshot: &ScannerActivitySnapshot, local_node: &str) -> [u8; 32] {
|
||||
pub(crate) fn scanner_activity_structural_digest(snapshot: &ScannerActivitySnapshot) -> [u8; 32] {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(b"scanner-reuse-members-v2");
|
||||
hasher.update(u64::try_from(snapshot.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
for (host, activity) in scanner_activity_members(snapshot, local_node) {
|
||||
for (host, activity) in snapshot {
|
||||
let host = host.as_bytes();
|
||||
let instance_id = activity
|
||||
.checkpoint_instance_id
|
||||
.as_deref()
|
||||
.unwrap_or(&activity.instance_id)
|
||||
.as_bytes();
|
||||
let instance_id = activity.instance_id.as_bytes();
|
||||
hasher.update(u64::try_from(host.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
hasher.update(host);
|
||||
hasher.update(u64::try_from(instance_id.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
@@ -1176,64 +1150,33 @@ pub(crate) fn scanner_activity_structural_digest(snapshot: &ScannerActivitySnaps
|
||||
hasher.finalize().into()
|
||||
}
|
||||
|
||||
fn scanner_activity_members<'a>(
|
||||
snapshot: &'a ScannerActivitySnapshot,
|
||||
local_node: &'a str,
|
||||
) -> Vec<(&'a str, &'a ScannerNodeActivity)> {
|
||||
// The local marker moves to another member when leadership changes.
|
||||
// Hash the actual member identities in canonical order instead.
|
||||
let mut nodes = snapshot
|
||||
.iter()
|
||||
.map(|(host, activity)| {
|
||||
(
|
||||
if host == LOCAL_SCANNER_ACTIVITY_NODE {
|
||||
local_node
|
||||
} else {
|
||||
host.as_str()
|
||||
},
|
||||
activity,
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
nodes.sort_unstable_by_key(|(host, _)| *host);
|
||||
nodes
|
||||
}
|
||||
|
||||
/// The coverage identity: hash the activity inputs under which a bucket
|
||||
/// sweep's traversal stays one traversal, so a checkpoint resumed in a later
|
||||
/// cycle is not mixed coverage. It binds the peer set, the protocol, the
|
||||
/// topology and the data-movement state. A coordinator restart, a
|
||||
/// maintenance publication or a publication pause changes none of the
|
||||
/// namespace a sweep walks, so the RPC `instance_id`, `maintenance_generation`
|
||||
/// and `publication_blocked` are left out. Protocol v8 instead binds the
|
||||
/// checkpoint identity inherited only from a durable read-only handoff;
|
||||
/// a lost write hint must invalidate partial as well as completed coverage. The
|
||||
/// standalone node's namespace and dirty generations are left out too: the
|
||||
/// namespace a sweep walks, so `instance_id`, `maintenance_generation` and
|
||||
/// `publication_blocked` are left out; hashing them made every resumed sweep
|
||||
/// restart as mixed coverage after such an event (rustfs/rustfs#8494). The
|
||||
/// local node's namespace and dirty generations are left out too: the
|
||||
/// bucket's own dirty generation joins the checkpoint plan in
|
||||
/// `scanner_bucket_cache_digest`, which is what certification compares.
|
||||
/// Distributed plans retain every member's generations, including the
|
||||
/// coordinator's, so a leader rotation cannot remove a mutation witness.
|
||||
pub(crate) fn scanner_activity_coverage_digest(snapshot: &ScannerActivitySnapshot, local_node: &str) -> [u8; 32] {
|
||||
/// `scanner_bucket_cache_digest`, which is what certification compares. A
|
||||
/// remote peer's generations stay in, because its dirty buckets are still
|
||||
/// acknowledged per cycle rather than per bucket, and a write there between
|
||||
/// the two halves of a sweep would otherwise be acknowledged without having
|
||||
/// been walked.
|
||||
pub(crate) fn scanner_activity_coverage_digest(snapshot: &ScannerActivitySnapshot) -> [u8; 32] {
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(b"scanner-checkpoint-coverage-v2");
|
||||
hasher.update(u64::try_from(snapshot.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
for (host, activity) in scanner_activity_members(snapshot, local_node) {
|
||||
for (host, activity) in snapshot {
|
||||
let host_bytes = host.as_bytes();
|
||||
hasher.update(u64::try_from(host_bytes.len()).unwrap_or(u64::MAX).to_be_bytes());
|
||||
hasher.update(host_bytes);
|
||||
if let Some(instance) = &activity.checkpoint_instance_id {
|
||||
hasher.update(instance.as_bytes());
|
||||
}
|
||||
hasher.update(activity.protocol_version.to_be_bytes());
|
||||
hasher.update(activity.topology_digest);
|
||||
hasher.update([u8::from(activity.data_movement_active)]);
|
||||
hasher.update(activity.movement_generation.to_be_bytes());
|
||||
if host == LOCAL_SCANNER_ACTIVITY_NODE {
|
||||
// Standalone writes are fenced by the bucket's own generation.
|
||||
hasher.update(0_u64.to_be_bytes());
|
||||
hasher.update(0_u64.to_be_bytes());
|
||||
hasher.update([0]);
|
||||
} else {
|
||||
if host.as_str() != LOCAL_SCANNER_ACTIVITY_NODE {
|
||||
hasher.update(activity.namespace_generation.to_be_bytes());
|
||||
hasher.update(activity.dirty_usage_generation.to_be_bytes());
|
||||
hasher.update([u8::from(activity.dirty_usage_pending)]);
|
||||
@@ -1371,10 +1314,8 @@ pub(crate) fn scanner_peer_dirty_usage_moved(
|
||||
pub(crate) fn scanner_activity_allows_usage_publication(snapshot: &ScannerActivitySnapshot) -> bool {
|
||||
!snapshot.is_empty()
|
||||
&& snapshot.values().all(|activity| {
|
||||
matches!(
|
||||
activity.protocol_version,
|
||||
crate::SCANNER_ACTIVITY_V7_PROTOCOL_VERSION | SCANNER_ACTIVITY_PROTOCOL_VERSION
|
||||
) && activity.movement_generation != u64::MAX
|
||||
activity.protocol_version == SCANNER_ACTIVITY_PROTOCOL_VERSION
|
||||
&& activity.movement_generation != u64::MAX
|
||||
&& !activity.data_movement_active
|
||||
&& !activity.publication_blocked
|
||||
})
|
||||
@@ -1481,7 +1422,6 @@ where
|
||||
LOCAL_SCANNER_ACTIVITY_NODE.to_string(),
|
||||
ScannerNodeActivity {
|
||||
instance_id: local_instance_id,
|
||||
checkpoint_instance_id: Some(crate::scanner_checkpoint_instance_id().to_owned()),
|
||||
namespace_generation,
|
||||
maintenance_generation,
|
||||
protocol_version: SCANNER_ACTIVITY_PROTOCOL_VERSION,
|
||||
@@ -1531,7 +1471,7 @@ where
|
||||
SCANNER_ACTIVITY_V6_PROTOCOL_VERSION
|
||||
));
|
||||
}
|
||||
crate::SCANNER_ACTIVITY_V7_PROTOCOL_VERSION | SCANNER_ACTIVITY_PROTOCOL_VERSION => (
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION => (
|
||||
activity
|
||||
.topology_digest
|
||||
.ok_or_else(|| format!("scanner activity peer {host} omitted its storage topology"))?,
|
||||
@@ -1570,7 +1510,6 @@ where
|
||||
host.clone(),
|
||||
ScannerNodeActivity {
|
||||
instance_id: activity.instance_id,
|
||||
checkpoint_instance_id: activity.checkpoint_instance_id,
|
||||
namespace_generation: activity.namespace_generation,
|
||||
maintenance_generation: activity.maintenance_generation,
|
||||
protocol_version: activity.protocol_version,
|
||||
|
||||
@@ -899,7 +899,7 @@ async fn checkpoint_fixture_runtime_deadline_save_reload_resume() {
|
||||
.await
|
||||
.expect("initial cache revisions");
|
||||
checkpoint
|
||||
.save_with_revisions_for_epoch(store.clone(), cache_name, &cache_revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), cache_name, &cache_revisions, 0)
|
||||
.await
|
||||
.expect("publish the current checkpoint into the fenced store");
|
||||
let cache_path = rustfs_utils::path::path_join_buf(&[crate::BUCKET_META_PREFIX, cache_name]);
|
||||
@@ -8947,7 +8947,7 @@ fn clean_idle_backoff_grows_to_cap() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retry_backoff_grows_caps_and_resets_after_convergence() {
|
||||
fn superseded_retry_backoff_grows_caps_and_resets_after_convergence() {
|
||||
let mut backoff = ScannerRetryBackoff::default();
|
||||
assert_eq!(backoff.retry_interval(Duration::from_secs(24 * 60 * 60)), None);
|
||||
|
||||
@@ -8971,7 +8971,7 @@ fn retry_backoff_grows_caps_and_resets_after_convergence() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retry_backoff_respects_a_faster_configured_cycle() {
|
||||
fn superseded_retry_backoff_respects_a_faster_configured_cycle() {
|
||||
let mut backoff = ScannerRetryBackoff::default();
|
||||
backoff.record_retryable_cycle(true);
|
||||
|
||||
@@ -8983,7 +8983,7 @@ fn retry_backoff_respects_a_faster_configured_cycle() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retry_backoff_grows_from_the_default_cycle() {
|
||||
fn superseded_retry_backoff_grows_from_the_default_cycle() {
|
||||
let mut backoff = ScannerRetryBackoff::default();
|
||||
// The first race after a write burst retries in seconds, not a whole
|
||||
// cycle, while repeated supersedes still climb toward the cap.
|
||||
@@ -9003,7 +9003,7 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(
|
||||
scanner_superseded_retry_interval(backoff, &default_config),
|
||||
scanner_superseded_retry_interval(backoff, &default_config, ScannerMaintenanceFeatures::default()),
|
||||
Some(SCANNER_RETRY_BASE_INTERVAL),
|
||||
"the default adaptive cadence must retain fast convergence"
|
||||
);
|
||||
@@ -9013,24 +9013,24 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
|
||||
ScannerRuntimeConfigSource::Config,
|
||||
ScannerRuntimeConfigSource::ScannerCompatConfig,
|
||||
] {
|
||||
for cycle_secs in [1, 3, 600, 6 * 60 * 60] {
|
||||
let runtime_config = ScannerRuntimeConfig {
|
||||
cycle_interval: Duration::from_secs(cycle_secs),
|
||||
cycle_interval_source: source,
|
||||
let runtime_config = ScannerRuntimeConfig {
|
||||
cycle_interval: Duration::from_secs(6 * 60 * 60),
|
||||
cycle_interval_source: source,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
for features in [
|
||||
ScannerMaintenanceFeatures::default(),
|
||||
ScannerMaintenanceFeatures {
|
||||
lifecycle: true,
|
||||
..Default::default()
|
||||
};
|
||||
let mut backoff = ScannerRetryBackoff::default();
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
|
||||
for _ in 0..40 {
|
||||
backoff.record_retryable_cycle(true);
|
||||
assert_eq!(
|
||||
scanner_superseded_retry_interval(backoff, &runtime_config),
|
||||
Some(runtime_config.cycle_interval),
|
||||
"{source:?} cycle cadence must hold throughout convergence retries"
|
||||
);
|
||||
}
|
||||
backoff.record_retryable_cycle(false);
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
|
||||
},
|
||||
] {
|
||||
assert_eq!(
|
||||
scanner_superseded_retry_interval(backoff, &runtime_config, features),
|
||||
Some(runtime_config.cycle_interval),
|
||||
"{source:?} cycle cadence must not be shortened by convergence retries"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9051,7 +9051,7 @@ fn superseded_retry_does_not_delay_required_maintenance() {
|
||||
for consecutive_cycles in 1..=12 {
|
||||
backoff.record_retryable_cycle(true);
|
||||
assert_eq!(
|
||||
scanner_superseded_retry_interval(backoff, &config),
|
||||
scanner_superseded_retry_interval(backoff, &config, features),
|
||||
Some(config.cycle_interval),
|
||||
"superseded cycle {consecutive_cycles} must preserve regular maintenance"
|
||||
);
|
||||
@@ -9137,50 +9137,19 @@ fn scope_fallback_retry_preserves_explicit_and_fast_cycle_cadence() {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn superseded_retry_caps_at_the_default_cycle_and_resets_after_convergence() {
|
||||
fn superseded_retry_only_caps_regular_maintenance() {
|
||||
let config = ScannerRuntimeConfig {
|
||||
cycle_interval: Duration::from_secs(60),
|
||||
..Default::default()
|
||||
};
|
||||
let mut backoff = ScannerRetryBackoff::default();
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
|
||||
for expected in [5, 10, 20, 40, 60, 60, 60, 60] {
|
||||
backoff.record_retryable_cycle(true);
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(Duration::from_secs(expected)));
|
||||
}
|
||||
backoff.consecutive_cycles = u32::MAX;
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
|
||||
backoff.record_retryable_cycle(true);
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
|
||||
backoff.record_retryable_cycle(false);
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn superseded_retry_caps_fast_cycles_without_changing_the_generic_retry_cap() {
|
||||
let mut backoff = ScannerRetryBackoff::default();
|
||||
for _ in 0..40 {
|
||||
for _ in 0..6 {
|
||||
backoff.record_retryable_cycle(true);
|
||||
}
|
||||
assert_eq!(
|
||||
backoff.retry_interval(Duration::from_secs(60)),
|
||||
Some(Duration::from_secs(30 * 60)),
|
||||
"deferred and transient retries must retain the generic cap"
|
||||
scanner_superseded_retry_interval(backoff, &config, ScannerMaintenanceFeatures::default()),
|
||||
Some(Duration::from_secs(160)),
|
||||
);
|
||||
for cycle_secs in [0, 1, 3, 60, 6 * 60 * 60] {
|
||||
let config = ScannerRuntimeConfig {
|
||||
cycle_interval: Duration::from_secs(cycle_secs),
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(
|
||||
scanner_superseded_retry_interval(backoff, &config),
|
||||
Some(Duration::from_secs(cycle_secs.max(1))),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn superseded_retry_timer_keeps_required_maintenance_full_scans() {
|
||||
for features in [
|
||||
ScannerMaintenanceFeatures {
|
||||
lifecycle: true,
|
||||
@@ -9195,23 +9164,28 @@ fn superseded_retry_timer_keeps_required_maintenance_full_scans() {
|
||||
..Default::default()
|
||||
},
|
||||
] {
|
||||
assert!(features.requires_full_scan(Some(7), 7, ScannerCycleWakeReason::Timer, true));
|
||||
assert_eq!(scanner_superseded_retry_interval(backoff, &config, features), Some(config.cycle_interval));
|
||||
assert_eq!(scanner_superseded_retry_interval(ScannerRetryBackoff::default(), &config, features), None);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
#[serial]
|
||||
async fn superseded_retry_wait_keeps_convergence_fencing_at_the_default_cycle() {
|
||||
async fn superseded_maintenance_wait_keeps_convergence_fencing() {
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
let config = ScannerRuntimeConfig {
|
||||
cycle_interval: Duration::from_secs(60),
|
||||
..Default::default()
|
||||
};
|
||||
let features = ScannerMaintenanceFeatures {
|
||||
lifecycle: true,
|
||||
..Default::default()
|
||||
};
|
||||
let mut backoff = ScannerRetryBackoff::default();
|
||||
for _ in 0..12 {
|
||||
backoff.record_retryable_cycle(true);
|
||||
}
|
||||
let interval = scanner_superseded_retry_interval(backoff, &config);
|
||||
let interval = scanner_superseded_retry_interval(backoff, &config, features);
|
||||
assert_eq!(interval, Some(config.cycle_interval));
|
||||
let generations = ScannerCycleObservedGenerations::for_wait(
|
||||
&config,
|
||||
@@ -9226,28 +9200,19 @@ async fn superseded_retry_wait_keeps_convergence_fencing_at_the_default_cycle()
|
||||
let mut seen = Some(BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]));
|
||||
let changed = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 8, 3))]);
|
||||
let started = tokio::time::Instant::now();
|
||||
let mut wait = Box::pin(wait_for_next_scanner_cycle_with_activity(
|
||||
let reason = wait_for_next_scanner_cycle_with_activity(
|
||||
&ctx,
|
||||
interval.expect("superseded retry must retain its convergence interval"),
|
||||
interval.expect("maintenance must retain its convergence interval"),
|
||||
Some(Duration::from_secs(1)),
|
||||
&mut seen,
|
||||
generations,
|
||||
|| false,
|
||||
|| std::future::ready(Ok(changed.clone())),
|
||||
));
|
||||
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
|
||||
for _ in 0..3 {
|
||||
tokio::time::advance(Duration::from_secs(15)).await;
|
||||
crate::scanner_io::record_dirty_usage_bucket("photos");
|
||||
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
|
||||
}
|
||||
let reason = wait.await;
|
||||
)
|
||||
.await;
|
||||
assert_eq!(reason, ScannerCycleWakeReason::Timer);
|
||||
assert_eq!(started.elapsed(), config.cycle_interval);
|
||||
assert_eq!(seen, Some(changed));
|
||||
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -10392,7 +10357,6 @@ async fn recovery_wake_interrupts_normal_cycle_wait() {
|
||||
|
||||
fn scanner_node_activity(epoch: &str, namespace_generation: u64, maintenance_generation: u64) -> ScannerNodeActivity {
|
||||
ScannerNodeActivity {
|
||||
checkpoint_instance_id: None,
|
||||
instance_id: epoch.to_string(),
|
||||
namespace_generation,
|
||||
maintenance_generation,
|
||||
@@ -10413,10 +10377,7 @@ fn scoped_scan_remote_dirty_coverage_invalidates_local_bucket_current() {
|
||||
let remote = after.get_mut("remote").expect("remote activity should exist");
|
||||
remote.dirty_usage_generation += 1;
|
||||
remote.dirty_usage_pending = true;
|
||||
assert_eq!(
|
||||
scanner_activity_structural_digest(&before, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&after, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
);
|
||||
assert_eq!(scanner_activity_structural_digest(&before), scanner_activity_structural_digest(&after));
|
||||
let old_plan = crate::scanner_io::checkpoint_fixture_bucket_digest(
|
||||
DataUsageScanPlanDigest(scanner_activity_snapshot_digest(&before)),
|
||||
None,
|
||||
@@ -10458,7 +10419,6 @@ fn scoped_scan_remote_dirty_coverage_invalidates_local_bucket_current() {
|
||||
tier_registry_generation: None,
|
||||
checkpoint_identity: None,
|
||||
checkpoint_plan_digest: None,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
},
|
||||
),
|
||||
crate::scanner_io::DataUsageCacheScanState::Prepared {
|
||||
@@ -10648,14 +10608,6 @@ fn scanner_activity_snapshot_fences_data_movement() {
|
||||
assert!(scanner_activity_allows_usage_publication(&idle));
|
||||
assert!(!scanner_activity_allows_usage_publication(&moving));
|
||||
assert_ne!(scanner_activity_snapshot_digest(&idle), scanner_activity_snapshot_digest(&moving));
|
||||
let mut v7 = idle;
|
||||
v7.get_mut("node-2").expect("node").protocol_version = crate::SCANNER_ACTIVITY_V7_PROTOCOL_VERSION;
|
||||
assert!(
|
||||
scanner_activity_allows_usage_publication(&v7),
|
||||
"v7 still authenticates terminal movement state"
|
||||
);
|
||||
v7.get_mut("node-2").expect("node").protocol_version = crate::SCANNER_ACTIVITY_V6_PROTOCOL_VERSION;
|
||||
assert!(!scanner_activity_allows_usage_publication(&v7));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -10684,8 +10636,8 @@ fn scanner_activity_structural_digest_ignores_regular_bucket_writes() {
|
||||
|
||||
assert_ne!(scanner_activity_snapshot_digest(&baseline), scanner_activity_snapshot_digest(&written));
|
||||
assert_eq!(
|
||||
scanner_activity_structural_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&written, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&baseline),
|
||||
scanner_activity_structural_digest(&written),
|
||||
"bucket writes are refreshed through the dirty-bucket scope rather than invalidating every cache"
|
||||
);
|
||||
}
|
||||
@@ -10707,24 +10659,18 @@ fn scanner_activity_structural_digest_fences_restart_and_maintenance_but_not_pub
|
||||
// maintenance rule needs a maintenance pass, so both invalidate every
|
||||
// reusable result (the baseline and the set caches).
|
||||
assert_ne!(
|
||||
scanner_activity_structural_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&restarted, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
scanner_activity_structural_digest(&baseline),
|
||||
scanner_activity_structural_digest(&restarted)
|
||||
);
|
||||
assert_ne!(
|
||||
scanner_activity_structural_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&maintained, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
scanner_activity_structural_digest(&baseline),
|
||||
scanner_activity_structural_digest(&maintained)
|
||||
);
|
||||
// A publication pause describes nothing about the heap: it vetoes the
|
||||
// cycle it moves in and is part of no identity.
|
||||
assert_eq!(
|
||||
scanner_activity_structural_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&paused, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
);
|
||||
assert_eq!(scanner_activity_structural_digest(&baseline), scanner_activity_structural_digest(&paused));
|
||||
assert_eq!(scanner_activity_snapshot_digest(&baseline), scanner_activity_snapshot_digest(&paused));
|
||||
assert_eq!(
|
||||
scanner_activity_coverage_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&paused, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
);
|
||||
assert_eq!(scanner_activity_coverage_digest(&baseline), scanner_activity_coverage_digest(&paused));
|
||||
assert_eq!(
|
||||
scanner_activity_guardrail_veto(&baseline, &paused).map(|veto| veto.field),
|
||||
Some("publication_blocked")
|
||||
@@ -10753,8 +10699,8 @@ fn scanner_activity_coverage_digest_survives_restart_and_maintenance_but_fences_
|
||||
let mut changed = baseline.clone();
|
||||
mutate(changed.get_mut(host).expect("host should be present"));
|
||||
assert_eq!(
|
||||
scanner_activity_coverage_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&changed, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&baseline),
|
||||
scanner_activity_coverage_digest(&changed),
|
||||
"{field} on {host} must not change the coverage identity"
|
||||
);
|
||||
assert_eq!(
|
||||
@@ -10773,8 +10719,8 @@ fn scanner_activity_coverage_digest_survives_restart_and_maintenance_but_fences_
|
||||
let mut changed = baseline.clone();
|
||||
mutate(changed.get_mut(LOCAL_SCANNER_ACTIVITY_NODE).expect("local activity"));
|
||||
assert_eq!(
|
||||
scanner_activity_coverage_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&changed, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&baseline),
|
||||
scanner_activity_coverage_digest(&changed),
|
||||
"local {field} is certified per bucket through the bucket's own dirty generation"
|
||||
);
|
||||
// A peer's generations stay in: its dirty buckets are acknowledged
|
||||
@@ -10783,8 +10729,8 @@ fn scanner_activity_coverage_digest_survives_restart_and_maintenance_but_fences_
|
||||
let mut peer_changed = baseline.clone();
|
||||
mutate(peer_changed.get_mut("node-2").expect("peer activity"));
|
||||
assert_ne!(
|
||||
scanner_activity_coverage_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&peer_changed, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&baseline),
|
||||
scanner_activity_coverage_digest(&peer_changed),
|
||||
"remote {field} must change the coverage identity"
|
||||
);
|
||||
}
|
||||
@@ -10799,63 +10745,20 @@ fn scanner_activity_coverage_digest_survives_restart_and_maintenance_but_fences_
|
||||
let mut changed = baseline.clone();
|
||||
mutate(changed.get_mut(host).expect("host should be present"));
|
||||
assert_ne!(
|
||||
scanner_activity_coverage_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&changed, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&baseline),
|
||||
scanner_activity_coverage_digest(&changed),
|
||||
"{field} on {host} must change the coverage identity"
|
||||
);
|
||||
assert_ne!(
|
||||
scanner_activity_structural_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&changed, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&baseline),
|
||||
scanner_activity_structural_digest(&changed),
|
||||
"{field} on {host} must change the reuse identity"
|
||||
);
|
||||
}
|
||||
}
|
||||
let mut peer_lost = baseline.clone();
|
||||
peer_lost.remove("node-2");
|
||||
assert_ne!(
|
||||
scanner_activity_coverage_digest(&baseline, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_coverage_digest(&peer_lost, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scanner_checkpoint_coverage_survives_clean_leader_rotation() {
|
||||
let first = BTreeMap::from([
|
||||
(LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("leader-a", 0, 0)),
|
||||
("node-b".to_string(), scanner_node_activity("peer-b", 0, 0)),
|
||||
("node-c".to_string(), scanner_node_activity("peer-c", 0, 0)),
|
||||
]);
|
||||
let second = BTreeMap::from([
|
||||
(LOCAL_SCANNER_ACTIVITY_NODE.to_string(), scanner_node_activity("leader-b", 0, 0)),
|
||||
("node-a".to_string(), scanner_node_activity("peer-a", 0, 0)),
|
||||
("node-c".to_string(), scanner_node_activity("peer-c", 0, 0)),
|
||||
]);
|
||||
assert_eq!(
|
||||
scanner_activity_coverage_digest(&first, "node-a"),
|
||||
scanner_activity_coverage_digest(&second, "node-b"),
|
||||
"a quiet namespace must retain checkpoint coverage when a different node becomes leader"
|
||||
);
|
||||
assert_ne!(
|
||||
scanner_activity_structural_digest(&first, "node-a"),
|
||||
scanner_activity_structural_digest(&second, "node-b"),
|
||||
"restarted processes must invalidate completed results even with reset mutation counters"
|
||||
);
|
||||
let mut rotated = first.clone();
|
||||
let previous_leader = rotated.remove(LOCAL_SCANNER_ACTIVITY_NODE).expect("previous leader");
|
||||
let next_leader = rotated.remove("node-b").expect("next leader");
|
||||
rotated.insert("node-a".to_string(), previous_leader);
|
||||
rotated.insert(LOCAL_SCANNER_ACTIVITY_NODE.to_string(), next_leader);
|
||||
assert_eq!(
|
||||
scanner_activity_structural_digest(&first, "node-a"),
|
||||
scanner_activity_structural_digest(&rotated, "node-b"),
|
||||
"a leadership change alone must retain completed coverage"
|
||||
);
|
||||
rotated.get_mut("node-a").expect("previous leader").namespace_generation += 1;
|
||||
assert_ne!(
|
||||
scanner_activity_coverage_digest(&first, "node-a"),
|
||||
scanner_activity_coverage_digest(&rotated, "node-b"),
|
||||
"a write on the previous coordinator must invalidate its completed coverage"
|
||||
);
|
||||
assert_ne!(scanner_activity_coverage_digest(&baseline), scanner_activity_coverage_digest(&peer_lost));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -10887,18 +10790,10 @@ fn bucket_checkpoint_plan_survives_a_coordinator_restart() {
|
||||
// restarted coordinator requests, or every resumed sweep restarts as
|
||||
// mixed coverage and the bucket can never certify (rustfs/rustfs#8494).
|
||||
// The reuse plan does change: finished results are walked again.
|
||||
let coverage_plan = |snapshot| {
|
||||
crate::scanner_io::scanner_bucket_plan_digest(
|
||||
&buckets,
|
||||
scanner_activity_coverage_digest(snapshot, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
)
|
||||
};
|
||||
let reuse_plan = |snapshot| {
|
||||
crate::scanner_io::scanner_bucket_plan_digest(
|
||||
&buckets,
|
||||
scanner_activity_structural_digest(snapshot, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
)
|
||||
};
|
||||
let coverage_plan =
|
||||
|snapshot| crate::scanner_io::scanner_bucket_plan_digest(&buckets, scanner_activity_coverage_digest(snapshot));
|
||||
let reuse_plan =
|
||||
|snapshot| crate::scanner_io::scanner_bucket_plan_digest(&buckets, scanner_activity_structural_digest(snapshot));
|
||||
assert_eq!(coverage_plan(&before), coverage_plan(&restarted));
|
||||
assert_ne!(reuse_plan(&before), reuse_plan(&restarted));
|
||||
|
||||
@@ -10906,10 +10801,7 @@ fn bucket_checkpoint_plan_survives_a_coordinator_restart() {
|
||||
extra_bucket.push(bucket_info("gamma"));
|
||||
assert_ne!(
|
||||
coverage_plan(&before),
|
||||
crate::scanner_io::scanner_bucket_plan_digest(
|
||||
&extra_bucket,
|
||||
scanner_activity_coverage_digest(&before, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
),
|
||||
crate::scanner_io::scanner_bucket_plan_digest(&extra_bucket, scanner_activity_coverage_digest(&before)),
|
||||
"the bucket list stays part of the plan"
|
||||
);
|
||||
}
|
||||
@@ -11105,8 +10997,8 @@ fn generation_movement_shades_buckets_without_a_guardrail_veto() {
|
||||
assert_eq!(scanner_activity_guardrail_veto(&before, &after), None, "{field} on {host} must not veto");
|
||||
assert!(SCANNER_ACTIVITY_SHADE_FIELDS.contains(field));
|
||||
assert_eq!(
|
||||
scanner_activity_structural_digest(&before, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&after, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
scanner_activity_structural_digest(&before),
|
||||
scanner_activity_structural_digest(&after),
|
||||
"{field} must stay outside the structural digest"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -395,7 +395,7 @@ async fn assert_coordinator_partial_progress(resume_mixed_plan: bool, fail_cycle
|
||||
progress.started_plan.0[0] ^= 1;
|
||||
let epoch = checkpoint.info.scan_identity.expect("checkpoint identity").publication_epoch;
|
||||
checkpoint
|
||||
.save_with_revisions_for_epoch(set.clone(), &cache_name, &revisions, epoch, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(set.clone(), &cache_name, &revisions, epoch)
|
||||
.await
|
||||
.expect("persist a prefix from a different process plan");
|
||||
|
||||
|
||||
@@ -932,7 +932,7 @@ async fn usage_durable_root_ack_requires_fresh_contributions_from_every_source()
|
||||
progress.started_plan.0[0] ^= 1;
|
||||
let epoch = cache.info.scan_identity.expect("checkpoint identity").publication_epoch;
|
||||
cache
|
||||
.save_with_revisions_for_epoch(set, &cache_name, &revisions, epoch, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(set, &cache_name, &revisions, epoch)
|
||||
.await
|
||||
.expect("persist a resumed interval mismatch on one source");
|
||||
mixed_source = Some(index);
|
||||
|
||||
@@ -59,7 +59,6 @@ use rustfs_scanner_metrics::metrics::{
|
||||
CloseDiskGuard, IlmAction, Metric, Metrics, ScannerReplicationRepairKind, ScannerSourceWorkUpdate, ScannerWorkSource,
|
||||
UpdateCurrentPathFn, current_path_updater, global_metrics,
|
||||
};
|
||||
use rustfs_scanner_metrics::scan_observation::{ScannerCacheReuseDecision, ScannerCacheReuseGate, ScannerCacheReuseKind};
|
||||
use rustfs_utils::path::{SLASH_SEPARATOR, path_join_buf};
|
||||
use sha2::{Digest, Sha256};
|
||||
use time::OffsetDateTime;
|
||||
@@ -1061,51 +1060,38 @@ impl FolderScanner {
|
||||
child_hash: &DataUsageHash,
|
||||
abandoned_children: &DataUsageHashMap,
|
||||
) -> bool {
|
||||
let decision = if let Some(prefix_scan_scope) = &self.prefix_scan_scope {
|
||||
if folder.parent.is_some() || folder.name != self.old_cache.info.name {
|
||||
ScannerCacheReuseDecision::NotRoot
|
||||
} else if into.compacted {
|
||||
ScannerCacheReuseDecision::CompactedParent
|
||||
} else if !abandoned_children.contains(&child_hash.0) {
|
||||
ScannerCacheReuseDecision::MissingCachedChild
|
||||
} else if let Some(entry) = child
|
||||
.name
|
||||
.strip_prefix(folder.name.as_str())
|
||||
.and_then(|entry| entry.strip_prefix('/'))
|
||||
{
|
||||
if entry.is_empty() || entry.contains('/') {
|
||||
ScannerCacheReuseDecision::InvalidPrefix
|
||||
} else if prefix_scan_scope.contains(entry) {
|
||||
ScannerCacheReuseDecision::DirtyPrefix
|
||||
} else if self.is_erasure_mode {
|
||||
// EC scans must still perform independent object-health work.
|
||||
global_metrics().scan_observation().record_erasure_clean_prefix_walk();
|
||||
ScannerCacheReuseDecision::ErasureHealthRequired
|
||||
} else {
|
||||
ScannerCacheReuseDecision::Eligible
|
||||
}
|
||||
} else {
|
||||
ScannerCacheReuseDecision::InvalidPrefix
|
||||
}
|
||||
} else {
|
||||
ScannerCacheReuseDecision::MissingScope
|
||||
let Some(prefix_scan_scope) = &self.prefix_scan_scope else {
|
||||
return false;
|
||||
};
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, decision);
|
||||
decision == ScannerCacheReuseDecision::Eligible
|
||||
// Erasure-mode usage scans also perform probabilistic object-health
|
||||
// work. Reusing a clean subtree here would silently suppress that
|
||||
// independent maintenance path, so prefix reuse is limited to the
|
||||
// non-erasure data-usage scanner.
|
||||
if self.is_erasure_mode
|
||||
|| folder.parent.is_some()
|
||||
|| folder.name != self.old_cache.info.name
|
||||
|| into.compacted
|
||||
|| !abandoned_children.contains(&clone_cache_value(&child_hash.0))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
let Some(entry) = child
|
||||
.name
|
||||
.strip_prefix(folder.name.as_str())
|
||||
.and_then(|entry| entry.strip_prefix('/'))
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
!entry.is_empty() && !entry.contains('/') && !prefix_scan_scope.contains(entry)
|
||||
}
|
||||
|
||||
fn reuse_clean_root_child(&mut self, child: &CachedFolder, child_hash: &DataUsageHash, into: &mut DataUsageEntry) {
|
||||
let started = Instant::now();
|
||||
self.new_cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
|
||||
if let Some(updates) = &mut self.usage_updates {
|
||||
updates.cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
|
||||
}
|
||||
into.add_child(child_hash);
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, started.elapsed());
|
||||
}
|
||||
|
||||
fn failed_retry_suppressed(&self, path: &str) -> bool {
|
||||
@@ -1292,60 +1278,14 @@ impl FolderScanner {
|
||||
}
|
||||
|
||||
fn can_reuse_covered_namespace(&self, name: &str, compacted: bool) -> bool {
|
||||
let decision = if self.old_cache.info.scan_progress.is_none() {
|
||||
ScannerCacheReuseDecision::NoResumeSweep
|
||||
} else if compacted {
|
||||
ScannerCacheReuseDecision::CompactedParent
|
||||
} else if !self
|
||||
.resume_frontier
|
||||
.as_deref()
|
||||
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
|
||||
{
|
||||
ScannerCacheReuseDecision::MissingCoverage
|
||||
} else if self.old_cache.find(name).is_none() {
|
||||
ScannerCacheReuseDecision::MissingCachedChild
|
||||
} else if self.resume_subtree_has_failures(name) {
|
||||
ScannerCacheReuseDecision::CachedFailures
|
||||
} else {
|
||||
ScannerCacheReuseDecision::Eligible
|
||||
};
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, decision);
|
||||
decision == ScannerCacheReuseDecision::Eligible
|
||||
}
|
||||
|
||||
// None preserves work that cannot use sampling; false requests this cycle's rescan.
|
||||
fn sampled_namespace_reuse(&self, hash: &DataUsageHash, compacted: bool, forward_sweep: bool) -> Option<bool> {
|
||||
let decision = if self.scan_mode == HealScanMode::Deep {
|
||||
ScannerCacheReuseDecision::DeepScan
|
||||
} else if forward_sweep {
|
||||
ScannerCacheReuseDecision::ForwardSweep
|
||||
} else if self
|
||||
.old_cache
|
||||
.info
|
||||
.replication
|
||||
.as_ref()
|
||||
.is_some_and(|config| config.has_active_rules("", true))
|
||||
{
|
||||
ScannerCacheReuseDecision::ActiveReplication
|
||||
} else if compacted {
|
||||
ScannerCacheReuseDecision::CompactedParent
|
||||
} else if !self.old_cache.is_compacted(hash) {
|
||||
ScannerCacheReuseDecision::UncompactedChild
|
||||
} else if hash.mod_(self.old_cache.info.next_cycle as u32, data_usage_update_dir_cycles()) {
|
||||
ScannerCacheReuseDecision::SelectedForWalk
|
||||
} else {
|
||||
ScannerCacheReuseDecision::Eligible
|
||||
};
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_cache_reuse_decision(ScannerCacheReuseGate::NamespaceSampling, decision);
|
||||
match decision {
|
||||
ScannerCacheReuseDecision::Eligible => Some(true),
|
||||
ScannerCacheReuseDecision::SelectedForWalk => Some(false),
|
||||
_ => None,
|
||||
}
|
||||
self.old_cache.info.scan_progress.is_some()
|
||||
&& !compacted
|
||||
&& self
|
||||
.resume_frontier
|
||||
.as_deref()
|
||||
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
|
||||
&& self.old_cache.find(name).is_some()
|
||||
&& !self.resume_subtree_has_failures(name)
|
||||
}
|
||||
|
||||
fn resume_subtree_has_failures(&self, folder: &str) -> bool {
|
||||
@@ -2137,11 +2077,7 @@ impl FolderScanner {
|
||||
// The caller stitches this cached child into its
|
||||
// parent; do not clone a growing child set into the
|
||||
// unused provisional accumulator as well.
|
||||
let started = Instant::now();
|
||||
self.new_cache.copy_with_children(&self.old_cache, &this_hash, &folder.parent);
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_cache_reuse(ScannerCacheReuseKind::ClassifiedNamespace, started.elapsed());
|
||||
self.budget.record_entries_visited(pending_entry_progress);
|
||||
self.finish_raw_enumeration_parent(&folder.name);
|
||||
done_folder();
|
||||
@@ -2728,12 +2664,8 @@ impl FolderScanner {
|
||||
FolderScanMode::Namespace
|
||||
};
|
||||
if child_mode == FolderScanMode::Namespace && reuse_covered {
|
||||
let started = Instant::now();
|
||||
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
|
||||
into.add_child(&h);
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_cache_reuse(ScannerCacheReuseKind::CoveredNamespace, started.elapsed());
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -2773,15 +2705,22 @@ impl FolderScanner {
|
||||
}
|
||||
FolderScanSource::Existing => {
|
||||
// Usage sampling is not proof that a Deep check ran.
|
||||
if let Some(reuse) = self.sampled_namespace_reuse(&h, into.compacted, forward_sweep) {
|
||||
if reuse {
|
||||
if self.scan_mode != HealScanMode::Deep
|
||||
&& !forward_sweep
|
||||
&& !self
|
||||
.old_cache
|
||||
.info
|
||||
.replication
|
||||
.as_ref()
|
||||
.is_some_and(|config| config.has_active_rules("", true))
|
||||
&& !into.compacted
|
||||
&& self.old_cache.is_compacted(&h)
|
||||
{
|
||||
let next_cycle = self.old_cache.info.next_cycle as u32;
|
||||
if !h.mod_(next_cycle, data_usage_update_dir_cycles()) {
|
||||
if child_mode == FolderScanMode::Namespace {
|
||||
let started = Instant::now();
|
||||
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
|
||||
into.add_child(&h);
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_cache_reuse(ScannerCacheReuseKind::SampledNamespace, started.elapsed());
|
||||
self.record_scan_resume_hint(&folder_item.name);
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -3103,112 +3103,6 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
|
||||
assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Directories));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn cache_reuse_gates_report_only_the_first_production_condition() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir);
|
||||
scanner.old_cache.info.name = "bucket".to_string();
|
||||
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
|
||||
scanner.old_cache.replace(
|
||||
"bucket/child",
|
||||
"bucket",
|
||||
DataUsageEntry {
|
||||
compacted: true,
|
||||
objects: 2,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let hash = hash_path("bucket/child");
|
||||
let folder = CachedFolder {
|
||||
name: "bucket".to_string(),
|
||||
parent: None,
|
||||
object_heal_prob_div: 1,
|
||||
};
|
||||
let child = CachedFolder {
|
||||
name: "bucket/child".to_string(),
|
||||
parent: Some(hash_path("bucket")),
|
||||
object_heal_prob_div: 1,
|
||||
};
|
||||
let abandoned = HashSet::from([hash.0.clone()]);
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
|
||||
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
|
||||
let missing_scope = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(
|
||||
missing_scope.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"],
|
||||
before.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"] + 1
|
||||
);
|
||||
scanner.prefix_scan_scope = ScannerBucketPrefixScanScope::from_dirty_top_level_entries(HashSet::from(["dirty".to_string()]));
|
||||
scanner.is_erasure_mode = true;
|
||||
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
|
||||
let erasure = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(
|
||||
erasure.reuse_decisions["clean_prefix"].blocked["erasure_health_required"],
|
||||
before.reuse_decisions["clean_prefix"].blocked["erasure_health_required"] + 1
|
||||
);
|
||||
assert_eq!(erasure.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
|
||||
|
||||
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
|
||||
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
|
||||
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
|
||||
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
|
||||
});
|
||||
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
|
||||
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
|
||||
scanner.resume_frontier = Some("bucket/child".to_string());
|
||||
scanner.resume_failed_paths = vec!["bucket/child/failed".to_string()];
|
||||
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
|
||||
scanner.resume_failed_paths.clear();
|
||||
assert!(scanner.can_reuse_covered_namespace("bucket/child", false));
|
||||
let covered = global_metrics().scan_observation().snapshot();
|
||||
let reasons = &covered.reuse_decisions["covered_namespace"];
|
||||
let prior = &before.reuse_decisions["covered_namespace"];
|
||||
assert_eq!(reasons.not_evaluated["no_resume_sweep"], prior.not_evaluated["no_resume_sweep"] + 1);
|
||||
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
|
||||
assert_eq!(reasons.blocked["missing_coverage"], prior.blocked["missing_coverage"] + 1);
|
||||
assert_eq!(reasons.blocked["cached_failures"], prior.blocked["cached_failures"] + 1);
|
||||
assert_eq!(reasons.eligible, prior.eligible + 1);
|
||||
|
||||
with_var(ENV_DATA_USAGE_UPDATE_DIR_CYCLES, Some("16"), || {
|
||||
scanner.scan_mode = HealScanMode::Deep;
|
||||
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
|
||||
scanner.scan_mode = HealScanMode::Normal;
|
||||
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
|
||||
let replication: ReplicationConfig = serde_json::from_value(serde_json::json!({
|
||||
"config": { "role": "", "rules": [{ "destination": { "bucket": "arn:test" }, "id": "status", "prefix": "", "priority": 1, "status": "Enabled" }] },
|
||||
"remotes": crate::storage_api::EcstoreBucketTargets::default(),
|
||||
})).expect("active replication fixture");
|
||||
scanner.old_cache.info.replication = Some(Arc::new(replication));
|
||||
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
|
||||
scanner.old_cache.info.replication = None;
|
||||
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
|
||||
let rescan_cycle = (0..16)
|
||||
.find(|cycle| hash.mod_(*cycle, 16))
|
||||
.expect("hash is visited in one sampling cycle");
|
||||
scanner.old_cache.info.next_cycle = u64::from(rescan_cycle);
|
||||
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(false));
|
||||
scanner.old_cache.info.next_cycle = u64::from((rescan_cycle + 1) % 16);
|
||||
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(true));
|
||||
});
|
||||
let after = global_metrics().scan_observation().snapshot();
|
||||
let reasons = &after.reuse_decisions["namespace_sampling"];
|
||||
let prior = &before.reuse_decisions["namespace_sampling"];
|
||||
for label in ["deep_scan", "forward_sweep", "active_replication"] {
|
||||
assert_eq!(
|
||||
reasons.not_evaluated[label],
|
||||
prior.not_evaluated[label] + 1,
|
||||
"first sampling condition {label}"
|
||||
);
|
||||
}
|
||||
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
|
||||
assert_eq!(reasons.blocked["selected_for_walk"], prior.blocked["selected_for_walk"] + 1);
|
||||
assert_eq!(reasons.eligible, prior.eligible + 1);
|
||||
for (kind, reuse) in &after.cache_reuse {
|
||||
assert_eq!(reuse.count, before.cache_reuse[kind].count, "gate admission alone does not copy {kind}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entries() {
|
||||
@@ -3250,21 +3144,12 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
|
||||
parent: None,
|
||||
object_heal_prob_div: 1,
|
||||
};
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
let mut root = DataUsageEntry::default();
|
||||
scanner
|
||||
.scan_folder(CancellationToken::new(), folder, &mut root)
|
||||
.await
|
||||
.expect("scoped root scan should finish successfully");
|
||||
|
||||
let observed = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count + 1);
|
||||
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks);
|
||||
assert_eq!(
|
||||
observed.reuse_decisions["clean_prefix"].eligible,
|
||||
before.reuse_decisions["clean_prefix"].eligible + 1
|
||||
);
|
||||
|
||||
let clean = scanner
|
||||
.new_cache
|
||||
.size_recursive("bucket/clean")
|
||||
@@ -3282,78 +3167,6 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
|
||||
assert_eq!((bucket.size, bucket.objects), (17, 3));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn scoped_cache_observations_distinguish_checkpoint_and_periodic_sampling() {
|
||||
for covered in [true, false] {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
|
||||
tokio::fs::create_dir_all(temp_dir.join("bucket/child"))
|
||||
.await
|
||||
.expect("create cached namespace");
|
||||
scanner.old_cache.info.name = "bucket".to_string();
|
||||
scanner.new_cache.info.name = "bucket".to_string();
|
||||
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
|
||||
scanner.old_cache.replace(
|
||||
"bucket/child",
|
||||
"bucket",
|
||||
DataUsageEntry {
|
||||
size: 17,
|
||||
objects: 5,
|
||||
compacted: true,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let kind = if covered {
|
||||
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
|
||||
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
|
||||
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
|
||||
});
|
||||
scanner.resume_frontier = Some("bucket/child".to_string());
|
||||
"covered_namespace"
|
||||
} else {
|
||||
let hash = hash_path("bucket/child");
|
||||
let cycle = (0..data_usage_update_dir_cycles())
|
||||
.find(|cycle| !hash.mod_(*cycle, data_usage_update_dir_cycles()))
|
||||
.expect("a cycle samples other work");
|
||||
scanner.old_cache.info.next_cycle = u64::from(cycle);
|
||||
"sampled_namespace"
|
||||
};
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
let mut root = DataUsageEntry::default();
|
||||
scanner
|
||||
.scan_folder(
|
||||
CancellationToken::new(),
|
||||
CachedFolder {
|
||||
name: "bucket".to_string(),
|
||||
parent: None,
|
||||
object_heal_prob_div: 1,
|
||||
},
|
||||
&mut root,
|
||||
)
|
||||
.await
|
||||
.expect("cached namespace reuses accounting");
|
||||
let summary = scanner.new_cache.size_recursive("bucket").expect("complete bucket totals");
|
||||
assert_eq!((summary.size, summary.objects), (17, 5));
|
||||
let after = global_metrics().scan_observation().snapshot();
|
||||
for label in [
|
||||
"clean_prefix",
|
||||
"covered_namespace",
|
||||
"sampled_namespace",
|
||||
"classified_namespace",
|
||||
] {
|
||||
assert_eq!(
|
||||
after.cache_reuse[label].count,
|
||||
before.cache_reuse[label].count + u64::from(label == kind),
|
||||
"reuse reason {label}"
|
||||
);
|
||||
}
|
||||
assert_eq!(after.metadata_reads.attempts, before.metadata_reads.attempts);
|
||||
let gate = if covered { "covered_namespace" } else { "namespace_sampling" };
|
||||
assert_eq!(after.reuse_decisions[gate].eligible, before.reuse_decisions[gate].eligible + 1);
|
||||
}
|
||||
}
|
||||
|
||||
async fn scan_hot_cold_segment_fixture(scoped: bool) -> (DataUsageEntry, Vec<String>) {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let visits = metadata_visits::observe(&mut scanner);
|
||||
@@ -3465,7 +3278,6 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
|
||||
parent: None,
|
||||
object_heal_prob_div: 1,
|
||||
};
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
let mut root = DataUsageEntry::default();
|
||||
scanner
|
||||
.scan_folder(CancellationToken::new(), folder, &mut root)
|
||||
@@ -3477,9 +3289,6 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
|
||||
.size_recursive("bucket/clean")
|
||||
.expect("erasure scan should visit the clean entry");
|
||||
assert_eq!((clean.size, clean.objects), (0, 0));
|
||||
let observed = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
|
||||
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -623,7 +623,7 @@ async fn checkpoint_fixture_existing_uncovered_cursor_cannot_skip_to_complete()
|
||||
cache.info.scan_coverage_receipt.as_mut().expect("receipt").through = "bucket/z-stale".into();
|
||||
}
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist corrupted existing cursor");
|
||||
let mut loaded = store.strict_load().await;
|
||||
@@ -663,7 +663,7 @@ async fn checkpoint_fixture_existing_uncovered_cursor_cannot_skip_to_complete()
|
||||
assert_eq!(cache.checked_flatten("bucket/b-pending").expect("revisited prefix").size, 7);
|
||||
assert!(!cache.info.snapshot_complete);
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist verified partial");
|
||||
assert!(!store.strict_load().await.info.snapshot_complete);
|
||||
@@ -827,7 +827,7 @@ async fn check_complete_sampling_resumption(resume_mode: HealScanMode) {
|
||||
.await
|
||||
.expect("fixture revision");
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist sampling partial");
|
||||
if resume_mode == HealScanMode::Deep {
|
||||
@@ -876,7 +876,7 @@ async fn check_complete_sampling_resumption(resume_mode: HealScanMode) {
|
||||
assert_eq!(cache.find("bucket/prefix/0000").expect("Deep revisits earlier prefix").size, 7);
|
||||
}
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("save bounded recovery");
|
||||
if complete {
|
||||
@@ -964,7 +964,6 @@ async fn run_hot_bucket_certification(advance_epoch: bool, logs: &CapturedLogs)
|
||||
tier_registry_generation: Some(identity.tier_registry_generation),
|
||||
checkpoint_identity: Some(identity),
|
||||
checkpoint_plan_digest: None,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
},
|
||||
);
|
||||
cache.info.skip_healing = true;
|
||||
@@ -996,7 +995,7 @@ async fn run_hot_bucket_certification(advance_epoch: bool, logs: &CapturedLogs)
|
||||
};
|
||||
assert_eq!(budget.budget_elapsed(), max_objects.is_some(), "round {round} budget");
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("save hot bucket sweep");
|
||||
let saved = store.strict_load().await;
|
||||
@@ -1210,7 +1209,7 @@ pub(super) async fn scan_budgeted_metadata_repair_tail() -> DataUsageCache {
|
||||
assert!(!cache.info.skip_healing);
|
||||
assert!(!cache.info.snapshot_complete);
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist scanner repair obligation with CAS");
|
||||
let saved = store.strict_load().await;
|
||||
@@ -1283,7 +1282,6 @@ where
|
||||
tier_registry_generation: None,
|
||||
checkpoint_identity: Some(identity),
|
||||
checkpoint_plan_digest: None,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
},
|
||||
);
|
||||
let prepared = retained(&cache);
|
||||
@@ -1320,7 +1318,7 @@ where
|
||||
assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects));
|
||||
let scanned = retained(&cache);
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist partial checkpoint");
|
||||
let mut loaded = DataUsageCache::default();
|
||||
@@ -1359,14 +1357,14 @@ where
|
||||
.await
|
||||
.expect("read durable checkpoint bytes");
|
||||
let epoch_error = loaded
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 1, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 1)
|
||||
.await
|
||||
.expect_err("stale publication epoch must reject persistence");
|
||||
assert!(epoch_error.to_string().contains(crate::SCANNER_PUBLICATION_EPOCH_CHANGED));
|
||||
store.reject_save.store(true, Ordering::SeqCst);
|
||||
loaded.info.next_cycle += 1;
|
||||
loaded
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect_err("injected save failure must not report durable progress");
|
||||
assert_eq!(
|
||||
@@ -1424,7 +1422,6 @@ where
|
||||
tier_registry_generation: Some(identity.tier_registry_generation),
|
||||
checkpoint_identity: Some(identity),
|
||||
checkpoint_plan_digest: None,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
},
|
||||
);
|
||||
cache.info.skip_healing = true;
|
||||
@@ -1455,7 +1452,7 @@ where
|
||||
ScannerDiskScanOutcome::NamespaceNotFound(_) => panic!("fixture namespace exists"),
|
||||
};
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("save each bounded sweep");
|
||||
let saved = store.strict_load().await;
|
||||
|
||||
@@ -134,7 +134,7 @@ async fn assert_completed_compaction_resumes(mode: HealScanMode) {
|
||||
.await
|
||||
.expect("initial fixture revisions");
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist compacted partial cache through the production codec and CAS path");
|
||||
let mut reloaded = store.strict_load().await;
|
||||
|
||||
@@ -42,7 +42,7 @@ async fn save_reload(store: &Arc<FixtureStore>, cache: &DataUsageCache) -> DataU
|
||||
.await
|
||||
.expect("fixture save revisions");
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("save compacted checkpoint through real codec and revision checks");
|
||||
let loaded = store.strict_load().await;
|
||||
|
||||
@@ -150,7 +150,7 @@ async fn nested_periodic_snapshot(
|
||||
.await
|
||||
.expect("initial periodic fixture revisions");
|
||||
snapshot
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist real mid-walk periodic snapshot");
|
||||
let mut loaded = DataUsageCache::default();
|
||||
@@ -223,7 +223,7 @@ async fn periodic_mixed_tail_requires_terminal_persistence_before_fresh_certific
|
||||
.await
|
||||
.expect("initial periodic fixture revisions");
|
||||
periodic
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("seed durable periodic snapshot");
|
||||
|
||||
@@ -281,7 +281,7 @@ async fn periodic_mixed_tail_requires_terminal_persistence_before_fresh_certific
|
||||
|
||||
store.reject_save.store(round == 0, Ordering::SeqCst);
|
||||
let saved = terminal
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await;
|
||||
if round == 0 {
|
||||
assert!(saved.is_err(), "injected terminal persistence failure must be observable");
|
||||
@@ -369,7 +369,7 @@ async fn periodic_nested_snapshot_keeps_durable_frontier_across_early_repeated_h
|
||||
.await
|
||||
.expect("initial repeated-handoff revisions");
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("save the inherited valid subtree frontier");
|
||||
let (mut scanner, unused_root) = build_test_scanner().await;
|
||||
@@ -442,7 +442,7 @@ async fn periodic_nested_snapshot_keeps_durable_frontier_across_early_repeated_h
|
||||
"inherited completed object and both newly read objects must survive"
|
||||
);
|
||||
candidate
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist later coverage without losing the inherited prefix");
|
||||
let mut latest = DataUsageCache::default();
|
||||
@@ -958,7 +958,7 @@ async fn periodic_failed_subtree_resumes_healthy_descendants_and_compacted_debt(
|
||||
.await
|
||||
.expect("fresh revisions");
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist failure-bearing coverage");
|
||||
let mut adopted = DataUsageCache::default();
|
||||
@@ -1220,7 +1220,7 @@ async fn checkpoint_debt_restart_round(request: &CheckpointDebtRequest) -> serde
|
||||
let expected = checkpoint_debt_observation(&cache, state.identity, frontier, failed_objects);
|
||||
assert_eq!(cache.info.leader_epoch, 8 + u64::from(request.round));
|
||||
cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("persist combined healthy coverage and failure debt with CAS");
|
||||
let saved = store.strict_load().await;
|
||||
|
||||
@@ -303,7 +303,7 @@ async fn walk_and_save(observe: bool) -> (Vec<String>, serde_json::Value) {
|
||||
.expect("read empty fixture revisions");
|
||||
scanner
|
||||
.new_cache
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0, CancellationToken::new())
|
||||
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
|
||||
.await
|
||||
.expect("save actual walker output through the cache codec and revision gate");
|
||||
let loaded = store.strict_load().await;
|
||||
|
||||
@@ -774,7 +774,6 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
|
||||
},
|
||||
);
|
||||
scanner.budget = budget.clone();
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
scanner
|
||||
.scan_folder(
|
||||
budget.token(),
|
||||
@@ -791,15 +790,6 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
|
||||
assert_eq!(budget.progress(), (0, 1), "only the parent consumes namespace quota");
|
||||
assert_eq!(charged_ops(&root), 4, "list both directories once without re-reading covered metadata");
|
||||
assert_eq!(scanner.new_cache.find(&folder.name).expect("reused candidate").objects, 1);
|
||||
let observed = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(
|
||||
observed.cache_reuse["classified_namespace"].count,
|
||||
before.cache_reuse["classified_namespace"].count + 1
|
||||
);
|
||||
assert_eq!(
|
||||
observed.metadata_reads.attempts, before.metadata_reads.attempts,
|
||||
"classification retains the existing zero metadata read path"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -879,7 +869,6 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
|
||||
},
|
||||
);
|
||||
scanner.budget = budget.clone();
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
scanner
|
||||
.scan_folder(
|
||||
budget.token(),
|
||||
@@ -900,15 +889,6 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
|
||||
"sampling still requires full current classification, without metadata re-read"
|
||||
);
|
||||
assert_eq!(scanner.new_cache.find(&folder.name).expect("sampled candidate").objects, 1);
|
||||
let observed = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(
|
||||
observed.cache_reuse["classified_namespace"].count,
|
||||
before.cache_reuse["classified_namespace"].count + 1
|
||||
);
|
||||
assert_eq!(
|
||||
observed.metadata_reads.attempts, before.metadata_reads.attempts,
|
||||
"classification retains the existing zero metadata read path"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -40,7 +40,6 @@ use rustfs_lock::{LockError, NamespaceLockGuard};
|
||||
use rustfs_scanner_metrics::metrics::{
|
||||
Metric, Metrics, emit_scan_bucket_drive_complete, emit_scan_bucket_drive_partial, global_metrics,
|
||||
};
|
||||
use rustfs_scanner_metrics::scan_observation::{PREFIX_REUSE_BLOCKERS, ScannerBaselineRejection, ScannerPrefixReuseStage};
|
||||
use rustfs_utils::path::path_join_buf;
|
||||
use sha2::{Digest as _, Sha256};
|
||||
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
|
||||
@@ -113,7 +112,16 @@ pub(crate) const SCANNER_SEGMENT_ACTIVATION_PROOF_INPUTS: [&str; 9] = [
|
||||
"durable_dirty_producer_journal",
|
||||
"ack_generation_guard",
|
||||
];
|
||||
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = PREFIX_REUSE_BLOCKERS;
|
||||
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = [
|
||||
"missing_producer_identity",
|
||||
"missing_durable_journal_replay",
|
||||
"restart_gap",
|
||||
"generation_gap",
|
||||
"overflow",
|
||||
"stale_ack_generation",
|
||||
"missing_cold_zero_walk_oracle",
|
||||
"distributed_without_peer_invalidation",
|
||||
];
|
||||
|
||||
pub type DirtyUsageBuckets = HashMap<String, u64>;
|
||||
|
||||
@@ -189,7 +197,6 @@ struct VerifiedRemoteDirtyUsage {
|
||||
}
|
||||
|
||||
struct ScannerBucketScopeResolutionResult {
|
||||
segment_reuse_activation_preflight: Option<ScannerSegmentReuseActivationPreflight>,
|
||||
scope: ScannerBucketScanScope,
|
||||
remote_dirty_usage_acknowledgements: Vec<crate::scanner::ScannerDirtyUsageAcknowledgement>,
|
||||
distributed_segment_invalidation_evidence: Option<DistributedSegmentInvalidationEvidence>,
|
||||
@@ -229,13 +236,6 @@ pub(crate) struct ScannerSegmentReuseActivationPreflight {
|
||||
}
|
||||
|
||||
impl ScannerSegmentReuseActivationPreflight {
|
||||
fn record_observation(&self, stage: ScannerPrefixReuseStage) {
|
||||
global_metrics().scan_observation().record_prefix_reuse(
|
||||
stage,
|
||||
self.scanner_segment_reuse_activated,
|
||||
&self.fail_closed_blockers,
|
||||
);
|
||||
}
|
||||
#[cfg(test)]
|
||||
pub(crate) fn fail_closed_blockers(&self) -> impl Iterator<Item = &'static str> + '_ {
|
||||
self.fail_closed_blockers.iter().filter_map(|blocker| *blocker)
|
||||
@@ -319,7 +319,6 @@ fn resolve_remote_dirty_usage_scope(
|
||||
baseline_proof: ScannerCacheBaselineProof<'_>,
|
||||
) -> ScannerBucketScopeResolutionResult {
|
||||
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
|
||||
segment_reuse_activation_preflight: None,
|
||||
scope,
|
||||
remote_dirty_usage_acknowledgements: Vec::new(),
|
||||
distributed_segment_invalidation_evidence: None,
|
||||
@@ -411,7 +410,6 @@ fn resolve_remote_dirty_usage_scope(
|
||||
};
|
||||
|
||||
ScannerBucketScopeResolutionResult {
|
||||
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
|
||||
scope,
|
||||
remote_dirty_usage_acknowledgements: scoped_acknowledgements,
|
||||
distributed_segment_invalidation_evidence,
|
||||
@@ -429,81 +427,43 @@ fn complete_scanner_cache_snapshot_plan_digest(
|
||||
proof: ScannerCacheBaselineProof<'_>,
|
||||
expected_converged: Option<bool>,
|
||||
) -> Option<DataUsageScanPlanDigest> {
|
||||
validate_scanner_cache_snapshot_plan_digest(snapshot, proof, expected_converged).ok()
|
||||
}
|
||||
|
||||
fn validate_scanner_cache_snapshot_plan_digest(
|
||||
snapshot: &DataUsageInfo,
|
||||
proof: ScannerCacheBaselineProof<'_>,
|
||||
expected_converged: Option<bool>,
|
||||
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineRejection> {
|
||||
// The observational mark disqualifies only the authoritative slot: an
|
||||
// observed candidate carries it by construction.
|
||||
let marks_accepted = match expected_converged {
|
||||
Some(expected) => snapshot.usage_snapshot_converged == Some(expected),
|
||||
None => snapshot.usage_snapshot_converged.is_some() && !snapshot.usage_snapshot_observational,
|
||||
};
|
||||
if !snapshot.is_complete_bucket_usage_snapshot() {
|
||||
return Err(ScannerBaselineRejection::Incomplete);
|
||||
}
|
||||
if snapshot.usage_snapshot_partial {
|
||||
return Err(ScannerBaselineRejection::Partial);
|
||||
}
|
||||
if !marks_accepted {
|
||||
return Err(ScannerBaselineRejection::Marks);
|
||||
}
|
||||
if snapshot.scanner_epoch != Some(proof.leader_epoch) {
|
||||
return Err(ScannerBaselineRejection::Epoch);
|
||||
}
|
||||
if snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len() {
|
||||
return Err(ScannerBaselineRejection::SourceCount);
|
||||
if !snapshot.is_complete_bucket_usage_snapshot()
|
||||
|| snapshot.usage_snapshot_partial
|
||||
|| !marks_accepted
|
||||
|| snapshot.scanner_epoch != Some(proof.leader_epoch)
|
||||
|| snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len()
|
||||
{
|
||||
return None;
|
||||
}
|
||||
|
||||
// Completed maintenance also covers ordinary usage. Keep its exact stored
|
||||
// proof for cache reuse, and reject mixtures of different set work proofs.
|
||||
let baseline_plan_digest = DataUsageScanPlanDigest(
|
||||
snapshot
|
||||
.usage_snapshot_set_states
|
||||
.first()
|
||||
.and_then(|state| state.scan_plan_digest)
|
||||
.ok_or(ScannerBaselineRejection::PlanMissing)?,
|
||||
);
|
||||
let baseline_plan_digest = DataUsageScanPlanDigest(snapshot.usage_snapshot_set_states.first()?.scan_plan_digest?);
|
||||
if !scanner_plan_digest_is_work_variant(proof.scan_plan_digest, baseline_plan_digest) {
|
||||
return Err(ScannerBaselineRejection::PlanMismatch);
|
||||
return None;
|
||||
}
|
||||
let mut states = HashSet::with_capacity(snapshot.usage_snapshot_set_states.len());
|
||||
for state in &snapshot.usage_snapshot_set_states {
|
||||
let source = DataUsageCacheSource::new(
|
||||
usize::try_from(state.pool_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
|
||||
usize::try_from(state.set_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
|
||||
);
|
||||
if !proof.expected_sources.contains(&source) {
|
||||
return Err(ScannerBaselineRejection::SourceMembership);
|
||||
}
|
||||
if !states.insert(source) {
|
||||
return Err(ScannerBaselineRejection::SourceDuplicate);
|
||||
}
|
||||
if !state.complete {
|
||||
return Err(ScannerBaselineRejection::SetIncomplete);
|
||||
}
|
||||
if state.tombstone {
|
||||
return Err(ScannerBaselineRejection::SetTombstone);
|
||||
}
|
||||
if state.scanner_epoch != Some(proof.leader_epoch) {
|
||||
return Err(ScannerBaselineRejection::SetEpoch);
|
||||
}
|
||||
if state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle) {
|
||||
return Err(ScannerBaselineRejection::SetCycle);
|
||||
}
|
||||
if state.scan_plan_digest != Some(baseline_plan_digest.0) {
|
||||
return Err(ScannerBaselineRejection::SetPlan);
|
||||
let source = DataUsageCacheSource::new(usize::try_from(state.pool_index).ok()?, usize::try_from(state.set_index).ok()?);
|
||||
if !proof.expected_sources.contains(&source)
|
||||
|| !states.insert(source)
|
||||
|| !state.complete
|
||||
|| state.tombstone
|
||||
|| state.scanner_epoch != Some(proof.leader_epoch)
|
||||
|| state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle)
|
||||
|| state.scan_plan_digest != Some(baseline_plan_digest.0)
|
||||
{
|
||||
return None;
|
||||
}
|
||||
}
|
||||
|
||||
if states != *proof.expected_sources {
|
||||
return Err(ScannerBaselineRejection::SourceCoverage);
|
||||
}
|
||||
Ok(baseline_plan_digest)
|
||||
(states == *proof.expected_sources).then_some(baseline_plan_digest)
|
||||
}
|
||||
|
||||
/// Whether `digest` is `structural` under any work flag: the same bucket
|
||||
@@ -533,35 +493,10 @@ fn scanner_baseline_pending_buckets(proof: &ScannerCacheBaselineProof<'_>) -> Ve
|
||||
}
|
||||
|
||||
fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<'_>) -> Option<DataUsageScanPlanDigest> {
|
||||
validate_scanner_cache_baseline_plan_digest(proof).ok()
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
struct ScannerBaselineValidationFailure {
|
||||
authoritative: ScannerBaselineRejection,
|
||||
observed: Option<ScannerBaselineRejection>,
|
||||
}
|
||||
|
||||
fn validate_scanner_cache_baseline_plan_digest(
|
||||
proof: ScannerCacheBaselineProof<'_>,
|
||||
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineValidationFailure> {
|
||||
let authoritative_error = |authoritative| ScannerBaselineValidationFailure {
|
||||
authoritative,
|
||||
observed: None,
|
||||
};
|
||||
let authoritative_data = proof
|
||||
.authoritative_data
|
||||
.ok_or_else(|| authoritative_error(ScannerBaselineRejection::Missing))?;
|
||||
let authoritative = serde_json::from_slice::<DataUsageInfo>(authoritative_data)
|
||||
.map_err(|_| authoritative_error(ScannerBaselineRejection::Decode))?;
|
||||
let authoritative_rejection = match validate_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
|
||||
Ok(validated_digest) => return Ok(validated_digest),
|
||||
Err(reason) => reason,
|
||||
};
|
||||
let observed_error = |observed| ScannerBaselineValidationFailure {
|
||||
authoritative: authoritative_rejection,
|
||||
observed: Some(observed),
|
||||
};
|
||||
let authoritative = serde_json::from_slice::<DataUsageInfo>(proof.authoritative_data?).ok()?;
|
||||
if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
|
||||
return Some(validated_digest);
|
||||
}
|
||||
|
||||
// A complete but superseded observation may reuse its per-set cache only
|
||||
// when it was explicitly tied to the durable authoritative baseline. It
|
||||
@@ -571,18 +506,14 @@ fn validate_scanner_cache_baseline_plan_digest(
|
||||
&& !authoritative.usage_snapshot_observational)
|
||||
|| crate::scanner::data_usage_info_is_bootstrap_pending(&authoritative);
|
||||
if !authoritative_has_identity {
|
||||
return Err(observed_error(ScannerBaselineRejection::AuthoritativeIdentity));
|
||||
return None;
|
||||
}
|
||||
let observed_data = proof
|
||||
.observed_candidate_data
|
||||
.ok_or_else(|| observed_error(ScannerBaselineRejection::Missing))?;
|
||||
let observed =
|
||||
serde_json::from_slice::<DataUsageInfo>(observed_data).map_err(|_| observed_error(ScannerBaselineRejection::Decode))?;
|
||||
let observed = serde_json::from_slice::<DataUsageInfo>(proof.observed_candidate_data?).ok()?;
|
||||
if !observed_data_usage_is_newer(&observed, &authoritative) {
|
||||
return Err(observed_error(ScannerBaselineRejection::NotNewer));
|
||||
return None;
|
||||
}
|
||||
|
||||
validate_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false)).map_err(observed_error)
|
||||
complete_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false))
|
||||
}
|
||||
|
||||
fn scanner_segment_reuse_baseline_producer_evidence(
|
||||
@@ -727,12 +658,6 @@ impl ScannerScopeFallbackReason {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
struct ScannerScopeFallback {
|
||||
reason: ScannerScopeFallbackReason,
|
||||
baseline_rejection: Option<ScannerBaselineValidationFailure>,
|
||||
}
|
||||
|
||||
/// Classify a resolved full-namespace scope against the preconditions
|
||||
/// `resolve_scanner_bucket_scan_scope` checks, in the same order. Returns
|
||||
/// `None` when the cycle was required to walk everything or was scoped.
|
||||
@@ -743,24 +668,18 @@ fn scanner_scope_fallback_reason(
|
||||
all_buckets: &[BucketInfo],
|
||||
baseline_proof: ScannerCacheBaselineProof<'_>,
|
||||
distributed: bool,
|
||||
) -> Option<ScannerScopeFallback> {
|
||||
let fallback = |reason| {
|
||||
Some(ScannerScopeFallback {
|
||||
reason,
|
||||
baseline_rejection: None,
|
||||
})
|
||||
};
|
||||
) -> Option<ScannerScopeFallbackReason> {
|
||||
if !scope_eligible || !resolved_scope.is_default() {
|
||||
return None;
|
||||
}
|
||||
if dirty_usage_snapshot.generation == u64::MAX {
|
||||
return fallback(ScannerScopeFallbackReason::DirtyGenerationExhausted);
|
||||
return Some(ScannerScopeFallbackReason::DirtyGenerationExhausted);
|
||||
}
|
||||
if !dirty_usage_snapshot.covers_listed_pending {
|
||||
return fallback(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
|
||||
return Some(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
|
||||
}
|
||||
if dirty_usage_snapshot.buckets.len() > crate::SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES {
|
||||
return fallback(ScannerScopeFallbackReason::DirtyBucketOverflow);
|
||||
return Some(ScannerScopeFallbackReason::DirtyBucketOverflow);
|
||||
}
|
||||
// Peers may contribute dirty buckets the local snapshot does not hold.
|
||||
if !distributed {
|
||||
@@ -770,19 +689,13 @@ fn scanner_scope_fallback_reason(
|
||||
.keys()
|
||||
.any(|bucket| listed.contains(bucket.as_str()))
|
||||
{
|
||||
return fallback(ScannerScopeFallbackReason::NoListedDirtyBuckets);
|
||||
return Some(ScannerScopeFallbackReason::NoListedDirtyBuckets);
|
||||
}
|
||||
}
|
||||
if let Err(failure) = validate_scanner_cache_baseline_plan_digest(baseline_proof) {
|
||||
return Some(ScannerScopeFallback {
|
||||
reason: ScannerScopeFallbackReason::BaselineUnavailable,
|
||||
baseline_rejection: Some(failure),
|
||||
});
|
||||
if complete_scanner_cache_baseline_plan_digest(baseline_proof).is_none() {
|
||||
return Some(ScannerScopeFallbackReason::BaselineUnavailable);
|
||||
}
|
||||
distributed.then_some(ScannerScopeFallback {
|
||||
reason: ScannerScopeFallbackReason::PeerScopeUnavailable,
|
||||
baseline_rejection: None,
|
||||
})
|
||||
distributed.then_some(ScannerScopeFallbackReason::PeerScopeUnavailable)
|
||||
}
|
||||
|
||||
fn scanner_segment_reuse_activation_preflight() -> ScannerSegmentReuseActivationPreflight {
|
||||
@@ -1018,7 +931,6 @@ pub struct ScannerBucketScanPlan {
|
||||
/// coordinator restart or a maintenance publication does not turn a
|
||||
/// resumed sweep into mixed coverage.
|
||||
checkpoint_coverage_digest: DataUsageScanPlanDigest,
|
||||
completed_checkpoint_cycle_floor: Option<u64>,
|
||||
requires_full_scan: bool,
|
||||
service_cohort: Option<Arc<StdMutex<ScannerServiceCohort>>>,
|
||||
// Cache work must invalidate on namespace completion even when its scoped baseline remains reusable.
|
||||
@@ -2233,8 +2145,8 @@ use guards::*;
|
||||
|
||||
pub(crate) use cache::{
|
||||
DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerCheckpointPersistContext, ScannerCheckpointPersistResult,
|
||||
acquire_scanner_cache_locks, completed_checkpoint_cycle_floor, current_cache_root_or_prepare_with_generation,
|
||||
log_saved_partial_bucket_cache, persist_scanner_checkpoint,
|
||||
acquire_scanner_cache_locks, current_cache_root_or_prepare_with_generation, log_saved_partial_bucket_cache,
|
||||
persist_scanner_checkpoint,
|
||||
};
|
||||
pub use dirty_usage::{
|
||||
ScannerDirtyUsageAckError, ScannerDirtyUsageBucket, ScannerDirtyUsageClearObserver, ScannerDirtyUsageMutationObserver,
|
||||
@@ -2244,10 +2156,8 @@ pub use dirty_usage::{
|
||||
probe_scoped_dirty_usage, record_dirty_usage_bucket, record_dirty_usage_bucket_from_producer,
|
||||
record_dirty_usage_bucket_from_producers, record_dirty_usage_object, record_dirty_usage_object_from_producer,
|
||||
record_scanner_maintenance_change, record_status_class_mutation, replay_durable_dirty_usage_producer_record,
|
||||
restore_scanner_checkpoint_instance_id, scanner_activity_epoch, scanner_checkpoint_instance_id,
|
||||
scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state,
|
||||
scanner_maintenance_generation, scanner_process_was_read_only, set_scanner_dirty_usage_clear_observer,
|
||||
set_scanner_dirty_usage_mutation_observer,
|
||||
scanner_activity_epoch, scanner_dirty_usage_bucket_generation, scanner_dirty_usage_snapshot, scanner_dirty_usage_state,
|
||||
scanner_maintenance_generation, set_scanner_dirty_usage_clear_observer, set_scanner_dirty_usage_mutation_observer,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use dirty_usage::{clear_dirty_usage_buckets_for_tests, dirty_usage_buckets_for_tests};
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
// limitations under the License.
|
||||
use super::*;
|
||||
/// scanner cache locks and the cache snapshot persist/publish path.
|
||||
use crate::data_usage_define::{DataUsageCacheBaselineRevisions, clone_cache_value};
|
||||
use crate::data_usage_define::clone_cache_value;
|
||||
use crate::{ScannerConfigObjectDelete, ScannerObjectIO};
|
||||
use rustfs_scanner_metrics::usage_candidate::{
|
||||
ScannerUsageCandidateKind as CandidateKind, ScannerUsageCandidateOutcome as CandidateOutcome,
|
||||
@@ -97,15 +97,11 @@ where
|
||||
return ScannerCheckpointPersistResult::FenceChanged;
|
||||
}
|
||||
|
||||
let checkpoint_save = checkpoint
|
||||
.save_with_revisions_for_epoch(
|
||||
store.clone(),
|
||||
cache_name,
|
||||
revisions,
|
||||
context.expected_publication_epoch,
|
||||
context.ctx.clone(),
|
||||
)
|
||||
.await;
|
||||
let checkpoint_save = crate::storage_api::ecstore_with_background_disk_io(
|
||||
context.ctx.clone(),
|
||||
checkpoint.save_with_revisions_for_epoch(store.clone(), cache_name, revisions, context.expected_publication_epoch),
|
||||
)
|
||||
.await;
|
||||
if let Err(error) = checkpoint_save {
|
||||
record_checkpoint_save_failure(cache_name, checkpoint, &error);
|
||||
return if matches!(error, StorageError::PreconditionFailed) {
|
||||
@@ -363,39 +359,6 @@ pub(crate) struct DataUsageCacheReuseOptions {
|
||||
/// (`scanner_activity_coverage_digest`); `None` resumes under the reuse
|
||||
/// plan itself for requests from older remote coordinators.
|
||||
pub(crate) checkpoint_plan_digest: Option<DataUsageScanPlanDigest>,
|
||||
/// The last authoritative aggregate's cycle. Later completed bucket
|
||||
/// sweeps are still checkpoints of an unpublished aggregate.
|
||||
pub(crate) completed_checkpoint_cycle_floor: Option<u64>,
|
||||
}
|
||||
|
||||
pub(crate) fn completed_checkpoint_cycle_floor(
|
||||
data: Option<&[u8]>,
|
||||
expected_sources: &HashSet<DataUsageCacheSource>,
|
||||
next_cycle: u64,
|
||||
leader_epoch: u64,
|
||||
) -> Option<u64> {
|
||||
let usage: DataUsageInfo = serde_json::from_slice(data?).ok()?;
|
||||
if usage.scanner_epoch.is_some_and(|epoch| epoch > leader_epoch) {
|
||||
return None;
|
||||
}
|
||||
if crate::scanner::data_usage_info_is_bootstrap_pending(&usage) {
|
||||
return Some(0);
|
||||
}
|
||||
let cycle = usage.scanner_cycle.filter(|cycle| *cycle < next_cycle)?;
|
||||
let plan = DataUsageScanPlanDigest(usage.usage_snapshot_set_states.first()?.scan_plan_digest?);
|
||||
complete_scanner_cache_snapshot_plan_digest(
|
||||
&usage,
|
||||
ScannerCacheBaselineProof {
|
||||
authoritative_data: None,
|
||||
observed_candidate_data: None,
|
||||
expected_sources,
|
||||
leader_epoch: usage.scanner_epoch?,
|
||||
want_cycle: cycle,
|
||||
scan_plan_digest: plan,
|
||||
},
|
||||
None,
|
||||
)?;
|
||||
Some(cycle)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -420,7 +383,6 @@ pub(crate) fn current_cache_root_or_prepare(
|
||||
tier_registry_generation: None,
|
||||
checkpoint_identity: None,
|
||||
checkpoint_plan_digest: None,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
},
|
||||
)
|
||||
}
|
||||
@@ -435,38 +397,6 @@ pub(crate) fn current_cache_root_or_prepare_with_generation(
|
||||
options: DataUsageCacheReuseOptions,
|
||||
) -> DataUsageCacheScanState {
|
||||
let checkpoint_plan_digest = options.checkpoint_plan_digest.unwrap_or(scan_plan_digest);
|
||||
// A new leader must not discard finished progress while other buckets
|
||||
// still need budget. Adopt only completed, unpublished coverage; after
|
||||
// the aggregate is published the cycle floor forces the usual refresh.
|
||||
// Keep the saved cycle unchanged so reuse cannot extend this authority.
|
||||
// The reuse plan still binds every process incarnation: a restart may
|
||||
// lose unflushed dirty hints and must invalidate completed coverage.
|
||||
let unpublished_completion = options.completed_checkpoint_cycle_floor.is_some_and(|floor| {
|
||||
cache.info.next_cycle > floor && cache.info.next_cycle < next_cycle && cache.info.leader_epoch < leader_epoch
|
||||
}) && options.checkpoint_identity.is_some_and(|identity| {
|
||||
identity.is_valid()
|
||||
&& cache.info.scan_identity == Some(identity)
|
||||
&& cache.info.tier_registry_generation == Some(identity.tier_registry_generation)
|
||||
&& options.tier_registry_generation == Some(identity.tier_registry_generation)
|
||||
}) && name != DATA_USAGE_ROOT
|
||||
&& cache.info.name == name
|
||||
&& cache.info.source == Some(source)
|
||||
&& cache.info.cache_key_format == DATA_USAGE_CACHE_KEY_FORMAT
|
||||
&& cache.info.snapshot_complete
|
||||
&& cache.info.last_update.is_some()
|
||||
&& cache.info.scan_progress.is_none()
|
||||
&& !cache.has_scan_position()
|
||||
&& cache.info.scan_plan_digest == Some(scan_plan_digest)
|
||||
&& options.checkpoint_plan_digest.is_some()
|
||||
&& cache.info.scan_coverage_digest == Some(checkpoint_plan_digest)
|
||||
&& !cache
|
||||
.info
|
||||
.replication
|
||||
.as_ref()
|
||||
.is_some_and(|config| config.has_active_rules("", true));
|
||||
if unpublished_completion && let Ok(root) = cache_root_entry_info(cache) {
|
||||
return DataUsageCacheScanState::Current(Box::new(root));
|
||||
}
|
||||
if cache.info.next_cycle <= next_cycle
|
||||
&& cache.info.leader_epoch <= leader_epoch
|
||||
&& cache.info.scan_identity != options.checkpoint_identity
|
||||
@@ -1549,13 +1479,7 @@ pub(super) async fn persist_scanner_service_ledger(
|
||||
ledger.leader_epoch = leader_epoch;
|
||||
persisted.info.scan_bucket_service = Some(ledger);
|
||||
match persisted
|
||||
.save_with_revisions_for_epoch(
|
||||
store,
|
||||
DATA_USAGE_CACHE_NAME,
|
||||
&revisions,
|
||||
expected_publication_epoch,
|
||||
CancellationToken::new(),
|
||||
)
|
||||
.save_with_revisions_for_epoch(store, DATA_USAGE_CACHE_NAME, &revisions, expected_publication_epoch)
|
||||
.await
|
||||
{
|
||||
Ok(()) => true,
|
||||
@@ -1570,7 +1494,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
|
||||
store: Arc<SetDisks>,
|
||||
updates: &mpsc::Sender<DataUsageCache>,
|
||||
mut cache_snapshot: DataUsageCache,
|
||||
initial_revisions: Option<&DataUsageCacheBaselineRevisions>,
|
||||
initial_revisions: Option<&DataUsageCacheRevisions>,
|
||||
cache_cycle_floor: &AtomicU64,
|
||||
expected_publication_epoch: u64,
|
||||
) -> Option<SystemTime> {
|
||||
@@ -1612,10 +1536,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
|
||||
};
|
||||
|
||||
let mut persisted = DataUsageCache::default();
|
||||
let current_baseline = match persisted
|
||||
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
{
|
||||
let revisions = match persisted.load_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME).await {
|
||||
Ok(revisions) => revisions,
|
||||
Err(err) => {
|
||||
error!(
|
||||
@@ -1637,7 +1558,6 @@ pub(super) async fn persist_and_publish_cache_snapshot(
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let revisions = current_baseline.revisions();
|
||||
let Some(scan_plan_digest) = cache_snapshot.info.scan_plan_digest else {
|
||||
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "plan_digest");
|
||||
return None;
|
||||
@@ -1701,9 +1621,9 @@ pub(super) async fn persist_and_publish_cache_snapshot(
|
||||
{
|
||||
cache_snapshot = persisted;
|
||||
} else {
|
||||
// Another execution must not advance the baseline. A backup catching
|
||||
// up to the same decoded primary changes no baseline content.
|
||||
if !initial_revisions.is_some_and(|initial| current_baseline.permits_unchanged_baseline_since(initial)) {
|
||||
// A later execution may have completed while this scan was walking.
|
||||
// Only replace the cache revision from which this scan started.
|
||||
if initial_revisions != Some(&revisions) {
|
||||
warn!(
|
||||
target: "rustfs::scanner::io",
|
||||
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
|
||||
@@ -1742,13 +1662,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
|
||||
|
||||
let done_save = Metrics::time(Metric::SaveUsage);
|
||||
if let Err(e) = cache_snapshot
|
||||
.save_with_revisions_for_epoch(
|
||||
store.clone(),
|
||||
DATA_USAGE_CACHE_NAME,
|
||||
revisions,
|
||||
expected_publication_epoch,
|
||||
CancellationToken::new(),
|
||||
)
|
||||
.save_with_revisions_for_epoch(store.clone(), DATA_USAGE_CACHE_NAME, &revisions, expected_publication_epoch)
|
||||
.await
|
||||
{
|
||||
error!(
|
||||
|
||||
@@ -40,8 +40,6 @@ static DIRTY_USAGE_MUTATION_OBSERVER: LazyLock<StdRwLock<Option<ScannerDirtyUsag
|
||||
pub(super) static DIRTY_USAGE_PRODUCER_COVERAGE: AtomicU64 = AtomicU64::new(0);
|
||||
pub(super) static DIRTY_USAGE_BUCKET_NOTIFY: LazyLock<Notify> = LazyLock::new(Notify::new);
|
||||
pub(super) static SCANNER_ACTIVITY_EPOCH: LazyLock<String> = LazyLock::new(|| format!("{:032x}", rand::random::<u128>()));
|
||||
static SCANNER_CHECKPOINT_INSTANCE: std::sync::OnceLock<String> = std::sync::OnceLock::new();
|
||||
static SCANNER_PROCESS_MUTATED: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
|
||||
pub(super) static SCANNER_MAINTENANCE_GENERATION: AtomicU64 = AtomicU64::new(0);
|
||||
pub(super) static SCANNER_MAINTENANCE_NOTIFY: LazyLock<Notify> = LazyLock::new(Notify::new);
|
||||
const SCANNER_DURABLE_DIRTY_USAGE_REPLAY_SCHEMA: u16 = 1;
|
||||
@@ -154,7 +152,6 @@ fn notify_dirty_usage_mutation(
|
||||
if bucket.is_empty() {
|
||||
return;
|
||||
}
|
||||
SCANNER_PROCESS_MUTATED.store(true, Ordering::Release);
|
||||
let observer = DIRTY_USAGE_MUTATION_OBSERVER
|
||||
.read()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner())
|
||||
@@ -1375,30 +1372,6 @@ pub fn scanner_activity_epoch() -> &'static str {
|
||||
SCANNER_ACTIVITY_EPOCH.as_str()
|
||||
}
|
||||
|
||||
/// Identity used only for checkpoint continuity, never for RPC or dirty ACKs.
|
||||
/// Without a consumed clean-shutdown receipt it is the fresh process identity.
|
||||
pub fn scanner_checkpoint_instance_id() -> &'static str {
|
||||
SCANNER_CHECKPOINT_INSTANCE.get_or_init(|| scanner_activity_epoch().to_owned())
|
||||
}
|
||||
|
||||
/// Dirty replay is not a new mutation. Live invalidations prevent handing
|
||||
/// checkpoint authority to another process even after their dirty ACKs drain.
|
||||
pub fn scanner_process_was_read_only() -> bool {
|
||||
!SCANNER_PROCESS_MUTATED.load(Ordering::Acquire) && scanner_maintenance_generation() == 0
|
||||
}
|
||||
|
||||
/// Restore continuity before starting any scanner or mutation producer.
|
||||
/// The caller must first consume matching, durable, read-only shutdown receipts
|
||||
/// on every local disk. Returns false for invalid or already initialized state.
|
||||
pub fn restore_scanner_checkpoint_instance_id(instance: &str) -> bool {
|
||||
instance.len() == 32
|
||||
&& instance
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte))
|
||||
&& instance.bytes().any(|byte| byte != b'0')
|
||||
&& SCANNER_CHECKPOINT_INSTANCE.set(instance.to_owned()).is_ok()
|
||||
}
|
||||
|
||||
pub fn scanner_dirty_usage_state() -> ScannerDirtyUsageState {
|
||||
let dirty_buckets = dirty_usage_buckets();
|
||||
ScannerDirtyUsageState {
|
||||
|
||||
@@ -14,7 +14,6 @@
|
||||
use super::*;
|
||||
/// ScannerIOCache implementation for SetDisks: bucket ordering, worker fan-out, merge, and publish.
|
||||
use crate::data_usage_define::clone_cache_value;
|
||||
use rustfs_scanner_metrics::scan_observation::{ScannerBucketReuseKind, ScannerScanObservation};
|
||||
|
||||
pub(super) async fn collect_completed_bucket_results(
|
||||
ctx: CancellationToken,
|
||||
@@ -143,32 +142,6 @@ pub(super) struct PreparedScopedSetScan {
|
||||
pub(super) buckets: Vec<BucketInfo>,
|
||||
pub(super) cache: DataUsageCache,
|
||||
pub(super) cold_bucket_reuse_proof: Option<ScopedColdBucketReuseProof>,
|
||||
cold_bucket_reuse_count: u64,
|
||||
cold_bucket_reuse_elapsed: Duration,
|
||||
}
|
||||
|
||||
impl PreparedScopedSetScan {
|
||||
pub(super) fn record_cold_bucket_reuse(&self, observation: &ScannerScanObservation) {
|
||||
observation.record_bucket_reuse(
|
||||
ScannerBucketReuseKind::ScopedColdBucket,
|
||||
self.cold_bucket_reuse_count,
|
||||
self.cold_bucket_reuse_elapsed,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Counts only roots delivered to the collector; cancellation or channel failure is not reuse.
|
||||
pub(super) async fn send_current_cache_root_entry(
|
||||
bucket_result_tx: &mpsc::Sender<DataUsageEntryInfo>,
|
||||
root: DataUsageEntryInfo,
|
||||
cache: &DataUsageCache,
|
||||
pending_maintenance_work: &AtomicBool,
|
||||
observation: &ScannerScanObservation,
|
||||
) -> std::result::Result<(), ScannerError> {
|
||||
let started = Instant::now();
|
||||
send_cache_root_entry(bucket_result_tx, root, cache, pending_maintenance_work).await?;
|
||||
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, started.elapsed());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) struct ScopedColdBucketReuseProof {
|
||||
@@ -252,8 +225,6 @@ pub(super) fn prepare_scoped_set_scan(
|
||||
};
|
||||
cache.replace(DATA_USAGE_ROOT, "", DataUsageEntry::default());
|
||||
let mut current_bucket_names = HashSet::with_capacity(all_buckets.len());
|
||||
let mut cold_bucket_reuse_count = 0u64;
|
||||
let mut cold_bucket_reuse_elapsed = Duration::ZERO;
|
||||
for bucket in all_buckets {
|
||||
if !current_bucket_names.insert(bucket.name.as_str()) {
|
||||
return None;
|
||||
@@ -262,14 +233,11 @@ pub(super) fn prepare_scoped_set_scan(
|
||||
cache.info.scan_bucket_value_origins.remove(&bucket.name);
|
||||
cache.replace(&bucket.name, DATA_USAGE_ROOT, DataUsageEntry::default());
|
||||
} else {
|
||||
let started = Instant::now();
|
||||
cache.copy_with_children(
|
||||
old_cache,
|
||||
&rustfs_data_usage::hash_path(&bucket.name),
|
||||
&Some(rustfs_data_usage::hash_path(DATA_USAGE_ROOT)),
|
||||
);
|
||||
cold_bucket_reuse_count = cold_bucket_reuse_count.saturating_add(1);
|
||||
cold_bucket_reuse_elapsed = cold_bucket_reuse_elapsed.saturating_add(started.elapsed());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -281,8 +249,6 @@ pub(super) fn prepare_scoped_set_scan(
|
||||
.collect(),
|
||||
cache,
|
||||
cold_bucket_reuse_proof,
|
||||
cold_bucket_reuse_count,
|
||||
cold_bucket_reuse_elapsed,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -423,7 +389,6 @@ impl ScannerIOCache for SetDisks {
|
||||
digest: scan_plan_digest,
|
||||
bucket_coverage_digest,
|
||||
checkpoint_coverage_digest,
|
||||
completed_checkpoint_cycle_floor,
|
||||
requires_full_scan,
|
||||
service_cohort,
|
||||
execution_digest,
|
||||
@@ -460,10 +425,7 @@ impl ScannerIOCache for SetDisks {
|
||||
.ok_or_else(|| StorageError::other("scanner cache publication is blocked by data movement"))?,
|
||||
};
|
||||
let mut old_cache = DataUsageCache::default();
|
||||
let initial_revisions = match old_cache
|
||||
.load_with_baseline_revisions(self.clone(), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
{
|
||||
let initial_revisions = match old_cache.load_with_revisions(self.clone(), DATA_USAGE_CACHE_NAME).await {
|
||||
Ok(revisions) => Some(revisions),
|
||||
Err(e) => {
|
||||
warn!(
|
||||
@@ -593,8 +555,6 @@ impl ScannerIOCache for SetDisks {
|
||||
})
|
||||
});
|
||||
let mut scoped_cache = scoped_scan.map(|mut prepared| {
|
||||
// The copies are completed work even if this turn later fails publication.
|
||||
prepared.record_cold_bucket_reuse(global_metrics().scan_observation());
|
||||
buckets = prepared.buckets;
|
||||
prepared.cache.info.scan_coverage_digest = Some(bucket_coverage_digest);
|
||||
prepared.cache
|
||||
@@ -1555,7 +1515,6 @@ impl ScannerIOCache for SetDisks {
|
||||
tier_registry_generation: Some(tier_registry_generation),
|
||||
checkpoint_identity,
|
||||
checkpoint_plan_digest: Some(bucket_checkpoint_plan_digest),
|
||||
completed_checkpoint_cycle_floor,
|
||||
},
|
||||
);
|
||||
let outcome = match scan_state {
|
||||
@@ -1599,14 +1558,8 @@ impl ScannerIOCache for SetDisks {
|
||||
}
|
||||
record_completed_usage_watch(&usage_watch_completions_clone, source, &bucket.name, bucket_usage_watch.as_ref()).await;
|
||||
if let Err(e) =
|
||||
send_current_cache_root_entry(
|
||||
&bucket_result_tx_clone,
|
||||
*root,
|
||||
&cache,
|
||||
&pending_maintenance_work_clone,
|
||||
global_metrics().scan_observation(),
|
||||
)
|
||||
.await
|
||||
send_cache_root_entry(&bucket_result_tx_clone, *root, &cache, &pending_maintenance_work_clone)
|
||||
.await
|
||||
{
|
||||
record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await;
|
||||
error!(
|
||||
@@ -1921,11 +1874,10 @@ impl ScannerIOCache for SetDisks {
|
||||
} else {
|
||||
// The sweep finished under its coverage plan; the
|
||||
// complete cache is current under the reuse plan,
|
||||
// while its coverage plan remains available until
|
||||
// the coordinated aggregate is published.
|
||||
// which a restart or a maintenance publication
|
||||
// changes so the bucket is walked again next time.
|
||||
if completed_cache.info.snapshot_complete {
|
||||
completed_cache.info.scan_plan_digest = Some(bucket_scan_plan_digest);
|
||||
completed_cache.info.scan_coverage_digest = Some(bucket_checkpoint_plan_digest);
|
||||
}
|
||||
(completed_cache, false)
|
||||
}
|
||||
@@ -1977,7 +1929,6 @@ impl ScannerIOCache for SetDisks {
|
||||
cache_name.as_str(),
|
||||
&revisions,
|
||||
expected_publication_epoch_clone,
|
||||
CancellationToken::new(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -2091,7 +2042,6 @@ impl ScannerIOCache for SetDisks {
|
||||
&cache_name,
|
||||
&revisions,
|
||||
expected_publication_epoch_clone,
|
||||
CancellationToken::new(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
|
||||
@@ -357,7 +357,6 @@ where
|
||||
S: ScannerStorage,
|
||||
{
|
||||
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
|
||||
segment_reuse_activation_preflight: None,
|
||||
scope,
|
||||
remote_dirty_usage_acknowledgements: Vec::new(),
|
||||
distributed_segment_invalidation_evidence: None,
|
||||
@@ -507,7 +506,7 @@ where
|
||||
distributed,
|
||||
resolution.baseline_proof,
|
||||
);
|
||||
let scope = scoped_scan_scope_from_dirty_buckets(
|
||||
default_result(scoped_scan_scope_from_dirty_buckets(
|
||||
resolution.requested_scope,
|
||||
dirty_buckets,
|
||||
(!distributed).then_some(resolution.dirty_usage_snapshot.scopes.as_ref()),
|
||||
@@ -515,11 +514,7 @@ where
|
||||
segment_reuse_activation_preflight.scanner_segment_reuse_activated,
|
||||
resolution.all_buckets,
|
||||
resolution.baseline_proof,
|
||||
);
|
||||
ScannerBucketScopeResolutionResult {
|
||||
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
|
||||
..default_result(scope)
|
||||
}
|
||||
))
|
||||
}
|
||||
|
||||
/// Runs once between the bucket listing and the dirty-usage snapshot, the
|
||||
@@ -624,14 +619,6 @@ where
|
||||
};
|
||||
|
||||
let distributed = store.setup_is_dist_erasure().await;
|
||||
let local_node = if distributed {
|
||||
store
|
||||
.scanner_local_node_name()
|
||||
.filter(|name| !name.is_empty())
|
||||
.ok_or_else(|| Error::other("scanner local member identity is unavailable"))?
|
||||
} else {
|
||||
crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE.to_string()
|
||||
};
|
||||
#[cfg(test)]
|
||||
let distributed = distributed || peer_observations.is_some();
|
||||
let activity_before = match scanner_activity_preflight(
|
||||
@@ -736,23 +723,15 @@ where
|
||||
let activity_before = remote_start
|
||||
.as_ref()
|
||||
.map_or(activity_before, ScannerRemoteDirtyUsageWindow::activity_for_walk);
|
||||
let structural_scan_plan_digest = scanner_bucket_plan_digest(
|
||||
&all_buckets,
|
||||
crate::scanner::scanner_activity_structural_digest(&activity_before, &local_node),
|
||||
);
|
||||
let structural_scan_plan_digest =
|
||||
scanner_bucket_plan_digest(&all_buckets, crate::scanner::scanner_activity_structural_digest(&activity_before));
|
||||
let scan_plan_digest = scanner_bucket_work_digest(structural_scan_plan_digest, scan_mode, requires_full_scan);
|
||||
let activity_digest = crate::scanner::scanner_activity_snapshot_digest(&activity_before);
|
||||
let bucket_coverage_digest = scanner_bucket_plan_digest(
|
||||
&all_buckets,
|
||||
crate::scanner::scanner_activity_bucket_reuse_digest(&activity_before, &local_node),
|
||||
);
|
||||
// A bucket checkpoint must survive unrelated inventory changes.
|
||||
let checkpoint_coverage_digest =
|
||||
DataUsageScanPlanDigest(crate::scanner::scanner_activity_coverage_digest(&activity_before, &local_node));
|
||||
let completed_checkpoint_cycle_floor =
|
||||
completed_checkpoint_cycle_floor(persisted_usage_baseline.as_deref(), &expected_sources, want_cycle, leader_epoch);
|
||||
let execution_digest =
|
||||
scanner_bucket_work_digest(scanner_bucket_plan_digest(&all_buckets, activity_digest), scan_mode, requires_full_scan);
|
||||
let bucket_coverage_digest = scanner_bucket_plan_digest(&all_buckets, activity_digest);
|
||||
// A bucket checkpoint must not be invalidated by another bucket joining
|
||||
// the inventory. The set plan above still binds the complete inventory.
|
||||
let checkpoint_coverage_digest = DataUsageScanPlanDigest(crate::scanner::scanner_activity_coverage_digest(&activity_before));
|
||||
let execution_digest = scanner_bucket_work_digest(bucket_coverage_digest, scan_mode, requires_full_scan);
|
||||
let dirty_usage_snapshot = Arc::new(snapshot_dirty_usage_buckets_after_listing(&all_buckets, dirty_usage_cutoff));
|
||||
let baseline_proof = ScannerCacheBaselineProof {
|
||||
authoritative_data: persisted_usage_baseline.as_ref(),
|
||||
@@ -780,9 +759,6 @@ where
|
||||
},
|
||||
)
|
||||
.await;
|
||||
if let Some(preflight) = scope_resolution.segment_reuse_activation_preflight {
|
||||
preflight.record_observation(ScannerPrefixReuseStage::ScopeResolution);
|
||||
}
|
||||
let remote_dirty_usage_acknowledgements = scope_resolution.remote_dirty_usage_acknowledgements;
|
||||
let distributed_segment_invalidation_evidence = scope_resolution.distributed_segment_invalidation_evidence;
|
||||
let scan_scope = scope_resolution.scope;
|
||||
@@ -794,14 +770,8 @@ where
|
||||
baseline_proof,
|
||||
distributed,
|
||||
);
|
||||
if let Some(fallback) = scope_fallback {
|
||||
let reason = fallback.reason;
|
||||
if let Some(reason) = scope_fallback {
|
||||
global_metrics().record_scanner_scope_fallback(reason.as_str());
|
||||
if let Some(failure) = fallback.baseline_rejection {
|
||||
global_metrics()
|
||||
.scan_observation()
|
||||
.record_baseline_rejection(failure.authoritative, failure.observed);
|
||||
}
|
||||
tracing::info!(
|
||||
target: "rustfs::scanner::io",
|
||||
event = EVENT_SCANNER_SET_STATE,
|
||||
@@ -815,7 +785,6 @@ where
|
||||
"Scanner cycle walks every bucket because its dirty-bucket scope was not admitted"
|
||||
);
|
||||
}
|
||||
let scope_fallback = scope_fallback.map(|fallback| fallback.reason);
|
||||
let segment_invalidation_proof = dirty_usage_producer_evidence(&dirty_usage_snapshot).segment_invalidation_proof();
|
||||
#[cfg(test)]
|
||||
if let Some(observer) = resolved_scope_observer {
|
||||
@@ -892,7 +861,6 @@ where
|
||||
None,
|
||||
false,
|
||||
);
|
||||
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
|
||||
return Ok(ScannerCycleResult::new(status, dirty_usage_clear)
|
||||
.with_publication_epoch(publication_epoch)
|
||||
.with_activity_digest(activity_digest)
|
||||
@@ -1046,7 +1014,6 @@ where
|
||||
digest: structural_scan_plan_digest,
|
||||
bucket_coverage_digest,
|
||||
checkpoint_coverage_digest,
|
||||
completed_checkpoint_cycle_floor,
|
||||
requires_full_scan,
|
||||
service_cohort: service_cohort.clone(),
|
||||
execution_digest,
|
||||
@@ -1240,7 +1207,6 @@ where
|
||||
distributed_segment_invalidation_evidence,
|
||||
cold_zero_walk_oracle,
|
||||
);
|
||||
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
|
||||
let pending_maintenance_work = pending_maintenance_work_for_cycle(&pending_maintenance_work, &results);
|
||||
let observed_cycle_floor = cache_cycle_floor.load(Ordering::Acquire);
|
||||
let required_cycle_floor = (observed_cycle_floor > want_cycle).then_some(observed_cycle_floor);
|
||||
|
||||
@@ -15,7 +15,6 @@ use super::*;
|
||||
use crate::UNKNOWN_TIER;
|
||||
/// ScannerIODisk implementation for Disk: get_size and the per-disk bucket scan.
|
||||
use crate::data_usage_define::clone_cache_value;
|
||||
use rustfs_scanner_metrics::scan_observation::ScannerMetadataReadOutcome;
|
||||
|
||||
///
|
||||
/// Seed [`SizeSummary::tier_stats`] from the cached tier-name list.
|
||||
@@ -63,20 +62,8 @@ impl ScannerIODisk for Disk {
|
||||
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
let metadata_object_path = item.object_path();
|
||||
let metadata_result = crate::workload_admission::with_scanner_metadata_io(async {
|
||||
let read = global_metrics().start_scanner_metadata_read();
|
||||
let result = self.read_metadata(&item.bucket, &metadata_object_path).await;
|
||||
let (outcome, bytes) = match &result {
|
||||
Ok(data) => (ScannerMetadataReadOutcome::Success, u64::try_from(data.len()).unwrap_or(u64::MAX)),
|
||||
Err(e) if DiskError::is_err_object_not_found(e) || DiskError::is_err_version_not_found(e) => {
|
||||
(ScannerMetadataReadOutcome::NotFound, 0)
|
||||
}
|
||||
Err(_) => (ScannerMetadataReadOutcome::Error, 0),
|
||||
};
|
||||
read.finish(outcome, bytes);
|
||||
result
|
||||
})
|
||||
.await;
|
||||
let metadata_result =
|
||||
crate::workload_admission::with_scanner_metadata_io(self.read_metadata(&item.bucket, &metadata_object_path)).await;
|
||||
ensure_scanner_node_operation_active(cancellation.as_ref())?;
|
||||
let data = match metadata_result {
|
||||
Ok(data) => data,
|
||||
|
||||
@@ -635,7 +635,6 @@ mod tests {
|
||||
tier_registry_generation: None,
|
||||
checkpoint_identity: None,
|
||||
checkpoint_plan_digest: None,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
};
|
||||
assert!(matches!(
|
||||
current_cache_root_or_prepare_with_generation(&mut cache.clone(), "cold-bucket", source, 7, 11, old_plan, options()),
|
||||
|
||||
@@ -827,7 +827,6 @@ fn current_cache_root_with_new_tier_generation_resets_old_cache() {
|
||||
tier_registry_generation: Some(2),
|
||||
checkpoint_identity: None,
|
||||
checkpoint_plan_digest: None,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
},
|
||||
);
|
||||
|
||||
|
||||
@@ -27,10 +27,10 @@ use crate::storage_api::owner::{
|
||||
};
|
||||
use crate::storage_api::scan::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _};
|
||||
use crate::{
|
||||
BUCKET_META_PREFIX, DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints,
|
||||
InstanceContext, PoolEndpoints, ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER,
|
||||
init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx,
|
||||
new_disk, path2_bucket_object_with_base_path, save_config,
|
||||
DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints,
|
||||
ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests,
|
||||
init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
|
||||
save_config,
|
||||
};
|
||||
use rustfs_concurrency::{
|
||||
AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
|
||||
@@ -104,7 +104,6 @@ fn scanner_activity_preflight_defers_a_temporarily_offline_peer() {
|
||||
|
||||
#[test]
|
||||
fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
let preflight = scanner_segment_reuse_activation_preflight();
|
||||
|
||||
assert!(preflight.production_activation);
|
||||
@@ -125,9 +124,6 @@ fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
|
||||
"distributed_without_peer_invalidation",
|
||||
]
|
||||
);
|
||||
let after = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
|
||||
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -551,102 +547,11 @@ async fn setup_two_pool_scanner_store() -> (tempfile::TempDir, Arc<ECStore>) {
|
||||
pub(crate) async fn setup_local_scanner_store(pool_count: usize, drives_per_set: usize) -> (tempfile::TempDir, Arc<ECStore>) {
|
||||
init_ecstore_config_for_scanner_tests();
|
||||
let temp_dir = tempfile::tempdir().expect("multi-pool scanner test directory should be created");
|
||||
let store = open_local_scanner_store(temp_dir.path(), pool_count, drives_per_set, false).await;
|
||||
(temp_dir, store)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn epoch_fenced_cache_save_uses_background_admission_on_every_disk() {
|
||||
#[derive(Clone)]
|
||||
struct AdmissionRecorder {
|
||||
root: String,
|
||||
counts: Arc<std::sync::Mutex<HashMap<String, Arc<AtomicU64>>>>,
|
||||
}
|
||||
impl metrics::Recorder for AdmissionRecorder {
|
||||
fn describe_counter(&self, _: metrics::KeyName, _: Option<metrics::Unit>, _: metrics::SharedString) {}
|
||||
fn describe_gauge(&self, _: metrics::KeyName, _: Option<metrics::Unit>, _: metrics::SharedString) {}
|
||||
fn describe_histogram(&self, _: metrics::KeyName, _: Option<metrics::Unit>, _: metrics::SharedString) {}
|
||||
fn register_counter(&self, key: &metrics::Key, _: &metrics::Metadata<'_>) -> metrics::Counter {
|
||||
if key.name() != "rustfs_disk_io_admission_total"
|
||||
|| !key
|
||||
.labels()
|
||||
.any(|label| label.key() == "class" && label.value() == "background")
|
||||
{
|
||||
return metrics::Counter::noop();
|
||||
}
|
||||
let Some(disk) = key
|
||||
.labels()
|
||||
.find(|label| label.key() == "disk" && label.value().starts_with(&self.root))
|
||||
else {
|
||||
return metrics::Counter::noop();
|
||||
};
|
||||
metrics::Counter::from_arc(
|
||||
self.counts
|
||||
.lock()
|
||||
.expect("admission counters")
|
||||
.entry(disk.value().to_string())
|
||||
.or_default()
|
||||
.clone(),
|
||||
)
|
||||
}
|
||||
fn register_gauge(&self, _: &metrics::Key, _: &metrics::Metadata<'_>) -> metrics::Gauge {
|
||||
metrics::Gauge::noop()
|
||||
}
|
||||
fn register_histogram(&self, _: &metrics::Key, _: &metrics::Metadata<'_>) -> metrics::Histogram {
|
||||
metrics::Histogram::noop()
|
||||
}
|
||||
}
|
||||
let (root, store) = setup_local_scanner_store(1, 4).await;
|
||||
let set = store.pools[0].disk_set[0].clone();
|
||||
let epoch = scanner_publication_epoch(set.clone()).await.expect("publication epoch");
|
||||
let cache_name = "background-admission/.usage-cache.bin";
|
||||
let mut cache = DataUsageCache::default();
|
||||
cache.info.name = "background-admission".to_string();
|
||||
let recorder = AdmissionRecorder {
|
||||
root: root.path().to_str().expect("fixture path").to_string(),
|
||||
counts: Arc::default(),
|
||||
};
|
||||
let counts = recorder.counts.clone();
|
||||
// nextest runs this test in its own process; the recorder also observes the
|
||||
// detached commit tasks and filters out every disk outside this fixture.
|
||||
metrics::set_global_recorder(recorder).expect("native save recorder");
|
||||
cache.save(set.clone(), cache_name).await.expect("unclassified control save");
|
||||
assert!(counts.lock().expect("control counters").is_empty());
|
||||
let revisions = DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("cache revisions");
|
||||
cache.info.next_cycle = 7;
|
||||
cache
|
||||
.save_with_revisions_for_epoch(set.clone(), cache_name, &revisions, epoch, CancellationToken::new())
|
||||
.await
|
||||
.expect("background primary and backup save");
|
||||
{
|
||||
let counts = counts.lock().expect("background counters");
|
||||
assert_eq!(counts.len(), 4, "every local disk must receive background cache I/O");
|
||||
assert!(counts.values().all(|counter| counter.load(Ordering::Relaxed) > 0));
|
||||
}
|
||||
for name in [cache_name.to_string(), format!("{cache_name}.bkp")] {
|
||||
let body = crate::read_config(set.clone(), &path_join_buf(&[crate::BUCKET_META_PREFIX, &name]))
|
||||
.await
|
||||
.expect("persisted cache copy");
|
||||
let saved: DataUsageCache = rmp_serde::from_slice(&body).expect("decode persisted cache");
|
||||
assert_eq!(saved.info.name, cache.info.name);
|
||||
assert_eq!(saved.info.next_cycle, 7);
|
||||
}
|
||||
}
|
||||
|
||||
async fn open_local_scanner_store(
|
||||
root: &std::path::Path,
|
||||
pool_count: usize,
|
||||
drives_per_set: usize,
|
||||
process_boot: bool,
|
||||
) -> Arc<ECStore> {
|
||||
init_ecstore_config_for_scanner_tests();
|
||||
let mut pools = Vec::new();
|
||||
for pool_index in 0..pool_count {
|
||||
let mut endpoints = Vec::new();
|
||||
for disk_index in 0..drives_per_set {
|
||||
let disk_path = root.join(format!("pool{pool_index}-disk{disk_index}"));
|
||||
let disk_path = temp_dir.path().join(format!("pool{pool_index}-disk{disk_index}"));
|
||||
tokio::fs::create_dir_all(&disk_path)
|
||||
.await
|
||||
.expect("multi-pool scanner test disk should be created");
|
||||
@@ -668,16 +573,6 @@ async fn open_local_scanner_store(
|
||||
}
|
||||
|
||||
let endpoint_pools = EndpointServerPools::from(pools);
|
||||
if process_boot {
|
||||
crate::storage_api::ecstore_init_local_disks(endpoint_pools.clone())
|
||||
.await
|
||||
.expect("process disk registry");
|
||||
let store = ECStore::new("127.0.0.1:0".parse().expect("test address"), endpoint_pools, CancellationToken::new())
|
||||
.await
|
||||
.expect("process scanner store");
|
||||
init_bucket_metadata_sys_for_scanner_tests(store.clone()).await;
|
||||
return store;
|
||||
}
|
||||
let instance_ctx = Arc::new(InstanceContext::new());
|
||||
instance_ctx.set_endpoints(endpoint_pools.clone());
|
||||
init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools.clone())
|
||||
@@ -706,7 +601,7 @@ async fn open_local_scanner_store(
|
||||
.expect("scanner fixture must expose its live metadata owner");
|
||||
assert!(Arc::ptr_eq(&owner, &store), "scanner watch metadata owner must be this fixture");
|
||||
|
||||
store
|
||||
(temp_dir, store)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -2491,260 +2386,6 @@ async fn set_cache_send_failure_logs_receiver_and_snapshot_without_changing_retu
|
||||
assert_set_cache_context(set_cache_event(&events, "publish_failed"), Some((0, 0)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn set_snapshot_accepts_backup_catchup_to_the_same_decoded_primary() {
|
||||
use crate::data_usage_define::read_config_with_revision;
|
||||
|
||||
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
|
||||
let set = Arc::clone(&store.pools[0].disk_set[0]);
|
||||
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
|
||||
let backup_path = format!("{primary_path}.bkp");
|
||||
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
|
||||
.await
|
||||
.expect("read decoded primary bytes");
|
||||
let primary_body = primary_body.expect("primary exists");
|
||||
let mut older = cache.clone();
|
||||
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
|
||||
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode older backup"))
|
||||
.await
|
||||
.expect("seed a lagging backup");
|
||||
let mut loaded = DataUsageCache::default();
|
||||
let initial = loaded
|
||||
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("capture primary with lagging backup");
|
||||
save_config(Arc::clone(&set), &backup_path, primary_body)
|
||||
.await
|
||||
.expect("backup catches up without changing primary");
|
||||
let current = loaded
|
||||
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("capture the reconciled backup");
|
||||
assert_ne!(initial.revisions(), current.revisions());
|
||||
assert!(current.permits_unchanged_baseline_since(&initial));
|
||||
|
||||
let mut candidate = cache;
|
||||
candidate.info.next_cycle += 1;
|
||||
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
|
||||
candidate.replace(
|
||||
"photos",
|
||||
DATA_USAGE_ROOT,
|
||||
DataUsageEntry {
|
||||
size: 20,
|
||||
objects: 1,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let cycle = candidate.info.next_cycle;
|
||||
let cycle_floor = AtomicU64::new(cycle);
|
||||
let (tx, mut rx) = mpsc::channel(1);
|
||||
assert!(
|
||||
persist_and_publish_cache_snapshot(Arc::clone(&set), &tx, candidate, Some(&initial), &cycle_floor, epoch)
|
||||
.await
|
||||
.is_some(),
|
||||
"content-preserving backup catchup must not veto the new snapshot"
|
||||
);
|
||||
let published = rx.try_recv().expect("new snapshot reached the collector");
|
||||
assert_eq!(published.info.next_cycle, cycle);
|
||||
assert_eq!(published.find("photos").expect("new bucket entry").size, 20);
|
||||
let (primary, primary_revision) = read_config_with_revision(Arc::clone(&set), &primary_path)
|
||||
.await
|
||||
.expect("read new primary");
|
||||
let (backup, backup_revision) = read_config_with_revision(set, &backup_path).await.expect("read new backup");
|
||||
assert_eq!(primary, backup, "backup CAS must use the reconciled current revision");
|
||||
assert_eq!(primary_revision, backup_revision);
|
||||
let persisted = DataUsageCache::unmarshal(&primary.expect("new primary body")).expect("decode new primary");
|
||||
assert_eq!(persisted.info.next_cycle, cycle);
|
||||
assert_eq!(persisted.find("photos").expect("persisted new entry").size, 20);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn set_snapshot_backup_catchup_keeps_revision_provenance_and_epoch_fences() {
|
||||
use crate::data_usage_define::read_config_with_revision;
|
||||
|
||||
for case in ["different_backup", "different_primary", "corrupt_primary", "changed_epoch"] {
|
||||
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
|
||||
let set = Arc::clone(&store.pools[0].disk_set[0]);
|
||||
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
|
||||
let backup_path = format!("{primary_path}.bkp");
|
||||
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
|
||||
.await
|
||||
.expect("read original primary");
|
||||
let primary_body = primary_body.expect("original primary exists");
|
||||
let mut older = cache.clone();
|
||||
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
|
||||
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode lagging backup"))
|
||||
.await
|
||||
.expect("seed lagging backup");
|
||||
if case == "corrupt_primary" {
|
||||
save_config(Arc::clone(&set), &primary_path, b"corrupt-primary".to_vec())
|
||||
.await
|
||||
.expect("inject primary corruption before the initial load");
|
||||
}
|
||||
let mut loaded = DataUsageCache::default();
|
||||
let initial = loaded
|
||||
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("capture original baseline source and revisions");
|
||||
let mut replacement = cache.clone();
|
||||
replacement.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
|
||||
match case {
|
||||
"different_backup" => {
|
||||
save_config(Arc::clone(&set), &backup_path, replacement.marshal_msg().expect("encode changed backup"))
|
||||
.await
|
||||
.expect("backup changes to different contents");
|
||||
}
|
||||
"different_primary" => {
|
||||
let body = replacement.marshal_msg().expect("encode changed primary");
|
||||
save_config(Arc::clone(&set), &primary_path, body.clone())
|
||||
.await
|
||||
.expect("primary advances");
|
||||
save_config(Arc::clone(&set), &backup_path, body)
|
||||
.await
|
||||
.expect("backup follows changed primary");
|
||||
}
|
||||
"corrupt_primary" => {
|
||||
save_config(Arc::clone(&set), &backup_path, b"corrupt-primary".to_vec())
|
||||
.await
|
||||
.expect("backup matches the corrupt primary ETag without a decoded primary proof");
|
||||
}
|
||||
"changed_epoch" => {
|
||||
save_config(Arc::clone(&set), &backup_path, primary_body)
|
||||
.await
|
||||
.expect("backup catches up before epoch refusal");
|
||||
}
|
||||
_ => unreachable!("fixed test cases"),
|
||||
}
|
||||
let before_primary = read_config_with_revision(Arc::clone(&set), &primary_path)
|
||||
.await
|
||||
.expect("current primary");
|
||||
let before_backup = read_config_with_revision(Arc::clone(&set), &backup_path)
|
||||
.await
|
||||
.expect("current backup");
|
||||
let mut candidate = cache;
|
||||
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
|
||||
let cycle_floor = AtomicU64::new(candidate.info.next_cycle);
|
||||
let requested_epoch = if case == "changed_epoch" {
|
||||
epoch.checked_add(1).expect("fixture epoch can advance")
|
||||
} else {
|
||||
epoch
|
||||
};
|
||||
let (tx, mut rx) = mpsc::channel(1);
|
||||
let (result, events) = capture_set_cache_logs(persist_and_publish_cache_snapshot(
|
||||
Arc::clone(&set),
|
||||
&tx,
|
||||
candidate,
|
||||
Some(&initial),
|
||||
&cycle_floor,
|
||||
requested_epoch,
|
||||
))
|
||||
.await;
|
||||
assert!(result.is_none(), "{case} must still refuse publication");
|
||||
assert!(
|
||||
matches!(rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
|
||||
"{case} forwards no snapshot"
|
||||
);
|
||||
let expected_state = if case == "changed_epoch" {
|
||||
"save_failed"
|
||||
} else {
|
||||
"scan_baseline_revision_changed"
|
||||
};
|
||||
assert_set_cache_context(set_cache_event(&events, expected_state), Some((0, 0)));
|
||||
assert_eq!(
|
||||
read_config_with_revision(Arc::clone(&set), &primary_path)
|
||||
.await
|
||||
.expect("protected primary"),
|
||||
before_primary,
|
||||
"{case} does not overwrite primary"
|
||||
);
|
||||
assert_eq!(
|
||||
read_config_with_revision(set, &backup_path).await.expect("protected backup"),
|
||||
before_backup,
|
||||
"{case} does not overwrite backup"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn set_cache_cas_preserves_updates_after_backup_catchup_validation() {
|
||||
use crate::data_usage_define::read_config_with_revision;
|
||||
|
||||
for changed_copy in ["primary", "backup"] {
|
||||
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
|
||||
let set = Arc::clone(&store.pools[0].disk_set[0]);
|
||||
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
|
||||
let backup_path = format!("{primary_path}.bkp");
|
||||
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
|
||||
.await
|
||||
.expect("read initial primary");
|
||||
let mut lagging = cache.clone();
|
||||
lagging.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
|
||||
save_config(Arc::clone(&set), &backup_path, lagging.marshal_msg().expect("lagging backup bytes"))
|
||||
.await
|
||||
.expect("seed backup lag");
|
||||
let mut loaded = DataUsageCache::default();
|
||||
let initial = loaded
|
||||
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("read initial baseline");
|
||||
save_config(Arc::clone(&set), &backup_path, primary_body.expect("initial primary exists"))
|
||||
.await
|
||||
.expect("finish backup catchup");
|
||||
let current = loaded
|
||||
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("read reconciled baseline");
|
||||
assert!(current.permits_unchanged_baseline_since(&initial));
|
||||
let changed_path = if changed_copy == "primary" {
|
||||
&primary_path
|
||||
} else {
|
||||
&backup_path
|
||||
};
|
||||
let mut concurrent = cache.clone();
|
||||
concurrent.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
|
||||
let concurrent_body = concurrent.marshal_msg().expect("encode concurrent update");
|
||||
save_config(Arc::clone(&set), changed_path, concurrent_body.clone())
|
||||
.await
|
||||
.expect("copy changes after the baseline validation");
|
||||
let mut candidate = cache;
|
||||
candidate.info.next_cycle += 1;
|
||||
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
|
||||
let result = candidate
|
||||
.save_with_revisions_for_epoch(
|
||||
Arc::clone(&set),
|
||||
DATA_USAGE_CACHE_NAME,
|
||||
current.revisions(),
|
||||
epoch,
|
||||
CancellationToken::new(),
|
||||
)
|
||||
.await;
|
||||
if changed_copy == "primary" {
|
||||
assert!(result.is_err(), "primary CAS refuses the intervening update");
|
||||
} else {
|
||||
result.expect("primary remains durable when the best-effort backup CAS refuses an update");
|
||||
}
|
||||
let (protected, _) = read_config_with_revision(Arc::clone(&set), changed_path)
|
||||
.await
|
||||
.expect("read protected concurrent copy");
|
||||
assert_eq!(
|
||||
protected,
|
||||
Some(concurrent_body),
|
||||
"{changed_copy} CAS must not overwrite the concurrent update"
|
||||
);
|
||||
let (primary, _) = read_config_with_revision(set, &primary_path)
|
||||
.await
|
||||
.expect("read resulting primary");
|
||||
let primary = DataUsageCache::unmarshal(&primary.expect("primary exists")).expect("primary remains decodable");
|
||||
assert_eq!(
|
||||
primary.info.scan_execution_digest,
|
||||
Some(DataUsageScanPlanDigest([if changed_copy == "primary" { 9 } else { 3 }; 32])),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers() {
|
||||
@@ -2759,7 +2400,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
|
||||
.expect("seed legacy set cache");
|
||||
let mut persisted = DataUsageCache::default();
|
||||
let initial = persisted
|
||||
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("capture the shared starting revision");
|
||||
let mut fresh = legacy.clone();
|
||||
@@ -2785,7 +2426,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
|
||||
assert_eq!(published.find("photos").expect("published bucket").size, 20);
|
||||
assert_eq!(published.info.scan_execution_digest, fresh.info.scan_execution_digest);
|
||||
let current = persisted
|
||||
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
|
||||
.await
|
||||
.expect("capture the current revision for the unidentified execution");
|
||||
|
||||
@@ -2853,7 +2494,6 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
|
||||
digest: DataUsageScanPlanDigest([6; 32]),
|
||||
bucket_coverage_digest: DataUsageScanPlanDigest([6; 32]),
|
||||
checkpoint_coverage_digest: DataUsageScanPlanDigest([6; 32]),
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
requires_full_scan: false,
|
||||
execution_digest: empty_execution,
|
||||
leader_epoch: 11,
|
||||
@@ -3077,130 +2717,6 @@ fn scoped_scan_requires_a_complete_authoritative_baseline_with_exact_set_provena
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scoped_scan_baseline_rejections_preserve_the_existing_first_failed_check() {
|
||||
let source = DataUsageCacheSource::new(1, 2);
|
||||
let expected_sources = HashSet::from([source]);
|
||||
let digest = DataUsageScanPlanDigest([9; 32]);
|
||||
let baseline = complete_usage_baseline(source, digest, 7, 11);
|
||||
let snapshot: DataUsageInfo = serde_json::from_slice(&baseline).expect("decode complete baseline");
|
||||
let proof = ScannerCacheBaselineProof {
|
||||
authoritative_data: Some(&baseline),
|
||||
observed_candidate_data: None,
|
||||
expected_sources: &expected_sources,
|
||||
leader_epoch: 11,
|
||||
want_cycle: 8,
|
||||
scan_plan_digest: digest,
|
||||
};
|
||||
type RejectionCase = (ScannerBaselineRejection, fn(&mut DataUsageInfo));
|
||||
let cases: [RejectionCase; 13] = [
|
||||
(ScannerBaselineRejection::Incomplete, |s| s.usage_snapshot_complete = false),
|
||||
(ScannerBaselineRejection::Partial, |s| s.usage_snapshot_partial = true),
|
||||
(ScannerBaselineRejection::Marks, |s| s.usage_snapshot_converged = None),
|
||||
(ScannerBaselineRejection::Epoch, |s| s.scanner_epoch = Some(10)),
|
||||
(ScannerBaselineRejection::SourceCount, |s| s.usage_snapshot_set_states.clear()),
|
||||
(ScannerBaselineRejection::PlanMissing, |s| {
|
||||
s.usage_snapshot_set_states[0].scan_plan_digest = None
|
||||
}),
|
||||
(ScannerBaselineRejection::PlanMismatch, |s| {
|
||||
s.usage_snapshot_set_states[0].scan_plan_digest = Some([8; 32])
|
||||
}),
|
||||
(ScannerBaselineRejection::SourceMembership, |s| {
|
||||
s.usage_snapshot_set_states[0].set_index = 3
|
||||
}),
|
||||
(ScannerBaselineRejection::SetIncomplete, |s| {
|
||||
s.usage_snapshot_set_states[0].complete = false
|
||||
}),
|
||||
(ScannerBaselineRejection::SetTombstone, |s| {
|
||||
s.usage_snapshot_set_states[0].tombstone = true
|
||||
}),
|
||||
(ScannerBaselineRejection::SetEpoch, |s| {
|
||||
s.usage_snapshot_set_states[0].scanner_epoch = None
|
||||
}),
|
||||
(ScannerBaselineRejection::SetCycle, |s| {
|
||||
s.usage_snapshot_set_states[0].scanner_cycle = Some(9)
|
||||
}),
|
||||
(ScannerBaselineRejection::SetCycle, |s| {
|
||||
s.usage_snapshot_set_states[0].scanner_cycle = None
|
||||
}),
|
||||
];
|
||||
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&snapshot, proof, None), Ok(digest));
|
||||
for (expected, mutate) in cases {
|
||||
let mut invalid = snapshot.clone();
|
||||
mutate(&mut invalid);
|
||||
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None), Err(expected));
|
||||
assert_eq!(complete_scanner_cache_snapshot_plan_digest(&invalid, proof, None), None);
|
||||
}
|
||||
let mut invalid = snapshot.clone();
|
||||
invalid.usage_snapshot_partial = true;
|
||||
invalid.scanner_epoch = Some(10);
|
||||
assert_eq!(
|
||||
validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None),
|
||||
Err(ScannerBaselineRejection::Partial)
|
||||
);
|
||||
|
||||
let two_sources = HashSet::from([source, DataUsageCacheSource::new(1, 3)]);
|
||||
let mut two_sets = snapshot.clone();
|
||||
two_sets
|
||||
.usage_snapshot_set_states
|
||||
.push(two_sets.usage_snapshot_set_states[0].clone());
|
||||
let two_set_proof = ScannerCacheBaselineProof {
|
||||
expected_sources: &two_sources,
|
||||
..proof
|
||||
};
|
||||
assert_eq!(
|
||||
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
|
||||
Err(ScannerBaselineRejection::SourceDuplicate)
|
||||
);
|
||||
two_sets.usage_snapshot_set_states[1].set_index = 3;
|
||||
two_sets.usage_snapshot_set_states[1].scan_plan_digest = Some([8; 32]);
|
||||
assert_eq!(
|
||||
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
|
||||
Err(ScannerBaselineRejection::SetPlan)
|
||||
);
|
||||
|
||||
let undecodable = Bytes::from_static(b"not-json");
|
||||
for (data, expected) in [
|
||||
(None, ScannerBaselineRejection::Missing),
|
||||
(Some(&undecodable), ScannerBaselineRejection::Decode),
|
||||
] {
|
||||
assert_eq!(
|
||||
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
|
||||
authoritative_data: data,
|
||||
..proof
|
||||
}),
|
||||
Err(ScannerBaselineValidationFailure {
|
||||
authoritative: expected,
|
||||
observed: None
|
||||
})
|
||||
);
|
||||
}
|
||||
let mut wrong_epoch = snapshot;
|
||||
wrong_epoch.scanner_epoch = Some(10);
|
||||
let wrong_epoch = Bytes::from(serde_json::to_vec(&wrong_epoch).expect("encode stale epoch baseline"));
|
||||
let stale_proof = ScannerCacheBaselineProof {
|
||||
authoritative_data: Some(&wrong_epoch),
|
||||
..proof
|
||||
};
|
||||
assert_eq!(
|
||||
validate_scanner_cache_baseline_plan_digest(stale_proof),
|
||||
Err(ScannerBaselineValidationFailure {
|
||||
authoritative: ScannerBaselineRejection::Epoch,
|
||||
observed: Some(ScannerBaselineRejection::Missing),
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
|
||||
observed_candidate_data: Some(&undecodable),
|
||||
..stale_proof
|
||||
}),
|
||||
Err(ScannerBaselineValidationFailure {
|
||||
authoritative: ScannerBaselineRejection::Epoch,
|
||||
observed: Some(ScannerBaselineRejection::Decode)
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_baseline() {
|
||||
let source = DataUsageCacheSource::new(1, 2);
|
||||
@@ -3634,12 +3150,6 @@ fn distributed_segment_reuse_activation_keeps_remote_dirty_buckets_at_bucket_sco
|
||||
result.scope.prefix_scope_for("photos").is_some(),
|
||||
"durable local producer proof may activate local segment reuse after distributed ACK capability evidence"
|
||||
);
|
||||
assert!(
|
||||
result
|
||||
.segment_reuse_activation_preflight
|
||||
.expect("actual scope proof is retained for observation")
|
||||
.scanner_segment_reuse_activated
|
||||
);
|
||||
assert!(
|
||||
result.scope.prefix_scope_for("archive").is_none(),
|
||||
"remote dirty usage has only bucket-granularity evidence and must not be narrowed to a local prefix"
|
||||
@@ -4058,10 +3568,6 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
|
||||
.checked_flatten("stable")
|
||||
.expect("selected bucket placeholder should exist");
|
||||
assert_eq!((stable.size, stable.objects), (0, 0));
|
||||
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
|
||||
prepared.record_cold_bucket_reuse(&observation);
|
||||
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
|
||||
|
||||
assert!(prepared.cache.find("stable/prefix").is_none());
|
||||
assert_eq!(prepared.cache.find("dirty").map(|entry| (entry.size, entry.objects)), Some((0, 0)));
|
||||
assert!(prepared.cache.find("deleted").is_none());
|
||||
@@ -4073,124 +3579,6 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
|
||||
assert_eq!(prepared.cache.info.lkg_scan_plan_digest, Some(baseline_digest));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cold_bucket_reuse_observation_rejects_unbound_or_failed_preparation() {
|
||||
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
|
||||
let mut old_cache = complete_set_usage_cache(&[("stable", 10), ("dirty", 20)], baseline_digest);
|
||||
old_cache.info.scan_bucket_incarnations = test_bucket_incarnations(&["stable", "dirty"]);
|
||||
let incarnations = old_cache.info.scan_bucket_incarnations.clone();
|
||||
let buckets = vec![
|
||||
bucket_info_with_created_time("stable"),
|
||||
bucket_info_with_created_time("dirty"),
|
||||
];
|
||||
let scope = ScannerBucketScanScope {
|
||||
selected_buckets: Some(Arc::new(HashSet::from(["dirty".to_string()]))),
|
||||
selected_bucket_prefixes: None,
|
||||
baseline_scan_plan_digest: Some(baseline_digest),
|
||||
};
|
||||
let generation = ScannerSetCacheGeneration {
|
||||
want_cycle: 8,
|
||||
leader_epoch: 11,
|
||||
tier_registry_generation: 13,
|
||||
source: DataUsageCacheSource::new(1, 2),
|
||||
scan_plan_digest: DataUsageScanPlanDigest([2; 32]),
|
||||
};
|
||||
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
|
||||
let prepared = prepare_scoped_set_scan(&old_cache, &buckets, &buckets, &scope, generation, Some(&incarnations))
|
||||
.expect("bound complete cache permits a cold bucket copy");
|
||||
prepared.record_cold_bucket_reuse(&observation);
|
||||
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 1);
|
||||
|
||||
for case in ["incomplete", "duplicate", "nil_incarnation"] {
|
||||
let mut invalid_cache = old_cache.clone();
|
||||
let mut invalid_buckets = buckets.clone();
|
||||
let mut invalid_incarnations = incarnations.clone();
|
||||
match case {
|
||||
"incomplete" => invalid_cache.info.snapshot_complete = false,
|
||||
"duplicate" => invalid_buckets.push(buckets[0].clone()),
|
||||
"nil_incarnation" => {
|
||||
invalid_incarnations.insert("stable".to_string(), Uuid::nil());
|
||||
}
|
||||
_ => unreachable!("fixed test cases"),
|
||||
}
|
||||
let prepared = prepare_scoped_set_scan(
|
||||
&invalid_cache,
|
||||
&invalid_buckets,
|
||||
&invalid_buckets,
|
||||
&scope,
|
||||
generation,
|
||||
Some(&invalid_incarnations),
|
||||
);
|
||||
assert!(prepared.is_none(), "{case} must not return a reusable prepared cache");
|
||||
if let Some(prepared) = prepared {
|
||||
prepared.record_cold_bucket_reuse(&observation);
|
||||
}
|
||||
assert_eq!(
|
||||
observation.snapshot().bucket_reuse["scoped_cold_bucket"].count,
|
||||
1,
|
||||
"{case} does not count completed reuse"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn current_root_reuse_counts_only_successful_collector_delivery() {
|
||||
use rustfs_scanner_metrics::scan_observation::ScannerScanObservation;
|
||||
|
||||
let observation = ScannerScanObservation::default();
|
||||
let cache = DataUsageCache::default();
|
||||
let pending = AtomicBool::new(false);
|
||||
let root = || DataUsageEntryInfo {
|
||||
name: "bucket".to_string(),
|
||||
parent: DATA_USAGE_ROOT.to_string(),
|
||||
entry: DataUsageEntry {
|
||||
size: 10,
|
||||
objects: 2,
|
||||
..Default::default()
|
||||
},
|
||||
bucket_incarnation: Some(Uuid::from_u128(7)),
|
||||
tier_registry_generation: None,
|
||||
value_origin: None,
|
||||
};
|
||||
let (sender, mut receiver) = mpsc::channel(1);
|
||||
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
|
||||
.await
|
||||
.expect("current root is delivered to the collector");
|
||||
let delivered = receiver.recv().await.expect("collector received the root");
|
||||
assert_eq!((delivered.entry.size, delivered.entry.objects), (10, 2));
|
||||
assert_eq!(observation.snapshot().bucket_reuse["current_root"].count, 1);
|
||||
drop(receiver);
|
||||
assert!(
|
||||
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
assert_eq!(
|
||||
observation.snapshot().bucket_reuse["current_root"].count,
|
||||
1,
|
||||
"closed collector is not completed reuse"
|
||||
);
|
||||
|
||||
let (sender, _receiver) = mpsc::channel(1);
|
||||
sender.send(root()).await.expect("fill bounded collector channel");
|
||||
{
|
||||
let mut send = Box::pin(super::io_cache::send_current_cache_root_entry(
|
||||
&sender,
|
||||
root(),
|
||||
&cache,
|
||||
&pending,
|
||||
&observation,
|
||||
));
|
||||
assert!(futures::poll!(send.as_mut()).is_pending(), "full collector keeps reuse unfinished");
|
||||
}
|
||||
assert_eq!(
|
||||
observation.snapshot().bucket_reuse["current_root"].count,
|
||||
1,
|
||||
"dropping unfinished delivery is not reuse"
|
||||
);
|
||||
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
|
||||
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
|
||||
@@ -4232,12 +3620,6 @@ fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
|
||||
)
|
||||
.expect("matching bucket incarnations should authorize cold bucket reuse");
|
||||
|
||||
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
|
||||
prepared.record_cold_bucket_reuse(&observation);
|
||||
let observed = observation.snapshot();
|
||||
assert_eq!(observed.bucket_reuse["scoped_cold_bucket"].count, 1);
|
||||
assert_eq!(observed.bucket_reuse["current_root"].count, 0);
|
||||
assert!(observed.cache_reuse.values().all(|reuse| reuse.count == 0));
|
||||
assert_eq!(prepared.buckets.iter().map(|bucket| bucket.name.as_str()).collect::<Vec<_>>(), ["dirty"]);
|
||||
let stable = prepared
|
||||
.cache
|
||||
@@ -5243,36 +4625,12 @@ async fn get_size_treats_missing_metadata_as_skip_file() {
|
||||
debug: false,
|
||||
};
|
||||
|
||||
let before = global_metrics().scan_observation().snapshot().metadata_reads;
|
||||
let err = disk
|
||||
.get_size(item.clone())
|
||||
.get_size(item)
|
||||
.await
|
||||
.expect_err("missing metadata should be skipped instead of reported as a scanner failure");
|
||||
assert!(matches!(err, StorageError::Io(ref io) if io.to_string() == SCANNER_SKIP_FILE_ERROR));
|
||||
|
||||
let missing = global_metrics().scan_observation().snapshot().metadata_reads;
|
||||
assert_eq!(missing.attempts, before.attempts + 1);
|
||||
assert_eq!(missing.not_found, before.not_found + 1);
|
||||
assert_eq!(missing.bytes, before.bytes);
|
||||
|
||||
#[cfg(unix)]
|
||||
{
|
||||
tokio::fs::create_dir_all(&object_dir)
|
||||
.await
|
||||
.expect("recreate object directory");
|
||||
std::os::unix::fs::symlink(STORAGE_FORMAT_FILE, &metadata_path).expect("inject a metadata symlink loop");
|
||||
let err = disk
|
||||
.get_size(item)
|
||||
.await
|
||||
.expect_err("symlink loop must remain a read failure");
|
||||
assert!(is_scanner_metadata_transient_error(&err), "read error: {err:?}");
|
||||
let failed = global_metrics().scan_observation().snapshot().metadata_reads;
|
||||
assert_eq!(failed.attempts, missing.attempts + 1);
|
||||
assert_eq!(failed.errors, missing.errors + 1);
|
||||
assert_eq!(failed.succeeded, before.succeeded);
|
||||
assert_eq!(failed.bytes, before.bytes);
|
||||
}
|
||||
|
||||
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
|
||||
}
|
||||
|
||||
@@ -5323,7 +4681,6 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
|
||||
debug: false,
|
||||
};
|
||||
|
||||
let before = global_metrics().scan_observation().snapshot().metadata_reads;
|
||||
for contents in [b"".as_slice(), b"not-valid-filemeta".as_slice()] {
|
||||
tokio::fs::write(&metadata_path, contents)
|
||||
.await
|
||||
@@ -5335,16 +4692,6 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
|
||||
assert!(is_scanner_metadata_corrupt_error(&err));
|
||||
}
|
||||
|
||||
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
|
||||
assert_eq!(reads.attempts, before.attempts + 2);
|
||||
assert_eq!(
|
||||
reads.succeeded,
|
||||
before.succeeded + 2,
|
||||
"storage reads succeed before decoding rejects corruption"
|
||||
);
|
||||
assert_eq!(reads.errors, before.errors);
|
||||
assert_eq!(reads.bytes, before.bytes + 18);
|
||||
|
||||
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
|
||||
}
|
||||
|
||||
@@ -5417,17 +4764,11 @@ async fn get_size_counts_delete_markers_separately_from_versions() {
|
||||
debug: false,
|
||||
};
|
||||
|
||||
let before = global_metrics().scan_observation().snapshot().metadata_reads;
|
||||
let expected_bytes = tokio::fs::metadata(&metadata_path).await.expect("stat stored metadata").len();
|
||||
let summary = disk.get_size(item).await.expect("scanner should read versioned metadata");
|
||||
|
||||
assert_eq!(summary.versions, 2);
|
||||
assert_eq!(summary.delete_markers, 1);
|
||||
assert_eq!(summary.total_size, 30);
|
||||
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
|
||||
assert_eq!(reads.attempts, before.attempts + 1);
|
||||
assert_eq!(reads.succeeded, before.succeeded + 1);
|
||||
assert_eq!(reads.bytes, before.bytes + expected_bytes);
|
||||
|
||||
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
|
||||
}
|
||||
|
||||
@@ -14,143 +14,6 @@ use super::*;
|
||||
const HOT_OBJECTS: [&str; 6] = ["p0/a", "p1/b", "p2/c", "p3/d", "p4/e", "p5/f"];
|
||||
const LEADER_EPOCH: u64 = 11;
|
||||
|
||||
#[test]
|
||||
fn completed_checkpoint_reuse_requires_unpublished_unchanged_coverage() {
|
||||
let source = DataUsageCacheSource::new(0, 0);
|
||||
let plan = DataUsageScanPlanDigest([1; 32]);
|
||||
let coverage = DataUsageScanPlanDigest([2; 32]);
|
||||
let identity = crate::DataUsageScanIdentity {
|
||||
version: 1,
|
||||
bucket_incarnation: Uuid::new_v4(),
|
||||
set_layout: DataUsageScanPlanDigest([3; 32]),
|
||||
publication_epoch: 1,
|
||||
tier_registry_generation: 2,
|
||||
scan_mode: HealScanMode::Normal,
|
||||
};
|
||||
let mut saved = DataUsageCache {
|
||||
info: DataUsageCacheInfo {
|
||||
name: "completed".to_string(),
|
||||
next_cycle: 8,
|
||||
leader_epoch: 11,
|
||||
last_update: Some(SystemTime::now()),
|
||||
source: Some(source),
|
||||
snapshot_complete: true,
|
||||
scan_plan_digest: Some(plan),
|
||||
scan_coverage_digest: Some(coverage),
|
||||
scan_identity: Some(identity),
|
||||
tier_registry_generation: Some(2),
|
||||
cache_key_format: DATA_USAGE_CACHE_KEY_FORMAT,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
};
|
||||
saved.replace(
|
||||
"completed",
|
||||
"",
|
||||
DataUsageEntry {
|
||||
objects: 3,
|
||||
size: 48,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
for case in [
|
||||
"unpublished",
|
||||
"published",
|
||||
"missing_floor",
|
||||
"restart",
|
||||
"mutation",
|
||||
"legacy",
|
||||
"newer_cycle",
|
||||
"newer_leader",
|
||||
"same_leader",
|
||||
"tier",
|
||||
"incarnation",
|
||||
"source",
|
||||
"deep",
|
||||
"corrupt",
|
||||
"partial",
|
||||
] {
|
||||
let mut cache = saved.clone();
|
||||
let mut options = DataUsageCacheReuseOptions {
|
||||
require_source: true,
|
||||
tier_registry_generation: Some(2),
|
||||
checkpoint_identity: Some(identity),
|
||||
checkpoint_plan_digest: Some(coverage),
|
||||
completed_checkpoint_cycle_floor: Some(7),
|
||||
};
|
||||
match case {
|
||||
"published" => options.completed_checkpoint_cycle_floor = Some(8),
|
||||
"missing_floor" => options.completed_checkpoint_cycle_floor = None,
|
||||
"restart" => cache.info.scan_plan_digest = Some(DataUsageScanPlanDigest([4; 32])),
|
||||
"mutation" => cache.info.scan_coverage_digest = Some(DataUsageScanPlanDigest([4; 32])),
|
||||
"legacy" => cache.info.scan_coverage_digest = None,
|
||||
"newer_cycle" => cache.info.next_cycle = 10,
|
||||
"newer_leader" => cache.info.leader_epoch = 13,
|
||||
"same_leader" => cache.info.leader_epoch = 12,
|
||||
"tier" => options.tier_registry_generation = Some(3),
|
||||
"incarnation" => options.checkpoint_identity.as_mut().expect("identity").bucket_incarnation = Uuid::new_v4(),
|
||||
"source" => cache.info.source = Some(DataUsageCacheSource::new(1, 0)),
|
||||
"deep" => options.checkpoint_identity.as_mut().expect("identity").scan_mode = HealScanMode::Deep,
|
||||
"corrupt" => cache.cache.clear(),
|
||||
"partial" => cache.info.scan_resume_after = Some("completed/a".to_string()),
|
||||
"unpublished" => {}
|
||||
_ => unreachable!("all fixture cases are handled"),
|
||||
}
|
||||
let state = current_cache_root_or_prepare_with_generation(&mut cache, "completed", source, 9, 12, plan, options);
|
||||
assert_eq!(matches!(state, DataUsageCacheScanState::Current(_)), case == "unpublished", "{case}");
|
||||
if case == "unpublished" {
|
||||
assert_eq!(cache.info.next_cycle, 8, "adoption must not extend the saved receipt's lifetime");
|
||||
let DataUsageCacheScanState::Current(root) = state else { unreachable!() };
|
||||
assert_eq!(root.entry.objects, 3);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn completed_checkpoint_floor_requires_an_authoritative_baseline() {
|
||||
let source = DataUsageCacheSource::new(0, 0);
|
||||
let sources = HashSet::from([source]);
|
||||
let baseline: serde_json::Value =
|
||||
serde_json::from_slice(&complete_usage_baseline(source, DataUsageScanPlanDigest([1; 32]), 7, 11))
|
||||
.expect("complete baseline");
|
||||
for case in [
|
||||
"complete",
|
||||
"bootstrap",
|
||||
"observational",
|
||||
"partial",
|
||||
"incomplete",
|
||||
"legacy",
|
||||
"no_cycle",
|
||||
] {
|
||||
let mut value = baseline.clone();
|
||||
match case {
|
||||
"bootstrap" => {
|
||||
value = serde_json::to_value(crate::scanner::scanner_usage_bootstrap_marker(SystemTime::now(), Some(11)))
|
||||
.expect("bootstrap marker");
|
||||
}
|
||||
"observational" => value["usage_snapshot_observational"] = true.into(),
|
||||
"partial" => value["usage_snapshot_partial"] = true.into(),
|
||||
"incomplete" => value["usage_snapshot_complete"] = false.into(),
|
||||
"legacy" => value["scanner_epoch"] = serde_json::Value::Null,
|
||||
"no_cycle" => value["scanner_cycle"] = serde_json::Value::Null,
|
||||
"complete" => {}
|
||||
_ => unreachable!("all fixture cases are handled"),
|
||||
}
|
||||
let bytes = serde_json::to_vec(&value).expect("baseline fixture");
|
||||
assert_eq!(
|
||||
completed_checkpoint_cycle_floor(Some(&bytes), &sources, 9, 12),
|
||||
match case {
|
||||
"complete" => Some(7),
|
||||
"bootstrap" => Some(0),
|
||||
_ => None,
|
||||
},
|
||||
"{case}"
|
||||
);
|
||||
}
|
||||
assert_eq!(completed_checkpoint_cycle_floor(None, &sources, 9, 12), None);
|
||||
assert_eq!(completed_checkpoint_cycle_floor(Some(b"invalid"), &sources, 9, 12), None);
|
||||
}
|
||||
|
||||
async fn put_fixture_object(store: &Arc<ECStore>, bucket: &str, object: &str) {
|
||||
let mut reader = ScannerPutObjReader::from_vec(vec![7; 16]);
|
||||
store
|
||||
@@ -164,18 +27,6 @@ async fn run_full_cycle(
|
||||
cycle: u64,
|
||||
max_objects: Option<u64>,
|
||||
baseline: Option<&DataUsageInfo>,
|
||||
) -> (ScannerCycleResult, Option<DataUsageInfo>, Arc<ScannerCycleBudget>) {
|
||||
let baseline = baseline
|
||||
.map(|snapshot| Bytes::from(serde_json::to_vec(snapshot).expect("the actual candidate must encode as a baseline")));
|
||||
run_full_cycle_at_epoch(store, cycle, LEADER_EPOCH, max_objects, baseline).await
|
||||
}
|
||||
|
||||
async fn run_full_cycle_at_epoch(
|
||||
store: &Arc<ECStore>,
|
||||
cycle: u64,
|
||||
leader_epoch: u64,
|
||||
max_objects: Option<u64>,
|
||||
persisted_usage_baseline: Option<Bytes>,
|
||||
) -> (ScannerCycleResult, Option<DataUsageInfo>, Arc<ScannerCycleBudget>) {
|
||||
let ctx = CancellationToken::new();
|
||||
let budget = ScannerCycleBudget::new_with_progress_tracking(
|
||||
@@ -198,10 +49,12 @@ async fn run_full_cycle_at_epoch(
|
||||
budget: Arc::clone(&budget),
|
||||
updates,
|
||||
want_cycle: cycle,
|
||||
leader_epoch,
|
||||
leader_epoch: LEADER_EPOCH,
|
||||
scan_mode: HealScanMode::Normal,
|
||||
scan_scope: ScannerBucketScanScope::default(),
|
||||
persisted_usage_baseline,
|
||||
persisted_usage_baseline: baseline.map(|snapshot| {
|
||||
Bytes::from(serde_json::to_vec(snapshot).expect("the actual candidate must encode as a baseline"))
|
||||
}),
|
||||
observed_usage_candidate: None,
|
||||
requires_full_scan: true,
|
||||
service_cohort: None,
|
||||
@@ -220,265 +73,6 @@ async fn run_full_cycle_at_epoch(
|
||||
(result, snapshot, budget)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn checkpoint_small_budget_accumulates_completed_buckets_across_leaders() {
|
||||
let (_temp_dir, store) = setup_local_scanner_store(1, 1).await;
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
let names = ["bounded-a", "bounded-b", "bounded-c", "bounded-d"];
|
||||
for bucket in names {
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("bounded fixture bucket");
|
||||
for object in ["a", "b", "c"] {
|
||||
put_fixture_object(&store, bucket, object).await;
|
||||
}
|
||||
record_dirty_usage_bucket(bucket);
|
||||
}
|
||||
wait_for_namespace_commit_tails(store.as_ref()).await;
|
||||
let baseline = Bytes::from(
|
||||
serde_json::to_vec(&crate::scanner::scanner_usage_bootstrap_marker(SystemTime::now(), Some(LEADER_EPOCH)))
|
||||
.expect("bootstrap baseline"),
|
||||
);
|
||||
let mut observations = Vec::new();
|
||||
for cycle in 1..=20 {
|
||||
let (result, usage, budget) =
|
||||
run_full_cycle_at_epoch(&store, cycle, LEADER_EPOCH + cycle, Some(2), Some(baseline.clone())).await;
|
||||
observations.push((cycle, result.status, budget.progress()));
|
||||
if result.status == ScannerCycleStatus::Complete {
|
||||
let usage = usage.expect("completed bounded sweep publishes usage");
|
||||
assert_eq!(usage.objects_total_count, 12);
|
||||
assert_eq!(usage.usage_snapshot_converged, Some(true));
|
||||
for bucket in names {
|
||||
assert_eq!(usage.buckets_usage[bucket].objects_count, 3);
|
||||
}
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
return;
|
||||
}
|
||||
}
|
||||
panic!("fixed budget must accumulate the completed buckets: {observations:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn checkpoint_small_budget_accumulates_across_process_restarts() {
|
||||
const ROOT_ENV: &str = "RUSTFS_CHECKPOINT_RESTART_ROOT";
|
||||
const CYCLE_ENV: &str = "RUSTFS_CHECKPOINT_RESTART_CYCLE";
|
||||
const TEST_NAME: &str =
|
||||
"scanner_io::tests::checkpoint_coverage_identity::checkpoint_small_budget_accumulates_across_process_restarts";
|
||||
let names = ["restart-a", "restart-b", "restart-c", "restart-d"];
|
||||
if let Some(root) = std::env::var_os(ROOT_ENV) {
|
||||
let root = std::path::PathBuf::from(root);
|
||||
let cycle: u64 = std::env::var(CYCLE_ENV)
|
||||
.expect("worker cycle")
|
||||
.parse()
|
||||
.expect("numeric cycle");
|
||||
let store = open_local_scanner_store(&root, 1, 1, true).await;
|
||||
let registry = scanner_set_disk_inventory(&store.pools[0].disk_set[0])
|
||||
.await
|
||||
.into_iter()
|
||||
.map(Some)
|
||||
.collect::<Vec<_>>();
|
||||
let previous = rustfs_heal::heal::consume_read_only_scanner_handoff("test-node", ®istry)
|
||||
.await
|
||||
.expect("consume previous receipt");
|
||||
if let Some(previous) = previous {
|
||||
assert!(crate::restore_scanner_checkpoint_instance_id(&previous));
|
||||
}
|
||||
if cycle == 0 {
|
||||
for bucket in names {
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("fixture bucket");
|
||||
for object in ["a", "b", "c"] {
|
||||
put_fixture_object(&store, bucket, object).await;
|
||||
}
|
||||
}
|
||||
wait_for_namespace_commit_tails(store.as_ref()).await;
|
||||
rustfs_heal::heal::clear_unclean_shutdown_markers().await;
|
||||
return;
|
||||
}
|
||||
let mutation_path = root.join("mutate.json");
|
||||
let mutated = root.join("mutated");
|
||||
if mutation_path.exists() && !mutated.exists() {
|
||||
let mutation: serde_json::Value =
|
||||
serde_json::from_slice(&tokio::fs::read(&mutation_path).await.expect("mutation request")).expect("mutation JSON");
|
||||
let bucket = mutation["bucket"].as_str().expect("mutation bucket");
|
||||
assert!(names.contains(&bucket));
|
||||
put_fixture_object(&store, bucket, "after-checkpoint").await;
|
||||
wait_for_namespace_commit_tails(store.as_ref()).await;
|
||||
assert!(store.scanner_namespace_mutation_generation() > 0 || !crate::scanner_process_was_read_only());
|
||||
if mutation["clean"] == true {
|
||||
rustfs_heal::heal::clear_unclean_shutdown_markers().await;
|
||||
}
|
||||
tokio::fs::write(&mutated, b"committed").await.expect("mutation receipt");
|
||||
let report = serde_json::json!({"instance": scanner_activity_epoch(), "mutated": true});
|
||||
tokio::fs::write(
|
||||
root.join(format!("round-{cycle}.json")),
|
||||
serde_json::to_vec(&report).expect("writer report"),
|
||||
)
|
||||
.await
|
||||
.expect("writer report file");
|
||||
return;
|
||||
}
|
||||
let baseline = if root.join("published.json").exists() {
|
||||
Bytes::from(tokio::fs::read(root.join("published.json")).await.expect("published floor"))
|
||||
} else {
|
||||
Bytes::from(
|
||||
serde_json::to_vec(&crate::scanner::scanner_usage_bootstrap_marker(SystemTime::now(), Some(LEADER_EPOCH)))
|
||||
.expect("bootstrap baseline"),
|
||||
)
|
||||
};
|
||||
let (result, usage, budget) = run_full_cycle_at_epoch(&store, cycle, LEADER_EPOCH + cycle, Some(2), Some(baseline)).await;
|
||||
let complete = result.status == ScannerCycleStatus::Complete;
|
||||
if complete {
|
||||
let usage = usage.expect("complete sweep must publish");
|
||||
let mutation: Option<serde_json::Value> = if mutated.exists() {
|
||||
Some(
|
||||
serde_json::from_slice(&tokio::fs::read(&mutation_path).await.expect("mutation request"))
|
||||
.expect("mutation JSON"),
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
assert_eq!(usage.objects_total_count, if mutation.is_some() { 13 } else { 12 });
|
||||
assert_eq!(usage.usage_snapshot_converged, Some(true));
|
||||
for bucket in names {
|
||||
let expected = if mutation.as_ref().is_some_and(|mutation| mutation["bucket"] == bucket) {
|
||||
4
|
||||
} else {
|
||||
3
|
||||
};
|
||||
assert_eq!(usage.buckets_usage[bucket].objects_count, expected, "{bucket}");
|
||||
}
|
||||
tokio::fs::write(root.join("published.json"), serde_json::to_vec(&usage).expect("complete usage"))
|
||||
.await
|
||||
.expect("published baseline");
|
||||
}
|
||||
let mut completed_buckets = Vec::new();
|
||||
for bucket in names {
|
||||
let mut cache = DataUsageCache::default();
|
||||
if cache
|
||||
.load(store.pools[0].disk_set[0].clone(), &path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]))
|
||||
.await
|
||||
.is_ok()
|
||||
&& cache.info.snapshot_complete
|
||||
&& !cache.has_scan_position()
|
||||
{
|
||||
completed_buckets.push(bucket);
|
||||
}
|
||||
}
|
||||
assert_eq!(
|
||||
store.scanner_namespace_mutation_generation(),
|
||||
0,
|
||||
"scanner must not mint namespace mutations"
|
||||
);
|
||||
assert!(
|
||||
crate::scanner_process_was_read_only(),
|
||||
"a scanner-only process must be eligible for a read-only handoff"
|
||||
);
|
||||
rustfs_heal::heal::save_read_only_scanner_handoff("test-node", crate::scanner_checkpoint_instance_id())
|
||||
.await
|
||||
.expect("persist read-only handoff");
|
||||
let report = serde_json::json!({
|
||||
"instance": scanner_activity_epoch(), "checkpoint_instance": crate::scanner_checkpoint_instance_id(),
|
||||
"complete": complete, "objects_processed": budget.progress().0, "completed_buckets": completed_buckets,
|
||||
});
|
||||
tokio::fs::write(
|
||||
root.join(format!("round-{cycle}.json")),
|
||||
serde_json::to_vec(&report).expect("worker report"),
|
||||
)
|
||||
.await
|
||||
.expect("worker report file");
|
||||
return;
|
||||
}
|
||||
for scenario in ["quiet", "clean-writer", "crashed-writer"] {
|
||||
let root = tempfile::tempdir().expect("restart fixture root");
|
||||
let mut instances = HashSet::new();
|
||||
let mut observations = Vec::new();
|
||||
let mut injected = false;
|
||||
let mut complete = false;
|
||||
let mut before_mutation = None;
|
||||
let mut expect_fresh = false;
|
||||
for cycle in 0..=40 {
|
||||
let log_path = root.path().join(format!("round-{cycle}.log"));
|
||||
let output = std::fs::File::create(&log_path).expect("worker log");
|
||||
let mut child = std::process::Command::new(std::env::current_exe().expect("test executable"))
|
||||
.args(["--exact", TEST_NAME, "--nocapture"])
|
||||
.env(ROOT_ENV, root.path())
|
||||
.env(CYCLE_ENV, cycle.to_string())
|
||||
.env("RUST_MIN_STACK", "4194304")
|
||||
.stdout(output.try_clone().expect("worker stdout"))
|
||||
.stderr(output)
|
||||
.spawn()
|
||||
.expect("restart worker");
|
||||
let status = tokio::time::timeout(Duration::from_secs(75), async {
|
||||
loop {
|
||||
if let Some(status) = child.try_wait().expect("worker status") {
|
||||
break status;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(25)).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|_| {
|
||||
child.kill().expect("stop expired worker");
|
||||
child.wait().expect("reap expired worker");
|
||||
panic!("{scenario} round {cycle} exceeded its deadline");
|
||||
});
|
||||
assert!(
|
||||
status.success(),
|
||||
"{scenario} round {cycle}: {}",
|
||||
std::fs::read_to_string(log_path).expect("worker diagnostics")
|
||||
);
|
||||
if cycle == 0 {
|
||||
continue;
|
||||
}
|
||||
let report: serde_json::Value = serde_json::from_slice(
|
||||
&tokio::fs::read(root.path().join(format!("round-{cycle}.json")))
|
||||
.await
|
||||
.expect("worker report file"),
|
||||
)
|
||||
.expect("worker report JSON");
|
||||
assert!(instances.insert(report["instance"].as_str().expect("process identity").to_owned()));
|
||||
if report["mutated"] == true {
|
||||
expect_fresh = true;
|
||||
continue;
|
||||
}
|
||||
if expect_fresh {
|
||||
assert_ne!(
|
||||
before_mutation.as_ref(),
|
||||
Some(&report["checkpoint_instance"]),
|
||||
"writer must invalidate continuity"
|
||||
);
|
||||
assert_eq!(report["complete"], false, "old completed roots must not hide the mutation");
|
||||
expect_fresh = false;
|
||||
}
|
||||
if scenario != "quiet" && !injected && !report["completed_buckets"].as_array().expect("completed buckets").is_empty()
|
||||
{
|
||||
assert_eq!(report["complete"], false, "mutation must precede global publication");
|
||||
let mutation = serde_json::json!({"bucket": report["completed_buckets"][0], "clean": scenario == "clean-writer"});
|
||||
tokio::fs::write(root.path().join("mutate.json"), serde_json::to_vec(&mutation).expect("mutation request"))
|
||||
.await
|
||||
.expect("mutation request file");
|
||||
before_mutation = Some(report["checkpoint_instance"].clone());
|
||||
injected = true;
|
||||
}
|
||||
if report["complete"] == true {
|
||||
assert!(cycle > 1);
|
||||
assert!(scenario == "quiet" || injected);
|
||||
complete = true;
|
||||
eprintln!("{scenario}: exact convergence in {cycle} bounded fresh processes");
|
||||
break;
|
||||
}
|
||||
observations.push(report);
|
||||
}
|
||||
assert!(complete, "{scenario}: fixed budget did not converge in 40 restarts: {observations:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn checkpoint_resumes_across_a_maintenance_publication() {
|
||||
@@ -585,13 +179,13 @@ async fn checkpoint_resumes_across_a_maintenance_publication() {
|
||||
.await
|
||||
.expect("the fixture activity should be observable");
|
||||
assert_ne!(
|
||||
crate::scanner::scanner_activity_structural_digest(&before, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
crate::scanner::scanner_activity_structural_digest(&after, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
crate::scanner::scanner_activity_structural_digest(&before),
|
||||
crate::scanner::scanner_activity_structural_digest(&after),
|
||||
"a maintenance publication invalidates reusable results"
|
||||
);
|
||||
assert_eq!(
|
||||
crate::scanner::scanner_activity_coverage_digest(&before, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
crate::scanner::scanner_activity_coverage_digest(&after, crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
crate::scanner::scanner_activity_coverage_digest(&before),
|
||||
crate::scanner::scanner_activity_coverage_digest(&after),
|
||||
"a maintenance publication keeps the coverage plan"
|
||||
);
|
||||
|
||||
@@ -629,31 +223,6 @@ async fn checkpoint_resumes_across_a_maintenance_publication() {
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn remote_checkpoint_resumes_across_restart_and_maintenance_identity_changes() {
|
||||
use tokio::io::AsyncReadExt;
|
||||
|
||||
#[derive(serde::Deserialize)]
|
||||
struct Envelope {
|
||||
payload: Vec<u8>,
|
||||
}
|
||||
#[derive(Debug, serde::Deserialize)]
|
||||
struct Progress {
|
||||
objects_scanned: u64,
|
||||
}
|
||||
#[derive(Debug, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
enum FrameResult {
|
||||
Progress,
|
||||
Complete(serde::de::IgnoredAny),
|
||||
Partial,
|
||||
NamespaceNotFound,
|
||||
CycleAhead(serde::de::IgnoredAny),
|
||||
Error(serde::de::IgnoredAny),
|
||||
}
|
||||
#[derive(Debug, serde::Deserialize)]
|
||||
struct Frame {
|
||||
progress: Progress,
|
||||
result: FrameResult,
|
||||
}
|
||||
#[derive(serde::Serialize)]
|
||||
struct WorkerRequest<'a> {
|
||||
version: u16,
|
||||
@@ -676,24 +245,20 @@ async fn remote_checkpoint_resumes_across_restart_and_maintenance_identity_chang
|
||||
move || resolved.upgrade()
|
||||
)));
|
||||
rustfs_credentials::set_global_rpc_secret("remote-checkpoint-test-secret".to_string()).expect("test RPC secret");
|
||||
let bucket = "remote-checkpoint";
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("remote bucket");
|
||||
for object in HOT_OBJECTS {
|
||||
put_fixture_object(&store, bucket, object).await;
|
||||
}
|
||||
wait_for_namespace_commit_tails(store.as_ref()).await;
|
||||
let set = store.pools[0].disk_set[0].clone();
|
||||
let disk = scanner_set_disk_inventory(&set).await.pop().expect("local worker disk");
|
||||
let cache_name = path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]);
|
||||
|
||||
for (index, restart) in [true, false].into_iter().enumerate() {
|
||||
let bucket = if restart {
|
||||
"remote-checkpoint-restart"
|
||||
} else {
|
||||
"remote-checkpoint-maintenance"
|
||||
};
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("remote bucket");
|
||||
for object in HOT_OBJECTS {
|
||||
put_fixture_object(&store, bucket, object).await;
|
||||
}
|
||||
wait_for_namespace_commit_tails(store.as_ref()).await;
|
||||
let cache_name = path_join_buf(&[bucket, DATA_USAGE_CACHE_NAME]);
|
||||
let before = if restart {
|
||||
std::collections::BTreeMap::from([(
|
||||
"<local>".to_string(),
|
||||
@@ -704,64 +269,36 @@ async fn remote_checkpoint_resumes_across_restart_and_maintenance_identity_chang
|
||||
.await
|
||||
.expect("activity before maintenance")
|
||||
};
|
||||
let coverage = DataUsageScanPlanDigest(crate::scanner::scanner_activity_coverage_digest(
|
||||
&before,
|
||||
crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE,
|
||||
));
|
||||
let reuse = DataUsageScanPlanDigest(crate::scanner::scanner_activity_structural_digest(
|
||||
&before,
|
||||
crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE,
|
||||
));
|
||||
let cycle = 100 + u64::try_from(index).expect("fixture index") * 4;
|
||||
for step in 0..4_u64 {
|
||||
let resumed = step > 0;
|
||||
let coverage = DataUsageScanPlanDigest(crate::scanner::scanner_activity_coverage_digest(&before));
|
||||
let reuse = DataUsageScanPlanDigest(crate::scanner::scanner_activity_structural_digest(&before));
|
||||
let cycle = 100 + u64::try_from(index).expect("fixture index") * 2;
|
||||
for resumed in [false, true] {
|
||||
let activity = if resumed && restart {
|
||||
std::collections::BTreeMap::from([(
|
||||
"<local>".to_string(),
|
||||
crate::scanner::scanner_node_activity_for_tests("instance-after", 0, 0, false),
|
||||
)])
|
||||
} else if resumed {
|
||||
if step == 1 {
|
||||
record_scanner_maintenance_change(bucket);
|
||||
}
|
||||
record_scanner_maintenance_change(bucket);
|
||||
crate::scanner::probe_scanner_activity(store.as_ref(), false)
|
||||
.await
|
||||
.expect("activity after maintenance")
|
||||
} else {
|
||||
before.clone()
|
||||
};
|
||||
let requested_reuse = DataUsageScanPlanDigest(crate::scanner::scanner_activity_structural_digest(
|
||||
&activity,
|
||||
crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE,
|
||||
));
|
||||
let requested_reuse = DataUsageScanPlanDigest(crate::scanner::scanner_activity_structural_digest(&activity));
|
||||
assert_eq!(
|
||||
coverage,
|
||||
DataUsageScanPlanDigest(crate::scanner::scanner_activity_coverage_digest(
|
||||
&activity,
|
||||
crate::scanner::LOCAL_SCANNER_ACTIVITY_NODE
|
||||
))
|
||||
DataUsageScanPlanDigest(crate::scanner::scanner_activity_coverage_digest(&activity))
|
||||
);
|
||||
if resumed {
|
||||
assert_ne!(reuse, requested_reuse, "the event must invalidate completed cache reuse");
|
||||
}
|
||||
let next_cycle = cycle + step;
|
||||
let leader_epoch = LEADER_EPOCH + next_cycle;
|
||||
if step >= 2 {
|
||||
let baseline = if step == 2 {
|
||||
serde_json::to_vec(&crate::scanner::scanner_usage_bootstrap_marker(SystemTime::now(), Some(leader_epoch)))
|
||||
.expect("bootstrap baseline")
|
||||
} else {
|
||||
complete_usage_baseline(DataUsageCacheSource::new(0, 0), requested_reuse, cycle + 1, leader_epoch - 2)
|
||||
.to_vec()
|
||||
};
|
||||
save_config(store.clone(), crate::DATA_USAGE_OBJ_NAME_PATH.as_str(), baseline)
|
||||
.await
|
||||
.expect("authoritative aggregate floor");
|
||||
}
|
||||
let next_cycle = cycle + u64::from(resumed);
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_BLOOM_NAME_PATH.as_str(),
|
||||
crate::scanner::encode_scanner_cycle_fence_for_test(next_cycle, leader_epoch),
|
||||
crate::scanner::encode_scanner_cycle_fence_for_test(next_cycle, LEADER_EPOCH),
|
||||
)
|
||||
.await
|
||||
.expect("remote request cycle fence");
|
||||
@@ -773,39 +310,28 @@ async fn remote_checkpoint_resumes_across_restart_and_maintenance_identity_chang
|
||||
session_sequence: 0,
|
||||
bucket,
|
||||
next_cycle,
|
||||
leader_epoch,
|
||||
leader_epoch: LEADER_EPOCH,
|
||||
scan_plan_digest: requested_reuse,
|
||||
checkpoint_plan_digest: coverage,
|
||||
skip_healing: true,
|
||||
scan_mode: HealScanMode::Normal,
|
||||
budget: serde_json::json!({"max_objects": if step == 1 { None } else { Some(2_u64) }}),
|
||||
budget: serde_json::json!({"max_objects": if resumed { None } else { Some(2_u64) }}),
|
||||
};
|
||||
let body = rmp_serde::to_vec_named(&body).expect("remote worker request body");
|
||||
let request = crate::decode_remote_scanner_request(&body).expect("remote worker decodes coverage identity");
|
||||
let mut response = Vec::new();
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(60),
|
||||
crate::serve_remote_scanner_request(disk.clone(), request, &mut response, CancellationToken::new()),
|
||||
crate::serve_remote_scanner_request(disk.clone(), request, tokio::io::sink(), CancellationToken::new()),
|
||||
)
|
||||
.await
|
||||
.expect("bounded remote worker")
|
||||
.expect("remote worker stream");
|
||||
let mut frames = response.as_slice();
|
||||
let mut terminal: Option<Frame> = None;
|
||||
while !frames.is_empty() {
|
||||
let length = usize::try_from(frames.read_u32().await.expect("frame length")).expect("frame size");
|
||||
let envelope: Envelope = rmp_serde::from_slice(&frames[..length]).expect("response envelope");
|
||||
terminal = Some(rmp_serde::from_slice(&envelope.payload).expect("response frame"));
|
||||
frames = &frames[length..];
|
||||
}
|
||||
let terminal = terminal.expect("terminal worker response");
|
||||
let mut cache = DataUsageCache::default();
|
||||
cache
|
||||
.load(set.clone(), &cache_name)
|
||||
.await
|
||||
.expect("durable remote worker cache");
|
||||
if step == 1 || step == 2 {
|
||||
assert!(matches!(terminal.result, FrameResult::Complete(_)), "step {step}: {terminal:?}");
|
||||
if resumed {
|
||||
assert!(
|
||||
cache.info.snapshot_complete,
|
||||
"a resumed remote sweep must complete without mixed coverage"
|
||||
@@ -813,12 +339,7 @@ async fn remote_checkpoint_resumes_across_restart_and_maintenance_identity_chang
|
||||
assert!(cache.info.scan_progress.is_none());
|
||||
assert_eq!(cache.info.scan_plan_digest, Some(requested_reuse));
|
||||
assert_eq!(cache.flatten(cache.find(bucket).expect("bucket root")).objects, 6);
|
||||
if step == 2 {
|
||||
assert_eq!(terminal.progress.objects_scanned, 0);
|
||||
assert_eq!(cache.info.next_cycle, cycle + 1, "reuse must not renew the completed receipt");
|
||||
}
|
||||
} else {
|
||||
assert!(matches!(terminal.result, FrameResult::Partial), "step {step}: {terminal:?}");
|
||||
assert!(!cache.info.snapshot_complete);
|
||||
assert!(cache.has_scan_position(), "budget must leave real checkpoint progress");
|
||||
let progress = cache.info.scan_progress.expect("interrupted worker records coverage");
|
||||
|
||||
@@ -27,7 +27,6 @@ async fn checkpoint_plan(set: &Arc<SetDisks>, owner: &CancellationToken) -> Scan
|
||||
digest,
|
||||
bucket_coverage_digest: digest,
|
||||
checkpoint_coverage_digest: digest,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
requires_full_scan: true,
|
||||
service_cohort: None,
|
||||
execution_digest: digest,
|
||||
|
||||
@@ -130,7 +130,6 @@ async fn scan_plan(set: &Arc<SetDisks>, names: &[&str], cycle: u64) -> ScannerBu
|
||||
digest,
|
||||
bucket_coverage_digest: digest,
|
||||
checkpoint_coverage_digest: digest,
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
requires_full_scan: true,
|
||||
service_cohort: Some(Arc::new(StdMutex::new(cohort))),
|
||||
execution_digest: digest,
|
||||
|
||||
@@ -183,11 +183,9 @@ async fn dirty_entry_removed_during_listing_still_forces_a_full_scan() {
|
||||
);
|
||||
assert_eq!(second.buckets_usage["hot-bucket"].objects_count, 2);
|
||||
assert_eq!(second.buckets_usage["cold-bucket"].objects_count, 2);
|
||||
let report = global_metrics().report().await;
|
||||
let freshness = report.usage_freshness;
|
||||
let freshness = global_metrics().report().await.usage_freshness;
|
||||
assert_eq!(freshness.scope_fallback_total, fallbacks_before + 1);
|
||||
assert_eq!(freshness.last_scope_fallback_reason, "dirty_snapshot_incomplete");
|
||||
assert_eq!(report.scan_observation.scope_fallbacks["dirty_snapshot_incomplete"], 1);
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
}
|
||||
|
||||
@@ -201,53 +199,11 @@ async fn required_full_scan_is_not_reported_as_a_scope_fallback() {
|
||||
wait_for_namespace_commit_tails(store.as_ref()).await;
|
||||
super::io_cycle::set_bucket_listing_window_hook_for_tests(|| record_dirty_usage_bucket("created-after-listing"));
|
||||
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
let (result, _, scope) = run_cycle(&store, 2, Some(baseline), true).await;
|
||||
|
||||
assert!(scope.is_default());
|
||||
assert_eq!(result.pending_dirty_usage_buckets(), 1);
|
||||
assert_eq!(result.scope_fallback(), None, "scheduled maintenance walks are not degradations");
|
||||
assert_eq!(result.completed_scope_fallback(), None);
|
||||
let after = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(after.scope_fallbacks, before.scope_fallbacks);
|
||||
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
|
||||
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn baseline_unavailable_cycle_records_real_scope_and_baseline_rejections_once() {
|
||||
let (_temp_dir, store) = setup_local_scanner_store(1, 1).await;
|
||||
let _baseline = converged_baseline(&store).await;
|
||||
put_fixture_object(&store, "hot-bucket", "second").await;
|
||||
record_dirty_usage_bucket("hot-bucket");
|
||||
wait_for_namespace_commit_tails(store.as_ref()).await;
|
||||
let before = global_metrics().scan_observation().snapshot();
|
||||
let (result, snapshot, scope) = run_cycle(&store, 2, Some(Bytes::from_static(b"not-json")), false).await;
|
||||
assert!(scope.is_default());
|
||||
assert_eq!(result.status, ScannerCycleStatus::Complete);
|
||||
assert_eq!(result.scope_fallback(), Some(ScannerScopeFallbackReason::BaselineUnavailable));
|
||||
assert_eq!(snapshot.buckets_usage["hot-bucket"].objects_count, 2);
|
||||
let after = global_metrics().scan_observation().snapshot();
|
||||
assert_eq!(
|
||||
after.scope_fallbacks["baseline_unavailable"],
|
||||
before.scope_fallbacks["baseline_unavailable"] + 1
|
||||
);
|
||||
assert_eq!(
|
||||
after.authoritative_baseline_rejections["decode"],
|
||||
before.authoritative_baseline_rejections["decode"] + 1
|
||||
);
|
||||
assert_eq!(
|
||||
after.observed_baseline_rejections, before.observed_baseline_rejections,
|
||||
"an undecodable authority never admits an observation"
|
||||
);
|
||||
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated + 1);
|
||||
assert_eq!(after.prefix_scope_resolution.activated, before.prefix_scope_resolution.activated);
|
||||
assert_eq!(
|
||||
after.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"],
|
||||
before.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"] + 1
|
||||
);
|
||||
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
|
||||
clear_dirty_usage_buckets_for_tests();
|
||||
}
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
// Copyright 2026 RustFS Team
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
//! A bucket whose walk stops early can publish its last certified value as
|
||||
//! pending. This fixture tests usage-scan status, not the outer scanner cycle
|
||||
//! outcome: budget exhaustion may leave complete validated usage while the
|
||||
//! coordinator records `Partial`. A continuation under a newer plan exercises
|
||||
//! `mixed_coverage` (rustfs/rustfs#8494). A bucket that never certified keeps
|
||||
//! the usage scan incomplete.
|
||||
//! A bucket whose walk stops early without exhausting the cycle budget (here
|
||||
//! a sweep resumed under a newer plan, the `mixed_coverage` case of
|
||||
//! rustfs/rustfs#8494) no longer makes the whole cycle `Incomplete`. The set
|
||||
//! publishes the bucket's last certified value, the cycle completes with the
|
||||
//! bucket pending, and the next cycle walks it again. A bucket that never
|
||||
//! certified keeps the cycle incomplete.
|
||||
|
||||
use super::*;
|
||||
|
||||
@@ -38,7 +38,8 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
// Real checkpoint commits need the runtime drain grace after the count budget stops the walk.
|
||||
// These cycles persist real bucket checkpoints and source caches. Keep
|
||||
// CI storage latency inside the production drain window after a count stop.
|
||||
budget.use_production_drain_grace_for_tests();
|
||||
let (updates, mut receiver) = mpsc::channel(1);
|
||||
let result = tokio::time::timeout(
|
||||
@@ -48,7 +49,7 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
|
||||
ScannerCycleRequest {
|
||||
owner_ctx: ctx.clone(),
|
||||
ctx: budget.token(),
|
||||
budget: Arc::clone(&budget),
|
||||
budget,
|
||||
updates,
|
||||
want_cycle: cycle,
|
||||
leader_epoch: LEADER_EPOCH,
|
||||
@@ -68,13 +69,6 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
|
||||
.await
|
||||
.expect("cycle should finish within the test deadline")
|
||||
.expect("cycle should succeed");
|
||||
if options.max_objects.is_some() {
|
||||
assert_eq!(
|
||||
budget.reason(),
|
||||
Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects),
|
||||
"a count-limited fixture must stop for its object budget"
|
||||
);
|
||||
}
|
||||
let snapshot = receiver.recv().await;
|
||||
assert!(receiver.recv().await.is_none(), "a cycle publishes at most one terminal snapshot");
|
||||
(result, snapshot)
|
||||
@@ -131,8 +125,9 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
|
||||
assert!(result.clear_verified_usage().is_empty());
|
||||
assert!(dirty_usage_buckets_for_tests().is_empty());
|
||||
|
||||
// Cycle 3: the budget stops the walk and saves a checkpoint. Usage can
|
||||
// carry the certified value as pending; the outer coordinator records Partial.
|
||||
// Cycle 3: the object budget stops the hot bucket's walk and leaves a
|
||||
// checkpoint under this cycle's plan. Its certified value is carried, so
|
||||
// budget exhaustion leaves the bucket pending without blocking the cycle.
|
||||
for object in ["p4/third", "p5/third"] {
|
||||
put_fixture_object(&store, "hot-bucket", object).await;
|
||||
}
|
||||
@@ -149,38 +144,12 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert_eq!(
|
||||
result.status,
|
||||
ScannerCycleStatus::Complete,
|
||||
"certified carried usage remains complete after a budget stop"
|
||||
);
|
||||
let third = third.expect("validated carried usage publishes after the budget stop");
|
||||
let third = third.expect("a carried partial cycle publishes");
|
||||
assert_eq!(result.status, ScannerCycleStatus::Complete);
|
||||
assert_eq!(result.pending_dirty_usage_buckets(), 1);
|
||||
assert!(!third.usage_snapshot_observational);
|
||||
assert_eq!(third.usage_snapshot_converged, Some(false));
|
||||
assert_eq!(third.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]);
|
||||
assert_eq!(third.usage_snapshot_pending_bucket_count, 1);
|
||||
assert!(third.is_complete_bucket_usage_snapshot());
|
||||
let carried_hot = &third.buckets_usage["hot-bucket"];
|
||||
let certified_hot = &second.buckets_usage["hot-bucket"];
|
||||
assert_eq!(carried_hot.usage_value_state, Some(rustfs_data_usage::BucketUsageValueState::Carried));
|
||||
assert!(certified_hot.usage_certification.is_some(), "cycle 2 certified the prior hot value");
|
||||
assert_eq!(
|
||||
carried_hot.usage_certification, certified_hot.usage_certification,
|
||||
"carrying must not mint a new certificate"
|
||||
);
|
||||
assert_eq!(carried_hot.objects_count, 4);
|
||||
assert_eq!(carried_hot.size, certified_hot.size);
|
||||
assert_eq!(third.buckets_usage["cold-bucket"].objects_count, 1);
|
||||
assert_eq!(third.buckets_usage["idle-bucket"].objects_count, 1);
|
||||
assert_eq!(third.objects_total_count, 6);
|
||||
// Only hot was dirty in this scoped scan; it received no fresh certification or ACK.
|
||||
assert!(result.clear_verified_usage().is_empty());
|
||||
assert_eq!(
|
||||
dirty_usage_buckets_for_tests().keys().cloned().collect::<Vec<_>>(),
|
||||
vec!["hot-bucket".to_string()],
|
||||
"the budget-stopped hot bucket retains its dirty responsibility"
|
||||
);
|
||||
assert_eq!(third.buckets_usage["hot-bucket"].objects_count, 4);
|
||||
let set = Arc::clone(&store.pools[0].disk_set[0]);
|
||||
let mut interrupted = DataUsageCache::default();
|
||||
interrupted
|
||||
|
||||
@@ -8,22 +8,6 @@ const EPOCH: u64 = 7;
|
||||
const RETIRED: &str = "retirement-generation";
|
||||
const LIVE: &str = "retirement-survivor";
|
||||
|
||||
async fn close_fixture(directory: tempfile::TempDir, store: Arc<ECStore>) {
|
||||
store.background_cancel_token().expect("fixture shutdown token").cancel();
|
||||
tokio::time::timeout(Duration::from_secs(30), async {
|
||||
for pool in &store.pools {
|
||||
pool.stop_endpoint_monitor_for_test().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("endpoint monitors must stop before removing fixture disks");
|
||||
wait_for_namespace_commit_tails(&store).await;
|
||||
drop(store);
|
||||
let path = directory.path().to_path_buf();
|
||||
directory.close().expect("remove source retirement fixture");
|
||||
assert!(!path.exists());
|
||||
}
|
||||
|
||||
async fn run_turn(
|
||||
set: &Arc<SetDisks>,
|
||||
buckets: &[BucketInfo],
|
||||
@@ -53,7 +37,6 @@ async fn run_turn(
|
||||
digest: DataUsageScanPlanDigest([31; 32]),
|
||||
bucket_coverage_digest: DataUsageScanPlanDigest([32; 32]),
|
||||
checkpoint_coverage_digest: DataUsageScanPlanDigest([33; 32]),
|
||||
completed_checkpoint_cycle_floor: None,
|
||||
requires_full_scan: true,
|
||||
execution_digest: DataUsageScanPlanDigest([34; 32]),
|
||||
leader_epoch: EPOCH,
|
||||
@@ -160,7 +143,10 @@ async fn assert_retired_source_turn_discards_only_deleted_generation_and_retains
|
||||
failures.partial.lock().await.contains(RETIRED),
|
||||
"responsibility remains until namespace replanning"
|
||||
);
|
||||
close_fixture(directory, store).await;
|
||||
drop(store);
|
||||
let path = directory.path().to_path_buf();
|
||||
directory.close().expect("remove source retirement fixture");
|
||||
assert!(!path.exists());
|
||||
}
|
||||
|
||||
async fn assert_recreated_source_turn_rescans_successor_without_old_value_or_identity() {
|
||||
@@ -202,7 +188,10 @@ async fn assert_recreated_source_turn_rescans_successor_without_old_value_or_ide
|
||||
assert_eq!((successor.objects, successor.size), (2, 28));
|
||||
assert_eq!(after.info.scan_bucket_incarnations[RETIRED], new_identity);
|
||||
assert_eq!(after.find(LIVE).expect("unrelated bucket").size, 13);
|
||||
close_fixture(directory, store).await;
|
||||
drop(store);
|
||||
let path = directory.path().to_path_buf();
|
||||
directory.close().expect("remove source retirement fixture");
|
||||
assert!(!path.exists());
|
||||
}
|
||||
|
||||
async fn assert_missing_retirement_proof_and_foreign_source_fences_cannot_clear_completion() {
|
||||
@@ -229,7 +218,10 @@ async fn assert_missing_retirement_proof_and_foreign_source_fences_cannot_clear_
|
||||
.expect_err("unconfirmed identity changes cannot retire completed work");
|
||||
assert!(error.to_string().contains("identity"), "{field}: {error}");
|
||||
}
|
||||
close_fixture(directory, store).await;
|
||||
drop(store);
|
||||
let path = directory.path().to_path_buf();
|
||||
directory.close().expect("remove source retirement fixture");
|
||||
assert!(!path.exists());
|
||||
}
|
||||
|
||||
async fn assert_missing_directories_and_unreadable_retirement_proof_remain_unconfirmed() {
|
||||
@@ -259,7 +251,10 @@ async fn assert_missing_directories_and_unreadable_retirement_proof_remain_uncon
|
||||
.await
|
||||
.expect_err("an unreadable retirement must preserve failed completion validation");
|
||||
assert!(error.to_string().contains("identity"));
|
||||
close_fixture(directory, store).await;
|
||||
drop(store);
|
||||
let path = directory.path().to_path_buf();
|
||||
directory.close().expect("remove source retirement fixture");
|
||||
assert!(!path.exists());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -168,8 +168,8 @@ fn watch_structure_ignores_unrelated_dirty_generations_but_requires_every_owner(
|
||||
unrelated.insert("peer".into(), crate::scanner::scanner_node_activity_for_tests("remote", 103, 104, false));
|
||||
assert_eq!(original, watch_structure_digest(&unrelated, &routes).unwrap());
|
||||
assert_ne!(
|
||||
crate::scanner::scanner_activity_coverage_digest(&before, LOCAL_SCANNER_ACTIVITY_NODE),
|
||||
crate::scanner::scanner_activity_coverage_digest(&unrelated, LOCAL_SCANNER_ACTIVITY_NODE)
|
||||
crate::scanner::scanner_activity_coverage_digest(&before),
|
||||
crate::scanner::scanner_activity_coverage_digest(&unrelated)
|
||||
);
|
||||
unrelated.remove("peer");
|
||||
assert!(watch_structure_digest(&unrelated, &routes).is_none());
|
||||
|
||||
@@ -170,8 +170,6 @@ pub(crate) use rustfs_ecstore::api::set_disk::SetDisks as EcstoreSetDisks;
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::storage::SCANNER_PUBLICATION_LEASE_TTL_MS as ECSTORE_SCANNER_PUBLICATION_LEASE_TTL_MS;
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::storage::init_local_disks as ecstore_init_local_disks;
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::storage::init_local_disks_with_instance_ctx as ecstore_init_local_disks_with_instance_ctx;
|
||||
pub(crate) use rustfs_ecstore::api::storage::{
|
||||
ECStore as EcstoreStore, ScannerDataMovementPauseStatus as EcstoreScannerDataMovementPauseStatus,
|
||||
@@ -412,9 +410,9 @@ pub(crate) mod scan {
|
||||
#[cfg(test)]
|
||||
pub(crate) use super::storage_contracts::{DeleteBucketOptions, MakeBucketOptions, ObjectIO, StorageAdminApi};
|
||||
pub use super::storage_contracts::{
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_ACTIVITY_V7_PROTOCOL_VERSION,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES, SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE, SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES,
|
||||
SCANNER_ACTIVITY_PROTOCOL_VERSION, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES,
|
||||
SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
|
||||
SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use super::{
|
||||
@@ -462,9 +460,6 @@ pub(crate) trait ScannerStorage:
|
||||
+ storage_contracts::NamespaceLocking<Error = EcstoreErrorType, NamespaceLock = NamespaceLockWrapper>
|
||||
{
|
||||
fn scanner_topology_digest(&self) -> [u8; 32];
|
||||
fn scanner_local_node_name(&self) -> Option<String> {
|
||||
None
|
||||
}
|
||||
fn scanner_namespace_mutation_generation(&self) -> u64;
|
||||
async fn scanner_data_movement_activity(&self) -> (bool, bool, u64);
|
||||
async fn scanner_data_usage_publication_blocked(&self) -> bool;
|
||||
@@ -517,15 +512,6 @@ impl ScannerStorage for EcstoreStore {
|
||||
crate::scanner::scanner_topology_digest(self)
|
||||
}
|
||||
|
||||
fn scanner_local_node_name(&self) -> Option<String> {
|
||||
self.endpoints()
|
||||
.0
|
||||
.iter()
|
||||
.flat_map(|pool| pool.endpoints.as_ref().iter())
|
||||
.find(|endpoint| endpoint.is_local)
|
||||
.map(|endpoint| endpoint.grid_host())
|
||||
}
|
||||
|
||||
fn scanner_namespace_mutation_generation(&self) -> u64 {
|
||||
EcstoreStore::scanner_namespace_mutation_generation(self)
|
||||
}
|
||||
|
||||
@@ -58,10 +58,7 @@ pub const SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION: u32 = 5;
|
||||
/// a scanner must not use it as a publication proof because terminal movement
|
||||
/// state is not authenticated by that version.
|
||||
pub const SCANNER_ACTIVITY_V6_PROTOCOL_VERSION: u32 = 6;
|
||||
/// Authenticates terminal movement state, but carries no checkpoint continuity.
|
||||
pub const SCANNER_ACTIVITY_V7_PROTOCOL_VERSION: u32 = 7;
|
||||
/// Authenticates read-only checkpoint continuity separately from the RPC epoch.
|
||||
pub const SCANNER_ACTIVITY_PROTOCOL_VERSION: u32 = 8;
|
||||
pub const SCANNER_ACTIVITY_PROTOCOL_VERSION: u32 = 7;
|
||||
pub const SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION: u32 = 2;
|
||||
pub const SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES: usize = 32;
|
||||
pub const SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES: usize = SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES;
|
||||
|
||||
@@ -255,7 +255,18 @@ walk. Field roles:
|
||||
| `maintenance_generation` | veto: no cycle today records whether it published maintenance results, and the field is already part of the structural digest; it becomes a shade signal only once that distinction exists |
|
||||
| `namespace_generation`, `dirty_usage_generation`, `dirty_usage_pending` | shade signal: consumed per bucket; never a veto on their own |
|
||||
|
||||
The activity snapshot feeds separate traversal, bucket-reuse and publication identities. Traversal binds the canonical peer set, bucket list and creation times, topology, protocol, movement state and checkpoint continuity. Bucket reuse also binds namespace, maintenance and dirty generations. Protocol v8 authenticates a checkpoint identity separately from the fresh RPC process identity: only a clean, read-only shutdown with `RUSTFS_SCANNER_READ_ONLY_HANDOFF=true` can hand that identity to the next process. The flag defaults to false and requires reader-first deployment plus receipt-clearing preparation before legacy rollback. Missing, corrupt, mismatched or unconsumed receipts, a writer, or a crash expire it and invalidate old coverage. Protocol v7 remains readable and retains its fresh process fence. `publication_blocked` belongs to neither cache identity; it remains a cycle veto. Publication, dirty ACKs and leases always bind the actual process identity, even across a verified handoff. See [the shutdown receipt contract](scanner-usage-publication.md#read-only-restart-continuity).
|
||||
The activity snapshot feeds two plan identities with different jobs. The
|
||||
coverage identity (bucket list and creation times, `topology_digest`,
|
||||
`protocol_version`, `data_movement_active`, `movement_generation`) decides
|
||||
whether a resumed bucket checkpoint still describes the same traversal; only
|
||||
it may mark a resumed sweep as mixed coverage. The reuse identity adds
|
||||
`instance_id` and `maintenance_generation` and decides whether a completed
|
||||
cache or a scoped baseline may be reused: a peer restart can lose dirty hints
|
||||
inside the journal's coalescing window and a maintenance change needs a fresh
|
||||
discovery walk, so both force a full walk, but neither changes what a
|
||||
full-walk checkpoint has already covered. `publication_blocked` belongs to
|
||||
neither identity. The guardrail compares the reuse identity plus the movement
|
||||
fields.
|
||||
|
||||
The guardrail is bounded in three ways:
|
||||
|
||||
|
||||
@@ -110,17 +110,21 @@ snapshot for a missing or corrupt authoritative object.
|
||||
|
||||
## Cache Execution Identity
|
||||
|
||||
The structural scan-plan digest remains stable across ordinary bucket writes so a scoped scan can retain unaffected baseline buckets. Bucket reuse additionally binds every member's namespace, maintenance and dirty generations; its process component may survive only the read-only handoff described below. The current cycle's execution digest always binds the full activity snapshot with fresh RPC process identities. A set cache carrying buckets whose walk stopped early also binds those bucket names (`scanner_carried_execution_digest`), so it cannot replace a full walk of the same cycle. Completed bucket progress from an older leader is reusable only above the authoritative aggregate's published cycle floor, with identical incarnation, source, layout, tier, scan mode, checkpoint coverage and bucket reuse identity. Adoption never advances the saved cycle. A published aggregate expires this unpublished-progress exception.
|
||||
|
||||
## Read-Only Restart Continuity
|
||||
|
||||
`RUSTFS_SCANNER_READ_ONLY_HANDOFF` defaults to `false`. Unset, false or invalid values disable both checkpoint-identity restoration and receipt creation; startup still durably consumes existing receipts. Deploy these readers to every possible writer before enabling receipt writes with `true`. Cross-process reuse of unpublished completed buckets requires this opt-in.
|
||||
|
||||
The existing `.rustfs.sys/unclean-shutdown` marker can contain a versioned read-only handoff receipt. It binds the node owner, disk UUID, complete local endpoint/disk inventory and checkpoint identity. Startup consumes every receipt by durably replacing it with `running` before constructing the store, because store construction can start recovery writers. Missing or incompatible receipts provide no continuity; an existing or unreadable receipt that cannot be overwritten aborts startup. Marker writes always sync the file and parent directory, including under the legacy data-sync-off setting. A missing local drive, replacement disk, different inventory or inconsistent receipt identity expires continuity.
|
||||
|
||||
When enabled, shutdown saves matching receipts only after frontend, heal, capacity, table-maintenance, scanner and detached-mutation shutdown fences succeed, and only if the process observed no namespace or maintenance mutation. Live dirty notifications permanently disqualify that process even if the dirty generation is later acknowledged; replaying an existing durable dirty hint is not a new mutation. A writer follows ordinary marker cleanup and receives a fresh checkpoint identity on restart. Cancellation, failed shutdown, crash and partial receipt writes cannot grant continuity. Activity protocol v8 signs the checkpoint identity while keeping RPC instances, ACKs, publication proofs and leases fresh. v7 responses omit this field and use the old fresh-process fence.
|
||||
|
||||
Before any older binary opens the storage, disable the flag on every node, start the current reader with all local drives present, wait for readiness on every node, then shut down cleanly. This preparation consumes receipts without restoring their identities or writing replacements. If a node cannot complete it, rollback must wait. Direct rollback from an enabled writer is unsupported: an older process can start recovery before replacing its marker, and a 1.0.4 writer with Scanner and Heal disabled can acknowledge a foreground write and crash while leaving the receipt unchanged. Metadata synchronization alone does not close that gap. A later opt-in restart after the supported preparation starts a fresh checkpoint lineage.
|
||||
The structural scan-plan digest can remain stable across ordinary bucket writes
|
||||
so a scoped scan can retain unaffected baseline buckets. It is not sufficient
|
||||
proof for reusing a completed result within the same cycle. Bucket work uses an
|
||||
execution digest combining the structural plan and the full activity snapshot,
|
||||
with the bucket's dirty generation included in its cache identity. A bucket
|
||||
sweep's checkpoint is keyed separately by the coverage identity
|
||||
(`scanner_activity_coverage_digest`, see
|
||||
[scanner-per-bucket-convergence.md](scanner-per-bucket-convergence.md)), so
|
||||
the reuse identity can change without restarting a sweep in progress. Completed
|
||||
set caches carry the same execution digest separately from their structural
|
||||
plan; a set cache that stands in for buckets whose walk stopped early binds
|
||||
the carried bucket names into it (`scanner_carried_execution_digest`), so it
|
||||
is never adopted in place of a full walk of the same cycle, nor the reverse.
|
||||
The persisted set-root fast path requires equal execution identities as well as
|
||||
the existing source, cycle, leader, tier, and cache-structure checks.
|
||||
|
||||
Remote workers receive the reuse plan as `scan_plan_digest` and the independent
|
||||
coverage plan as optional `checkpoint_plan_digest`. The v3 capability proof
|
||||
|
||||
@@ -73,8 +73,6 @@ All names are defined in `crates/io-metrics/src/internode_metrics.rs`.
|
||||
|
||||
Explicit channel dial metrics start after endpoint settings validate and cover TLS configuration, waiting for its blocking task, and connection establishment. Each completed or cancelled attempt records one result. TLS configuration errors and acquisition cancellation contribute to `rustfs_system_network_internode_dial_errors_total`. Only connection failures follow the existing peer-unreachable classification. Cache hits reuse the existing channel, and coalesced callers share the dial owner's result. Tonic's internal TCP reconnects and HTTP/2 PING outcomes require their own transport observations.
|
||||
|
||||
Internode clients retain up to 8,192 locally reset streams per connection so late responses to cancelled quorum reads can be ignored. The locked HTTP/2 library's one-second retention period and 1,024 local protocol-error reset limit remain unchanged. This increases bounded transport bookkeeping on both shared and bulk channels; measure memory alongside cancellation and GOAWAY counts. Responses arriving after expiration or beyond the retention capacity can still consume the error budget. This client-side setting requires no wire-format migration; rollback restores the previous capacity.
|
||||
|
||||
| Metric | Stage signal |
|
||||
| --- | --- |
|
||||
| `rustfs_system_network_internode_operation_duration_ms{operation,backend}` | control-plane RTT (P0), lock/bulk latency (P1), first-op latency (P3) |
|
||||
|
||||
@@ -29,8 +29,6 @@ Use rustfs_memory_allocator_malloc_requested_bytes only when rustfs_memory_alloc
|
||||
|
||||
A previously published gauge can remain visible after a statistic becomes unavailable. Filter it by the availability metric. The malloc_requested_total_bytes metric is cumulative allocation activity and can grow indefinitely in a healthy process. Do not use its slope as evidence of retained memory.
|
||||
|
||||
The `rustfs_memory_allocator_allocated_bytes` gauge reports the allocator's `allocated.current` block accounting when `rustfs_memory_allocator_allocated_bytes_available` is one. It includes size-class rounding and can lag page or thread statistics merges. Availability means the field was present and valid, not that accounting was enabled. This observation is distinct from live requested bytes, committed bytes and RSS; zero or a change alone does not prove a drained heap or a leak.
|
||||
|
||||
When live requested bytes are unavailable, capture a supported heap profile or allocation/free trace. Preserve attribution to allocation stacks and lifetime; committed bytes and RSS alone cannot establish a leak. Other raw allocator statistics, including page accounting, depend on the allocator build and are not interchangeable with live application bytes.
|
||||
|
||||
### Diagnostic allocator and runtime metrics
|
||||
|
||||
@@ -314,8 +314,6 @@ last successful timestamp and values unchanged. Missing/invalid limits, invisibl
|
||||
membership, or oversized files must not be interpreted as host capacity or zero
|
||||
pressure. Other platforms and cgroup v1 have no sample from this observer.
|
||||
|
||||
A missing `cpu.max` or `memory.max` is accepted only when `cgroup.controllers` proves that the controller belongs to the verified v2 hierarchy but is not enabled for that cgroup. Visible ancestor limits and sibling memory use still apply. Missing limits for an available controller, unreadable controller lists, and controllers absent from the v2 root remain unavailable; missing pressure counters remain unknown.
|
||||
|
||||
The `cpu_usage_microseconds`, `cpu_throttled_microseconds`, `cpu_stall_microseconds`,
|
||||
`memory_stall_microseconds`, and `io_stall_microseconds` gauges, under the same
|
||||
prefix, expose cumulative leaf-cgroup counters. Each requires its own
|
||||
|
||||
@@ -566,7 +566,7 @@ Scanner startup records an empty scope to establish a genuine zero observation.
|
||||
|
||||
`RUSTFS_HEAL_ADAPTIVE_ADMISSION=true` enables Linux cgroup-v2 feedback shared by the heal manager’s four object-heal entry points and local Scanner namespace reads and object accounting. Scanner directory and local metadata reads acquire the physical leaves backing their local disk; object Heal admits the selected erasure sets across its requested pools. It is off by default while full SSD performance acceptance is pending. Unsupported platforms reject an enabled setting. Physical admission does not yet cover cache publication, lifecycle actions, the abandoned-child streaming list coordinator or independent bucket/format repair. Existing worker, disk and byte budgets still apply.
|
||||
|
||||
The node samples its effective CPU quota/cpuset, ancestor memory headroom, CPU usage/throttling and CPU/memory/I/O pressure once per second. Sampling uses one blocking worker at a time. After a failed resource read, it may make one complete fresh read while time remains in the original 20 ms bracket; it never substitutes a cached observation. Both attempts share their original start time, and two failed reads leave admission closed. Resource reads spanning over 20 ms or older than one second after all inputs return cannot renew admission. `rustfs_background_resource_rereads_total{result="success"|"error"}` counts second-read outcomes; a successful read can still be rejected by the timing checks. Counter rates use both read brackets conservatively. The concurrency cap is at most the rounded-up effective CPU count and 256 scheduling units; existing operator limits still apply independently. At most 1% memory headroom blocks new work. This scheduling safeguard alone does not reserve memory for repair buffers.
|
||||
The node samples its effective CPU quota/cpuset, ancestor memory headroom, CPU usage/throttling and CPU/memory/I/O pressure once per second. Sampling uses one blocking worker at a time. Resource reads retain their real start/end times: reads spanning over 20 ms or older than one second after all inputs return cannot renew admission. Counter rates use both read brackets conservatively. The concurrency cap is at most the rounded-up effective CPU count and 256 scheduling units; existing operator limits still apply independently. At most 1% memory headroom blocks new work. This scheduling safeguard alone does not reserve memory for repair buffers.
|
||||
|
||||
The same sampler also controls the reusable staging storage of admitted buffered file readers and writers, whole-file results, and positioned-read results. Its process-wide cap is the smaller of 10% of the effective memory limit and the remaining headroom above 20% of that limit. Missing or expired samples block new allocations. A buffer reserves its payload size before allocation; growth counts both the old and replacement allocations until the old storage is freed. Shrinking the cap does not revoke an existing allocation, and reuse needs no new reservation. A denied allocation returns `WouldBlock`; reusable file streams accept no write bytes and preserve their read position. Cancellation and runtime shutdown keep storage charged until the actual native job releases it. Returned `Bytes` retain the allocation's charge through the last clone or slice. Standard-backend direct reads charge both the aligned bounce allocation and logical output while they coexist. Native io_uring reads retain their full allocation, including alignment padding, through the returned result; chunked reads also charge the assembled output while a native chunk is pending. Memory-budget denial does not switch either read path to a different backend.
|
||||
|
||||
@@ -584,8 +584,6 @@ Namespace frame payload and envelope encoding use the memory owner captured by t
|
||||
|
||||
This cap does not yet cover other encode/decode paths, other RPC serializers and transport copies, allocator overhead or total process RSS. Already resident format-cache bytes are shared without a new payload allocation. The `rustfs_background_buffer_memory_limit_bytes` and `rustfs_background_buffer_memory_retained_bytes` gauges expose the sampled cap and retained payload. Like the node gauges, they can remain stale when the sampler stalls; the in-process admission deadline still expires after three seconds.
|
||||
|
||||
`rustfs_background_resource_observations_total` distinguishes `valid`, `unavailable`, `stale`, `wide` and `invalid_timing` observations used by admission. `rustfs_background_resource_sample_duration_seconds` records the available sampling brackets. These metrics use the admission sample, unlike the independent process-memory sampler, and stop progressing if that worker stalls. Buffer-denial errors include the requested bytes, retained bytes, configured limit and whether the lease expired, captured under the same budget lock. These diagnostics do not change the cap, freshness limits or retry behavior.
|
||||
|
||||
Each complete sample leases admission for three seconds. Missing resource caps block new work; missing or low-confidence feedback retains at most one maintenance unit. CPU use above 90%, throttling or pressure above 1% of elapsed time, memory headroom at most 10%, foreground errors/cancellations, a foreground queue or a comparable foreground p99 regression reduce concurrency immediately. Recovery requires CPU use at most 70%, throttling/stalls at most 0.1% of elapsed time, at least 20% memory headroom, fresh foreground evidence and actual successful work. The existing controller probes one additional unit only after eight consecutive healthy seconds and cooldown. Limits affect new admission and do not revoke active work.
|
||||
|
||||
The Scanner captures the same node owner once per bucket walk. Directory open/refill, entry type and size lookup, and object accounting try admission before their first storage poll; each scheduling unit ends when its operation future finishes. Physical leaves take FIFO turns in a stable order before node admission, and partial grants are returned before any resource wait. On contention, local and remote coordinators suspend the same traversal, release the bucket cache lease, and keep their physical/node FIFO position while topology remains current. The existing cycle budget and cancellation bound this wait. Topology validity is rechecked every 250 ms; expired or replaced plans yield a typed deferral. The coordinators reacquire the cache lease and verify both cache revisions before resuming with a permit or a typed deferral that preserves partial progress. A changed cache rejects the suspended result. Cancellation also drops the queued request.
|
||||
@@ -596,13 +594,11 @@ Native directory work and the local metadata read retain only that disk's physic
|
||||
|
||||
Resource-controlled peer requests carry `x-rustfs-rpc-background-admission: v1`, bound to their HTTP or gRPC signature. Stream retries preserve that classification without retaining the sender's disk quota. The receiver uses its existing shared node owner and admits the selected local physical device before storage locks or I/O. Repeated phases reuse the request's reservation, including when capacity is one. Missing capacity, an unknown UUID mapping, or an unavailable physical plan defers the request instead of probing other disks or reporting object absence. Cancellation returns node capacity; native work retains only its selected physical resources until it finishes.
|
||||
|
||||
An authenticated namespace-scan POST can queue its initial target admission for up to 15 seconds, before storage locks or body processing. The caller's existing response-header deadline and cancellation still apply. The receiver revalidates the active disk attachment after waiting; unknown UUID mappings and unavailable physical plans defer immediately. Other RPC entry points and additional device attachments within an admitted request remain nonblocking.
|
||||
|
||||
An HTTP file response releases node admission after opening its reader, but records progress only after producing the declared byte count or reaching an unbounded EOF. Body errors, early EOF and cancellation do not credit successful preparation. The deferred progress record retains counter identities without keeping node slots or physical units.
|
||||
|
||||
Enable this experimental setting only after all peers support the admission marker. Older peers reject marked signatures, and a new peer with admission disabled rejects marked work; neither path silently runs it outside resource control. Requests without the marker retain their existing signature and foreground behavior. The receiver reports deferred admission as HTTP 503 or gRPC `RESOURCE_EXHAUSTED`.
|
||||
|
||||
Storage operations without a coordinating cache lease retain immediate resource deferral and useful partial-cache handling. Existing remote single-session ownership and local-disk admission shares remain in force. Disabled admission creates no request channel. Enabled walks use one bounded channel, with a response allocation and cache revision reads only on contention; the uncontended per-entry path adds no global lookup or task. The streaming abandoned-child coordinator requires a separate ownership boundary: retaining a node unit while it waits for a bounded-channel consumer that needs the same unit would deadlock at capacity one.
|
||||
Calls without a coordinating cache lease retain immediate resource deferral and useful partial-cache handling. Existing remote single-session ownership and local-disk admission shares remain in force. Disabled admission creates no request channel. Enabled walks use one bounded channel, with a response allocation and cache revision reads only on contention; the uncontended per-entry path adds no global lookup or task. The streaming abandoned-child coordinator requires a separate ownership boundary: retaining a node unit while it waits for a bounded-channel consumer that needs the same unit would deadlock at capacity one.
|
||||
|
||||
Foreground comparison uses the existing complete response-body windows and shared admission snapshots. Each active class needs at least 100 completions and a finite p99 interval; this permits the percentile rank but is not a statistical confidence guarantee. Comparisons require the same active classes and rates within 80–125% of a retained reference. A changed workload resets recovery. The reference is an observed workload window, not an independent background-off baseline. A p99 interval can justify stability only when its upper bound is within 5% of the reference lower bound; a lower bound above 105% of the reference upper bound is pressure. Overlapping/uncertain comparisons cannot justify growth. Missing queues, stale or repeated cuts, uncertain cut boundaries and low counts are not idle evidence.
|
||||
|
||||
@@ -902,42 +898,6 @@ done
|
||||
|
||||
The `aggregated.scanner` payload preserves the same scanner progress, checkpoint, pacing, source work, maintenance control, lifecycle expiry, and lifecycle transition fields used by the local scanner status, but only for the responding node; `by_host.*.scanner` keeps that node's host view. Compare the per-node artifacts externally to find old active paths, partial checkpoints, pacing pressure, source-level control pressure, or downstream queue admission problems across the deployment.
|
||||
|
||||
## Reading Cached Bucket Statistics
|
||||
|
||||
Admin usage reads cache the selected scanner baseline for the existing 30-second TTL, then verify every returned bucket against uncached storage-owned incarnation metadata. Identity reads use at most four concurrent operations and a shared two-second deadline after releasing snapshot/refresh locks. Each bucket read linearizes independently against metadata transactions; the response is not an atomic snapshot of the entire namespace. Delayed peer invalidation cannot authorize a predecessor's statistics for a recreated bucket.
|
||||
|
||||
Missing, failed, timed-out, nil, mismatched, or legacy absent identities leave the bucket value absent (unknown), rather than returning the predecessor's value or a confirmed zero. Legacy statistics become visible after a scanner snapshot supplies the current identity. Account-info and usage display apply the existing single-process memory overlay before the final identity filter; distributed local counters are not promoted. When a projection omits values or changes an overlay, totals are partial and `usage_snapshot_converged=false`, and per-tier/replication aggregates are unavailable. Verified cold bucket values remain visible. Such response-only projections cannot be persisted, used for quota admission, or acknowledge dirty work.
|
||||
|
||||
## Reading Scanner Work Observations
|
||||
|
||||
`GET /rustfs/admin/v3/scanner/status` exposes `metrics.scan_observation` for the responding node. Capture timestamped snapshots on every node, including the scanner leader, and compare their deltas over the same workload window. These process-lifetime counters reset on restart; a missing field on an older node means unobserved. They are loaded independently and may briefly disagree while operations finish. This telemetry does not change scan cadence, cached usage reads, quota authority, dirty acknowledgement, or maintenance requirements.
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `metadata_reads.attempts` | Disk metadata read futures started after scanner node admission. Invalid paths and operations cancelled before starting the read do not count. |
|
||||
| `metadata_reads.succeeded`, `not_found`, `errors`, `cancelled` | Storage outcomes, including object/version disappearance and dropped read futures. Successful storage reads can still contain corrupt metadata; decoding and lifecycle failures do not become read errors. Cancellation observes the future, not proof that physical I/O stopped. |
|
||||
| `metadata_reads.bytes` | Total returned buffer lengths for successful reads, including inline data and corrupt metadata. Failed reads contribute no known bytes; this is not physical disk traffic. |
|
||||
| `metadata_reads.elapsed_nanos` | Sum of read-future elapsed time, including disk-permit or RPC waits, failures, and cancellation. Folder pacing, scanner node admission, decoding, and lifecycle work are outside this interval. Concurrent reads overlap, so the sum is not cycle wall time. |
|
||||
| `cache_reuse.clean_prefix` | Clean root subtrees copied under an admitted prefix scope by the non-erasure scanner. |
|
||||
| `cache_reuse.covered_namespace` | Namespace subtrees copied from validated checkpoint coverage. |
|
||||
| `cache_reuse.sampled_namespace` | Cached namespace subtrees copied when the existing periodic sampling selected other work. |
|
||||
| `cache_reuse.classified_namespace` | EC candidate namespace subtrees copied only after the current listing finished classifying them. Health/classification work still runs. |
|
||||
| `cache_reuse.*.count`, `elapsed_nanos` | Completed subtree reuse decisions and time spent copying/attaching their cache entries. Counts are not objects skipped or a cache hit ratio; times exclude classification/listing and are not total cache maintenance cost. |
|
||||
| `bucket_reuse.scoped_cold_bucket` | Whole cold buckets copied and attached to a successfully prepared scoped set cache. Count is copied buckets, not plans selected; later turn publication can still fail. Time covers copying/attachment only. |
|
||||
| `bucket_reuse.current_root` | Existing current bucket roots successfully sent to the local collector after watch, namespace lease and publication epoch checks. Time covers channel send only; failure or cancellation contributes no completed reuse. Collector acceptance and durable publication remain separate. |
|
||||
| `reuse_decisions.*.not_evaluated`, `blocked`, `eligible` | First conditions reached by production clean-prefix, checkpoint-coverage and namespace-sampling gates. Policy exclusions (`missing_scope`, `no_resume_sweep`, `deep_scan`, `forward_sweep`, `active_replication`) precede cache proof checks. `blocked` reports the first failed work/cache condition; later checks are not repeated for diagnostics. `eligible` does not certify a copy: an EC candidate can still require classification. |
|
||||
| `erasure_clean_prefix_walks` | Clean root children that still require walking because EC scans perform independent object-health work. |
|
||||
| `prefix_scope_resolution.evaluated`, `activated`, `blockers` | Proof evaluations retained by the final scope resolver, admitted proof gates, and every failed prerequisite. An admitted proof does not guarantee a reusable child exists; actual subtree copies appear in `cache_reuse`. Startup/required full scans, early scope rejection, and discarded peer/ACK resolutions do not count here. |
|
||||
| `prefix_cycle_completion.evaluated`, `activated`, `blockers` | End-of-cycle proof evaluations for subsequent reuse. An activated completion proof does not mean this cycle reused prefixes or that its later publication/persistence succeeded. One evaluation can increment several blocker counters. |
|
||||
| `scope_fallbacks` | Per-reason breakdown of the existing `usage_freshness.scope_fallback_total`; required full scans are excluded. |
|
||||
| `authoritative_baseline_rejections`, `observed_baseline_rejections` | First failed validation checks for `baseline_unavailable` fallback cycles, preserving the authoritative rejection when the observed fallback also fails. Missing/undecodable authority never reaches observed validation; `authoritative_identity` means it could not anchor an observed fallback. |
|
||||
|
||||
`freshness` describes the responding node's completed-cycle age, not whether every bucket has incorporated new writes. Read `metrics.usage_freshness` for node-local live pending work, guardrail vetoes, durable success and publication state; read `metrics.usage_publication` for the confirmed snapshot's pending count and publication class/outcome, and `cycle_schedule` for effective retry intervals and consecutive superseded cycles. Missing or unknown publication data does not establish zero pending work. A follower or a single node's empty dirty map cannot establish cluster-wide convergence.
|
||||
|
||||
Whole cold bucket copies, current-root reuse and the four subtree categories have different prerequisites and are not interchangeable hit rates. EC clean prefixes retain health walks. Deep or required full work must run, but a completed cache matching that exact work digest may satisfy a retry of the same cycle. A normal workload need not exercise current-root reuse or every subtree category; use controlled tests for positive emission evidence and capture real gate reasons when a VM window has zero copies.
|
||||
|
||||
The existing `life_time_ops.read_metadata` and `last_minute.actions.read_metadata` now include finished and cancelled read attempts; the latter reports count, accumulated whole seconds, and returned bytes in its existing window. Use `metadata_reads.elapsed_nanos` for subsecond read costs. Use read-count and byte deltas to assess work growth, and elapsed-time deltas to assess read-future cost. Compare those with reused-subtree deltas, scope fallback reasons, cycle wall time, pacing waits, and source work before attributing freshness delays to metadata I/O or proposing a scan optimization.
|
||||
|
||||
## Reading Lifecycle Transition Status
|
||||
|
||||
`metrics.lifecycle_transition`:
|
||||
|
||||
@@ -85,8 +85,3 @@ RUST_MIN_STACK=4194304 cargo test -p rustfs-ecstore --lib segment_observation_eq
|
||||
```
|
||||
|
||||
[W19](https://github.com/rustfs/backlog/issues/2272) remains open for trustworthy producer coverage, source/incarnation binding, and production shadow observations. No production stream, durable journal, runtime feature switch, scan skipping, or performance claim is introduced here. No restart/gap detection or restart-safe production coverage is established, and the revision-aware file backend does not prove EC publication durability.
|
||||
|
||||
|
||||
## Whole-cycle read-only restart regression
|
||||
|
||||
`scanner_io::tests::checkpoint_coverage_identity::checkpoint_small_budget_accumulates_across_process_restarts` runs real scanner/store workers in fresh OS processes against four buckets. Every scan keeps the same two-object budget and must publish exact per-bucket totals within 40 rounds. Separate cases inject a write into an already completed bucket before global publication, then either cleanly stop the writer or leave its running marker behind. Both must expire checkpoint continuity and eventually include the extra object; accepting the old total as converged fails the test. This is a local storage regression, separate from whole-daemon multi-node and physical-device acceptance.
|
||||
|
||||
@@ -20,7 +20,7 @@ use crate::admin::service::caller_identity::oidc_profile_fields;
|
||||
use crate::admin::storage_api::bucket::versioning_sys::BucketVersioningSys;
|
||||
use crate::admin::storage_api::contract::admin::StorageAdminApi;
|
||||
use crate::admin::storage_api::contract::bucket::{BucketOperations, BucketOptions};
|
||||
use crate::admin::storage_api::data_usage::load_admin_data_usage_with_memory_overlay;
|
||||
use crate::admin::storage_api::data_usage::{apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached};
|
||||
use crate::admin::storage_api::metadata_sys;
|
||||
use crate::auth::get_condition_values;
|
||||
use crate::server::ADMIN_PREFIX;
|
||||
@@ -278,7 +278,8 @@ impl Operation for AccountInfoHandler {
|
||||
// process-local absolute counters to cluster-wide bucket totals.
|
||||
// This path never triggers a live full-version listing
|
||||
// (rustfs/backlog#1306); freshness is owned by the scanner.
|
||||
let data_usage_info = map_data_usage_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
|
||||
let mut data_usage_info = map_data_usage_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
|
||||
apply_bucket_usage_memory_overlay(&mut data_usage_info).await;
|
||||
account_info.usage_snapshot_converged = data_usage_info.usage_snapshot_converged;
|
||||
|
||||
for bucket in buckets.iter() {
|
||||
|
||||
@@ -259,7 +259,7 @@ mod tests {
|
||||
}
|
||||
|
||||
// Note: Testing the actual async handler implementations requires:
|
||||
// 1. Owned admin request setup with proper headers, URI, and credentials
|
||||
// 1. S3Request setup with proper headers, URI, and credentials
|
||||
// 2. Global object store initialization
|
||||
// 3. IAM system initialization
|
||||
// 4. Mock or real backend services
|
||||
|
||||
@@ -77,6 +77,37 @@ fn table_catalog_handler_request(context: Arc<AppContext>, access_key: &str, sec
|
||||
load_table_request_builder(context, access_key, secret_key).build()
|
||||
}
|
||||
|
||||
/// Converts a legacy request the way the admin router edge does, without its body.
|
||||
fn legacy_request_view(req: &s3s::S3Request<s3s::Body>) -> AdminRequest {
|
||||
AdminRequest::new(RequestEnvelope::from_s3s(req), Body::empty())
|
||||
}
|
||||
|
||||
/// Builds a legacy request to exercise conversion into the owned request.
|
||||
fn legacy_table_catalog_handler_request(
|
||||
context: Arc<AppContext>,
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> s3s::S3Request<s3s::Body> {
|
||||
let slot = ServerContextSlot::new();
|
||||
assert!(slot.install(context));
|
||||
let mut extensions = http::Extensions::new();
|
||||
extensions.insert(slot);
|
||||
s3s::S3Request {
|
||||
input: s3s::Body::empty(),
|
||||
method: Method::GET,
|
||||
uri: LOAD_TABLE_URI.parse().expect("load table URI"),
|
||||
headers: HeaderMap::new(),
|
||||
extensions,
|
||||
credentials: Some(s3s::auth::Credentials {
|
||||
access_key: access_key.to_string(),
|
||||
secret_key: s3s::auth::SecretKey::from(secret_key.to_string()),
|
||||
}),
|
||||
region: None,
|
||||
service: None,
|
||||
trailing_headers: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn with_access_delegation(request: TestRequest, values: &[&str]) -> AdminRequest {
|
||||
values
|
||||
.iter()
|
||||
@@ -150,13 +181,15 @@ async fn call_catalog_backup_prepare_handler(request: AdminRequest) -> S3Result<
|
||||
PrepareTableCatalogBackupObjectHandler {}.call(request, matched.params).await
|
||||
}
|
||||
|
||||
async fn call_disable_table_bucket_handler(request: AdminRequest) -> S3Result<AdminResponse> {
|
||||
/// Calls the disable handler with a request converted the same way the admin router edge converts it.
|
||||
async fn call_disable_table_bucket_handler(request: s3s::S3Request<s3s::Body>) -> S3Result<AdminResponse> {
|
||||
let mut router = matchit::Router::new();
|
||||
router.insert("/iceberg/v1/buckets/{warehouse}", ()).unwrap();
|
||||
router.insert("/_iceberg/v1/buckets/{warehouse}", ()).unwrap();
|
||||
let path = request.uri().path().to_string();
|
||||
let path = request.uri.path().to_string();
|
||||
let matched = router.at(&path).unwrap();
|
||||
DisableTableBucketHandler {}.call(request, matched.params).await
|
||||
let owned = AdminRequest::new(RequestEnvelope::from_s3s(&request), request.input.into());
|
||||
DisableTableBucketHandler {}.call(owned, matched.params).await
|
||||
}
|
||||
|
||||
async fn call_maintenance_worker_handler(request: AdminRequest) -> S3Result<AdminResponse> {
|
||||
@@ -1334,11 +1367,14 @@ async fn warehouse_location_claim_checks_request_identity_and_relocation() {
|
||||
..Default::default()
|
||||
}));
|
||||
// Exercise the actual owned internal authorization path, including its temporary target.
|
||||
let mut owned = table_catalog_handler_request(context.clone(), access_key, secret_key);
|
||||
let legacy = legacy_table_catalog_handler_request(context.clone(), access_key, secret_key);
|
||||
let mut owned = legacy_request_view(&legacy);
|
||||
let principal = table_catalog_request_principal(&owned).await.expect("resolve principal");
|
||||
// The owned request keeps the region it was given. An already installed region wins.
|
||||
// Copying a legacy s3s region into the envelope is covered by from_s3s_copies_every_named_extension.
|
||||
// Both request boundaries retain the effective region and the authenticated principal.
|
||||
{
|
||||
let mut legacy_region = legacy_table_catalog_handler_request(context.clone(), access_key, secret_key);
|
||||
legacy_region.region = Some("eu-west-1".parse().expect("valid legacy region"));
|
||||
let legacy_request = legacy_request_view(&legacy_region);
|
||||
let owned_request = load_table_request_builder(context.clone(), access_key, secret_key)
|
||||
.region(Some("eu-west-1".to_owned()))
|
||||
.build();
|
||||
@@ -1351,6 +1387,7 @@ async fn warehouse_location_claim_checks_request_identity_and_relocation() {
|
||||
})
|
||||
.build();
|
||||
for (mut request, expected_region, expected_bucket) in [
|
||||
(legacy_request, "eu-west-1", None),
|
||||
(owned_request, "eu-west-1", None),
|
||||
(preserved_request, "ap-south-1", Some("original-bucket")),
|
||||
] {
|
||||
@@ -1393,9 +1430,11 @@ async fn warehouse_location_claim_checks_request_identity_and_relocation() {
|
||||
// The exact-key check selects internal-object mode: a request header the object
|
||||
// write does not apply (a malformed x-amz-tagging) must not reach the decision,
|
||||
// while the ordinary owned entry point still evaluates it and rejects it.
|
||||
let mut tagged = load_table_request_builder(context.clone(), access_key, secret_key)
|
||||
.header("x-amz-tagging", "classification=%ZZ")
|
||||
.build();
|
||||
let mut tagged = legacy_table_catalog_handler_request(context.clone(), access_key, secret_key);
|
||||
tagged
|
||||
.headers
|
||||
.insert("x-amz-tagging", http::HeaderValue::from_static("classification=%ZZ"));
|
||||
let mut tagged = legacy_request_view(&tagged);
|
||||
install_table_catalog_s3_request_info(&mut tagged, &principal).expect("install authorization context");
|
||||
authorize_table_catalog_s3_actions(&mut tagged, "warehouse", "owned/writable", &[S3Action::PutObjectAction])
|
||||
.await
|
||||
@@ -1416,7 +1455,8 @@ async fn warehouse_location_claim_checks_request_identity_and_relocation() {
|
||||
.await;
|
||||
objects.seed_object("warehouse", "shared/denied", b"unchanged".to_vec()).await;
|
||||
let backend = || {
|
||||
let mut request = table_catalog_handler_request(context.clone(), access_key, secret_key);
|
||||
let legacy = legacy_table_catalog_handler_request(context.clone(), access_key, secret_key);
|
||||
let mut request = legacy_request_view(&legacy);
|
||||
install_table_catalog_s3_request_info(&mut request, &principal).expect("install authorization context");
|
||||
TableCommitObjectBackend::for_request(objects.clone(), request)
|
||||
};
|
||||
@@ -1548,7 +1588,7 @@ async fn warehouse_location_claim_checks_request_identity_and_relocation() {
|
||||
] {
|
||||
let backend = TableCommitObjectBackend::for_request(
|
||||
TestTableCatalogObjectBackend::default(),
|
||||
table_catalog_handler_request(context.clone(), access_key, secret_key),
|
||||
legacy_request_view(&legacy_table_catalog_handler_request(context.clone(), access_key, secret_key)),
|
||||
);
|
||||
let mut target = metadata.clone();
|
||||
target["location"] = serde_json::json!(location);
|
||||
@@ -18310,13 +18350,10 @@ async fn cleanup_empty_table_bucket_handler_enforces_request_object_permissions(
|
||||
let metadata_objects =
|
||||
create_dropped_empty_tables_for_cleanup(&catalog, &backend, object_store.as_ref(), &bucket, 2).await;
|
||||
let request = |prefix: &str| {
|
||||
table_catalog_request_builder(
|
||||
context.clone(),
|
||||
Some((access_key, secret_key)),
|
||||
Method::DELETE,
|
||||
&format!("{prefix}/v1/buckets/{bucket}?cleanup=true"),
|
||||
)
|
||||
.build()
|
||||
let mut request = legacy_table_catalog_handler_request(context.clone(), access_key, secret_key);
|
||||
request.method = Method::DELETE;
|
||||
request.uri = format!("{prefix}/v1/buckets/{bucket}?cleanup=true").parse().unwrap();
|
||||
request
|
||||
};
|
||||
let object_resource = format!("arn:aws:s3:::{bucket}/*");
|
||||
let first_metadata_resource = format!("arn:aws:s3:::{bucket}/{}", metadata_objects[0]);
|
||||
|
||||
@@ -945,6 +945,10 @@ pub(crate) static ERR_TIER_RESERVED_NAME: AdminErrorRef = AdminErrorRef(|| &ecst
|
||||
pub(crate) mod data_usage {
|
||||
use std::sync::Arc;
|
||||
|
||||
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
|
||||
crate::storage::storage_api::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
|
||||
}
|
||||
|
||||
pub(crate) async fn load_data_usage_from_backend(
|
||||
store: Arc<crate::storage::storage_api::ECStore>,
|
||||
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
|
||||
@@ -956,12 +960,6 @@ pub(crate) mod data_usage {
|
||||
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
|
||||
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_from_backend_cached(store).await
|
||||
}
|
||||
|
||||
pub(crate) async fn load_admin_data_usage_with_memory_overlay(
|
||||
store: Arc<crate::storage::storage_api::ECStore>,
|
||||
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
|
||||
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_with_memory_overlay(store).await
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) mod access {
|
||||
@@ -1085,10 +1083,20 @@ pub(crate) mod runtime {
|
||||
}
|
||||
|
||||
pub(crate) mod s3 {
|
||||
pub(crate) use http::header;
|
||||
// Keep auth types behind the existing s3 facade so test-only callers do
|
||||
// not widen the repository's direct s3s import footprint.
|
||||
pub(crate) use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
|
||||
#[allow(unused_imports)]
|
||||
pub(crate) use s3s::auth;
|
||||
pub(crate) use s3s::header;
|
||||
|
||||
/// Build an owned `S3Error` from a code and a message.
|
||||
/// Build an `S3Error` without reaching for the `s3s` error macro.
|
||||
///
|
||||
/// The macro expands to the very constructor this calls, but it puts an
|
||||
/// `s3s` dependency in every file that reports an error. Routing the
|
||||
/// construction through here keeps that dependency in this facade, which is
|
||||
/// the boundary the s3gate migration replaces
|
||||
/// (`scripts/check_s3s_footprint.sh`, rustfs/backlog#1677 F1).
|
||||
pub(crate) fn error(code: S3ErrorCode, message: impl Into<std::borrow::Cow<'static, str>>) -> S3Error {
|
||||
S3Error::with_message(code, message)
|
||||
}
|
||||
|
||||
@@ -49,7 +49,8 @@ pub(crate) async fn read_compatible_admin_body(
|
||||
path: &str,
|
||||
secret_key: &str,
|
||||
) -> S3Result<Vec<u8>> {
|
||||
// Callers pass an owned body. One length limit applies to every read.
|
||||
// Legacy handlers still hand over the s3s body; owned handlers pass the
|
||||
// RustFS body. Both read through the owned type so one limit applies.
|
||||
let mut input: rustfs_s3_types::Body = input.into();
|
||||
let body = input.store_all_limited(max_len).await.map_err(|e| {
|
||||
if e.is::<rustfs_s3_types::BodyTooLarge>() {
|
||||
@@ -241,6 +242,13 @@ mod tests {
|
||||
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
|
||||
assert_eq!(err.message(), Some("failed to read request body: length limit exceeded"));
|
||||
|
||||
let legacy_body = s3s::Body::http_body(http_body_util::Full::new(bytes::Bytes::from_static(b"nine bytes!")));
|
||||
let legacy_err = read_compatible_admin_body(legacy_body, 8, "/rustfs/admin/v3/list-users", "secret")
|
||||
.await
|
||||
.expect_err("legacy streaming bodies retain the same limit error");
|
||||
assert_eq!(legacy_err.code(), err.code());
|
||||
assert_eq!(legacy_err.message(), err.message());
|
||||
|
||||
let exact =
|
||||
read_compatible_admin_body(rustfs_s3_types::Body::from("12345678"), 8, "/rustfs/admin/v3/list-users", "secret")
|
||||
.await
|
||||
|
||||
@@ -21,7 +21,9 @@ use super::storage_api::admin_usecase::capacity::{
|
||||
};
|
||||
use super::storage_api::admin_usecase::contract::StorageAdminApi;
|
||||
use super::storage_api::admin_usecase::contract::bucket::{BucketOperations as _, BucketOptions};
|
||||
use super::storage_api::admin_usecase::data_usage::load_admin_data_usage_with_memory_overlay;
|
||||
use super::storage_api::admin_usecase::data_usage::{
|
||||
apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached,
|
||||
};
|
||||
use super::storage_api::admin_usecase::{ECStore, EndpointServerPools};
|
||||
use crate::app::runtime_sources::{
|
||||
AppContext, current_app_context, current_endpoints_handle, current_object_store_handle_for_context,
|
||||
@@ -270,7 +272,7 @@ impl DefaultAdminUsecase {
|
||||
info.buckets_usage.retain(|bucket, _| current_buckets.contains(bucket));
|
||||
info.bucket_sizes.retain(|bucket, _| current_buckets.contains(bucket));
|
||||
|
||||
if info.buckets_usage.is_empty() && !current_buckets.is_empty() && !info.usage_snapshot_admin_projection {
|
||||
if info.buckets_usage.is_empty() && !current_buckets.is_empty() {
|
||||
*info = DataUsageInfo::default();
|
||||
return;
|
||||
}
|
||||
@@ -285,14 +287,19 @@ impl DefaultAdminUsecase {
|
||||
}
|
||||
}
|
||||
|
||||
/// Keep already identity-bound display values without accepting a display
|
||||
/// projection as a persistable or quota-authoritative snapshot.
|
||||
fn present_identity_bound_data_usage_snapshot(
|
||||
mut info: DataUsageInfo,
|
||||
buckets: impl IntoIterator<Item = String>,
|
||||
) -> DataUsageInfo {
|
||||
let snapshot_measured =
|
||||
info.usage_snapshot_admin_projection || info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
|
||||
/// Overlay single-process counts on a persisted snapshot and narrow it
|
||||
/// for display.
|
||||
///
|
||||
/// Whether the snapshot measured anything is decided before the overlay.
|
||||
/// The overlay clears `usage_snapshot_complete` when it holds counts it
|
||||
/// cannot reconcile with the snapshot, such as writes into a bucket that
|
||||
/// has no verified baseline. That makes the totals unconverged, not the
|
||||
/// scanned buckets unusable; treating it as structural incompleteness
|
||||
/// blanked usage for every bucket for as long as one such bucket kept
|
||||
/// receiving writes, because each write restarts its scanner fence.
|
||||
async fn present_data_usage_snapshot(mut info: DataUsageInfo, buckets: impl IntoIterator<Item = String>) -> DataUsageInfo {
|
||||
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
|
||||
apply_bucket_usage_memory_overlay(&mut info).await;
|
||||
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, snapshot_measured, buckets);
|
||||
info
|
||||
}
|
||||
@@ -338,7 +345,7 @@ impl DefaultAdminUsecase {
|
||||
/// This path never triggers a live full-version listing
|
||||
/// (rustfs/backlog#1306); freshness is owned by the scanner.
|
||||
pub(crate) async fn query_data_usage_info_with_store(store: Arc<ECStore>) -> AdminUsecaseResult<DataUsageInfo> {
|
||||
let info = Self::map_data_usage_load_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
|
||||
let info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
|
||||
let buckets = store
|
||||
.list_bucket(&BucketOptions {
|
||||
cached: true,
|
||||
@@ -347,7 +354,7 @@ impl DefaultAdminUsecase {
|
||||
})
|
||||
.await
|
||||
.map_err(|_| Self::app_error(S3ErrorCode::InternalError, "list_bucket failed"))?;
|
||||
let mut info = Self::present_identity_bound_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name));
|
||||
let mut info = Self::present_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name)).await;
|
||||
|
||||
let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
|
||||
|
||||
@@ -972,52 +979,6 @@ mod tests {
|
||||
assert_eq!(info.buckets_count, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identity_bound_partial_display_keeps_valid_buckets_and_unknown_state() {
|
||||
use std::collections::HashMap;
|
||||
|
||||
let mut projected = DataUsageInfo {
|
||||
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
|
||||
usage_snapshot_admin_projection: true,
|
||||
usage_snapshot_partial: true,
|
||||
usage_snapshot_converged: Some(false),
|
||||
buckets_count: 1,
|
||||
objects_total_count: 7,
|
||||
objects_total_size: 70,
|
||||
buckets_usage: HashMap::from([(
|
||||
"cold".to_string(),
|
||||
rustfs_data_usage::BucketUsageInfo {
|
||||
size: 70,
|
||||
objects_count: 7,
|
||||
..Default::default()
|
||||
},
|
||||
)]),
|
||||
bucket_sizes: HashMap::from([("cold".to_string(), 70)]),
|
||||
..Default::default()
|
||||
};
|
||||
projected = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
|
||||
projected,
|
||||
["cold".to_string(), "recreated".to_string()],
|
||||
);
|
||||
assert_eq!(projected.buckets_usage["cold"].objects_count, 7);
|
||||
assert!(!projected.buckets_usage.contains_key("recreated"));
|
||||
assert_eq!(projected.objects_total_count, 7);
|
||||
assert_eq!(projected.usage_snapshot_converged, Some(false));
|
||||
assert!(!projected.is_complete_bucket_usage_snapshot());
|
||||
assert!(!projected.is_valid_partial_snapshot());
|
||||
|
||||
projected.buckets_usage.clear();
|
||||
projected.bucket_sizes.clear();
|
||||
projected.calculate_totals();
|
||||
projected.buckets_count = 0;
|
||||
let unknown = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(projected, ["recreated".to_string()]);
|
||||
assert!(unknown.buckets_usage.is_empty());
|
||||
assert!(unknown.usage_snapshot_admin_projection);
|
||||
assert_eq!(unknown.usage_snapshot_converged, Some(false));
|
||||
assert!(!unknown.is_complete_bucket_usage_snapshot());
|
||||
assert_eq!(unknown.last_update, Some(std::time::SystemTime::UNIX_EPOCH));
|
||||
}
|
||||
|
||||
/// A write into a bucket without a verified baseline leaves process-local
|
||||
/// counts the overlay cannot reconcile, so it clears
|
||||
/// `usage_snapshot_complete`. That must flag the response non-converged,
|
||||
@@ -1048,15 +1009,11 @@ mod tests {
|
||||
|
||||
crate::app::storage_api::test::data_usage::record_bucket_object_write_memory(unverified_bucket, None, 1_024).await;
|
||||
|
||||
// Exercise the production overlay, followed by the same identity-bound
|
||||
// presentation function used by the admin query.
|
||||
crate::app::storage_api::test::data_usage::apply_bucket_usage_memory_overlay(&mut persisted).await;
|
||||
persisted.usage_snapshot_admin_projection = true;
|
||||
persisted.usage_snapshot_converged = Some(false);
|
||||
let info = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
|
||||
let info = DefaultAdminUsecase::present_data_usage_snapshot(
|
||||
persisted,
|
||||
[measured_bucket.to_string(), unverified_bucket.to_string()],
|
||||
);
|
||||
)
|
||||
.await;
|
||||
|
||||
assert_eq!(
|
||||
info.buckets_usage
|
||||
|
||||
@@ -906,7 +906,9 @@ mod tests {
|
||||
.expect_err("full queue rejects immediately");
|
||||
assert_eq!(error.code(), &S3ErrorCode::SlowDown);
|
||||
assert_eq!(error.headers().expect("retry headers")[http::header::RETRY_AFTER], "1");
|
||||
let response = error.to_http_response().expect("serialize admission error");
|
||||
let response = s3s::S3Error::from(error)
|
||||
.to_http_response()
|
||||
.expect("serialize admission error");
|
||||
assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE);
|
||||
assert_eq!(response.headers()[http::header::RETRY_AFTER], "1");
|
||||
use http_body_util::BodyExt as _;
|
||||
|
||||
@@ -108,17 +108,6 @@ async fn persist_scanner_snapshot(ecstore: Arc<ECStore>, info: DataUsageInfo) {
|
||||
rustfs_scanner::scanner::store_data_usage_in_backend(CancellationToken::new(), ecstore, receiver).await;
|
||||
}
|
||||
|
||||
async fn bind_current_bucket_incarnations(ecstore: &ECStore, info: &mut DataUsageInfo) {
|
||||
for (bucket, usage) in &mut info.buckets_usage {
|
||||
let incarnation = ecstore
|
||||
.bucket_incarnation_id_from_disk(bucket)
|
||||
.await
|
||||
.expect("load the current incarnation for the covered snapshot");
|
||||
assert!(!incarnation.is_nil(), "covered snapshot must bind a non-nil bucket identity");
|
||||
usage.bucket_incarnation = Some(incarnation.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
|
||||
@@ -185,7 +174,6 @@ async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
|
||||
seeded_info.bucket_sizes.insert(overlay_bucket.clone(), 0);
|
||||
seeded_info.buckets_count = u64::try_from(seeded_info.buckets_usage.len()).expect("test bucket count should fit into u64");
|
||||
seeded_info.calculate_totals();
|
||||
bind_current_bucket_incarnations(&ecstore, &mut seeded_info).await;
|
||||
persist_scanner_snapshot(ecstore.clone(), seeded_info).await;
|
||||
|
||||
record_bucket_object_write_memory(&overlay_bucket, None, 512).await;
|
||||
@@ -353,7 +341,6 @@ async fn data_usage_endpoint_treats_legacy_partial_stats_as_unknown_until_covere
|
||||
completed.bucket_sizes.insert(small_bucket.clone(), 57_344);
|
||||
completed.buckets_count = u64::try_from(completed.buckets_usage.len()).expect("test bucket count should fit into u64");
|
||||
completed.calculate_totals();
|
||||
bind_current_bucket_incarnations(&ecstore, &mut completed).await;
|
||||
store_data_usage_in_backend(completed, ecstore.clone())
|
||||
.await
|
||||
.expect("persist complete covered snapshot");
|
||||
|
||||
@@ -208,7 +208,7 @@ impl DefaultMultipartUsecase {
|
||||
let (body, total) =
|
||||
crate::app::object::cuobject::rdma_put_body(bucket, key, &token, true).map_err(S3Error::from)?;
|
||||
let total = i64::try_from(total).map_err(|_| S3Error::new(S3ErrorCode::EntityTooLarge))?;
|
||||
(Some(body), Some(total), Some(total))
|
||||
(Some(crate::app::object::legacy_put_body(body)), Some(total), Some(total))
|
||||
}
|
||||
_ => (body, content_length, None),
|
||||
};
|
||||
|
||||
@@ -27,6 +27,7 @@ use crate::app::storage_api::object_usecase::runtime::get_global_endpoints_opt;
|
||||
#[cfg(not(feature = "nixl"))]
|
||||
use crate::app::storage_api::object_usecase::runtime::global_rustfs_port;
|
||||
use bytes::Bytes;
|
||||
use futures::Stream;
|
||||
use http::HeaderMap;
|
||||
use rustfs_config::{
|
||||
DEFAULT_ERASURE_SHARD_BYTES, DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT, DEFAULT_RUSTFS_CUOBJECT_GET_FANOUT_BUDGET,
|
||||
@@ -55,8 +56,9 @@ pub(crate) mod transport {
|
||||
};
|
||||
}
|
||||
use crate::app::storage_api::object_usecase::rdma::{RegisteredSlabArena, ShardBuf, ShardEmitter, install_shard_arena};
|
||||
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Result};
|
||||
use rustfs_gateway_stream::Body;
|
||||
use crate::app::storage_api::s3::dto::StreamingBlob;
|
||||
use crate::app::storage_api::s3::stream::{ByteStream, RemainingLength};
|
||||
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Result, StdError};
|
||||
use rustfs_utils::{get_env_bool, get_env_opt_str, get_env_u64, get_env_usize};
|
||||
use std::collections::VecDeque;
|
||||
use std::io;
|
||||
@@ -778,6 +780,36 @@ pub(crate) async fn serve_get_over_rdma(
|
||||
Ok(moved)
|
||||
}
|
||||
|
||||
/// Request body of a cuObject PUT: the object bytes are RDMA-read from the
|
||||
/// client's buffer into staging, one chunk ahead of the consumer, and handed
|
||||
/// to the normal PUT pipeline as an ordinary byte stream.
|
||||
struct RdmaPutBody {
|
||||
inner: Pin<Box<dyn Stream<Item = Result<Bytes, StdError>> + Send + Sync + 'static>>,
|
||||
remaining: usize,
|
||||
}
|
||||
|
||||
impl Stream for RdmaPutBody {
|
||||
type Item = Result<Bytes, StdError>;
|
||||
|
||||
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
|
||||
let next = self.inner.as_mut().poll_next(cx);
|
||||
if let Poll::Ready(Some(Ok(bytes))) = &next {
|
||||
self.remaining = self.remaining.saturating_sub(bytes.len());
|
||||
}
|
||||
next
|
||||
}
|
||||
|
||||
fn size_hint(&self) -> (usize, Option<usize>) {
|
||||
(self.remaining, Some(self.remaining))
|
||||
}
|
||||
}
|
||||
|
||||
impl ByteStream for RdmaPutBody {
|
||||
fn remaining_length(&self) -> RemainingLength {
|
||||
RemainingLength::new_exact(self.remaining)
|
||||
}
|
||||
}
|
||||
|
||||
type StagedRead = JoinHandle<Result<StagingChunk, CuObjectError>>;
|
||||
|
||||
/// The staged bytes of `chunk`: with `zero_copy`, a `Bytes` that owns the
|
||||
@@ -814,14 +846,13 @@ fn spawn_staged_read(server: &Arc<CuObjectServer>, object: &str, token: &RdmaTok
|
||||
/// consumer that keeps a chunk keeps it out of the pool: only callers whose
|
||||
/// body is consumed and dropped with the request (UploadPart) may set it.
|
||||
/// PutObject may hand its body to the object data cache, so it copies.
|
||||
pub(crate) fn rdma_put_body(bucket: &str, key: &str, token: &RdmaToken, zero_copy: bool) -> S3Result<(Body, u64)> {
|
||||
pub(crate) fn rdma_put_body(bucket: &str, key: &str, token: &RdmaToken, zero_copy: bool) -> S3Result<(StreamingBlob, u64)> {
|
||||
let service = SERVICE
|
||||
.get()
|
||||
.ok_or_else(|| S3Error::with_message(S3ErrorCode::NotImplemented, "rdma put is not enabled"))?;
|
||||
let server = Arc::clone(service.pick());
|
||||
let read_ahead = service.put_read_ahead;
|
||||
let total = token.buffer_len;
|
||||
let length = usize::try_from(total).map_err(|_| S3Error::with_message(S3ErrorCode::EntityTooLarge, "rdma put too large"))?;
|
||||
let chunk_bytes = server.chunk_bytes();
|
||||
let object = format!("{bucket}/{key}");
|
||||
let token = token.clone();
|
||||
@@ -862,7 +893,13 @@ pub(crate) fn rdma_put_body(bucket: &str, key: &str, token: &RdmaToken, zero_cop
|
||||
Ok::<_, io::Error>(Some((bytes, (pending, next_offset))))
|
||||
}
|
||||
});
|
||||
Ok((super::native_body::from_io_stream_with_length(stream, Some(length)), total))
|
||||
let stream = futures::TryStreamExt::map_err(stream, |error: io::Error| -> StdError { Box::new(error) });
|
||||
let body = RdmaPutBody {
|
||||
inner: Box::pin(stream),
|
||||
remaining: usize::try_from(total)
|
||||
.map_err(|_| S3Error::with_message(S3ErrorCode::EntityTooLarge, "rdma put too large"))?,
|
||||
};
|
||||
Ok((StreamingBlob::new(body), total))
|
||||
}
|
||||
|
||||
/// State shared by the segment writers of one RDMA GET: the server, the
|
||||
|
||||
@@ -929,7 +929,6 @@ pin_project! {
|
||||
struct GetObjectReaderStream<R> {
|
||||
#[pin]
|
||||
reader: Option<R>,
|
||||
primed_byte: Option<u8>,
|
||||
capacity: usize,
|
||||
strategy: &'static str,
|
||||
buffer_source: &'static str,
|
||||
@@ -994,7 +993,6 @@ where
|
||||
}
|
||||
Self {
|
||||
reader: Some(reader),
|
||||
primed_byte: None,
|
||||
capacity,
|
||||
strategy,
|
||||
buffer_source,
|
||||
@@ -1439,14 +1437,9 @@ where
|
||||
Some(reader) => reader,
|
||||
None => return Poll::Ready(None),
|
||||
};
|
||||
let (poll_read, buf) = if let Some(byte) = this.primed_byte.take() {
|
||||
(Poll::Ready(Ok(1)), BytesMut::from(&[byte][..]))
|
||||
} else {
|
||||
let read_capacity = (*this.capacity).min(*this.remaining);
|
||||
let mut buf = BytesMut::with_capacity(read_capacity);
|
||||
let poll_read = poll_read_buf(reader, cx, &mut buf);
|
||||
(poll_read, buf)
|
||||
};
|
||||
let read_capacity = (*this.capacity).min(*this.remaining);
|
||||
let mut buf = BytesMut::with_capacity(read_capacity);
|
||||
let poll_read = poll_read_buf(reader, cx, &mut buf);
|
||||
|
||||
let result: Poll<Option<Self::Item>> = match poll_read {
|
||||
Poll::Ready(Ok(bytes_read)) if bytes_read > 0 => {
|
||||
@@ -2521,21 +2514,19 @@ impl DefaultObjectUsecase {
|
||||
lifecycle,
|
||||
resume,
|
||||
);
|
||||
let primed_byte = if expected == 0 {
|
||||
None
|
||||
let mut prefix = [0_u8; 1];
|
||||
let prefix_len = if expected == 0 {
|
||||
0
|
||||
} else {
|
||||
let mut prefix = [0_u8; 1];
|
||||
reader
|
||||
.read_exact(&mut prefix)
|
||||
.await
|
||||
.map_err(|error| map_get_object_reader_error(StorageError::from(error)))?;
|
||||
Some(prefix[0])
|
||||
1
|
||||
};
|
||||
let mut stream =
|
||||
GetObjectReaderStream::new(reader, stream_buffer_size, expected, stream_strategy.as_str(), buffer_source)
|
||||
.with_diagnostics(bucket, key, request_id);
|
||||
// Keep the preflight byte in the stream until the transport asks for it.
|
||||
stream.primed_byte = primed_byte;
|
||||
let reader = std::io::Cursor::new(prefix).take(prefix_len).chain(reader);
|
||||
let stream = GetObjectReaderStream::new(reader, stream_buffer_size, expected, stream_strategy.as_str(), buffer_source)
|
||||
.with_diagnostics(bucket, key, request_id);
|
||||
let blob = GetObjectBody::Stream(Box::pin(stream));
|
||||
if let Some(handoff_start) = handoff_start {
|
||||
rustfs_io_metrics::record_get_object_response_handoff(
|
||||
@@ -9019,7 +9010,7 @@ mod tests {
|
||||
let error = result.expect_err("a read quorum failure before the first byte must reject response construction");
|
||||
assert_eq!(error.code(), &S3ErrorCode::Custom("SlowDownRead".into()));
|
||||
assert_eq!(error.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE));
|
||||
assert_eq!(error.message(), Some("The object could not be read from storage."));
|
||||
assert_eq!(error.message(), Some("Resource requested is unreadable, please reduce your request rate"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -9046,20 +9037,7 @@ mod tests {
|
||||
.expect("the first byte should be available before response handoff");
|
||||
|
||||
assert_eq!(reads.load(AtomicOrdering::Relaxed), 1, "response construction must prime one byte");
|
||||
let first = body
|
||||
.next()
|
||||
.await
|
||||
.expect("the primed byte should be present")
|
||||
.expect("the first chunk should be readable");
|
||||
assert_eq!(&first[..], b"h");
|
||||
let first = first.try_into_mut().expect("the first chunk should own its buffer");
|
||||
assert!(first.capacity() < 5, "the preflight byte must not reserve the remaining body capacity");
|
||||
assert_eq!(
|
||||
reads.load(AtomicOrdering::Relaxed),
|
||||
1,
|
||||
"emitting the primed byte must not poll storage again"
|
||||
);
|
||||
let mut received = first.to_vec();
|
||||
let mut received = Vec::new();
|
||||
while let Some(chunk) = body.next().await {
|
||||
received.extend_from_slice(&chunk.expect("the primed body should remain readable"));
|
||||
}
|
||||
|
||||
@@ -52,31 +52,38 @@ fn website_and_object_lambda_ranges_match_the_legacy_wire_reader() {
|
||||
|
||||
#[test]
|
||||
fn website_and_object_lambda_conditions_match_the_legacy_wire_reader() {
|
||||
// Answers measured on s3s 5761ddfe (`ETagCondition::parse_http_header` and
|
||||
// `Timestamp::parse(HttpDate)`). The owned checks must keep those answers.
|
||||
for (value, accepted) in [
|
||||
(b"*".as_slice(), true),
|
||||
(b"abc-2", true),
|
||||
(b"\"\"", true),
|
||||
(b"W/\"x\"", true),
|
||||
(b"\"a \tb\"", true),
|
||||
(b"\"a\"b\"", true),
|
||||
(b"", false),
|
||||
(b"bad etag", false),
|
||||
(b"w/\"x\"", false),
|
||||
(b"\"\xff\"", false),
|
||||
(b"\"\x7f\"", false),
|
||||
(b"\"x", false),
|
||||
use s3s::dto::{ETagCondition, Timestamp, TimestampFormat};
|
||||
for value in [
|
||||
b"*".as_slice(),
|
||||
b"abc-2",
|
||||
b"\"\"",
|
||||
b"W/\"x\"",
|
||||
b"\"a \tb\"",
|
||||
b"\"a\"b\"",
|
||||
b"",
|
||||
b"bad etag",
|
||||
b"w/\"x\"",
|
||||
b"\"\xff\"",
|
||||
b"\"\x7f\"",
|
||||
b"\"x",
|
||||
] {
|
||||
assert_eq!(GetObjectFields::valid_etag_condition(value), accepted, "{value:?}");
|
||||
assert_eq!(
|
||||
GetObjectFields::valid_etag_condition(value),
|
||||
ETagCondition::parse_http_header(value).is_ok(),
|
||||
"{value:?}"
|
||||
);
|
||||
}
|
||||
for (value, accepted) in [
|
||||
("Sun, 06 Nov 1994 08:49:37 GMT", true),
|
||||
("Sunday, 06-Nov-94 08:49:37 GMT", false),
|
||||
("2025-01-01", false),
|
||||
("Sun, 32 Nov 1994 08:49:37 GMT", false),
|
||||
for value in [
|
||||
"Sun, 06 Nov 1994 08:49:37 GMT",
|
||||
"Sunday, 06-Nov-94 08:49:37 GMT",
|
||||
"2025-01-01",
|
||||
"Sun, 32 Nov 1994 08:49:37 GMT",
|
||||
] {
|
||||
assert_eq!(GetObjectFields::valid_date_condition(value), accepted, "{value}");
|
||||
assert_eq!(
|
||||
GetObjectFields::valid_date_condition(value),
|
||||
Timestamp::parse(TimestampFormat::HttpDate, value).is_ok(),
|
||||
"{value}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -182,26 +182,17 @@ impl Stream for BodyChunks {
|
||||
}
|
||||
|
||||
pub(super) fn from_io_stream<S>(stream: S) -> Body
|
||||
where
|
||||
S: Stream<Item = std::io::Result<Bytes>> + Send + 'static,
|
||||
{
|
||||
from_io_stream_with_length(stream, None)
|
||||
}
|
||||
|
||||
pub(super) fn from_io_stream_with_length<S>(stream: S, length: Option<usize>) -> Body
|
||||
where
|
||||
S: Stream<Item = std::io::Result<Bytes>> + Send + 'static,
|
||||
{
|
||||
Body::from_payload(Payload::Stream(Box::pin(IoPayload {
|
||||
stream: Box::pin(stream),
|
||||
remaining: length,
|
||||
ended: false,
|
||||
})))
|
||||
}
|
||||
|
||||
struct IoPayload<S> {
|
||||
stream: Pin<Box<S>>,
|
||||
remaining: Option<usize>,
|
||||
ended: bool,
|
||||
}
|
||||
|
||||
@@ -214,12 +205,7 @@ impl<S: Stream<Item = std::io::Result<Bytes>>> PayloadStream for IoPayload<S> {
|
||||
match self.stream.as_mut().poll_next(cx) {
|
||||
Poll::Pending => return Poll::Pending,
|
||||
Poll::Ready(Some(Ok(bytes))) if bytes.is_empty() => continue,
|
||||
Poll::Ready(Some(Ok(bytes))) => {
|
||||
if let Some(remaining) = &mut self.remaining {
|
||||
*remaining = remaining.saturating_sub(bytes.len());
|
||||
}
|
||||
return Poll::Ready(Ok(PayloadRead::Chunk(bytes)));
|
||||
}
|
||||
Poll::Ready(Some(Ok(bytes))) => return Poll::Ready(Ok(PayloadRead::Chunk(bytes))),
|
||||
Poll::Ready(Some(Err(error))) => {
|
||||
self.ended = true;
|
||||
return Poll::Ready(Err(StreamError::upstream(Box::new(ProducerSource(Box::new(error))))));
|
||||
@@ -236,12 +222,10 @@ impl<S: Stream<Item = std::io::Result<Bytes>>> PayloadStream for IoPayload<S> {
|
||||
Poll::Pending
|
||||
}
|
||||
fn caps(&self) -> PayloadCaps {
|
||||
let mut caps = PayloadCaps::PUSH;
|
||||
caps.set(PayloadCaps::KNOWN_LENGTH, self.remaining.is_some());
|
||||
caps
|
||||
PayloadCaps::PUSH
|
||||
}
|
||||
fn len_hint(&self) -> Option<u64> {
|
||||
self.remaining.map(|length| length as u64)
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
@@ -419,35 +403,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn io_stream_with_length_tracks_data_and_preserves_producer_error() {
|
||||
let mut body = from_io_stream_with_length(
|
||||
futures::stream::iter([
|
||||
Ok(Bytes::new()),
|
||||
Ok(Bytes::from_static(b"abc")),
|
||||
Err(std::io::Error::new(std::io::ErrorKind::TimedOut, InternalProducerFailure)),
|
||||
]),
|
||||
Some(5),
|
||||
);
|
||||
assert_eq!(body.size_hint().exact(), Some(5));
|
||||
let frame = std::future::poll_fn(|cx| Pin::new(&mut body).poll_frame(cx))
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(frame.into_data().unwrap(), Bytes::from_static(b"abc"));
|
||||
assert_eq!(body.size_hint().exact(), Some(2));
|
||||
|
||||
let (mut chunks, trailers) = BodyChunks::new(body);
|
||||
let error = chunks.next().await.unwrap().unwrap_err();
|
||||
let error = error
|
||||
.downcast_ref::<std::io::Error>()
|
||||
.expect("the original io::Error reaches RIO");
|
||||
assert_eq!(error.kind(), std::io::ErrorKind::TimedOut);
|
||||
assert!(error.get_ref().unwrap().is::<InternalProducerFailure>());
|
||||
assert_eq!(trailers.lookup("x-test"), TrailerValue::Pending);
|
||||
assert!(chunks.next().await.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reader_boundary_is_send_and_sync() {
|
||||
fn assert_send_sync<T: Send + Sync>() {}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user