diff --git a/docs/operations/scanner-benchmark-runbook.md b/docs/operations/scanner-benchmark-runbook.md index f1c173a49..3d4811fcf 100644 --- a/docs/operations/scanner-benchmark-runbook.md +++ b/docs/operations/scanner-benchmark-runbook.md @@ -177,6 +177,25 @@ response per endpoint listed in `--metrics-endpoints`, and one by-host admin metrics response per listed endpoint. When `--metrics-endpoints` is omitted, the harness captures background-heal status only from `--endpoint`. +For bucket metrics freshness validation, use the same harness around a +post-start bucket creation workload: + +1. Start RustFS from an empty data path. +2. Start the harness before creating buckets. +3. Create a bucket, upload objects, and keep the harness running until at + least one usage save is observed. +4. Compare `scanner-summary.csv` with + `/rustfs/admin/v3/metrics?types=1&n=1` bucket metrics. + +This covers the issue 3496 timing where scanner startup sees no buckets, the +bucket is created after startup, and the first metrics collection must not +confuse cold usage cache with real zero usage. The expected evidence is that +dirty usage is marked, `life_time_scan_cycle` or +`life_time_scan_bucket_drive` advances, `life_time_scan_object` advances for +object workloads, and `life_time_save_usage` plus +`usage_last_save_result=success` appear before accepting non-zero bucket usage +metrics as fresh. + For distributed runs, capture scanner admin metrics from every node with `by-host=true`. The metrics endpoint reports the node that handles the request; `by-host=true` preserves that node's host view but does not collect peer nodes. @@ -327,6 +346,16 @@ Compare these fields between baseline and tuned runs: | `metrics.lifecycle_transition.queue_full` | Shows transition queue pressure outside the scanner walk itself. | | `metrics.lifecycle_transition.compensation_pending` | Shows transition compensation still pending or running after queue pressure. | | `metrics.lifecycle_transition.failed` | Shows transition worker failures, which should also surface as lifecycle source failure. | +| `metrics.current_cycle_usage_saves` | Shows usage cache saves produced by the active scan cycle. | +| `metrics.last_cycle_usage_saves` | Shows usage cache saves produced by the previous completed or partial scan cycle. | +| `metrics.usage_freshness.dirty_pending_buckets` | Shows whether bucket/object mutations are still waiting for usage refresh. | +| `metrics.usage_freshness.last_cycle_dirty_buckets` | Shows how many dirty buckets were picked up by the last cycle. | +| `metrics.usage_freshness.last_cycle_cleared_dirty_buckets` | Shows how many dirty bucket marks were cleared by a successful cycle. | +| `metrics.usage_freshness.last_usage_save_result` | Confirms whether the last usage save succeeded, failed, or was skipped. | +| `metrics.life_time_ops.scan_cycle` | Confirms scanner cycles actually started after the workload. | +| `metrics.life_time_ops.scan_bucket_drive` | Confirms bucket-drive scan work reached the storage layer. | +| `metrics.life_time_ops.scan_object` | Confirms object metadata scanning advanced for object workloads. | +| `metrics.life_time_ops.save_usage` | Confirms `DataUsageInfo` save work happened; this is the key freshness signal for bucket metrics. | | `metrics.scan_checkpoint` | Confirms partial cycles preserve resume context. | | `metrics.oldest_active_path_age_seconds` | Helps identify scanner paths that may be stuck. | @@ -353,6 +382,23 @@ the background-heal status snapshots captured across `--metrics-endpoints`. | `heal_admin_queued` | Manual/admin heal work waiting in the queue. | | `heal_auto_heal_queued` | Auto-heal work waiting in the queue, typically from disk/set recovery paths. | +`scanner-summary.csv` also includes usage freshness columns for quick +post-start bucket metrics validation: + +| Field | Why it matters | +|---|---| +| `current_cycle_usage_saves` | Usage saves during the current cycle. | +| `last_cycle_usage_saves` | Usage saves from the last finished or partial cycle. | +| `usage_dirty_pending_buckets` | Dirty buckets still waiting for scanner refresh. | +| `usage_last_cycle_dirty_buckets` | Dirty buckets selected by the last cycle. | +| `usage_last_cycle_cleared_dirty_buckets` | Dirty bucket marks cleared by the last cycle. | +| `usage_last_save_result` | Last `DataUsageInfo` save result. | +| `usage_last_save_unix_secs` | Last `DataUsageInfo` save timestamp. | +| `life_time_scan_cycle` | Total scanner cycles observed by the node. | +| `life_time_scan_bucket_drive` | Total bucket-drive scans completed by the node. | +| `life_time_scan_object` | Total object scan operations observed by the node. | +| `life_time_save_usage` | Total usage save operations observed by the node. | + ## Interpreting Results A useful tuning result has all of these properties: @@ -377,6 +423,8 @@ Treat these as failure signals: - lifecycle transition `scanner_missed`, `queue_full`, `compensation_pending`, or `failed` grows during a run that was expected to reduce backlog; +- bucket metrics show zero usage after post-start uploads while dirty usage + remains pending and `life_time_save_usage` does not advance; - heal or bitrot work moves from `queued` to `missed` after a scanner pacing change. @@ -392,6 +440,8 @@ For scanner behavior PRs, include this evidence when available: - Observation window and sample interval. - Scanner status snapshots or time series. - Host CPU and disk telemetry. +- Usage freshness fields from `scanner-summary.csv` when validating bucket + metrics or issue 3496-style timing. - Short conclusion that separates pressure reduction from scanner progress. - `scanner-validation-report.md` from the harness when using the scripted collection path. @@ -405,6 +455,7 @@ at least these runs: | Run | Required evidence | |---|---| | Single-node, single-disk small-object idle | Scanner status series, host telemetry, `scanner-summary.csv`, and a conclusion that CPU or disk pressure is lower without scan progress stopping. | +| Single-node post-start bucket metrics freshness | Empty data path startup, post-start bucket creation/upload, bucket metrics snapshots, `scanner-summary.csv` usage freshness columns, and evidence that `DataUsageInfo` save work occurred before accepting bucket usage metrics. | | Single-node erasure or multi-disk | Checkpoint movement, active path age, set/disk scan pressure, data usage freshness, and before/after scanner config. | | Distributed lifecycle backlog | `maintenance_control`, lifecycle expiry/transition queue fields, source work missed/failed counts, and by-host admin metrics. | | Distributed replication backlog | Bucket replication repair kind counters, site replication passive/active boundary counters, source work queued/skipped/missed counts, and by-host admin metrics. | diff --git a/scripts/run_scanner_validation_harness.sh b/scripts/run_scanner_validation_harness.sh index ec69bf139..12c962203 100755 --- a/scripts/run_scanner_validation_harness.sh +++ b/scripts/run_scanner_validation_harness.sh @@ -161,7 +161,7 @@ setup_output() { mkdir -p "$OUT_DIR/heal" mkdir -p "$OUT_DIR/metrics" SUMMARY_CSV="$OUT_DIR/scanner-summary.csv" - echo "timestamp,primary_pressure,current_cycle_objects_scanned,current_cycle_directories_scanned,last_cycle_result,last_cycle_partial_reason,last_cycle_partial_source,lifecycle_transition_scanner_missed,source_work_missed_total,heal_queue_length,heal_active_tasks,heal_scanner_queued,heal_admin_queued,heal_auto_heal_queued" >"$SUMMARY_CSV" + echo "timestamp,primary_pressure,current_cycle_objects_scanned,current_cycle_directories_scanned,last_cycle_result,last_cycle_partial_reason,last_cycle_partial_source,lifecycle_transition_scanner_missed,source_work_missed_total,current_cycle_usage_saves,last_cycle_usage_saves,usage_dirty_pending_buckets,usage_last_cycle_dirty_buckets,usage_last_cycle_cleared_dirty_buckets,usage_last_save_result,usage_last_save_unix_secs,life_time_scan_cycle,life_time_scan_bucket_drive,life_time_scan_object,life_time_save_usage,heal_queue_length,heal_active_tasks,heal_scanner_queued,heal_admin_queued,heal_auto_heal_queued" >"$SUMMARY_CSV" } git_value() { @@ -403,6 +403,17 @@ capture_status_sample() { (.metrics.last_cycle_partial_source // ""), (.metrics.lifecycle_transition.scanner_missed // 0), ((.metrics.source_work // []) | map(.missed // 0) | add // 0), + (.metrics.current_cycle_usage_saves // 0), + (.metrics.last_cycle_usage_saves // 0), + (.metrics.usage_freshness.dirty_pending_buckets // 0), + (.metrics.usage_freshness.last_cycle_dirty_buckets // 0), + (.metrics.usage_freshness.last_cycle_cleared_dirty_buckets // 0), + (.metrics.usage_freshness.last_usage_save_result // ""), + (.metrics.usage_freshness.last_usage_save_unix_secs // 0), + (.metrics.life_time_ops.scan_cycle // 0), + (.metrics.life_time_ops.scan_bucket_drive // 0), + (.metrics.life_time_ops.scan_object // 0), + (.metrics.life_time_ops.save_usage // 0), ($heal.healOperations.queueLength // $heal.healQueueLength // 0), ($heal.healOperations.activeTasks // $heal.healActiveTasks // 0), ($heal.healOperations.queuedBySource.scanner // 0), @@ -455,11 +466,14 @@ Interval seconds: $INTERVAL_SECS - Background heal status snapshots: $heal_status_count - Distributed admin metrics snapshots: $metrics_count - Summary CSV: scanner-summary.csv +- Usage freshness summary: current/last usage saves, dirty bucket state, last usage save result, and scanner life_time_ops. - Run metadata: run-metadata.env ## Review Checklist - Compare scanner progress and pressure in scanner-summary.csv. +- Confirm dirty usage marks lead to usage saves when validating post-start bucket metrics freshness. +- Check life_time_ops scan_cycle, scan_bucket_drive, scan_object, and save_usage before accepting bucket metrics freshness results. - Check source work missed totals before accepting pressure reductions. - Check healOperations queued and active counts when heal or bitrot pressure is involved. - Use distributed admin metrics snapshots for by-host investigation when metrics endpoints were provided. diff --git a/scripts/test_scanner_validation_harness.sh b/scripts/test_scanner_validation_harness.sh index 1104b6f0f..9107926b1 100755 --- a/scripts/test_scanner_validation_harness.sh +++ b/scripts/test_scanner_validation_harness.sh @@ -85,6 +85,12 @@ cat <<'JSON' "delay": { "value": 30, "source": "config" } }, "metrics": { + "life_time_ops": { + "scan_cycle": 9, + "scan_bucket_drive": 11, + "scan_object": 13, + "save_usage": 5 + }, "pacing_pressure": { "primary_pressure": "active_scans" }, @@ -96,6 +102,18 @@ cat <<'JSON' "lifecycle_transition": { "scanner_missed": 0 }, + "current_cycle_usage_saves": 2, + "last_cycle_usage_saves": 3, + "usage_freshness": { + "dirty_pending_buckets": 1, + "last_dirty_mark_unix_secs": 1800000001, + "last_dirty_clear_unix_secs": 1800000002, + "last_cycle_dirty_buckets": 1, + "last_cycle_cleared_dirty_buckets": 1, + "last_usage_save_unix_secs": 1800000003, + "last_usage_save_result": "success", + "last_usage_save_result_code": 1 + }, "source_work": [ { "source": "usage", "missed": 0 }, { "source": "lifecycle", "missed": 0 } @@ -150,7 +168,7 @@ fi if [[ "$1" == "-r" ]]; then while [[ $# -gt 0 ]]; do if [[ "$1" == "--arg" && "${2:-}" == "ts" ]]; then - printf '"%s","active_scans",5,2,"success","","",0,0,8,2,4,2,2\n' "$3" + printf '"%s","active_scans",5,2,"success","","",0,0,2,3,1,1,1,"success",1800000003,9,11,13,5,8,2,4,2,2\n' "$3" exit 0 fi shift @@ -205,8 +223,10 @@ if [[ "$(wc -l <"$OUT_DIR/scanner-summary.csv" | tr -d ' ')" != "3" ]]; then echo "Expected scanner summary header plus 2 rows" >&2 exit 1 fi +grep -q 'current_cycle_usage_saves,last_cycle_usage_saves,usage_dirty_pending_buckets' "$OUT_DIR/scanner-summary.csv" +grep -q 'life_time_scan_cycle,life_time_scan_bucket_drive,life_time_scan_object,life_time_save_usage' "$OUT_DIR/scanner-summary.csv" grep -q 'heal_queue_length,heal_active_tasks,heal_scanner_queued,heal_admin_queued,heal_auto_heal_queued' "$OUT_DIR/scanner-summary.csv" -grep -q '"active_scans",5,2,"success","","",0,0,8,2,4,2,2' "$OUT_DIR/scanner-summary.csv" +grep -q '"active_scans",5,2,"success","","",0,0,2,3,1,1,1,"success",1800000003,9,11,13,5,8,2,4,2,2' "$OUT_DIR/scanner-summary.csv" grep -q '^deployment=single-disk$' "$OUT_DIR/run-metadata.env" grep -q '^workload_label=small-object-idle$' "$OUT_DIR/run-metadata.env" @@ -214,6 +234,7 @@ grep -q '^metrics_endpoints=http://node-a:9000,http://node-b:9000$' "$OUT_DIR/ru grep -q '## Scanner Validation Report' "$OUT_DIR/scanner-validation-report.md" grep -q 'Distributed admin metrics snapshots: 4' "$OUT_DIR/scanner-validation-report.md" grep -q 'Background heal status snapshots: 4' "$OUT_DIR/scanner-validation-report.md" +grep -q 'Usage freshness summary:' "$OUT_DIR/scanner-validation-report.md" grep -q 'admin config get rustfs-local scanner' "$mc_log" grep -q 'admin config get rustfs-local heal' "$mc_log" grep -q -- '--request GET' "$awscurl_log"