test(scanner): add usage freshness validation evidence (#3627)

docs/operations/scanner-benchmark-runbook.md

@@ -177,6 +177,25 @@ response per endpoint listed in `--metrics-endpoints`, and one by-host admin
 metrics response per listed endpoint. When `--metrics-endpoints` is omitted,
 the harness captures background-heal status only from `--endpoint`.
 
+For bucket metrics freshness validation, use the same harness around a
+post-start bucket creation workload:
+
+1. Start RustFS from an empty data path.
+2. Start the harness before creating buckets.
+3. Create a bucket, upload objects, and keep the harness running until at
+   least one usage save is observed.
+4. Compare `scanner-summary.csv` with
+   `/rustfs/admin/v3/metrics?types=1&n=1` bucket metrics.
+
+This covers the issue 3496 timing where scanner startup sees no buckets, the
+bucket is created after startup, and the first metrics collection must not
+confuse cold usage cache with real zero usage. The expected evidence is that
+dirty usage is marked, `life_time_scan_cycle` or
+`life_time_scan_bucket_drive` advances, `life_time_scan_object` advances for
+object workloads, and `life_time_save_usage` plus
+`usage_last_save_result=success` appear before accepting non-zero bucket usage
+metrics as fresh.
+
 For distributed runs, capture scanner admin metrics from every node with
 `by-host=true`. The metrics endpoint reports the node that handles the request;
 `by-host=true` preserves that node's host view but does not collect peer nodes.
@@ -327,6 +346,16 @@ Compare these fields between baseline and tuned runs:
 | `metrics.lifecycle_transition.queue_full` | Shows transition queue pressure outside the scanner walk itself. |
 | `metrics.lifecycle_transition.compensation_pending` | Shows transition compensation still pending or running after queue pressure. |
 | `metrics.lifecycle_transition.failed` | Shows transition worker failures, which should also surface as lifecycle source failure. |
+| `metrics.current_cycle_usage_saves` | Shows usage cache saves produced by the active scan cycle. |
+| `metrics.last_cycle_usage_saves` | Shows usage cache saves produced by the previous completed or partial scan cycle. |
+| `metrics.usage_freshness.dirty_pending_buckets` | Shows whether bucket/object mutations are still waiting for usage refresh. |
+| `metrics.usage_freshness.last_cycle_dirty_buckets` | Shows how many dirty buckets were picked up by the last cycle. |
+| `metrics.usage_freshness.last_cycle_cleared_dirty_buckets` | Shows how many dirty bucket marks were cleared by a successful cycle. |
+| `metrics.usage_freshness.last_usage_save_result` | Confirms whether the last usage save succeeded, failed, or was skipped. |
+| `metrics.life_time_ops.scan_cycle` | Confirms scanner cycles actually started after the workload. |
+| `metrics.life_time_ops.scan_bucket_drive` | Confirms bucket-drive scan work reached the storage layer. |
+| `metrics.life_time_ops.scan_object` | Confirms object metadata scanning advanced for object workloads. |
+| `metrics.life_time_ops.save_usage` | Confirms `DataUsageInfo` save work happened; this is the key freshness signal for bucket metrics. |
 | `metrics.scan_checkpoint` | Confirms partial cycles preserve resume context. |
 | `metrics.oldest_active_path_age_seconds` | Helps identify scanner paths that may be stuck. |
 
@@ -353,6 +382,23 @@ the background-heal status snapshots captured across `--metrics-endpoints`.
 | `heal_admin_queued` | Manual/admin heal work waiting in the queue. |
 | `heal_auto_heal_queued` | Auto-heal work waiting in the queue, typically from disk/set recovery paths. |
 
+`scanner-summary.csv` also includes usage freshness columns for quick
+post-start bucket metrics validation:
+
+| Field | Why it matters |
+|---|---|
+| `current_cycle_usage_saves` | Usage saves during the current cycle. |
+| `last_cycle_usage_saves` | Usage saves from the last finished or partial cycle. |
+| `usage_dirty_pending_buckets` | Dirty buckets still waiting for scanner refresh. |
+| `usage_last_cycle_dirty_buckets` | Dirty buckets selected by the last cycle. |
+| `usage_last_cycle_cleared_dirty_buckets` | Dirty bucket marks cleared by the last cycle. |
+| `usage_last_save_result` | Last `DataUsageInfo` save result. |
+| `usage_last_save_unix_secs` | Last `DataUsageInfo` save timestamp. |
+| `life_time_scan_cycle` | Total scanner cycles observed by the node. |
+| `life_time_scan_bucket_drive` | Total bucket-drive scans completed by the node. |
+| `life_time_scan_object` | Total object scan operations observed by the node. |
+| `life_time_save_usage` | Total usage save operations observed by the node. |
+
 ## Interpreting Results
 
 A useful tuning result has all of these properties:
@@ -377,6 +423,8 @@ Treat these as failure signals:
 - lifecycle transition `scanner_missed`, `queue_full`,
   `compensation_pending`, or `failed` grows during a run that was expected to
   reduce backlog;
+- bucket metrics show zero usage after post-start uploads while dirty usage
+  remains pending and `life_time_save_usage` does not advance;
 - heal or bitrot work moves from `queued` to `missed` after a scanner pacing
   change.
 
@@ -392,6 +440,8 @@ For scanner behavior PRs, include this evidence when available:
 - Observation window and sample interval.
 - Scanner status snapshots or time series.
 - Host CPU and disk telemetry.
+- Usage freshness fields from `scanner-summary.csv` when validating bucket
+  metrics or issue 3496-style timing.
 - Short conclusion that separates pressure reduction from scanner progress.
 - `scanner-validation-report.md` from the harness when using the scripted
   collection path.
@@ -405,6 +455,7 @@ at least these runs:
 | Run | Required evidence |
 |---|---|
 | Single-node, single-disk small-object idle | Scanner status series, host telemetry, `scanner-summary.csv`, and a conclusion that CPU or disk pressure is lower without scan progress stopping. |
+| Single-node post-start bucket metrics freshness | Empty data path startup, post-start bucket creation/upload, bucket metrics snapshots, `scanner-summary.csv` usage freshness columns, and evidence that `DataUsageInfo` save work occurred before accepting bucket usage metrics. |
 | Single-node erasure or multi-disk | Checkpoint movement, active path age, set/disk scan pressure, data usage freshness, and before/after scanner config. |
 | Distributed lifecycle backlog | `maintenance_control`, lifecycle expiry/transition queue fields, source work missed/failed counts, and by-host admin metrics. |
 | Distributed replication backlog | Bucket replication repair kind counters, site replication passive/active boundary counters, source work queued/skipped/missed counts, and by-host admin metrics. |

scripts/run_scanner_validation_harness.sh

@@ -161,7 +161,7 @@ setup_output() {
   mkdir -p "$OUT_DIR/heal"
   mkdir -p "$OUT_DIR/metrics"
   SUMMARY_CSV="$OUT_DIR/scanner-summary.csv"
-  echo "timestamp,primary_pressure,current_cycle_objects_scanned,current_cycle_directories_scanned,last_cycle_result,last_cycle_partial_reason,last_cycle_partial_source,lifecycle_transition_scanner_missed,source_work_missed_total,heal_queue_length,heal_active_tasks,heal_scanner_queued,heal_admin_queued,heal_auto_heal_queued" >"$SUMMARY_CSV"
+  echo "timestamp,primary_pressure,current_cycle_objects_scanned,current_cycle_directories_scanned,last_cycle_result,last_cycle_partial_reason,last_cycle_partial_source,lifecycle_transition_scanner_missed,source_work_missed_total,current_cycle_usage_saves,last_cycle_usage_saves,usage_dirty_pending_buckets,usage_last_cycle_dirty_buckets,usage_last_cycle_cleared_dirty_buckets,usage_last_save_result,usage_last_save_unix_secs,life_time_scan_cycle,life_time_scan_bucket_drive,life_time_scan_object,life_time_save_usage,heal_queue_length,heal_active_tasks,heal_scanner_queued,heal_admin_queued,heal_auto_heal_queued" >"$SUMMARY_CSV"
 }
 
 git_value() {
@@ -403,6 +403,17 @@ capture_status_sample() {
       (.metrics.last_cycle_partial_source // ""),
       (.metrics.lifecycle_transition.scanner_missed // 0),
       ((.metrics.source_work // []) | map(.missed // 0) | add // 0),
+      (.metrics.current_cycle_usage_saves // 0),
+      (.metrics.last_cycle_usage_saves // 0),
+      (.metrics.usage_freshness.dirty_pending_buckets // 0),
+      (.metrics.usage_freshness.last_cycle_dirty_buckets // 0),
+      (.metrics.usage_freshness.last_cycle_cleared_dirty_buckets // 0),
+      (.metrics.usage_freshness.last_usage_save_result // ""),
+      (.metrics.usage_freshness.last_usage_save_unix_secs // 0),
+      (.metrics.life_time_ops.scan_cycle // 0),
+      (.metrics.life_time_ops.scan_bucket_drive // 0),
+      (.metrics.life_time_ops.scan_object // 0),
+      (.metrics.life_time_ops.save_usage // 0),
       ($heal.healOperations.queueLength // $heal.healQueueLength // 0),
       ($heal.healOperations.activeTasks // $heal.healActiveTasks // 0),
       ($heal.healOperations.queuedBySource.scanner // 0),
@@ -455,11 +466,14 @@ Interval seconds: $INTERVAL_SECS
 - Background heal status snapshots: $heal_status_count
 - Distributed admin metrics snapshots: $metrics_count
 - Summary CSV: scanner-summary.csv
+- Usage freshness summary: current/last usage saves, dirty bucket state, last usage save result, and scanner life_time_ops.
 - Run metadata: run-metadata.env
 
 ## Review Checklist
 
 - Compare scanner progress and pressure in scanner-summary.csv.
+- Confirm dirty usage marks lead to usage saves when validating post-start bucket metrics freshness.
+- Check life_time_ops scan_cycle, scan_bucket_drive, scan_object, and save_usage before accepting bucket metrics freshness results.
 - Check source work missed totals before accepting pressure reductions.
 - Check healOperations queued and active counts when heal or bitrot pressure is involved.
 - Use distributed admin metrics snapshots for by-host investigation when metrics endpoints were provided.

scripts/test_scanner_validation_harness.sh

@@ -85,6 +85,12 @@ cat <<'JSON'
     "delay": { "value": 30, "source": "config" }
   },
   "metrics": {
+    "life_time_ops": {
+      "scan_cycle": 9,
+      "scan_bucket_drive": 11,
+      "scan_object": 13,
+      "save_usage": 5
+    },
     "pacing_pressure": {
       "primary_pressure": "active_scans"
     },
@@ -96,6 +102,18 @@ cat <<'JSON'
     "lifecycle_transition": {
       "scanner_missed": 0
     },
+    "current_cycle_usage_saves": 2,
+    "last_cycle_usage_saves": 3,
+    "usage_freshness": {
+      "dirty_pending_buckets": 1,
+      "last_dirty_mark_unix_secs": 1800000001,
+      "last_dirty_clear_unix_secs": 1800000002,
+      "last_cycle_dirty_buckets": 1,
+      "last_cycle_cleared_dirty_buckets": 1,
+      "last_usage_save_unix_secs": 1800000003,
+      "last_usage_save_result": "success",
+      "last_usage_save_result_code": 1
+    },
     "source_work": [
       { "source": "usage", "missed": 0 },
       { "source": "lifecycle", "missed": 0 }
@@ -150,7 +168,7 @@ fi
 if [[ "$1" == "-r" ]]; then
   while [[ $# -gt 0 ]]; do
     if [[ "$1" == "--arg" && "${2:-}" == "ts" ]]; then
-      printf '"%s","active_scans",5,2,"success","","",0,0,8,2,4,2,2\n' "$3"
+      printf '"%s","active_scans",5,2,"success","","",0,0,2,3,1,1,1,"success",1800000003,9,11,13,5,8,2,4,2,2\n' "$3"
       exit 0
     fi
     shift
@@ -205,8 +223,10 @@ if [[ "$(wc -l <"$OUT_DIR/scanner-summary.csv" | tr -d ' ')" != "3" ]]; then
   echo "Expected scanner summary header plus 2 rows" >&2
   exit 1
 fi
+grep -q 'current_cycle_usage_saves,last_cycle_usage_saves,usage_dirty_pending_buckets' "$OUT_DIR/scanner-summary.csv"
+grep -q 'life_time_scan_cycle,life_time_scan_bucket_drive,life_time_scan_object,life_time_save_usage' "$OUT_DIR/scanner-summary.csv"
 grep -q 'heal_queue_length,heal_active_tasks,heal_scanner_queued,heal_admin_queued,heal_auto_heal_queued' "$OUT_DIR/scanner-summary.csv"
-grep -q '"active_scans",5,2,"success","","",0,0,8,2,4,2,2' "$OUT_DIR/scanner-summary.csv"
+grep -q '"active_scans",5,2,"success","","",0,0,2,3,1,1,1,"success",1800000003,9,11,13,5,8,2,4,2,2' "$OUT_DIR/scanner-summary.csv"
 
 grep -q '^deployment=single-disk$' "$OUT_DIR/run-metadata.env"
 grep -q '^workload_label=small-object-idle$' "$OUT_DIR/run-metadata.env"
@@ -214,6 +234,7 @@ grep -q '^metrics_endpoints=http://node-a:9000,http://node-b:9000$' "$OUT_DIR/ru
 grep -q '## Scanner Validation Report' "$OUT_DIR/scanner-validation-report.md"
 grep -q 'Distributed admin metrics snapshots: 4' "$OUT_DIR/scanner-validation-report.md"
 grep -q 'Background heal status snapshots: 4' "$OUT_DIR/scanner-validation-report.md"
+grep -q 'Usage freshness summary:' "$OUT_DIR/scanner-validation-report.md"
 grep -q 'admin config get rustfs-local scanner' "$mc_log"
 grep -q 'admin config get rustfs-local heal' "$mc_log"
 grep -q -- '--request GET' "$awscurl_log"