3f20fbd77b
fix(scanner): report active first-cycle status ( #5397 )
...
* fix(scanner): report active first-cycle status
* fix(scanner): publish cycle activity consistently
* test(common): satisfy Rust 1.97 waker lint
* fix(scanner): satisfy Rust 1.97 clippy
---------
Co-authored-by: houseme <housemecn@gmail.com >
2026-07-29 22:33:25 +08:00
1397a4e7ca
feat(ilm): expose lifecycle expiry scanner counters ( #5262 )
...
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-26 12:11:03 +08:00
a63b79004c
fix(scanner): make distributed usage convergence authoritative ( #5151 )
...
* fix(scanner): make distributed usage cycles authoritative
* fix(scanner): close distributed refresh races
* fix(config): align scanner reload integration
* fix(admin): scope config test helpers
* fix(scanner): harden distributed usage convergence
* fix(scanner): preserve rolling activity compatibility
* fix(admin): expose non-secret optional config values
* fix(scanner): acknowledge distributed dirty usage
* fix(ecstore): make bucket mutations cancellation safe
* fix(scanner): preserve pending dirty acknowledgements
* test(obs): account for superseded scanner metric
* fix(api): reject excess detached bucket mutations
* test: close scanner convergence coverage gaps
* fix(scanner): make path tracking cleanup one-shot
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-07-25 18:45:16 +08:00
889a45ad4d
fix(scanner): back off clean idle scans across erasure clusters ( #4984 )
...
* fix(scanner): back off clean single-disk cycles
* fix(scanner): extend idle backoff across erasure clusters
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-07-18 10:49:45 +08:00
6ab8636734
fix(obs): count scanned versions independent of ILM ( #4516 )
...
versions_scanned for both the scanner and ILM collectors was read from
the Lifecycle work source's `checked` counter, which is never recorded on
the production scan path — so rustfs_scanner_versions_scanned_total and
rustfs_ilm_versions_scanned_total sat at zero even while objects_scanned
climbed. The two metrics also have distinct intended meanings that were
conflated: "versions scanned" (all versions, any bucket) vs "versions
checked for ILM actions" (lifecycle-configured buckets only).
Add a lifetime `versions_scanned` counter recorded for every version the
scanner walks (independent of ILM), and record the Lifecycle source's
`checked` counter from the ILM evaluator so the ILM metric reflects the
real checked subset. The scanner collector now reports total scanned
versions; the ILM collector keeps the ILM-checked subset.
Closes backlog#995 (OBS-09).
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-08 16:50:51 +00:00
76124423a4
fix: stabilize s3-tests delete key-limit coverage ( #4283 )
...
* fix: tighten list handling and s3 test support
* chore: tidy imports and metric updates
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
Signed-off-by: Zhengchao An <anzhengchao@gmail.com >
* fix: address s3tests review follow-ups
---------
Signed-off-by: Zhengchao An <anzhengchao@gmail.com >
Co-authored-by: Zhengchao An <anzhengchao@gmail.com >
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
2026-07-05 20:35:32 +08:00
0ff261864a
fix(scanner): harden data scanner integrity handling ( #4012 )
...
* fix(scanner): heal metadata scan failures
* fix(scanner): preserve dirty buckets after scan failures
* feat(scanner): record leader lock liveness
* fix(scanner): preserve failed deep scan state
* fix(scanner): reject untimestamped stale usage
* fix(scanner): report topology-derived admission limit
* fix(scanner): accumulate tier usage stats
* fix(scanner): guard data usage cache recursion
* feat(scanner): expose startup enabled status
* fix(scanner): continue after heal admission rejection
* fix(scanner): avoid cyclic root usage double count
* fix(scanner): preserve dirty markers on cache save failure
* feat(scanner): expose leader liveness status
* fix(scanner): avoid heal escalation for transient metadata reads
* feat(scanner): persist rejected heal retry candidates
* test(scanner): satisfy freshness status clippy
* test(app): avoid global store reinit in context test
* test(app): gate global store helper to tests
2026-06-29 08:19:58 +08:00
7bdb25ae9d
feat(scanner): define replication boundary contract ( #3630 )
...
* feat(scanner): define replication boundary contract
* fix(scanner): propagate replication boundary metrics
* fix(scanner): preserve repair metadata on merge
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-23 21:35:36 +08:00
80144ec886
feat(scanner): expose usage freshness status ( #3577 )
...
* feat(scanner): expose usage freshness status
* fix(scanner): reset dirty usage cycle state
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-18 21:14:55 +08:00
d28aedfbb3
feat(scanner): expose replication repair kind metrics ( #3476 )
2026-06-15 16:52:39 +08:00
dd6b4c35ad
fix(scanner): harden lifecycle and tiering backlog ( #3469 )
...
* fix(scanner): expose lifecycle transition backlog
* fix(scanner): expose lifecycle expiry backlog
* fix(scanner): preserve lifecycle backlog states
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-15 15:01:21 +08:00
46fb4bdc2f
feat(scanner): add source-aware maintenance controls ( #3461 )
...
* feat(scanner): add source-aware maintenance controls
* fix(scanner): preserve source control between cycles
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-15 12:12:05 +08:00
91f80a5585
feat(scanner): expose lifecycle transition status ( #3326 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-10 16:45:02 +00:00
66fd55a8e0
feat(scanner): expose pacing pressure status ( #3319 )
...
* feat(scanner): expose pacing pressure status
* fix(scanner): preserve merged pause pressure
* fix(scanner): default missing primary pressure
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-10 07:33:41 +00:00
ad9bf41fc8
feat(heal): expose scanner heal admission outcomes ( #3292 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: cxymds <Cxymds@qq.com >
2026-06-09 03:39:57 +00:00
f49df41db9
fix(lifecycle): harden scanner ILM expiry accounting ( #3257 )
...
* fix(lifecycle): harden scanner ILM expiry accounting
* fix(scanner): gate ILM action accounting on enqueue
* fix(metrics): avoid scanner source work argument list
* fix(scanner): gate local ILM accounting on enqueue
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-07 16:18:34 +00:00
4ae070c8cc
feat(scanner): expose scan partial source status ( #3247 )
...
* feat(scanner): expose scan partial source status
* fix(scanner): expose partial source in aggregated metrics
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: cxymds <Cxymds@qq.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-07 06:40:04 +00:00
83a4e5712e
feat(scanner): track scan cycle source work ( #3240 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-06 12:53:48 +00:00
b36a730e48
feat(scanner): expose checkpoint and source work status ( #3230 )
...
* feat(scanner): expose checkpoint and source work status
* fix(scanner): count ignored checkpoints once per scan
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-06 10:47:39 +00:00
ad1a489f75
feat(scanner): add scanner budget progress controls ( #3185 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-03 14:37:58 +00:00
cc07946782
feat(scanner): add cycle budget observability ( #3166 )
...
* feat(scanner): add cycle budget observability
* fix(scanner): clear cycle state after budget stop
* test(scanner): stabilize timeout-based scanner tests
* fix(scanner): keep cycle ILM counts scanner-only
* test(scanner): avoid test-only pending import
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
2026-06-03 03:32:41 +00:00
1d46047d6f
feat(scanner): expand scanner observability metrics ( #3159 )
...
* feat(scanner): expand scanner observability metrics
* chore(scanner): align bucket-drive metric wording
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-02 01:29:53 +00:00
f3bd838925
feat(scanner): expose cycle progress metrics ( #3152 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-01 07:54:39 +00:00
76da2a48d0
feat(scanner): expose cycle observability controls ( #3147 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-05-31 21:55:46 +00:00
a99ef64db2
feat(scanner): add scanner budgets and progress metrics ( #3145 )
...
* fix(scanner): preserve maintenance scan cadence
* feat(scanner): add scanner concurrency budget
* feat(scanner): expose scanner runtime progress
* fix(scanner): address scanner review feedback
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-05-31 16:42:38 +00:00
73bde843d6
refactor(s3): consolidate semantic boundaries and remove s3-common ( #3012 )
...
* refactor(common): introduce rustfs-data-usage core crate
* refactor(concurrency): migrate workers crate into concurrency
* refactor(crypto): migrate appauth token APIs into crypto
* fix docs urls
* remove unused crate
* refactor(data-usage): switch consumers to rustfs-data-usage
* chore(fmt): apply cargo fmt and lockfile sync
* refactor(common): remove data_usage compatibility re-export
* refactor(capacity): move capacity_scope to object-capacity
* refactor(io-metrics): relocate internode metrics from common
* refactor(common): decouple scanner report from madmin
* chore(fmt): normalize import ordering after pre-commit
* refactor(s3): split s3 types and ops crates
* refactor(s3): centralize event version and safe parsing
* refactor(s3): add op-event compatibility guardrails
* refactor(s3): add runtime op-event mismatch observability
* refactor(s3): extract delete event mapping helper
* refactor(s3): extract put event mapping helper
* refactor(s3): consolidate remaining event semantic helpers
* refactor(s3): add op-event coverage checks and observability alerts
* refactor(s3-ops): consolidate op-event semantic mapping
* refactor(scanner): remove last_minute wrapper module
* refactor(scanner): consolidate duplicated data usage models
2026-05-19 12:50:25 +00:00
49366ee200
chore(lint): clippy rules redundant_clone ( #2554 )
2026-04-15 13:54:07 +00:00
ec8f059506
feat: add timing and metrics for query execution phases ( #2419 )
...
Signed-off-by: Shreyan Jain <shreyan11@duck.com >
Co-authored-by: Shreyan Jain <shreyan11@duck.com >
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: loverustfs <hello@rustfs.com >
Co-authored-by: cxymds <Cxymds@qq.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com >
2026-04-07 20:32:32 +08:00
9786d9b004
metrics(scanner): Add metrics to scanner ( #1823 )
...
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: loverustfs <hello@rustfs.com >
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com >
2026-02-15 18:36:40 +08:00
29d86036b1
feat: implement bucket quota system ( #1461 )
...
Signed-off-by: yxrxy <1532529704@qq.com >
Co-authored-by: loverustfs <hello@rustfs.com >
2026-01-12 11:42:07 +08:00
75b144b7d4
Fixing URL output format in IPv6 environments #1343 and Incorrect time in UI #1350 ( #1363 )
2026-01-04 14:56:54 +08:00
8d7cd4cb1b
chore: upgrade dependencies and migrate to aws-lc-rs ( #1333 )
2026-01-02 00:02:34 +08:00
d934e3905b
Refactor telemetry initialization for non-production environments ( #789 )
...
* add dep `scopeguard`
* improve for tracing
* fix
* fix
* improve code for import
* add logger trace id
* fix
* fix
* fix
* fix
* fix
2025-11-05 00:55:08 +08:00
e7d0a8d4b9
feat: integrate global metrics system into AHM scanner
...
- Add global metrics system to common crate for cross-module usage
- Integrate global metrics collection into AHM scanner operations
- Update ECStore to use common metrics system instead of local implementation
- Add chrono dependency to AHM crate for timestamp handling
- Re-export IlmAction from common metrics in ECStore lifecycle module
- Update scanner methods to use global metrics for cycle, disk, and volume scans
- Maintain backward compatibility with local metrics collector
- Fix clippy warnings and ensure proper code formatting
This change enables unified metrics collection across the entire RustFS system,
allowing better monitoring and observability of scanner operations.
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-07-24 12:12:49 +08:00
Henry Guo