gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-11 15:46:53 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,371 Commits 22 Branches 121 Tags
36428bc490ea4cc0619a728fc36fb9b75ae8f014
Commit Graph

53 Commits

Author SHA1 Message Date
houseme 6ab8636734 fix(obs): count scanned versions independent of ILM (#4516) 
versions_scanned for both the scanner and ILM collectors was read from
the Lifecycle work source's `checked` counter, which is never recorded on
the production scan path — so rustfs_scanner_versions_scanned_total and
rustfs_ilm_versions_scanned_total sat at zero even while objects_scanned
climbed. The two metrics also have distinct intended meanings that were
conflated: "versions scanned" (all versions, any bucket) vs "versions
checked for ILM actions" (lifecycle-configured buckets only).

Add a lifetime `versions_scanned` counter recorded for every version the
scanner walks (independent of ILM), and record the Lifecycle source's
`checked` counter from the ILM evaluator so the ILM metric reflects the
real checked subset. The scanner collector now reports total scanned
versions; the ILM collector keeps the ILM-checked subset.

Closes backlog#995 (OBS-09).

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-07-08 16:50:51 +00:00
Zhengchao An e0972f19ac fix(scanner): cache object lock config (#4422) 
* fix(scanner): cache object lock config

* test(scanner): update lifecycle scanner items
 2026-07-08 15:01:57 +08:00
cxymds 902cf1d3dc fix: handle scanner task failures (#4185) 2026-07-02 16:34:56 +08:00
Zhengchao An 14016dbe8c refactor(scanner): hide replication queue DTOs (#4150) 2026-07-02 00:17:03 +08:00
cxymds 0ff261864a fix(scanner): harden data scanner integrity handling (#4012) 
* fix(scanner): heal metadata scan failures

* fix(scanner): preserve dirty buckets after scan failures

* feat(scanner): record leader lock liveness

* fix(scanner): preserve failed deep scan state

* fix(scanner): reject untimestamped stale usage

* fix(scanner): report topology-derived admission limit

* fix(scanner): accumulate tier usage stats

* fix(scanner): guard data usage cache recursion

* feat(scanner): expose startup enabled status

* fix(scanner): continue after heal admission rejection

* fix(scanner): avoid cyclic root usage double count

* fix(scanner): preserve dirty markers on cache save failure

* feat(scanner): expose leader liveness status

* fix(scanner): avoid heal escalation for transient metadata reads

* feat(scanner): persist rejected heal retry candidates

* test(scanner): satisfy freshness status clippy

* test(app): avoid global store reinit in context test

* test(app): gate global store helper to tests
 2026-06-29 08:19:58 +08:00
GatewayJ 675597ec16 fix(ecstore): handle stalled recovery reads and listings (#3790) 
* fix(ecstore): handle stalled recovery reads and listings

* fix(rio): start HTTP stall timeout on read

* fix(ecstore): handle stalled reads and partial lists

* fix(ecstore): retire stalled shards and list errors

* fix(ecstore): preserve list merge lookahead entries

* fix(ecstore): bound zero-copy shard reads

* fix(ecstore): hedge stalled shard reads

* fix(ecstore): retire abandoned shard reads

* fix(ecstore): include part identity in metadata quorum

* fix(ecstore): validate heal shard sources

* fix(ecstore): verify reconstructed read shards

* chore(ecstore): log slow object read stages

* fix(heal): throttle auto heal during recovery

* fix(scanner): yield to foreground reads

* fix(scanner): track streaming object reads

* fix(ecstore): avoid false read heal fanout

* fix(ecstore): verify codec streaming reconstruction sources

* fix(ecstore): preserve quorum progress on slow shards

* fix(storage): restore read timeout facade

* fix(ecstore): retain fallback readers after quorum

* chore: allow decode helper argument lists

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2026-06-27 10:21:09 +08:00
Henry Guo a8327f8901 fix(scanner): account versioned delete markers in usage (#3904) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-26 18:34:21 +08:00
Zhengchao An 1f7e159388 refactor: segment external storage api boundaries (#3903) 2026-06-26 15:10:49 +08:00
Zhengchao An 4d6ea453a7 refactor: route scanner heal storage boundaries (#3888) 2026-06-26 05:24:41 +08:00
Henry Guo 30f781871d fix(scanner): preserve dirty usage convergence (#3844) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-25 21:09:12 +08:00
Zhengchao An a935854b32 refactor: centralize scanner runtime source helpers (#3827) 
* refactor: batch lifecycle runtime source handles

* refactor: centralize scanner runtime source helpers
 2026-06-24 22:05:03 +08:00
Zhengchao An d3796d6c10 refactor: remove external owner compat bridges (#3741) 2026-06-22 18:28:35 +08:00
安正超 cca9e83a8b refactor: use relative standalone compat consumers (#3734) 2026-06-22 15:07:58 +08:00
安正超 c13f42e9a0 refactor: wrap disk rpc compat trait methods (#3707) 2026-06-22 06:31:24 +08:00
安正超 7eb9a4759e refactor: wrap bucket compat trait methods (#3706) 2026-06-22 04:32:37 +08:00
安正超 6b3d96fde3 refactor: prune trait import compat re-exports (#3699) 2026-06-21 23:17:58 +08:00
GatewayJ c47dea0009 fix(scanner): preserve complete usage snapshots (#3654) 2026-06-21 08:21:20 +08:00
Henry Guo a689ff3c39 feat(scanner): wake cycles for dirty usage refresh (#3617) 2026-06-19 18:36:48 +08:00
Henry Guo 80144ec886 feat(scanner): expose usage freshness status (#3577) 
* feat(scanner): expose usage freshness status

* fix(scanner): reset dirty usage cycle state

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-18 21:14:55 +08:00
安正超 bdb2461b55 refactor: clean scanner heal runtime boundaries (#3571) 2026-06-18 16:12:27 +08:00
安正超 acdf439371 refactor: clean external DTO consumers (#3566) 
* refactor: clean external DTO consumers

* fix: handle empty erasure shard recovery
 2026-06-18 14:22:44 +08:00
安正超 99941f7e7c refactor: clean external operation consumers (#3565) 2026-06-18 12:39:41 +08:00
Henry Guo 46e200c22b feat(scanner): prioritize dirty usage refresh work (#3538) 2026-06-18 11:03:23 +08:00
安正超 ed55857bca refactor: move bucket operations contract (#3507) 2026-06-17 09:10:38 +08:00
安正超 84f7027e90 refactor: add object store resolver for standalone crates (#3437) 2026-06-14 18:02:39 +08:00
houseme e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
houseme 7da10db852 refactor(logging): standardize heal and scanner events (#3414) 
* refactor(logging): standardize heal and scanner events

* chore(git): untrack local logging governance note

* chore(git): ignore local logging governance note
 2026-06-14 01:47:39 +08:00
安正超 7146c893cb refactor(storage-api): remove legacy bucket DTO re-export (#3364) 2026-06-11 21:24:56 +08:00
安正超 d4ee44f49f refactor(scanner): narrow cache storage bounds (#3348) 2026-06-11 13:52:35 +08:00
安正超 87968275a7 refactor(storage): route maintenance inventory reads (#3337) 2026-06-11 10:32:48 +08:00
cxymds 452c003341 fix(scanner): ignore missing rebalance metadata (#3282) 
* fix(ecstore): harden rebalance data movement

* fix(ecstore): preserve failed rebalance status

* fix(ecstore): avoid rebalance walkdir total timeout

* fix(ecstore): retry rebalance listing timeouts

* fix(ecstore): restrict data movement resume target

* refactor(ecstore): simplify multipart movement target

* fix(ecstore): restore resume target checks

* perf(ecstore): speed up rebalance bucket merges

* fix: keep rebalance listing alive on transient failures

* fix(scanner): ignore missing rebalance metadata

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-08 14:24:16 +00:00
cxymds 9504dff595 fix(ecstore): harden rebalance data movement (#3234) 
* fix(ecstore): harden rebalance data movement

* fix(ecstore): preserve failed rebalance status

* fix(ecstore): avoid rebalance walkdir total timeout

* fix(ecstore): retry rebalance listing timeouts

* fix(ecstore): restrict data movement resume target

* refactor(ecstore): simplify multipart movement target

* fix(ecstore): restore resume target checks

* perf(ecstore): speed up rebalance bucket merges

* fix: keep rebalance listing alive on transient failures

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-08 10:40:26 +00:00
Henry Guo f40dd2f93c fix(scanner): publish partial usage for compacted scans (#3277) 
* fix(scanner): publish partial usage for compacted scans

* fix(scanner): publish first partial usage immediately

* fix(scanner): skip startup delay for cold usage cache

* fix(scanner): tighten cold usage publish gate

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-08 06:18:35 +00:00
Henry Guo 6850457707 feat(scanner): add runtime scanner controls and status (#3203) 
* feat(scanner): add runtime scanner controls and status

* fix(scanner): validate persisted scanner config

* docs(scanner): clarify start delay behavior

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-04 05:16:26 +00:00
Henry Guo ad1a489f75 feat(scanner): add scanner budget progress controls (#3185) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-03 14:37:58 +00:00
Henry Guo cc07946782 feat(scanner): add cycle budget observability (#3166) 
* feat(scanner): add cycle budget observability

* fix(scanner): clear cycle state after budget stop

* test(scanner): stabilize timeout-based scanner tests

* fix(scanner): keep cycle ILM counts scanner-only

* test(scanner): avoid test-only pending import

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-06-03 03:32:41 +00:00
Henry Guo 1d46047d6f feat(scanner): expand scanner observability metrics (#3159) 
* feat(scanner): expand scanner observability metrics

* chore(scanner): align bucket-drive metric wording

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-02 01:29:53 +00:00
Henry Guo a99ef64db2 feat(scanner): add scanner budgets and progress metrics (#3145) 
* fix(scanner): preserve maintenance scan cadence

* feat(scanner): add scanner concurrency budget

* feat(scanner): expose scanner runtime progress

* fix(scanner): address scanner review feedback

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-31 16:42:38 +00:00
唐小鸭 e16f1ae639 fix(window): Compatible with Windows Path (#2691) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-28 14:36:59 +00:00
houseme 9677320f23 fix(scanner): stabilize data usage cache persistence under slow metadata I/O (#2594) 2026-04-18 16:36:28 +00:00
houseme 28edfd6190 fix(storage): harden offline drive fail-fast paths (#2564) 
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2026-04-16 09:21:45 +00:00
Tunglies 579b124726 lint: clippy rules or_fun_call (#2561) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-16 02:48:39 +00:00
weisd 41dcebda44 fix(tier): sweep transitioned copies from delete handlers (#2287) 2026-03-25 16:06:36 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
evan slack b3da8ae269 feat(scanner): Add dynamic throttling presets (#2095) 
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: weisd <im@weisd.in>
 2026-03-10 16:12:56 +08:00
evan slack d52a10c5fb chore(obs): Improve tracing instrumentation (#2086) 
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-03-07 20:03:20 +08:00
weisd ed18b3da75 Fix data usage cache and scanner (#2074) 2026-03-04 19:55:01 +08:00
安正超 af6c32efac refactor: improve code quality with safer error handling, trait decomposition, and dead code cleanup (#1997) 2026-02-28 01:19:47 +08:00
LoganZ2 e73b17aff6 fix(scanner): skip recent IO-error objects (#1860) 
Signed-off-by: LoganZ2 <103290230+LoganZ2@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-02-27 22:25:52 +08:00
evan slack 9786d9b004 metrics(scanner): Add metrics to scanner (#1823) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
 2026-02-15 18:36:40 +08:00