Commit Graph

100 Commits

Author SHA1 Message Date
安正超 0e7e2660bb refactor: clean remaining storage DTO imports (#3568) 2026-06-18 14:44:46 +08:00
安正超 acdf439371 refactor: clean external DTO consumers (#3566)
* refactor: clean external DTO consumers

* fix: handle empty erasure shard recovery
2026-06-18 14:22:44 +08:00
安正超 99941f7e7c refactor: clean external operation consumers (#3565) 2026-06-18 12:39:41 +08:00
Henry Guo 46e200c22b feat(scanner): prioritize dirty usage refresh work (#3538) 2026-06-18 11:03:23 +08:00
安正超 57403525ee refactor: move heal and namespace contracts (#3560) 2026-06-18 11:00:21 +08:00
安正超 e5cad7ed20 refactor: move object operation contracts (#3559) 2026-06-18 09:48:13 +08:00
安正超 3fb4cb3d65 refactor: move list operations contract (#3550) 2026-06-18 08:48:18 +08:00
安正超 53337a6f71 refactor: move HTTP range helper contracts (#3533) 2026-06-18 00:47:23 +08:00
安正超 919deeb816 refactor: move object option helper contracts (#3521) 2026-06-17 22:56:10 +08:00
Henry Guo a9aba323c6 fix(scanner): restore prompt bucket usage scans (#3516)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-17 21:49:56 +08:00
安正超 ed55857bca refactor: move bucket operations contract (#3507) 2026-06-17 09:10:38 +08:00
Henry Guo b387689f26 feat(heal): expose scanner-aware operations status (#3483)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-15 22:28:53 +08:00
Henry Guo d28aedfbb3 feat(scanner): expose replication repair kind metrics (#3476) 2026-06-15 16:52:39 +08:00
abdullahnah92 c2e0792f6f fix(scanner): skip startup delay when replication is active so failed… (#3455)
fix(scanner): skip startup delay when replication is active so failed objects heal promptly after restart

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-15 09:02:21 +08:00
安正超 84f7027e90 refactor: add object store resolver for standalone crates (#3437) 2026-06-14 18:02:39 +08:00
houseme e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
houseme 7da10db852 refactor(logging): standardize heal and scanner events (#3414)
* refactor(logging): standardize heal and scanner events

* chore(git): untrack local logging governance note

* chore(git): ignore local logging governance note
2026-06-14 01:47:39 +08:00
Henry Guo 0d16a86d9a docs(scanner): add benchmark runbook (#3412)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-13 22:39:13 +08:00
escapecode 7a8514bdfa feat(sftp): add macOS and Windows platform support (#3372)
The session watchdog now selects its detection method per
platform. It previously probed kernel TCP state through a
Linux-only procfs path, so on macOS every healthy idle session
was killed within a minute, and on Windows the watchdog never
spawned at all, leaving wedged sessions with no cleanup. Linux
keeps its fast-kill watchdog unchanged. Other platforms get a
silence-only backstop that kills a session only at the
documented 30-minute idle ceiling.

The host-key loader now has a Windows arm. It loads OpenSSH
format host keys from the configured directory and logs a
one-time warning to restrict NTFS ACLs on the key directory,
the same operator-managed approach FTPS, WebDAV, KMS, and IAM
already use on Windows. Startup previously aborted with
UnsupportedPlatform because the Unix mode-bit permission check
has no Windows equivalent. tokio's io-uring feature is now
enabled only in Linux builds. io-uring is a Linux kernel
interface and enabling it unconditionally broke the Windows
build.

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-12 18:08:23 +08:00
Henry Guo 19ff378e9b fix(scanner): preserve newer usage snapshots (#3370)
* fix(scanner): preserve newer usage snapshots

* fix(table-catalog): require namespace locking backend

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-12 07:06:08 +08:00
安正超 a85cc0354c refactor(storage-api): remove namespace lock from StorageAPI (#3365)
* refactor(storage-api): remove namespace lock from StorageAPI

* test(scanner): wait for runtime budget cancellation

---------

Co-authored-by: loverustfs <hello@rustfs.com>
2026-06-11 22:42:12 +08:00
安正超 7146c893cb refactor(storage-api): remove legacy bucket DTO re-export (#3364) 2026-06-11 21:24:56 +08:00
安正超 b4524033e3 refactor(config): move global config accessors (#3360) 2026-06-11 19:16:02 +08:00
安正超 69549634ea refactor(config): migrate scanner config consumer (#3356) 2026-06-11 17:30:26 +08:00
安正超 d4ee44f49f refactor(scanner): narrow cache storage bounds (#3348) 2026-06-11 13:52:35 +08:00
Henry Guo 7191a3abae docs(scanner): document runtime scanner controls (#3339)
* docs(scanner): document runtime scanner controls

* docs(scanner): split English and Chinese README

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
2026-06-11 12:04:34 +08:00
安正超 87968275a7 refactor(storage): route maintenance inventory reads (#3337) 2026-06-11 10:32:48 +08:00
Henry Guo 91f80a5585 feat(scanner): expose lifecycle transition status (#3326)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-10 16:45:02 +00:00
houseme b1339a98ed fix(scanner): delay deep verify for fresh objects (#3329)
Gate scanner-triggered deep heal behind a short new-object cooldown and retry transient bitrot shard size mismatches during local verification.

Add targeted tests for cooldown mode selection and size-mismatch classification, and log when deep scans are downgraded or size-mismatch retries occur.
2026-06-10 15:50:44 +00:00
Henry Guo 62e9c5b94f feat(scanner): add adaptive pacing controls (#3315)
* feat(scanner): add adaptive pacing controls

* fix(scanner): bound pacing delay status precision

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-10 03:46:02 +00:00
Henry Guo e79c793803 feat(heal): add scanner-aware bitrot controls (#3297)
* feat(heal): add scanner-aware bitrot controls

* fix(config): register heal defaults

* test(admin): avoid heal config queue conflict

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-09 13:43:18 +00:00
Henry Guo ad9bf41fc8 feat(heal): expose scanner heal admission outcomes (#3292)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-09 03:39:57 +00:00
Henry Guo 6413df4b7f feat(replication): expose scanner repair outcomes (#3278)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-08 14:24:27 +00:00
cxymds 452c003341 fix(scanner): ignore missing rebalance metadata (#3282)
* fix(ecstore): harden rebalance data movement

* fix(ecstore): preserve failed rebalance status

* fix(ecstore): avoid rebalance walkdir total timeout

* fix(ecstore): retry rebalance listing timeouts

* fix(ecstore): restrict data movement resume target

* refactor(ecstore): simplify multipart movement target

* fix(ecstore): restore resume target checks

* perf(ecstore): speed up rebalance bucket merges

* fix: keep rebalance listing alive on transient failures

* fix(scanner): ignore missing rebalance metadata

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 14:24:16 +00:00
cxymds 9504dff595 fix(ecstore): harden rebalance data movement (#3234)
* fix(ecstore): harden rebalance data movement

* fix(ecstore): preserve failed rebalance status

* fix(ecstore): avoid rebalance walkdir total timeout

* fix(ecstore): retry rebalance listing timeouts

* fix(ecstore): restrict data movement resume target

* refactor(ecstore): simplify multipart movement target

* fix(ecstore): restore resume target checks

* perf(ecstore): speed up rebalance bucket merges

* fix: keep rebalance listing alive on transient failures

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 10:40:26 +00:00
Henry Guo f40dd2f93c fix(scanner): publish partial usage for compacted scans (#3277)
* fix(scanner): publish partial usage for compacted scans

* fix(scanner): publish first partial usage immediately

* fix(scanner): skip startup delay for cold usage cache

* fix(scanner): tighten cold usage publish gate

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 06:18:35 +00:00
Henry Guo f49df41db9 fix(lifecycle): harden scanner ILM expiry accounting (#3257)
* fix(lifecycle): harden scanner ILM expiry accounting

* fix(scanner): gate ILM action accounting on enqueue

* fix(metrics): avoid scanner source work argument list

* fix(scanner): gate local ILM accounting on enqueue

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 16:18:34 +00:00
Henry Guo 4ae070c8cc feat(scanner): expose scan partial source status (#3247)
* feat(scanner): expose scan partial source status

* fix(scanner): expose partial source in aggregated metrics

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 06:40:04 +00:00
Henry Guo b36a730e48 feat(scanner): expose checkpoint and source work status (#3230)
* feat(scanner): expose checkpoint and source work status

* fix(scanner): count ignored checkpoints once per scan

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-06 10:47:39 +00:00
Henry Guo 0d445afda9 feat(scanner): add versioned scan checkpoints (#3220)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-05 13:49:41 +00:00
Henry Guo 542720a1f7 feat(scanner): add partial scan resume hints (#3207)
* feat(scanner): add partial scan resume hints

* test(scanner): cover clearing scan resume hints

* fix(scanner): apply resume hint to combined child order

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-04 09:21:30 +00:00
Henry Guo 6850457707 feat(scanner): add runtime scanner controls and status (#3203)
* feat(scanner): add runtime scanner controls and status

* fix(scanner): validate persisted scanner config

* docs(scanner): clarify start delay behavior

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-04 05:16:26 +00:00
Henry Guo ad1a489f75 feat(scanner): add scanner budget progress controls (#3185)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-03 14:37:58 +00:00
Henry Guo cc07946782 feat(scanner): add cycle budget observability (#3166)
* feat(scanner): add cycle budget observability

* fix(scanner): clear cycle state after budget stop

* test(scanner): stabilize timeout-based scanner tests

* fix(scanner): keep cycle ILM counts scanner-only

* test(scanner): avoid test-only pending import

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-06-03 03:32:41 +00:00
Henry Guo 1d46047d6f feat(scanner): expand scanner observability metrics (#3159)
* feat(scanner): expand scanner observability metrics

* chore(scanner): align bucket-drive metric wording

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-02 01:29:53 +00:00
Henry Guo f3bd838925 feat(scanner): expose cycle progress metrics (#3152)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-01 07:54:39 +00:00
Henry Guo 76da2a48d0 feat(scanner): expose cycle observability controls (#3147)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-05-31 21:55:46 +00:00
安正超 a14657f517 fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of (#3142)
* fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of

Bug #1: add() had inverted logic — returned early when children were present,
making the recursive child traversal dead code. Leaf compaction path was
completely non-functional. Fixed by recursing into children when present and
collecting leaves only at leaf nodes.

Bug #2: reduce_children_of used  which underflows in debug
(panic) or release (wraps to usize::MAX, compacts entire cache). Fixed with
saturating_sub.

Both bugs existed identically in crates/scanner and crates/data-usage.

Added 5 tests covering add() subtree traversal, edge cases, and
reduce_children_of compaction behavior.

Closes rustfs/backlog#652

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: collect internal nodes as compaction candidates, not leaves

The previous fix for add() collected leaf nodes (children empty) as
compaction candidates. This is incorrect: total_children_rec returns 0
for leaves, so compacting them never decrements  — the
compaction loop makes no progress.

Collect internal nodes (children non-empty) instead. Compacting an
internal node removes its subtree via delete_recursive, which actually
reduces the total child count. Leaf nodes are skipped since they have
no children to remove.

Updated tests to match the corrected semantics.

* refactor: rename leaves→candidates, add data-usage regression tests

- Rename misleading  variable/parameter to  in
  add() and reduce_children_of() for both crates
- Add 4 regression tests in crates/data-usage covering add() candidate
  selection and reduce_children_of() compaction + saturating_sub

* fix: address data usage compaction review feedback

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-31 21:52:53 +00:00
Henry Guo a99ef64db2 feat(scanner): add scanner budgets and progress metrics (#3145)
* fix(scanner): preserve maintenance scan cadence

* feat(scanner): add scanner concurrency budget

* feat(scanner): expose scanner runtime progress

* fix(scanner): address scanner review feedback

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-31 16:42:38 +00:00
Henry Guo 92104cb354 fix(scanner): reduce single-disk scanner churn (#3135)
* fix(scanner): slow single-disk default scans

* fix(scanner): reduce single-disk scanner churn

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-31 05:05:04 +00:00