gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-27 16:48:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,170 Commits 15 Branches 117 Tags
a7be7c558d99b9bcf2f73b230577fc4d260abc9d
Commit Graph

59 Commits

Author SHA1 Message Date
Henry Guo 6850457707 feat(scanner): add runtime scanner controls and status (#3203) 
* feat(scanner): add runtime scanner controls and status

* fix(scanner): validate persisted scanner config

* docs(scanner): clarify start delay behavior

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-04 05:16:26 +00:00
Henry Guo ad1a489f75 feat(scanner): add scanner budget progress controls (#3185) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-03 14:37:58 +00:00
Henry Guo cc07946782 feat(scanner): add cycle budget observability (#3166) 
* feat(scanner): add cycle budget observability

* fix(scanner): clear cycle state after budget stop

* test(scanner): stabilize timeout-based scanner tests

* fix(scanner): keep cycle ILM counts scanner-only

* test(scanner): avoid test-only pending import

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-06-03 03:32:41 +00:00
Henry Guo 1d46047d6f feat(scanner): expand scanner observability metrics (#3159) 
* feat(scanner): expand scanner observability metrics

* chore(scanner): align bucket-drive metric wording

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-02 01:29:53 +00:00
Henry Guo f3bd838925 feat(scanner): expose cycle progress metrics (#3152) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-01 07:54:39 +00:00
Henry Guo 76da2a48d0 feat(scanner): expose cycle observability controls (#3147) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-05-31 21:55:46 +00:00
安正超 a14657f517 fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of (#3142) 
* fix(scanner,data-usage): fix add() logic inversion and usize underflow in reduce_children_of

Bug #1: add() had inverted logic — returned early when children were present,
making the recursive child traversal dead code. Leaf compaction path was
completely non-functional. Fixed by recursing into children when present and
collecting leaves only at leaf nodes.

Bug #2: reduce_children_of used  which underflows in debug
(panic) or release (wraps to usize::MAX, compacts entire cache). Fixed with
saturating_sub.

Both bugs existed identically in crates/scanner and crates/data-usage.

Added 5 tests covering add() subtree traversal, edge cases, and
reduce_children_of compaction behavior.

Closes rustfs/backlog#652

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: collect internal nodes as compaction candidates, not leaves

The previous fix for add() collected leaf nodes (children empty) as
compaction candidates. This is incorrect: total_children_rec returns 0
for leaves, so compacting them never decrements  — the
compaction loop makes no progress.

Collect internal nodes (children non-empty) instead. Compacting an
internal node removes its subtree via delete_recursive, which actually
reduces the total child count. Leaf nodes are skipped since they have
no children to remove.

Updated tests to match the corrected semantics.

* refactor: rename leaves→candidates, add data-usage regression tests

- Rename misleading  variable/parameter to  in
  add() and reduce_children_of() for both crates
- Add 4 regression tests in crates/data-usage covering add() candidate
  selection and reduce_children_of() compaction + saturating_sub

* fix: address data usage compaction review feedback

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-31 21:52:53 +00:00
Henry Guo a99ef64db2 feat(scanner): add scanner budgets and progress metrics (#3145) 
* fix(scanner): preserve maintenance scan cadence

* feat(scanner): add scanner concurrency budget

* feat(scanner): expose scanner runtime progress

* fix(scanner): address scanner review feedback

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-31 16:42:38 +00:00
Henry Guo 92104cb354 fix(scanner): reduce single-disk scanner churn (#3135) 
* fix(scanner): slow single-disk default scans

* fix(scanner): reduce single-disk scanner churn

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-31 05:05:04 +00:00
Henry Guo d921af6ef7 fix(heal): normalize completed root heal state (#3140) 
Clear persisted background heal active state after a deep scanner cycle completes so root heal status does not remain in progress between cycles.

Treat path-based stop for an already missing heal task as stopped while keeping client-token cancellation strict.

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-05-31 04:47:36 +00:00
Henry Guo 8e809f005d fix(scanner): support PBS subfolder alert threshold (#3129) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-05-30 11:03:13 +00:00
Henry Guo 62d1f80dbf fix(data-usage): refresh admin usage after object changes (#3081) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-05-25 14:24:00 +00:00
weisd 69345fe059 fix(scanner): preserve background heal compatibility (#3041) 2026-05-22 14:38:35 +08:00
houseme dcbffb084f chore(deps): refresh workspace deps and linux fs_type gating (#3030) 
* chore(deps): refresh workspace deps and linux fs_type gating

- refresh workspace dependency pins and lockfile updates

- remove now-unused crate dependency entries in multiple Cargo.toml files

- enable profiling export defaults in config and scripts/run.sh

- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings

* fix(utils): simplify fs_type linux gating

- keep fs_type module-level linux cfg in os::mod

- remove redundant linux cfg on get_fs_type and test module

* chore(deps): bump s3s git revision

- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5

- refresh Cargo.lock source entry for s3s

* chore(dev): allow mysql_async git source and env overrides

- allow mysql_async git source in deny.toml allow-git list

- make scripts/run.sh core env vars overrideable via existing shell env

* fix(utils): import get_fs_type in fs_type tests

- add explicit super::get_fs_type import in fs_type test module

- fix Linux E0425 unresolved function errors in unit tests

* chore(dev): tune run script observability defaults

- make profiling export env overrideable in scripts/run.sh

- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0

- update allow-git review window comments in deny.toml

* test(obs): stabilize profiling env alias tests
 2026-05-20 15:03:07 +00:00
houseme 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012) 
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
 2026-05-19 12:50:25 +00:00
安正超 e65aebba53 fix(ecstore): guard transition worker max (#3003) 
* fix(ecstore): guard transition worker max

* test: clarify env mutation safety

* test(ecstore): document env mutation concurrency contract
 2026-05-19 02:25:51 +00:00
houseme bd1e57293f fix: harden lifecycle transition compensation and regression coverage (#2995) 
* fix(ecstore): honor transition worker configuration

* fix(ecstore): add transition queue backpressure metrics

* fix(ecstore): schedule transition compensation on enqueue pressure

* fix(ecstore): log transition compensation scheduling

* test(rustfs): add transition compensation fault-injection coverage

* test(rustfs): cover delete after transition compensation

* test(scanner): cover cleanup after transition compensation

* test(rustfs): extend compensation transition coverage

* test(scanner): cover backfill idempotency after compensation

* test(scanner): cover noncurrent expiry after compensation

* test(rustfs): cover versioned delete after compensation

* test(rustfs): cover delete marker lifecycle after compensation

* test(scanner): extend versioned lifecycle compensation coverage

* test(scanner): model versioned delete after compensation

* test(scanner): clarify modeled versioned delete helper

* refactor(ecstore): optimize transition enqueue hot path

* refactor(ecstore): centralize transition runtime constants

* style(ecstore): apply rustfmt for transition timeout helper

* fix(ilm): align queue-full metric semantics

* refactor(ecstore): unify immediate enqueue failure handling

* refactor(ecstore): reuse transition worker env constant

* ci(actions): update setup action inputs
 2026-05-18 12:50:43 +00:00
唐小鸭 e16f1ae639 fix(window): Compatible with Windows Path (#2691) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-28 14:36:59 +00:00
cxymds c698a0f2b6 fix(policy): allow AssumeRole in system policies (#2718) 2026-04-28 11:01:46 +00:00
houseme 50d03ef021 perf(memory): add reclaim signals and cache controls (#2689) 2026-04-26 16:42:35 +00:00
weisd d4dcb2ac9d fix(scanner): avoid stalls after abandoned child listings (#2632) 2026-04-21 09:06:26 +00:00
安正超 457f4e0170 fix(s3): improve GitLab registry compatibility (#2596) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-19 02:10:42 +00:00
houseme 9677320f23 fix(scanner): stabilize data usage cache persistence under slow metadata I/O (#2594) 2026-04-18 16:36:28 +00:00
GatewayJ dc5ce7d0af test(scanner): avoid flaky noncurrent version counting (#2589) 
Co-authored-by: GatewayJ <8352692332qq.com>
 2026-04-18 11:13:35 +00:00
houseme f77ccd5b23 chore: move mcp crate to standalone repository (#2578) 2026-04-17 12:42:26 +00:00
houseme 478720d2ee Centralize lifecycle state updates and fix systemd running status (#2567) 2026-04-17 03:20:11 +00:00
houseme 6ce24f3b63 feat(lifecycle): improve ILM compatibility and scanner runtime config (#2534) 
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-04-16 10:28:03 +00:00
houseme 28edfd6190 fix(storage): harden offline drive fail-fast paths (#2564) 
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2026-04-16 09:21:45 +00:00
Tunglies 579b124726 lint: clippy rules or_fun_call (#2561) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-16 02:48:39 +00:00
Tunglies 49366ee200 chore(lint): clippy rules redundant_clone (#2554) 2026-04-15 13:54:07 +00:00
weisd 27971f5c03 fix(scanner): preserve totals for compacted folder scans (#2530) 2026-04-15 07:32:13 +00:00
安正超 2a3d127a86 revert: restore PutObjReader and remove PUT zero-copy chunk path (#2351 phase 2) (#2532) 2026-04-14 16:35:03 +08:00
weisd 9ed6487e2f fix(scanner): continue scans after traversal errors (#2414) 2026-04-07 14:19:26 +08:00
houseme 32bf8f5bf3 feat(storage): add direct chunk GET fast path (#2351) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-04-07 08:33:46 +08:00
weisd 41dcebda44 fix(tier): sweep transitioned copies from delete handlers (#2287) 2026-03-25 16:06:36 +08:00
cxymds 5ea6d8a7e6 fix(ecstore): preserve transition object metadata (#2263) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-03-24 14:06:29 +08:00
weisd 05dc131a49 perf(storage): optimize internode RPC transfer path (#2262) 
Co-authored-by: momoda693 <momoda693@gmail.com>
 2026-03-23 17:09:49 +08:00
cxymds 236142a682 fix(ecstore): repair lifecycle transition and restore flows (#2240) 
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: weisd <im@weisd.in>
 2026-03-23 12:29:13 +08:00
heihutu 3c28f0a0ba feat(metrics): migrate system monitoring from rustfs-obs to rustfs-metrics (#2242) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-20 18:52:33 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
安正超 83fb530609 refactor(config): normalize scanner env naming (#2129) 2026-03-11 22:41:41 +08:00
安正超 fdbe12ec95 fix(scanner): respect configured scan start delay (#2119) 2026-03-11 21:56:48 +08:00
安正超 ac43a44a00 [codex] fix scanner first cycle startup delay (#2137) 2026-03-11 20:02:01 +08:00
evan slack b3da8ae269 feat(scanner): Add dynamic throttling presets (#2095) 
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: weisd <im@weisd.in>
 2026-03-10 16:12:56 +08:00
evan slack d52a10c5fb chore(obs): Improve tracing instrumentation (#2086) 
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-03-07 20:03:20 +08:00
weisd ed18b3da75 Fix data usage cache and scanner (#2074) 2026-03-04 19:55:01 +08:00
安正超 af6c32efac refactor: improve code quality with safer error handling, trait decomposition, and dead code cleanup (#1997) 2026-02-28 01:19:47 +08:00
LoganZ2 e73b17aff6 fix(scanner): skip recent IO-error objects (#1860) 
Signed-off-by: LoganZ2 <103290230+LoganZ2@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-02-27 22:25:52 +08:00
houseme 024a3107d8 build(deps): bump the dependencies group with 4 updates (#1955) 2026-02-25 19:21:20 +08:00
evan slack 9786d9b004 metrics(scanner): Add metrics to scanner (#1823) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
 2026-02-15 18:36:40 +08:00