gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-05 19:55:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,309 Commits 61 Branches 132 Tags
1638e9b51606054a2dd4c998827bec96397cf5d2
Commit Graph

286 Commits

Author SHA1 Message Date
Henry Guo 923bde6904 feat(scanner): prepare scoped bucket cache scans (#7136) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-09-04 15:09:58 +00:00
Henry Guo f16a30b231 feat(scanner): expose authenticated dirty bucket snapshots (#7122) 
* feat(scanner): add peer bucket dirty snapshots

* fix(scanner): keep dirty snapshot errors stable

* test(protos): satisfy dirty snapshot clippy

* fix(scanner): satisfy dirty snapshot clippy

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-09-04 18:23:13 +08:00
cxymds 80c88a9031 fix(ilm): delete historical null versions by exact identity (#7109) 2026-09-04 08:14:47 +08:00
Henry Guo 3a914b429d fix(scanner): preserve explicit cycle retry cadence (#7107) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-09-03 22:30:46 +08:00
houseme 62e66baf89 feat(scanner): add storage seam (#7105) 2026-09-03 20:31:59 +08:00
cxymds a6cb34c7a4 fix: fence transition transaction recovery (#7095) 2026-09-03 10:38:30 +00:00
houseme 0e6ee3bf62 feat(scanner): coordinate usage and workload boundaries (#7093) 
* test(scanner): wire usage and heal rebuild gates

* docs(scanner): define usage authority protocol

* docs(heal): clarify scanner and ecstore boundaries

* refactor(scanner): split metrics from contracts

* feat(scanner): use shared workload snapshots

* fix(ecstore): recheck capacity before decommission drain
 2026-09-03 17:02:43 +08:00
houseme 2e2bc814b1 fix(scanner): align usage floor compat marker (#7064) 
fix(scanner): accept bootstrap marker on timeout fence
 2026-09-02 14:03:04 +00:00
houseme 99f85ca2b1 fix(scanner): recover fenced incomplete usage floors (#7055) 
* fix(scanner): recover fenced incomplete usage floors

* fix(scanner): validate legacy usage floor shape

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-09-02 12:32:31 +00:00
cxymds afc66b7182 fix(ilm): enqueue committed tier free versions (#7041) 
* fix(ilm): enqueue committed tier free versions

* fix(ilm): stabilize causal cleanup CI coverage

* test(ilm): make expire GET race deterministic

* test(ilm): synchronize expiry with active GET
 2026-09-02 11:08:28 +00:00
cxymds 6e26769265 fix(ecstore): make transitioned cleanup crash-safe (#6978) 
* fix(ecstore): fence transitioned object cleanup

* fix(ecstore): address ILM recovery review findings

* fix(ecstore): complete crash-safe tier cleanup recovery

* test(ecstore): avoid typo false positive

* fix(ecstore): stabilize decommission error buckets

* fix(ecstore): stabilize transition delete validation

* fix(ecstore): resume authorized tier delete dispatch

* fix(ecstore): satisfy feature clippy
 2026-09-01 19:09:22 +08:00
houseme ab44ae7e83 fix(scanner): add supported usage state reset (#6972) 
Add an authenticated scanner usage-state reset endpoint that publishes a fenced bootstrap marker for full rebuilds instead of requiring operators to delete usage metadata by hand.

Guard the reset with the scanner leader lock, storage publication epoch, and per-slot revision preconditions, and make startup resumable across stale cleanup leftovers while still rejecting newer conflicting usage state.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
 2026-09-01 01:51:46 +00:00
houseme e44007012b fix(scanner): recover legacy usage floor from backup (#6964) 
* fix(scanner): recover legacy usage floor from backup

Allow scanner usage-floor startup and leadership fencing to use a valid legacy backup when the legacy primary read fails with a corruption-shaped error.

Keep v2 primary read failures, stale metadata, transient I/O, and missing or invalid backups fail-closed.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): cover legacy backup fencing gaps (#6966)

* fix(scanner): recover legacy usage from valid backup

* fix(scanner): recover legacy usage floor from backup

Allow scanner usage-floor startup and leadership fencing to use a valid legacy backup when the legacy primary read fails with a corruption-shaped error.

Keep v2 primary read failures, stale metadata, transient I/O, and missing or invalid backups fail-closed.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): cover legacy backup fencing gaps

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
 2026-09-01 00:01:38 +08:00
houseme 281e40f1cc fix(scanner): fit usage persistence within publication lease (#6967) 
Lower the default scanner cache save timeout so the derived usage persistence budget stays inside the effective distributed publication lease window.

Add focused regressions for the default publication budget and bootstrap-pending observational baselines, and update operator docs with the new default.

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-31 22:58:29 +08:00
唐小鸭 35456bcede test(scanner): serialize tests sharing process-global scanner state (#6940) 
Under the cargo test fallback (threads in one process), tests that touch
the process-global scanner cycle recovery status or the global usage-save
metrics raced each other and failed randomly in full-suite runs.

Mark all touchers with #[serial] per docs/testing/README.md:
- 22 tests reading or writing scanner_cycle_recovery_status() via
  load_scanner_cycle_state_for_startup / reset_scanner_cycle_recovery
- 24 tests mutating global_metrics() usage-save counters via
  store_data_usage_in_backend*, which raced the existing serial
  test_deferred_usage_save_keeps_last_real_save_metric

No-op under nextest, which isolates each test in its own process.
 2026-08-31 18:20:25 +08:00
Zhengchao An c4ac11d22e fix(scanner): persist decommission catch-up debt (#6922) 2026-08-31 08:45:36 +08:00
houseme 8ecd8f2520 fix(scanner): preserve cache cycle during usage recovery (#6921) 
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-31 00:19:54 +00:00
houseme fca1514aac fix(scanner): recover legacy empty usage floor (#6914) 
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-31 06:17:26 +08:00
唐小鸭 1370434f3a fix(scanner): unblock quota usage baseline on never-converged sites (#6896) 2026-08-31 00:20:04 +08:00
houseme ee39e4fccb fix(scanner): own publication mutations through storage drain (#6867) 
* fix(scanner): own publication mutations through storage drain

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(storage): remove unused rename data shim

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-30 02:39:07 +00:00
houseme ff3ad30f0c fix(scanner): bound publication proof retries on main (#6870) 
* fix(scanner): retain completed publication candidates

* fix(scanner): export publication activity helper

* test(ecstore): retain activity snapshot across retries

* fix(scanner): rebase publication proof retry onto main

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): resolve publication proof retry conflicts

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-29 22:41:36 +00:00
houseme c235f7c05d fix(scanner): retain usage across transient peer failures (#6859) 
* test(scanner): cover bucket drive guard lifecycle

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): recover usage floor from fenced backups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): retry transient activity probes

Retry one failed scanner activity probe after a bounded reconnect when the failure is transport-like or timed out. Keep protocol and response validation failures fail-closed.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): retain post-scan observations

Preserve a complete scanner walk as a non-converged observation when the final activity probe is unavailable. Advance the cycle as partial without acknowledging dirty usage.\n\nCo-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): classify publication lease deferrals

Distinguish persistence budget and lease deadline deferrals from unavailable activity baselines, and ensure lease-gate deferrals update usage metrics. Keep the fixed lease gate fail-closed while storage-owned commit scope work remains pending.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): recover usage floor from fenced backups

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): preserve publication lease defer reasons

Keep lease expiry and release failures distinct from activity baseline failures so scanner freshness metrics and cycle outcomes identify the publication barrier that blocked progress. Preserve fail-closed behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): reuse recovered usage baseline for publication

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): fence legacy usage floor fallback

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(scanner): use typed activity timeout error

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-30 02:54:20 +08:00
houseme c0155f0dfa fix(logging): bound ECStore debug output (#6809) 
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-29 04:51:37 +00:00
Zhengchao An 1585308f0f fix(test): restore #[serial] markers the fallback runner still needs (#6767) 2026-08-28 08:41:36 +08:00
Zhengchao An a96dd7d289 refactor: migrate consumers off rustfs-common heal/scanner shims (#6623) 
* refactor(ecstore): import heal/scanner contracts crates directly (backlog#1843)

* refactor(heal): import heal/scanner contracts crates directly (backlog#1843)

* refactor(lifecycle): import heal/scanner contracts crates directly (backlog#1843)

* refactor(obs): import heal/scanner contracts crates directly (backlog#1843)

* refactor(protos): import heal/scanner contracts crates directly (backlog#1843)

* refactor(scanner): import heal/scanner contracts crates directly (backlog#1843)

* refactor(rustfs): import heal/scanner contracts crates directly (backlog#1843)
 2026-08-26 21:13:18 +08:00
Henry Guo 1590d9107b fix(scanner): rebuild missing usage floor after upgrade (#6624) 
* fix(scanner): rebuild missing usage floor after upgrade

* fix(scanner): preserve missing-floor reset across conflicts

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
 2026-08-26 12:30:53 +08:00
Zhengchao An 7cac528de3 refactor(protos): move compat manifest send-site assertions into owning crates (#6618) 
refactor(protos): move internode compat manifest send-site assertions into owning crates

Promotes the rolling-upgrade dual-write manifest from a test-only constant in rustfs-protos into the public rustfs_protos::compat_manifest module, moves the JSON-encoder send-site assertions into the crates that own the asserted sources (ecstore remote_disk.rs for requests, the rustfs binary node_service/disk.rs for responses), and splits the scanner Phase-0 overlap inventory so its heal- and ecstore-owned halves live in those crates. Adds a cross-crate include_str!/include! guard with fixture self-tests to scripts/check_layer_dependencies.sh so a library crate can never again read another crate's Rust source at compile time, and records the rule in docs/architecture/crate-boundaries.md.

Part of rustfs/backlog#1884.
 2026-08-26 11:11:16 +08:00
houseme 0f987714a1 fix(ecstore): handle metadata-less bucket residue (#6591) 
* fix(ecstore): handle metadata-less bucket residue

Diagnose metadata-less on-disk residue before non-force DeleteBucket reaches physical deletion, and keep scanner-discovered metadata-missing objects on a non-destructive heal path.

Add explicit heal --remove cleanup for unversioned metadata-less data directories, using the existing data-dir delete primitive and fail-closed shape checks so pre-commit or unknown residue is preserved.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(connect): adapt offline array validator

Wrap the filesystem summary validator in a closure so Option::is_some_and can pass the concrete array reference accepted by serde_json::Value::as_array.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(connect): remove redundant offline test clones

Move the temporary path into the swap closure after deriving the output path, keeping clippy's redundant-clone lint clean for offline bundle tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-26 09:17:19 +08:00
Zhengchao An ade7e320da test(scanner): align pristine startup fixtures (#6516) 2026-08-24 18:29:47 +08:00
Henry Guo f06b004f2d fix(scanner): clarify follower status 2026-08-24 16:12:29 +08:00
Zhengchao An 7d3f5545e7 fix(ci): repair post-merge build gates 2026-08-24 16:10:24 +08:00
Zhengchao An cd1363d519 fix(scanner): restore s3s footprint baseline (#6486) 
chore(scanner): narrow s3s DTO references
 2026-08-24 14:30:16 +08:00
Zhengchao An 170a4c7640 fix(scanner): bootstrap pristine usage baseline (#6471) 
* fix(ecstore): fence pool metadata replica updates

* fix(ecstore): block decommission on unsafe pool metadata

* fix(ecstore): block writes after pool metadata save errors

* fix(ecstore): latch pool metadata writes before await

* fix(scanner): bootstrap pristine usage baseline
 2026-08-24 14:29:09 +08:00
cxymds eec0e0e056 fix(scanner): fence movement generation publication (#6461) 
* feat(scanner): add movement generation fencing

* fix(scanner): prioritize unverified cycle deferral

* feat(ecstore): add scanner publication lease fence

* feat(rpc): add scanner publication lease protocol

* feat(scanner): hold remote leases through usage publish

* test(scanner): cover publication lease fencing

* fix(scanner): fence remote leases across restart and delay

* feat(rpc): fence scanner publication rename writes

* fix(scanner): fence observed cleanup deletes

* fix(proto): qualify lease release test types

* fix(scanner): pin movement notifications

* fix(scanner): clean publication imports

* fix(ecstore): satisfy scanner fence clippy

* refactor(scanner): group wait and publication options

* fix(scanner): satisfy final lint and facade guards

* fix(rpc): resolve facade export conflicts

* fix(ci): remove unused decommission and healing facades

* fix(ci): cfg-gate test-only usage overlay import

* fix(scanner): wake on remote scanner restart
 2026-08-24 14:17:35 +08:00
cxymds 1c5c28842a fix(scanner): reject duplicate usage updates (#6445) 
* fix(scanner): reject duplicate usage updates

* style: format decommission test imports

* fix(ci): remove unused decommission and healing facades

* fix(ci): cfg-gate test-only usage overlay import

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-08-24 14:17:11 +08:00
Zhengchao An 0e015360cc test(scanner): repair post-fence fixtures (#6478) 2026-08-24 09:33:21 +08:00
Zhengchao An 2bd1df3075 fix(scanner): preserve default usage cache wire format (#6477) 2026-08-24 09:33:16 +08:00
Zhengchao An 99fb77b164 fix(ci): restore main checks (#6469) 
* fix(ci): restore main checks

* fix(scanner): bootstrap pristine usage state

* fix(scanner): reject empty usage snapshots
 2026-08-24 09:33:00 +08:00
houseme 201c653dcd fix(ci): restore workspace lint compatibility (#6460) 2026-08-23 22:35:43 +08:00
cxymds a8e4b67d99 feat(metrics): expose deferred usage freshness (#6449) 2026-08-23 19:31:01 +08:00
cxymds b2e60be647 fix(scanner): fence unknown tier accounting (#6396) 2026-08-23 19:28:43 +08:00
cxymds 76a863b3ea fix(scanner): unify unknown metadata size accounting (#6394) 
* fix(scanner): unify unknown metadata size accounting

* fix(scanner): preserve restore expiry semantics

* fix(ci): resolve ecstore clippy warnings

* fix(scanner): close lifecycle review gaps

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-08-23 17:28:52 +08:00
cxymds e196a134cc fix(scanner): fence system metadata publication (#6444) 
* feat(scanner): fence usage publication during data movement

* fix(scanner): detect movement refresh state changes

* fix(scanner): fence publication during data movement

* fix(scanner): close movement epoch publication races

* fix(scanner): fence movement-sensitive publication paths

* fix(scanner): fence cache and heal recovery paths

* fix(scanner): carry publication epoch through scan cycle

* fix(scanner): recheck remote cache epoch after save

* fix(scanner): recheck local cache epoch before publish

* fix(scanner): fence data usage writers and baseline

* fix(scanner): expose decommission activity to publication fence

* fix(scanner): release publication gate before reads

* fix(scanner): complete publication fence integration

* fix(scanner): avoid empty usage baseline publication

* chore(scanner): gate test-only helpers

* fix: use decommission canceler in reload test

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-08-23 17:28:21 +08:00
cxymds 9cda615519 fix(scanner): discover sub-quorum heal candidates (#6384) 
* fix(scanner): preserve unversioned heal retries

* fix(scanner): bound orphan heal discovery fallback

* fix(filemeta): fence unsafe heal key components

* fix(scanner): preserve exact overflow heal versions

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-08-23 16:46:36 +08:00
cxymds 32cc7c8fcf fix(heal): coalesce duplicate MRF intents (#6425) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-08-23 16:45:22 +08:00
cxymds 2bb0ab18b2 docs(scanner): baseline scanner heal admission (#6426) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-08-23 16:45:07 +08:00
cxymds 8dc2537178 fix(scanner): publish per-set usage freshness (#6432) 
* fix(scanner): publish partial usage observations

* fix(ecstore): preserve quota baseline across restart

* style: format usage freshness changes

* fix(scanner): correct observational usage arguments

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-08-23 16:44:45 +08:00
cxymds 2fccfdeabe fix(scanner): fence timed out scan cycles (#6352) 
* fix(scanner): fence timed out scan cycles

* fix(scanner): cancel scan workers with cycle scope

* fix(scanner): reject persisted timer overflow

* fix(scanner): reject terminal leadership epochs

* fix(scanner): reject trailing cycle state bytes
 2026-08-22 14:38:18 +00:00
Zhengchao An 12a9e654b5 refactor(data-usage): rename ReplicationStats to ReplicationTargetUsage (#6345) 
* refactor(data-usage): ReplicationStats -> ReplicationTargetUsage

Rename the data-usage crate's ReplicationStats to ReplicationTargetUsage.
Serde field names are byte-identical (only the Rust type name changed;
field identifiers that rmp encodes are untouched). An rmp round-trip test
guards against future drift.

Scanner test imports updated to match.

* style: cargo fmt
 2026-08-22 13:56:58 +00:00
cxymds 04e1ea227a fix(scanner): isolate corrupt cycle state (#6354) 
* fix(scanner): isolate corrupt cycle state

* fix(scanner): preserve newer state during recovery reset

* fix(scanner): fence recovery reset state

* fix(scanner): reject terminal recovery epochs

* fix(scanner): reject trailing cycle state bytes

* fix(scanner): reject terminal leadership epochs

* fix(scanner): retain recovery wake notifications

* fix(scanner): recover from oversized markers
 2026-08-22 11:11:48 +00:00