Commit Graph

21 Commits

Author SHA1 Message Date
Zhengchao An ebd0531124 chore(ecstore): drop the data_usage dead_code blanket (#6089)
Removing the blanket exposes eighteen items. Six are deleted; the rest belong to one feature that was never wired up.

crates/ecstore/src/data_usage/local_snapshot.rs and its aggregation entry point, aggregate_local_snapshots, form a complete per-disk usage-snapshot feature: read/write of snapshot files under the metadata bucket, cross-disk aggregation, and tests. Nothing calls it. It landed in #5307 on 2026-07-27 and git log -S over the whole history shows the entry point has never had a caller; the live data-usage path is load_data_usage_from_backend / store_data_usage_in_backend. Rather than delete a recent, tested feature on cleanup grounds, the module gets a header explaining the situation and its items carry individual allows, so the gap stays greppable without reintroducing a blanket. One commit flips this to deletion if that is preferred.

Deleted:

- DATA_USAGE_BLOOM_NAME together with DATA_USAGE_BLOOM_NAME_PATH. Both are a dead duplicate of the pair in crates/scanner/src/data_usage_define.rs, which has roughly fifty consumers across scanner.rs and remote_scanner.rs; the ecstore copies have none.
- increment_bucket_usage_memory and decrement_bucket_usage_memory, thin wrappers over the live record_bucket_object_write_memory and record_bucket_object_delete_memory.
- sync_memory_cache_with_backend, whose doc comment says it is "called by scanner" — nothing calls it.
- create_cache_entry_from_summary and cache_to_data_usage_info, neither with a consumer in any lane.

resolve_loaded_snapshot is kept with an allow: nine tests in the same file assert its primary/backup fallback.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. rustfs-scanner still compiles clean. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 2).
2026-08-14 21:57:29 +08:00
cxymds d60a77b750 fix(quota): enforce durable hard quota reservations (#6058)
* fix(quota): enforce durable hard quota reservations

* fix(quota): close reservation bypasses

* fix(quota): isolate tests and box object futures

* fix(quota): close legacy and deferred settlement bypasses

* fix(app): keep object futures off caller stacks

* fix(metrics): preserve object operation labels

* fix(logging): retain GET trace guard contract
2026-08-14 06:26:00 +00:00
Zhengchao An d31bd3cd10 fix(data-usage): make thin usage-cache types a read-only projection (#5981)
Delete the dead ecstore save_data_usage_cache and the thin
DataUsageCache::marshal_msg it was the only caller of, drop the
Serialize derive (and the dead DataUsageCacheStorage trait with its
save path) from the thin projection types so no write path can exist
outside the scanner's canonical map-encoded writer, and pin the
persisted .usage-cache.bin wire bytes with cross-crate fixture tests
on both the scanner writer and the thin reader.

Refs rustfs/backlog#1828 (T1-T3).
2026-08-12 09:09:09 +00:00
cxymds 6cce3d60bb fix(quota): reject oversized multipart completion (#5958)
* fix(quota): reject oversized multipart completion

* fix(arch): route quota test through app facade
2026-08-11 21:30:05 +08:00
anthonymartin 706a8b6061 fix(scanner): publish bounded observational usage (#5742)
* fix(scanner): publish bounded observational usage

* test(ci): serialize embedded integration ports

* test(cache): isolate generation-change timeout

* fix(scanner): address observational usage review

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-07 08:52:34 +08:00
Zhengchao An 4042bc0a5e fix(quota): admit writes against a persisted usage baseline while authoritative usage is unavailable (#5722)
Upgrading from a pre-v2 release leaves only the legacy .usage.json snapshot, which has no completeness marker and is demoted to non-authoritative, so every write to a quota-enabled bucket failed closed with a retryable 503 until the scanner's first complete cycle persisted .usage.v2.json — a production outage on large namespaces (issue #5716).

Quota admission now degrades to the last persisted per-bucket size: normalize_loaded_data_usage returns the pre-discard bucket sizes, the TTL-bounded snapshot cache retains them (carried forward through failed refreshes), and QuotaChecker::get_real_time_usage falls back to that baseline when the authoritative caches miss. The baseline is static between snapshot loads, so hard-quota enforcement is advisory for the duration of the degraded window — strictly tighter than beta.11 (usage treated as 0) and strictly more available than a blanket 503. Buckets absent from every persisted snapshot still fail closed, and removing a bucket's usage from the backend purges the baseline so a recreated bucket cannot inherit the dead incarnation's size.

Refs #5716
2026-08-05 01:42:09 +00:00
Henry Guo d5df66ac4f fix(scanner): require authoritative usage snapshots (#5333) 2026-07-28 07:29:57 +08:00
Henry Guo a63b79004c fix(scanner): make distributed usage convergence authoritative (#5151)
* fix(scanner): make distributed usage cycles authoritative

* fix(scanner): close distributed refresh races

* fix(config): align scanner reload integration

* fix(admin): scope config test helpers

* fix(scanner): harden distributed usage convergence

* fix(scanner): preserve rolling activity compatibility

* fix(admin): expose non-secret optional config values

* fix(scanner): acknowledge distributed dirty usage

* fix(ecstore): make bucket mutations cancellation safe

* fix(scanner): preserve pending dirty acknowledgements

* test(obs): account for superseded scanner metric

* fix(api): reject excess detached bucket mutations

* test: close scanner convergence coverage gaps

* fix(scanner): make path tracking cleanup one-shot

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-25 18:45:16 +08:00
Zhengchao An 87128682b0 fix(ecstore): recover data usage snapshots safely (#4979) 2026-07-17 19:02:16 +00:00
Zhengchao An 4589148f48 fix(admin): serve data usage endpoints from scanner snapshot instead of live listing (#4980) 2026-07-18 00:09:02 +08:00
Zhengchao An 627c396649 fix(scanner): allow usage snapshot save when existing timestamp is future-dated beyond clock tolerance (#4982) 2026-07-17 16:04:43 +00:00
houseme 13bdca6762 build(toolchain): switch Rust channel to stable (#4775)
* Change Rust toolchain channel to stable

Signed-off-by: houseme <housemecn@gmail.com>

* style: apply clippy --fix and cargo fix lint suggestions

Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:

- collapse needless borrows in `format!` args, prefer `?` over explicit
  early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
  `select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
  only by the test factory) with `#[allow(dead_code)]`

Behavior is unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-12 18:59:43 +08:00
Henry Guo 3f25426534 fix(ecstore): reject incomplete listing usage refreshes (#4698)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-11 08:47:49 +08:00
Henry Guo 24d0c09347 fix(admin): keep datausage live refresh stable (#4630)
fix(ecstore): keep fuller data usage overlay

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-10 09:17:35 +08:00
Zhengchao An 4f999bb6b8 perf(ecstore): backfill rename_data old size and gate the PUT prelookup (#4598) 2026-07-09 12:08:01 +08:00
Henry Guo 757f9b3b7b fix(admin): refresh accountinfo bucket usage (#4472) 2026-07-08 18:34:11 +08:00
wood 7484a61fa3 feat(compression): show cluster-level compression stat in grafana (#4112) 2026-06-30 19:05:05 +08:00
Zhengchao An 055d3ad825 refactor(ecstore): scope dead code allowances (#4098) 2026-06-30 06:07:45 +08:00
Henry Guo 1e303e5be0 fix(data-usage): refresh versioned usage state (#3969)
fix(data-usage): refresh versioned usage from authoritative state

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-28 07:58:07 +08:00
Zhengchao An 0a5b1b1b3a refactor: consolidate ecstore owner module layout (#3934)
* refactor: shrink ecstore root owner facades

* refactor: remove ecstore core store root shims

* refactor: move ecstore erasure owner modules

* refactor: remove ecstore root rpc facade

* refactor: move ecstore services domain modules
2026-06-27 09:03:20 +08:00
Zhengchao An c6ecfae39e refactor: move ecstore owner layout modules (#3932) 2026-06-27 05:54:25 +08:00