Commit Graph

9 Commits

Author SHA1 Message Date
Zhengchao An 5d820df79c chore(obs): rename ReplicationStats to ReplicationMetricsSnapshot (#6344)
* chore(obs): ReplicationStats -> ReplicationMetricsSnapshot, BucketReplicationStats -> BucketReplicationMetricsSnapshot

Rename in-obs-crate ReplicationStats and BucketReplicationStats to
ReplicationMetricsSnapshot and BucketReplicationMetricsSnapshot respectively.
No serde impact (these types are Prometheus metric collectors, not serialized).
No external consumers found outside the obs crate.

* cleanup: remove #[serial] annotations from e2e_test, scanner, lifecycle, and object-capacity crates

Remove no-op #[serial] attributes (nextest ignores serial_test) and the
serial_test dependency from four crates. All tests already use temp_env
for env-var isolation, making #[serial] purely redundant.

Crates cleaned:
- e2e_test (37 annotations, 9 imports, removed serial_test dep)
- rustfs-scanner (115 annotations across 7 files, removed serial_test dep)
- rustfs-lifecycle (46 annotations, removed serial_test dep)
- rustfs-object-capacity (38 annotations, removed serial_test dep)

Also converted scanner/tests/lifecycle_integration_test.rs
with_forced_immediate_enqueue_timeout helper from unsafe raw
env::set_var/remove_var to temp_env::async_with_vars for proper
isolation, and added async_closure feature to scanner's temp-env dep.

* fix(lifecycle): restore #[serial] on 2 tests that read env vars without temp_env

eval_inner_expires_latest_object_after_days_due and
eval_inner_does_not_panic_on_many_equal_due_events call eval_inner()
which reads ENV_ILM_PROCESS_TIME via std::env::var(). Without #[serial]
they race with other tests that set these vars via temp_env.

* style: cargo fmt
2026-08-22 00:30:30 +00:00
cxymds 2a44985037 fix(capacity): stop background schedulers on shutdown (#5797) 2026-08-07 19:02:42 +08:00
houseme 78d6918c52 feat: extend hotpath coverage across crates (#5505)
Add opt-in hotpath feature surfaces to every workspace crate and wire the root rustfs feature passthrough for function, allocation, and CPU profiling.

Add a focused set of function-level measurements for scanner, heal, lock, target replay, IAM, KMS, Keystone, trusted proxy, and capacity paths without adding request-scoped primitive wrappers.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-31 05:42:19 +00:00
houseme 56179210ab chore(deps): simplify dependency features (#4890)
* chore(deps): remove redundant dependency features

Remove manifest feature entries that are implied by other requested features in the same dependency declaration.

Verified that the resolved Cargo feature graph is unchanged after the cleanup.

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore(deps): narrow tokio and reqwest features

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-16 05:20:43 +00:00
houseme f3a7a4b0da chore(deps): localize workspace dependency features (#4888)
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.

Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-16 03:55:27 +00:00
Zhengchao An 73a30178f5 fix(object-capacity): make timeout fallback keep accumulated work and enter sampling within the time budget (#4517)
The capacity scan's timeout fallback required the exact prefix
(threshold + sample_rate files, default ~200k) to be fully enumerated
before any sample existed. Cold-cache HDDs cannot stat that many files
inside the 15s budget, so large slow disks — exactly the disks sampling
was designed for — always timed out with sampled_count == 0, discarded
the accumulated exact-prefix work with a hard error, and re-ran 15s of
useless full-disk I/O every 120s (S03). When a sample did exist, the
estimate only extrapolated over files the walker had seen, silently
under-reporting disks whose tail was never reached (S10).

- Enter sampling early: once half the (possibly dynamic) time budget is
  consumed and the exact prefix hasn't filled, freeze the threshold at
  the current position so the remaining budget collects samples and the
  timeout fallback is always reachable.
- Compensate the unseen tail: when the scan root is a dedicated mount
  point (stat st_dev differs from the parent), take the max of the
  seen-files extrapolation and the filesystem-level used bytes. On
  shared filesystems (multi-disk dev layouts) statvfs would overcount,
  so it is not trusted and behavior is unchanged.
- sampled_count == 0 at timeout no longer hard-fails when a dedicated
  mount provides filesystem usage; the exact prefix is kept as a floor.
  Without any estimator the original error still surfaces.
- Extract ScanLimits from env reading so the blocking scanner is
  parameterizable in tests.

Ref: rustfs/backlog#1013 (S03+S10 from audit rustfs/backlog#1010)
2026-07-08 16:53:35 +00:00
cxymds 4cb28abfc9 fix(scanner): reduce non-actionable scan noise (#3840) 2026-06-25 14:35:10 +08:00
houseme 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012)
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
2026-05-19 12:50:25 +00:00
houseme 064e21062d fix(capacity): harden scope registry, scan symlink guard, and test temp dir cleanup (#2432)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-04-08 20:58:17 +08:00