Commit Graph

69 Commits

Author SHA1 Message Date
houseme 8cf3c0bfbd fix(ecstore): support MinIO DARE fixture compatibility (#3590) 2026-06-19 10:13:39 +08:00
安正超 c28fee0013 refactor: continue storage api contract cleanup (#3580)
* refactor: move delete object contracts to storage api

* refactor: narrow store api compatibility exports

* refactor: route table catalog test through storage compat
2026-06-18 22:42:02 +08:00
安正超 57403525ee refactor: move heal and namespace contracts (#3560) 2026-06-18 11:00:21 +08:00
安正超 e5cad7ed20 refactor: move object operation contracts (#3559) 2026-06-18 09:48:13 +08:00
安正超 3fb4cb3d65 refactor: move list operations contract (#3550) 2026-06-18 08:48:18 +08:00
安正超 53337a6f71 refactor: move HTTP range helper contracts (#3533) 2026-06-18 00:47:23 +08:00
安正超 919deeb816 refactor: move object option helper contracts (#3521) 2026-06-17 22:56:10 +08:00
安正超 ed55857bca refactor: move bucket operations contract (#3507) 2026-06-17 09:10:38 +08:00
安正超 0c259547d1 refactor: move multipart DTO contracts (#3505) 2026-06-17 00:33:06 +08:00
安正超 2ff69ae21c refactor: remove ecstore bucket DTO aliases (#3503) 2026-06-16 22:10:07 +08:00
安正超 0baf90ee88 refactor: remove storage api facade (#3490) 2026-06-16 07:46:32 +08:00
安正超 a85cc0354c refactor(storage-api): remove namespace lock from StorageAPI (#3365)
* refactor(storage-api): remove namespace lock from StorageAPI

* test(scanner): wait for runtime budget cancellation

---------

Co-authored-by: loverustfs <hello@rustfs.com>
2026-06-11 22:42:12 +08:00
安正超 b4524033e3 refactor(config): move global config accessors (#3360) 2026-06-11 19:16:02 +08:00
安正超 ca58d7f0ec refactor(config): migrate server config consumers (#3353) 2026-06-11 17:04:50 +08:00
安正超 6b86e3875c refactor(storage): remove old admin surfaces (#3340) 2026-06-11 11:28:09 +08:00
安正超 4b1714438e feat(storage-api): bind ecstore admin contract (#3331) 2026-06-11 05:49:14 +08:00
安正超 e2fe99d8bc refactor(ecstore): add server configuration accessors (#3238)
* refactor(ecstore): add server configuration accessors

Phase 4 of global singleton consolidation. Add ECStore accessor
methods for server-level configuration globals.

New methods:
- port() — delegates to global_rustfs_port()
- host() — delegates to GLOBAL_RUSTFS_HOST
- addr() — delegates to GLOBAL_RUSTFS_ADDR
- region() — delegates to get_global_region()
- endpoints() — delegates to get_global_endpoints()
- server_config() — delegates to get_global_server_config()
- storage_class() — delegates to get_global_storage_class()
- tier_config_mgr() — delegates to get_global_tier_config_mgr()
- notification_system() — delegates to get_global_notification_sys()
- bucket_metadata_sys() — delegates to get_global_bucket_metadata_sys()

1160 tests pass, clippy clean, formatting clean.

* fix(ecstore): avoid duplicate server accessors
2026-06-06 21:38:45 +08:00
安正超 06acf49672 refactor(ecstore): add accessor methods for service globals (#3226)
* refactor(ecstore): add accessor methods for service globals

Phase 3 of global singleton consolidation. Add ECStore accessor
methods that delegate to process-global service singletons,
providing a unified API through ECStore.

New methods:
- notification_system() — delegates to notification_sys global
- bucket_metadata_sys() — delegates to bucket metadata global
- endpoints() — delegates to endpoints global
- region() — delegates to region global
- tier_config_mgr() — delegates to tier config global
- server_config() — delegates to server config global
- storage_class() — delegates to storage class global

1161 tests pass, clippy clean, formatting clean.

* fix: use imported functions instead of fully-qualified paths

Address review comments: use imported functions instead of
crate:: paths to avoid unused import warnings.

1160 tests pass, clippy clean.

* fix(ecstore): remove unused service imports
2026-06-05 15:29:38 +00:00
安正超 8185376b90 refactor(ecstore): migrate config globals into ECStore struct fields (#3219)
* refactor(ecstore): migrate config globals into ECStore struct fields

Phase 2 of global singleton consolidation. Add server_config and
storage_class fields to ECStore, sharing the same underlying data
as the process-global LazyLock statics.

New ECStore fields:
- server_config: RwLock<Option<Config>>
- storage_class: RwLock<storageclass::Config>

New async accessor methods:
- get_server_config() / set_server_config()
- get_storage_class() / set_storage_class()

Fields initialized with defaults in ECStore::new(), synced from
globals after init() completes. Global functions preserved for
backward compatibility.

1160 tests pass, clippy clean, full workspace compiles.

* fix: address PR #3219 review comments - delegate to globals

Accessors now delegate to process-global statics to avoid state drift:
- get_server_config() delegates to get_global_server_config()
- set_server_config() updates both global and local field
- get_storage_class() delegates to get_global_storage_class()
- set_storage_class() updates both global and local field
- Removed stale sync code from init()
- Changed fields to std::sync::RwLock for sync access

1160 tests pass.

* fix: correct import order for cargo fmt

* fix: simplify config accessors - delegate to globals without local state

Remove local server_config and storage_class fields from ECStore.
Accessors now purely delegate to process-global statics, eliminating
the state drift risk identified in review.

1160 tests pass, formatting clean.
2026-06-05 01:06:08 +00:00
安正超 d6120f5788 refactor(ecstore): migrate mutable globals into ECStore struct fields (#3214)
* refactor(ecstore): migrate mutable globals into ECStore struct fields

Phase 1 of global singleton consolidation. Move mutable globals from
lazy_static into ECStore struct fields as the first step toward
dependency injection and multi-instance support.

New ECStore fields:
- is_erasure, is_dist_erasure, is_erasure_sd (erasure type flags)
- local_disk_map, local_disk_id_map, local_disk_set_drives
- root_disk_threshold
- tier_config_mgr, event_notifier, bucket_monitor

New accessor methods:
- is_erasure(), is_dist_erasure(), is_erasure_sd()
- update_erasure_type()
- tier_config_mgr(), event_notifier(), bucket_monitor()

Global functions in global.rs preserved for backward compatibility.
1151 ecstore tests pass.

* fix: address PR #3214 review comments

- Sync ECStore fields from globals after init()
- Enforce DistErasure => is_erasure invariant in update_erasure_type()
- Change bucket_monitor from Option to OnceLock for deferred initialization
- Restrict TypeLocalDiskSetDrives to pub(crate)
- 1151 tests pass

* fix: address PR #3150 review comments

- Restore get_host_addr as best-effort wrapper (String return type)
- Replace bare expect("err") with descriptive messages in unsigned trailer
- Simplify aws-chunked header construction
- 20 signer + 96 io-core tests pass

* fix(ecstore): format store imports

* fix(ecstore): keep migrated accessors in sync

* fix(signer): preserve host fallback and unsigned trailer errors

* fix(ecstore): defer migrated global accessors

* fix(signer): box unsigned trailer signing errors

* fix(ecstore): avoid lock awaits during sync

* fix(ecstore): narrow phase one globals

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-04 17:27:21 +00:00
houseme 8577bd825e feat(admin): restore config admin compatibility (#3133)
* feat(admin): restore config admin compatibility

Co-authored-by: weisd <im@weisd.in>

* fix(admin): align config admin clean rebuild

Co-authored-by: weisd <im@weisd.in>

* fix(admin): align config history and peer signals

* fix(admin): harden config admin mutations

* fix(admin): tighten config review follow-ups

* perf(admin): reuse env snapshot in config render

* fix(ecstore): clean up config admin and listing error handling

Remove redundant is_all_volume_not_found check in list_merged, add
storage class encode/decode roundtrip tests, fresh boot integration
test, and config admin clean rebuild improvements.

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(admin): sync global server config on mutation and reload

Change GLOBAL_SERVER_CONFIG from OnceLock to RwLock so config mutations
(set/del/restore/reload) are visible to readers without restart. Call
set_global_server_config after every store save and on snapshot reload.
Register storage_class as a dynamic config subsystem.

Co-authored-by: hehutu <heihutu@gmail.com>

* style: apply rustfmt to config and admin tests

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(test): update signal_service test for storage_class dynamic subsystem

storage_class is now a valid dynamic config subsystem, so the
"requires object layer" test should expect "storage layer not initialized"
instead of "unsupported dynamic config subsystem".

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(config): publish storage_class runtime config on dynamic reload

Change GLOBAL_STORAGE_CLASS from OnceLock to RwLock so runtime updates
are possible. apply_storage_class_runtime_config now actually publishes
the parsed config via set_global_storage_class instead of dropping it.

Addresses review feedback: storage_class was marked as dynamically
applied but the parsed result was discarded, so mc admin config set
returned config_applied=true while the runtime kept using stale parity
settings until restart.

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(admin): harden config init, history ordering, and env redaction

- Change GLOBAL_SERVER_CONFIG from RwLock<Config> to RwLock<Option<Config>>
  initialized with None, preserving "not initialized" detection via None
- Move save_server_config_history before save_server_config_to_store in
  SetConfigKVHandler, DelConfigKVHandler, and SetConfigHandler so a
  restore point exists before mutations are persisted
- Redact sensitive env override values with *redacted* instead of
  silently omitting the line, improving admin visibility
- Add code comment explaining VolumeNotFound removal rationale in
  list_merged for listing paths

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(config): keep in-memory config in sync after set/restore/reload

GLOBAL_SERVER_CONFIG was a OnceLock set once at startup and never
updated. After mc admin config set writes to the store, any fallback
to get_global_server_config() returned stale init-time data. Similarly,
reload_runtime_config_snapshot read from the store but discarded the
result.

- Replace OnceLock with RwLock for GLOBAL_SERVER_CONFIG and
  GLOBAL_STORAGE_CLASS so they can be updated at runtime
- Add set_global_server_config / set_global_storage_class setters
- Call set_global_server_config after every config save (set-kv,
  del-kv, set-config, restore-history)
- Re-apply dynamic subsystems (storage_class, audit_webhook,
  audit_mqtt) and signal peers in reload_runtime_config_snapshot
  and full-config operations
- Fix render_selected_config scope boundary check: track per-scope
  line count instead of checking global lines.is_empty()
- Include STORAGE_CLASS_SUB_SYS in is_dynamic_config_subsystem so
  apply_storage_class_runtime_config is reachable

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(storageclass): use CLASS_RRS key in lookup_config for RRS parity

lookup_config used kvs.get(RRS) where RRS="REDUCED_REDUNDANCY", but the
admin config path writes the key as CLASS_RRS="rrs". This caused RRS
values to never be read back, always falling back to default parity.

- Changed kvs.get(RRS) to kvs.get(CLASS_RRS) in lookup_config
- Added regression tests verifying RRS read/write consistency

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(config): add peer-side logging and don't swallow apply errors

- Add tracing::warn! in reload_dynamic_config_runtime_state and
  reload_runtime_config_snapshot when config read or subsystem apply
  fails, so on-host diagnostics show which signal failed and why
- Change `let _ = apply_dynamic_config_for_subsystem(...)` to
  `if let Err(err) = ... { warn!(...) }` in reload_runtime_config_snapshot
  so per-subsystem failures are logged instead of silently swallowed
- Remove weak test global_server_config_returns_none_before_init that
  had no meaningful assertion due to shared global state

Co-authored-by: hehutu <heihutu@gmail.com>

* style: apply rustfmt to config and storageclass tests

Co-authored-by: hehutu <heihutu@gmail.com>

---------

Co-authored-by: weisd <im@weisd.in>
Co-authored-by: hehutu <heihutu@gmail.com>
2026-05-31 11:50:13 +00:00
weisd 8dc5ef6ef5 fix(storage): avoid startup UUID disk lookup misses (#2576) 2026-04-17 07:55:47 +00:00
houseme 6ce24f3b63 feat(lifecycle): improve ILM compatibility and scanner runtime config (#2534)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-04-16 10:28:03 +00:00
安正超 2a3d127a86 revert: restore PutObjReader and remove PUT zero-copy chunk path (#2351 phase 2) (#2532) 2026-04-14 16:35:03 +08:00
安正超 458086dc80 fix(get): remove GET chunk fast path (#2507) 2026-04-12 23:12:23 +08:00
houseme 32bf8f5bf3 feat(storage): add direct chunk GET fast path (#2351)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-04-07 08:33:46 +08:00
weisd 28f57b228c feat(s3): advance parity coverage (#2278) 2026-03-24 17:29:33 +08:00
weisd 05dc131a49 perf(storage): optimize internode RPC transfer path (#2262)
Co-authored-by: momoda693 <momoda693@gmail.com>
2026-03-23 17:09:49 +08:00
cxymds 236142a682 fix(ecstore): repair lifecycle transition and restore flows (#2240)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: weisd <im@weisd.in>
2026-03-23 12:29:13 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
evan slack d52a10c5fb chore(obs): Improve tracing instrumentation (#2086)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-03-07 20:03:20 +08:00
安正超 af6c32efac refactor: improve code quality with safer error handling, trait decomposition, and dead code cleanup (#1997) 2026-02-28 01:19:47 +08:00
安正超 095b77795d refactor(ecstore): split store.rs into store submodules (#1942) 2026-02-25 06:35:27 +08:00
LeonWang0735 06d12a8ec8 feat(replication):add replication bandwidth throttle monitor and reader (#1885)
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-24 15:21:45 +08:00
evan slack 23f7ffe36b fix(startup): Only monitor disk health after format loading (#1854)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-02-21 20:46:14 +08:00
evan slack 2093a13308 logging(disks): Propogate storage disk init error, improve logging (#1825)
Co-authored-by: houseme <housemecn@gmail.com>
2026-02-15 15:28:07 +08:00
houseme 0b870d6301 build(deps): bump the dependencies group with 19 updates (#1745) 2026-02-07 12:22:14 +08:00
weisd dce117840c refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes (#1664)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: weisd <2057561+weisd@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-30 13:13:41 +08:00
houseme 2ee81496b0 fix: deduplicate disks in capacity calculation to prevent inflation (#1656) 2026-01-30 00:03:21 +08:00
weisd 6631407416 feat: Add RustFS Scanner Module and Multiple Bug Fixes (#1579) 2026-01-22 13:39:38 +08:00
houseme 2d58eea702 fix: exclude matching key from ListObjects results when using marker/startAfter (#1506) 2026-01-14 19:21:51 +08:00
安正超 60103f0f72 fix: s3 api compatibility (#1370)
Co-authored-by: houseme <housemecn@gmail.com>
2026-01-05 16:54:16 +08:00
weisd ab752458ce Fix Path Traversal and Enhance Object Validation (#1387) 2026-01-05 15:57:15 +08:00
houseme 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
houseme 82a6e78845 Inject GlobalReadiness into HTTP server pipeline and gate traffic until FullReady (#1255) 2025-12-25 00:19:03 +08:00
houseme 4abfc9f554 Fix/fix event 1216 (#1191)
Signed-off-by: loverustfs <hello@rustfs.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-19 12:07:07 +08:00
0xdx2 7c6cbaf837 feat: enhance error handling and add precondition checks for object o… (#1008) 2025-12-06 20:39:03 +08:00
weisd 55d44622ed list object include deleted support (#882)
Co-authored-by: houseme <housemecn@gmail.com>
2025-11-18 21:51:10 +08:00
houseme d934e3905b Refactor telemetry initialization for non-production environments (#789)
* add dep `scopeguard`

* improve for tracing

* fix

* fix

* improve code for import

* add logger trace id

* fix

* fix

* fix

* fix

* fix
2025-11-05 00:55:08 +08:00
安正超 dd47fcf2a8 fix: restore localized samples in tests (#749)
* fix: restore required localized examples

* style: fix formatting issues
2025-10-29 13:16:31 +08:00