Commit Graph

58 Commits

Author SHA1 Message Date
Zhengchao An 0a5b1b1b3a refactor: consolidate ecstore owner module layout (#3934)
* refactor: shrink ecstore root owner facades

* refactor: remove ecstore core store root shims

* refactor: move ecstore erasure owner modules

* refactor: remove ecstore root rpc facade

* refactor: move ecstore services domain modules
2026-06-27 09:03:20 +08:00
Zhengchao An b38976d5ee refactor: segment ECStore storage contracts by domain (#3910) 2026-06-26 17:47:36 +08:00
Zhengchao An c9614eb7cb refactor: route ecstore storage api boundaries (#3892) 2026-06-26 09:35:18 +08:00
Zhengchao An 94034ef406 refactor: centralize ecstore batch sources and aliases (#3817) 2026-06-24 14:12:48 +08:00
Zhengchao An 5b6c486b13 refactor: centralize ecstore locality runtime sources (#3808) 2026-06-24 08:42:36 +08:00
安正超 54ffbedbc8 refactor: remove ecstore operation compat facades (#3608) 2026-06-19 17:01:12 +08:00
安正超 57403525ee refactor: move heal and namespace contracts (#3560) 2026-06-18 11:00:21 +08:00
安正超 e5cad7ed20 refactor: move object operation contracts (#3559) 2026-06-18 09:48:13 +08:00
安正超 53337a6f71 refactor: move HTTP range helper contracts (#3533) 2026-06-18 00:47:23 +08:00
安正超 c26593fa7a refactor: clean config storage boundaries (#3489) 2026-06-16 06:35:32 +08:00
安正超 a85cc0354c refactor(storage-api): remove namespace lock from StorageAPI (#3365)
* refactor(storage-api): remove namespace lock from StorageAPI

* test(scanner): wait for runtime budget cancellation

---------

Co-authored-by: loverustfs <hello@rustfs.com>
2026-06-11 22:42:12 +08:00
安正超 a0b6636b61 refactor(config): remove legacy accessor re-export (#3362) 2026-06-11 19:32:57 +08:00
安正超 b4524033e3 refactor(config): move global config accessors (#3360) 2026-06-11 19:16:02 +08:00
安正超 ed3851782c refactor(config): remove legacy model re-export (#3357) 2026-06-11 18:09:01 +08:00
安正超 ca58d7f0ec refactor(config): migrate server config consumers (#3353) 2026-06-11 17:04:50 +08:00
安正超 2205991180 refactor(config): extract server config model (#3351) 2026-06-11 16:11:17 +08:00
安正超 686068d8bd refactor(storage): narrow metadata bounds (#3343) 2026-06-11 12:04:02 +08:00
安正超 6b86e3875c refactor(storage): remove old admin surfaces (#3340) 2026-06-11 11:28:09 +08:00
Henry Guo 62e9c5b94f feat(scanner): add adaptive pacing controls (#3315)
* feat(scanner): add adaptive pacing controls

* fix(scanner): bound pacing delay status precision

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-10 03:46:02 +00:00
Henry Guo e79c793803 feat(heal): add scanner-aware bitrot controls (#3297)
* feat(heal): add scanner-aware bitrot controls

* fix(config): register heal defaults

* test(admin): avoid heal config queue conflict

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-09 13:43:18 +00:00
cxymds 55590e38fb fix(config): accept Kafka SASL keys in legacy admin (#3300) 2026-06-09 12:01:15 +00:00
Henry Guo 6850457707 feat(scanner): add runtime scanner controls and status (#3203)
* feat(scanner): add runtime scanner controls and status

* fix(scanner): validate persisted scanner config

* docs(scanner): clarify start delay behavior

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-04 05:16:26 +00:00
安正超 97dd825468 fix(ecstore): optimize ObjectInfo clone and fix critical TODOs (#3149)
* fix(ecstore): optimize ObjectInfo clone and fix critical TODOs

## ObjectInfo hot-path clone optimization (issue #653 item 2)
- Wrap user_defined (HashMap), user_tags (String), parts (Vec) in Arc
- Clone cost reduced from ~20 heap allocations to O(1) ref count bump
- Updated 11 downstream access sites with explicit deref where needed

## Critical TODO/FIXME fixes (issue #653 item 7)
- rpc/peer_s3_client.rs: descriptive error message for empty peer response
- set_disk/write.rs: concurrent rollback deletes via tokio::spawn + join_all
- store_list_objects.rs: resolved FIXME with explanation + 4 regression tests
- store/bucket.rs: namespace write locks for make_bucket and delete_bucket

Skipped TODOs (too risky without broader context):
- multipart.rs:30 nslock — causes lock timeout in existing tests
- bucket.rs:88 cached list_bucket — needs cache invalidation strategy
- bucket.rs:149 replication delete — needs replication subsystem integration

1134 tests pass (1 flaky under parallel execution, passes individually).

* fix: address PR #3149 review comments

- Preserve NamespaceLockQuorumUnavailable variant in bucket create/delete
- Fix cancellation test to actually test cancellation (keep senders open)
- Update ObjectInfo consumers in app/ for Arc-backed fields
- Update bucket_usecase.rs user_tags to Arc<String>

* fix: update ObjectInfo Arc consumers

* fix: address ecstore merge review comments

* fix: cancel blocked merge output sends

* fix(ecstore): satisfy ObjectInfo clone clippy
2026-06-01 00:44:58 +00:00
houseme 8577bd825e feat(admin): restore config admin compatibility (#3133)
* feat(admin): restore config admin compatibility

Co-authored-by: weisd <im@weisd.in>

* fix(admin): align config admin clean rebuild

Co-authored-by: weisd <im@weisd.in>

* fix(admin): align config history and peer signals

* fix(admin): harden config admin mutations

* fix(admin): tighten config review follow-ups

* perf(admin): reuse env snapshot in config render

* fix(ecstore): clean up config admin and listing error handling

Remove redundant is_all_volume_not_found check in list_merged, add
storage class encode/decode roundtrip tests, fresh boot integration
test, and config admin clean rebuild improvements.

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(admin): sync global server config on mutation and reload

Change GLOBAL_SERVER_CONFIG from OnceLock to RwLock so config mutations
(set/del/restore/reload) are visible to readers without restart. Call
set_global_server_config after every store save and on snapshot reload.
Register storage_class as a dynamic config subsystem.

Co-authored-by: hehutu <heihutu@gmail.com>

* style: apply rustfmt to config and admin tests

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(test): update signal_service test for storage_class dynamic subsystem

storage_class is now a valid dynamic config subsystem, so the
"requires object layer" test should expect "storage layer not initialized"
instead of "unsupported dynamic config subsystem".

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(config): publish storage_class runtime config on dynamic reload

Change GLOBAL_STORAGE_CLASS from OnceLock to RwLock so runtime updates
are possible. apply_storage_class_runtime_config now actually publishes
the parsed config via set_global_storage_class instead of dropping it.

Addresses review feedback: storage_class was marked as dynamically
applied but the parsed result was discarded, so mc admin config set
returned config_applied=true while the runtime kept using stale parity
settings until restart.

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(admin): harden config init, history ordering, and env redaction

- Change GLOBAL_SERVER_CONFIG from RwLock<Config> to RwLock<Option<Config>>
  initialized with None, preserving "not initialized" detection via None
- Move save_server_config_history before save_server_config_to_store in
  SetConfigKVHandler, DelConfigKVHandler, and SetConfigHandler so a
  restore point exists before mutations are persisted
- Redact sensitive env override values with *redacted* instead of
  silently omitting the line, improving admin visibility
- Add code comment explaining VolumeNotFound removal rationale in
  list_merged for listing paths

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(config): keep in-memory config in sync after set/restore/reload

GLOBAL_SERVER_CONFIG was a OnceLock set once at startup and never
updated. After mc admin config set writes to the store, any fallback
to get_global_server_config() returned stale init-time data. Similarly,
reload_runtime_config_snapshot read from the store but discarded the
result.

- Replace OnceLock with RwLock for GLOBAL_SERVER_CONFIG and
  GLOBAL_STORAGE_CLASS so they can be updated at runtime
- Add set_global_server_config / set_global_storage_class setters
- Call set_global_server_config after every config save (set-kv,
  del-kv, set-config, restore-history)
- Re-apply dynamic subsystems (storage_class, audit_webhook,
  audit_mqtt) and signal peers in reload_runtime_config_snapshot
  and full-config operations
- Fix render_selected_config scope boundary check: track per-scope
  line count instead of checking global lines.is_empty()
- Include STORAGE_CLASS_SUB_SYS in is_dynamic_config_subsystem so
  apply_storage_class_runtime_config is reachable

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(storageclass): use CLASS_RRS key in lookup_config for RRS parity

lookup_config used kvs.get(RRS) where RRS="REDUCED_REDUNDANCY", but the
admin config path writes the key as CLASS_RRS="rrs". This caused RRS
values to never be read back, always falling back to default parity.

- Changed kvs.get(RRS) to kvs.get(CLASS_RRS) in lookup_config
- Added regression tests verifying RRS read/write consistency

Co-authored-by: hehutu <heihutu@gmail.com>

* fix(config): add peer-side logging and don't swallow apply errors

- Add tracing::warn! in reload_dynamic_config_runtime_state and
  reload_runtime_config_snapshot when config read or subsystem apply
  fails, so on-host diagnostics show which signal failed and why
- Change `let _ = apply_dynamic_config_for_subsystem(...)` to
  `if let Err(err) = ... { warn!(...) }` in reload_runtime_config_snapshot
  so per-subsystem failures are logged instead of silently swallowed
- Remove weak test global_server_config_returns_none_before_init that
  had no meaningful assertion due to shared global state

Co-authored-by: hehutu <heihutu@gmail.com>

* style: apply rustfmt to config and storageclass tests

Co-authored-by: hehutu <heihutu@gmail.com>

---------

Co-authored-by: weisd <im@weisd.in>
Co-authored-by: hehutu <heihutu@gmail.com>
2026-05-31 11:50:13 +00:00
houseme 81ad48dac2 feat(targets): add AMQP support for notify and audit (#2879)
Co-authored-by: Hyesook Yun <74169420+suk13574@users.noreply.github.com>
2026-05-09 01:56:26 +00:00
houseme 5431b9273d feat(targets): complete redis mysql postgres target wiring (#2842)
Signed-off-by: jaehanbyun <awbrg789@naver.com>
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Co-authored-by: jaehanbyun <awbrg789@naver.com>
Co-authored-by: Gunther Xing <jiengup@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: JaySon <tshent@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-07 10:00:59 +00:00
houseme 3dd0692917 refactor: unify credential env constants and deploy env usage (#2821)
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-06 04:30:29 +00:00
houseme 368ef0f16c refactor(targets): unify endpoint source/merge logic and bump rustfs-kafka-async to v1.2.0 (#2654)
Co-authored-by: Filipe Monteiro <a22407332@alunos.ulht.pt>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-04-23 09:14:36 +00:00
Andy Teijelo Pérez 989827e3b5 feat: add OTHER_AUDIENCES config (#2605)
Co-authored-by: GatewayJ <835269233@qq.com>
2026-04-21 03:48:13 +00:00
houseme 3796b684f0 feat(targets): add NATS and Pulsar target support (#2618) 2026-04-21 00:28:19 +00:00
houseme 478720d2ee Centralize lifecycle state updates and fix systemd running status (#2567) 2026-04-17 03:20:11 +00:00
Tunglies 49366ee200 chore(lint): clippy rules redundant_clone (#2554) 2026-04-15 13:54:07 +00:00
安正超 642d83f0e4 revert: remove #2351 chunk I/O and object-io crate (phase 7) (#2543)
Co-authored-by: houseme <housemecn@gmail.com>
2026-04-15 10:54:41 +08:00
GatewayJ 16b9189e9b feat(oidc): add roles_claim and jwt:roles policy support (#2509)
Co-authored-by: GatewayJ <8352692332qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-04-15 09:30:24 +08:00
安正超 2a3d127a86 revert: restore PutObjReader and remove PUT zero-copy chunk path (#2351 phase 2) (#2532) 2026-04-14 16:35:03 +08:00
houseme d6158c0481 feat(mqtt): migrate client and harden TLS config (#2413)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-07 15:28:29 +08:00
houseme 32bf8f5bf3 feat(storage): add direct chunk GET fast path (#2351)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-04-07 08:33:46 +08:00
houseme d2901fd78c feat(admin): add audit target APIs and harden target source handling (#2350)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
2026-04-04 09:07:22 +08:00
weisd e86c6b726f fix(lock): split distributed read and write quorum (#2355) 2026-03-31 16:16:13 +08:00
cxymds 75e6902f46 feat(admin): add persisted OIDC config APIs (#2267)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-03-24 12:13:41 +08:00
weisd 05dc131a49 perf(storage): optimize internode RPC transfer path (#2262)
Co-authored-by: momoda693 <momoda693@gmail.com>
2026-03-23 17:09:49 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
evan slack d52a10c5fb chore(obs): Improve tracing instrumentation (#2086)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-03-07 20:03:20 +08:00
安正超 09aa6d9f6f refactor(app): remove remaining global access in main init (#1978) 2026-02-26 22:14:51 +08:00
安正超 49579129c1 refactor(app): decouple AppContext adapters from GLOBAL statics (#1970) 2026-02-26 14:54:45 +08:00
Jeff Poegel c35ef84a8c feat(iam): add OpenID Connect SSO with claim-based policy resolution (#1875)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-02-26 14:03:17 +08:00
houseme d251b9fb35 fix: unify path handling to use S3-standard forward slashes on all platforms (#1555)
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: heihutu <30542132+heihutu@users.noreply.github.com>
Co-authored-by: LeonWang0735 <wlywly0735@126.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-26 18:49:21 +08:00
houseme 760cb1d734 Fix Windows Path Separator Handling in rustfs_utils (#1464)
Co-authored-by: reatang <tangtang1251@qq.com>
2026-01-11 19:53:51 +08:00
houseme 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
houseme 4abfc9f554 Fix/fix event 1216 (#1191)
Signed-off-by: loverustfs <hello@rustfs.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2025-12-19 12:07:07 +08:00