Commit Graph

3235 Commits

Author SHA1 Message Date
安正超 f5bb034ec8 feat(kms): migrate KMS handlers to dedicated actions (#3298)
feat: migrate KMS handlers to dedicated actions
2026-06-09 12:01:02 +00:00
安正超 4fec606dc4 feat: add KMS action taxonomy (#3294) 2026-06-09 05:06:15 +00:00
Henry Guo ad9bf41fc8 feat(heal): expose scanner heal admission outcomes (#3292)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-09 03:39:57 +00:00
Henry Guo 51c26278a4 feat(table-catalog): support standard REST create commit (#3287)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-09 02:18:33 +00:00
安正超 b9a3a2245a feat: add admin route policy inventory (#3286)
* feat: add admin route policy inventory

* fix: align table route policy actions

* fix: align load table route policy action
2026-06-09 02:17:50 +00:00
houseme aabda41ae5 build: upgrade Rust baseline to 1.96.0 (#3291)
* build: upgrade Rust baseline to 1.96.0

* ci: pin Rust workflows to 1.96.0

* remove

* ci: pin setup action to Rust 1.96.0

* build: pin toolchain to Rust 1.96.0
2026-06-08 20:32:37 +00:00
houseme 9354b939ca refactor(zip): simplify archive extraction path (#3290)
* refactor(zip): simplify archive extraction path

* build(deps): align datafusion and http updates

* fix(test): restore pre-commit compatibility
2026-06-08 20:03:02 +00:00
Henry Guo 6413df4b7f feat(replication): expose scanner repair outcomes (#3278)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-08 14:24:27 +00:00
cxymds 452c003341 fix(scanner): ignore missing rebalance metadata (#3282)
* fix(ecstore): harden rebalance data movement

* fix(ecstore): preserve failed rebalance status

* fix(ecstore): avoid rebalance walkdir total timeout

* fix(ecstore): retry rebalance listing timeouts

* fix(ecstore): restrict data movement resume target

* refactor(ecstore): simplify multipart movement target

* fix(ecstore): restore resume target checks

* perf(ecstore): speed up rebalance bucket merges

* fix: keep rebalance listing alive on transient failures

* fix(scanner): ignore missing rebalance metadata

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 14:24:16 +00:00
Henry Guo 8c3e52efb8 feat(table-catalog): refine table catalog permissions (#3283)
* feat(table-catalog): refine table catalog permissions

* fix(policy): scope table admin resources

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-08 14:23:52 +00:00
唐小鸭 f7724d223b feat(rio): rio_v2 is compatible with minio for storing data. (#3115)
* Set up a compatibility layer for replacing old Rio components with new ones.

* fix(rio). compress range

* feat(rio). Add the experimental feature rio_v2 to support minio data at the binary level.

* feat(rio_v2): add sse-c test

* test compression component

* simple fix

* fix minlz encode

* fix metadata

* fix kms key cache error

* Update launch.json

* ci: set nix crate download user agent

* fix: gate obs pyroscope backend

* ignore minio test

* fix encrypt check

* fix

* fix

* fix

* Update object_usecase.rs

* Update ci.yml

* fix

* ci add rio-v2 test

* fix

* ci fix

* fix

* Reconstructed into a more reasonable compatibility mode

* fix

* fix

---------

Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-06-08 11:59:14 +00:00
cxymds 9504dff595 fix(ecstore): harden rebalance data movement (#3234)
* fix(ecstore): harden rebalance data movement

* fix(ecstore): preserve failed rebalance status

* fix(ecstore): avoid rebalance walkdir total timeout

* fix(ecstore): retry rebalance listing timeouts

* fix(ecstore): restrict data movement resume target

* refactor(ecstore): simplify multipart movement target

* fix(ecstore): restore resume target checks

* perf(ecstore): speed up rebalance bucket merges

* fix: keep rebalance listing alive on transient failures

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 10:40:26 +00:00
Michael 2fc1cf90e0 feat(helm): add option to disable log PVCs and mounts (#3189)
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-08 10:35:12 +00:00
houseme ed73952cb6 perf(ecstore): improve erasure write diagnostics and single-block performance (#3280)
* docs(object-capacity): add localized crate docs

* fix(ecstore): improve quorum and transport diagnostics

* perf(ecstore): add safe single-block write fast path

* refactor(ecstore): collapse layered small write paths

* chore(docs): keep issue 662 design note tracked

* fix(docs): restore issue 662 design note

* chore(docs): keep issue 662 design local only

* feat(obs): add internode reliability metrics and dashboard

* feat(obs): extend internode diagnostics and service logging

* fix(docs): use AGENTS guide filename

* perf(ecstore): reuse owned buffer in small encode

* fix(ecstore): tighten small write diagnostics

---------

Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-08 09:45:56 +00:00
cxymds ddd35badad fix(ecstore): avoid offline disks on admin timeout (#3263)
* fix(ecstore): avoid offline disks on admin timeout

* fix(ecstore): handle poisoned admin cache locks

* fix(ecstore): recover poisoned admin cache on success

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-06-08 08:35:36 +00:00
Henry Guo f40dd2f93c fix(scanner): publish partial usage for compacted scans (#3277)
* fix(scanner): publish partial usage for compacted scans

* fix(scanner): publish first partial usage immediately

* fix(scanner): skip startup delay for cold usage cache

* fix(scanner): tighten cold usage publish gate

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 06:18:35 +00:00
Henry Guo 4751a9f4b9 feat(table-catalog): add recovery diagnostics (#3275)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 03:59:13 +00:00
安正超 3d0e6ce0da feat: add security governance policy contracts (#3271)
* feat: add security governance policy contracts

* fix: require signatures for release assets

* docs: update policy verification counts

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 02:42:03 +00:00
安正超 78df276d25 fix: return 503 on lock contention instead of 500 (#3274)
* fix(ecstore): skip hidden metadata in walk limit

* fix(ecstore): match walk limit to visible versions

* fix(ecstore): avoid decoding all versions for limit

* fix: return 503 on lock contention instead of 500

When concurrent PUTs to the same key contend for the namespace write
lock, lock timeout and conflict errors were wrapped as
StorageError::other(...) → StorageError::Io(...), which fell through
to S3ErrorCode::InternalError (500) in the error mapping.

Only QuorumNotReached was correctly mapped to ServiceUnavailable (503);
all other lock errors (Timeout, AlreadyLocked, etc.) became 500.

Fix: map_namespace_lock_error now returns StorageError::Lock(err) for
non-quorum lock errors, and StorageError::Lock is mapped to
S3ErrorCode::ServiceUnavailable in the HTTP error conversion.

Affected paths:
- crates/ecstore/src/set_disk/lock.rs
- crates/ecstore/src/store/object.rs
- crates/ecstore/src/store/bucket.rs (make_bucket, delete_bucket)
- rustfs/src/app/object_usecase.rs (copy_object)

Regression test: test_concurrent_put_same_key_never_returns_500

* test: fail on unexpected lock contention errors

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-08 02:20:21 +00:00
houseme c479f3d0cb fix(ecstore): gate rustix fs diagnostics on windows (#3267)
* fix(ecstore): gate rustix fs diagnostics on windows

* fix(ecstore): improve windows disk validation diagnostics

* build(deps): bump workspace dependency versions
2026-06-08 00:05:55 +00:00
Henry Guo c452dd8ad7 feat(table-catalog): add metadata maintenance cleanup (#3266)
* feat(table-catalog): add metadata maintenance dry run

* feat(table-catalog): add metadata-only maintenance delete

* fix(table-catalog): protect recent metadata cleanup candidates

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-08 00:05:41 +00:00
安正超 7a9bf707ee feat: add security governance contract types (#3270) 2026-06-07 23:14:11 +00:00
安正超 c03c0ebc36 test: add admin route matrix guard (#3268) 2026-06-08 05:30:18 +08:00
安正超 dee550a831 ci: add architecture migration rule checks (#3264)
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 16:23:38 +00:00
Henry Guo f49df41db9 fix(lifecycle): harden scanner ILM expiry accounting (#3257)
* fix(lifecycle): harden scanner ILM expiry accounting

* fix(scanner): gate ILM action accounting on enqueue

* fix(metrics): avoid scanner source work argument list

* fix(scanner): gate local ILM accounting on enqueue

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 16:18:34 +00:00
houseme cebea9a238 fix(ecstore): add disk validation diagnostics (#3265)
Log local endpoint disk-topology details during physical disk independence checks.
Include canonical paths, st_dev major:minor values, and resolved device ids in validation output to make Docker and bind-mount startup failures easier to diagnose.

Also assert the new diagnostics are present in the shared-disk regression test.
2026-06-07 15:32:38 +00:00
唐小鸭 74296761fa fix(replication): repair site replication setup (#3252)
* fix: repair site replication setup

* Optimized site status collection

---------

Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-07 13:19:50 +00:00
安正超 6f4d0b54a1 fix(ci): install ripgrep for script checks (#3260) 2026-06-07 22:03:25 +08:00
安正超 4254633c9e docs: update security advisory lessons (#3262) 2026-06-07 22:02:56 +08:00
安正超 e5b2bcc088 docs: complete config helper inventory (#3261) 2026-06-07 22:02:21 +08:00
Henry Guo dd0ef1f766 feat(table-catalog): tighten REST load/register compatibility (#3245)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-07 21:54:15 +08:00
安正超 3c71d5ef1c docs: inventory ecstore config consumers (#3259) 2026-06-07 21:44:55 +08:00
安正超 241e45d3b0 docs: add admin route action snapshot (#3258) 2026-06-07 21:20:16 +08:00
安正超 0f9584c8d9 docs: add startup timeline baseline (#3256) 2026-06-07 20:18:30 +08:00
安正超 ae9d25879d ci: stabilize architecture layer guard (#3255) 2026-06-07 19:26:35 +08:00
Marcelo Bartsch f00898d070 fix(tier): recover (#3182)
* fix(tier): stop sending nil/garbage versionId to warm backend S3

Three bugs caused NoSuchVersion errors when reading tiered objects:

1. warm_backend_s3sdk: GET and DELETE ignored rv/range opts entirely —
   fixed to forward version_id and byte-range to the SDK request.

2. version.rs (MetaObject + MetaDeleteMarker): transition_version_id was
   parsed with unwrap_or_default(), turning invalid/wrong-length bytes
   into Uuid::nil(). The nil UUID was then serialized and sent as
   ?versionId=00000000-... to the tier backend -> NoSuchVersion.
   Fixed: .and_then(.ok()).filter(!is_nil()) so only valid non-nil UUIDs
   are forwarded as versionId.

3. bucket_lifecycle_ops: add debug/error logs in
   get_transitioned_object_reader to record tier, tier_object, and
   tier_version_id before and on failure of the tier GET.

Also adds tier transition fields to dump_fileinfo example for offline
xl.meta inspection, and fixes Docker build (cargo path + entrypoint).
Adds CLAUDE.md with tier architecture and debugging notes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* more fixes for versionId

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>

* remove branch

* Add tests and fix cargo path, add load to build-docker

* update documentation (CLAUDE.md)

* more fixes for recover

* More fixes to ILM recover

* final fix

* chore: add missing-shard first-scene diagnostics (#3213)

chore(ecstore): add missing-shard first-scene diagnostics

Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.

Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.

* fix test anmd fmt

* fix cargo path
fix test

* fix(tier): format copy_object self-copy guard

---------

Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-06-07 09:38:28 +00:00
安正超 069d1e5a75 test: bypass proxy for embedded readiness probe (#3254) 2026-06-07 09:04:29 +00:00
Henry Guo 4ae070c8cc feat(scanner): expose scan partial source status (#3247)
* feat(scanner): expose scan partial source status

* fix(scanner): expose partial source in aggregated metrics

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 06:40:04 +00:00
安正超 f8aa4fa221 docs: add architecture migration guardrails (#3253) 2026-06-07 14:43:55 +08:00
cxymds 61f0dfbc40 fix(ecstore): invalidate wiped disk id cache (#3251) 2026-06-07 04:25:38 +00:00
cxymds 04b5e8f988 fix(replication): normalize local site endpoint port (#3249)
* fix(replication): normalize local site endpoint port

* fix(replication): respect runtime console endpoint

* fix(replication): allow valid peer endpoint ports

---------

Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 04:19:04 +00:00
cxymds 316540aaf8 fix(docker): align TLS compose healthchecks (#3246)
* fix(docker): align TLS compose healthchecks

* fix(docker): keep simple compose TLS opt-in
2026-06-07 01:27:03 +00:00
cxymds dd5e2c8ae9 docs(docker): clarify bind mount permission setup (#3248)
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 09:33:24 +08:00
houseme e73b1c11c0 chore(deps): update flake.lock (#3250)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/e9a7635' (2026-05-29)
  → 'github:NixOS/nixpkgs/cbb5cf3' (2026-06-06)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/85570ef' (2026-05-30)
  → 'github:oxalica/rust-overlay/27b7e78' (2026-06-06)
2026-06-07 01:07:23 +00:00
GatewayJ 4a28e3d671 fix: clean old data dirs on object overwrite (#3244)
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-07 00:18:56 +00:00
GatewayJ 8c742ede14 fix(admin): format policy JSON and improve error messages in service … (#3242)
* fix(admin): format policy JSON and improve error messages in service account API

- Use serde_json::to_string_pretty for policy serialization to match MinIO behavior
- Replace technical error messages with user-friendly ones for policy validation
- Fixes rustfs/rustfs#3233 (policy JSON not formatted)
- Fixes rustfs/rustfs#3232 (error messages not shown to users)

* Delete scripts/tempfile.zip

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-06-06 16:39:59 +00:00
Henry Guo 7d84e89c36 feat(table-catalog): wire REST commit handler (#3239)
* feat(table-catalog): wire REST commit handler

* fix(table-catalog): hide MVP commit endpoint from config

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-06 13:38:12 +00:00
安正超 e2fe99d8bc refactor(ecstore): add server configuration accessors (#3238)
* refactor(ecstore): add server configuration accessors

Phase 4 of global singleton consolidation. Add ECStore accessor
methods for server-level configuration globals.

New methods:
- port() — delegates to global_rustfs_port()
- host() — delegates to GLOBAL_RUSTFS_HOST
- addr() — delegates to GLOBAL_RUSTFS_ADDR
- region() — delegates to get_global_region()
- endpoints() — delegates to get_global_endpoints()
- server_config() — delegates to get_global_server_config()
- storage_class() — delegates to get_global_storage_class()
- tier_config_mgr() — delegates to get_global_tier_config_mgr()
- notification_system() — delegates to get_global_notification_sys()
- bucket_metadata_sys() — delegates to get_global_bucket_metadata_sys()

1160 tests pass, clippy clean, formatting clean.

* fix(ecstore): avoid duplicate server accessors
2026-06-06 21:38:45 +08:00
Henry Guo 83a4e5712e feat(scanner): track scan cycle source work (#3240)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-06 12:53:48 +00:00
Henry Guo fade13d950 feat(table-catalog): wire REST catalog MVP handlers (#3227)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-06 10:47:49 +00:00