Commit Graph

3026 Commits

Author SHA1 Message Date
安正超 a9e62dc2c2 perf: avoid blocking hop for owned disk writes (#3004)
* perf: avoid blocking hop for owned disk writes

* fix(ecstore): sync owned write path
2026-05-19 02:27:14 +00:00
yihong ecb6704679 fix: make help color not right (#3005)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2026-05-19 02:26:23 +00:00
安正超 e65aebba53 fix(ecstore): guard transition worker max (#3003)
* fix(ecstore): guard transition worker max

* test: clarify env mutation safety

* test(ecstore): document env mutation concurrency contract
2026-05-19 02:25:51 +00:00
weisd a66337bd28 fix: keep scanner walk timeouts from offlining drives (#2996)
* fix: keep scanner walk timeouts from offlining drives

Scanner walk operations can time out on large or slow directory listings without proving the backing drive is faulty. Keep the timeout error local to the scan while preserving failure marking for ordinary disk operations.

Constraint: Scanner walk_dir can include listing work that exceeds the drive timeout under slow storage.

Rejected: Disable timeout failure marking globally | real stuck disk operations must still affect drive health

Confidence: high

Scope-risk: narrow

Directive: Do not route scanner/listing timeout back into drive offline state without reproducing issue #2651

Tested: cargo test -p rustfs-ecstore timeout

Tested: cargo fmt --all --check

Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings

Tested: make pre-commit with en_US.UTF-8 locale

Related: https://github.com/rustfs/rustfs/issues/2651

* fix: keep remote scanner walks from offlining drives

Remote walk_dir uses a streaming request that can hit total or stall timeouts during large scans without proving the remote drive is faulty. Route that scanner path through an explicit ignore action while preserving failure marking for ordinary remote operations.

Also treat Duration::ZERO as no operation timeout for remote health tracking, matching the existing local disk wrapper contract while still marking network-like errors on default paths.

Constraint: Scanner walk_dir streams can be slow because of directory size or consumer backpressure.

Rejected: Disable remote timeout failure marking globally | normal remote disk operations still need to evict failed connections and update drive health

Confidence: high

Scope-risk: narrow

Directive: Do not reintroduce remote scanner timeout failure marking without reproducing issue #2651 against remote disks.

Tested: cargo test -p rustfs-ecstore execute_with_timeout

Tested: cargo test -p rustfs-ecstore timeout

Tested: cargo fmt --all --check

Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings

Tested: make pre-commit with en_US.UTF-8 locale

Related: https://github.com/rustfs/rustfs/issues/2651

* test: prove scanner walk backpressure keeps drives online

Add a local scanner walk regression test where the output writer never makes progress. The test confirms the walk timeout returns without marking the drive faulty, covering a stall source that is not caused by object count or network transfer speed.

Constraint: Scanner walk can block on downstream writer backpressure as well as disk or network IO.\nConfidence: high\nScope-risk: narrow\nTested: cargo test -p rustfs-ecstore walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure\nTested: cargo test -p rustfs-ecstore timeout\nTested: cargo fmt --all --check\nTested: cargo clippy --workspace --all-features --all-targets -- -D warnings\nTested: make pre-commit

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-05-18 15:04:05 +00:00
houseme bd1e57293f fix: harden lifecycle transition compensation and regression coverage (#2995)
* fix(ecstore): honor transition worker configuration

* fix(ecstore): add transition queue backpressure metrics

* fix(ecstore): schedule transition compensation on enqueue pressure

* fix(ecstore): log transition compensation scheduling

* test(rustfs): add transition compensation fault-injection coverage

* test(rustfs): cover delete after transition compensation

* test(scanner): cover cleanup after transition compensation

* test(rustfs): extend compensation transition coverage

* test(scanner): cover backfill idempotency after compensation

* test(scanner): cover noncurrent expiry after compensation

* test(rustfs): cover versioned delete after compensation

* test(rustfs): cover delete marker lifecycle after compensation

* test(scanner): extend versioned lifecycle compensation coverage

* test(scanner): model versioned delete after compensation

* test(scanner): clarify modeled versioned delete helper

* refactor(ecstore): optimize transition enqueue hot path

* refactor(ecstore): centralize transition runtime constants

* style(ecstore): apply rustfmt for transition timeout helper

* fix(ilm): align queue-full metric semantics

* refactor(ecstore): unify immediate enqueue failure handling

* refactor(ecstore): reuse transition worker env constant

* ci(actions): update setup action inputs
2026-05-18 12:50:43 +00:00
wood 0e888cfef2 fix(ecstore): list_object_v2 error when scanning multipart folder (#2946)
* fix(ecstore): list_object_v2 error when scanning multipart directory with prefix

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): cover prefix dir scan with multipart folder support

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): harden test shape to improve regression detection

Signed-off-by: w0od <dingboning02@163.com>

* refactor(ecstore): move multipart dir filter into recursion to reduce I/O

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): replace scan_dir test with walk_dir integration coverage

Signed-off-by: w0od <dingboning02@163.com>

* refactor(ecstore): remove unnecessary .clone() calls

Signed-off-by: w0od <dingboning02@163.com>

---------

Signed-off-by: w0od <dingboning02@163.com>
2026-05-18 11:01:20 +00:00
安正超 4c9fd789ea docs: update security advisory skill lessons (#2991) 2026-05-18 12:34:27 +08:00
Henry Guo 17ae9f34c7 fix(notify): accept case-insensitive filter rule names (#2990) 2026-05-18 03:41:48 +00:00
houseme cdfe83877b chore(deps): update flake.lock (#2986)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/b3da656' (2026-05-08)
  → 'github:NixOS/nixpkgs/d233902' (2026-05-15)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/1d634a9' (2026-05-09)
  → 'github:oxalica/rust-overlay/61ec6a4' (2026-05-16)
2026-05-17 03:09:08 +00:00
安正超 be37fd285e ci(nix): avoid requesting review from PR author (#2987) 2026-05-17 11:13:45 +08:00
安正超 095337100e test(policy): validate default policies (#2985) 2026-05-17 01:41:03 +00:00
GatewayJ e0729f5f4d fix(policy): align action-family validation and defaults (#2984)
* fix(policy): align action-family validation and defaults

* test(e2e): add accountinfo service-account roundtrip

* test(policy): add mixed action family cases
2026-05-16 11:19:04 +00:00
houseme 6e12289339 fix(runtime): finalize issue 2941 profiling cleanup (#2983)
* perf(runtime): narrow profiling support and upgrade starshard

* style(notify): normalize starshard imports

* perf(ecstore): reduce list_path_raw coordination overhead

* docs(scripts): add issue 2941 perf capture workflow

* fix(runtime): finalize issue 2941 profiling cleanup

* build(deps): bump quick-xml to 0.40.0

* chore(scripts): untrack local perf capture guide

* fix(scripts): honor label in perf capture output
2026-05-16 11:09:04 +00:00
weisd 9dcf8cb7ea fix: stabilize rebalance start and listing (#2961)
* fix: unblock empty chunked admin rebalance start

Some admin clients send empty POST requests with chunked framing and no explicit content length. The admin rebalance route carries no request body, so normalize that known route before downstream validation sees a missing length.

Constraint: Keep normalization scoped to known empty-body admin routes

Rejected: Relax transfer-encoding handling for S3 routes | broader protocol risk

Confidence: high

Scope-risk: narrow

Tested: cargo test -p rustfs --lib server::layer::tests::

Tested: cargo fmt --all --check

Tested: git diff --check

* fix: keep rebalance listing from stalling

Large object migration was awaited inside the listing callback, so metacache readers could stop being drained while drive walk operations were still writing listing data. Move entry migration into bounded background tasks and wait for them after each set listing completes.

Constraint: Preserve existing rebalance cancellation and first-error propagation

Rejected: Only increase drive walk timeouts | masks callback backpressure and still fails for slower migrations

Confidence: medium

Scope-risk: moderate

Tested: cargo test -p rustfs-ecstore rebalance_unit_tests

Tested: cargo test -p rustfs-ecstore cache_value::metacache_set::tests

Tested: cargo test -p rustfs --lib server::layer::tests::

Tested: cargo clippy -p rustfs-ecstore --lib --all-features -- -D warnings

Tested: cargo clippy -p rustfs-ecstore --tests --all-features -- -D warnings

Tested: cargo fmt --all --check

Tested: git diff --check

---------

Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-16 05:17:01 +00:00
安正超 6898e720dd fix(security): harden proxy auth and default credentials (#2981)
* fix(security): harden proxy auth and default credentials

* fix(security): address proxy and credential feedback
2026-05-16 04:01:50 +00:00
yihong 824c4f7673 docs: fix some dead links (#2975)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-16 10:06:53 +08:00
安正超 c0c92cb048 ci(build): honor console asset download fallback (#2980) 2026-05-16 10:06:23 +08:00
Henry Guo bca8b08c2b fix: handle Windows paths in pre-commit tests (#2974)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-15 14:04:51 +00:00
Henry Guo 738fb86611 fix(admin): surface access key policy errors (#2970)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-05-15 08:26:38 +00:00
安正超 092c6bc135 ci(build): pin macOS x86 release runner (#2971) 2026-05-15 11:56:13 +08:00
escapecode 5cda460451 fix(sftp): preserve OPEN-time client attrs as object metadata (#2913)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-15 00:32:06 +00:00
GatewayJ fc8322ed64 bucket policy notify & pba (#2968) 2026-05-14 14:31:40 +00:00
houseme cd2cd74314 ci: force Node24 in Nix workflows with pinned actions (#2966) 2026-05-14 11:13:42 +00:00
Henry Guo ced390b02b test(ecstore): cover managed sse range reads (#2962)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-14 06:56:21 +00:00
Henry Guo 11be278c40 test(notify): cover prefix suffix event dispatch (#2960) 1.0.0-beta.3 2026-05-14 05:38:38 +00:00
Henry Guo 0893c05540 fix(ecstore): use hex sha256 for delete objects (#2958)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-14 05:34:26 +00:00
houseme 81754d80b3 refactor(targets): unify queue/connectivity handling and coverage (#2953)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: marshawcoco <marshawcoco@gmail.com>
2026-05-14 04:31:23 +00:00
houseme bdb98598d2 chore(release): prepare 1.0.0-beta.3 (#2957) 2026-05-14 04:27:26 +00:00
安正超 5fccfceb91 test(ecstore): cover walk listing error success paths (#2954) 2026-05-14 01:03:31 +00:00
Henry Guo c1bcee327c fix(notify): keep live listen events active when disabled (#2952)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 17:10:14 +00:00
Henry Guo d4d07095f8 fix(ecstore): propagate walk listing errors (#2949)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 16:59:49 +00:00
Sergei Z. 8c49671161 Fix #2775 recursive list handling in LocalDisk::scan_dir() (#2923)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-13 14:36:23 +00:00
cxymds c65d4071a3 fix(ecstore): map missing metadata to not found (#2944)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 13:03:15 +00:00
weisd 4ea805cbc0 fix: preserve pagination when max keys exceed limit (#2943)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 12:48:38 +00:00
Henry Guo 26341742e0 fix(ecstore): fail listing on walk_dir producer errors (#2937) 2026-05-13 11:08:05 +00:00
安正超 fd3fb77ed5 fix(sftp): avoid metadata on multipart copy (#2935)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 07:44:09 +00:00
weisd c5125b20bd fix(storage): keep storage info RPC map encoded (#2942)
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-13 06:19:43 +00:00
weisd 9b83de53e1 fix(ecstore): surface prefix listing storage errors (#2940) 2026-05-13 04:24:54 +00:00
houseme d72b8cb88e fix(server): fail fast when configured TLS parsing fails (#2939) 2026-05-13 04:16:39 +00:00
Henry Guo 17eff5a97b fix(tls): ignore Kubernetes secret projection dirs (#2938) 2026-05-13 03:45:42 +00:00
houseme 129cb0f920 fix: make HeadObject consistent after write completion (#2936) 2026-05-13 03:27:38 +00:00
安正超 28c9358482 test(notify): cover encoded key target union (#2934)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-12 15:20:06 +00:00
Henry Guo fe5690ca70 fix(sftp): preserve open attrs metadata (#2929)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-12 13:34:52 +00:00
安正超 25cf164461 test(protocols): cover SFTP host key reload failure (#2928)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-12 10:40:23 +00:00
安正超 8a501846f4 test(iam): cover mixed STS claim policy names (#2932) 2026-05-12 10:39:53 +00:00
GatewayJ b2ba2e5bb3 iam: handle sts claim policy names (#2902)
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-12 07:10:42 +00:00
安正超 8169bd63e9 test(protocols): cover TLS reload fingerprint ordering (#2927) 2026-05-12 02:50:49 +00:00
houseme 6185d38911 fix(protocols): add hot reload for WebDAV FTPS and SFTP (#2922)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-11 16:25:20 +00:00
cxymds 8ef2c0a5e1 fix(storage): sync transition tier config across peers (#2918)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-11 15:23:29 +00:00
Henry Guo d6813b53a2 fix(ecstore): preserve list marker set index (#2919)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-11 15:20:43 +00:00