gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-21 03:46:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,572 Commits 21 Branches 125 Tags
7cc92ac93cd8b970cbb22b2322b8bf77f4065140
Commit Graph

194 Commits

Author SHA1 Message Date
houseme 6264be437c fix(storage): add scoped timeout policy and startup fs guardrail (#3056) 
* fix(storage): add RUSTFS_NETWORK_MOUNT_MODE for CIFS/NFS backends

* style: fix cargo fmt formatting in disk_store.rs

* fix(storage): add RUSTFS_NETWORK_MOUNT_MODE for CIFS/NFS backends

Extend the TimeoutHealthAction introduced in #2996 to read_metadata,
list_dir, and disk_info operations when RUSTFS_NETWORK_MOUNT_MODE=true.
Also raises all drive operation timeouts to 60s (explicit per-operation
overrides still take precedence).

Closes #2790

* feat(startup): add unsupported filesystem policy guardrail

* chore(deps): refresh lockfile and dependency pins

* feat(ecstore): add scoped timeout health-action policy

* docs(config): document drive timeout health-action policy

* refactor(ecstore): cache timeout health policy per disk wrapper

* fix(storage): add RUSTFS_NETWORK_MOUNT_MODE for CIFS/NFS backends (#2838)

* fix(storage): add RUSTFS_NETWORK_MOUNT_MODE for CIFS/NFS backends

* style: fix cargo fmt formatting in disk_store.rs

* fix(storage): add RUSTFS_NETWORK_MOUNT_MODE for CIFS/NFS backends

Extend the TimeoutHealthAction introduced in #2996 to read_metadata,
list_dir, and disk_info operations when RUSTFS_NETWORK_MOUNT_MODE=true.
Also raises all drive operation timeouts to 60s (explicit per-operation
overrides still take precedence).

Closes #2790

* fix(utils): map verified Linux filesystem magic values (#3051)

* fix(utils): cover sha256 checksum validation (#3052)

* fix(utils): cover sha256 checksum validation

* docs: clarify sha256 checksum validation

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>

* refactor(config): replace network mount mode with timeout profile preset

* fix(review): align fallback defaults and extend fs-type detection

* fix(review): cache timeout profile and restore probe timeout semantics

* refactor(ecstore): cache timeout health policy lookup

* perf(ecstore): cache active probe timeout per monitor task

---------

Co-authored-by: mistik <mistiklord4@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-05-22 06:37:30 +00:00
houseme e69e32285b fix(ecstore): harden multipart part metadata visibility (#3042) 
* fix(ecstore): harden multipart part metadata visibility

* fix(ecstore): address PR3042 review follow-ups
 2026-05-21 14:47:52 +00:00
Henry Guo 45b04316b9 refactor(ecstore): decouple transport construction from RemoteDisk (#3027) 
Move build_internode_data_transport_from_env() out of RemoteDisk::new()
into new_disk(), so RemoteDisk depends only on the
Arc<dyn InternodeDataTransport> trait object. TCP/HTTP remains the
default backend; all data paths unchanged.

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-05-20 11:55:08 +00:00
安正超 dd35538e90 fix(ecstore): remove stale disk TODOs (#3022) 2026-05-20 03:59:34 +00:00
安正超 a8a5f25af3 perf(ecstore): remove owned write sync regression (#3018) 2026-05-19 22:14:35 +00:00
houseme 25c6bdf490 perf(filemeta): phase-1~3 rename_data metadata optimization (#3011) 
* chore(perf): harden amd64 profiling benchmark flow

* fix(profiling): isolate bench buckets and map protobuf conflict

* perf: avoid blocking owned local writes

* style: format profile admin handler

* docs: clarify observability trace validation

* perf: reduce mkdir overhead on local writes

* perf: add rename_data meta microbenchmark

* perf(filemeta): fast-path data_dir decode in version meta

* perf(filemeta): collapse data-dir lookup into one scan

* perf(filemeta): reduce scan allocs and refresh meta bench

* perf(ecstore): skip mkdir path on read-only open

* perf(filemeta): single-pass unshared data-dir scan

* perf(filemeta): add two-key inline remove fast path

* perf(filemeta): compare remove-two keys by bytes first

* bench(ecstore): add remove_two-only micro benchmark

* bench(ecstore): stabilize rename_data meta benchmark timing

* bench(ecstore): align rename_data path with remove_two

* perf(filemeta): avoid uuid string alloc in remove_two

* perf(filemeta): add fast-path for empty inline data

* perf(filemeta): streamline add_version match branch

* perf(filemeta): fast-return remove_key on miss

* perf(filemeta): speed up add_version insertion lookup

* style(ecstore): normalize formatting in perf-tuning files

* refactor(filemeta): unify inline data removal paths
 2026-05-19 10:20:24 +00:00
安正超 a9e62dc2c2 perf: avoid blocking hop for owned disk writes (#3004) 
* perf: avoid blocking hop for owned disk writes

* fix(ecstore): sync owned write path
 2026-05-19 02:27:14 +00:00
weisd a66337bd28 fix: keep scanner walk timeouts from offlining drives (#2996) 
* fix: keep scanner walk timeouts from offlining drives

Scanner walk operations can time out on large or slow directory listings without proving the backing drive is faulty. Keep the timeout error local to the scan while preserving failure marking for ordinary disk operations.

Constraint: Scanner walk_dir can include listing work that exceeds the drive timeout under slow storage.

Rejected: Disable timeout failure marking globally | real stuck disk operations must still affect drive health

Confidence: high

Scope-risk: narrow

Directive: Do not route scanner/listing timeout back into drive offline state without reproducing issue #2651

Tested: cargo test -p rustfs-ecstore timeout

Tested: cargo fmt --all --check

Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings

Tested: make pre-commit with en_US.UTF-8 locale

Related: https://github.com/rustfs/rustfs/issues/2651

* fix: keep remote scanner walks from offlining drives

Remote walk_dir uses a streaming request that can hit total or stall timeouts during large scans without proving the remote drive is faulty. Route that scanner path through an explicit ignore action while preserving failure marking for ordinary remote operations.

Also treat Duration::ZERO as no operation timeout for remote health tracking, matching the existing local disk wrapper contract while still marking network-like errors on default paths.

Constraint: Scanner walk_dir streams can be slow because of directory size or consumer backpressure.

Rejected: Disable remote timeout failure marking globally | normal remote disk operations still need to evict failed connections and update drive health

Confidence: high

Scope-risk: narrow

Directive: Do not reintroduce remote scanner timeout failure marking without reproducing issue #2651 against remote disks.

Tested: cargo test -p rustfs-ecstore execute_with_timeout

Tested: cargo test -p rustfs-ecstore timeout

Tested: cargo fmt --all --check

Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings

Tested: make pre-commit with en_US.UTF-8 locale

Related: https://github.com/rustfs/rustfs/issues/2651

* test: prove scanner walk backpressure keeps drives online

Add a local scanner walk regression test where the output writer never makes progress. The test confirms the walk timeout returns without marking the drive faulty, covering a stall source that is not caused by object count or network transfer speed.

Constraint: Scanner walk can block on downstream writer backpressure as well as disk or network IO.\nConfidence: high\nScope-risk: narrow\nTested: cargo test -p rustfs-ecstore walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure\nTested: cargo test -p rustfs-ecstore timeout\nTested: cargo fmt --all --check\nTested: cargo clippy --workspace --all-features --all-targets -- -D warnings\nTested: make pre-commit

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-18 15:04:05 +00:00
wood 0e888cfef2 fix(ecstore): list_object_v2 error when scanning multipart folder (#2946) 
* fix(ecstore): list_object_v2 error when scanning multipart directory with prefix

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): cover prefix dir scan with multipart folder support

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): harden test shape to improve regression detection

Signed-off-by: w0od <dingboning02@163.com>

* refactor(ecstore): move multipart dir filter into recursion to reduce I/O

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): replace scan_dir test with walk_dir integration coverage

Signed-off-by: w0od <dingboning02@163.com>

* refactor(ecstore): remove unnecessary .clone() calls

Signed-off-by: w0od <dingboning02@163.com>

---------

Signed-off-by: w0od <dingboning02@163.com>
 2026-05-18 11:01:20 +00:00
Henry Guo bca8b08c2b fix: handle Windows paths in pre-commit tests (#2974) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-15 14:04:51 +00:00
Sergei Z. 8c49671161 Fix #2775 recursive list handling in LocalDisk::scan_dir() (#2923) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-05-13 14:36:23 +00:00
Henry Guo 26341742e0 fix(ecstore): fail listing on walk_dir producer errors (#2937) 2026-05-13 11:08:05 +00:00
weisd c5125b20bd fix(storage): keep storage info RPC map encoded (#2942) 
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-05-13 06:19:43 +00:00
weisd 9b83de53e1 fix(ecstore): surface prefix listing storage errors (#2940) 2026-05-13 04:24:54 +00:00
houseme c90bfe2b23 fix(ecstore): harden runtime read-path quorum handling (#2872) 2026-05-08 09:56:39 +00:00
安正超 e209fc6eef test(ecstore): cover store init health reset delegation (#2865) 2026-05-08 05:42:05 +00:00
weisd 97a2775434 fix(ecstore): reset drive health between store init format retries (#2848) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-05-08 03:16:58 +00:00
安正超 4728abcff1 fix(security): document unsafe and TLS overrides (#2835) 2026-05-06 15:09:02 +00:00
安正超 50ddec3ffc fix: preserve data on self metadata copy (#2819) 
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-05-06 06:28:36 +00:00
houseme 2953558f41 fix(lifecycle): prevent eager date-expiry deletion on config update (#2708) 2026-04-28 10:26:14 +00:00
weisd e0b8c4fd42 fix(storage): avoid faulting local drives on transient timeouts (#2714) 2026-04-28 06:38:31 +00:00
houseme 50d03ef021 perf(memory): add reclaim signals and cache controls (#2689) 2026-04-26 16:42:35 +00:00
houseme 81854762d4 fix(ecstore): log missing local paths during physical disk independence checks (#2680) 2026-04-25 08:01:01 +00:00
houseme 1e9c75a201 feat(ecstore): enforce local disk topology guardrails and expose device ids (#2679) 2026-04-25 06:02:02 +00:00
weisd f08b592c6f fix(storage): list prefix children behind marker objects (#2643) 2026-04-22 11:23:27 +00:00
houseme 28edfd6190 fix(storage): harden offline drive fail-fast paths (#2564) 
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2026-04-16 09:21:45 +00:00
Tunglies 579b124726 lint: clippy rules or_fun_call (#2561) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-16 02:48:39 +00:00
安正超 b2b92de26c test(ecstore): cover fresh tmp cleanup path (#2562) 2026-04-16 02:12:16 +00:00
Tunglies 49366ee200 chore(lint): clippy rules redundant_clone (#2554) 2026-04-15 13:54:07 +00:00
weisd da9dfb51e7 fix(ecstore): cleanup stale put object temp data (#2529) 2026-04-15 14:40:58 +08:00
安正超 6d476ae9a1 revert: remove BlockReadable trait and chunk-based I/O from rio, ecstore, disk layers (#2533) 2026-04-14 19:09:02 +08:00
weisd a8a2aaa460 fix(ecstore): avoid duplicate keys in ListObjectsV2 (#2467) 2026-04-10 11:29:23 +08:00
houseme 32bf8f5bf3 feat(storage): add direct chunk GET fast path (#2351) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-04-07 08:33:46 +08:00
安正超 d5f05993a3 test(ecstore): cover read offset overflow (#2341) 2026-03-30 20:48:03 +08:00
houseme 7172e151de fix: address correctness, safety, and concurrency issues (#2327) 
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-30 00:30:57 +08:00
weisd 05dc131a49 perf(storage): optimize internode RPC transfer path (#2262) 
Co-authored-by: momoda693 <momoda693@gmail.com>
 2026-03-23 17:09:49 +08:00
weisd 2e7abfbd63 fix(ecstore): preserve raw metadata read semantics (#2258) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-03-23 10:42:16 +08:00
weisd a42320848c fix(ecstore): invalidate xl.meta cache after writes and rename (#2255) 2026-03-22 11:30:29 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
evan slack d52a10c5fb chore(obs): Improve tracing instrumentation (#2086) 
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-03-07 20:03:20 +08:00
安正超 9d2b8822cf refactor: stabilize heal format recovery integration tests (#1984) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-02-27 15:26:19 +08:00
weisd 368bba3345 fix(ecstore): adjust unformatted disk error mapping (#1988) 2026-02-27 13:45:33 +08:00
安正超 4211652991 refactor(app): add application layer module entry (#1907) 2026-02-22 22:15:37 +08:00
evan slack 23f7ffe36b fix(startup): Only monitor disk health after format loading (#1854) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-02-21 20:46:14 +08:00
loverustfs 5d737eaeb7 fix(ecstore): invalidate GlobalFileCache after write_all_private to fix DeleteMarker visibility (#1890) 2026-02-20 22:47:45 +08:00
安正超 2fc36bb52e fix: restore s3 compatibility regressions and CI coverage (#1793) 2026-02-13 12:26:52 +08:00
Tyooughtul 1184806c3f Fix/resolve pr 1710 (#1743) 2026-02-11 08:24:55 +08:00
weisd 4d19b069c3 fix: replication delete (#1714) 2026-02-04 13:39:35 +08:00
weisd dce117840c refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes (#1664) 
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: weisd <2057561+weisd@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-01-30 13:13:41 +08:00
weisd 6ab7b75fd4 fix bug (#1615) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-01-27 09:54:13 +08:00