Commit Graph

356 Commits

Author SHA1 Message Date
houseme 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012)
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
2026-05-19 12:50:25 +00:00
houseme 25c6bdf490 perf(filemeta): phase-1~3 rename_data metadata optimization (#3011)
* chore(perf): harden amd64 profiling benchmark flow

* fix(profiling): isolate bench buckets and map protobuf conflict

* perf: avoid blocking owned local writes

* style: format profile admin handler

* docs: clarify observability trace validation

* perf: reduce mkdir overhead on local writes

* perf: add rename_data meta microbenchmark

* perf(filemeta): fast-path data_dir decode in version meta

* perf(filemeta): collapse data-dir lookup into one scan

* perf(filemeta): reduce scan allocs and refresh meta bench

* perf(ecstore): skip mkdir path on read-only open

* perf(filemeta): single-pass unshared data-dir scan

* perf(filemeta): add two-key inline remove fast path

* perf(filemeta): compare remove-two keys by bytes first

* bench(ecstore): add remove_two-only micro benchmark

* bench(ecstore): stabilize rename_data meta benchmark timing

* bench(ecstore): align rename_data path with remove_two

* perf(filemeta): avoid uuid string alloc in remove_two

* perf(filemeta): add fast-path for empty inline data

* perf(filemeta): streamline add_version match branch

* perf(filemeta): fast-return remove_key on miss

* perf(filemeta): speed up add_version insertion lookup

* style(ecstore): normalize formatting in perf-tuning files

* refactor(filemeta): unify inline data removal paths
2026-05-19 10:20:24 +00:00
Henry Guo f695870626 feat(internode): add transport observability (#3007)
* docs: add internode data transport RFC

* feat: add internode operation metrics

* fix feedback

* fix(ci): fallback protoc token to github.token

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-05-19 07:16:05 +00:00
安正超 aa3f13c0d3 fix(ecstore): stop listing after reaching result limit (#3001)
* fix(ecstore): stop listing after reaching result limit

* fix(ecstore): ignore intentional list cancellation

* fix(ecstore): satisfy list cancellation clippy lint

* fix(ecstore): include entry before limit early return

* fix(ecstore): cancel on limit in gather_results

* fix(ecstore): sync timeout branch with reviewed patch

* fix(ecstore): satisfy cancellation clippy lint

---------

Co-authored-by: loverustfs <hello@rustfs.com>
2026-05-19 05:20:21 +00:00
安正超 a9e62dc2c2 perf: avoid blocking hop for owned disk writes (#3004)
* perf: avoid blocking hop for owned disk writes

* fix(ecstore): sync owned write path
2026-05-19 02:27:14 +00:00
安正超 e65aebba53 fix(ecstore): guard transition worker max (#3003)
* fix(ecstore): guard transition worker max

* test: clarify env mutation safety

* test(ecstore): document env mutation concurrency contract
2026-05-19 02:25:51 +00:00
weisd a66337bd28 fix: keep scanner walk timeouts from offlining drives (#2996)
* fix: keep scanner walk timeouts from offlining drives

Scanner walk operations can time out on large or slow directory listings without proving the backing drive is faulty. Keep the timeout error local to the scan while preserving failure marking for ordinary disk operations.

Constraint: Scanner walk_dir can include listing work that exceeds the drive timeout under slow storage.

Rejected: Disable timeout failure marking globally | real stuck disk operations must still affect drive health

Confidence: high

Scope-risk: narrow

Directive: Do not route scanner/listing timeout back into drive offline state without reproducing issue #2651

Tested: cargo test -p rustfs-ecstore timeout

Tested: cargo fmt --all --check

Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings

Tested: make pre-commit with en_US.UTF-8 locale

Related: https://github.com/rustfs/rustfs/issues/2651

* fix: keep remote scanner walks from offlining drives

Remote walk_dir uses a streaming request that can hit total or stall timeouts during large scans without proving the remote drive is faulty. Route that scanner path through an explicit ignore action while preserving failure marking for ordinary remote operations.

Also treat Duration::ZERO as no operation timeout for remote health tracking, matching the existing local disk wrapper contract while still marking network-like errors on default paths.

Constraint: Scanner walk_dir streams can be slow because of directory size or consumer backpressure.

Rejected: Disable remote timeout failure marking globally | normal remote disk operations still need to evict failed connections and update drive health

Confidence: high

Scope-risk: narrow

Directive: Do not reintroduce remote scanner timeout failure marking without reproducing issue #2651 against remote disks.

Tested: cargo test -p rustfs-ecstore execute_with_timeout

Tested: cargo test -p rustfs-ecstore timeout

Tested: cargo fmt --all --check

Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings

Tested: make pre-commit with en_US.UTF-8 locale

Related: https://github.com/rustfs/rustfs/issues/2651

* test: prove scanner walk backpressure keeps drives online

Add a local scanner walk regression test where the output writer never makes progress. The test confirms the walk timeout returns without marking the drive faulty, covering a stall source that is not caused by object count or network transfer speed.

Constraint: Scanner walk can block on downstream writer backpressure as well as disk or network IO.\nConfidence: high\nScope-risk: narrow\nTested: cargo test -p rustfs-ecstore walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure\nTested: cargo test -p rustfs-ecstore timeout\nTested: cargo fmt --all --check\nTested: cargo clippy --workspace --all-features --all-targets -- -D warnings\nTested: make pre-commit

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-05-18 15:04:05 +00:00
houseme bd1e57293f fix: harden lifecycle transition compensation and regression coverage (#2995)
* fix(ecstore): honor transition worker configuration

* fix(ecstore): add transition queue backpressure metrics

* fix(ecstore): schedule transition compensation on enqueue pressure

* fix(ecstore): log transition compensation scheduling

* test(rustfs): add transition compensation fault-injection coverage

* test(rustfs): cover delete after transition compensation

* test(scanner): cover cleanup after transition compensation

* test(rustfs): extend compensation transition coverage

* test(scanner): cover backfill idempotency after compensation

* test(scanner): cover noncurrent expiry after compensation

* test(rustfs): cover versioned delete after compensation

* test(rustfs): cover delete marker lifecycle after compensation

* test(scanner): extend versioned lifecycle compensation coverage

* test(scanner): model versioned delete after compensation

* test(scanner): clarify modeled versioned delete helper

* refactor(ecstore): optimize transition enqueue hot path

* refactor(ecstore): centralize transition runtime constants

* style(ecstore): apply rustfmt for transition timeout helper

* fix(ilm): align queue-full metric semantics

* refactor(ecstore): unify immediate enqueue failure handling

* refactor(ecstore): reuse transition worker env constant

* ci(actions): update setup action inputs
2026-05-18 12:50:43 +00:00
wood 0e888cfef2 fix(ecstore): list_object_v2 error when scanning multipart folder (#2946)
* fix(ecstore): list_object_v2 error when scanning multipart directory with prefix

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): cover prefix dir scan with multipart folder support

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): harden test shape to improve regression detection

Signed-off-by: w0od <dingboning02@163.com>

* refactor(ecstore): move multipart dir filter into recursion to reduce I/O

Signed-off-by: w0od <dingboning02@163.com>

* test(ecstore): replace scan_dir test with walk_dir integration coverage

Signed-off-by: w0od <dingboning02@163.com>

* refactor(ecstore): remove unnecessary .clone() calls

Signed-off-by: w0od <dingboning02@163.com>

---------

Signed-off-by: w0od <dingboning02@163.com>
2026-05-18 11:01:20 +00:00
houseme 6e12289339 fix(runtime): finalize issue 2941 profiling cleanup (#2983)
* perf(runtime): narrow profiling support and upgrade starshard

* style(notify): normalize starshard imports

* perf(ecstore): reduce list_path_raw coordination overhead

* docs(scripts): add issue 2941 perf capture workflow

* fix(runtime): finalize issue 2941 profiling cleanup

* build(deps): bump quick-xml to 0.40.0

* chore(scripts): untrack local perf capture guide

* fix(scripts): honor label in perf capture output
2026-05-16 11:09:04 +00:00
weisd 9dcf8cb7ea fix: stabilize rebalance start and listing (#2961)
* fix: unblock empty chunked admin rebalance start

Some admin clients send empty POST requests with chunked framing and no explicit content length. The admin rebalance route carries no request body, so normalize that known route before downstream validation sees a missing length.

Constraint: Keep normalization scoped to known empty-body admin routes

Rejected: Relax transfer-encoding handling for S3 routes | broader protocol risk

Confidence: high

Scope-risk: narrow

Tested: cargo test -p rustfs --lib server::layer::tests::

Tested: cargo fmt --all --check

Tested: git diff --check

* fix: keep rebalance listing from stalling

Large object migration was awaited inside the listing callback, so metacache readers could stop being drained while drive walk operations were still writing listing data. Move entry migration into bounded background tasks and wait for them after each set listing completes.

Constraint: Preserve existing rebalance cancellation and first-error propagation

Rejected: Only increase drive walk timeouts | masks callback backpressure and still fails for slower migrations

Confidence: medium

Scope-risk: moderate

Tested: cargo test -p rustfs-ecstore rebalance_unit_tests

Tested: cargo test -p rustfs-ecstore cache_value::metacache_set::tests

Tested: cargo test -p rustfs --lib server::layer::tests::

Tested: cargo clippy -p rustfs-ecstore --lib --all-features -- -D warnings

Tested: cargo clippy -p rustfs-ecstore --tests --all-features -- -D warnings

Tested: cargo fmt --all --check

Tested: git diff --check

---------

Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-16 05:17:01 +00:00
Henry Guo bca8b08c2b fix: handle Windows paths in pre-commit tests (#2974)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-15 14:04:51 +00:00
Henry Guo ced390b02b test(ecstore): cover managed sse range reads (#2962)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-14 06:56:21 +00:00
Henry Guo 0893c05540 fix(ecstore): use hex sha256 for delete objects (#2958)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-14 05:34:26 +00:00
安正超 5fccfceb91 test(ecstore): cover walk listing error success paths (#2954) 2026-05-14 01:03:31 +00:00
Henry Guo d4d07095f8 fix(ecstore): propagate walk listing errors (#2949)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 16:59:49 +00:00
Sergei Z. 8c49671161 Fix #2775 recursive list handling in LocalDisk::scan_dir() (#2923)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-13 14:36:23 +00:00
cxymds c65d4071a3 fix(ecstore): map missing metadata to not found (#2944)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 13:03:15 +00:00
weisd 4ea805cbc0 fix: preserve pagination when max keys exceed limit (#2943)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-13 12:48:38 +00:00
Henry Guo 26341742e0 fix(ecstore): fail listing on walk_dir producer errors (#2937) 2026-05-13 11:08:05 +00:00
weisd c5125b20bd fix(storage): keep storage info RPC map encoded (#2942)
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-13 06:19:43 +00:00
weisd 9b83de53e1 fix(ecstore): surface prefix listing storage errors (#2940) 2026-05-13 04:24:54 +00:00
houseme 129cb0f920 fix: make HeadObject consistent after write completion (#2936) 2026-05-13 03:27:38 +00:00
cxymds 8ef2c0a5e1 fix(storage): sync transition tier config across peers (#2918)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-11 15:23:29 +00:00
Henry Guo d6813b53a2 fix(ecstore): preserve list marker set index (#2919)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-11 15:20:43 +00:00
Henry Guo 5ca99a8e32 fix(ecstore): fail listing on stalled reader (#2920)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-11 15:19:46 +00:00
唐小鸭 09c2d15057 fix(sse): Temporarily refactored the SSE design for ECStore (#2813)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-09 14:06:35 +00:00
安正超 8182e2dd38 test(ecstore): cover empty runtime listing candidates (#2889) 2026-05-09 01:57:54 +00:00
houseme 81ad48dac2 feat(targets): add AMQP support for notify and audit (#2879)
Co-authored-by: Hyesook Yun <74169420+suk13574@users.noreply.github.com>
2026-05-09 01:56:26 +00:00
安正超 1582f216fe test(ecstore): cover offline capacity snapshots (#2880) 2026-05-08 16:33:34 +00:00
安正超 5081b8396d test(ecstore): cover system path failure classifier (#2874) 2026-05-08 14:51:11 +00:00
houseme c90bfe2b23 fix(ecstore): harden runtime read-path quorum handling (#2872) 2026-05-08 09:56:39 +00:00
安正超 e209fc6eef test(ecstore): cover store init health reset delegation (#2865) 2026-05-08 05:42:05 +00:00
weisd 97a2775434 fix(ecstore): reset drive health between store init format retries (#2848)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-08 03:16:58 +00:00
cxymds 2a0fbb8d77 fix(ecstore): repair decommission pool quorum (#2847)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-07 11:14:11 +00:00
houseme 5431b9273d feat(targets): complete redis mysql postgres target wiring (#2842)
Signed-off-by: jaehanbyun <awbrg789@naver.com>
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Co-authored-by: jaehanbyun <awbrg789@naver.com>
Co-authored-by: Gunther Xing <jiengup@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: JaySon <tshent@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-07 10:00:59 +00:00
cxymds b7815b0694 fix(ecstore): remove startup order sensitivity (#2850) 2026-05-07 09:07:25 +00:00
安正超 5e0ca006f0 test(replication): cover ETag comparison edge cases (#2840) 2026-05-07 02:48:12 +00:00
Nikita Bakun 4d6171e996 fix(replication): handle version ID format mismatch with AWS S3 (#2829)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-06 21:30:07 +00:00
安正超 9b9e0db9f6 test(lifecycle): cover ILM process time aliases (#2839) 2026-05-06 21:12:22 +00:00
安正超 4728abcff1 fix(security): document unsafe and TLS overrides (#2835) 2026-05-06 15:09:02 +00:00
cxymds 70be0804ee fix: 2827 lifecycle days next midnight (#2833)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-06 15:00:53 +00:00
cxymds 9f07029373 fix: reload bucket metadata after lifecycle updates (#2822)
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-06 13:01:44 +00:00
安正超 50ddec3ffc fix: preserve data on self metadata copy (#2819)
Co-authored-by: cxymds <Cxymds@qq.com>
2026-05-06 06:28:36 +00:00
houseme 3dd0692917 refactor: unify credential env constants and deploy env usage (#2821)
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-06 04:30:29 +00:00
Alexander Kharkevich 995e26f5ee fix(rpc): use map-encoded msgpack for all internode RPC responses (#2771)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-05 03:42:29 +00:00
安正超 eb8868397e fix: address security review follow-ups (#2781) 2026-05-03 13:53:36 +00:00
安正超 66c38b629d Harden admin and RPC security checks (#2773)
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-03 11:55:09 +00:00
Ramakrishna Chilaka 4978f60254 test(ecstore): cover ranged decode and harden offset bounds (#2758)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-05-02 09:02:50 +00:00
安正超 a455b4377c fix: handle empty multipart list-parts (#2765)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-05-02 02:47:52 +00:00