Commit Graph

25 Commits

Author SHA1 Message Date
安正超 54ffbedbc8 refactor: remove ecstore operation compat facades (#3608) 2026-06-19 17:01:12 +08:00
安正超 c28fee0013 refactor: continue storage api contract cleanup (#3580)
* refactor: move delete object contracts to storage api

* refactor: narrow store api compatibility exports

* refactor: route table catalog test through storage compat
2026-06-18 22:42:02 +08:00
安正超 fbab160c2b refactor: narrow replication storage consumers (#3485) 2026-06-15 22:29:38 +08:00
abdullahnah92 a19560da41 fix(bucket-repl): persist MRF retry queue to disk and reload on startup (#3456)
* fix(bucket-repl): persist MRF retry queue to disk and reload on startup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(bucket-repl): address three blocking MRF issues from review

1. MRF replay loses delete operations — add `MrfOpKind` discriminator to
   `MrfReplicateEntry` (Object | Delete, default=Object for backward
   compat).  `DeletedObjectReplicationInfo::to_mrf_entry` now persists
   `op=Delete`, `version_id`, `delete_marker_version_id`, and
   `delete_marker`.  `start_mrf_processor` branches on `op`: delete
   entries skip `get_object_info` and replay via
   `schedule_replication_delete` with `ReplicationType::Heal`; object
   entries follow the existing heal path.

2. `flush_mrf_to_disk` cleared the in-memory batch even on encode/write
   failure — changed return type to `bool` and callers now only
   `pending.clear()` on `true`, so a transient storage error retries
   on the next tick instead of silently dropping the batch.

3. Add focused tests: encode/decode roundtrips for object, delete-marker,
   versioned-delete, and mixed-batch entries; a routing test confirming
   op-kind propagates correctly and that the default is Object for
   legacy files; a legacy-compat test verifying old entries round-trip
   cleanly through the new format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: fix clippy redundant-clone in MRF tests

Replace &[entry.clone()] with std::slice::from_ref(&entry) in two
encode_mrf_file call sites flagged by clippy's redundant_clone lint
under --all-targets --features rio-v2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(bucket-repl): strengthen legacy MRF compat test with hand-built msgpack

The previous mrf_legacy_file_without_op_field_decoded_as_object test
round-tripped through encode_mrf_file, so it exercised the new format
and never touched a truly-legacy payload.

Replace it with a hand-built msgpack payload that genuinely omits the
"op", "deleteMarker", and "deleteMarkerVersionID" keys — exactly what
the old binary would have written before MrfOpKind existed.  The test
now fails if #[serde(default)] is removed from the op field, which
proves real backward compatibility rather than round-trip stability.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-15 11:56:50 +08:00
houseme efaf07d323 feat: preserve request ids across async recovery logs (#3451)
* feat(obs): promote request ids in structured logs

* refactor(tracing): propagate spans into request tasks

* test(ecstore): baseline recovery monitor log chains

* fix(replication): reduce startup resync log noise

* chore(docs): stop tracking local recovery baseline

* chore(obs): polish request id logging cleanup
2026-06-14 23:16:04 +08:00
abdullahnah92 22460243bf fix(bucket-repl): honor op_type in replicate_object so ExistingObject… (#3420)
fix(bucket-repl): honor op_type in replicate_object so ExistingObject resync respects DISABLED targets

replicate_object was calling filter_target_arns with hard-coded
op_type: Object and existing_object: false regardless of what was
stored in roi.op_type. This meant that a resync worker setting
roi.op_type = ExistingObject (resync_bucket, line 889) had no effect
on target filtering: all configured targets were included, even ones
whose rule had ExistingObjectReplicationStatus::DISABLED.

Fix: pass op_type: roi.op_type and derive existing_object from it
(true only for ExistingObject, not Heal — Heal intentionally bypasses
the existing-object opt-out to repair past failures).

Also add warn! logs at all four MRF channel-overflow sites that were
previously silently returning Missed with no observability.

Verified with a live two-instance test: after resync, objects reached
the ENABLED target and were correctly blocked from the DISABLED target.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-14 07:14:11 +08:00
安正超 a85cc0354c refactor(storage-api): remove namespace lock from StorageAPI (#3365)
* refactor(storage-api): remove namespace lock from StorageAPI

* test(scanner): wait for runtime budget cancellation

---------

Co-authored-by: loverustfs <hello@rustfs.com>
2026-06-11 22:42:12 +08:00
houseme 0a987d870b refactor(logging): reduce runtime noise (#3363) 2026-06-11 19:49:01 +08:00
安正超 2f7cc7cb9e refactor(storage): narrow resync metadata bounds (#3345) 2026-06-11 04:41:15 +00:00
Henry Guo 6413df4b7f feat(replication): expose scanner repair outcomes (#3278)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
2026-06-08 14:24:27 +00:00
安正超 97dd825468 fix(ecstore): optimize ObjectInfo clone and fix critical TODOs (#3149)
* fix(ecstore): optimize ObjectInfo clone and fix critical TODOs

## ObjectInfo hot-path clone optimization (issue #653 item 2)
- Wrap user_defined (HashMap), user_tags (String), parts (Vec) in Arc
- Clone cost reduced from ~20 heap allocations to O(1) ref count bump
- Updated 11 downstream access sites with explicit deref where needed

## Critical TODO/FIXME fixes (issue #653 item 7)
- rpc/peer_s3_client.rs: descriptive error message for empty peer response
- set_disk/write.rs: concurrent rollback deletes via tokio::spawn + join_all
- store_list_objects.rs: resolved FIXME with explanation + 4 regression tests
- store/bucket.rs: namespace write locks for make_bucket and delete_bucket

Skipped TODOs (too risky without broader context):
- multipart.rs:30 nslock — causes lock timeout in existing tests
- bucket.rs:88 cached list_bucket — needs cache invalidation strategy
- bucket.rs:149 replication delete — needs replication subsystem integration

1134 tests pass (1 flaky under parallel execution, passes individually).

* fix: address PR #3149 review comments

- Preserve NamespaceLockQuorumUnavailable variant in bucket create/delete
- Fix cancellation test to actually test cancellation (keep senders open)
- Update ObjectInfo consumers in app/ for Arc-backed fields
- Update bucket_usecase.rs user_tags to Arc<String>

* fix: update ObjectInfo Arc consumers

* fix: address ecstore merge review comments

* fix: cancel blocked merge output sends

* fix(ecstore): satisfy ObjectInfo clone clippy
2026-06-01 00:44:58 +00:00
houseme 13b4500212 feat(obs): improve telemetry stack, replication metrics, and Grafana alignment (#2672)
Co-authored-by: Filipe Monteiro <a22407332@alunos.ulht.pt>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-04-24 13:50:17 +00:00
Tunglies e05d07494e chore(lint): clippy rules large_futures (#2555)
Co-authored-by: houseme <housemecn@gmail.com>
2026-04-16 02:10:13 +00:00
Tunglies 49366ee200 chore(lint): clippy rules redundant_clone (#2554) 2026-04-15 13:54:07 +00:00
weisd 15995aae14 feat(admin): complete site replication support (#2346) 2026-03-31 13:01:38 +08:00
weisd 28f57b228c feat(s3): advance parity coverage (#2278) 2026-03-24 17:29:33 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
evan slack d52a10c5fb chore(obs): Improve tracing instrumentation (#2086)
Co-authored-by: loverustfs <hello@rustfs.com>
2026-03-07 20:03:20 +08:00
安正超 09aa6d9f6f refactor(app): remove remaining global access in main init (#1978) 2026-02-26 22:14:51 +08:00
houseme 0b870d6301 build(deps): bump the dependencies group with 19 updates (#1745) 2026-02-07 12:22:14 +08:00
weisd 6631407416 feat: Add RustFS Scanner Module and Multiple Bug Fixes (#1579) 2026-01-22 13:39:38 +08:00
houseme f795299d53 Optimization and collation of dependencies introduction processing (#1493) 2026-01-13 15:02:54 +08:00
houseme 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
weisd cd1e244c68 Refactor: Introduce content checksums and improve multipart/object metadata handling (#671)
* feat:  adapt to s3s typed etag support

* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug

* add head_object checksum, filter object metadata output

* fix multipart checksum

* fix multipart checksum

* add content md5,sha256 check

* fix test

* fix cargo

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2025-10-20 23:46:13 +08:00
weisd 90f21a9102 refactor: Reimplement bucket replication system with enhanced architecture (#590)
* feat:refactor replication

* use aws sdk for replication client

* refactor/replication

* merge main

* fix lifecycle test
2025-09-26 14:27:53 +08:00