mirror of https://github.com/rustfs/rustfs.git synced 2026-08-23 04:39:04 +00:00

Files

T

cxymds 319a03e638 fix(lifecycle): safely expire all object versions (#6291 )

* fix(lifecycle): safely expire all object versions

* fix(lifecycle): preserve delete-all replication purges

* fix(lifecycle): remove dead replication journal

* fix(ci): avoid lifecycle transition test stack overflow

* fix(lifecycle): release recovery locks before tier IO

* test(lifecycle): align object-lock error assertions

* test(lifecycle): avoid scanner restore stack overflow

* test(scanner): avoid stack overflow in transition and restore flow test (#6300)

* refactor(scanner): split remote_scanner.rs into stream child module (#6289)

Split the 3080-line remote_scanner.rs (47% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:

- remote_scanner.rs (~320): protocol constants, process statics, and
  the request decode/validate/admit/preflight/claim API plus root
  re-exports
- remote_scanner/stream.rs (~1340): wire/frame types, replay cache,
  FrameAuthenticator, serve path, local bucket scan + persist, client
  scan, and the bounded stream plumbing
- remote_scanner/stream/tests.rs (~1470): the inline test module as a
  child module of stream so it can reach both parents' private items

All crate paths are unchanged: lib.rs re-exports
(serve_remote_scanner_request, RemoteScannerRequest, ...) resolve
through root re-exports, and scanner_io's crate::remote_scanner::
{scan_remote_bucket, RemoteScannerScanSpec, RemoteScannerOutcome}
paths resolve through pub(crate) re-exports. Cross-module items gain
pub(super), whose scope equals the old single-module privacy domain;
no item's effective visibility widens. Code is moved verbatim apart
from those markers, per-module import headers, and rustfmt line
re-wraps.

Co-authored-by: heihutu <heihutu@gmail.com>

* refactor(heal): split resume.rs into focused child modules (#6290)

Split the 4242-line resume.rs (46% inline tests) into a canonical
foo.rs + foo/ module tree with zero behavior change:

- resume.rs (~1020): state file constants, PersistThrottle, ResumeState,
  ResumeManager core (constructors, load/discovery, progress mutators,
  ordinary persistence) plus root re-exports
- resume/replacement.rs (~690): replacement-intent/proof types and the
  ResumeManager replacement-lifecycle methods
- resume/checkpoint.rs (~350): ResumeCheckpoint + CheckpointManager
- resume/utils.rs (~310): ResumeUtils statics
- resume/tests.rs (~1980): the inline test module as a child module

All module paths are unchanged (heal::resume::CheckpointManager and
friends resolve through root re-exports), so no consumer inside or
outside the crate changes. Items defined in child modules keep
module-private visibility; only the ten cross-module helpers gain
pub(super), which is not part of the crate API. Code is moved verbatim
apart from those visibility markers, four super::storage_api path
fixes, and the new per-module import headers.

Co-authored-by: heihutu <heihutu@gmail.com>

* refactor(scanner): split scanner_io.rs into child modules (#6294)

Split the 5369-line scanner_io.rs (39% inline tests) into a canonical
scanner_io.rs + scanner_io/ module tree with zero behavior change:

- scanner_io.rs (~660): constants, metadata-error constructors, the
  bucket scan plan, cycle-status classification helpers, the ScannerIO /
  ScannerIOCache / ScannerIODisk traits, and ScannerCycleResult
- scanner_io/dirty_usage.rs (~300): process-wide dirty-usage statics
  and the acknowledgment protocol
- scanner_io/guards.rs (~270): concurrency gauges and RAII guards
- scanner_io/cache.rs (~410): scanner cache locks and the snapshot
  persist/publish path
- scanner_io/io_cycle.rs (~390), io_cache.rs (~1160), io_disk.rs
  (~230): the ECStore / SetDisks / Disk trait implementations
- scanner_io/publish_gate_tests.rs (~750) and tests.rs (~1340): the two
  inline test modules as child modules

All crate paths are unchanged: the lib.rs scanner_io re-exports and
every crate::scanner_io:: consumer (scanner.rs, remote_scanner,
scanner_folder, and cross-crate rustfs users) resolve through root
re-exports with their original visibilities (pub stays pub, pub(crate)
stays pub(crate)). Cross-module items gain pub(super), whose scope
equals the old single-module privacy domain. Code is moved verbatim
apart from those markers, per-module import headers, and rustfmt
re-wraps.

The logging-guardrail nsscanner_disk skip-set_disks rule now points at
scanner_io/io_disk.rs where the function moved; the pattern and
thresholds are unchanged.

Co-authored-by: heihutu <heihutu@gmail.com>

* refactor(scanner): split data_usage_define persistence and tests (#6292)

Split the 3655-line data_usage_define.rs (59% inline tests) into a
canonical foo.rs + foo/ module tree with zero behavior change:

- data_usage_define.rs (~950): cache constants and revision helpers,
  the data-usage tree types, DataUsageCacheInfo with its hand-written
  Serialize, the in-memory tree operations, dui, and marshal/unmarshal
- data_usage_define/persistence.rs (~580): the load/backup/restore
  ladder (load, try_load_inner, revision_for_path) and the CAS save
  path with its retry policy and save metrics
- data_usage_define/tests.rs (~2155): the inline test module as a child
  module

All module paths are unchanged (the lib.rs data_usage_define::* glob
re-export and every crate::data_usage_define:: consumer resolve as
before). The hand-written map-encoded Serialize for
DataUsageCacheInfo is moved byte-for-byte per the AGENTS.md
cross-cutting invariant; on-disk names and the cache key format const
stay in the root. Four persistence helpers used by tests gain
pub(super), whose scope equals the old single-module privacy domain.
Code is moved verbatim apart from those markers, per-module import
headers, and rustfmt re-wraps.

Co-authored-by: heihutu <heihutu@gmail.com>

* chore(deps): bump datafusion to 55.0.0 (#6288)

* refactor(heal): split task.rs per heal kind (#6293)

* feat(ecstore): batch small file fdatasync commits (#6297)

* feat(ecstore): batch small file fdatasync commits

Add a default-off experimental file fdatasync group commit path for small rename_data shard directories. The coordinator batches same-disk waiters into one blocking task while preserving per-directory source fsync after shard contents are durable.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): wait for compression S3 readiness

Reuse the shared S3 API readiness probe for compression test servers so multipart requests do not race the startup readiness gate after the TCP port opens.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>

* fix(tier): recover multi-committed mutation intents (#6296)

* fix(tier): recover multi-committed mutation intents

* fix(tier): recover committed mutations on standalone nodes

* test(scanner): avoid stack overflow in transition test

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>

2026-08-20 09:42:09 +00:00

bucket_lifecycle_audit.rs

refactor(lifecycle): route events through audit sink (#4066 )

2026-06-29 22:21:23 +08:00

bucket_lifecycle_ops.rs

fix(lifecycle): safely expire all object versions (#6291 )

2026-08-20 09:42:09 +00:00

config_boundary.rs

fix(ecstore): retry manual ILM job CAS updates (#6012 )

2026-08-12 18:40:10 +00:00

core.rs

refactor(lifecycle): extract core rule contracts (#4258 )

2026-07-04 20:05:56 +08:00

evaluator.rs

refactor(lifecycle): extract core rule contracts (#4258 )

2026-07-04 20:05:56 +08:00

manual_transition_job.rs

chore(ecstore): drop the bucket dead_code blanket (#6147 )

2026-08-16 21:39:04 +08:00

metadata_boundary.rs

fix(table-catalog): harden commit publication (#5779 )

2026-08-11 19:36:46 +08:00

mod.rs

feat(heal): add progress and trace observability (#6179 )

2026-08-18 08:29:29 +08:00

object_lock_boundary.rs

fix: address rc.1 release blockers (#5648 )

2026-08-03 19:25:43 +00:00

README.md

refactor(runtime): batch owner boundary guards (#4092 )

2026-06-30 04:34:54 +08:00

replication_sink.rs

chore(ecstore): drop the bucket dead_code blanket (#6147 )

2026-08-16 21:39:04 +08:00

rule.rs

refactor(lifecycle): extract core rule contracts (#4258 )

2026-07-04 20:05:56 +08:00

runtime_boundary.rs

refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13) (#4586 )

2026-07-08 22:06:31 +00:00

tagging_boundary.rs

chore(ecstore): drop the bucket dead_code blanket (#6147 )

2026-08-16 21:39:04 +08:00

tier_delete_journal.rs

fix(lifecycle): safely expire all object versions (#6291 )

2026-08-20 09:42:09 +00:00

tier_free_version_recovery.rs

chore(ecstore): drop the bucket dead_code blanket (#6147 )

2026-08-16 21:39:04 +08:00

tier_last_day_stats.rs

chore(ecstore): adjudicate 32 bare dead_code allows (#6173 )

2026-08-17 23:51:56 +00:00

tier_sweeper.rs

fix(lifecycle): safely expire all object versions (#6291 )

2026-08-20 09:42:09 +00:00

transition_transaction.rs

chore(ecstore): drop the bucket dead_code blanket (#6147 )

2026-08-16 21:39:04 +08:00

README.md

ECStore Lifecycle Split Inventory

This directory still belongs to ECStore. It is not ready to become a standalone crate because lifecycle workers currently depend on ECStore runtime state, object IO, bucket metadata, replication scheduling, notification/audit sinks, and tier services.

Current Modules

Module	Current role	Split blocker
`core.rs`	Lifecycle rule model, action evaluation, object options, and transition/expiry decisions.	Uses ECStore object metadata types and compatibility DTO re-exports.
`bucket_lifecycle_ops.rs`	Worker orchestration, expiry, transition, stale multipart cleanup, audit, replication delete scheduling, and queue state.	Depends on `ECStore`, `SetDisks`, runtime globals, bucket metadata/versioning, disk internals, event notification, tier services, and lifecycle-local object-lock/replication boundaries.
`evaluator.rs`	Bucket lifecycle evaluation wrapper.	Uses lifecycle-local object-lock boundary and replication state through lifecycle-local replication sink.
`rule.rs`	Lifecycle rule filter helpers.	Uses lifecycle-local tagging boundary.
`tier_delete_journal.rs`	Remote tier delete journal persistence and recovery.	Uses lifecycle-local config persistence boundary, object IO contracts, metadata bucket paths, and `ECStore`.
`tier_free_version_recovery.rs`	Free-version recovery queue and object restoration path.	Depends on `ECStore`, object metadata, storage-api contracts, and lifecycle queue callbacks.
`tier_last_day_stats.rs`	Tier statistics helpers.	Pure data/stat logic, but still part of lifecycle worker reporting.
`tier_sweeper.rs`	Remote tier deletion worker and transition cleanup.	Depends on runtime sources, `ECStore`, tier journal persistence, signer-error handling, and lifecycle object options.
`bucket_lifecycle_audit.rs`	Lifecycle audit event source labels.	Must remain wired to lifecycle audit and notification sinks.

Required Contracts

Contract	Responsibility	Current dependency to remove
`LifecycleObjectStore`	Object stat, delete, transition, restore, multipart cleanup, and version-aware metadata operations.	Direct `ECStore`, `SetDisks`, disk, and object API access in worker paths.
`LifecycleMetadataStore`	Lifecycle, object-lock, replication, bucket versioning, and stale multipart metadata reads.	Direct bucket metadata/versioning imports; object-lock and replication are still backed by local ECStore boundaries.
`LifecycleRuntime`	Expiry state, transition state, tier config, deployment ID, local node name, queue metrics, cancellation, and worker sizing.	Direct runtime source/global access and process environment reads inside worker code.
`LifecycleConfigStore`	Persist, read, and remove lifecycle-owned journal/config objects.	Direct ECStore config persistence helper imports from worker paths.
`LifecycleTagFilter`	Decode object tag strings for lifecycle rule matching.	Direct bucket tagging helper imports from lifecycle rule paths.
`LifecycleObjectLockStore`	Object-lock retention and deletion checks used by lifecycle evaluation and worker deletion paths.	Direct object-lock module imports from lifecycle code.
`LifecycleReplicationSink`	Lifecycle-originated delete and version-purge replication scheduling.	Boundary is local, but still backed by ECStore bucket replication internals.
`LifecycleAuditSink`	Lifecycle audit and notification event emission.	Direct event notification service calls and audit-side effects from worker code.

Migration Rules

Do not move bucket/lifecycle into a new crate while any worker imports crate::store::ECStore, crate::set_disk::SetDisks, runtime globals, or bucket replication internals directly.
Extract pure contracts before runtime movement. The first code-bearing PR should introduce one contract boundary and keep the old ECStore call path.
Preserve lifecycle queue behavior, transition/expiry state, replication delete scheduling, notification/audit events, scanner-visible metrics, and stale multipart cleanup semantics.
Keep rustfs_ecstore::api::bucket::lifecycle compatibility until scanner, OBS, and test boundary files compile through replacement paths.
Verify each code-bearing step with focused lifecycle tests before attempting broad gates.

First Code-Bearing Step

Start with LifecycleRuntime or LifecycleAuditSink. Both can be introduced as narrow internal contracts while keeping the current ECStore worker behavior unchanged. Do not start with a crate move.

Current first boundary: runtime_boundary.rs centralizes lifecycle access to runtime state while preserving the existing ECStore-backed implementations. config_boundary.rs centralizes lifecycle-owned config object persistence for tier delete journal recovery while preserving the existing ECStore config store. tagging_boundary.rs centralizes lifecycle tag decoding while preserving the existing ECStore bucket tagging implementation. object_lock_boundary.rs centralizes lifecycle object-lock checks while preserving the existing ECStore object-lock implementation. replication_sink.rs centralizes lifecycle-originated replication config checks and delete scheduling while preserving the existing ECStore replication worker path.