mirror of https://github.com/rustfs/rustfs.git synced 2026-10-12 23:52:45 +00:00

Files

T

cppudge 99ae115731 fix(ecstore): make deletes cancellation-safe and reclaim their residue (#8378 )

* fix(ecstore): keep object deletes running when the caller is dropped

DeleteObject and DeleteObjects commit on every disk and then run a
separate cleanup pass, both inside the request future. When the client
disconnects, hyper drops that future and the delete stops between the
two passes: xl.meta is gone, but the rollback backup and the marked data
dir stay on disk, hidden from listings and never reclaimed (#6898).

Run the ECStore delete entry points on a detached task tracked by the
instance context. The task owns the object locks, both disk passes and
any rollback, so dropping the caller only stops the wait. Server
shutdown waits up to 30s for detached deletes after the HTTP drain.

Two smaller leaks of the same kind: a cancelled data-dir delete no
longer leaves its snapshot-lease entry marked as deleting forever, and
DiskStore::delete_versions releases its waiting slot through a guard.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): finish committed delete residue on repeated deletes

Residue that an older build stranded (#6898) has no xl.meta, so deleting
the key again reports it missing and leaves everything in place. On a
disk, a delete-data.<T> marker in one of the object's data dirs proves
that transaction T removed the metadata there and committed: the data it
marked can no longer be read, healed or rolled back on that disk.

When delete_versions or delete_version finds no xl.meta, remove what the
cleanup pass of every transaction proven that way would have removed,
under the object's metadata lease. The rollback backup goes before the
marked data dirs, so an attempt cut short still leaves the marker for
the next one. Reserved markers, rollback backups without a committed
marker and directories that also hold xl.meta or other keys are left
alone. The disk result stays FileNotFound; the extra cost is one
directory read per missing key inside the disk call that already runs,
with no new RPC, and the read stops at the first child that is not a
data dir. Finished transactions are counted in
rustfs_disk_delete_residue_finished_total instead of logged per object.

A single DeleteObject in an unversioned bucket answers a missing key
from the metadata read quorum without reaching the disks, so existing
residue is reclaimed by DeleteObjects or by the empty-listing purge.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): purge the rollback backup of a committed delete on listing

After an empty listing, the orphan purge (#7955, #8261) already removes
data dirs that carry a committed delete-data.<T> marker, but it treated
the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data
and blocked the whole object directory and its ancestors. Residue left
by an abandoned delete (#6898) therefore survived the listing, and
DeleteBucket kept failing with BucketNotEmpty.

Treat a UUID dir that holds only xl.meta.bkp as purgeable when a
purgeable data dir of the same object on the same disk carries the
committed marker of that transaction. A backup without such a marker
still blocks as before, since it may be needed to roll the delete back.
The other purge gates are unchanged: every disk of the set must be
online, and the backoff applies. xl.meta or unmarked data on any disk
blocks that directory and its ancestors on every disk. Like a marked
data dir before, a backup below such a directory is still removed from
a disk that holds no xl.meta there and proves the transaction with its
own marker.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): move caller-held delete locks into the detached delete

A force (recursive) delete takes the bucket lifecycle write lock in the
S3 handler and passes only its fence down, so the store does not take
that lock again. Since the delete runs on a detached task, dropping the
request released the lock while the delete kept running: a PUT under
the same prefix could succeed and then be removed by the old delete.
Lifecycle expiry had the same shape with its table-bucket publication
read lock, of which only the lock-lost signal reaches the delete.

Add ECStore::delete_object_with_tier_delete_journal_and_guards. It
moves the caller's guards into the detached task and releases them
after the delete and its listing bookkeeping. The force-delete handler
and lifecycle expiry hand their guards over; DeleteObjects holds none.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(startup): keep unclean-shutdown markers while deletes still run

Shutdown waits up to 30s for detached deletes, but it cleared the
unclean-shutdown markers even when that wait timed out. A delete then
cut short by the runtime teardown went unnoticed: the next startup
skipped the erasure-set heal that the marker triggers.

Clear the markers only when the heal handoff completed and every
detached mutation finished, and log that they are kept otherwise. Wait
after the FTP, FTPS, SFTP and WebDAV servers have stopped as well,
since their deletes take the same detached path.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* test(ecstore): simulate tier-delete crashes below the detached delete

The chunk crash-boundary tests model a crash by aborting the task that
runs delete_object_with_tier_delete_journal while it is paused at a
TierDeleteChunkTestBarrier. That entry point now runs the delete on a
detached task, so aborting the caller no longer stops it: the delete
resumed on the old store once the barrier was dropped, and building the
restarted store on the same disks tripped the tier-delete journal proof
assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources
failed in CI).

Run the delete through handle_delete_object_with_journal, the body
below the detached task, so the abort stops it at the barrier as before.
tier_delete_chunk_missing_child_with_journals_fails_closed used the same
pattern; its resumed delete only failed fast because the tier rejects
removes, so it gets the same change.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

---------

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

2026-10-08 02:58:14 -07:00

6.8 KiB

Raw Blame History

Global State And Crate Split Plan

Use this when: business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a new crate split out of ECStore. Source of truth: crates/ecstore/src/runtime/global.rs and crates/ecstore/src/runtime/sources.rs (ECStore-owned state and its adapter), rustfs/src/app/context.rs and the runtime_sources.rs owner modules under rustfs/src (RustFS resolvers), and the rustfs_ecstore::api::global boundary list in scripts/check_architecture_migration_rules.sh. The static inventory is global-state-inventory.md.

Broad resolver-fallback removal is complete: runtime resolver fallbacks live in explicit owner-local boundaries, not in the root facade. What remains is ECStore-owned bootstrap state and crate-split decisions.

Remaining Global Owners

Owner	Role	Stance
`rustfs/src/app/context.rs`	AppContext-first resolver facade.	Resolver helpers stay context-first and do not construct concrete no-AppContext defaults.
`rustfs/src/app/context/runtime_sources.rs`	Default adapters for KMS, IAM, object store, endpoints, config, metrics, and notification state used by AppContext construction.	Allowed adapter boundary, not a business-logic owner.
`rustfs/src/runtime_sources.rs`, `rustfs/src/admin/runtime_sources.rs`, `rustfs/src/app/runtime_sources.rs`, `rustfs/src/server/runtime_sources.rs`, `rustfs/src/storage/runtime_sources.rs`	Owner-local runtime-source boundaries.	Business modules use these instead of global state; owner facades decide when to apply no-AppContext compatibility defaults.
`rustfs/src/storage_api.rs`, `rustfs/src/admin/storage_api.rs`, `rustfs/src/app/storage_api.rs`, `rustfs/src/storage/storage_api.rs`	Owner-local storage contract/facade boundaries.	Storage helper and ECStore facade access stays visible at local owner boundaries.
`crates/*/storage_api.rs`	External crate-local storage facade boundaries (IAM, scanner, heal, notify, observability, Swift, S3 Select).	External runtime crates read ECStore runtime state through `rustfs_ecstore::api::runtime`, never the global facade.
`crates/ecstore/src/runtime/global.rs`	ECStore bootstrap/runtime state owner.	Internal until ECStore has explicit owner handles for all remaining bootstrap state.
`crates/ecstore/src/runtime/sources.rs`	ECStore runtime-source adapter over global state.	Preferred ECStore-internal access path while direct `runtime::global` reads shrink.

Runtime Source Boundaries

Runtime-source modules are the allowed compatibility layer between migrated consumers and process-global state. They keep these properties:

context-first lookup when an AppContext handle exists;
explicit fallback to the existing global only where compatibility still requires it, decided by the owner facade;
no hidden service construction in business logic;
the root rustfs/src/runtime_sources.rs is an entrypoint only: it composes no concrete fallback defaults (unwrap_or, unwrap_or_else, direct init_global or new_global calls);
production callers outside runtime-source and storage_api.rs boundary modules do not import ECStore global state directly.

Guarded Boundary List

The guard pins the production files allowed to reference rustfs_ecstore::api::global directly:

rustfs/src/storage/storage_api.rs

That boundary keeps only bootstrap writes and lifecycle controls (set_global_endpoints, set_global_region, set_global_rustfs_port, set_object_store_resolver, shutdown_background_services, update_erasure_type, wait_for_detached_mutations). Read-only runtime getters are exported through rustfs_ecstore::api::runtime and consumed through the local storage facade. A new direct use either moves behind an existing owner-local boundary or updates this plan and the guard in the same reviewed change.

Fallback Removal Plan

AppContext-first lookup is the stable resolver contract.
Concrete no-AppContext compatibility defaults exist only at the owner-local runtime-source facades that consume them.
Business logic does not call AppContext or ECStore globals directly when an owner-local runtime-source boundary exists.
Embedded startup and tests keep working before any remaining owner fallback is deleted.
ECStore bootstrap globals stay until ownership handles exist for local disks, endpoint pools, lock clients, notification state, tier config, lifecycle state, and object-store publication.

Crate Split Evaluation

ecstore-erasure and storage-cluster are proposal-only; neither is ready for code movement. Lifecycle and replication split status is tracked in ecstore-module-split-plan.md.

`ecstore-erasure`

Coupling: erasure decoding depends on disk errors, disk read timeouts, and set-disk shard sources; set-disk read/write/heal paths construct codecs in hot object I/O paths; bitrot readers/writers live in ECStore IO support and serve both erasure and set-disk code; rustfs_ecstore::api::erasure is still a public compatibility surface.

Decision: do not split. The boundary becomes a candidate only after shard-source, disk-error, bitrot, and metrics contracts are explicit enough to avoid a dependency cycle back into ECStore, backed by encode/decode/reconstruction benchmarks and a rollback plan that keeps read/write quorum and old-version decode unchanged.

`storage-cluster`

Coupling: cluster RPC remote-disk code depends on disk stores, disk health tracking, set-disk buffer sizing, local disk scan guards, internode metrics, and runtime credential/signature sources; peer S3 and peer REST clients share bucket metadata, disk quorum reduction, endpoint layout, local disk initialization, and store helpers; control-plane snapshots are separate from data-plane RPC, but remote disk and peer clients still own data-movement side effects inside ECStore.

Decision: do not split. The boundary becomes a candidate only after remote disk, peer health, lock/quorum, runtime metrics, and endpoint layout contracts can stand below ECStore without cycles, with compatibility plans for rustfs_ecstore::api::cluster and api::rpc and focused tests for remote disk error classification, peer health recovery, per-pool quorum reduction, lock behavior, and data-stream request paths.

Preservation Rules

Do not reintroduce AppContext resolver fallback families in broad cleanups.
Do not introduce direct global reads in admin, app, server, storage, scanner, heal, IAM, notify, observability, Swift, or S3 Select business logic.
Do not split crates in the same change that moves runtime state.
Do not change startup order, readiness, KMS fatal boundaries, IAM recovery, lock quorum, object placement, reader behavior, or notification/audit lifecycle while shrinking global state.

6.8 KiB Raw Blame History