* fix(ecstore): keep object deletes running when the caller is dropped
DeleteObject and DeleteObjects commit on every disk and then run a
separate cleanup pass, both inside the request future. When the client
disconnects, hyper drops that future and the delete stops between the
two passes: xl.meta is gone, but the rollback backup and the marked data
dir stay on disk, hidden from listings and never reclaimed (#6898).
Run the ECStore delete entry points on a detached task tracked by the
instance context. The task owns the object locks, both disk passes and
any rollback, so dropping the caller only stops the wait. Server
shutdown waits up to 30s for detached deletes after the HTTP drain.
Two smaller leaks of the same kind: a cancelled data-dir delete no
longer leaves its snapshot-lease entry marked as deleting forever, and
DiskStore::delete_versions releases its waiting slot through a guard.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): finish committed delete residue on repeated deletes
Residue that an older build stranded (#6898) has no xl.meta, so deleting
the key again reports it missing and leaves everything in place. On a
disk, a delete-data.<T> marker in one of the object's data dirs proves
that transaction T removed the metadata there and committed: the data it
marked can no longer be read, healed or rolled back on that disk.
When delete_versions or delete_version finds no xl.meta, remove what the
cleanup pass of every transaction proven that way would have removed,
under the object's metadata lease. The rollback backup goes before the
marked data dirs, so an attempt cut short still leaves the marker for
the next one. Reserved markers, rollback backups without a committed
marker and directories that also hold xl.meta or other keys are left
alone. The disk result stays FileNotFound; the extra cost is one
directory read per missing key inside the disk call that already runs,
with no new RPC, and the read stops at the first child that is not a
data dir. Finished transactions are counted in
rustfs_disk_delete_residue_finished_total instead of logged per object.
A single DeleteObject in an unversioned bucket answers a missing key
from the metadata read quorum without reaching the disks, so existing
residue is reclaimed by DeleteObjects or by the empty-listing purge.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): purge the rollback backup of a committed delete on listing
After an empty listing, the orphan purge (#7955, #8261) already removes
data dirs that carry a committed delete-data.<T> marker, but it treated
the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data
and blocked the whole object directory and its ancestors. Residue left
by an abandoned delete (#6898) therefore survived the listing, and
DeleteBucket kept failing with BucketNotEmpty.
Treat a UUID dir that holds only xl.meta.bkp as purgeable when a
purgeable data dir of the same object on the same disk carries the
committed marker of that transaction. A backup without such a marker
still blocks as before, since it may be needed to roll the delete back.
The other purge gates are unchanged: every disk of the set must be
online, and the backoff applies. xl.meta or unmarked data on any disk
blocks that directory and its ancestors on every disk. Like a marked
data dir before, a backup below such a directory is still removed from
a disk that holds no xl.meta there and proves the transaction with its
own marker.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): move caller-held delete locks into the detached delete
A force (recursive) delete takes the bucket lifecycle write lock in the
S3 handler and passes only its fence down, so the store does not take
that lock again. Since the delete runs on a detached task, dropping the
request released the lock while the delete kept running: a PUT under
the same prefix could succeed and then be removed by the old delete.
Lifecycle expiry had the same shape with its table-bucket publication
read lock, of which only the lock-lost signal reaches the delete.
Add ECStore::delete_object_with_tier_delete_journal_and_guards. It
moves the caller's guards into the detached task and releases them
after the delete and its listing bookkeeping. The force-delete handler
and lifecycle expiry hand their guards over; DeleteObjects holds none.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(startup): keep unclean-shutdown markers while deletes still run
Shutdown waits up to 30s for detached deletes, but it cleared the
unclean-shutdown markers even when that wait timed out. A delete then
cut short by the runtime teardown went unnoticed: the next startup
skipped the erasure-set heal that the marker triggers.
Clear the markers only when the heal handoff completed and every
detached mutation finished, and log that they are kept otherwise. Wait
after the FTP, FTPS, SFTP and WebDAV servers have stopped as well,
since their deletes take the same detached path.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* test(ecstore): simulate tier-delete crashes below the detached delete
The chunk crash-boundary tests model a crash by aborting the task that
runs delete_object_with_tier_delete_journal while it is paused at a
TierDeleteChunkTestBarrier. That entry point now runs the delete on a
detached task, so aborting the caller no longer stops it: the delete
resumed on the old store once the barrier was dropped, and building the
restarted store on the same disks tripped the tier-delete journal proof
assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources
failed in CI).
Run the delete through handle_delete_object_with_journal, the body
below the detached task, so the abort stops it at the barrier as before.
tier_delete_chunk_missing_child_with_journals_fails_closed used the same
pattern; its resumed delete only failed fast because the tier rejects
removes, so it gets the same change.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
---------
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(scanner): scope observational publication leases
Separate ordinary namespace commit churn from data-movement fences and retry transient lease acquisition conflicts. Bind observational leases to the observed usage object while preserving strict authoritative publication checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): expect namespace admission conflict
Assert the retryable namespace-specific result when a commit finishes while strict lease admission awaits its snapshot.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(connect): add headroom to real performance probes
Give real filesystem and object integration probes enough bounded time on slow CI runners, and keep cleanup independent from measurement latency. Use a genuinely unwritable Linux fixture when tests run as root.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
* refactor(sse): decouple encryption from ecstore
* feat(kms): enhance KMS service manager with runtime state and persistence support
* feat(kms): add local key export functionality for SSE-S3 migration tests
* fix(kms): keep local key export narrowly scoped
* fix(sse): validate copy source customer algorithm
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* fix(ilm): recover uploaded transition transactions
Persist transition transaction records through production transition uploads, use transaction-scoped remote object names, and start a conservative recovery loop for uploaded candidates.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): reconcile committed transition records
Drop LocalCommitStarted transaction records only after object metadata confirms that the local transition commit already points at the same tier object and version.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
The OnceLock introduced in 5fbd49a80 for bucket_metadata_sys panics when
multiple test modules initialize their own ECStore against the shared
bootstrap context. Switch to Mutex<Option> so test fixtures can
reinitialize safely.
Extract shared gating test environment to gating_test_env.rs to avoid
duplicate ECStore setup across delete_objects_stat_gating_test and
put_prelookup_gating_test.
Fixes 8 test failures:
- capacity_dirty_scope_test (2 tests)
- delete_objects_stat_gating_test (3 tests)
- put_prelookup_gating_test (3 tests)
backlog#1052 S3, third slice — the hard blocker for a second embedded
server's service stage. GLOBAL_BUCKET_METADATA_SYS was a process-global
OnceLock whose init ran .set().unwrap(): a second instance's service
startup panicked the whole process. The system it guards is inherently
per-store (it holds the store handle and that store's bucket→metadata
cache), so it now lives on the store's own InstanceContext:
- init_bucket_metadata_sys writes the cell of the store it was handed
(api.ctx) — no signature change — and keeps the double-init fail-fast,
now scoped to the instance (assert on the per-context set). The
dist-erasure check for the refresh loop reads the same context.
- get_global_bucket_metadata_sys / the crate-private getter behind the
~20 get_*_config helpers resolve through the current_ctx() facade —
the published store's context, or bootstrap before that — so every
existing reader keeps its exact single-instance behavior.
- New acceptance test: two real stores in one process each initialize
their own metadata system (the old global cell panicked right there),
plus a shared builder extracted from the store-graph test.
201 bucket/metadata regression tests green.
* refactor(ecstore): migrate the background-services cancel token into InstanceContext (Phase 5 Slice 13)
Phase 5 Slice 13 (backlog#939): move the background-services cancellation token
out of the process static into the per-instance InstanceContext, so cancelling
one instance's background workers (scanner/heal/tier/lifecycle) no longer
touches another instance.
- InstanceContext gains `background_cancel_token: OnceLock<CancellationToken>`
with `init_background_cancel_token` (set-once) and `background_cancel_token()`
returning an owned clone.
- global.rs `init_/get_/create_/shutdown_` helpers keep their signatures and
route through the current instance's context; the static is removed. The
getter now returns an owned `Option<CancellationToken>` instead of a
`Option<&'static _>`, which is what lets the token live in the context.
- Callers adapt to the owned token: the metadata-refresh loop drops `.cloned()`;
the lifecycle worker/loops take the owned token (a shared fallback is cloned
when the token is somehow uninitialized). No Arc cycle is introduced —
workers hold a token clone, not the instance context.
Single-instance behavior is unchanged: startup creates one token in the
bootstrap context the ECStore adopts, and shutdown cancels that same token.
Tests: the token is set-once and cancelling one instance's token leaves a
distinct instance uninitialized.
Verification: cargo test -p rustfs-ecstore (23 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 13)
* test(ecstore): prove multi-instance isolation; document embedded guard retention (Phase 5 Slice 14) (#4588)
Phase 5 (backlog#939) capstone. The prior 13 slices moved every piece of
per-instance runtime state out of process globals into ECStore's
InstanceContext. This slice proves the result and records the remaining work.
- Add `two_instances_isolate_all_migrated_state`: an end-to-end acceptance test
that constructs two independent InstanceContexts and verifies NONE of the
migrated state is shared — erasure setup, lock manager, region, deployment id,
the four service handles (tier/notifier/expiry/transition), the local disk
registry, the bucket monitor, and the background cancel token. This is the
object-graph isolation carrier working end to end.
- Document why the embedded single-instance guard (EMBEDDED_SERVER_STARTED) is
intentionally retained: storage startup still publishes into the process-level
bootstrap context (write-once region/endpoints/deployment id) and the single
GLOBAL_OBJECT_API handle, so a second startup would fail-fast on that shared
state. Lifting the guard requires threading a per-instance context through
startup — a follow-up beyond migrating the globals. The guard is NOT removed:
rejecting the second start is safer than the panic it would otherwise become.
Verification: cargo test -p rustfs-ecstore (acceptance test + all instance-context
tests green), cargo clippy -p rustfs-ecstore --all-targets (clean), make
pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 14). Stacked on Slice 13 (#4586).
Phase 5 Slice 12 (backlog#939): move the local disk registry — the
path->disk map, the disk-id->endpoint map, and the pool/set/drive layout —
out of the three GLOBAL_LOCAL_DISK_* process statics into the per-instance
InstanceContext.
This is the migration the owned-guard prerequisite (#4501) unblocked.
- InstanceContext gains three `Arc<RwLock<..>>` registry fields, constructed
eagerly (empty), plus pub(crate) handle accessors.
- The three runtime-source handle functions (`local_disk_map_handle` etc.) now
return the current instance's registry via `current_ctx()`; every direct
`GLOBAL_LOCAL_DISK_*` access in runtime::sources (~15 sites) routes through
those handles instead. Accessors that hold a write guard across `.await`
(`record_local_disks`, `initialize_local_disk_maps`, `replace_local_disk_id`)
bind the handle first, so the same locks are held across the same awaits.
- The three statics are removed; the test-reset helper clears the current
context's registry.
Construction window: the registry is populated during storage startup, before
the ECStore exists, via the bootstrap context that `ECStore::new` then adopts —
so startup writes and post-construction reads share one registry (no
split-brain). Single-instance behavior is byte-for-byte unchanged; the
GLOBAL_LOCAL_DISK_* boundary arch guard passes with the statics gone.
Verification: cargo test -p rustfs-ecstore (32 disk/instance tests green) and
-p rustfs-heal (201 green), cargo clippy -p rustfs-ecstore -p rustfs-heal
--all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 12; disk-registry migration)
Phase 5 Slice 11 (backlog#939): move the background replication pool and stats —
the last service handles, and the only async ones — out of the process statics
into the per-instance InstanceContext.
- InstanceContext gains `replication_stats: OnceCell<Arc<ReplicationStats>>` and
`replication_pool: OnceCell<Arc<DynReplicationPool>>` (tokio async OnceCell),
with sync read accessors (`replication_stats`/`replication_pool`/
`replication_initialized`) and pub(crate) cell accessors for the async init.
- `init_background_replication` initializes the current instance's cells via the
same `get_or_init(async {…}).await` (workers still spawned once on first
init). The lifecycle owner helpers and the runtime-source accessors keep their
signatures and route through the current instance's context; the two statics
(and the now-unused lazy_static import) are removed. The replication-boundary
arch guard still passes.
Single-instance: init materializes one shared pool/stats via the bootstrap
context — byte-for-byte the same as the eager statics.
Tests: replication state is None until set and independent across instances.
Verification: cargo test -p rustfs-ecstore (22 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 11). Stacked on Slice 10 (#4494).
refactor(ecstore): migrate the bucket bandwidth monitor into InstanceContext (Phase 5 Slice 10)
Phase 5 Slice 10 (backlog#939): move the bucket bandwidth monitor out of the
process static into the per-instance InstanceContext.
- InstanceContext gains `bucket_monitor: OnceLock<Arc<Monitor>>` with
`init_bucket_monitor(num_nodes)` (set-once; returns false if already set) and
`bucket_monitor() -> Option<..>`.
- global.rs `init_global_bucket_monitor` / `get_global_bucket_monitor` keep
their signatures and route through the current instance's context; the static
is removed. The ignore-and-warn-on-reinit behavior is preserved (the warn
stays in global.rs).
Tests: the monitor is None until initialized, set-once (second init is a no-op),
and independent across instances.
Verification: cargo test -p rustfs-ecstore (21 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 10).
Phase 5 Slice 4 (backlog#939): move the S3 region — a write-once identity
scalar — out of the GLOBAL_REGION process static into the per-instance
InstanceContext, so two instances can serve different regions.
- InstanceContext gains `region: OnceLock<Region>` with `set_region()` /
`region()`. `set_region` keeps the write-once fail-fast contract: a second
write panics, exactly as the process global did (not downgraded to a warn).
- global.rs `set_global_region` / `get_global_region` keep their signatures and
forward to the current instance's context; the GLOBAL_REGION static is
removed. Single-instance: startup writes the bootstrap context (which the
ECStore adopts), so reads are unchanged.
Tests: set/get round-trip, two contexts hold distinct regions, and a second
set_region panics (fail-fast preserved).
Verification: cargo test -p rustfs-ecstore (9 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 4). Stacked on Slice 3 (#4417).
* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type
Phase 5 of the global-singleton consolidation (backlog#939): begin moving
runtime identity state out of process globals so multiple ECStore instances
can coexist in one process. Isolation is carried by the object graph
(ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a
task-local, which does not propagate across the many internal tokio::spawn
boundaries in the data/background paths.
This first slice migrates the erasure setup type -- previously three
independent process-global bools -- into a single per-instance
RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd,
removing a triple source of truth that could drift out of sync.
- New runtime::instance module: InstanceContext + process bootstrap context.
- The legacy free-function facade (is_erasure/update_erasure_type/...) keeps
its signatures and forwards to the current instance's context, falling back
to the bootstrap context before a store is published.
- ECStore gains a pub(crate) ctx field and setup_is_* accessors; its
constructors adopt the bootstrap context (never mint a fresh one) so startup
writes and post-construction reads share one cell -- single-instance
behavior is byte-for-byte unchanged.
Tests: erasure predicate derivation vs the legacy behavior, object-graph
carrier isolation across two ECStore instances, and bootstrap adoption.
Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8)
* refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415)
* refactor(ecstore): source the namespace lock manager per-instance (#4417)
refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3)
Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by
sourcing SetDisks' lock manager from the instance context instead of the
process singleton. This removes the false cross-instance mutual exclusion (and
attendant ABBA risk) that a shared GlobalLockManager would cause once multiple
instances coexist.
- InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints
a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the
process singleton via get_global_lock_manager(), so a single-instance
deployment keeps exactly one shared namespace.
- SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx
it already adopts), not `runtime_sources::global_lock_manager()`. Single
instance: same Arc as before, so behavior is unchanged.
- Remove the now-unused `runtime_sources::global_lock_manager()` wrapper.
Tests: bootstrap lock manager aliases the process singleton; two fresh contexts
own distinct managers; a SetDisks' lock manager is the one from its context and
aliases the global singleton in a single-instance build.
Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking
regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean),
make pre-commit (pass).
Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2).
cppudge