* fix(ecstore): keep object deletes running when the caller is dropped
DeleteObject and DeleteObjects commit on every disk and then run a
separate cleanup pass, both inside the request future. When the client
disconnects, hyper drops that future and the delete stops between the
two passes: xl.meta is gone, but the rollback backup and the marked data
dir stay on disk, hidden from listings and never reclaimed (#6898).
Run the ECStore delete entry points on a detached task tracked by the
instance context. The task owns the object locks, both disk passes and
any rollback, so dropping the caller only stops the wait. Server
shutdown waits up to 30s for detached deletes after the HTTP drain.
Two smaller leaks of the same kind: a cancelled data-dir delete no
longer leaves its snapshot-lease entry marked as deleting forever, and
DiskStore::delete_versions releases its waiting slot through a guard.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): finish committed delete residue on repeated deletes
Residue that an older build stranded (#6898) has no xl.meta, so deleting
the key again reports it missing and leaves everything in place. On a
disk, a delete-data.<T> marker in one of the object's data dirs proves
that transaction T removed the metadata there and committed: the data it
marked can no longer be read, healed or rolled back on that disk.
When delete_versions or delete_version finds no xl.meta, remove what the
cleanup pass of every transaction proven that way would have removed,
under the object's metadata lease. The rollback backup goes before the
marked data dirs, so an attempt cut short still leaves the marker for
the next one. Reserved markers, rollback backups without a committed
marker and directories that also hold xl.meta or other keys are left
alone. The disk result stays FileNotFound; the extra cost is one
directory read per missing key inside the disk call that already runs,
with no new RPC, and the read stops at the first child that is not a
data dir. Finished transactions are counted in
rustfs_disk_delete_residue_finished_total instead of logged per object.
A single DeleteObject in an unversioned bucket answers a missing key
from the metadata read quorum without reaching the disks, so existing
residue is reclaimed by DeleteObjects or by the empty-listing purge.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): purge the rollback backup of a committed delete on listing
After an empty listing, the orphan purge (#7955, #8261) already removes
data dirs that carry a committed delete-data.<T> marker, but it treated
the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data
and blocked the whole object directory and its ancestors. Residue left
by an abandoned delete (#6898) therefore survived the listing, and
DeleteBucket kept failing with BucketNotEmpty.
Treat a UUID dir that holds only xl.meta.bkp as purgeable when a
purgeable data dir of the same object on the same disk carries the
committed marker of that transaction. A backup without such a marker
still blocks as before, since it may be needed to roll the delete back.
The other purge gates are unchanged: every disk of the set must be
online, and the backoff applies. xl.meta or unmarked data on any disk
blocks that directory and its ancestors on every disk. Like a marked
data dir before, a backup below such a directory is still removed from
a disk that holds no xl.meta there and proves the transaction with its
own marker.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): move caller-held delete locks into the detached delete
A force (recursive) delete takes the bucket lifecycle write lock in the
S3 handler and passes only its fence down, so the store does not take
that lock again. Since the delete runs on a detached task, dropping the
request released the lock while the delete kept running: a PUT under
the same prefix could succeed and then be removed by the old delete.
Lifecycle expiry had the same shape with its table-bucket publication
read lock, of which only the lock-lost signal reaches the delete.
Add ECStore::delete_object_with_tier_delete_journal_and_guards. It
moves the caller's guards into the detached task and releases them
after the delete and its listing bookkeeping. The force-delete handler
and lifecycle expiry hand their guards over; DeleteObjects holds none.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(startup): keep unclean-shutdown markers while deletes still run
Shutdown waits up to 30s for detached deletes, but it cleared the
unclean-shutdown markers even when that wait timed out. A delete then
cut short by the runtime teardown went unnoticed: the next startup
skipped the erasure-set heal that the marker triggers.
Clear the markers only when the heal handoff completed and every
detached mutation finished, and log that they are kept otherwise. Wait
after the FTP, FTPS, SFTP and WebDAV servers have stopped as well,
since their deletes take the same detached path.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* test(ecstore): simulate tier-delete crashes below the detached delete
The chunk crash-boundary tests model a crash by aborting the task that
runs delete_object_with_tier_delete_journal while it is paused at a
TierDeleteChunkTestBarrier. That entry point now runs the delete on a
detached task, so aborting the caller no longer stops it: the delete
resumed on the old store once the barrier was dropped, and building the
restarted store on the same disks tripped the tier-delete journal proof
assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources
failed in CI).
Run the delete through handle_delete_object_with_journal, the body
below the detached task, so the abort stops it at the barrier as before.
tier_delete_chunk_missing_child_with_journals_fails_closed used the same
pattern; its resumed delete only failed fast because the tier rejects
removes, so it gets the same change.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
---------
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
Reuse the existing UTF-8 byte minimum for credential creation so long service account secrets remain intact. Keep generated secrets at 40 bytes and document older receiver and import constraints.
* fix: resume stalled GET bodies
Retry a stalled source through the existing identity-fenced resume budget so transient stalls do not truncate committed responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix: preserve data usage read quorum errors
Apply the existing system-volume error mapping to snapshot body reads so a metadata-volume outage remains a retryable quorum failure.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix: handle transient heal lock failures
Align erasure-set heal classification with the shared lock taxonomy and avoid duplicate error logging for retried scanner journal writes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): scope observational publication leases
Separate ordinary namespace commit churn from data-movement fences and retry transient lease acquisition conflicts. Bind observational leases to the observed usage object while preserving strict authoritative publication checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): expect namespace admission conflict
Assert the retryable namespace-specific result when a commit finishes while strict lease admission awaits its snapshot.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(connect): add headroom to real performance probes
Give real filesystem and object integration probes enough bounded time on slow CI runners, and keep cleanup independent from measurement latency. Use a genuinely unwritable Linux fixture when tests run as root.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(targets): support Kafka SASL without TLS
Propagate the explicit Kafka transport mode through notify and audit targets while retaining TLS as the default.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): update RustFS Kafka crates to 1.4.0
Use the published Kafka clients after the SASL plaintext transport support release.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): skip warning for absent .minio.sys volume
Startup MinIO migrations read from .minio.sys and treat its absence as expected, but resolve_read_all_result logged a disk_local_read_version_fallback warning for every such read. Keep the access check and the VolumeNotFound return, and skip only the warning for the migrating metadata volume.
Refs #8372
* test(ecstore): cover .minio.sys descendants and lookalike volumes
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* feat(table-catalog): support disabling empty catalogs
* fix(admin): use the storage error domain module
* style(admin): format the publication fence error
* fix(table-catalog): use shared API error constructors
* fix(tier): drop reserved internal headers from transition PUT
* test(tier): transition PUT against a target rejecting reserved headers
The fake S3 target gains a mode that answers 400 InvalidArgument to any
PutObject or UploadPart carrying an x-minio-internal-* request header,
the rule MinIO and the Storj gateway enforce. RT-14 transitions an
inline and a non-inline object through a real server to that target and
asserts the stored bytes, both transition identity pairs and read-back.
The e2e-full linux selection digest is refreshed for the new test.
---------
Co-authored-by: Vector <vector@wcomp.io>
Co-authored-by: cxymds <cxymds@gmail.com>
The test used
with default stack size, but
the 16-disk deep heal path overflows the default macOS thread stack
(~8MB). Switch to an explicit with 32MB stack
and manual tokio runtime, matching the pattern used by sibling tests
in the same file.
Verified: test passes in isolation after the change.
* fix(auth): return ExpiredToken for retained STS credentials
Validate session identity independently of credential expiry in both IAM
rejection and claims validation, and preserve typed S3 errors at the caller.
Keep token binding, disabled-account precedence, and service-account behavior.
Add exact expiry, IAM, and HTTP/XML regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): synchronize the metadata slowtail hedge regression
Park the slow metadata task at a test-only fault checkpoint and release
healthy reads after the fault is active. Assert the spare response and
cancelled-task drain without imposing a 300 ms disk/scheduler deadline.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ilm): synchronize copy compensation transition stages
Observe automatic compensation reaching the mock tier PUT checkpoint before
waiting for the final metadata commit. Assert forced enqueue failure,
automatic backfill, one completed tier upload, and payload integrity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): isolate and coalesce dirty usage journals
Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): collapse confirmed journal clear condition
Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(oidc): ignore groups without a matching policy at login
OIDC login failed with "OIDC policy mapping did not resolve to current
policies" whenever any mapped group lacked a policy of the same name.
Directory-backed providers such as Active Directory always emit groups
like `DOMAIN\Domain Users` that can never be mapped, so group-based
authorization was impossible there.
Keep only policy names that resolve to an existing policy and are valid
in session claims, and reject the login only when none remain. The
signed `policy` claim still lists only resolved names, so request-time
evaluation and site replication receivers keep their invariant.
Refs #8163
* fix(oidc): only ignore unmapped groups-claim values
Limit the relaxed resolution to policy names derived solely from the
groups claim. Names from ROLE_POLICY, a dedicated CLAIM_NAME claim, or
an explicit IAM policy mapping must still all resolve.
Reject the login when a mapped name refers to an existing policy whose
name is not allowed in session claims: dropping it could remove an
explicit Deny and broaden access.
Claim-unsafe names are only checked against the in-memory policy cache
and never passed to storage-backed policy loading.
Refs #8163
* docs(oidc): document ignorable roles claim values and cover wiring
Roles claim values are merged into the canonical groups, so values
without a matching policy are ignored like groups-claim values. This
covers built-in provider roles such as Keycloak's `offline_access`.
Document that in the field and method docs and in the provider
requirements, and pin it with a test.
Assert that the OIDC authorization carries the group claim policies so
a regression in the wiring cannot silently disable the relaxation.
Refs #8163
* fix(oidc): ignore group policies only after confirmed absence
merge_policies drops names whose load fails, so a storage or decode
error for an uncached group policy was indistinguishable from a missing
one. The group name was then ignored and the session signed without it,
which could remove an existing Deny that request-time checks cannot
restore.
Add IamSys::policy_exists, which consults the cache and then storage and
reports false only for NoSuchPolicy while returning every other error.
The OIDC binding now ignores a group-derived name only after a confirmed
absence and rejects the login when a lookup fails.
Refs #8163
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
* feat(iam): expose stored bucket tags to OPA
* fix(ci): lint bucket tag fixtures and register new e2e tests
* fix(ecstore): restore strict Clippy compatibility on Rust 1.99
Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 537c15277a)
* fix(deps): replace yanked yoke-derive release
(cherry picked from commit b54b32b8a9)
* perf(iam): borrow unchanged OPA bucket tag conditions
Reuse clean conditions for bucketless and untagged OPA decisions. Keep the existing clone-and-sanitize path for tagged requests, reserve enrichment capacity, and preserve authoritative lookup errors. Cover borrowed input, reserved namespaces, immutable request state, and both paths in the OPA input regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(iam): clarify OPA bucket policy fallback semantics
Document that an OPA tag mismatch remains an implicit IAM denial which a bucket-policy Allow can supplement. Extend the existing S3 contract test to contrast a normal OPA denial with corrupt bucket metadata, whose lookup error must abort the same Allow fallback.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): isolate durable progress fixtures from competing IO
Reserve the nextest execution budget for schedule, streaming lock, retirement, MRF checkpoint/crash, and native scanner progress scenarios. Preserve their internal concurrency, existing deadlines, typed failure assertions, and zero-retry policy in default and CI profiles.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: rjregenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(scanner): preserve checkpoint children in large prefixes
* fix(ecstore): restore strict Clippy compatibility on Rust 1.99
Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 537c15277a)
* fix(deps): replace yanked yoke-derive release
(cherry picked from commit b54b32b8a9)
* chore: keep scanner repair separate from CI corrections
* test(scanner): verify large-prefix resume and compaction lifecycle
Exercise Normal and Deep scans across cycle and leader changes. Validate
persisted frontiers, bound replay to interrupted boundary objects, and
prove that a completed same-plan sweep restores compaction without losing
object or byte totals.
Refs: rustfs/backlog#2710
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
GatewayJ