* fix(ecstore): skip warning for absent .minio.sys volume
Startup MinIO migrations read from .minio.sys and treat its absence as expected, but resolve_read_all_result logged a disk_local_read_version_fallback warning for every such read. Keep the access check and the VolumeNotFound return, and skip only the warning for the migrating metadata volume.
Refs #8372
* test(ecstore): cover .minio.sys descendants and lookalike volumes
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* feat(table-catalog): support disabling empty catalogs
* fix(admin): use the storage error domain module
* style(admin): format the publication fence error
* fix(table-catalog): use shared API error constructors
* fix(tier): drop reserved internal headers from transition PUT
* test(tier): transition PUT against a target rejecting reserved headers
The fake S3 target gains a mode that answers 400 InvalidArgument to any
PutObject or UploadPart carrying an x-minio-internal-* request header,
the rule MinIO and the Storj gateway enforce. RT-14 transitions an
inline and a non-inline object through a real server to that target and
asserts the stored bytes, both transition identity pairs and read-back.
The e2e-full linux selection digest is refreshed for the new test.
---------
Co-authored-by: Vector <vector@wcomp.io>
Co-authored-by: cxymds <cxymds@gmail.com>
fix(compose): Refactor volume paths and add service_completed_successfully
The volume paths for the volume-permission-helper service have been updated and the command has been adjusted to reflect the new paths.
Ensure that the volume-permission-helper service runs successfully before starting the rustfs service.
Signed-off-by: Sven Gottwald <2502366+svengo@users.noreply.github.com>
The test used
with default stack size, but
the 16-disk deep heal path overflows the default macOS thread stack
(~8MB). Switch to an explicit with 32MB stack
and manual tokio runtime, matching the pattern used by sibling tests
in the same file.
Verified: test passes in isolation after the change.
* fix(auth): return ExpiredToken for retained STS credentials
Validate session identity independently of credential expiry in both IAM
rejection and claims validation, and preserve typed S3 errors at the caller.
Keep token binding, disabled-account precedence, and service-account behavior.
Add exact expiry, IAM, and HTTP/XML regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): synchronize the metadata slowtail hedge regression
Park the slow metadata task at a test-only fault checkpoint and release
healthy reads after the fault is active. Assert the spare response and
cancelled-task drain without imposing a 300 ms disk/scheduler deadline.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ilm): synchronize copy compensation transition stages
Observe automatic compensation reaching the mock tier PUT checkpoint before
waiting for the final metadata commit. Assert forced enqueue failure,
automatic backfill, one completed tier upload, and payload integrity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
* fix(scanner): isolate and coalesce dirty usage journals
Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): collapse confirmed journal clear condition
Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(oidc): ignore groups without a matching policy at login
OIDC login failed with "OIDC policy mapping did not resolve to current
policies" whenever any mapped group lacked a policy of the same name.
Directory-backed providers such as Active Directory always emit groups
like `DOMAIN\Domain Users` that can never be mapped, so group-based
authorization was impossible there.
Keep only policy names that resolve to an existing policy and are valid
in session claims, and reject the login only when none remain. The
signed `policy` claim still lists only resolved names, so request-time
evaluation and site replication receivers keep their invariant.
Refs #8163
* fix(oidc): only ignore unmapped groups-claim values
Limit the relaxed resolution to policy names derived solely from the
groups claim. Names from ROLE_POLICY, a dedicated CLAIM_NAME claim, or
an explicit IAM policy mapping must still all resolve.
Reject the login when a mapped name refers to an existing policy whose
name is not allowed in session claims: dropping it could remove an
explicit Deny and broaden access.
Claim-unsafe names are only checked against the in-memory policy cache
and never passed to storage-backed policy loading.
Refs #8163
* docs(oidc): document ignorable roles claim values and cover wiring
Roles claim values are merged into the canonical groups, so values
without a matching policy are ignored like groups-claim values. This
covers built-in provider roles such as Keycloak's `offline_access`.
Document that in the field and method docs and in the provider
requirements, and pin it with a test.
Assert that the OIDC authorization carries the group claim policies so
a regression in the wiring cannot silently disable the relaxation.
Refs #8163
* fix(oidc): ignore group policies only after confirmed absence
merge_policies drops names whose load fails, so a storage or decode
error for an uncached group policy was indistinguishable from a missing
one. The group name was then ignored and the session signed without it,
which could remove an existing Deny that request-time checks cannot
restore.
Add IamSys::policy_exists, which consults the cache and then storage and
reports false only for NoSuchPolicy while returning every other error.
The OIDC binding now ignores a group-derived name only after a confirmed
absence and rejects the login when a lookup fails.
Refs #8163
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
* feat(iam): expose stored bucket tags to OPA
* fix(ci): lint bucket tag fixtures and register new e2e tests
* fix(ecstore): restore strict Clippy compatibility on Rust 1.99
Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 537c15277a)
* fix(deps): replace yanked yoke-derive release
(cherry picked from commit b54b32b8a9)
* perf(iam): borrow unchanged OPA bucket tag conditions
Reuse clean conditions for bucketless and untagged OPA decisions. Keep the existing clone-and-sanitize path for tagged requests, reserve enrichment capacity, and preserve authoritative lookup errors. Cover borrowed input, reserved namespaces, immutable request state, and both paths in the OPA input regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(iam): clarify OPA bucket policy fallback semantics
Document that an OPA tag mismatch remains an implicit IAM denial which a bucket-policy Allow can supplement. Extend the existing S3 contract test to contrast a normal OPA denial with corrupt bucket metadata, whose lookup error must abort the same Allow fallback.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): isolate durable progress fixtures from competing IO
Reserve the nextest execution budget for schedule, streaming lock, retirement, MRF checkpoint/crash, and native scanner progress scenarios. Preserve their internal concurrency, existing deadlines, typed failure assertions, and zero-retry policy in default and CI profiles.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: rjregenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix: emit NextPartNumberMarker only when ListParts is truncated
ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.
Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.
Fixes#8208
* fix(s3): honor sparse ListParts markers and verify termination
Resume part listings at the first part above the numeric marker, even
when that marker is absent. Use binary search over the sorted part
numbers and retain the existing exact-tail empty-slice path.
Add storage, XML, and real AWS SDK paginator regressions for empty and
terminal pages, sparse markers, and multipart completion integrity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): register ListParts E2E in selection manifests
Register the intentional paginator regression in the smoke selection
and Linux full-suite inventory. Audit the actual CI test identities
and verify that removing the new case restores both prior digests.
Keep the profile filters, strict digest checks, and Darwin full entry
unchanged. The latter has an independent pre-existing selection drift.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(scanner): preserve checkpoint children in large prefixes
* fix(ecstore): restore strict Clippy compatibility on Rust 1.99
Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 537c15277a)
* fix(deps): replace yanked yoke-derive release
(cherry picked from commit b54b32b8a9)
* chore: keep scanner repair separate from CI corrections
* test(scanner): verify large-prefix resume and compaction lifecycle
Exercise Normal and Deep scans across cycle and leader changes. Validate
persisted frontiers, bound replay to interrupted boundary objects, and
prove that a completed same-plan sweep restores compaction without losing
object or byte totals.
Refs: rustfs/backlog#2710
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Lindroos Hsu