Keep internode JSON compatibility fields unless operators explicitly confirm fleet-wide msgpack-only readiness. This prevents a single legacy rollout flag from emptying JSON fields in mixed-version clusters where older peers may still read the legacy JSON payload.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ilm): bound manual transition duration
Add maxDurationSeconds handling for manual transition runs so operators can bound long scoped scans without changing the existing enqueue_only job contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ilm): document manual transition run limits
Document the enqueue_only manual transition contract, secret-handling guidance, and best-effort duration budget while pinning the new duration flag in the e2e response model.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(ilm): report manual transition backfill outcomes
Add scoped lifecycle transition backfill reporting for backlog #1478 and expose enqueue outcomes needed by #1479 without changing the existing scanner/compensation bool API.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(admin): add manual transition run endpoint
Add a bounded POST /rustfs/admin/v3/ilm/transition/run API for backlog #1477 and cover the route, policy, query parsing, and partial status contract needed by #1481. Console operations from #1480 are intentionally left for a later client integration.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(ilm): allow manual transition resume markers
Accept additive marker and versionMarker parameters on the bounded manual transition run API so clients can continue from a partial report without changing the existing default scan behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): harden manual transition partial reports
Preserve the null-version cursor contract, stop manual scans on enqueue pressure without skipping the failed object, and keep raw resume markers out of admin JSON responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ilm): add manual transition e2e coverage
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(ilm): keep transition enqueue hot path direct
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A federated OIDC / AssumeRoleWithWebIdentity login derives parent_user from the external identity via session_identity() (username -> email -> sub -> oidc-user-unknown). At IAM request time a temporary credential whose parent_user equals the root access key is treated as owner (see rustfs/src/admin/auth.rs and the rustfs_iam owner resolution used by the policy engine and service-account paths). A federated identity whose display name happens to equal the root access key would therefore be silently granted full owner access on a string match alone.
Reject such issuance up front: immediately after parent_user is derived, and before credential generation, set_temp_user, and the site-replication hook, deny the binding when parent_user equals the root access key. Both federated flows funnel through the single DefaultFederatedSessionBinding::bind -> issue_credentials chokepoint, so this covers browser OIDC callback and AssumeRoleWithWebIdentity alike.
- rustfs/src/admin/service/federated_identity.rs: issue_credentials resolves the root access key via current_action_credentials() and rejects with FederatedSessionBindingError::InvalidRequest on collision. Added the pure helper parent_user_is_reserved(parent_user, root_access_key) so the collision decision is unit-testable without process globals; the comparison is exact and case-sensitive, mirroring the request-time owner comparison.
This is the federated-principal / root isolation pre-work (batch 0A, the immediate issuance-time block) from the OIDC review in rustfs/backlog#1437. Layer 2 (a signature-protected federated credential origin that forces is_owner=false at request time regardless of the parent_user string) is a separate follow-up. Already-issued colliding credentials continue on their existing TTL and should be revoked proactively. Legitimate federated identities are unaffected; only an exact collision with the configured root access key is denied.
* perf(ecstore): seek encrypted multipart Range GETs to the covering part boundary on the Legacy rio v1 backend
Phase A of https://github.com/rustfs/backlog/issues/1316. Encrypted Range GETs on the default (non rio-v2) backend previously planned storage_offset=0, storage_length=oi.size and discarded the decrypted prefix, so a small Range on a large SSE object read, erasure-decoded, and decrypted the whole object. Eligible multipart objects now seek to the covering part boundary using only the per-part size/actual_size metadata facts; the multipart decrypt reader already handles streams starting at any part boundary, so rio and the on-disk format are untouched. Single-part objects, compressed payloads, defective parts tables, and zero-length ranges keep the previous full read, and RUSTFS_ENCRYPTED_RANGE_SEEK=false restores it globally. A new histogram rustfs_get_encrypted_range_read_amplification plus a full|part_seek path counter record the physical/plaintext amplification at the ReadPlan decision point.
* fix(ecstore): guard encrypted multipart range seeks
* fix(io-metrics): align encrypted range metric names with the rustfs_io_ prefix
Every other metric in rustfs-io-metrics uses the rustfs_io_ prefix; the
two encrypted-range-seek metrics were the only exception. Rename before
first release so dashboards never see the unprefixed names.
Stop the OIDC subsystem from writing credential-grade secrets into logs and returned errors.
- crates/iam/src/oidc.rs: redact sensitive header values (authorization, proxy-authorization, cookie, set-cookie) in format_http_headers, emitting only name and length; drop the raw request/response body from the DEBUG events (keep byte length); stop logging and stop splicing the raw token response body into the error returned on token_response_parse_failed (the TokenResponseBodyShape summary and length are retained); remove the now-unused format_http_body helper.
- rustfs/src/admin/handlers/oidc.rs: stop logging the raw authorization code and state on the code-exchange error path (code_len/state_len are kept).
This is the OIDC log/error redaction pre-work (batch 0B) from the OIDC review in rustfs/backlog#1437. It changes diagnostic content only; HTTP/STS status codes and legitimate request results are unchanged.
* fix(admin): confine service-account parent to caller scope (GHSA-5354)
AddServiceAccount gated creation only on CreateServiceAccountAdminAction and did not constrain the targetUser (the new service account's parent) to the caller's own scope. Combined with the deliberate root-credential exemption in the existence check, a non-owner principal holding that admin action could create a service account parented to the root credential, which then authenticates as owner via prepare_service_account_auth. This is the direct-handler analogue of the ImportIam parent-scope check (GHSA-566f), which already enforces the invariant via imported_service_account_parent_allowed.
Enforce the same invariant through add_service_account_parent_within_scope: a non-owner may create a service account only for itself (or, for a derived credential, its own parent); owners retain cross-user creation. Add named regression test ghsa_5354_non_owner_service_account_parent_confined_to_scope.
Refs GHSA-5354-r3w2-34m8.
* test(admin): pin GHSA-5354 scope guard for derived credentials
The GHSA-5354 fix (#5141) added `add_service_account_parent_within_scope` and a
named-boundary test, but that test only covers the self-scoped case with
`req_user == req_parent_user`. A derived credential — a service account or STS
token holding CreateServiceAccountAdminAction — has `req_user` (its own key)
distinct from `req_parent_user` (its parent), which is the realistic attacker
shape and was left unexercised.
Add a test that pins the guard's allow set to exactly `owner || is_svc_acc`
across that derived-credential shape, including a derived non-owner aiming at the
root credential, so a later change to either the guard or the `is_svc_acc`
rewrite cannot silently let a derived non-owner escape its own parent's scope.
Also note at the check site that the guard is evaluated on the original
`target_user`, before the derived-credential rewrite to `req_parent_user`.
Refs GHSA-5354-r3w2-34m8.
PRs #5141 and #5142 fixed GHSA-5354 (service-account parent-scope enforcement) and GHSA-3ppv (versioned object-read authorization) but did not add the advisories to the security-regressions inventory that repo policy requires for every fixed GHSA. Add both rows to the advisory -> test mapping (advisory, class, fix PR, named regression test, layer) and list ghsa_5354_* / ghsa_3ppv_* under the default-CI unit-test execution map, since both tests live in the rustfs lib target and run in the default `cargo nextest run --profile ci --all --exclude e2e_test` pass.
get_object, the CopyObject source, and the UploadPartCopy source authorized reads that name an explicit versionId against s3:GetObject rather than s3:GetObjectVersion. A principal holding s3:GetObject but not s3:GetObjectVersion could therefore read historical object versions. get_object_attributes was already version-aware; these three paths were not.
Add a shared versioned_read_action helper that selects s3:GetObjectVersion when a version is named and s3:GetObject otherwise, and apply it to the three read/copy-source authorization sites. Add named regression test ghsa_3ppv_versioned_read_selects_get_object_version_action.
The ActionSet::is_match GetObjectVersion->GetObject compatibility mapping is intentionally retained for now; its removal is gated on a follow-up audit of the remaining version-aware read paths (HeadObject, GetObjectAcl, tagging). It does not affect this fix, which only prevents a GetObject grant from satisfying a versioned read.
Refs GHSA-3ppv-fx5m-m749.
AddServiceAccount gated creation only on CreateServiceAccountAdminAction and did not constrain the targetUser (the new service account's parent) to the caller's own scope. Combined with the deliberate root-credential exemption in the existence check, a non-owner principal holding that admin action could create a service account parented to the root credential, which then authenticates as owner via prepare_service_account_auth. This is the direct-handler analogue of the ImportIam parent-scope check (GHSA-566f), which already enforces the invariant via imported_service_account_parent_allowed.
Enforce the same invariant through add_service_account_parent_within_scope: a non-owner may create a service account only for itself (or, for a derived credential, its own parent); owners retain cross-user creation. Add named regression test ghsa_5354_non_owner_service_account_parent_confined_to_scope.
Refs GHSA-5354-r3w2-34m8.
* fix(site-replication): replicate object-lock retention and legal hold
Object Lock did not survive site replication, so a WORM-protected object was
either missing from the peer entirely or present there unprotected. Two
independent defects caused this.
1. The site-replicator service account policy granted no object-lock actions.
Any replicated object carrying x-amz-object-lock-mode /
x-amz-object-lock-retain-until-date was rejected by the peer with
AccessDenied, so an object created under a retention rule never replicated
at all. Grant s3:GetObjectRetention, s3:PutObjectRetention,
s3:GetObjectLegalHold and s3:PutObjectLegalHold. s3:BypassGovernanceRetention
is deliberately NOT granted: replication must never be able to erase a
retained version on the peer.
2. PutObjectRetention and PutObjectLegalHold only rewrote object metadata. The
object PUT path is what normally computes a replication decision and
schedules replication, so a lock applied after upload stayed local and the
peer kept its previous, unprotected state. Schedule replication explicitly
after both metadata writes.
Verified live on a two-site cluster: an object created under a COMPLIANCE
retention rule now replicates, the peer copy reports COMPLIANCE, a
version-scoped delete on the peer is refused, and the content matches by MD5.
Retention and legal hold applied after upload propagate to the peer within
about 15s, and the replicated copy is likewise undeletable. No AccessDenied
replication errors remain.
Adds a regression test asserting the policy allows the four object-lock actions
and still denies governance bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(site-replication): persist pending marker before object-lock metadata commit
Review follow-up on the object-lock replication fix.
PutObjectRetention and PutObjectLegalHold scheduled replication but did not
persist a pending replication marker first, unlike the canonical object PUT
path. If the process restarted between the metadata commit and the replication
worker write-back, the in-memory task was lost and the object carried no
Pending/Failed marker, so the scanner heal skipped it and the peer silently
stayed unprotected.
Both handlers now mirror the PUT path ordering: compute the replication
decision once BEFORE the commit, fold SUFFIX_REPLICATION_STATUS and
SUFFIX_REPLICATION_TIMESTAMP into the eval_metadata passed to
put_object_metadata, then schedule after the commit. eval_metadata is merged
into the object metadata, so the lock fields are preserved. When the object
info cannot be read the decision is empty and nothing is scheduled.
Also aligns the two handlers to build their options the same way, and notes at
both schedule sites that this triggers a full object re-upload because
metadata-only replication is not implemented yet.
Adds an integration test asserting both handlers compute a replication decision
exactly once, so a regression that drops the scheduling is caught.
Verified live on a two-site cluster: applying retention while the peer is down
now leaves Replication Status: PENDING on disk at commit time, and once the
peer returns the lock propagates in about 15s and the status becomes COMPLETED.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
On macOS with DNS interception services, .invalid TLD domains resolve
to an interception address (e.g. 198.18.16.173) instead of failing DNS
resolution. The health probe then classifies the error as Unreachable
rather than DnsFailure. Accept both outcomes since both are correct
non-reachable error classifications.
Keep remote peer topology slots observable when peer client construction cannot build a dialing client, and publish admin server_info cache/failure state only after a complete probe round.
Covers rustfs/backlog#1426 and rustfs/backlog#1430.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(admin): use raw topology host for peer mapping
Backlog: rustfs/backlog#1427
Keep endpoint topology membership keyed by raw host:port when mapping peers to grid hosts, while preserving DNS resolution only for the PeerRestClient dialing host.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(admin): reconcile v3 info with topology
Backlog: rustfs/backlog#1425
Backlog: rustfs/backlog#1428
Synthesize additive unknown server rows from configured endpoint topology before v3 backend counters are computed, and add exact drive-identity coverage tests so balanced totals cannot hide a missing member.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(admin): collapse topology reconciliation
Build a single endpoint topology index for v3 admin server reconciliation and completeness reporting, and reuse the raw peer/grid mapping when constructing peer clients.
Backlog: rustfs/backlog#1424
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Update selected workspace dependencies and lockfile entries.
Keep async-nats and rcgen on explicit feature sets while preserving the RustFS targets and TLS test surfaces.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): gate exact remote version consumption
Reject non-empty remote tier versions before transitioned GET and remote delete backend I/O when the tier backend does not support exact version operations.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(tier): split mock remote version validation fault
Separate one-shot mock remote version validation failures from persistent unsupported-backend behavior so cleanup durability tests can still verify exact-version recovery while #1358 fail-closed gate tests keep asserting no backend I/O.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Advance transition transactions to UploadOutcomeUnknown before remote tier PUT so response-loss windows can be recovered through provider-authoritative probing.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep coordinator Prepared intents durable when peer abort recovery fails, then make reload retry abort or commit based on the persisted tier config digest.
Co-authored-by: heihutu <heihutu@gmail.com>
A single instance with a malformed `enable` value (e.g. the typo `enable`
instead of `enabled`/`on`) made every configured notification/audit target
of every type fail to load. `create_targets_from_config_with_store_mode`
collected instance configs through the strict `try_collect_target_configs`,
whose `.collect::<Result<Vec, _>>()` short-circuits on the first error and
drops all siblings; the surrounding `?` then aborted the loop over every
plugin type.
This contradicted the function's own documented contract ("Creation is
fault-isolated per instance") and was inconsistent: target construction
failures were already isolated into the `failures` accumulator, only config
collection failures were fatal.
Add `collect_target_config_results[_from_env]`, which returns the enabled
`(id, config)` pairs plus a per-instance failure summary, and use it in the
create path. Per-instance collection errors now flow into the same
`failures`/`creation_failures` accumulator as construction failures: healthy
siblings and other target types still load, while the malformed instance is
surfaced so the notify lifecycle stays non-converged/retryable and an Admin
write cannot report a false success. Audit likewise keeps logging through
its valid targets instead of going fully Stopped.
Follow-up to #5088.
* feat(tier): add transition candidate probe contract
Add a fail-closed WarmBackend probe contract for provider-authoritative transition candidate state. Default providers report Unsupported, while the shared mock backend can now model missing, unversioned, and exact-version candidates for follow-up recovery tests.
This is a forward-compatible foundation for #1352/#1358 recovery work and does not change production cleanup behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(tier): probe transition candidates from providers
Implement provider-authoritative transition candidate probing for S3-family warm backends by querying ListObjectVersions with exact-key filtering and fail-closed classification for delete markers, multiple versions, truncation, and unknown versioning state.
This keeps non-S3 providers on the default Unsupported probe result and forwards MinIO, RustFS, and R2 through the S3 probe implementation.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
houseme