The add preflight unconditionally rejected any topology with data on
more than one site, which made `replicate remove` a one-way door: a DR
cluster whose sites both hold data could never be re-paired, and the
only way out was wiping one side by hand.
Admit a multi-non-empty add when every bucket held by more than one
requested site is provably safe to merge through the existing
backfill/resync convergence: versioning must be Enabled on every holder
(so a same-key object from the peer lands as another version instead of
replacing the only copy) and object-lock enablement must match (lock
cannot be toggled after bucket creation). Incompatible adds are still
rejected, now with the operator recovery steps (empty one side, re-run
replicate add, resync) instead of a bare refusal. Bucket configs that
fail to decode fail the preflight closed.
A committed add now also clears this site's own half-finished
pending_remove, mirroring the join receiver (rustfs/rustfs#5963);
otherwise the reconcile tick would replay the stale removal against the
freshly re-paired peer and dismantle the new pairing.
Refs rustfs/backlog#2070
* fix(admin): replay recorded IAM deletions in site replication retry drain
An IAM deletion whose peer delivery failed during an outage window was
previously unrecoverable without a manual repair: the collapsed retry
entry carries no body, the snapshot resend cannot express "this entity
no longer exists", and the drain escalated the entry into a permanent
marker. The deleted user kept working credentials on the peer until an
operator intervened — a security exposure (backlog#2071).
Record the verbatim deletion body (user/policy/group-removal/
policy-mapping-clear/service-account) in the persisted state, in the
same transaction as the retry-event upsert. The drain now replays the
recorded deletions before the snapshot resend — snapshot-after ordering
restores any entity recreated locally in the meantime — and settles the
collapsed entry when its whole liability is provably replayed. Entries
that predate recording, merge with legacy rows, or overflow the
per-peer record cap keep the escalation semantics: only explicitly
recorded deletion events are ever replayed, never a cross-site diff.
The peer apply handlers become idempotent for deletion shapes (missing
policy/group/member tolerated, matching the existing user-delete
tolerance), so a replayed deletion that already converged settles
instead of wedging the drain. The IAM change hook now attempts every
peer instead of failing fast, so a multi-site broadcast books a retry
entry (and deletion record) for each unreachable peer rather than only
the first. Repair success and peer removal clear the affected peer's
records alongside the entries they accompany.
* fix(admin): probe recovered peers to lift retry drain backoff
A bucket created while a peer was unreachable accumulated three failed
deliveries and entered exponential backoff (2400s and up, capped at a
day). After the peer recovered, the reconcile tick's drain kept
skipping the entry until the backoff elapsed, so the site stayed
diverged — NoSuchBucket resync noise on the source, missing bucket on
the peer — for up to 24 hours with nothing else driving convergence
(backlog#2071, round-four R1.6).
Split reachability from replay: the drain now probes each peer whose
replayable backlog is held back only by backoff (one cheap devnull POST
per peer per tick) and promotes the backlog when the peer answers, so a
recovered peer converges at the next 600s tick. A failed probe advances
nothing — retry counts only move on real delivery attempts, keeping the
exponential schedule intact for a peer that is genuinely down. The base
backoff still floors re-attempts against a reachable peer that keeps
rejecting a delivery. SITE_REPLICATION_RETRY_FAILED_AFTER stays at 3:
the flag is retryStats visibility only, and with the probe in place an
early failed mark is a timely operator signal rather than a dead end.
The drain tick also logs an operator-visible warning whenever the queue
holds failed or escalated entries, instead of backing off in silence.
Align object-capacity refresh defaults with the production-oriented environment values and keep docs, script examples, and tests in sync.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(kms): restore persisted configuration after restart
* docs(kms): cover the reload route and startup load states
The admin contract matrix pins every dynamic KMS route for the rc and
console handoff, so the new POST /kms/reload needs a row there, and the
reload response reuses the configure snapshot shape rather than adding a
wire type. The observability runbook gains the operator procedure the
reload exists for: telling a load_failed startup apart from a server
that was never configured, and recovering without resubmitting secrets.
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
Merge newer partial observed usage into the complete authoritative admin baseline instead of replacing the full bucket set.
Keep the merged view partial and non-converged so shared consumers do not treat it as quota-authoritative.
Co-authored-by: heihutu <heihutu@gmail.com>
Validate lifecycle tier references through the tier reference proof path, preserve S3 list CommonPrefix XML compatibility, and make GetObject audit completion use real S3 error status codes.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(s3): round-trip null-version delete-marker identity through listing and delete responses
On a versioning-suspended bucket, a null delete marker's identity was lost on the way back to the client at three points (issue #6745): ListObjectVersions advertised the marker's VersionId as the literal nil UUID instead of null; deleting by that id succeeded but the DeleteObjects/DeleteObject response reported the identity as null with no way to correlate it to the request; and the response lacked DeleteMarker/DeleteMarkerVersionId because the marker-ness comparison mixed the client-facing identity (Some(nil)) with the storage identity (None), so the removal also mis-recorded accounting and fired DeleteMarkerCreated semantics on later paths.
- Listing (bucket_usecase, s3_api/bucket, build_list_versions_next_marker) now maps the synthesized nil UUID to the literal null everywhere it reaches the wire, and VersionMarker::parse folds a nil-UUID marker from older listings into VersionMarker::Null so pagination resumes correctly.
- delete_objects normalizes both sides of the marker-ness comparison via delete_file_info_version_id (matching the adjacent explicit_delete_marker admission check) and reports DeleteMarkerVersionId as null for an explicit null-marker removal.
- resolve_delete_version_state reports delete_marker for an explicit-version delete whose target is a delete marker even when the bucket is versioning-suspended, fixing x-amz-delete-marker on the single-object path.
- The DeleteObjects response entry echoes the version identity the request addressed for marker removals, marker-removal accounting no longer records a marker creation, and notification events fire DeleteMarkerCreated only for actual marker creation.
Fixes#6745
* fix(s3): keep null-marker removal write shape undeleted and report marker semantics response-side
The first cut marked the storage delete request deleted for a null-marker removal, which FileMeta::delete_version interprets as the suspended-bucket delete-mints-a-marker write and re-creates the marker just removed. Carry marker-ness to responses via explicit_delete_removed_marker (single path) and a response-only branch flag (batch path) instead, keeping every storage write shape byte-identical to the pre-fix behavior. Adds an embedded end-to-end regression test covering the full issue #6745 round trip.
* refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case
`crates/utils/src/http/metadata_compat.rs` owns the internal metadata key helpers, including `starts_with_ignore_ascii_case`. Two files carried their own byte-identical copies of that predicate: `SetDisks::starts_with_ignore_ascii_case` in ecstore and a free function in the S3 options layer. Both drive internal metadata key classification (`internal_metadata_suffix` and quorum hashing on one side, `should_skip_object_metadata_key` and `is_reserved_user_metadata_key` on the other), so keeping three implementations of one predicate is an avoidable drift risk on a path that decides whether an internal key is treated as user metadata.
Delete both local copies and call the canonical implementation. Every prefix used at these call sites is an ASCII constant or literal, where the canonical byte-slice comparison and the removed `str::get(..n)` form are equivalent; that equivalence was checked differentially over 4.6M (key, prefix) pairs, including keys with multi-byte characters straddling the prefix boundary. No other logic in `internal_metadata_suffix` or `should_skip_object_metadata_key` changed.
Add regression tests on both sides pinning the two properties the switch depends on: internal prefixes match case-insensitively (a mixed-case `X-RustFS-Internal-*` key stays internal), and keys shorter than a prefix never match (they stay ordinary user metadata).
Refs rustfs/backlog#2051
* fix(rustfs): avoid typos-checker false positive in prefix-length test
The test literal "x-rustfs-encryptio" (a deliberate truncation of the
x-rustfs-encryption- prefix, used to assert that a key shorter than every
internal prefix falls through to user metadata) reads as a likely typo of
"encryption" to the repo's typos CI check. Derive it from
RUSTFS_ENCRYPTION_PREFIX via slicing instead of a hand-typed literal, which
both satisfies the linter and ties the truncation to the real constant
instead of a copy-typed guess.
Refs rustfs/backlog#2051
The import_bucket_metadata handler carried eight match arms whose bodies were
byte-identical apart from the type a payload is validated against and the pair
of BucketMetadata fields it lands in, so every arm repeated the same warn! call
and the same metadata lookup. Fold them into one or-pattern arm backed by
apply_imported_bucket_config, where a single conf_name match owns both the
validated type and the destination field pair and can no longer drift apart.
Validation still runs before the metadata lookup, the warn! event, fields, and
label are unchanged, and the BUCKET_POLICY_CONFIG and BUCKET_QUOTA_CONFIG_FILE
arms keep their own handling. Regression tests drive the full mapping table:
each config file's payload lands only in the field it owns, an unparsable
payload leaves the field untouched, and a rejected entry does not stop the
remaining ones from being imported.
Refs rustfs/backlog#2052
houseme