Use the seqlock version as the publication fence and keep payload reads relaxed while validating the final version. This reduces ordering overhead in disk metric recording and snapshot collection without changing the rolling-window contract.
Co-authored-by: heihutu <heihutu@gmail.com>
fix(ecstore): classify system metadata volume failures
Preserve retryable quorum errors when system metadata reads or writes encounter missing volumes, and cover the create-bucket data-usage path with regressions.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(kms): restore persisted configuration after restart
* docs(kms): cover the reload route and startup load states
The admin contract matrix pins every dynamic KMS route for the rc and
console handoff, so the new POST /kms/reload needs a row there, and the
reload response reuses the configure snapshot shape rather than adding a
wire type. The observability runbook gains the operator procedure the
reload exists for: telling a load_failed startup apart from a server
that was never configured, and recovering without resubmitting secrets.
fix(replication): surface per-object failures and abort orphaned multipart uploads
Replication could mark an object FAILED with no server-log line naming
the object: the target-offline skip paths logged at debug without the
object key, and several failure branches omitted the key entirely. A
failed multipart transfer also leaked its incomplete upload on the
target, since nothing ever aborted it after CreateMultipartUpload
succeeded.
Log the offline skips at warn with the object key (they report the
object FAILED, matching the per-object put_object failure level), add
the object field to the remaining failure branches, and abort the
target-side multipart upload best-effort on any post-create failure
without masking the original transfer error.
Fixes#6825
fix(replication): never mint delete markers when replicating a version purge
Heal/resync/MRF rebuilds of a delete-marker version purge carry
delete_marker: true together with a purge-shaped entry. Passing that flag
straight into replication_delete_remove_options made the target DELETE
omit the versionId (marker-creation semantics), so a generic S3 target
that ignores the internal source-version headers minted a fresh delete
marker on every retry instead of purging one — the marker count on the
target grew monotonically (rustfs#6823).
- Gate marker-creation semantics on the new pure helper
delete_replication_creates_marker (delete_marker && !version purge) so
a purge always addresses the exact version.
- Stop falling through to the marker-creation send when the pre-send
source delete-marker verification fails with a transient error; fail
the entry instead so the MRF replay / heal scanner retries without
minting a marker on the target.
- Pin the purge-shape contract with unit tests in
crates/replication/src/delete.rs.
Also replace deprecated Atomic::fetch_update calls with try_update so the
current Rust toolchain keeps lint and CI jobs warning-clean.
Co-authored-by: heihutu <heihutu@gmail.com>
Triages the three divergences backlog#2054 found between the audit and
notify default KVS tables, cross-checked against MinIO upstream
(internal/logger/config.go, internal/config/notify/parse.go):
- webhook: audit's extra batch_size/max_retry/retry_interval/http_timeout
keys match MinIO's DefaultAuditWebhookKVS byte-for-byte, while notify's
table matches MinIO's notify DefaultWebhookKVS (which lacks them).
Intentional, not a copy/paste gap — documented with a doc comment on
each table instead of changed.
- mqtt: audit's stronger QoS/keep-alive/reconnect defaults have no MinIO
precedent (MinIO's audit logging has no MQTT target at all), while
notify's 0/0s/0s defaults match MinIO's DefaultMQTTKVS exactly.
Documented as an intentional RustFS-original choice, not changed.
- auth_token hidden_if_empty: audit had false, notify had true, with no
MinIO precedent either way (this KVS version has no per-key hidden
flag upstream). Fixed audit to true, matching notify and every other
sensitive key in both files (MQTT_PASSWORD, *_TLS_*). Non-empty tokens
were already redacted identically on both sides via ends_with("_token")
pattern matching in config_admin.rs — this only changes how an *unset*
audit webhook auth_token renders in admin config output (omitted
instead of shown as an empty value).
Refs rustfs/backlog#2054
TierAzure.storage_class and .sp_auth round-trip faithfully through the
admin API and on-disk config (ExternalTierAzure encode/decode in
tier.rs), so an operator can configure them, read them back via
ListTier, and never learn they do nothing. They are dropped only at the
WarmBackendAzure construction boundary: the Azure warm backend goes
through the same S3-compatible TransitionClient as every other
provider and has no Azure Blob-native client or Azure AD dependency
(confirmed: no azure_* crate anywhere in the workspace), so neither
field can actually be honored today. MinIO's reference implementation
(cmd/warm-backend-azure.go) treats both as first-class: storage_class
sets the blob access tier on every PUT, and sp_auth is a full
alternative to access/secret-key auth via azidentity, mutually
exclusive with it.
Rather than the larger, riskier options (add a native Azure SDK
dependency and a parallel non-S3 client path, or break the persisted
config format by removing the fields), this closes the silent-failure
gap with the minimal safe fix: TierConfigMgr::add now rejects an Azure
tier config with either field set, before backend construction,
returning ERR_TIER_INVALID_CONFIG with an explicit message instead of
accepting and ignoring. The fields stay in the config type (no format
break); already-persisted tiers with these fields set are grandfathered
in un-rejected (edit does not touch sp_auth or storage_class either).
Full support remains a larger follow-up if ever prioritized.
Also removes TierAzure::is_sp_enabled(), which had zero callers
repo-wide (backlog#2055 flagged this) and would have been misleading
dead weight once this decision was made — reusing it for the new gate
would also have been wrong, since it requires *all three* sp_auth
fields non-empty (&&), while the gate must reject on *any* one being set.
Refs rustfs/backlog#2055
(cherry picked from commit 8d148c4e9b2507a1c5075e3d9513adb8b5851ef5)
Merge newer partial observed usage into the complete authoritative admin baseline instead of replacing the full bucket set.
Keep the merged view partial and non-converged so shared consumers do not treat it as quota-authoritative.
Co-authored-by: heihutu <heihutu@gmail.com>
A PUT with HTTP preconditions took the per-object namespace write lock
before ingesting the request body and held it until commit, so any
concurrent read of the same object queued behind client-paced body
ingestion until the 5s acquire timeout and surfaced as 503. Exposed as
a deterministic S3 Implemented Tests gate failure when #6770 routed
1 MB conditional writes onto the streaming path (rustfs/backlog#2074).
Keep a lock-free advisory precondition check before the body for fast
412/404, and evaluate the authoritative check under the put_object
commit lock, reusing the deferred shape data movement already uses.
Reads during ingestion now return the last committed version, and a
precondition invalidated mid-stream fails closed with 412 at commit.
Validate lifecycle tier references through the tier reference proof path, preserve S3 list CommonPrefix XML compatibility, and make GetObject audit completion use real S3 error status codes.
Co-authored-by: heihutu <heihutu@gmail.com>
refactor(ecstore): huaweicloud/tencent reuse the shared S3 constructor
Migrates the Huaweicloud and Tencent tier warm backends onto the shared
S3-compatible constructor (backlog#2040). Also makes the shared
constructor's outbound-URL validation injectable per provider
(S3CompatibleWarmBackendParams::validate_endpoint) so it can centralize
rustfs/rustfs#6764's SSRF check for the providers that don't need an
exception, while accommodating rustfs/rustfs#6773's RustFS-specific
debug-only loopback opt-in without weakening the other six providers.
Updates scripts/error-other-format-baseline.txt: the one ::other(format!)
call site moves from the two per-provider files into the new shared
call site in warm_backend.rs (net call-site count unchanged).
Refs rustfs/backlog#2042
The heal manager carried its own byte-identical copy of the foreground pressure type and threshold computation that ecstore's data-movement backpressure also carries, so every change to the admission-utilization rules had to be mirrored by hand across two crates. The shared `ForegroundPressure` and `foreground_pressure` added to `rustfs-concurrency` now own that logic, and heal already depends on that crate, so this removes the duplicate without adding a crate edge.
`mainline_throttle_active` keeps the parts that are specific to this call site: the `mainline_throttle_enable` and both-thresholds-zero short circuit that avoids touching the provider at all, the optional-provider unwrap, and the heal-side threshold fields. Everything downstream is untouched — the `reason()` labels `foreground_read_pressure`, `foreground_write_pressure`, and `foreground_pressure` are byte-identical to the removed implementation, so the `rustfs_heal_mainline_throttle_total` reason label and the `heal_mainline_throttle` log fields keep their observability contract.
Refs rustfs/backlog#2049
(cherry picked from commit ec491bcbd8939e5978cd94f9a44cffb70d09fade)
(cherry picked from commit e800f29d6806591689204b3712300800b480beae)
Co-authored-by: houseme <housemecn@gmail.com>
refactor(ecstore): use shared ForegroundPressure for data movement
The data movement backpressure module carried its own byte-identical copy of ForegroundPressure, its reason() label mapping, and the foreground utilization computation. rustfs-concurrency now owns that logic as workload::ForegroundPressure and workload::foreground_pressure, so the local copy was a cross-crate synchronization point that could silently drift from the heal-side and admission-side behavior.
Delete the local type and computation and call the shared function instead. The call site keeps what is specific to data movement: the config.enabled short circuit, the optional provider unwrap, and the read/write threshold percentages read from DataMovementBackpressureConfig. The reason() labels emitted into the rustfs_data_movement_backpressure_total metric and the data_movement_backpressure log event are unchanged, as are the existing tests and their assertions.
Refs rustfs/backlog#2048
(cherry picked from commit 6a26e144e06ced53a8dfd1712ab7aa24589646ff)
(cherry picked from commit ab5ab417e80179265c32b22a5e671eac0b9e43ae)
refactor(ecstore): migrate notify default KVS to shared constructors
The amqp, nats, pulsar, redis, postgres, kafka and mysql default KVS tables in config/notify.rs duplicated the corresponding tables in config/audit.rs literally, leaving seven cross-file sync points that a future default or key-order edit had to keep aligned by hand. Replace those seven table bodies with calls to the shared constructors added in config/target_defaults.rs, passing the notify-side literals where the two subsystems genuinely differ: NOTIFY_REDIS_DEFAULT_CHANNEL for the redis channel and "rustfs_events" for the mysql table.
Key order is part of the admin config contract, so this is a pure restructuring: for all seven tables the ordered key sequence and every key's value and hidden_if_empty flag are unchanged.
DEFAULT_NOTIFY_WEBHOOK_KVS and DEFAULT_NOTIFY_MQTT_KVS are deliberately left untouched. Those two tables really do diverge between audit and notify, so folding them into shared constructors would change runtime behavior; the divergence is tracked separately in rustfs/backlog#2054.
Refs rustfs/backlog#2046
(cherry picked from commit 6df9b53027ef2f0cf9aa7b82ecb2af8c5108f11f)
(cherry picked from commit 14bbf756bea2ee6daacbfdc7d7a452effa649cff)
refactor(ecstore): migrate audit KVS defaults to shared constructors
The amqp, nats, pulsar, redis, postgres, kafka and mysql default KVS tables in config/audit.rs duplicated the corresponding tables in config/notify.rs, leaving seven cross-file sync points where a default could silently drift between the two subsystems. Build them from the shared constructors added in config/target_defaults.rs instead, passing in the two literals that are genuinely audit-specific: the redis pub/sub channel (AUDIT_REDIS_DEFAULT_CHANNEL) and the mysql destination table ("rustfs_audit_logs").
Key order, every default value and every hidden_if_empty flag are preserved exactly, since the key order drives the order admin config output lists keys in. DEFAULT_AUDIT_WEBHOOK_KVS and DEFAULT_AUDIT_MQTT_KVS are left untouched: those two tables really do differ from their notify counterparts, so unifying them would change runtime behavior.
Refs rustfs/backlog#2045
(cherry picked from commit 4de580e6d8901485eef268191924e035322d4d4e)
(cherry picked from commit e5e301fa78e7f27ffa85b05cf5e7962301a5ff00)
- common.rs gains an AdminTransport knob (Signed | Awscurl) with admin_execute_at plus three family wrappers: admin_create_user_via, admin_add_canned_policy_via, admin_attach_user_policy_via; the existing admin_create_user now delegates over the Signed transport.
- Deleted the four signed admin request clones in admin_mfa_test, admin_auth_test, reliant/tiering, and inline_fast_path_cluster_test; each keeps a thin local wrapper over common::admin_request so call sites keep their Option<&str> body shape.
- Deduped the notification_webhook signer onto common::signed_request and the webdav_core signer plus its three admin helpers onto the shared _via helpers.
- Consolidated the S3-client-with-credentials builders: admin_auth s3_client_with, existing_object_tag user_client/sts_session_client, bucket_policy_check create_user_client, and the create_user_s3_client copies in group_delete_test and replication_extension_test now delegate to create_s3_client_with_credentials / build_test_s3_config; replication_extension admin_add_canned_policy and admin_attach_policy_to_user route through the _via helpers on the Signed transport.
- The awscurl-gated suites (existing_object_tag_policy, bucket_policy_check, policy/policy_variables) keep going through the external awscurl binary via AdminTransport::Awscurl, preserving their wire behavior.
Part of rustfs/backlog#1846 (cluster 2).
* fix(s3): round-trip null-version delete-marker identity through listing and delete responses
On a versioning-suspended bucket, a null delete marker's identity was lost on the way back to the client at three points (issue #6745): ListObjectVersions advertised the marker's VersionId as the literal nil UUID instead of null; deleting by that id succeeded but the DeleteObjects/DeleteObject response reported the identity as null with no way to correlate it to the request; and the response lacked DeleteMarker/DeleteMarkerVersionId because the marker-ness comparison mixed the client-facing identity (Some(nil)) with the storage identity (None), so the removal also mis-recorded accounting and fired DeleteMarkerCreated semantics on later paths.
- Listing (bucket_usecase, s3_api/bucket, build_list_versions_next_marker) now maps the synthesized nil UUID to the literal null everywhere it reaches the wire, and VersionMarker::parse folds a nil-UUID marker from older listings into VersionMarker::Null so pagination resumes correctly.
- delete_objects normalizes both sides of the marker-ness comparison via delete_file_info_version_id (matching the adjacent explicit_delete_marker admission check) and reports DeleteMarkerVersionId as null for an explicit null-marker removal.
- resolve_delete_version_state reports delete_marker for an explicit-version delete whose target is a delete marker even when the bucket is versioning-suspended, fixing x-amz-delete-marker on the single-object path.
- The DeleteObjects response entry echoes the version identity the request addressed for marker removals, marker-removal accounting no longer records a marker creation, and notification events fire DeleteMarkerCreated only for actual marker creation.
Fixes#6745
* fix(s3): keep null-marker removal write shape undeleted and report marker semantics response-side
The first cut marked the storage delete request deleted for a null-marker removal, which FileMeta::delete_version interprets as the suspended-bucket delete-mints-a-marker write and re-creates the marker just removed. Carry marker-ness to responses via explicit_delete_removed_marker (single path) and a response-only branch flag (batch path) instead, keeping every storage write shape byte-identical to the pre-fix behavior. Adds an embedded end-to-end regression test covering the full issue #6745 round trip.
WarmBackendS3::new already rejects loopback, private, link-local, and
cloud metadata-service endpoints via validate_outbound_url, but the
Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS, and GCS warm
backend constructors built their transition clients directly from
conf.endpoint without the same check.
The endpoint comes from the AddTier admin API, gated only by
SetTierAction, which can be a narrower IAM grant than root. Any
principal holding it could point one of these eight tier types at an
internal address (loopback, RFC1918, link-local, or a cloud metadata
IP) and have the server issue authenticated outbound requests to it, a
server-side SSRF vector that the S3 and Wasabi tier types were already
closed against.
Apply the same validate_outbound_url check at construction time for
all eight providers, before any credentials or network client are
built, mirroring the existing WarmBackendS3 pattern. GCS keeps its
default-endpoint behavior when conf.endpoint is empty and only
validates an explicitly configured endpoint.
Add a regression test per provider asserting that a loopback endpoint
is rejected before any backend/network setup, matching the existing
WarmBackendS3 coverage.
Update the error(format!) ratchet baseline: these are one-shot admin
tier-configuration validation errors returned once per AddTier call,
not per-disk I/O errors that flow through reduce_errs quorum
aggregation (backlog#1845), so the new ::other(format!) call sites do
not introduce a quorum-bucketing hazard. They mirror the pre-existing,
already-baselined warm_backend_s3.rs call site.
TransitionClient::new() in crates/s3-client/src/transition_api.rs computes
trailing_header_support = opts.trailing_headers && override_signer_type == SignatureV4,
but override_signer_type is hardcoded to SignatureDefault at construction
and never mutated afterwards, so the expression is always false regardless
of opts.trailing_headers. The resulting field also has no live reader: its
only reference is inside PutObjectOptions::validate() in
crates/s3-client/src/api_put_object.rs, which is itself
#[allow(dead_code, reason = "MinIO-parity ... no caller in this port")],
and even there the reference to trailing_header_support is commented out.
So trailing_headers: true in the seven warm_backend_*.rs constructors has
never had any effect on request signing or chunked/trailing-header
behavior (stream_sha256 signing is gated separately by
metadata.stream_sha256 && !self.secure). Remove the misleading dead
configuration from the seven provider constructors so it doesn't look
like intentional, load-bearing behavior to future readers.
Found during adversarial self-check while implementing rustfs/backlog#2040 (out of that issue's scope).
* refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case
`crates/utils/src/http/metadata_compat.rs` owns the internal metadata key helpers, including `starts_with_ignore_ascii_case`. Two files carried their own byte-identical copies of that predicate: `SetDisks::starts_with_ignore_ascii_case` in ecstore and a free function in the S3 options layer. Both drive internal metadata key classification (`internal_metadata_suffix` and quorum hashing on one side, `should_skip_object_metadata_key` and `is_reserved_user_metadata_key` on the other), so keeping three implementations of one predicate is an avoidable drift risk on a path that decides whether an internal key is treated as user metadata.
Delete both local copies and call the canonical implementation. Every prefix used at these call sites is an ASCII constant or literal, where the canonical byte-slice comparison and the removed `str::get(..n)` form are equivalent; that equivalence was checked differentially over 4.6M (key, prefix) pairs, including keys with multi-byte characters straddling the prefix boundary. No other logic in `internal_metadata_suffix` or `should_skip_object_metadata_key` changed.
Add regression tests on both sides pinning the two properties the switch depends on: internal prefixes match case-insensitively (a mixed-case `X-RustFS-Internal-*` key stays internal), and keys shorter than a prefix never match (they stay ordinary user metadata).
Refs rustfs/backlog#2051
* fix(rustfs): avoid typos-checker false positive in prefix-length test
The test literal "x-rustfs-encryptio" (a deliberate truncation of the
x-rustfs-encryption- prefix, used to assert that a key shorter than every
internal prefix falls through to user metadata) reads as a likely typo of
"encryption" to the repo's typos CI check. Derive it from
RUSTFS_ENCRYPTION_PREFIX via slicing instead of a hand-typed literal, which
both satisfies the linter and ties the truncation to the real constant
instead of a copy-typed guess.
Refs rustfs/backlog#2051
cxymds