* refactor(kms): share the DEK spec mapping and stop re-parsing opened envelopes
- generate_key_material is now the single spec->length mapping for every
backend that mints DEKs itself; the inline copies in the Static and Local
backends are gone, and ChaCha20 (32 bytes, same as AES_256) is accepted
uniformly instead of only by Static.
- The pub(crate) client decrypt of the Local, Vault KV2 and Vault Transit
backends returns (plaintext, master_key_id), so KmsBackend::decrypt no
longer re-parses the envelope it just opened (one JSON parse per SSE GET
instead of two, and unknown-field observability is no longer double-counted).
- Malformed-envelope parse failures now report CryptographicError("parse")
on all backends; Local was the last one mapping them to SerializationError.
- The four KmsBackend::generate_data_key adapters take fields out of
DataKeyInfo instead of cloning, dropping a redundant un-zeroized plaintext
DEK copy and a full ciphertext clone per call; a missing plaintext now
fails closed everywhere instead of returning an empty key on three of four
backends.
* test(kms): pin legacy header fallback, stored-AAD, and decrypt key-id contracts
- a_legacy_aws_kms_object_without_the_cipher_header_still_opens rebuilds the
true pre-internal-header shape (aws:kms mode + S3 key-id header, no
x-rustfs-* headers) and asserts the fallback normalizes the cipher and
re-projects it.
- a_rewritten_sse_c_context_header_fails_authentication is the SSE-C flank of
the stored-AAD tamper check; metadata_without_stored_context_bytes_still_opens
covers the derived-AAD path for both flavours and pins the seal side to the
canonical bytes (mutation-verified).
- data_key_spec_controls_the_length_of_the_generated_key requires every
backend in the matrix to honour all three specs, asserts the envelope
records the requested spec, and round-trips each blob.
- corrupt_ciphertext_fails_cleanly pins unparseable ciphertext to
CryptographicError instead of merely not-InternalError.
- Deleted the never-called assert_validation_error / assert_cryptographic_error
helpers.
#5804 added the on-disk segment budget to check_bucket_and_object_names, but PUT validates through check_put_object_args, which has its own checks and never calls it. An over-NAME_MAX key therefore still reached the disk layer and came back to the client as ENAMETOOLONG → InternalError 500, exactly the behavior #5785 reported.
Caught by re-running the acceptance suite against the locked build 4b2d79f5d, which contains #5804: S3-003 still failed with a 512-byte key.
Multipart is unaffected — check_new_multipart_args and check_multipart_object_args both route through check_object_args → check_bucket_and_object_names, which already carries the budget.
Verification: new test pins the same boundaries on check_put_object_args (255 ok / 256 rejected, byte-based via CJK, multi-segment long keys ok, __XLDIR__ budget for directory keys); cargo test -p rustfs-ecstore --lib -- bucket::utils 17 passed; cargo clippy -p rustfs-ecstore --all-targets clean; make pre-commit green.
A superseded cycle is the expected outcome of the dirty-usage fast path, not a signal of pathological load: a write burst marks buckets dirty, the scanner wakes within milliseconds, and the still-landing writes then supersede the snapshot it just took. Charging that first race SUPERSEDED_RETRY_BASE_INTERVAL = 60s meant the burst surfaced in usage and quota accounting roughly two cycles late.
Measured on an idle single-node instance (fresh data dir, 10 PUTs, polling /rustfs/admin/v3/datausageinfo every 5s): the dirty-usage wake fires 0.3s after the PUTs, its cycle is superseded 0.2s later, and the retry was then scheduled 55.6s out; usage first became visible at t+120s. With the base at 5s the retry is scheduled 4.7s out and usage becomes visible at t+70s.
The exponential growth in retry_interval is what protects against a persistently hot bucket driving an unbroken full-scan loop, so the base does not need to be a whole cycle: 5s, 10s, 20s, 40s ... still reaches minute-scale backoff within a handful of consecutive supersedes and keeps the SUPERSEDED_RETRY_MAX_INTERVAL cap. A configured cycle shorter than the base still wins, since retrying faster than the operator's own cadence buys nothing.
Verification: cargo test -p rustfs-scanner --lib (444 passed) with the three superseded-backoff tests updated to the new schedule; make pre-commit green; end-to-end probe above.
Implement a local AsyncFileReader over DataFusion's object store re-export so Parquet metadata loading no longer uses the deprecated ParquetObjectReader adapter.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): memoize bucket-incarnation fence validation under lifecycle read-lock coverage
The PUT commit fence from #5648 validated the bucket incarnation with an uncached read (a distributed metadata-transaction read lock plus an EC quorum read of bucket metadata) on every PUT commit. Under 64-concurrency 4KiB PUT load this adds two quorum round-trips per PUT and the resulting lock-manager pressure produced ~1,000 client-visible 'Lock acquisition timeout' failures per 5-minute window (see rustfs/backlog#1776).
Memoize the validation per node while lifecycle read-lock coverage is continuous: bucket deletion/recreation requires the lifecycle WRITE lock, so while at least one read guard on this node has been held continuously the incarnation cannot have changed. The first fenced PUT in a coverage window performs the exact authoritative disk validation as before; overlapping PUTs reuse its result. The memo clears when the node's last guard drops or any guard observes a lost lock, so the next PUT revalidates from disk. Fence semantics are unchanged; only the redundant re-validations under continuous coverage are elided.
Also right-size the s3s footprint ratchet baselines: -1 s3_error! line from this change's error-path consolidation, and +1 s3s-importing file inherited from #5763 (crates/obs/src/telemetry/filter.rs) which landed on main without the baseline bump.
* fix(ecstore): carry the bucket fence registry through the rebalance test store
The rebalance entry test constructor landed on main after this branch was cut and needs the new field.
* fix(admin): align replication-reset responses with madmin ResyncTargetsInfo shape
The replication-reset and replication-reset-status responses serialized
their shell as "Targets" and per-target fields in PascalCase, while
madmin-go ResyncTargetsInfo/ResyncTarget expect the "target" shell key
and lowercase field tags (arn/resetid/resyncStatus/replicationCount/
completedReplicationSize/failedReplicationCount/failedReplicationSize).
Go json decoding is case-insensitive per field, but Targets vs target,
Status vs resyncStatus and the size/count key names cannot match, so
mc replicate resync decoded empty results.
Rename the serde tags to the exact madmin wire shape, keep the
ResetBeforeDate/Error RustFS extension keys (unknown keys are ignored
by Go decoders), pin the shape with a snapshot unit test, and update
the e2e client DTO to decode the madmin shape.
* fix(admin): stream bare madmin DiffInfo documents from replication diff
POST /v3/replication/diff returned a single enveloped object
({Entries, IsTruncated, ScannedVersions}) while madmin-go
BucketReplicationDiff decodes the body with a json.Decoder loop over
bare DiffInfo documents. The envelope decoded as exactly one DiffInfo
with an empty object, so mc replicate diff printed a phantom empty row
instead of the real backlog.
Emit one DiffInfo JSON document per line by default, using the exact
madmin json tags (object/versionId/rStatus/deletemarker/lastModified;
Size stays as a RustFS extension key that Go decoders ignore). The
enveloped shape moves to the opt-in ?aggregate=true RustFS extension,
which remains the only carrier of scan-coverage metadata; a truncated
default-mode scan is surfaced via a warn tracing event instead of
in-stream. Pin both shapes with unit tests and tighten the e2e helper
to reject any envelope in the stream.
* feat(replication): validate replication config structure before persisting
PutBucketReplication accepted structurally invalid configurations that
MinIO's replication.Config.Validate rejects: empty or oversized rule
lists, duplicate or negative rule priorities, over-long rule IDs,
filters carrying more than one of Prefix/Tag/And, and delete marker
replication enabled on tag-filtered rules. Such configs persisted
silently and later produced undefined routing (e.g. ambiguous priority
ties) instead of failing the PUT.
Add validate_replication_config_structure as a pure function in
rustfs-replication (limits documented as constants), surface it through
the ecstore api facade, and run it first in the PUT capability gate so
defects are named before any metadata write. Missing Priority counts as
zero for the uniqueness check, matching Go's zero-value semantics. The
self-target rejection deliberately stays at set-remote-target, where the
endpoint is known; a config can never reference a self-pointing ARN.
Document the rule-level Destination.StorageClass contract (use the
remote target's storage_class instead) and renumber the acceptance
matrix e2e to unique priorities, which MinIO would also require.
* test(replication): pin duplicated wire types with boundary reconciliation tests
rustfs-filemeta (xl.meta disk format) and rustfs-replication (MRF/resync
persistence format) deliberately each own ReplicationStatusType,
VersionPurgeStatusType and ReplicationState; the boundary converts
between them via as_str(), whose From<&str> impls fall back to Empty on
unknown tokens — a variant added on one side silently degrades to Empty
on the other.
Add reconciliation tests in replication_filemeta_boundary: exhaustive
matches with no wildcard arm on both sides of both enums (a new variant
fails compilation until the mapping is reconsidered), string-token
round-trip asserts (a token the other side does not recognize fails
instead of quietly becoming Empty), and a full-field ReplicationState
round-trip. Cross-reference the tests from both type definitions.
Struct drift was already compile-guarded by the exhaustive struct
literals in the conversion functions.
* docs(replication): define split completion criteria and milestone sequence
The ecstore replication split plan had no completion measure — the
boundary scaffolding risked ossifying because nothing said when the
migration counts as done. Record the criteria in the module inventory:
done means the Required Contracts table's 'Current dependency to
remove' column is empty; the end state moves pool/resyncer/state into
crates/replication, with the boundary micro-files dissolving as code
crosses the crate line (batch-merging them beforehand is explicitly
rejected — the guard scripts anchor on their file names, so merging is
churn with zero functional gain; only datatypes.rs can retire early).
Sequence the remaining work as M2 (resyncer pure decision logic, after
the oversized function splits) → M3 (worker runtime, highest risk,
last) → M4 (retire boundaries and guard entries). Refresh the stale
first-step text — the event sink / runtime contracts already landed —
and update the split-plan status table accordingly.
* fix(replication): align structural validator with MinIO semantics after adversarial review
Three interop corrections found by adversarial review of the new
structural validator, plus review fallout fixes:
- Delete-marker replication is now rejected only for a direct Filter.Tag,
not for tags inside Filter.And — MinIO's validator only inspects the
direct tag, and mc replicate add --tags "k1=v1&k2=v2" (delete-marker
replication on by default) puts multiple tags into And.Tags, so the
stricter check rejected mc-generated configs MinIO accepts.
- Rule ID length is measured in bytes (Go len semantics), not chars —
a 255-char multibyte ID must not round-trip into a config MinIO
rejects.
- An empty <Tag/> element (no key) counts as absent, matching MinIO's
Tag.IsEmpty(); console form serializers emit empty tags, which would
otherwise trip the exactly-one-of and delete-marker checks.
Also: repair the store-uninitialized PUT test whose empty-rules fixture
now (correctly) fails structural validation before reaching the store
lookup; pin the previously untested startTime madmin key in the
reset-status shape test; and signal a truncated default-mode diff scan
via the x-rustfs-replication-diff-truncated response header — the bare
madmin stream has no envelope, so a truncated scan was otherwise
indistinguishable from a complete healthy one (madmin/mc ignore unknown
headers).
* test(e2e): activate SSE-S3 replication contract and pin resync fail-closed path
The SSE-S3 replication contract e2e was ignored under backlog#1291
(silent plaintext replication); the fail-closed gate in
replication_target_boundary.rs closed that hole, so the ignore reason
expired. Un-ignore the test — it now pins the current fail-closed
contract (FAILED status, failure event, readable encrypted source,
stable absence of all target versions), verified green.
Add test_bucket_replication_sse_s3_resync_stays_fail_closed: drives the
existing-object resync path (PUT ?replication-reset) over a FAILED
SSE-S3 object and asserts the resync generation reaches a terminal
state without ever materializing a target version, with the
stays-absent window also spanning fast-scanner heal cycles. The new
start_bucket_replication_reset helper doubles as the madmin
ResyncTargetsInfo shape assertion (target[0].arn/resetid) for the
reset-start response.
Refresh the stale nextest count commentary (the module is at 20 fast +
36 nightly = 56 tests by cargo nextest list; the SSE-S3-ignored note no
longer holds).
fix(ecstore): reject over-NAME_MAX object key segments up front and classify irreconcilable parity as corrupt metadata
Two defects found during release acceptance and the backlog#1776 investigation:
Object keys with any path segment longer than 255 bytes could never be stored (each segment maps to one on-disk directory entry), but the failure surfaced only when the disk layer hit ENAMETOOLONG, which leaked to clients as InternalError 500 (rustfs#5785). Validate the on-disk segment budget in check_bucket_and_object_names so such keys fail deterministically as ObjectNameInvalid (4xx) before any I/O. Directory-object keys (trailing '/') account for the __XLDIR__ suffix their final segment carries on disk.
object_quorum_from_meta conflated two very different no-quorum situations (rustfs#5801): stray or foreign metadata whose parity values are garbage produced the same retryable-looking ErasureReadQuorum (503) as a genuine partial outage, so clients retried unrecoverable reads and monitoring could not tell corruption from capacity loss. Now (a) parity counts outside [0, total_shards] are treated as invalid entries instead of being clamped to i32::MAX, which could poison common_parity's occurrence counting, and (b) when a full read quorum of disks answers but their parity values cannot be reconciled, the error is FileCorrupt — heal-actionable and non-retryable — while too-few-healthy-replies keeps returning ErasureReadQuorum.
Verification: 4 new unit tests (segment budget boundaries incl. byte-vs-char and __XLDIR__ budget; garbage parity sanitization; corrupt-vs-quorum classification), metadata::tests + utils::tests 62/62, set_disk+bucket suites 1214 passed with the single pre-existing heal_queue_marks_missing_versioning_state_as_missed cross-test flake also failing on a clean tree (not introduced here), clippy clean, make pre-commit green.
rustfs/backlog#1636 rejected a built-in rotation scheduler: rotation is a policy decision with a per-backend cost and a hard upgrade-ordering constraint, and a server that rotated on its own would make that decision on an operator's behalf at a moment they did not choose. This is what that issue resolved to deliver instead — the signal, without the actuator.
RUSTFS_KMS_ROTATION_MAX_AGE_SECS names the period. Unset leaves the verdict unreported rather than assuming a policy, because how often keys must be rotated is a compliance decision and a built-in default would report keys as overdue against a rule nobody wrote; an unparsable value is refused the same way, loudly. Values below an hour are raised to it, since a threshold of seconds reports every key as overdue moments after it was rotated and teaches operators to ignore the signal.
KeyInfo gains rotation_due and rotation_due_reason, both additive on the wire and both filled in by the manager rather than by each backend, so no two backends can disagree about what overdue means. A backend that does not advertise rotation reports unsupported and is never reported as due — it must not be told to do something it cannot. A key with no recorded rotation is measured from creation, which is how long its material has actually been in use, and is distinguished from a stale rotation so an operator can tell "overdue again" from "never once". Ages are computed saturating, so a timestamp from a node running ahead cannot manufacture an overdue key.
The verdict is advisory in the strongest sense: nothing consults it before encrypting or decrypting, a key reported as due keeps serving traffic, and readiness is unaffected.
The single-key describe response deliberately does not carry the verdict. Its type records a creation date but no rotation timestamp, so a verdict computed there could not tell a key rotated last week from one never rotated, and reporting never_rotated for a key that was in fact rotated is worse than reporting nothing.
The wraps-based branch the issue also specifies is not implemented: it depends on the per-key wrap accounting that does not exist yet.
The Local backend's durability argument rests on properties of the filesystem it runs on, and those properties were assumptions: the crate had no test touching symlinks, none asserting a published file's mode, and none on directory replacement or cross-device behavior. Writing that verification surfaced three gaps.
The key directory's own permissions were never set and never read. create_dir_all applies the process umask, which is 0 in a good many container images, and the platform picks the mode far more often than an operator does: kubelet creates an emptyDir 0777, several PVC provisioners mkdir -m 0777, a --tmpfs mount lands at 1777. Write access there is the power to delete a key, destroying every object it protects, or to plant a record for a key id that does not exist yet. The directory is now created 0700 through DirBuilder::mode, so intermediate components are covered and no create-then-chmod window exists, and anything wider is narrowed on every start and re-read to confirm it took. Narrowing rather than refusing matches what the observability stack already does with its own directory; refusing would turn each of those platform defaults into a server that will not start while leaving the exposure on disk. Only a directory this process cannot secure is fatal.
An unspecified file_permissions meant whatever the umask said. The field is optional in the persisted configuration and stays optional, but with it absent the entire mode-application block was skipped, so under a 0 umask master key records were published world-readable. Absent now resolves to owner-only inside the commit protocol rather than at each call site, so the backup restore path — which passed the unset value straight through and published a legacy cluster's restored records at 0644 — is covered by construction.
Startup left symlinked commit temps behind forever, because the orphan sweep required a regular file. The protocol only ever creates temps with create_new, so an entry wearing a temp name and any other file type is either its own leftover or something planted.
Eight tests pin the boundaries: that a requested mode survives the umask, that an absent one still resolves owner-only, that a directory at each mode a real platform produces is narrowed, that publishing replaces a symlink instead of writing through it on both the hard_link and rename paths, that a planted hard link cannot be adopted as a key record, that a symlinked commit temp is removed without harming the key it pointed at, and that commit temps never leave the destination's directory. A real cross-device operation and a directory swapped between rename and fsync cannot be verified without a second filesystem and directory file descriptors respectively; both are recorded in the operations documentation rather than left looking covered.
A key that cannot be described was handled two incompatible ways. Vault KV2 swallowed every describe failure and dropped the key from the page, so a damaged or newer-format record silently disappeared from the operator's inventory and from the deletion sweep's census. Local failed the whole listing instead, so one bad record stopped every scheduled deletion on the node for as long as the damage lasted. Both force a per-key problem into a whole-page answer.
ListKeysResponse now carries unreadable_key_ids, and the backends that read local key records classify per-key failures in one place: KeyNotFound is a concurrent deletion and is skipped, a material-level error names the key on the page, and anything else fails the listing, because it says nothing about a particular key and reporting it as key damage would turn a backend outage into a false data-loss alarm. A listing that covered the entire key set and found nothing readable still fails, since an empty page there is indistinguishable from a deployment with no keys; the guard is scoped to a page with no successor so a damaged key can never strand the keys behind it. The deletion sweep destroys the expired keys it can read, counts the unreadable ones, and withholds its lifecycle gauges rather than publishing a census over a key set it did not fully see.
Vault Transit needs the same treatment and is easy to miss: its per-key metadata records live in KV2 too, so folding every non-404 failure into a backend error left its per-key classification unreachable and one metadata record written by a newer build still failed every listing on the node.
Vault KV2 record reads gain the typed errors this needs: an unparseable body is MaterialCorrupt and an absent data envelope is MaterialMissing, where both were previously indistinguishable from Vault being unreachable. Only the parse failure's category and position are reported, because serde's own message embeds the offending scalar and that message reaches a log line and an admin HTTP body.
The admin list handlers refuse a malformed limit with 400 instead of silently substituting the default page size, and every page is capped at 1000 where it is cut, so a single request can no longer fan out one metadata lookup per key without bound. The four operation-level KMS metrics gain a backend label, since operation names are shared across backends and a Transit latency regression was previously indistinguishable from an AWS one. The Static backend captures its reported creation date once instead of reading the clock on every describe and list. POST /kms/clear-cache gains a named response type with an unchanged wire shape.
* test(site-replication): pin MinIO IAMUserType wire semantics for policy mappings
Red tests for P0-4: MinIO peers send SRPolicyMapping.UserType using the
madmin IAMUserType table (unknown=-1, regUser=0, stsUser=1, svcUser=2),
while RustFS deserializes the field as u64 and decodes it with the
internal RPC table (None=0, Svc=1, Sts=2, Reg=3).
- userType -1 (MinIO group mappings) fails to deserialize, rejecting the
whole IAM item: group mappings never sync from MinIO.
- stsUser=1 decodes as Svc, landing federated STS mappings under the
wrong prefix and silently dropping their effect.
* fix(site-replication): translate policy mapping userType at MinIO wire boundary
SRPolicyMapping.userType travels on the wire using MinIO's IAMUserType
table (unknown=-1, regUser=0, stsUser=1, svcUser=2), but RustFS stored
the field as u64 and reused the internal RPC encoding
UserType::to_u64/from_u64 (None=0, Svc=1, Sts=2, Reg=3) at the site
replication boundary. Consequences: MinIO group mappings (userType -1)
failed to deserialize and the whole IAM item was rejected, and MinIO STS
mappings (1) were stored as service-account mappings, silently dropping
federated users' policies.
- Widen SRPolicyMapping.user_type and SRCredInfo.iam_user_type to i64 so
MinIO's -1 deserializes.
- Add sr_wire_user_type / user_type_from_sr_wire in rustfs-iam as the
dedicated SR wire codec: MinIO table on both directions, groups always
encoded as 0, and wire value 3 kept forever as an alias for Reg so
mappings from pre-fix RustFS peers still decode; unknown values fail
closed.
- Route the SR inbound (apply_iam_item) and outbound
(mapped_policy_to_sr_mapping, policy-mapping change hooks) paths
through the codec.
The internal UserType::to_u64/from_u64 encoding is untouched: it is the
intra-cluster node RPC contract and changing it would break rolling
restarts. Outbound compatibility with old RustFS peers is preserved
because UserType::None and Reg share the users prefix in
get_mapped_policy_path, so wire 0 lands in the same location Reg=3 did.
* test(replication): expect CopyObject and snowball extract to schedule replication
Red-phase TDD tests for P0-6: CopyObject never consults the bucket
replication config (no pending stamp, no schedule, and the destination
inherits the source's stale replication status metadata wholesale), and
snowball auto-extract members are never scheduled either.
- usecase white-box: observe MUST_REPLICATE_OBJECT_CALLS for
execute_copy_object (currently 0, must be 1) and
execute_put_object_extract (currently 0, must be 2 for a two-member
archive), plus stale replication-status metadata cleanup assertions
(MinIO filterReplicationStatusMetadata parity).
- e2e: CopyObject destination and snowball-extracted members must appear
on the remote replication target and reach COMPLETED on the source.
Red evidence (before fix):
copy_object_computes_replication_decision_and_strips_stale_status
assertion failed: left: 0, right: 1
put_object_extract_computes_replication_decision_per_entry
assertion failed: left: 0, right: 2
* fix(replication): schedule replication for CopyObject and snowball extracted objects
CopyObject and snowball auto-extract never consulted the bucket
replication config: no PENDING stamp, no post-commit schedule, and no
scanner-heal backstop (heal only re-drives Pending/Failed objects, and
these objects carried no status at all). Worse, the copy path cloned the
source metadata wholesale, so a destination object inherited the
source's replication bookkeeping and could present a fake
COMPLETED/REPLICA state.
Mirroring the PUT path (single immutable decision drives both the
pending metadata and the post-commit schedule, rustfs/backlog#1320):
- execute_copy_object: strip the source's replication status metadata
(internal replication/replica status + timestamps under both
compatibility prefixes, plus x-amz-replication-status) for
non-inbound requests — MinIO filterReplicationStatusMetadata parity;
the cleanup runs before the decision so an inherited REPLICA status
cannot suppress it. Then compute must_replicate_object once, stamp
PENDING when it replicates, and schedule after the copy commits and
the self-copy lock guard is released. Inbound replica writes keep
their authorized metadata and are declined inside
must_replicate_object, so replicas are never re-scheduled outbound.
- execute_put_object_extract: same stamp + schedule per extracted
member object (MinIO PutObjectExtract parity).
- execute_put_object dispatch: an authorized inbound replication PUT is
stored verbatim instead of being re-dispatched into the extract path.
Extracted members keep x-amz-meta-snowball-auto-extract in their user
metadata and the replication client replays stored metadata as
headers, so the target used to try to untar each member's own bytes,
permanently failing replication for non-archive members (surfaced by
the new snowball e2e test).
Green evidence:
- copy_object_computes_replication_decision_and_strips_stale_status,
put_object_extract_computes_replication_decision_per_entry (red: 0
decisions; green: 1 and 2), plus the existing PUT/object-lock
decision-count tests stay green.
- e2e test_copy_object_replicates_to_target and
test_snowball_extract_replicates_members_to_target pass against two
live instances.
* test(site-replication): expect MinIO sts-account IAM item type
MinIO madmin-go replicates STS credentials with SRIAMItem type
"sts-account" (SRIAMItemSTSAcc), but RustFS emits and accepts only
"sts-credential", so cross-implementation STS replication fails in
both directions (MinIO returns errSRInvalidRequest, RustFS returns
NotImplemented).
Red-light tests:
- pin the outbound AssumeRole replication item type to "sts-account"
(construction extracted into assume_role_site_replication_item so it
is testable, behavior unchanged in this commit)
- update the federated identity replication item snapshot to
"sts-account"
- inbound apply_iam_item must dispatch both "sts-account" and the
legacy "sts-credential" alias to the STS arm instead of the
unknown-type NotImplemented fallback
* fix(site-replication): use MinIO-compatible sts-account IAM item type
MinIO madmin-go replicates STS credentials with SRIAMItem type
"sts-account" (SRIAMItemSTSAcc). RustFS emitted "sts-credential" and
accepted only that value inbound, so STS credential replication with
MinIO peers failed in both directions: MinIO rejected RustFS items as
errSRInvalidRequest and RustFS answered MinIO items with
NotImplemented.
- define SR_IAM_ITEM_STS_ACC ("sts-account") and
SR_IAM_ITEM_STS_ACC_LEGACY ("sts-credential") in rustfs-madmin
- emit "sts-account" from both outbound sites (AssumeRole hook and
federated identity OIDC hook)
- accept both types inbound; the legacy alias remains permanently for
mixed-version RustFS rolling upgrades
Token verification and the retry/event mechanism are unchanged.
* test(replication): accept madmin nanosecond healthCheckDuration payloads
Red-phase TDD tests for P0-7: mc 'replicate add' sends the madmin default
healthCheckDuration=60s as a Go time.Duration nanosecond integer
(60000000000), which RustFS currently rejects as an unsupported field and
would misread as seconds. Also pins the defensive seconds-or-nanos read
for persisted bucket-targets metadata and the capability contract listing
healthCheckDuration as writable.
Currently failing (red):
- remote_target_request_accepts_go_duration_wire_values
- remote_target_request_accepts_legacy_seconds_health_check
- remote_target_health_check_duration_is_declared_writable
- bucket_target_reads_go_nanosecond_durations_defensively
- runtime_capabilities_response_reports_missing_topology_before_storage_init
* fix(replication): accept remote target healthCheckDuration nanoseconds
mc 'replicate add' always sends the madmin default healthcheck-seconds=60
serialized as a Go time.Duration nanosecond integer (60000000000), so the
default mc link-creation path (and 'mc replicate update') failed with
InvalidRequest. Move healthCheckDuration from the unsupported to the
writable remote-target field list; the capability contract in the runtime
capabilities response follows the constants automatically.
Fix the unit mismatch in both directions:
- Request parsing and persisted bucket-targets reads decode the value
defensively: below 10^7 it is legacy RustFS seconds, otherwise Go
time.Duration nanoseconds (also covers MinIO-written metadata).
totalDowntime shares the same wire shape and gets the same handling.
- The list-remote-targets admin response re-encodes only these two fields
as nanoseconds via a dedicated serialization path, leaving the persisted
seconds-based wire format untouched for existing readers.
The per-target health-check interval is accepted for mc compatibility but
not yet applied; the heartbeat keeps its global env-configured interval,
and the explicit 'healthcheck' update op stays rejected. disableProxy,
edge, and edgeSyncBeforeExpiry remain explicitly rejected.
* chore(deps): refresh mimalloc revision
Update mimalloc and libmimalloc-sys to the requested git revision after running the dependency refresh flow.
Keep ratelimit excluded while accepting compatible dependency updates from cargo update and cargo upgrade.
Harden all-feature test compilation by giving heavy integration test crates their own recursion limit and avoiding a cross-thread spawn for the embedded startup barrier future.
Co-Authored-By: heihutu <heihutu@gmail.com>
* upgrade version
---------
Co-authored-by: heihutu <heihutu@gmail.com>
PR #5719 fixed the issue #5716 per-object heal log amplification (per-object statements demoted, heal spans forced to TRACE, raw metadata dumps banned by guardrail) and superseded the demotion originally proposed here. This PR now carries only the residual cleanup on top of it:
- Convert the remaining bare-field and format-arg heal logs in crates/ecstore/src/set_disk/ops/heal.rs to the file's structured convention (event/component/subsystem + context fields): missing-object skip, disk-marked-for-healing, cannot-reconstruct errors, dangling-cleanup error, missing data_dir error, xl.meta regeneration warn, and orphan-reclaim failure warn.
- Demote the last remaining info! in the file — the per-set heal_format "set disk formats success, NoHealRequired" no-op message — to a structured debug! (error_count instead of a raw errs dump), and drop its whitelist exclusion in scripts/check_logging_guardrails.sh so the no-INFO check for set-disk heal files is strict.
No control flow or behavior changes.
Multipart upload ids embed the process-global deployment id at both create time and list time. Under plain cargo test (thread-parallel, shared process globals) a concurrently running test that re-initializes a store can swap the global between the two reads, making full-upload-id equality assertions fail spuriously (observed: core::sets::tests::list_multipart_uploads_merges_all_sets_without_pagination_loss failing when run concurrently with bucket::quota tests, passing in isolation).
Add a test-only upload_uuid_suffix helper next to deployment_upload_id and make the affected assertions compare only the decoded <uuid>x<timestamp> suffix. Where suffix normalization changes within-key ordering (base64 alphabet order is not byte order), both sides are sorted before comparison. nextest/CI is unaffected (process-per-test); this only hardens local plain cargo test runs.
initialize_local_disk_maps appended pool entries to local_disk_set_drives and inserted into local_disk_map without ever clearing previous state. Every caller (both production startup entry points and all tests) passes the FULL topology, so re-initializing the same InstanceContext left the pool/set vectors sized for the stale topology and panicked with index-out-of-bounds for wider disk indices.
This surfaced as deterministic cross-test contamination under single-process cargo test: in crates/heal heal_b920_subquorum_union_test, a 4-disk test initialized pool 0 as [None; 4] on the process-level default context, and the two 8-disk tests then panicked at disk_idx 4. Each test passed alone, and CI never caught it because cargo nextest isolates every test in its own process.
Fix: clear both registries at the start of initialize_local_disk_maps so initialization is idempotent and last-topology-wins. Add a regression unit test in ecstore (process-isolation-proof, unlike the heal integration binary) that re-initializes the same context with a wider topology; it fails with the pre-fix code.
The multipart staging namespace is one flat set of sha256(bucket/object) directories shared by every bucket, and the cross-set listing rewrite reads every upload's metadata. Two shapes poisoned the whole ListMultipartUploads response with InternalError: Corrupted format: a healthy in-flight upload belonging to another bucket (its stored owner bucket fails the guard and fell into the corrupted-format arm), and a single upload directory whose xl.meta was torn by an unclean shutdown. Docker Distribution calls ListMultipartUploads on every PATCH/commit, so either shape broke OCI registry pushes entirely (issue #5716).
Foreign-bucket uploads are now skipped silently, and directories whose metadata is affirmatively corrupt at quorum are skipped with a debug log, while every other decode failure (quorum loss from offline disks, timeouts, transport errors) keeps failing the listing so clients retry instead of silently losing entries. The degrade-vs-propagate decision is a named corrupt-family classifier with a unit test pinning both sides. FileMeta::check_xl2_v1 now classifies a missing or wrong XL2 magic as FileCorrupt instead of an anonymous io error so damage is distinguishable from transient IO faults.
Refs #5716
cxymds