* fix(sse): classify bare SSE-KMS writes when no KMS is available
A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.
Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.
Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.
(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)
* fix(sse): keep KMS error classes on the encrypted read path
GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.
Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.
Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.
(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)
* fix(kms): classify key-management errors on the admin routes
`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.
Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.
The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.
(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)
* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key
`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.
Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.
Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)
* fix(s3): validate PutBucketEncryption rules before storing them
A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.
Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.
Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.
(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)
* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes
* test: align two unit tests with the new KMS and bucket-encryption contracts
`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.
(cherry picked from commit e2e6a2535a)
* fix(ci): share quick checks and lint workflows
* fix(ci): install actionlint from its verified release
* fix(ci): reject dependencies on required quick checks
* feat(test): verify the E2E server build and source identity
* test(e2e): register verified Darwin test membership
* test(e2e): record verified Linux receipt test membership
* test(e2e): record compiled Darwin receipt test membership
* test(e2e): record compiled Linux receipt test membership
* test(e2e): record Darwin e2e-full membership after merging main
* fix(test): route scanner/heal evidence E2E runs through the verified server binary
The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed.
* docs(e2e): run server-backed e2e commands through the verified binary wrapper
* test(e2e): record Linux e2e-full membership from the branch CI listing
(cherry picked from commit 2909b1bfe1)
* fix(replication): total-order rule sort and honor V1 top-level Prefix
Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):
- The actionable-rule sort compared same-destination rules by priority but
answered Equal for any other pair, which is not a total order; the
standard library sort panics on such comparators once a slice exceeds the
insertion-sort threshold, so an object matching more than 20 enabled rules
across two or more targets could panic the PUT or DELETE task. Rules now
sort by priority descending with destination and id as tie-breakers, and
filter_target_arns preserves that order instead of draining a HashSet.
- A V1 rule written without a <Filter> carries its prefix at the top level;
that field was never read, so <Prefix>logs/</Prefix> matched every object.
ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
precedence. The existing prefix fixtures were built this way and had been
asserting nothing.
* fix(admin): advertise data-usage and listen capabilities to rc
The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).
Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.
* fix(site-replication): stop notifying the local site on remove and rotate
The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).
Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.
* fix(site-replication): promote and settle IAM retries without a tick of slack
Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):
- The lightweight 30-second pass filtered its reachability probe to bucket
ops, so a backed-off IAM or bucket-metadata snapshot waited for the
600-second tick to notice the peer. It now probes every backed-off class
and still replays only bounded bucket ops; promotion is a state flip the
heavyweight tick acts on.
- Backoffs are multiples of the tick interval, so a failure stamped δ
seconds after a tick was 600 − δ old at the next tick and slipped a whole
extra interval. The heavyweight drain now evaluates backoff halfway to its
next tick.
- An IAM entry first created by a non-deletion failure (the add bootstrap's
snapshot send, the drain's own replay, an import-iam schedule) was never
stamped `deletions_recorded`, so a later recorded deletion could not
settle it and it escalated to the marker only `replicate repair` clears.
Entries created by this binary now start recorded; a row persisted by an
older binary keeps the escalation semantics.
* fix(site-replication): reload peer node caches after bucket wiring writes
Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).
Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.
The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.
* test(e2e): cover delete-marker replication from a multi-node source
The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.
* ci(e2e): refresh the distributed selection for the new replication cases
Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.
(cherry picked from commit aeaba86d73)
Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers.
(cherry picked from commit f02bc947cd)
* fix(s3): reject oversize single PUT early and map body errors to 4xx
A single PutObject above the 5 GiB single-request ceiling was only
rejected after the client had streamed 5 GiB into s3s's read-time body
budget, and the resulting BodySizeLimitExceeded surfaced from the erasure
writer as 500 InternalError. A body whose connection hit EOF before
Content-Length bytes arrived (hyper's IncompleteBody) was also a 500.
SDKs retry 500s, so one oversize upload was resent from offset 0 five
times.
- PutObject and UploadPart reject a declared length above
MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the
body; the constant moves to rustfs_config so the s3s limit and the
admission check share one value.
- ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body
EOF to IncompleteBody across both io::Error conversions.
Fixes#7596.
* test(s3): cover UploadPart admission, aws-chunked length, real s3s limit
- Poll-counting test body proves PutObject and UploadPart reject a
declared size above the ceiling with zero body polls; exact-cap and
zero-length parts pass admission.
- A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the
cap is admitted when the decoded length is within it and rejected when
the decoded length is over it.
- The display-based BodySizeLimitExceeded matcher is checked against the
real error produced by the pinned s3s Body budget.
(cherry picked from commit 50b31bc75b)
Queue committed tier free-version cleanup receipts for PUT and materialized CopyObject overwrites of transitioned null versions, while keeping remote deletion behind the existing persisted free-version cleanup path.
Tighten data-movement delete-marker retry equivalence by ignoring local bucket-incarnation fencing metadata, avoid retry fallback to the source pool, and keep version-list pagination from manufacturing an empty final page.
Refresh the e2e-distributed selector hash for the current release test set.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Treat non-authoritative usage floor startup as pending bootstrap rebuild work so reset-published bootstrap markers cannot sit behind clean-idle or empty pause-backlog delay.
Wire recovery wakeups into the normal scanner cycle wait and expose the pending rebuild state in scanner status.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Map PartMissingOrCorrupt to SlowDownRead only at the GetObject/CopyObject source-reader boundary so quota metadata corruption keeps its internal fail-closed response.
Add store and e2e coverage for Harbor-style multipart staging CopyObject boundaries.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore)!: bind bitrot shards to immutable part identities
Verify part, coding-index, and block identity across write, GET, and Heal
paths. Preserve identities across metadata-only copies and repair, include
them in multipart quorum selection, and require payload proof for receipts.
Keep legacy decoding with conservative parity and target-digest validation,
and document its unsupported cases and additional verification I/O.
BREAKING CHANGE: New bound-v1 shards require compatible readers throughout
the fleet. Legacy objects without sufficient integrity evidence return an
error; binary rollback after new writes requires verified data migration.
Refs: rustfs/backlog#2497
* fix(ecstore): preserve shard framing with independent integrity
Commit immutable part-generation Merkle roots and replicated proof indexes without changing existing checksum frames. Verify reads, reconstruction and Deep Heal against metadata quorum; keep legacy reads and explicitly defer unproven legacy data repair.
Preserve multipart rollback generations, require acknowledged durable index publication, and add decoder compatibility and donor-shard regression coverage.
* fix(heal): verify protected partial-write replay
* fix(heal): rebuild truncated xl.meta from healthy quorum
* fix(test): pass topology to heal overlap RPC regression
* fix(test): drive heal admission alongside partial PUT
Poll the partial PUT and its mock heal receiver together, bound their handshake, and retain the existing repair-scope assertions.
* fix(test): prepare durable MRF fixtures and Linux heal stack
* fix(test): drive tier cleanup recovery after deferred attempts
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Allow a confirmed scanner root data-usage CAS write to prove its own publication when the follow-up root readback cannot provide a proof. Keep AlreadyDurable and all stale or companion paths on the existing readback-only proof boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): persist and retry partial-write repairs
* test(heal): pass topology to overlap RPC tests
Use the existing coordinator endpoint fixture for the three overlap-test
calls to the endpoint-aware heal control executor. This repairs the E0061
test-build failure inherited from the release base.
Reuse the heal-control endpoint fixture in the overlap receipt regression so the test matches the updated execution helper signature and selector validation boundary.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: hehutu <hehutu@gmail.com>
The overnight run after #7708 proved the security verdict greps still
counted zero: real verdict lines are '\e[1;31m[FAIL]\e[0m STS-105 ...'
— the reset escape sits between the tag and the case id, and the
pattern only tolerated escapes before the tag. Allow escapes on both
sides; the fixture now emits the reset too, mirroring the real suite.
The tier case table is produced by rustfs_tier_report.py and its rows
lead with the topology column, so the case-ID-first table grep matched
nothing ('Product result: 0 passed, 0 failed'). Parse the PASS/FAIL
counts from the '## Case Summary' bullets the report always emits,
falling back to a topology-aware table grep.
First full serial pass with the green-on-case-failure semantics
(run 34693745171 / 34695021651) exposed three report-layer defects:
security: the report step referenced LOG_FILE, which is undefined in
this workflow (set -u killed the step before writing report.md), and
the verdict greps could not match the ANSI-escaped [PASS]/[FAIL] tags
in the real suite log. Point it at the artifacts suite.log, allow any
number of color escapes before the verdict tag, and count [SKIP]
lines separately (45 passed, 6 failed, 3 skipped was reported as an
unbound-variable crash).
pool: warp is stopped early (SIGINT) at the storage threshold and
only writes its final report on a clean exit, so an empty warp.log is
the expected shape of a healthy run - require its presence, not its
size. A mid-script die() abort or a FAIL step verdict must also turn
the validator red now that the run step is continue-on-error.
tier: add the standard 'Product result: N passed, M failed' summary
line computed from the case table, matching the other suites.
The contract test fixture previously injected LOG_FILE into the
environment and wrote verdict lines without ANSI escapes, which hid
both real-world defects; the fixture now mirrors the real suite
(stdout+tee with color tags) and asserts the pass/fail/skip counters.
Remove the superseded ServiceUnavailable-only PUT helper after the G14 outage PUT probe switched to the shared retry classifier.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Treat SlowDownRead as a retryable outage PUT probe response in the G14 multi-pool runner, and label terminal candidate failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Treat SlowDownRead as a bounded retryable deferred PUT response after the target pool rejoins, and label terminal deferred outage PUT failures with stage context.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A failing product case used to turn the whole workflow red, so the run
conclusion carried no signal beyond 'something failed' and the report
was suppressed. New semantics across the functional suites:
- Suite steps run with continue-on-error: the outcome is still recorded
for the report and the backlog issue manager (security/tier already
carried the flag).
- Generate report always publishes the full per-case table plus a
'Product result: N passed, M failed' summary, and its exit gate is
harness health: red only when the suite never reached case level (no
case verdicts), failed wholesale (zero passes, >=3 failures), or was
cancelled/skipped. performance is unchanged (parked).
- tier's structured gate no longer fails on case failures; it keeps red
for evidence-init and missing-gate-result breakdowns.
- pool/performance keep their existing red sources (install/benchmark).
Workflow contract tests updated to the new exit semantics: the security
report matrix keys green off the suite outcome, the evidence matrix
expects green for failure outcomes with recorded case rows (except
performance), the heal staged-rerun block expects the per-step table to
always publish, and run steps are now required to carry
continue-on-error.
Verified locally: actionlint clean; test_security_workflow.py 21/21.
RUSTFS_POOL_NODE_ENDPOINTS has no secret/var configured, so the suite
ran with the workflow's inline 3-endpoint fallback and the explicit
--node-endpoints flag overrode the script default fixed in
rustfs/auto-testing#61 — every dispatch died at startup with 'must
provide at least 4 direct node endpoints' (run 34677538650). Add
rustfs-node4 to the fallback; explicit secret/var still wins.
Allow the Scanner/Heal interruption oracle to retry transient retryable GET failures after replacement recovery has converged. The readback still verifies exact object bytes and keeps a bounded timeout, so permanently unreadable objects continue to fail the case.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Add a read-only descriptor ledger mode to the Scanner/Heal Linux evidence planner so release operators can separate current-head measured descriptors from old-head measured artifacts and case-level inputs before final bundle assembly.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(storage): alias legacy meta bucket over internode rpc
Retry read-only internode RPC metadata access from legacy .minio.sys to .rustfs.sys when mixed-version peers report missing metadata during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): target durable ILM receipt quorum fixture
Use the actual durable ILM receipt object path when taking target disks offline so the test exercises receipt write quorum instead of whichever set owns the source record path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* ci: manage backlog issues by signal instead of per-run filing
The suite workflows used to file one backlog issue per failed run
(dedup was by run ID, which never matched), so issues accumulated
without bound. Replace the inline filing step in every suite workflow
(s3, kms, tier, storage, heal, pool, security, replication, upgrade,
performance) with a single call to
auto-testing/scripts/issue_manager.py, which:
- dedups by signal: failing cases are searched among open issues by
label (suite category + case ID); covered cases become a coalesced
comment on the existing issue, only uncovered cases file a new one
- labels new issues with functional-test, the suite category, one
label per failing case ID (lazily created), and env for
bootstrap-class failures (no cases ran, wholesale failure, or
404/ssh/clone/dpkg signatures in the log)
- closes open issues of the suite after a fully green run, citing the
run as evidence; cancelled runs never file or close anything
The step is skipped cleanly when auto-testing (private checkout) does
not contain the manager, or when PF_TESTING_GH_TOKEN is unset.
* fix(ci): satisfy actionlint and workflow contract tests for the manager step
- heal and performance workflows have no rustfs_version dispatch input;
referencing `${{ inputs.rustfs_version }}` in the manager step failed
actionlint's expression type check. Their package source now resolves
from package_url with the nightly fallback.
- scripts/test_security_workflow.py pinned the removed inline filing
step. The wiring assertions now pin the manager step (manager path +
per-suite report argument), and the evidence/stale-file tests assert
the skip contract instead: without the private auto-testing checkout
present, the step exits 0, publishes nothing, and leaves stale
evidence untouched.
Verified locally: actionlint clean, shellcheck clean,
test_security_workflow.py 21/21.
The rustfs_version dispatch input defaulted to 1.0.0-rc.4-preview.1,
which shadowed the nightly fallback and started 404ing once that
release was deleted. Manual dispatches with no inputs now fall through
to the nightly package (same contract the pool suite already has);
passing rustfs_version or package_url still pins the build exactly as
before. Chain (repository_dispatch) runs are unaffected: the inputs
context is empty there, so they always used the nightly fallback.
唐小鸭