* fix: reject unsupported pool expansion with actionable errors
Report singleton-pool and persisted-topology constraints before misleading startup retries. Preserve single-node multi-drive admission and existing parity policies, and cover format preservation plus operator recovery guidance for issue #6186.
* fix: keep pool layout errors typed
Preserve actionable pool layout diagnostics without adding generic formatted errors. Tighten the shrink-only baseline and assert that both typed payloads survive the I/O boundary.
---------
Co-authored-by: houseme <housemecn@gmail.com>
* test(e2e): add distributed 4x4 validation
* ci(e2e): use tmpfs for distributed pool isolation
sm-standard-4 is an ARC pod without usable loop devices, so
mount -o loop fails with ENOENT before any pool filesystem is
attached. Sized tmpfs still gives each pool a distinct st_dev
and independent 1G statfs capacity.
Co-authored-by: RustFS <hello@rustfs.com>
* test(e2e): prove operations overlap data movement
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* test(odm): pin gcs source status-to-error-class mapping
The native GCS backend classifies every failure from the HTTP status
alone, because GCS states its error code in a body this backend never
reads. Only NotFound is negative-cached and only a retryable class may be
re-sent, so cover 401/403 -> AccessDenied, 429/503 -> Throttled,
500/502 -> ServerError and 404 -> NotFound over both HEAD and GET.
* ci(odm): require the source-backend contract tests in test-and-lint
The shared contract tests already run in ci/test-and-lint, but only
because gcs is a rustfs default feature; nothing failed if that
selection went away. Pin the S3, Azure and native GCS contracts in the
core required-test manifest so a lost selection fails the lane.
test(e2e): prove bucket config survives rc.5 upgrade and rollback
Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.
The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.
The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
test(e2e): add the outbound target matrix and the replication checksum postmortem
Defense work for rustfs#7082, the regression rustfs#6895 introduced while
fixing rustfs#6853: a fix for one target class changed a client default for
every target class and nothing in tree modeled the other classes.
- docs/postmortems: timeline, root cause, why four defense layers missed
it, and the SOP for changing any outbound client default; AGENTS.md and
the adversarial compatibility lens point at it; the two env knobs from
rustfs#6895 are documented in docs/operations.
- fake_s3_target: reject_aws_chunked_uploads, require_checksum_for_object_lock
(Content-MD5 always verified), create_bucket_with_object_lock with a
GetObjectLockConfiguration handler, and a TransportSnapshot on every
journal record.
- replication_target_matrix_test: six object shapes against four target
modes with an explicit expectation table; the two rustfs#7082 cells are
pinned KnownFailing and fail with an XPASS message once the fix lands.
Wired into e2e-repl-nightly, excluded from e2e-full.
Preview tags stay as the traceability record for the validated commit, but their GitHub Releases are internal validation state and should not accumulate on the Releases page next to real deliverables.
Add a cleanup-preview-releases job that runs after publish-release succeeds for a release or prerelease tag and deletes every Release whose tag is exactly <target>-preview.<digits>. The tags themselves are kept: the job never passes --cleanup-tag. Tag matching uses jq string operations rather than a regex over the version, so dots in the version cannot widen the match, and the release listing is fetched before filtering so an API failure aborts the job instead of looking like there was nothing to clean up.
Extend the preview release workflow guard with the job condition, the delete invocation, both tag-matching filters, and an absent check for --cleanup-tag.
* test(e2e): activate data usage regressions
* test(e2e): bind data usage selection to Linux listing
* test(e2e): refresh data usage Darwin selection
---------
Co-authored-by: houseme <housemecn@gmail.com>
Add a read-only Prometheus report helper for backlog#2007 so the 200us vs 50us coalescer delay experiment can capture RPC, batch distribution, stage latency, and host-cost signals with one fixed evidence format.
Co-authored-by: heihutu <heihutu@gmail.com>
* test(e2e): fold seven identical POST-policy exact-mismatch tests into one table
The seven *_policy_mismatch tests in multipart_auth_test.rs were body-identical after literal normalization: the policy pins one field to an exact value, the form sends a different value, and the upload must be rejected with 400 InvalidPolicyDocument naming the field. Each test booted its own full server.
They fold into one table-driven test on the run_post_object_policy_case helper introduced by the PR1 fold. Every row keeps its original test's exact bucket, key, field name, policy value, mismatched form value, body bytes, and expected error strings — including the three rows that asserted the stronger <Code>InvalidPolicyDocument</Code> form. The pinned condition is built with an explicit serde_json::Map since the field name is now a table parameter.
cargo nextest list reports 97 tests for this module; the inventory row is updated in the same diff (103 -> 97).
Ref rustfs/backlog#1838 (PR2).
* test(e2e): fold the remaining POST-policy duplicate groups (15 tests) (#6018)
Completes the multipart_auth table-driven fold: the six remaining body-identical groups collapse onto the shared run_post_object_policy_case helper.
- Seven single-field exact-mismatch tests (cache-control, expires, tagging, storage-class, content-type, success_action_status, metadata-field-exact) join the existing exact-condition mismatch table as rows — same shape as the PR2 fold.
- The two object-lock mismatch tests become a two-row table (policy pins mode + retain-until-date, one form field mismatches).
- The three SSE-KMS parameter mismatch tests become a three-row table (policy pins the SSE mode plus one KMS parameter, form differs).
- The three SSE-KMS outside-policy tests become a three-row table pinning the distinct contract: an undeclared KMS parameter sails past policy validation and is rejected at runtime with 501 NotImplemented, not a policy error.
Every row keeps its original test's exact bucket, key, field names, values, body bytes, and expected status/code strings. cargo nextest list reports 85 tests for the module; the inventory row is updated in the same diff (97 -> 85).
Ref rustfs/backlog#1838 (PR3).
The nine *_missing_from_policy_conditions tests in multipart_auth_test.rs were literal-for-literal identical after normalization: policy pins bucket + key + content-length-range, the form smuggles one extra field the policy never declared, and the upload must be rejected with 403 AccessDenied naming the field. Each one started its own full server.
This adds the run_post_object_policy_case helper (parameterized by bucket, key, policy conditions, extra form fields, file body, and expected status/code/mention, with a per-case assertion prefix) and folds the nine tests into one table-driven test with nine rows. Every row keeps its original test's exact bucket, key, field name/value, body bytes, and expected error strings — including the two rows that asserted the stronger <Code>AccessDenied</Code> form — so no poison value is lost. The helper's signature is general enough for the policy_mismatch and sse-kms groups planned as PR2/PR3.
cargo nextest list now reports 103 tests for this module; the inventory row said 109 while the file actually held 111 before this change (stale by two), so the inventory is set to the measured 103 in the same diff per the issue's hard constraint.
Ref rustfs/backlog#1838 (PR1).
Open the managed-SSE replication gate (backlog#1783, PR-B of 3, after
#5872): the replication reader already decrypts through the injected
object-encryption resolver, so the source sends plaintext plus an
encryption intent header (AES256 / aws:kms, never the source key id) and
the target re-encrypts on its normal PUT path with its own KMS. No DEK
crosses sites.
- replication_put_object_options: fail closed only on Unsupported;
insert the SSE intent after the strip loop.
- TargetClient::create_multipart_upload sends the full opts.header()
set, fixing multipart replicas losing content-type/user metadata
(plaintext included).
- Preserve source ETag and mtime on replicas (authorized replication
only): receiver wires x-rustfs-source-etag into preserve_etag for PUT
and CompleteMultipartUpload, resolve_complete_etag consumes it, and
complete options carry source_etag/source_mtime (absent mtime
degrades to epoch, not now_utc). Without this every replication HEAD
comparison re-drives re-encrypted objects forever.
- e2e: managed SSE contracts flip to success on an independent-KMS
dual-process pair (byte-identical plain GET proves target-owned
envelopes; ETag/mtime preserved; version stable across scanner
cycles; resync converges; multipart keeps structure and metadata);
new target-without-KMS fail-closed contract; SSE-C stays FAILED.
Co-authored-by: houseme <housemecn@gmail.com>
houseme