mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-06 13:27:43 +00:00
Zhengchao An
98d3619613
* fix: address rc.1 release blockers
* fix: route release guards through architecture boundaries
* fix: close remaining rc.1 regression gaps
* refactor: group multipart listing options
* fix: resolve rc.1 CI regressions
* fix(ecstore): keep bucket-config writes off the caller's stack
A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.
Two CI lanes aborted with SIGABRT on this:
ILM Integration (serial)
rustfs app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions
Test and Lint (swift)
rustfs-protocols::swift_metadata_persistence::
swift_metadata_writes_are_durable
Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.
Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.
* fix(lifecycle): unbreak restore under the bucket generation fence
The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.
1. RestoreObject and ListMultipartUploads hard-required
`opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
deliberately leaves it unset when no guard extension is present — only the
S3 access layer installs one. Every direct caller therefore got
`InternalError: ... bucket generation guard is missing`. Resolve the
current generation instead, the way the copy path already does. The fence
is unaffected: RestoreObject still re-reads the incarnation from disk and
compares before admitting the restore, and the multipart listing is
filtered by the value it resolves.
2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
restored-copy expiry whose `restore_expires` had not already elapsed.
Whether the restored copy is due to expire is the ILM evaluator's
decision, made when it emitted DeleteRestoredAction; re-deriving it in
the set layer only adds a way for a legitimate action to be rejected.
The stale-event risk it appears to guard is already covered by the
surrounding snapshot match — a re-restore rewrites `restore_expires`,
so a replayed event fails the equality check. Drop the clause; the
fifteen identity clauses are unchanged.
Fixed:
rustfs app::lifecycle_transition_api_test::
restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
restore_object_usecase_completes_suspended_null_version_in_place
restore_object_usecase_reports_ongoing_conflict
rustfs-scanner::lifecycle_integration_test serial_tests::
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore
Verification: the CI ILM lane filter now runs 53/53 green locally.
* chore: address review follow-ups on this branch
Four items from the adversarial review that were still open.
- Restore the assertion `test_bucket_replication_replayed_delete_marker_
preserves_source_mtime_without_source_restart` is named for. The branch
had replaced the backlog#867 mtime check with `assert_replication_
converged`, which any successful replication satisfies, and deleted the
two helpers it needed — so the regression the test exists to catch would
now pass. This matters here specifically because the branch changes the
flag feeding `replication_delete_remove_options` and routes replay
through a new file and ordering.
- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
one real consumer calls the `_with_metadata` variant directly). Its test
stanza now exercises that variant, so the coverage moves to live code
rather than being deleted.
- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
2.7.0 before this branch and is untouched, so the caret range already
resolved there. Nothing in the diff uses the crate.
- Split the AGENTS.md "Adversarial Validation" policy change out of this
branch. The edit is defensible on its own, but it relaxes the review gate
that this branch has to pass, so it should land as its own PR reviewed on
its own merits rather than bundled with the change that benefits from it.
The reverted hunks are unchanged and ready to re-apply.
Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.
* test(ci): serialize the new bucket-incarnation tests
The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.
Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.
Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.
* test(ecstore): pin that a drained bucket is actually deletable
`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.
That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.
Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.
The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.
* fix(ecstore): let an explicit null-version delete purge its delete marker
Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.
On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:
explicit null-version purge of the null delete marker must succeed,
got [Some(MethodNotAllowed)]
so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.
The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.
This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.
Normalize both sides through `delete_file_info_version_id`.
The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.
* fix(app): drop the now-needless struct update
Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.
Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.
* test(e2e): assert the fresh-volume classification
four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.
Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.
* fix(bucket): apply the requested Object Lock to existing buckets
Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.
Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.
An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.
Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".
cargo nextest run -p rustfs-ecstore --lib: 3633 passed.
* fix(ecstore): box the generation-checked config mutation paths too
The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.
* fix(restore): keep the nil-version normalization the strip removed
Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:
- self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
+ self.version_id == fi.version_id
For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.
Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
358 lines
24 KiB
TOML
358 lines
24 KiB
TOML
# nextest configuration for RustFS.
|
|
#
|
|
# Serialize the ecstore tests that share the process-wide disk registry or
|
|
# exercise a multi-disk commit handoff across nextest process boundaries.
|
|
#
|
|
# * store::bucket::tests::bucket_delete_* share process/global state (disk
|
|
# registry, lock client) and race make_bucket into InsufficientWriteQuorum
|
|
# when run concurrently with other ecstore tests.
|
|
# * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
|
|
# uses the shared multipart fixture and a deterministic uploadId-lock
|
|
# handoff, so it must not overlap another process mutating that fixture.
|
|
# * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
|
|
# uses the shared transaction lock and must not overlap other ecstore tests.
|
|
#
|
|
# serial_test's #[serial] attribute does NOT serialize these across runs:
|
|
# nextest executes each test in its own process, where the in-process
|
|
# serial_test mutex has no effect. A nextest test-group with max-threads = 1 is
|
|
# the mechanism that actually serializes across nextest's process boundary.
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# Profiles
|
|
# ---------------------------------------------------------------------------
|
|
# The `default` profile is what local `cargo nextest run` uses. It NEVER
|
|
# retries: a red test locally means a real failure to investigate, not noise to
|
|
# paper over. The `ci` profile (below) is the strict CI gate: global
|
|
# retries = 0 so a new race's first occurrence is never masked, plus a
|
|
# narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN
|
|
# flake issue. Flake policy lives in docs/testing/README.md.
|
|
|
|
[test-groups]
|
|
ecstore-serial-flaky = { max-threads = 1 }
|
|
|
|
# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
|
|
# server and manipulate its disk directories at runtime (crates/e2e_test:
|
|
# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
|
|
# are correct in isolation but resource-heavy; serialize them under nextest's
|
|
# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
|
|
# servers never run at once. ci-7's nightly picks these up via the e2e suite;
|
|
# they are deliberately NOT in the fast PR `e2e-smoke` filter.
|
|
e2e-reliability = { max-threads = 1 }
|
|
e2e-inline-boundaries = { max-threads = 1 }
|
|
|
|
# --- default profile (local): serialize the flaky groups, never retry --------
|
|
[[profile.default.overrides]]
|
|
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
|
|
# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
|
|
# commit + GET cycles — the same cross-disk-commit IO shape that made
|
|
# concurrent_resend load-sensitive. Preventive serialization only, no retries.
|
|
# The matching ci-profile override is after [profile.ci].
|
|
[[profile.default.overrides]]
|
|
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# Serialize the durable manual-transition checkpoint test across nextest's
|
|
# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
|
|
[[profile.default.overrides]]
|
|
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
|
|
# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
|
|
# OnceLock state that serial_test's #[serial] cannot protect across nextest's
|
|
# process boundary, and they delete+recreate buckets — the same shape that
|
|
# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
|
|
# retries. The matching ci-profile override is after [profile.ci].
|
|
[[profile.default.overrides]]
|
|
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# Serialize the 4-disk reliability / degraded-read e2e tests (see the
|
|
# e2e-reliability test-group note above). The matching ci-profile override is at
|
|
# the end of the file, after [profile.ci] is declared.
|
|
[[profile.default.overrides]]
|
|
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
|
|
test-group = 'e2e-reliability'
|
|
|
|
[[profile.default.overrides]]
|
|
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
|
|
test-group = 'e2e-inline-boundaries'
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
|
|
# ---------------------------------------------------------------------------
|
|
[profile.ci]
|
|
# Strict: a new race must fail on its first occurrence, never be retried away.
|
|
retries = 0
|
|
# Report every failure in one run instead of bailing on the first.
|
|
fail-fast = false
|
|
|
|
[profile.ci.junit]
|
|
# Emitted to target/nextest/ci/junit.xml; uploaded as a CI artifact.
|
|
# Tests that pass only after a quarantine retry are marked `flaky` here — that
|
|
# marker is the observable signal the flake policy is built around.
|
|
path = "junit.xml"
|
|
|
|
# ===========================================================================
|
|
# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
|
|
#
|
|
# RULES (enforced by review, see docs/testing/README.md):
|
|
# * Every entry MUST link exactly one OPEN issue tracking the flake.
|
|
# * An entry stays until the issue is fixed (test made robust) or the test is
|
|
# deleted — 30-day policy. No entry may exist without a live issue link.
|
|
#
|
|
# Each entry also re-declares the `ecstore-serial-flaky` test-group so the
|
|
# serialization holds under the ci profile (nextest evaluates a named
|
|
# profile's own overrides list, not the default profile's).
|
|
# ===========================================================================
|
|
|
|
# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
|
|
# make_bucket into InsufficientWriteQuorum via shared global state under load.
|
|
[[profile.ci.overrides]]
|
|
filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
|
|
test-group = 'ecstore-serial-flaky'
|
|
retries = 2
|
|
|
|
# Keep deterministic ECStore write handoffs isolated across nextest processes.
|
|
[[profile.ci.overrides]]
|
|
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
|
|
# on producer/consumer timing windows that stretch past the budget on loaded
|
|
# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
|
|
[[profile.ci.overrides]]
|
|
filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
|
|
retries = 2
|
|
|
|
# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
|
|
# profile too (see the e2e-reliability test-group note near the top). Not a
|
|
# quarantine: no retries, just single-threaded so several 4-disk servers never
|
|
# run concurrently when ci-7's nightly runs the full e2e suite.
|
|
[[profile.ci.overrides]]
|
|
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
|
|
test-group = 'e2e-reliability'
|
|
|
|
# Serialize the multipart crash-consistency scenarios under the ci profile too
|
|
# (see the matching default-profile override near the top). Not a quarantine:
|
|
# no retries, just serialized 4-disk cross-disk-commit IO.
|
|
[[profile.ci.overrides]]
|
|
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# Serialize the durable manual-transition checkpoint test under the ci profile
|
|
# too. No retries: failures stay visible.
|
|
[[profile.ci.overrides]]
|
|
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
|
|
# too (see the matching default-profile override near the top). No retries.
|
|
[[profile.ci.overrides]]
|
|
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
|
|
test-group = 'ecstore-serial-flaky'
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
|
|
# ---------------------------------------------------------------------------
|
|
# PR smoke subset of the e2e_test crate (backlog#1149 ci-4). This profile is
|
|
# the single wiring mechanism for e2e tests in CI: other suites join by
|
|
# extending this filter (or a sibling profile), never by adding ad-hoc e2e
|
|
# jobs to ci.yml. Admission criteria (see crates/e2e_test/README.md): fast,
|
|
# single-node topology, no external dependencies (no awscurl / Vault / fixed
|
|
# ports / pre-started server), no #[ignore].
|
|
#
|
|
# Each e2e test spawns its own rustfs server on a random port with an isolated
|
|
# temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
|
|
#
|
|
# Replication failure harness (backlog#1147 repl-8): the first clause admits
|
|
# its four in-process fake-target self-tests. They bind random loopback ports,
|
|
# use no external service, and finish in under a second.
|
|
#
|
|
# Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
|
|
# FAST bucket-replication tests from replication_extension_test — the
|
|
# target-registration / replication-check / list / remove / delete admin paths
|
|
# that validate config synchronously and never wait for asynchronous
|
|
# replication convergence. Each spawns its own single-node rustfs server(s) on
|
|
# random ports (source, plus an independent single-node target for the pair
|
|
# checks — NOT a cluster), so the subset stays parallel-safe and single-digit
|
|
# seconds. The data-plane tests that poll for convergence and all
|
|
# `_real_dual_node` / `_real_single_node`
|
|
# site-replication tests run in the [profile.e2e-repl-nightly] lane below, NOT
|
|
# here. This allowlist is the single source of truth for the PR/nightly split:
|
|
# the nightly profile derives its set as "the replication module MINUS this
|
|
# allowlist", so any new replication test lands in nightly by default (never
|
|
# silently unrun) until it is explicitly blessed as fast here. Keep the two
|
|
# regexes byte-identical. Count invariant: 20 here + 28 nightly = 48 total
|
|
# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
|
|
# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
|
|
# (#4724) because they set a loopback (127.0.0.1) replication target that the
|
|
# SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
|
|
# the guard now honours an off-by-default opt-in and this suite's source servers
|
|
# set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
|
|
# restored.
|
|
#
|
|
# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
|
|
# S3 auth-rejection suites join the first clause above by module name —
|
|
# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
|
|
# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
|
|
# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
|
|
# smoke admission criteria unchanged. This is the wiring step that makes those
|
|
# merged suites actually execute on every PR (they were dead until listed here).
|
|
# A rename that drops any of them out of this filter would silently thin the
|
|
# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
|
|
# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
|
|
# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
|
|
# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
|
|
# (protocols::test_protocol_core_suite) binds fixed ports and needs the
|
|
# ftps,webdav features, so it cannot join this random-port, default-feature
|
|
# profile; its GHSA-r5qv sibling is a unit test that already runs in the
|
|
# test-and-lint `--all --exclude e2e_test` pass. See
|
|
# docs/testing/security-regressions.md for the full CI-execution map.
|
|
#
|
|
# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
|
|
# admits the hermetic transition e2e. Like the fast replication pair checks it
|
|
# spawns a second independent single-node server (the cold RustFS tier), not a
|
|
# cluster, so it keeps the lane's parallel-safe / no-external-dependency
|
|
# properties. The RustFS warm backend has no loopback guard (that guard is
|
|
# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
|
|
[profile.e2e-smoke]
|
|
default-filter = """
|
|
package(e2e_test) & (
|
|
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
|
|
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
|
|
| test(/^reliant::lifecycle::/)
|
|
| test(/^reliant::tiering::/)
|
|
)
|
|
"""
|
|
fail-fast = false
|
|
|
|
[profile.e2e-smoke.junit]
|
|
path = "junit.xml"
|
|
|
|
# The pagination boundary cases can stall when a server/listing regression
|
|
# prevents the continuation request from completing. Keep the timeout scoped
|
|
# to those known failure modes so legitimate lifecycle/tiering waits retain
|
|
# their test-level timing budget.
|
|
[[profile.e2e-smoke.overrides]]
|
|
filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
|
|
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
|
|
# ---------------------------------------------------------------------------
|
|
# backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
|
|
# tests that are unfit for the per-PR e2e-smoke gate:
|
|
#
|
|
# * 2 remote-target TLS validation tests.
|
|
# * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
|
|
# and poll until source and target converge; two replicate over HTTPS, two
|
|
# pin active SSE failure contracts, and one guards event/history observers.
|
|
# The SSE-S3 contract remains ignored under backlog#1291.
|
|
# * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
|
|
# servers and drives the cross-process site-replication control plane.
|
|
# * 1 `_real_three_node` site-replication test.
|
|
# * 1 `_real_single_node` service-account round-trip test.
|
|
#
|
|
# The set is defined as "everything in replication_extension_test that is NOT
|
|
# in the e2e-smoke PR allowlist above" (the negated clause is byte-identical to
|
|
# the allowlist), so a newly added replication test automatically runs here
|
|
# until it is explicitly promoted to the fast PR subset — no replication test
|
|
# is ever silently left out of CI.
|
|
#
|
|
# #[serial] does NOT serialize under nextest (process-per-test; see the file
|
|
# header). These tests need no cross-test serialization: each spawns its own
|
|
# server(s) on random ports with isolated temp dirs, so they are parallel-safe
|
|
# by construction — the same property the e2e-smoke subset relies on. If load
|
|
# on the runner surfaces a real flake, quarantine the specific test with an
|
|
# OPEN issue link (ci-10 / backlog#937 policy), never blanket-retry or exclude.
|
|
#
|
|
# Wired by .github/workflows/e2e-replication-nightly.yml (schedule +
|
|
# workflow_dispatch), which builds the rustfs binary once, installs awscurl so
|
|
# the STS dual-node test actually exercises its path (it skips gracefully with
|
|
# a visible log line when awscurl is absent), and routes scheduled failures
|
|
# through .github/actions/schedule-failure-issue (ci-8). Explicit division of
|
|
# labor with ci-5's future e2e-full merge gate: these tests run ONLY here, not
|
|
# double-run there. TODO(ci-7): fold this interim repl-owned lane into the ci
|
|
# domain's consolidated scheduled e2e workflow once it exists.
|
|
[profile.e2e-repl-nightly]
|
|
default-filter = """
|
|
package(e2e_test)
|
|
& test(/^replication_extension_test::/)
|
|
& !test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
|
|
"""
|
|
fail-fast = false
|
|
|
|
[profile.e2e-repl-nightly.junit]
|
|
# Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
|
|
# workflow as the failure-triage artifact.
|
|
path = "junit.xml"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
|
|
# ---------------------------------------------------------------------------
|
|
# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
|
|
# workflow_dispatch). Runs the never-automated user-visible suites — KMS (40),
|
|
# object_lock (33), multipart_auth (109), quota, checksum, encryption,
|
|
# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
|
|
# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
|
|
# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
|
|
#
|
|
# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
|
|
# * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
|
|
# ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
|
|
# * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
|
|
# (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
|
|
# namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
|
|
# object_lambda) — too heavy for the merge budget; they run in ci-7's
|
|
# nightly 4-node lane.
|
|
# * replication_extension_test — repl-1 already splits it into the PR
|
|
# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
|
|
# it for those, so e2e-full does not double-run it.
|
|
# * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
|
|
# manual-localhost:9000 reliant/policy tests are ci-13's migration.
|
|
#
|
|
# Each e2e test spawns its own single-node rustfs server on a random port with
|
|
# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
|
|
# parallel-safe — the same property e2e-smoke relies on. The exception is the
|
|
# 4-disk reliability / degraded-read fault-injection tests, serialized below
|
|
# (identical to the ci profile) so several 4-disk servers never run at once.
|
|
# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
|
|
# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
|
|
# product failures cannot be quarantined away with retries, so each family is
|
|
# excluded here with its tracking issue, under the same discipline as the
|
|
# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
|
|
# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
|
|
# negative-path siblings of each family stay in as regression guards.
|
|
# * rustfs#4843 — over-limit archive entry paths hard-reject the whole
|
|
# archive even under ignore-errors semantics.
|
|
[profile.e2e-full]
|
|
default-filter = """
|
|
package(e2e_test)
|
|
& !test(/^protocols::/)
|
|
& !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
|
|
& !test(/^replication_extension_test::/)
|
|
& !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
|
|
& !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
|
|
"""
|
|
fail-fast = false
|
|
|
|
[profile.e2e-full.junit]
|
|
# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
|
|
path = "junit.xml"
|
|
|
|
# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
|
|
# (see the e2e-reliability test-group note near the top of this file). Not a
|
|
# quarantine: no retries, just single-threaded so several 4-disk servers never
|
|
# run concurrently.
|
|
[[profile.e2e-full.overrides]]
|
|
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
|
|
test-group = 'e2e-reliability'
|
|
|
|
[[profile.e2e-full.overrides]]
|
|
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
|
|
test-group = 'e2e-inline-boundaries'
|