fix(rebalance): wait for readiness and admit peer workers (#8570)
* fix(rebalance): avoid contention when peers join the same run
* fix(rebalance): wait for fleet proof during initialization
* test(rebalance): check contextualized competing run errors
* fix(rebalance): retain typed worker admission lock errors
* test(rebalance): retry the init readiness 503 and pin the retry error shape (#8586)
The e2e harness only retried the pre-fix 500 carrying the fleet-proof
marker; the init stage now answers 503 + Retry-After after its own
readiness budget, so start_rebalance treats that as the same wait. The
missing-proof regression also asserts the typed error shape that the admin
handler's retry predicate depends on, not only the rendered message.
---------
(cherry picked from commit 7682a12b58)
Co-authored-by: cxymds <cxymds@gmail.com>
fix(multipart): retain commit guards across request cancellation (#8571)
* fix(multipart): retain commit guards across request cancellation
* fix(multipart): preserve commit task error context
* fix(multipart): reclaim verified orphan upload staging (#8572)
* fix(multipart): reclaim verified orphan upload staging
* fix(multipart): share orphan guards across matching lock domains
* fix(multipart): preserve orphan cleanup task error context
* fix(ecstore): box guarded delete transition identity
* fix(multipart): keep NoSuchUpload when orphan recovery cannot prove absence (#8587)
Recovery is a side path of an idempotent Abort. An offline disk, lock
contention, partially overlapping lock domains or unreadable bucket
metadata all stop the staging-absence proof before any deletion; those
used to propagate as 503/500, where the base answered NoSuchUpload. They
now keep that answer and delete nothing; only failures inside the
deletion task remain request failures, because a retry can finish them.
---------
(cherry picked from commit 65f30bd5c1)
Co-authored-by: cxymds <cxymds@gmail.com>
Backport of #8573 (main commit 7a32ca2adb) to release/1.0.2. The five touched files were identical between release/1.0.2 and the main base the fix was written against, so this applied without conflicts.
New encrypted writes store a keyed HMAC fingerprint as the object ETag
(docs/operations/encrypted-etags.md). Replication forwards that ETag on
x-rustfs-source-etag, but a target that computes its own ETag (AWS S3 and
most compatible stores) answers the plaintext PUT with the MD5 of what it
received. Both values are 32 hex characters, so verify_single_part_replica
treated the fingerprint as a comparable MD5 and failed every single-part
SSE-S3 / SSE-KMS replica with "replica etag mismatch"; MRF and resync then
retried the object forever. RustFS and MinIO targets adopt the source ETag
and were unaffected.
Exempt sources whose stored ETag cannot describe the wire bytes: any
managed-SSE or SSE-C source, decided by the existing replication
source-encryption classifier. The checksum header on the replication PUT
remains the integrity check for those objects; plaintext sources keep the
audit unchanged.
Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797
(cherry picked from commit 7a32ca2adb)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Backport of #8574 (main commit d7156290d5) to release/1.0.2.
A user-bucket delete fans out through the delete-transaction RPC. When a
participant answers Unimplemented (a peer still running a release without
that RPC, or a protocol-version mismatch) the coordinator reports
StorageError::UnsupportedDisk, which had no S3 mapping: clients received a
500 InternalError whose message was the variant's Display text,
"drive does not support O_DIRECT". That text describes the startup drive
probe and sends an operator mid-upgrade after the wrong cause.
Map UnsupportedDisk to 503 ServiceUnavailable with a constant message that
names the protocol-version cause and the remedy (finish upgrading every
node, then retry). The typed source is retained for diagnostics and no
Retry-After is added, since retrying the same node does not help until the
cluster upgrade completes.
Conflict resolution against release/1.0.2: the DiskFull arm and the
capacity-failure test from #8553 are not on this branch and were not
carried over; only the UnsupportedDisk mapping, its message, the mapping
table row and the new unit test are included.
(cherry picked from commit d7156290d5)
Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Add a one-second Retry-After hint at the canonical S3 error conversion
only for ServiceUnavailable responses retaining a typed lock timeout.
Keep existing error codes, source chains, and lock budgets unchanged.
Cover actual HTTP/XML serialization, nested causes, private-resource
redaction, and unrelated disk/quorum/text failures. Document the response
contract and the limits of log-count and five-second timeout attribution.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): remove needless returns from Linux disk helpers
* fix(test): verify encrypted purge replicas by version body
* fix(test): expire namespace timeout after fixture preparation
* fix(ci): remove remaining needless collection and borrows
* fix(test): wait for active admin heal at interruption checkpoint
* fix(heal): unblock automatic replacement recovery
Repair bucket publication directories before adopting survivor delete proofs.
Keep explicit empty-volume removal after proof settlement and preserve the
admitted bucket incarnation for administrator replay.
Discover unfinished intents across every local set and resume the durable
owner with its exact target scope before admitting new sibling replacements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): route scanner fixtures through the storage boundary
Expose the existing disk-layout bootstrap symbols as test-only aliases in
heal::storage_api and use that boundary from scanner recovery regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(heal): index replacement discovery within each scan
Keep a single endpoint-keyed disk inventory with first-snapshot precedence
and deterministic discovery order. Index local target lookups and move
resume state into generation candidates instead of cloning it.
Retain fresh durable reads and the existing conflict and mutation fences.
Cover duplicate disk handles, endpoint uniqueness and discovery ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): retain committed journals in recovery fixtures
Build one certified survivor delete decision for both replacement and
empty-volume removal tests. Clear target contents without moving its root
inode so Linux disk handles observe the same empty target. Drain PUT tails
and assert the proof remains before the real Heal entry.
Expose the existing recovery state through the ECStore disk API for typed
fixture checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): use server stack for replacement regression
The replacement regression polls the composed Heal future on libtest's
smaller calling-thread stack and aborts before reporting an assertion.
Run it on a bounded 8 MiB test thread and Tokio workers, matching the
debug server's stack budget. Keep all storage and object assertions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): prepare pinned Compose for Connect acceptance
Install the verified Compose CLI through one local action shared by all 11 connected acceptance entry points. Publish only checksum-verified binaries and keep daemon, provenance and timeout gates unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): use daemon-visible Connect temporary inputs
Match the existing official Connect acceptance TMPDIR contract in all 11 real E2E invocations so daemon bind mounts can read generated inputs. Preserve key modes, read-only mounts and cleanup checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The shared connected preflight uses uuid.uuid7, which is unavailable in older runner Python environments. Select Python 3.14 with the existing pinned setup action in all eleven acceptance callers.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(replication): skip version-addressed marker pre-check on targets that mint their own version ids
* fix(replication): keep bodiless 403 fail-closed and verify resynced markers on targets that mint their own ids
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(build): pin gnu glibc baseline to 2.31 and report compile target in version metadata
gnu release binaries were built natively on Ubuntu 24.04, embedding
GLIBC_2.38/2.39 symbol references while the RPM/DEB metadata promised
glibc >= 2.31, so Rocky/RHEL 9 (glibc 2.34) could not run them (#8062).
The two linux-gnu legs now build through cargo-zigbuild pinned to the
2.31 baseline (same-arch; zig acts as the linker), with post-build
guards asserting both the embedded version metadata and the highest
referenced GLIBC symbol version, keeping the package declarations
honest.
Version metadata itself now comes from cargo's TARGET cfg instead of
shadow-rs' host stamping, so builds whose host differs from their
target (cross compilation, musl targets on a glibc-hosted toolchain)
report the platform they were compiled for (#8048 follow-up).
The cross toolchain (zig 0.14.x, cargo-zigbuild 0.23.4) is pinned in
the setup action now that it is load-bearing for releases, the build
job gets an explicit matrix-independent name so the connect-*
acceptance workflows keep matching it when matrix fields change, and
the preview-release workflow contract is updated for the TARGET_ARG
build commands.
Fixes#8062
Follow-up on #8048
* fix(build): enforce GNU ABI floor and canonical target metadata
Pin Zig 0.17.0 and cargo-zigbuild, isolate cross-toolchain caches, and
build GNU releases against the declared glibc 2.31 floor. Validate both
release ELF binaries using their loader version requirements.
Generate target metadata once through shadow-rs and check the actual
release version output without adding a parallel metadata facade.
Refs: rustfs/backlog#2824
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(build): reuse GNU release profile across binaries
Switching the CLI back to the default strip profile restarted 419
compilation units after the server build. Keep the profiling build
profile for both binaries, then strip only the CLI artifact.
Retain serial links, server profiling symbols, and post-strip ABI gates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(build): limit release artifacts to the rustfs server
Build only rustfs in the release workflow and remove the CLI build,
strip, and packaging paths. Preserve GNU profiling symbols and catalog
sidecars, and align strict acceptance archive readers with the new set.
Exercise all six build targets, Linux package checks, and all four
strict acceptance extractors using their actual workflow scripts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344
Vendor the published Pyroscope package with an object 0.40 dependency and
lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add
bounded decoder regressions, and run the patched package in audit CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(deps): correct vendored pyroscope documentation typos
Correct copied comments and changelog spelling for the repository typo
check, and describe these comment-only changes in the vendor provenance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(ecstore): refresh compact transition proof per test case
Install the short-lived homogeneous fleet proof immediately before each
kill-point transaction. Earlier cancellation and recovery work must not
age the next case's writer capability fixture.
Report an early transition result instead of waiting for an unreachable
kill point. Preserve cancellation, remote-owner and payload assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
---------
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Tie watchdog cancellation to response future ownership before the first
poll, and prefer cancellation when the watchdog deadline is also ready.
Preserve completion logging and genuine slow-request warnings.
Add paused-clock regression coverage for dropped, completed, panicking,
and still-pending request futures.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
refactor(heal): index replacement generation reconciliation
Reconcile owned records and their definitive sources in one pass. Group claims by target slot and use a shared-anchor fast path while retaining pairwise ambiguity checks.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Read the optional socket peer extension supplied by external ingress so
API and Console client limits remain effective without trusted proxies.
Keep validated client precedence and the existing missing-peer policy.
Exercise the full external stack across HTTP request versions, direct
IPv4/IPv6 peers, forged headers, and distinct trusted proxy clients.
Align API and Console fixtures with the production extension contract.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
fix(ecstore): suppress clippy too_many_arguments on rename_data_commit_from_observations
The function takes 9 slice parameters for the renamedata commit phase.
Annotate with #[allow(clippy::too_many_arguments)] to match the existing
codebase pattern for internal multi-slice commit helpers.
Group disk observations into a borrowed struct to keep the commit helper
within Clippy's argument limit without changing quorum snapshot logic.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Capture the authenticated request access key when building the audit entry
and preserve it through completion instead of replacing it with root
credentials. Resolve remotehost from validated client context or the
socket peer, and ignore untrusted forwarding headers.
Cover request identities, completion outcomes, client address precedence,
and audit serialization without credential secrets.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep same-name buckets in independent clusters separate while deduplicating
replicated usage observations. Document the existing scanner-backed exporter
and cover its gauge contract and cluster-scoped queries.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): bound scanner witnesses by the reported schedule
Require a fresh completed observed walk and account once for planned
superseded waiting, preserving the active-cycle execution budget.
Cover invalid schedules and early wakeup, and refresh exact platform
selection digests without removing existing tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): refresh nightly scanner budget membership
Record the three scanner budget regressions in both platform nightly
selection digests, preserving every previously selected test.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Preserve canonical claim encoding for the signature tamper test and pause the admin cancellation fixture at its existing run fence instead of durable deletion.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reuse observations and expose complete usage
Allow superseded retries to reach the existing validated cache planner while
bootstrap remains pending. Serve complete observations in server info without
changing quota authority or dirty acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): route fixture setup through storage boundary
Keep the single-disk scanner fixture on the existing storage_api boundary
instead of importing the ECStore API facade directly.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The conditional_get_304_uses_metadata_without_decoding_damaged_data test
constructed the data directory path from info.data_dir.to_string(), but
the on-disk directory name may not match the UUID string form. Switch to
iterating subdirectories of the object path (matching the
delete_object_part_shards helper) so the test is robust regardless of
the data_dir naming convention.
Verified: cargo nextest run -p rustfs -- conditional_get_304 → PASS
* test(upgrade): separate rc.5 data and candidate query checks
Verify historical fixture bytes before upgrading without requiring the
pinned rc.5 reader to satisfy the candidate's cross-part range contract.
Reuse the complete-body oracle and retain all upgraded range, partNumber,
attributes and replication assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): replace timing races with recovery evidence
Prove no_lock batch progress with the existing cleanup barrier while the
caller retains its guard. Check namespace counters and every drive's
metadata absence rather than using a one-second disk I/O deadline.
Keep durable ILM's manual recovery as the sole tier journal consumer and
verify exact migrated input before retaining the original recovery stats,
receipts and remote-version assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
fix(ecstore): preserve retry evidence during delete cleanup
Bind normal cleanup to the coordinator transaction after quorum succeeds, and remove the rollback backup before marked data. Preserve data and markers if backup cleanup fails.
Add deterministic LocalDisk crash and retry regressions plus real four-disk single and batch coordinator coverage. This prevents one post-quorum cleanup window; it does not resolve ambiguous inline or historical residue in #8464.
Co-authored-by: Zhengchao An <overtrue.us@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): evaluate conditional reads before opening object bodies
Reuse locked metadata snapshots for conditional GETs and unify HEAD
preconditions and cache headers. Preserve body-cache hooks and validate
encryption metadata and key authorization before returning bodyless errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(s3): unify conditional reads at metadata quorum
Apply scoped metadata conditions to the existing reader pipeline instead
of maintaining parallel prepared-read branches. Preserve quorum checks,
early-stop, coalescing, and cache-hook behavior on condition misses.
Track metadata-only termination through selection and reject body access
or a changed decision. Keep concrete reader futures off request stacks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(s3): strengthen conditional read acceptance evidence
Assert zero shard, decode, cache-fill, and remote-body calls on matching
conditions with live positive controls. Exercise GET and HEAD through
s3s and Hyper with conditional-header and precedence matrices.
Add reproducible paired application-read benchmarks with isolated
metadata controls and strict drift, artifact, and resource gates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(s3): remove obsolete conditional comparison helper
Keep ETag comparison tests on the canonical read-condition evaluator,
remove unused date-trait imports, and name the HTTP case type.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): validate full conditional etag fields
Reject adjacent tags and malformed trailing list members before accepting a
match. Keep the existing request-scoped metadata read and authorization order,
while removing redundant validation wrappers and unused benchmark overrides.
Replace simulated phase coverage with real 304/412 cache-fill probes and extend
HTTP/1.1 and HTTP/2 ingress regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): cache terminal conditional metadata safely
Preserve quorum-validated conditional metadata when early termination
cancels the remaining fanout. Restrict those entries to terminal GET
readers; body and full-layout consumers still obtain their own metadata.
Serialize cache publication per key so conditional entries cannot replace
complete layouts or renew their TTL. Retain generation fencing and the
final selected-metadata predicate check before any body access.
Cover real four-disk hedging, nonterminal body and range reads, consumer
isolation, generation races, atomic publication and actual cache expiry.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(log-analyzer): align heal writer failure rule with current logs
Match the current structured all-targets failure, retain legacy log support, and reject partial or unrelated failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Reject causal cycles before source repair and share conservative backup
classification between discovery and inspection. Fence public raw deletion
and derive exact-key removals from successful settlement without an extra
metadata read. Preserve the existing physical phases and strict source checks.
Make outage and renewal fixtures deterministic without weakening their
convergence assertions, and complete strict TLS/RPC lint cleanup.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bound sentinel tests by the lock wait, not recreation IO
Both bucket incarnation sentinel tests gave the bucket recreation that
follows the paused delete (force delete_bucket, make_bucket, put_object
on four fsync'ing disks) 10 s in total. The delete releases the sentinel
milliseconds after the barrier, but on a runner starved for IO the
recreation alone can exceed 10 s:
delete_objects_holds_bucket_incarnation_sentinel_through_commit failed
that way on main in run 37760321449 (job 113259251979).
That bound is not what catches a kept sentinel: delete_bucket waits for
it on its lifecycle write lock, which fails after the lock acquire
timeout, and the recreation's expect reports the lock error. Keep that
check and give the recreation 120 s only as a hang guard.
The recreation starts that lock wait before the barrier is released, so
the wait also has to cover the delete's own commit, and the 5 s default
leaves a starved runner little room for it. Run both tests with
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30 and bound the delete, which was
not bounded, by the same 30 s. A kept sentinel still fails the tests,
after 30 s instead of 5 s.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* test(ecstore): keep background recovery out of dispatch rollback tests
dispatch_manifest_detached_workers_are_store_bounded_and_shutdown_cancellable
seeds six aborting dispatch manifests, installs a pause hook and expects
its own recovery passes to park four detached workers on it. Its first
store also ran the background tier-delete journal recovery loop, which
runs a pass every TIER_DELETE_JOURNAL_RECOVERY_INTERVAL (60 s). When the
setup took longer than that, a background pass rolled the manifests
back before the hook was installed, no worker paused, and the 30 s wait
failed. This happened on main on a runner starved for IO, in run
37760321449 (job 113259251979).
Build the first store with background recovery suppressed, as the
neighbouring dispatch tests do. The restarted store keeps it: the second
half of the test checks that it picks up the retained manifests.
dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member
and dispatch_manifest_page_timeout_detaches_one_deduplicated_worker_until_convergence
have the same exposure: each seeds an aborting manifest on a store with
background recovery and only then installs its slow-delete hook. They
have not failed in CI, but with the setup stretched past 60 s they fail
the same way. Suppress background recovery for them too.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
---------
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
overtrue