mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-27 06:18:43 +00:00
codex/plugin-runner-integration
247 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 overtrue
|
991d647c78 | chore: merge main into plugin runner integration | ||
|
Hauser
|
0f24511535 |
test(e2e): harden multipart fault and TLS acceptance (#8907)
* test(e2e): tighten scoped multipart fault acceptance Target Abort root deletion instead of child transaction cleanup, require actual metadata peer hits, and keep faulted APIs single-attempt. Validate bounded ordinary recovery and safe quorum refusal for partial mutations, with the recovery enhancement tracked in backlog#2940. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(e2e): drive concurrent TLS reads independently Give each coordinator its own supervised task and cooperative polling budget while retaining the existing workload, deadline and byte/hash assertions. Preserve node logs after fixture teardown and report the operation, response phase and received bytes when a transfer times out. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
overtrue
|
a5ad283c8c | merge: sync main for plugin integration validation | ||
|
overtrue
|
bdd7efc96d | fix(plugins): reconcile runner integration with current main | ||
|
Hauser
|
aaeedcd1d7 |
perf(rpc): remove stream boxing and verify distributed TLS reads (#8885)
* perf(rpc): remove redundant boxing from read streams Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks. Document protocol comparison commands and the old-node workaround during rolling upgrades. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): preserve TLS trust for alternate credentials Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): collapse acceptance report path guard Combine the optional report path and existing-file checks into a let chain without changing the report or PID sidecar collision behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
882217221e |
fix(lock): reach quorum behind silent voters (#8895)
* test(e2e): add MPU RPC quorum and recovery acceptance Exercise separate storage and lock RPC delays across multipart phases, including quorum boundaries, cancellation, bucket recreation, and bounded eight-node load. Honor TMPDIR for isolated cluster drive directories. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(lock): probe healthy voters behind silent peers Spread additional probes across the existing acquisition attempt budget when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds, fast-path fan-out, and existing ownership and late-response cleanup. Cover every four-node read-voter ordering, a stalled eight-node write quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and eleven metadata acknowledgements. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
8d5af5981d |
fix(s3): preserve aws-chunked error responses (#8875)
* fix(s3): preserve aws-chunked error responses Map named aws-chunked failures through the shared bounded body classifier, keeping typed source precedence and the dependency's S3 response contract. Cover exact responses, upload publication, and valid signed/trailer limits. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(s3): classify upload stream errors once per cause Reuse the typed UploadStreamError result for digest and incomplete-body classification while preserving the shared classifier's marker priorities. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): refresh aws-chunked profile selections Record the two new aws-chunked regression cases in smoke and full profiles. Verify exact additions against the CI Linux inventory and native Darwin listings, recovering each previous digest when those two cases are removed. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
911ea265c5 |
fix(storage): apply and index explicit delete-marker removals (#8854)
* fix(ecstore): purge every explicit delete-marker version Keep explicit marker removals on the canonical version-delete path so same-key batches are not truncated and pending replication does not retain source markers. Preserve marker identity and timestamps in the response. Add real four-disk regressions for mixed version cleanup, duplicate marker IDs, pending replication, and null/UUID marker removal. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(filemeta): index explicit delete-marker batches Reuse the batch index and final compaction for ordinary marker removals. Keep small marker-only histories on the scalar path and preserve fallback for free versions and complex mutations. Add scalar-equivalence, boundary, corruption and deterministic scan-count coverage, a real-disk mixed-history regression, and reproducible marker benchmark cases. Preserve existing benchmark defaults while honoring CLI sampling overrides. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(filemeta): remove redundant marker fixture clone Move the original metadata into its final test fixture after the last borrow, avoiding the redundant clone reported by strict Clippy checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): validate explicit marker deletion and recovery Exercise UUID/null and duplicate explicit marker removals over real HTTP with retries disabled, including immediate four-node visibility and exact retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process termination, rollback/commit recovery, and old-ID retries across a second restart without deleting a fresh marker. Use flat sampling and fixed per-case fixture batches in the metadata benchmark. Update profile selection digests for the added regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): synchronize crash fixture startup and flush readiness Give disk setup and the durable flush boundary separate readiness gates, keeping each phase bounded to ten seconds. Preserve the real SIGKILL and restart assertions while reporting early child exits and retaining output. Reap fixture children on assertion failure and cover early-exit diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
d9ab8760ee |
test(e2e): accept retried rebalance deferrals; retry busy decom preflight (#8813)
test(e2e): accept retried rebalance deferrals and retry busy decommission preflight |
||
|
Hauser
|
89c3282ae2 |
test(scanner): reserve capacity for all native backlog fixtures (#8802)
Include single-pool, status, and expansion fixtures in the existing native scanner reservation without changing budgets or assertions. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
5f5485e332 |
fix(s3): return InvalidPart and streamline multipart completion (#8780)
* fix(s3): classify empty multipart completion as InvalidPart Treat absent per-drive multipart transaction directories as empty listings so completion reaches the existing InvalidPart validation. Preserve quorum and transaction recovery, and cover the HTTP and degraded-drive boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): move multipart completion metadata in one pass Reuse quorum-validated part metadata instead of repeatedly inserting, sorting, finding and cloning parts. Keep legacy cleanup identities and ordering while moving the original part allocation into the completion manifest. Compare ETags by borrowing their normalized values. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
65f0599c70 |
feat(metrics): track usage outcomes and trim decommission probes (#8781)
* feat(metrics): expose usage candidate rejection outcomes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): borrow usage candidate fixtures Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): reserve capacity for decommission probes Reserve all nextest slots for the two eight-disk decommission fixtures under both default and CI profiles. Keep their internal concurrency, production lock budgets, assertions and zero-retry policy intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): scope prewrite capacity probes to the target pool After persisting a new capacity intent, refresh only the target pool baseline. Preserve the global admission and finalization scans, lock order, fencing, and accounting formulas. Verify the reduced probe count with real disk metrics and preserve injected snapshot/error behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
b06923beaf |
test(e2e): add scanner/heal topology protection matrix (#8705)
* test(e2e): add scanner/heal topology protection matrix Add the S00 protection baseline for the Scanner/Heal program (rustfs/backlog#2873, program rustfs/backlog#2872): seven scanner_heal_topology_ cases in the distributed e2e lane that pin pool expansion, rebalance and decommission behavior before the runtime changes. The fixture seeds a deterministic inventory (empty, inline, external, multipart, special and nested keys; a versioned bucket with an explicit null version, three history versions and a delete marker per key; an Object Lock bucket with legal hold and COMPLIANCE retention), records every version with its requested and listed version id, length and SHA-256, verifies the manifest through every node, reads physical placement from xl.meta on the pool drives, and reads the strict durability mode back from the admin API and the node logs. Fault cases hold the mover at proven windows through file barriers that compile only with the e2e-test-hooks feature: before the multipart target commit and before source cleanup (shared by decommission and rebalance), plus the existing pool-metadata before_publish cutpoint. An unreached window fails the case. DistCluster gains capture_logs_under so a case can read node startup events without a global environment variable. docs/testing/scanner-heal-topology-gates.md maps the gates to the tests and records the timeout basis. * test(e2e): keep the barrier failure message static The error(format!) ratchet rejects new formatted io::Error::other call sites in ecstore. The reached observation already records the stage and object, so the injected failure needs no per-call detail. * test(e2e): require e2e-test-hooks for topology fault cases The four scanner_heal_topology_ fault cases inject at server barriers that exist only with the e2e-test-hooks feature. Against a default server they waited for the barrier timeout and reported it as not reached. Check the verified binary's features first so the missing prerequisite fails immediately and is named. * ci(e2e): build the distributed lane with e2e-test-hooks Every other e2e lane builds and runs the fault-test hooks server. The distributed lane alone used the default server, so the topology fault cases could never reach their barriers there. Build and run with the same feature so the receipt matches, and update the documented local commands. The upgrade cases spawn the pinned previous release directly and do not compare feature sets. * test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR The topology fixture captured node logs, barrier observations and evidence in a temp dir removed on drop, so the distributed workflow uploaded nothing for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow uploads, when no explicit evidence directory is set. * test(e2e): remove retired pools before restart; race the pre-ACK start The queue case restarted with decommissioned pools 0 and 1 still on the command line, which the server rightly refuses. Add remove_single_node_pools and restart without them, then expect the remaining pools renumbered to 0 and 1. The pre-ACK case now races the start request against the barrier. A start that returns first, or stays in flight past the documented 60 s budget, is reported as such instead of as an unreached barrier, and the evidence records the reach latency. capture_logs_under now overrides a log path assigned from RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too. * test(e2e): bound fault-window waits by the movement budget On Linux CI a busy target capacity gate delayed the first fault object by about two minutes, past the 120 s wait borrowed from the server's release poll. Wait for a movement window within the 240 s movement budget instead, and fail at once when the decommission or rebalance ends without entering the window, so an uninjected fault is never counted. * test(e2e): bound status probes while waiting for a fault window The admin decommission status request can wait on a mover held at a test barrier until the barrier's own 120 s release timeout. The window wait issued that request each loop, so a reached barrier went unnoticed until the server timed it out and failed the decommission, which surfaced as a false resume failure. Bound each status probe to 5 s and treat a slow probe as still running. * test(e2e): retry a busy decommission start A decommission start that meets a busy movement gate can answer a retryable 503 instead of waiting. The harness treated any non-proof failure as fatal. Share one start helper that retries 503 within the case's movement budget and the fleet-proof 500 for 45 s, and keeps every other status fatal. The pre-ACK case retries 503 within its window budget. |
||
|
Hauser
|
2eeaca4cd0 |
fix(iam): derive bucket location conditions from XML (#8744)
* fix(iam): derive bucket location conditions from XML Capture the parsed CreateBucket LocationConstraint at the access boundary and move it into shared authorization conditions independently of signing and server regions. Cover conditional allow/deny across user, session and anonymous requests, including absent and spoofed values. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(iam): pass bucket location directly to authorization Replace temporary request-extension storage with an explicit owned XML argument to the single authorization core, keeping the public entry point compatible. Remove the marker allocation and typemap insert/remove path. Cover Object Lock and Versioning dependent authorizations with an absent location condition and verify their enabled state through real HTTP. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): isolate bounded object probe storage Use an independent temporary volume on Linux shared memory when available so the real endpoint/CLI fixture does not depend on contended CI disk IO. Await server shutdown before dropping the caller-owned storage directory. Retain the production cleanup deadline, measurement budget, assertions, and existing nextest capacity reservation without adding retries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
65a721d896 |
fix(ecstore): reindex pool metadata ids after removal (#8733)
* fix(ecstore): reindex pool metadata ids after removal * ci(nextest): reserve resources for table catalog lock probe * fix(ecstore): reindex retained decommission references * fix(ecstore): use typed pool metadata rebuild errors |
||
|
Hauser
|
13c37bf8b9 |
fix(s3): validate bucket location constraints (#8739)
* fix(s3): validate bucket location constraints Reject explicit bucket locations outside the configured server region before mutation admission, preserve site-default and EU alias compatibility, and return an empty GetBucketLocation element for us-east-1. Cover matching and rejected locations over real S3 HTTP requests, including rejection without creating a bucket. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): refresh bucket region profile selection digests Include both new region regression tests in the smoke and Darwin/Linux full selection pins without changing filters or checker behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
a5ae294f42 |
fix(quota): refresh peer metadata after admin quota changes (#8735)
* fix(quota): refresh peer metadata after admin quota changes * test(e2e): update distributed selection for quota regression |
||
|
GatewayJ
|
4dba635d48 | fix(table-catalog): authorize warehouse location claims (#8736) | ||
|
overtrue
|
7c85783a08 | chore: merge main into plugin runner integration | ||
|
Chris
|
e6f76cb64f |
fix(ci): refresh the e2e-full selection for the new policy tests (#8693)
* fix(ci): refresh the e2e-full selection for the new policy tests #8615 and #8671 added three e2e tests without refreshing the e2e-full selection digest, so "Verify e2e full membership" has failed on every main push since. Regenerate the Linux digest from the listing the failed run uploaded; the same procedure reproduces the previously committed value from the last green run's listing, and the name-level diff shows three additions and no removals. The darwin digest is refreshed separately from a real macOS listing. * fix(ci): refresh the darwin e2e-full selection digest Regenerated from a real `cargo nextest list --profile e2e-full -p e2e_test` on macOS (rustc 1.99.0, cargo-nextest 0.9.140), so the local check on macOS agrees with the three policy tests added by #8615 and #8671. |
||
|
overtrue
|
cf2e3af4d9 | merge: sync main into plugin runner integration | ||
|
overtrue
|
8233e3fe4c | feat(storage): enable distributed artifact PUT sealing | ||
|
唐小鸭
|
222beaba00 |
fix(scanner): certify dirty usage per bucket and complete cycles with pending buckets (#8542)
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): attribute activity guardrail vetoes per field
The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.
* feat(data-usage): carry pending-bucket certification in DataUsageInfo
Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.
* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets
A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.
* feat(scanner): accept unconverged complete publications as the cache baseline
complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.
Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.
* feat(admin): expose scanner guardrail vetoes in usage freshness
Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.
* docs(scanner): record per-bucket certification in the design and changelog
Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.
* refactor(scanner): drop the unreachable failed-bucket input from the pending set
A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.
* test(scanner): name the activity field mutation table type
* test(scanner): fill the post-walk gate in merged replication fixtures
The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.
* ci(nextest): run the admin target-repair fixtures on the ecstore base stack
repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (
|
||
|
Chris
|
3defc19d5e |
fix(ci): refresh E2E selection and isolate disk lock probes (#8638)
* fix(ci): include the baseline writer test in full E2E selection * fix(ci): reserve capacity for short-deadline disk lock probes |
||
|
cxymds
|
628663f468 |
fix(heal): recover replacements after pool removal (#8600)
* fix(heal): recover replacements after pool removal * test(e2e): gate upgrade probes on writable baseline nodes * test(e2e): register baseline readiness regression in smoke digest |
||
|
cxymds
|
ce00d8f7e5 |
feat(s3): support conditional bulk deletes (#8535)
* feat(s3): support conditional bulk deletes * fix(s3): report null marker identity in bulk delete responses * fix(s3): isolate conditional bulk delete actions and tier guards * fix(tier): preserve backend identity validation diagnostics --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
cxymds
|
388d1f17e6 |
fix(heal): safely recover missing bucket directories by default (#8517)
* fix(heal): safely recover missing bucket directories by default * fix(heal): allow bound admin recovery before namespace checks |
||
|
Hauser
|
c7848df9a0 |
fix(build): enforce GNU ABI floor and canonical target metadata (#8524)
* fix(build): pin gnu glibc baseline to 2.31 and report compile target in version metadata gnu release binaries were built natively on Ubuntu 24.04, embedding GLIBC_2.38/2.39 symbol references while the RPM/DEB metadata promised glibc >= 2.31, so Rocky/RHEL 9 (glibc 2.34) could not run them (#8062). The two linux-gnu legs now build through cargo-zigbuild pinned to the 2.31 baseline (same-arch; zig acts as the linker), with post-build guards asserting both the embedded version metadata and the highest referenced GLIBC symbol version, keeping the package declarations honest. Version metadata itself now comes from cargo's TARGET cfg instead of shadow-rs' host stamping, so builds whose host differs from their target (cross compilation, musl targets on a glibc-hosted toolchain) report the platform they were compiled for (#8048 follow-up). The cross toolchain (zig 0.14.x, cargo-zigbuild 0.23.4) is pinned in the setup action now that it is load-bearing for releases, the build job gets an explicit matrix-independent name so the connect-* acceptance workflows keep matching it when matrix fields change, and the preview-release workflow contract is updated for the TARGET_ARG build commands. Fixes #8062 Follow-up on #8048 * fix(build): enforce GNU ABI floor and canonical target metadata Pin Zig 0.17.0 and cargo-zigbuild, isolate cross-toolchain caches, and build GNU releases against the declared glibc 2.31 floor. Validate both release ELF binaries using their loader version requirements. Generate target metadata once through shadow-rs and check the actual release version output without adding a parallel metadata facade. Refs: rustfs/backlog#2824 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(build): reuse GNU release profile across binaries Switching the CLI back to the default strip profile restarted 419 compilation units after the server build. Keep the profiling build profile for both binaries, then strip only the CLI artifact. Retain serial links, server profiling symbols, and post-strip ABI gates. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(build): limit release artifacts to the rustfs server Build only rustfs in the release workflow and remove the CLI build, strip, and packaging paths. Preserve GNU profiling symbols and catalog sidecars, and align strict acceptance archive readers with the new set. Exercise all six build targets, Linux package checks, and all four strict acceptance extractors using their actual workflow scripts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
adf17bbfba | fix(heal): reject incomplete administrator bucket discovery (#8512) | ||
|
Hauser
|
34803e8165 |
perf(ecstore): isolate cleanup and preserve metadata authority (#8289)
* chore: pin s3s to upstream git revision Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4. Allow the official s3s git source in cargo-deny so source checks continue to pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): instrument put object hot path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): split cleanup worker wait metrics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * upgrade version * chore(metrics): add PUT post-store stage probes Add lightweight PUT post-store stage probes for issue 2615 validation. * fix cargo deny check warn * upgrade version * perf(ecstore): bound and isolate cleanup execution (#8151) * perf(ecstore): bound and isolate cleanup execution * fix(ecstore): preserve cleanup cancellation and lease release * feat(ecstore): default isolated cleanup to CPUs 1-2 * test(ecstore): decouple budget tests from CPU affinity * upgrade version * perf(ecstore): reuse pinned directory handles for object I/O * feat(observability): add sampled local GetObject stage summaries Add an opt-in GetObject stage aggregation path that works without the OTLP metrics export. When RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED is set, io-metrics keeps per-(path, stage, object class, size bucket) sampled counts and duration totals in a bounded in-process map, and a startup task flushes them periodically to /var/log/rustfs/get-stage-summary.log, tunable via the sample rate and interval env keys. * build(deps): bump workspace dependency releases Patch releases: async-compression 0.4.49, tokio-rustls 0.26.6, xxhash-rust 0.8.19, suppaftp 12.1.1, russh-sftp 3.0.1. Minor release: hotpath 0.27.0. * fix(ecstore): fence distributed metadata cache mutations Add authenticated cluster-wide mutation fences for the opt-in DistErasure GetObject metadata cache. Guard recursive prefix and batch deletes, fail closed on configuration mismatch or uncertain quorum, and cover replay, restart, peer loss, and mixed-configuration behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT observers on improve/base (#8039) * perf(object): instrument storage hot paths Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT stage observation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> * build(deps): bump workspace dependency releases * ci: allowlist rustix OFlags WRONLY in typos config * refactor(storage): route ecstore symbols through storage_api boundaries * perf(ecstore): cache quorum-selected early-stop metadata (#8308) Publish eligible latest-version GET metadata after a validated read quorum when early-stop cancels the remaining fanout. Keep metadata errors, selection quorum, online read quorum, and mutation-generation checks as publication gates. Add slow-peer repeat-hit and concurrent mutation-fence regression tests. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: merge latest main into improve/base (#8332) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): borrow metadata cache lookup key (#8333) Avoid request-scoped Arc<str> allocations on eligible cache lookups while preserving full key identity. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: sync improve/base with latest main (#8334) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish metrics by process identity (#8343) Attach process.pid to the OpenTelemetry resource and promote it for Prometheus so counter series from restarted processes remain distinct. Preserve the stable RustFS node and service instance attributes. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync origin/main into improve/base (#8345) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(storage): retain typed owners across borrowed writes (#8329) * fix(storage): retain typed owners across borrowed writes * fix(storage): preserve Copy write completion policy * refactor(storage): remove redundant write owner references * fix(ecstore): hide rollback backup residue in prefix listings * test(connect): isolate short HTTP fixtures in nextest * fix(protocols): enforce S3 policies for file protocol operations (#8262) * test(ecstore): enforce main core regression gates (#8338) * test(ecstore): enforce main core regression gates * test(ecstore): pin external shards and completion regressions * fix(test): restore authenticated cluster fault scenarios (#8340) * fix(test): restore authenticated cluster fault scenarios * fix(test): use the E2E storage API boundary * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * fix(ecstore): own listing workers through cancellation (#8342) Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route commit sync through guarded helper Use the Linux-capable guarded directory sync entry point for both commit sync sites. This keeps group commit and file-sync admission behavior while ensuring the already-open commit directory remains the sync target. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): expose rename directory guard in production The commit path now acquires the directory guard in production builds, not only tests, so destination metadata reads and durable sync share the guarded context. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> * fix(ecstore): remove unsafe metadata cache admission paths Keep distributed GET metadata on quorum reads and quota admission on authoritative storage. Preserve local cache invalidation, remove the experimental RPC surface, and gate diagnostic work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): cover cold degraded mutation admission Check first PUT and DELETE with one peer absent and no SDK retries, then verify that losing write quorum still rejects writes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain guarded Linux destination metadata reads Use the existing commit directory guard when reading old xl.meta on Linux. Cover symlinked destination metadata for inline and non-inline renames without changing the split commit structure. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(test): remove retired cache RPC import Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * merge: sync latest main into improve/base (#8349) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync main auth and scanner updates (#8352) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) * perf(scanner): reduce scan and checkpoint peak allocations (#8346) * perf(scanner): reduce scan and checkpoint peak allocations * test(scanner): isolate partial commit recovery scenarios * fix(auth): return ExpiredToken for retained STS credentials (#8344) * fix(auth): return ExpiredToken for retained STS credentials Validate session identity independently of credential expiry in both IAM rejection and claims validation, and preserve typed S3 errors at the caller. Keep token binding, disabled-account precedence, and service-account behavior. Add exact expiry, IAM, and HTTP/XML regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): synchronize the metadata slowtail hedge regression Park the slow metadata task at a test-only fault checkpoint and release healthy reads after the fault is active. Assert the spare response and cancelled-task drain without imposing a 300 ms disk/scheduler deadline. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ilm): synchronize copy compensation transition stages Observe automatic compensation reaching the mock tier PUT checkpoint before waiting for the final metadata commit. Assert forced enqueue failure, automatic backfill, one completed tier upload, and payload integrity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(ecstore): explain cleanup scheduler conflict Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(metrics): separate GET request and timeout outcomes (#8354) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * docs: refresh security advisory lessons (#8350) * fix(heal): bound durable MRF rescheduling (#8351) Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(rustfs): route recovery errors through S3 gateway Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(io-metrics): use is_multiple_of for sampling Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): satisfy strict Clippy pagination checks Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid cloning Copy purge identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * add ignore .mimosa * fix(zip): adapt archive readers to zip 9 APIs Upgrade zip and its required dependencies alongside fallible archive-name handling in log ingestion, report bundles, bucket metadata import, and archive tests. Preserve existing skip and error propagation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reuse quorum metadata for body GETs Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): align storage test stacks and peer reset classification Apply the existing bounded ECStore stack setup to the affected cross-crate storage fixtures. Validate conditional-only metadata publication separately from full body-layout cache reuse. Classify typed h2 shutdown resets as unreachable while retaining application, authentication, cancellation, and deadline distinctions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve omitted-disk GET recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore: integrate published workspace dependency updates Preserve the concurrent dependency refresh and retain the typed h2 reset classification and bounded storage-test stack setup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): preserve snapshot size budget after fallback move * fix(test): use one barrier for compensation COPY Let the forced-enqueue helper own the PUT barrier and compensation assertions. Waiting on an outer barrier overwritten by the helper always timed out after the upload had completed. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(ecstore): apply current lint fixes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update sha2 to 0.11.1 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
Hauser
|
0a6aeda191 |
test(e2e): bound scanner witnesses by the reported schedule (#8501)
* test(e2e): bound scanner witnesses by the reported schedule Require a fresh completed observed walk and account once for planned superseded waiting, preserving the active-cycle execution budget. Cover invalid schedules and early wakeup, and refresh exact platform selection digests without removing existing tests. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): refresh nightly scanner budget membership Record the three scanner budget regressions in both platform nightly selection digests, preserving every previously selected test. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
367f13f3f9 | ci: reduce repeated PR runner work (#8463) | ||
|
cxymds
|
b8889fccc9 |
fix(ecstore): allow cold-start recovery at EC read quorum (#8458)
* fix(ecstore): load existing formats at EC read quorum * fix(ecstore): complete startup at EC read quorum |
||
|
Hauser
|
a464b45367 |
fix(rustfs): keep recovery handler within S3 gateway (#8429)
* fix(rustfs): route recovery errors through S3 gateway Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): satisfy strict Clippy pagination checks Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid cloning Copy purge identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: restore one-second drive request budgets * fix(ecstore): resume pending lock bucket creation Allow retries to reuse an uncommitted Object Lock bucket intent when the same incarnation remains unretired and all physical volumes are absent. Isolate the recovery entrypoint test from the shared production leader lock. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test: provision stack for deep catalog test futures Apply the existing 4 MiB test-thread stack setup to table-catalog handler/store tests and the relocated-pool GET regression in both local and CI nextest profiles. These cases overflowed libtest's 2 MiB default stack on Linux. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Chris
|
0d1d1487f3 | test(ecstore): cover mixed metadata arrival orders (#8385) | ||
|
Chris
|
3824779520 | test(ecstore): cover late GET object generation changes (#8383) | ||
|
shurkus
|
47ed6717de |
fix(tier): drop reserved internal headers from transition PUT (#8363)
* fix(tier): drop reserved internal headers from transition PUT * test(tier): transition PUT against a target rejecting reserved headers The fake S3 target gains a mode that answers 400 InvalidArgument to any PutObject or UploadPart carrying an x-minio-internal-* request header, the rule MinIO and the Storj gateway enforce. RT-14 transitions an inline and a non-inline object through a real server to that target and asserts the stored bytes, both transition identity pairs and read-back. The e2e-full linux selection digest is refreshed for the new test. --------- Co-authored-by: Vector <vector@wcomp.io> Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
Chris
|
14adc5ed3f |
test(ecstore): cover real metadata cache and late identity reads (#8371)
* test(ecstore): cover real metadata cache and late identity reads * test(e2e): bound max-keys fixture upload concurrency |
||
|
Chris
|
4783adff36 |
test(ecstore): enforce main core regression gates (#8338)
* test(ecstore): enforce main core regression gates * test(ecstore): pin external shards and completion regressions |
||
|
overtrue
|
7c289001ed | test(connect): isolate short HTTP fixtures in nextest | ||
|
RJ Regenold
|
48a6104aad |
feat(iam): expose stored bucket tags to OPA (#8298)
* feat(iam): expose stored bucket tags to OPA * fix(ci): lint bucket tag fixtures and register new e2e tests * fix(ecstore): restore strict Clippy compatibility on Rust 1.99 Use try_update without changing atomic ordering or overflow behavior. Keep recursive storage futures boxed once at each frame and remove the redundant async-recursion macro, including its non-recursive SQL planner use. Remove needless closure borrows and orphaned dependency entries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> (cherry picked from commit |
||
|
Chris
|
641c4b3493 |
chore(release): merge 1.0.1 back into main and refresh installation references (#8314)
* fix(ci): include pagination regression in full E2E selection * fix(deps): replace yanked yoke-derive release * fix(scanner): expose pause backlog replica diagnostics (#8258) * fix(scanner): expose pause backlog replica diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(connect): stabilize runtime profile lease cancellation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): tolerate delayed schedule startup in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retry quota reads during usage warmup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid meta-bucket incarnation self-deadlock Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(test): use persisted incarnation in heal fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): reconcile stale counters after lifecycle expiration (#8108) * fix(usage): reconcile stale counters after lifecycle expiration * fix(usage): account lifecycle expiry during continuous writes * test(usage): run lifecycle usage scenarios on one scanner store * test(usage): use a Windows-representable pre-mutation offset * fix(usage): harden expiry accounting recovery and quota checks Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(usage): recover stale persisted counts through the scanner Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test. Refs rustfs/backlog#2689 Refs rustfs/backlog#2691 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): bound delimiter pagination fixture concurrency The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(upgrade): establish a persisted previous-release baseline Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions. Refs rustfs/backlog#2689 Refs rustfs/backlog#2384 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): bound cancellation metadata persistence waits Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions. Refs rustfs/backlog#2697 Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain startup and shutdown failure diagnostics * fix(test): supply CPU workload for sampler regression * fix(ci): locate security chain scripts in the workspace * fix(usage): recover historical counters with generation fencing (#8273) * fix(usage): recover historical counters during continued writes Use newer converged scanner snapshots to reconcile stale absolute usage baselines while preserving concurrent mutation and expiry receipt fences. Cover durable publication, admin and quota reads, and legacy generations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(usage): fence snapshots and move preserved cache entries Apply the cached scanner generation floor before every reconciliation path and retain it even when an older snapshot happens to match core counts. Move preserved usage entries instead of cloning their histogram maps under the cache lock, retaining expiry receipt identity and cancellation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): verify checkpoint takeover and repair dispatch (#8275) * test(scanner): cover checkpoint handoff and repair dispatch Drive runtime budget expiry, partial-cycle persistence, leadership claims, and stale checkpoint rejection between real disk-backed fixture scans. Verify that a metadata repair beyond the first bounded prefix is saved in the scanner ledger and dispatched by the MRF consumer. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: isolate scanner fixtures and refresh full e2e membership Reserve nextest capacity for the real-disk scanner publication and MRF admission fixtures. Bind both platform membership checks to the reviewed pagination deadline test added on main. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(scanner): consolidate checkpoint fixture lifecycle Keep one durable control store across timeout and leadership transitions, and inject generation advancement into the shared checkpoint scenario. Check the actual saved metadata path so late-write rejection also proves that existing checkpoint bytes remain intact. Centralize MRF fixture isolation and reuse nextest process isolation when the startup environment already satisfies the test contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish allocator counters from live memory (#8274) * fix(obs): distinguish allocator counters from live memory Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(obs): parse allocator statistics from one node Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate late parity recovery from metadata hedges Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): combine identical snapshot retention branches * fix(test): await HTTP sender readiness in Top RPC fixture * [release/1.0.1] Gate multipart copy through write admission (#8284) Gate multipart copy through write admission Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * Gate multipart copy through write admission (#8283) Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix: add UploadPart OOM validation guardrails (#8287) * docs(release): validate candidates on release branch * fix(scanner): validate checkpoints against global cycle fence (#8278) ## Related Issues Related to rustfs/backlog#2701. ## Summary of Changes Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission. ## Verification Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance. ## Impact Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired. ## Additional Notes Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior. * fix(ci): restore E2E membership and pagination timeouts (#8281) * ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279) ## Related Issues Follow-up to #8229. ## Summary of Changes Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence. ## Verification The exact PR head |
||
|
Chris
|
a33dd896e2 | fix(ci): restore E2E membership and pagination timeouts (#8281) | ||
|
Chris
|
2806a80c91 |
fix: prevent metadata lock reentry and stabilize CI fixtures (#8257)
* fix(test): establish a writable previous-release upgrade baseline * fix(ci): reserve capacity for durable admin fixtures * fix(ci): group durable IAM state fixtures by resource needs * ci: run E2E doctests with the E2E dependency graph * fix(ci): separate fixture startup from transport deadlines * fix(ci): bound pagination after seeding and revisit restored copies * fix(ci): make filesystem fixture timing deterministic * fix(ecstore): avoid metadata lock reentry during internal mutations * test: align recovery fixtures with durable ownership contracts |
||
|
Chris
|
5c3a2ff277 | fix(ci): reserve capacity for profile cancellation probe (#8259) | ||
|
Hauser
|
d60dfbb826 |
fix(heal): finalize durable MRF legacy responsibility lifecycle (#8254)
## Related Issues
Resolves the corrected lifecycle fixture findings in PR #8254.
## Summary of Changes
Preserve separate incarnation-bound durable MRF responsibilities and their lifecycle audit records, and align replay fixtures with their checkpoint identities.
## Verification
Two independent source reviews and changed-delta reviews are complete. The final review approved
|
||
|
Chris
|
3268c42e00 |
fix(ci): stabilize registration and rolling upgrade readiness (#8244)
## Related Issues Follow-up to #8233. ## Summary of Changes Registration runtime fixtures timed out in the workspace CI lane while the same cases passed in the feature lanes. Reserve nextest capacity for the exact `connect_registration` binary, as already done for related inventory and drive fixtures. Keep its existing deadlines, internal concurrency, assertions and zero-retry policy; report the last watch status on failure. Rolling upgrades could pass `ListBuckets` readiness while restarted peers still lacked write quorum. Before each mixed-version phase, probe writes through every node outside the asserted workload prefixes. A shared 30-second deadline includes requests and sleeps; only HTTP 503 with `ServiceUnavailable` is retryable, with SDK retries disabled for these probes. The actual compatibility writes, reads, multipart operations and listing assertions remain unchanged. Add four fast regression tests to the existing PR smoke profile, with matching exclusion from the full profile. No new workflow or job is introduced. ## Verification - `cargo nextest run --locked --profile ci -p rustfs --lib --test connect_registration --test-threads 4 -E 'binary(/^connect_registration$/) | test(=connect::diagnostics::trace_runtime::tests::local_runtime_rejects_non_private_state)' --no-tests fail --status-level pass --final-status-level fail` passed 23/23 twice in 5.088s and 5.097s: 22 macOS registration tests plus one unrelated control. JUnit intervals confirm capacity reservation; no retries or test-process leaks occurred. The Linux-only inventory case remains for CI. - `cargo nextest run -p e2e_test --lib --profile ci -E 'test(upgrade_write_readiness_tests)' --no-fail-fast --no-tests fail` passed 4/4 twice in 1.068s and 1.072s, with zero retries. Regressions cover metadata readiness followed by write unavailability, recovery through every writer, immediate permanent-error failure despite an incoming SDK retry configuration, and deadlines for repeated 503s and stalled requests. The original failure is recorded in [the mixed-version upgrade job](https://github.com/rustfs/rustfs/actions/runs/36569700716/job/109415806473). - `cargo fmt --all --check`, `git diff --check`, `python3 scripts/check_test_wiring.py` and compiled smoke/full membership checks passed. Smoke membership changes from 188 to 192 by adding exactly these four tests; full membership is unchanged. The expected Linux smoke digest was derived from the actual prior Linux listing plus those four platform-independent additions and still requires confirmation by this PR's CI. Local verification covers the exact source committed in `9b2ea836317ed035a91d1fc9fcf725f70c3098e2` on main `380e98a42cb4fcd0994fed79b30c2c7605deb0bc`. An independent final-diff correctness and reliability review found no findings. Fresh Linux workspace and real mixed-version upgrade runs are required before treating the remediation as fully verified; local fake-target tests do not establish that result. ## Impact Test scheduling and readiness only; no production behavior, API, dependency, test deadline or compatibility assertion changes. Reserving capacity serializes registration fixture processes within a nextest run. The bounded readiness probes may add startup time while peer write health converges; permanent errors still fail immediately. ## Additional Notes Rollback by reverting this PR. Existing CI restructuring from #8233 is independent of these follow-up fixes. |
||
|
Chris
|
a88225d208 |
fix(ci): reduce duplicate work and preserve reliable test failures (#8233)
* fix(ci): reduce duplicate work and preserve reliable test failures * fix(ci): retain protocol evidence and repair stale test fixtures * test(connect): honor parent deadline during API fixture readiness * fix(ci): reserve IO capacity for state writer proofs * test(connect): align RPC fixtures with service capture contracts * test(connect): cover pinned service capture failures |
||
|
Chris
|
47af5565c7 | fix(ci): preserve cluster startup failure evidence (#8213) | ||
|
Chris
|
af0ae798f8 | fix(test): serialize replacement_bucket_metadata heal test under parallel load (#8212) |