Keep native publication recording in main's native_io helpers and resource
writers, and adapt merged tests to the owned Body, Bytes write input, and
fallible find_disk.
Record native publications on the paths main's commit-fusion work opened
directly: the lazy truncate writer and the fused delete-recovery activation
and receipt renames. Keep extract_params_header in the shared audit builder.
* feat(scanner): share node admission with background heal
* fix(scanner): drain candidates after listing completion
* fix(scanner): yield cache leases for shared node admission
* test(scanner): attest adaptive admission in process receipts
* fix(concurrency): declare Tokio time dependency
* feat(concurrency): retain bounded physical resource subsets
* fix(scanner): retain resource ownership in native filesystem jobs
* feat(concurrency): acquire physical groups before node work
* feat(scanner): gate background reads by physical device
* feat(concurrency): combine preallocated resource selections
* feat(scanner): share physical topology across disk plans
* fix(storage): retain only native disk resource subsets
* feat(storage): admit auxiliary physical resources without waiting
* feat(heal): admit physical resources before object repair
* fix(storage): retain resources during native format probes
* fix(storage): retain resources through direct writer I/O
* fix(storage): retain resources through buffered writer I/O
* fix(storage): retain physical resources through streamed reads
* fix(storage): retain resources through native filesystem jobs
* fix(storage): retain physical ownership during bitrot reads
* fix(ecstore): retain resources during positioned reads
* fix(ecstore): retain staged metadata write ownership
* fix(ecstore): retain native io_uring read ownership
* fix(ecstore): enforce receiver-side background admission
* fix(ecstore): retain resources through whole-file native I/O
* fix(ecstore): retain physical ownership during durable rename sync
* fix(storage): account for retained background stream buffers
* fix(storage): retain memory charges for background read results
* fix(storage): account for native io_uring read allocations
* fix(storage): retain memory charges for direct-write buffers
* fix(ecstore): account for heal and RPC reader buffers
* test(scanner): verify samples and clean concurrent collectors
* fix(storage): retain inline and HTTP writer memory reservations
* perf(rio): reuse reserved HTTP writer storage
* fix(rpc): reserve background scanner pipe storage
* perf(rpc): reuse reserved response reader storage
* feat(scanner): measure cache copy allocation requests
* fix(rpc): budget namespace scanner responses
* fix(scanner): retain the enabled supervisor task handle
* fix(scanner): budget namespace frame read staging
* fix(scanner): account namespace envelope decode storage
* fix(heal): retain and join the MRF consumer on shutdown
* test(heal): isolate process-global MRF consumer fixture
* fix(heal): join manager workers during shutdown
* test(heal): verify shutdown at native read boundaries
* fix(heal): drain channel commands before stopping MRF
* fix(heal): admit resume scans and retain replacement workers
* build: pin profiling sources and trust fixed ownership dependencies
* fix(heal): yield node admission while accepted native work drains
* fix(startup): retain shutdown work and preserve deadline failures
* fix(heal): admit inventory probes and retain replacement work
* fix(shutdown): preserve maintenance and frontend join failures
* fix(shutdown): retain HTTP child tasks through real retirement
* fix(shutdown): join actual TLS reload work before retirement
* test(multipart): preserve failures before fault barriers
* fix: keep physical admission inside repository guards
Route the multipart lock diagnostics filter through the e2e storage
compatibility module, reuse the parent module's s3s body aliases in the
namespace memory test, and share one Reed-Solomon verify error mapping
across the reserved-buffer and default paths.
* fix: reconcile physical admission with main's new call sites
Main added a source-admission metadata write layer, a CAS write fence,
fixture writers and scanner handoff tests against the old borrowed-slice
and node-permit shapes. Pass owned Bytes through the new metadata layer,
answer the scanner handoff with ScannerNodePermit, gate the test-only
disk lookup re-export, and clear the lint findings that stable 1.99
reports for the shutdown and admission code without suppressions.
* test(concurrency): name reversed attachment groups clearly
* fix(ecstore): box disk operation futures below admission layers
Every LocalDiskWrapper operation reaches the LocalDisk future through the
health, admission and timeout layers (track_cleanup_mutation, the
track_disk_health family, the io-permit and native-resource admission
split, and track_admitted_disk_operation). Each layer embedded that future,
and unoptimized builds copy it into every awaiting frame, so one local
delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's
default 2 MiB stack.
Box the operation future once where track_admitted_disk_operation creates
it, through a non-async helper so the temporary never stays in a polled
frame. The layers above now hold only a pointer; the cost is one
allocation per local disk operation.
* fix(heal): box object heal futures below admission
The admission wrappers took the object heal future by value, so unoptimized
builds copied it into three awaiting frames, about 650 KiB per heal. Create
it through a non-async helper instead, leaving only a pointer in those frames.
* test(ecstore): count the I/O backend's root FD owner
* test(heal): isolate durable repair during root recovery
* test(heal): isolate the direct repair fallback
* fix(lock): use remaining acquire budget for retries
* test(lock): use virtual time for retry budget
* fix(lock): retain quorum failures across deadline retries
* test(heal): isolate multi-pool crash recovery
* fix(lock): retain quorum error on final timeout
* fix(heal): allow reads during read repair reconstruction
* fix(heal): retain deep scan retry for read repair
* test(heal): wait for metadata write tail
* perf(ecstore): fuse local PUT/DELETE commit I/O and eliminate redundant metadata disk reads (#9038)
- Fuse authoritative bucket metadata existence and parse reads via tokio::join!, add a LOCAL_META_MUTATION_GEN-validated local authority cache, and piggyback one-shot quota snapshot reads onto single-PUT Object Lock snapshot loads.
- Reuse prepared ObjectOptions in post-write lifecycle transition checks instead of re-reading bucket versioning and Object Lock config after commit.
- Defer temporary shard file open/mkdir/write into a single spawn_blocking call via LazyLocalFileWriter, pre-sync single-block shards before page-cache reclaim, and fuse rename_data preparation + commit into a single Fsync-lane dispatch on Linux.
- Enable rename_data old_current_source reuse by default across all-local sets, guard early-ack commit on quorum source stability, and skip redundant post-commit tmp_delete when all online local disks succeeded.
- Enable data-first batch_shard_pread when deferred parity handles are available and replace zero-filled allocation with Vec::with_capacity + read_to_end for offset-0 shard reads.
- Defer pre-delete get_object_info_for_delete to PrefixAccessDenied error handling, skip redundant single-pool unversioned delete pre-stats, and add a fused local delete preparation and 2-barrier settlement fast path.
Fixes#9038
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): preserve delete identity and durability policy
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): cover fused delete recovery stages
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve inline versions after commit sync failure
* perf(ecstore,s3): accelerate PutObjectLegalHold, CopyObject, and large PUT fdatasync (#9038)
- Fuse LocalDisk::update_metadata (read xl.meta + FileMeta::load + update_object_version_with_opts + marshal_msg + staged tmp write + rename_all_with_precondition_and_parent_sync_std) into a single Fsync-lane blocking operation, and skip redundant get_bucket_info / get_object_info calls in put_object_legal_hold when object lock is enabled and replication is not configured (PutObjectLegalHold: 314.8 -> 988.3 ops/s, 3.14x faster, 0.95x of MinIO).
- Pass destination bucket as orig_volume through create_bitrot_writer_with_orig_volume into LocalDisk::create_file so single-shot non-inline part.1 writes under syncs_data_shards() durability run fdatasync before fadvise(POSIX_FADV_DONTNEED) and skip the duplicate sync_dir_files_std(src_data_path) in FusedCommitAttempt without fsyncing the temporary parent directory (warp PUT 1MiB: 658.3 vs MinIO 656.8 obj/s, 1.00x parity, p50 26.10ms vs MinIO 46.60ms).
- Add single_pool() fast paths to ECStore::handle_copy_object and ECStore::handle_new_multipart_upload, skip no-op destination prelookup get_object_info when put_like_write_creates_new_version is true, and backfill previous_current_size from OldCurrentSize in copy_object (CopyObject 1KB: 398.4 -> 1,086.1 ops/s, 2.73x faster, 1.01x faster than MinIO).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): provide non-Linux has_rename_data_test_failpoints stub for Windows CI
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): preserve presynced shard directory durability
Sync the shard directory before publishing presynced payloads and keep the file sync test probe available on non-Linux targets.
* fix(ecstore,s3): fix non-consuming crash_inject check, test hooks, and rubber-duck findings (#9038)
- Add non-consuming crash_inject::is_armed(point, key) so FusedCommitAttempt and LocalDisk::update_metadata do not consume one-shot crash arms before the fallback path runs (fixes crash_consistency::* and meta_write_crash_consistency::* tests).
- Check INLINE_PREPARATION_BEFORE_BACKUP and RENAME_DATA_AFTER_FIRST_PUBLICATION in has_rename_data_test_failpoints so inline preparation admission and observed-owner panic tests run on their expected path.
- Resolve staged_dir via get_object_path_for_io in test_rename_data_presynced_shard_keeps_directory_sync and avoid premature orig_volume access check during tmp staging in put_object (fixes put_object_failure_cleans_tmp_workspace_inline).
- Address rubber-duck review findings: fall back from FusedCommitAttempt on any existing xl.meta when sync_commit_metadata is true, fsync object dir after activating delete rollback dir before unlinking xl.meta, classify data-dir entries before removal in fused delete finalization, include self_copy_lock in copy_prelookup_required, and avoid .expect() on non-test mutexes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(ecstore): defer shard sync until destination is known
* test(rustfs): cover legal hold replication fast path
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* fix(s3select): match request enum values case-insensitively
ExpressionType, CompressionType, CSV FileHeaderInfo, JSON Type and CSV
QuoteFields were compared exactly, so lowercase values such as
`lines`, `document` or `gzip` were rejected. MinIO accepts any case.
Match them ASCII case-insensitively in request validation and rewrite
them to the canonical uppercase spelling, so every later comparison
keeps working unchanged. Unknown values keep their existing errors.
* fix(s3select): keep source key order for JSON LINES input
Arrow's JSON schema inference walks each record as a serde_json::Map,
which sorts keys unless serde_json's process-wide preserve_order
feature is on. JSON LINES `SELECT *` therefore returned keys sorted,
and CSV output columns were permuted.
Infer the LINES schema from the same object bytes and record limit as
DataFusion's JsonFormat, keep Arrow's types, and reorder fields
(nested structs included) by first-seen key order. Enabling
preserve_order globally was rejected: it changes map ordering for the
whole binary.
Drop the test-only preserve_order dependency so the crate's tests run
with production features; the DOCUMENT inference reference now
compares against Arrow's types in source order.
* fix(s3select): name nested path projections after their last key
`SELECT s.obj0.id` returned the key `s.obj0[id]`, DataFusion's display
name for the field access. AWS names an unaliased path projection after
its last key, e.g. `{"id": ...}`.
Alias such projections to their last key when that name is unique among
the output and ORDER BY column names. Queries with a wildcard keep
today's names, since the expanded columns could collide.
* fix(s3select): preserve names beside computed projections
* fix(s3select): preserve names for ambiguous path projections
* fix(s3select): name path projections from the planned schema
Path projections that the SQL rewrite leaves unaliased are now named
after their last key once DataFusion has planned the query. A path takes
its key only if no other output column keeps or receives that name. A
colliding rename is dropped and the check repeats, so the final names
are unique whenever DataFusion's own names are.
The SQL rewrite is unchanged, so GROUP BY can still refer to the names
it assigns and every query that planned before still plans.
Review cases:
- s.obj0.id, s.obj1.id, s.obj2."s.obj0[id]" keeps the three generated
names: the third key would duplicate the first retained name.
- s.obj."lower(s.id)", LOWER(s.id), in either order, keeps the generated
path name: its key is the computed column's name.
Paths beside computed columns, beside duplicated keys, and in
parentheses now take their key, as AWS S3 Select names them. Paths stay
unnamed when an ORDER BY expression is not projected, as before.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
A request refused by the authorization layer was logged as
s3_authorization_denied but never reached the audit stream. Emit an audit
entry from DenialContext::deny with the account, bucket, object, denied
action and denial reason. Entries are bounded by a per-class token bucket
(lower for anonymous callers) with a suppressed count carried on the next
entry, quiet probe denials are skipped, DeleteObjects audits its refused
keys as one aggregate entry, and the S3 response is unchanged.
A replicated force-delete that a target rejected (for example a peer whose
credentials lack s3:ForceDeleteObject) was only logged; the per-target
replication counters never saw it, so `replication status` reported zero
failures while the sites had already diverged.
Build a ReplicatedTargetInfo for every target outcome of
replicate_force_delete_to_targets (missing client, offline target,
rejected remove, success) with a Pending previous status, and feed each
through ReplicationStats::update so rejections count as failures and
accepted force-deletes count as replicated.
Document that the managed site-replicator-0 policy must name
s3:ForceDeleteObject explicitly and that a bucket-replication target with
its own credentials needs the same grant.
* fix(audit): replay queued payloads for internal event names
Audit entries are persisted by the store targets and read back through
serde. Three internal events (CreateMultipartUpload, AbortMultipartUpload,
DeleteObjects) serialized to a string that EventName::deserialize rejected,
so every queued entry for them was dropped on replay. Read the persisted
event name through a lenient deserialize_with function; EventName's own
Deserialize and parse stay strict because they also back notification
configuration.
* fix(audit): emit entries for multipart parts and bucket config writes
UploadPart, UploadPartCopy, AbortMultipartUpload and the mutating bucket
configuration operations never produced an audit entry: the audit path
required a bucket notification event and these operations have none, or
(AbortMultipartUpload) never built a helper. Add an audit-only
OperationHelper constructor that builds no notification event and wrap
the S3 handlers for these operations with it. Operations without a
notification event use one coarse audit-only event name, with the
operation in api.name.
---------
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
* chore(deps): refresh crates and adopt released aws-sigv4
Run two update/upgrade rounds for the main and standalone fuzz workspaces.
Replace the temporary SigV4a Git override with aws-sigv4 1.6.4 from the
registry and remove its obsolete source permission.
Pin OpenID Connect to e20ddbb01ca68356f2979bc125f5613ab45b4e7e. Upgrade
async-nats to 0.51.0, hotpath to 0.30.0, libunftp to 0.24.0, and the TOML
test dependency to 1.1.8. Retain Arrow/Parquet 59.3.2 for DataFusion 55.2.0
and keep the fuzz PKCS#1 version compatible with pinned RSA rc.18.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): retain safe openidconnect pin against HMAC panic
The e20ddbb pin verifies HMAC signatures with mac.verify(signature.into()),
which panics in generic-array on malformed length instead of returning
InvalidIdentityToken. Retain the previous reviewed pin that uses
verify_slice and add a malformed-signature regression covering HS256,
HS384, and HS512.
* chore(deps): upgrade sysinfo to 0.40.0
Update the main and standalone fuzz lockfiles. Adapt fallible system
queries, CPU usage accessors, and OS-native network interface names.
Preserve process sampling fallbacks and report unavailable diagnostic
sources without panicking. Cover missing host and process collectors.
Validation: 167 focused tests passed; production and all five fuzz targets
passed cargo check. Formatting, logging guardrails, and the existing
dependency policy passed. Linux and Windows runtime checks were not run.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): use the admin error facade for sysinfo failures
Keep the sysinfo initialization error behind the existing admin S3 facade.
The facade builds the same error code and message without expanding the
direct S3 error-macro surface guarded by the migration ratchet.
Validation: S3 footprint guard, formatting, whitespace, and production
cargo check with sftp, hotpath, and connect_top_net passed.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh hotpath and ctutils (#9055)
Run two cargo update/upgrade rounds in both the main and fuzz workspaces
on top of PR #9047. Upgrade the hotpath family to 0.30.1 and ctutils to
0.4.7. Retain fuzz pkcs1 0.8.0-rc.4 for the pinned RSA 0.10.0-rc.18;
rc.5 fails to compile against that RSA version.
Validation: 1096 library tests passed with 19 skipped; upstream ctutils
and hotpath-drain property/concurrency suites passed. Production and all
five fuzz targets passed cargo check. Formatting, S3 footprint, and the
existing dependency policy passed.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Probe local data roots and fixed metadata directories before admitting disks. Report runtime and owner IDs with migration guidance for root-created volumes, while retaining unavailable registry slots for existing quorum handling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sync main 9601a7c4b into gateway/integration. Ports main-side additions to the
owned admin boundary (plugin delegation/attempt handlers, notification target
edit tests), drops a new ecstore s3s::dto reference in favour of the persisted
lifecycle filter, and refreshes the e2e selection digests and test-count
baseline for the merged tree.
* perf(ecstore): avoid eager parity reads for demand-bound GET
Reuse the existing batch setup and quorum targets to retain deferred parity
without materializing bytes that demand-bound reads immediately discard.
Keep legacy all-shard batching and verify degraded and ranged recovery.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): validate batch reads on the opened shard
Avoid repeated pathname traversal by reading metadata from the descriptor
used for positioned I/O. Reject nonregular shards without blocking and
preserve range, metrics, and cancellation behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Eliminate redundant .bucket-incarnation/xl.meta quorum reads and distributed
bucket-metadata transaction read locks under held lifecycle read locks and
Object Lock configuration snapshots across put_object_metadata, CopyObject,
UploadPartCopy, RestoreObject, read_object_versions_for_lifecycle, and
get_bucket_incarnation_id_for_options_in.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
Wait for eight completed delete attempts instead of auto-advancing through
300 seconds of replay scans against a durable on-disk store. Retain the
original virtual upper bound and assert hook/store state after cancelling
and joining the worker.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Reuse the owned frame's decoded record with bounded canonical size checks
and the existing scanner validation/application core. Keep the complete
outer parser and own the journal state fields without a JSON round trip.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): identify stale multipart cleanup failures
Preserve quorum failure metrics while avoiding an UploadPart warning for background cleanup. Keep the existing write quorum and source recovery checks intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): skip redundant stale upload part scans
Once the default deadline is due, lifecycle rules cannot make an upload ineligible. Avoid querying lifecycle and listing parts in that case while preserving the locked recheck.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): validate multipart lifecycle size
Skip part scans when no abort rule uses size. Treat unavailable, incomplete or overflowing sizes as unknown and keep size-independent lifecycle rules eligible.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* fix(ecstore): resolve dangling latest versions in metadata updates
A failed versioned PUT whose rollback also failed can leave the new
version on fewer drives than its data blocks. GET and HEAD of the
latest version already look past such versions, but read-before-write
callers without a version id still answered 503: object tagging read
each drive's top version with the quorum-error read, and
put_object_metadata and the CopyObject source read selected from
read_all_xl, whose merged latest version is missing on the other
drives, so object_quorum_from_meta failed before selection.
Object tagging now reads through a variant that runs the same walk past
dangling versions as a latest GET while keeping the quorum error for a
dangling key. put_object_metadata and the CopyObject source try the walk
when their quorum or selection step fails with read quorum, before the
existing dangling cleanup could report a key with a committed version
absent. A version-only copy takes the served version's own copies and
write quorum. The walk still applies only to S3 buckets, latest reads,
and sets where every drive answered, and the update still targets only
the served version's disks, so a version below write quorum is refused
before writing.
* fix(ecstore): refuse version-only copies below write quorum up front
Review of the dangling-latest metadata updates found three gaps:
- A version-only CopyObject wrote the new version and reverted on a
write-quorum miss; refuse it before writing when the served version's
disks cannot reach its write quorum, like update_object_meta_with_opts.
- Replication status write-back, including the zero-write ValidateOnly
mode, now keeps its read-quorum error and never walks past dangling
versions, since the walk queues heal for the versions it skips.
- The erasure-coding doc overstated that these callers never report a
dangling key as absent; put_object_metadata and CopyObject keep their
existing dangling cleanup when the walk finds no readable version.
Record local batch eligibility, blocking queue and execution time, shard
outcomes, and actual read bytes behind the existing GET-stage metrics gate.
Keep captured metric handles with the blocking worker after cancellation
and use scoped low-latency histogram buckets for the new duration series.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rebalance): reuse the target pool holding a dangling copy
Rebalance places a version with a fresh free-space choice whenever the
exact-version pool lookup finds no target copy. An attempt that loses
write quorum can leave the version on fewer drives than read quorum,
and object reads report such a copy as absent (#8963), so the retry
could commit the version on a different pool and leave it in two. The
distributed kill-before-source-cleanup case saw this: a delete marker
whose first copy failed with InsufficientWriteQuorum ended on pools 1
and 3 after the source was cleaned up.
Before the free-space fallback, data movement without a decommission
capacity owner now targets an eligible pool where any drive already
holds the exact version, so the retry rewrites that copy to full
quorum instead of creating a second one. This covers both the object
copy path and the delete-marker path.
* fix(rebalance): check free space before reusing a holding pool
The pool holding an earlier copy was returned before any space check,
so a full holder pinned every retry and could stall a deferred entry.
The holder now passes through the same filters as the free-space
choice (suspended, rebalancing and source pools, has_space_for for the
expected size, and the max-used filter), computed once per selection;
placement falls back to that weighted choice when the holder does not
qualify. The two free-space helpers share one implementation.
The tests no longer rely on random placement. A test-only override
pins each target pool's reported capacity so the free-space choice
always picks the non-holder pool, the earlier copies keep the version
ids but carry different bytes and modification times, and the moved
versions are checked for marker state, modification time, data on
every drive and readable bytes. A second test covers a full holder.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): select the latest version at read quorum like MinIO
GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.
Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.
* fix(ecstore): resolve selection ties by disk order and guard writes
Review of the read-quorum selection found three gaps:
- Rejecting a same-time half split made explicit-version reads of a
partially applied tagging update fail and blocked the retry that
repairs it. Resolve ties to the identity on the lowest-index disk,
for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
below the selected version's data blocks, so HEAD could report a
version GET cannot decode. Require that many copies, as MinIO's
getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
metadata writes are not rolled back. Refuse an update before writing
when its target disks cannot reach write quorum.
* fix(ecstore): count identity copies in the data-blocks guard
The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.
* fix(ecstore): keep commit-proof selection for system-bucket reads
Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.
Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
* fix(ecstore): serve the latest readable version past dangling versions
A failed versioned PUT whose rollback also failed can leave the new
version on top of fewer drives than its data blocks. Reading only each
drive's top version then left the committed version below it short of
read quorum, so GET and HEAD of the latest version answered 503 although
that version is intact in every stack. MinIO resolves the stacks in
pickLatestQuorumFilesInfo.
When every drive answered and an S3 latest read still cannot select a
version, walk the newest version headers of each stack through
exact-version reads: serve the readable version with the newest selected
modification time, skip versions whose exact read proves them dangling
(the NoSuchVersion an explicit GET reports) and queue heal for them, and
keep the read-quorum error on any other outcome. Candidates are ordered
by each version's newest modification time across drives, and at most
eight exact reads are spent.
Test PUTs now wait for every drive so tests that edit one drive's
metadata no longer race the PUT's background tail writes.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
* fix(ecstore): preserve canonical order in latest-version fallback
* test(decommission): cover held target lock and partial-copy resume
Port two tests from the closed lock-recovery branch onto the deferral
that defers any typed namespace lock timeout on a version copy.
- A version commits, then every attempt at the next one times out on
the target lock: the entry is deferred as a lock-timeout deferral, no
failed item is counted, both source versions stay, and the resumed
entry moves both versions with their exact bytes.
- A real write lock held on the target key makes target selection
return a typed object-lock timeout, the entry defers instead of
failing, and once the lock is released selection recovers and the
resumed entry completes the move.
* chore(ci): shrink the error(format!) ratchet baseline for rebalance
The rebalance control and meta modules on main now have fewer
Error::other(format!) call sites than the baseline records, which
fails the shrink-only ratchet in Quick Checks. Record the smaller
counts.
overtrue