Record local read_dir entry counts and duration for live-walker ListObjects scans so wide root/prefix amplification can be measured below the cross-set merge layer.
Add a focused LocalDisk scan_dir test showing a page limit of one still observes the whole parent directory enumeration, plus metric helper coverage.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): lock tier config mutations
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): add mutation RPC auth contract (#5082)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation handler core (#5084)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add mutation control rpc service (#5087)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover prepared mutation drains (#5093)
Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes.
Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): prove zero references before tier removal (#5092)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): clear peer mutation runtime blocks (#5094)
Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery.
Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation control client
Add signed tier mutation prepare, commit, and abort client calls for peer fanout while preserving the existing protobuf and RPC contract.
Verify response proofs before interpreting peer outcomes, fail closed on invalid states, and reject oversized payloads before dialing peers.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): lock tier config mutations
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(tier): add mutation RPC auth contract (#5082)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add peer mutation handler core (#5084)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): add mutation control rpc service (#5087)
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): recover prepared mutation drains (#5093)
Recover prepared tier mutation intent records into the local tier runtime so a restarted peer fails closed before issuing new remote-tier operation leases or conflicting admin publishes.
Reconcile the recovered block map on each scan so committed, aborted, or removed intents clear stale local blocks instead of wedging the peer until process restart.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): prove zero references before tier removal (#5092)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tier): clear peer mutation runtime blocks (#5094)
Install prepared mutation runtime blocks when peer prepare requests are applied or replayed so followers fail closed immediately before restart recovery.
Clear the in-memory block once peer commit or abort reaches a durable terminal state, including delayed duplicate prepare requests that observe a committed or aborted record.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Add idempotent terminal transition handling for tier mutation intents and a compare-and-swap record update helper backed by config-object ETags. Cover commit and abort retries, conflicting terminal updates, stale ETag rejection, and exact-limit scan pagination.
Co-authored-by: heihutu <heihutu@gmail.com>
Add canonical record-object naming and crate-private save, load, and idempotent delete helpers for tier mutation intents. Cover malformed record keys, mismatched persisted mutation IDs, and an ECStore config-object round trip under test-util.
Co-authored-by: heihutu <heihutu@gmail.com>
Add an internal durable tier mutation intent model for the #1357 distributed fencing work. The new model validates schema, checksum, canonical targets, mutation-specific target identity shape, config ETags, expiry, and prepared-to-terminal state transitions without changing the production mutation path yet.
Co-authored-by: heihutu <heihutu@gmail.com>
Retry cleanup-pending transition transaction recovery after restart, keeping committed remote bodies when local metadata already references them and preserving records when remote cleanup fails.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): accept null version transition sources
Treat nil/null source version IDs as null-version transition sources when building transition transaction source identity, while preserving fail-closed validation for truly missing IDs on versioned sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): preserve multipart source versions
Propagate bucket versioning into CompleteMultipartUpload, assign a concrete version ID for versioned multipart completions, and classify unversioned FileInfo sources as null-version transition sources.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): normalize multipart completion versions
Treat nil staged version IDs as missing for versioned multipart completion and clear staged version IDs when completion publishes a null-version object under suspended or unversioned semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ilm): recover uploaded transition transactions
Persist transition transaction records through production transition uploads, use transaction-scoped remote object names, and start a conservative recovery loop for uploaded candidates.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ilm): reconcile committed transition records
Drop LocalCommitStarted transaction records only after object metadata confirms that the local transition commit already points at the same tier object and version.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(tiering): make rejected upload cleanup durable
* fix(tiering): close transition upload cancellation gap
* test(tiering): cover failed upload without candidate
* test(tiering): synchronize cancelled cleanup recovery
* test(tiering): stabilize cancelled cleanup recovery
Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Restore the workspace reqwest default feature stack for RustFS outbound HTTPS clients, while keeping per-crate extra APIs such as json, stream, and multipart explicit. Lazily initialize the notification runtime from admin target access when RUSTFS_NOTIFY_ENABLE=true is already effective, and add regression coverage for HTTPS webhook custom CA handling and target-list visibility.
Fixes#5052.
Co-authored-by: heihutu <heihutu@gmail.com>
Commit aec2ee9ec (#5005) enforced that RPC secrets cannot be derived
from the public default credentials. This broke all multi-node cluster
E2E tests that relied on the constructor's DEFAULT_ACCESS_KEY /
DEFAULT_SECRET_KEY defaults.
Move the non-default credential and RUSTFS_RPC_SECRET blanking into
the RustFSTestClusterEnvironment constructor so every cluster test
starts with a valid RPC secret derivation base. Remove the per-test
overrides in cluster_multidrive_pool_test that were added as a partial
fix in #5005.
Add a deterministic transition matrix case for a stale prepared metadata snapshot followed by a committed transition, duplicate transition, and late GET. The test proves the stale metadata generation is physically reclaimed, duplicate transition does not upload a second remote candidate, and the late GET reads the committed remote body byte-for-byte instead of reopening the released local source.
Refs rustfs/backlog#1355
Co-authored-by: heihutu <heihutu@gmail.com>
houseme