mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-21 20:06:37 +00:00
perf(ecstore): k-way heap merge for ListObjects, drop clone-to-parse (#4347)
* perf(ecstore): replace linear merge scan with k-way heap and drop clone-to-parse (backlog#874 backlog#875) merge_entry_channels advanced the k-way merge with a linear scan over all channel heads (O(entries x channels)) and allocated two fresh Strings per pairwise comparison via path::clean. Every step also cloned MetaCacheEntry values, including entry.clone().xl_meta() clone-to-parse calls. - Introduce MergeHead with a cached cleaned name (allocated only when the raw name is not already clean) and drive the merge with a BinaryHeap of boxed heads: O(log channels) per entry, allocation-free comparisons. - Move entries through the merge instead of cloning; the winner is sent without an intermediate copy. - Remove the dead merge_file_meta_versions block: it only ran for prefix-dir groups whose entries have empty metadata, so xl_meta() always failed; cross-drive version merging happens in the resolve path. - Keep legacy same-name semantics (dir groups collapse, objects shadow prefix dirs, later object candidate wins) and add regression tests for interleaved ordering, dir/object precedence, uncleaned-name grouping, and prefix-dir collapse. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): honor ascending versions_sort in ListObjects walk (#4348) * fix(ecstore): honor ascending versions_sort in walk and document ordering invariant (backlog#876) The walk loop carried a bare `//TODO: SORT` inside the `WalkVersionsSortOrder::Ascending` branch, so the requested ascending order was silently ignored and versions streamed newest-first (the raw FileMeta order). WalkOptions defaults to Ascending, so every default walker -- notably replication resync, which replays versions and needs oldest-first to preserve the version-stack order -- received the exact opposite of the contract. FileMeta maintains versions newest-first (sort_by_mod_time is descending) and into_file_info_versions preserves that order, so ascending emission is the exact reverse of file_info_versions output. Reverse in place when ascending is requested and add a regression test locking the newest-first invariant plus the reversal contract. Key-ordering audit result (no gap found): per-disk walkers emit sorted streams, merge_entry_channels performs an ordered k-way merge, and gather_results only filters by marker/limit, so ListObjects key order is guaranteed upstream and needs no post-sort. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): enable GET metadata early-stop by default (#4349) * perf(ecstore): enable GET metadata early-stop by default with env opt-out (backlog#872) The metadata early-stop fanout (read_all_fileinfo_early_stop) has been implemented and instrumented for a while but stayed behind an opt-in flag, so default GETs always waited for every disk to answer the metadata read even after quorum agreement was reached. Flip RUSTFS_GET_METADATA_EARLY_STOP_ENABLE to default-on. The gate stays conservative: should_allow_metadata_early_stop only admits metadata-only reads (read_data=false) without version_id, healing, or free-version requirements, everything else falls back to the full-wait fanout, and setting the env var to false restores the old behavior entirely. The version-aware gate (RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE) remains opt-in because versioned reads carry a higher stale-selection risk profile. Also replace the stale "optimize concurrency" TODO in get_object_fileinfo with a pointer to the early-stop implementation and add regression tests for the new default plus the explicit opt-out path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): lazily construct codec streaming multipart readers (#4350) * perf(ecstore): lazily construct codec streaming multipart part readers (backlog#871) get_object_decode_reader_with_fileinfo opened shard readers for every part of a multipart object before returning the streaming reader, so TTFB paid for parts x disks file opens up front and an early client disconnect wasted the setup work for every unread part. Replace the eager loop with LazyMultipartCodecStreamingReader: the first part is still built eagerly so the dominant fallback conditions (missing shards / read quorum) are detected before any byte is streamed and the whole request can fall back to the legacy duplex path exactly as before. Each subsequent part is built on demand -- when the previous part hits EOF -- via a spawned task handle owned by the reader; dropping the reader aborts an in-flight build so disconnects stop all further IO. If a later part hits a fallback condition mid-stream (a shard vanished after the request started), the reader surfaces an explicit read error with a pipeline-failure metric instead of silently degrading; the client's retry then detects the condition on the eager first-part setup and takes the legacy path cleanly. Adds unit tests for in-order streaming across lazy boundaries, deferred construction (no build when the client stops within part 1), and the mid-stream fallback error path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): prefetch next multipart part reader setup during decode (#4351) * perf(ecstore): prefetch next multipart part reader setup during decode (backlog#870) get_object_with_fileinfo processed multipart parts strictly serially: the next part's bitrot reader setup (file opens + read-quorum wait across all disks) only started after the current part finished decoding, so large multipart reads paid full setup latency between every part. Overlap the two stages with a depth-one pipeline: right after the current part's readers are obtained, the next part's setup is spawned (shared inputs behind Arc) and joined when the loop reaches that part. The shared setup_multipart_part_readers helper keeps stage-duration metrics semantics identical for both paths; a failed or stale prefetch falls back to the synchronous setup, and the PrefetchedReaderSetup guard aborts the in-flight task on error returns, early breaks, or caller drop so disconnects stop background disk IO. Gate: RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH (default on, env opt-out). Adds a three-part end-to-end read test covering the prefetch hit path and cross-part content ordering. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): move FileInfo through GET shuffle instead of cloning (#4352) perf(ecstore): move FileInfo entries through the GET shuffle instead of cloning (backlog#873) shuffle_disks_and_parts_metadata_by_index deep-cloned every valid FileInfo (parts, erasure info, metadata map) once per disk on each GET. Add an ownership-taking variant that runs the same by-index consistency check as a read-only first pass and then moves entries into their shuffled slots with mem::take, and switch get_object_with_fileinfo to it -- that call site already owned the parts metadata vector. Disk handles are Arc clones and stay cheap. Scope notes from the backlog#873 audit: - get_object_fileinfo's disks.clone() stays: DiskStore is Arc<Disk>, so the clone is per-slot refcounting and correctly avoids holding the RwLock read guard across the metadata fanout awaits. - get_object_decode_reader_with_fileinfo keeps the borrowing shuffle: its caller must retain files/disks for the legacy fallback path, so an owned variant would just shift the same clone upstream. - The metadata-cache hit path still clones parts_metadata; sharing the cached entry via Arc changes the read-path return types and is left as a follow-up. Equivalence tests cover both the by-index placement and the mod-time fallback against the borrowing variant. Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> * fix(ecstore): gate merge emission on cleaned key and clear clippy redundant_clone Address review + CI findings on the ListObjects/GET optimization PR: - merge_entry_channels gated emission on the raw entry name while the heap orders by the cleaned key, so entries whose cleaned order and raw byte order disagree (e.g. redundant slashes) could be dropped. Gate on the same cleaned sort key the heap uses; add a regression test (`a//c` after `a/b`). - Drop three redundant `.clone()` calls in test code flagged by clippy::redundant_clone (owned-shuffle equivalence tests and the walk ascending-versions contract test) that failed the CI clippy gate. - Document the known mid-stream fallback limitation of the opt-in multipart codec streaming reader (default off) and mark the in-place per-part legacy degradation as a follow-up. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): force full metadata fanout for object tagging writes (backlog#872) put_object_tags reads the object fileinfo with read_data=false and then writes the updated tags to the online-disk set that read returned. With metadata early-stop enabled by default, that read now returns as soon as read quorum is reached, so the online-disk set is only a read-quorum subset. Writing tags to that subset fails write quorum -> ErasureWriteQuorum -> S3 SlowDown, which is exactly the s3-tests tagging failures (PutObjectTagging/DeleteObjectTagging, reached max retries). Thread a caller-controlled `allow_early_stop` gate through read_all_fileinfo_observed/_inner and add get_object_fileinfo_gated; put_object_tags calls it with allow_early_stop=false so the metadata read does the full quorum fanout and returns the complete online-disk set as the write target. Pure-read callers (GET/HEAD/tag read) keep the early-stop fast path unchanged. Extract metadata_early_stop_permitted() as the single gate and add a unit test locking the invariant: caller opt-out (and observe=false, and data reads) never early-stop even with the env flags on. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
@@ -39,14 +39,15 @@ use futures::future::join_all;
|
||||
use rand::seq::SliceRandom;
|
||||
use rustfs_filemeta::{
|
||||
FileMeta, FileMetaShallowVersion, MetaCacheEntries, MetaCacheEntriesSorted, MetaCacheEntriesSortedResult, MetaCacheEntry,
|
||||
MetacacheReader, MetadataResolutionParams, is_io_eof, merge_file_meta_versions,
|
||||
MetacacheReader, MetadataResolutionParams, is_io_eof,
|
||||
};
|
||||
use rustfs_io_metrics::{
|
||||
LIST_OBJECTS_GATHER_OUTCOME_INPUT_CLOSED, LIST_OBJECTS_GATHER_OUTCOME_LIMIT_REACHED, LIST_OBJECTS_SOURCE_WALKER,
|
||||
ListObjectsGatherObservation, ListObjectsIndexPageObservation,
|
||||
};
|
||||
use rustfs_utils::path::{self, SLASH_SEPARATOR, base_dir_from_prefix};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::cmp::Reverse;
|
||||
use std::collections::{BinaryHeap, HashMap, HashSet};
|
||||
use std::future::Future;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{
|
||||
@@ -4376,7 +4377,7 @@ impl ECStore {
|
||||
continue;
|
||||
}
|
||||
|
||||
let fvs = match if opts.include_free_versions {
|
||||
let mut fvs = match if opts.include_free_versions {
|
||||
entry.file_info_versions_with_free_versions(&bucket_clone)
|
||||
} else {
|
||||
entry.file_info_versions(&bucket_clone)
|
||||
@@ -4395,8 +4396,13 @@ impl ECStore {
|
||||
}
|
||||
};
|
||||
|
||||
// `FileMeta` keeps versions newest-first (`sort_by_mod_time`
|
||||
// sorts descending), so ascending emission is the exact
|
||||
// reverse. Callers such as replication resync walk with the
|
||||
// default (ascending) order so that re-applied versions
|
||||
// preserve the original version-stack order.
|
||||
if opts.versions_sort == WalkVersionsSortOrder::Ascending {
|
||||
//TODO: SORT
|
||||
fvs.versions.reverse();
|
||||
}
|
||||
|
||||
for fi in fvs.versions.iter() {
|
||||
@@ -4639,28 +4645,64 @@ async fn gather_results(
|
||||
Ok(GatherResultsState::InputClosed)
|
||||
}
|
||||
|
||||
async fn select_from(
|
||||
rx: &CancellationToken,
|
||||
in_channels: &mut [Receiver<MetaCacheEntry>],
|
||||
/// Head entry of one input channel inside the k-way merge.
|
||||
///
|
||||
/// `cleaned` caches `path::clean(name)` only when it differs from the raw
|
||||
/// name, so heap comparisons never allocate (walker-produced names are
|
||||
/// already clean in the common case).
|
||||
struct MergeHead {
|
||||
entry: MetaCacheEntry,
|
||||
cleaned: Option<String>,
|
||||
idx: usize,
|
||||
top: &mut [Option<MetaCacheEntry>],
|
||||
n_done: &mut usize,
|
||||
) -> Result<bool> {
|
||||
let entry = tokio::select! {
|
||||
entry = in_channels[idx].recv() => entry,
|
||||
_ = rx.cancelled() => return Ok(false),
|
||||
};
|
||||
}
|
||||
|
||||
match entry {
|
||||
Some(entry) => {
|
||||
top[idx] = Some(entry);
|
||||
}
|
||||
None => {
|
||||
top[idx] = None;
|
||||
*n_done += 1;
|
||||
}
|
||||
impl MergeHead {
|
||||
fn new(entry: MetaCacheEntry, idx: usize) -> Self {
|
||||
let cleaned = path::clean(&entry.name);
|
||||
let cleaned = if cleaned == entry.name { None } else { Some(cleaned) };
|
||||
Self { entry, cleaned, idx }
|
||||
}
|
||||
|
||||
fn sort_key(&self) -> &str {
|
||||
self.cleaned.as_deref().unwrap_or(&self.entry.name)
|
||||
}
|
||||
}
|
||||
|
||||
impl PartialEq for MergeHead {
|
||||
fn eq(&self, other: &Self) -> bool {
|
||||
self.cmp(other) == std::cmp::Ordering::Equal
|
||||
}
|
||||
}
|
||||
|
||||
impl Eq for MergeHead {}
|
||||
|
||||
impl PartialOrd for MergeHead {
|
||||
fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
|
||||
Some(self.cmp(other))
|
||||
}
|
||||
}
|
||||
|
||||
impl Ord for MergeHead {
|
||||
fn cmp(&self, other: &Self) -> std::cmp::Ordering {
|
||||
// Ascending channel index preserves the legacy scan order for equal keys.
|
||||
self.sort_key().cmp(other.sort_key()).then_with(|| self.idx.cmp(&other.idx))
|
||||
}
|
||||
}
|
||||
|
||||
enum MergePoll {
|
||||
Entry(Box<MergeHead>),
|
||||
Closed,
|
||||
Cancelled,
|
||||
}
|
||||
|
||||
async fn poll_merge_head(rx: &CancellationToken, in_channels: &mut [Receiver<MetaCacheEntry>], idx: usize) -> MergePoll {
|
||||
tokio::select! {
|
||||
entry = in_channels[idx].recv() => match entry {
|
||||
Some(entry) => MergePoll::Entry(Box::new(MergeHead::new(entry, idx))),
|
||||
None => MergePoll::Closed,
|
||||
},
|
||||
_ = rx.cancelled() => MergePoll::Cancelled,
|
||||
}
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
async fn send_or_cancel(rx: &CancellationToken, out_channel: &Sender<MetaCacheEntry>, entry: MetaCacheEntry) -> Result<bool> {
|
||||
@@ -4707,147 +4749,100 @@ async fn merge_entry_channels(
|
||||
}
|
||||
}
|
||||
|
||||
let mut top: Vec<Option<MetaCacheEntry>> = vec![None; in_channels.len()];
|
||||
let mut n_done = 0;
|
||||
|
||||
let in_channels_len = in_channels.len();
|
||||
|
||||
for idx in 0..in_channels_len {
|
||||
if !select_from(&rx, &mut in_channels, idx, &mut top, &mut n_done).await? {
|
||||
return Ok(());
|
||||
// K-way merge across per-channel sorted streams via a min-heap keyed by
|
||||
// the path-cleaned entry name: advancing the merge costs O(log channels)
|
||||
// per entry and comparisons are allocation-free (see `MergeHead`).
|
||||
//
|
||||
// Note on same-name resolution: prefix-dir groups collapse into a single
|
||||
// emission and objects shadow same-named prefix dirs. The legacy
|
||||
// `merge_file_meta_versions` block that used to live here was dead code:
|
||||
// it only ran for prefix-dir groups, whose entries have empty metadata,
|
||||
// so `xl_meta()` always failed. Cross-drive version merging happens in
|
||||
// the metadata resolve path (`MetaCacheEntries::resolve`), not here.
|
||||
let mut heap: BinaryHeap<Reverse<Box<MergeHead>>> = BinaryHeap::with_capacity(in_channels.len());
|
||||
for idx in 0..in_channels.len() {
|
||||
match poll_merge_head(&rx, &mut in_channels, idx).await {
|
||||
MergePoll::Entry(head) => heap.push(Reverse(head)),
|
||||
MergePoll::Closed => {}
|
||||
MergePoll::Cancelled => {
|
||||
info!("merge_entry_channels rx.recv() cancel");
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut last = String::new();
|
||||
let mut to_merge: Vec<usize> = Vec::new();
|
||||
loop {
|
||||
if n_done == in_channels.len() {
|
||||
let mut group: Vec<Box<MergeHead>> = Vec::new();
|
||||
let mut refill: Vec<usize> = Vec::with_capacity(in_channels.len());
|
||||
|
||||
while let Some(Reverse(first)) = heap.pop() {
|
||||
// Collect every channel head that resolves to the same cleaned key so
|
||||
// the duplicate rules below see the whole same-name group at once.
|
||||
group.clear();
|
||||
refill.clear();
|
||||
refill.push(first.idx);
|
||||
while heap.peek().is_some_and(|Reverse(next)| next.sort_key() == first.sort_key()) {
|
||||
let Some(Reverse(next)) = heap.pop() else { break };
|
||||
refill.push(next.idx);
|
||||
group.push(next);
|
||||
}
|
||||
|
||||
// Legacy same-name resolution rules (heads arrive in ascending
|
||||
// channel order):
|
||||
// - prefix dir vs prefix dir with the same suffix shape: the first
|
||||
// head wins and the rest collapse;
|
||||
// - prefix dir vs object: the object wins;
|
||||
// - object vs object: the later channel wins;
|
||||
// - both dirs with different suffix shape: the smaller raw name wins.
|
||||
let mut winner = first;
|
||||
for other in group.drain(..) {
|
||||
let dir_matches = winner.entry.is_dir() && other.entry.is_dir();
|
||||
let suffix_matches = winner.entry.name.ends_with(SLASH_SEPARATOR) == other.entry.name.ends_with(SLASH_SEPARATOR);
|
||||
|
||||
if dir_matches && suffix_matches {
|
||||
continue;
|
||||
}
|
||||
|
||||
if !dir_matches {
|
||||
if other.entry.is_dir() {
|
||||
continue;
|
||||
}
|
||||
winner = other;
|
||||
continue;
|
||||
}
|
||||
|
||||
if winner.entry.name > other.entry.name {
|
||||
winner = other;
|
||||
}
|
||||
}
|
||||
|
||||
// Gate emission on the same cleaned key the heap orders by, not the raw
|
||||
// name. The heap pops in non-decreasing cleaned order, so comparing the
|
||||
// raw name here could drop a legitimate entry whose cleaned order and
|
||||
// raw order disagree (e.g. redundant slashes or `./` segments).
|
||||
let emit = winner.sort_key() > last.as_str();
|
||||
if emit {
|
||||
last.clear();
|
||||
last.push_str(winner.sort_key());
|
||||
}
|
||||
let MergeHead { entry, .. } = *winner;
|
||||
if emit && !send_or_cancel(&rx, &out_channel, entry).await? {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let mut best = top[0].clone();
|
||||
let mut best_idx = 0;
|
||||
to_merge.clear();
|
||||
|
||||
// Note: `select_from` mutates `top[idx]` during the inner loop, but this is safe
|
||||
// because each borrow from `top[other_idx]` is only used before any later
|
||||
// `select_from` call that can mutate that slot.
|
||||
|
||||
for other_idx in 1..top.len() {
|
||||
if let Some(other_entry) = &top[other_idx] {
|
||||
if let Some(best_entry) = &best {
|
||||
if path::clean(&best_entry.name) == path::clean(&other_entry.name) {
|
||||
let dir_matches = best_entry.is_dir() && other_entry.is_dir();
|
||||
let suffix_matches =
|
||||
best_entry.name.ends_with(SLASH_SEPARATOR) == other_entry.name.ends_with(SLASH_SEPARATOR);
|
||||
|
||||
if dir_matches && suffix_matches {
|
||||
to_merge.push(other_idx);
|
||||
continue;
|
||||
}
|
||||
|
||||
if !dir_matches {
|
||||
// dir and object has the save name
|
||||
if other_entry.is_dir() {
|
||||
if !select_from(&rx, &mut in_channels, other_idx, &mut top, &mut n_done).await? {
|
||||
return Ok(());
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
to_merge.clear();
|
||||
|
||||
best = Some(other_entry.clone());
|
||||
best_idx = other_idx;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
if best_entry.name > other_entry.name {
|
||||
to_merge.clear();
|
||||
best = Some(other_entry.clone());
|
||||
best_idx = other_idx;
|
||||
}
|
||||
} else {
|
||||
best = Some(other_entry.clone());
|
||||
best_idx = other_idx;
|
||||
for &idx in &refill {
|
||||
match poll_merge_head(&rx, &mut in_channels, idx).await {
|
||||
MergePoll::Entry(head) => heap.push(Reverse(head)),
|
||||
MergePoll::Closed => {}
|
||||
MergePoll::Cancelled => {
|
||||
info!("merge_entry_channels rx.recv() cancel");
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if !to_merge.is_empty() {
|
||||
if let Some(entry) = &best {
|
||||
let mut versions = Vec::with_capacity(to_merge.len() + 1);
|
||||
|
||||
let mut has_xl = { entry.clone().xl_meta().ok() };
|
||||
|
||||
if let Some(x) = &has_xl {
|
||||
versions.push(x.versions.clone());
|
||||
}
|
||||
|
||||
for &idx in to_merge.iter() {
|
||||
let has_entry = top[idx].clone();
|
||||
|
||||
if let Some(entry) = has_entry {
|
||||
let xl2 = match entry.clone().xl_meta() {
|
||||
Ok(res) => res,
|
||||
Err(_) => {
|
||||
if !select_from(&rx, &mut in_channels, idx, &mut top, &mut n_done).await? {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
versions.push(xl2.versions.clone());
|
||||
|
||||
if has_xl.is_none() {
|
||||
if !select_from(&rx, &mut in_channels, best_idx, &mut top, &mut n_done).await? {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
best_idx = idx;
|
||||
best = Some(entry.clone());
|
||||
has_xl = Some(xl2);
|
||||
} else {
|
||||
if !select_from(&rx, &mut in_channels, best_idx, &mut top, &mut n_done).await? {
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(xl) = has_xl.as_mut()
|
||||
&& !versions.is_empty()
|
||||
{
|
||||
xl.versions = merge_file_meta_versions(read_quorum, true, 0, &versions);
|
||||
|
||||
if let Ok(meta) = xl.marshal_msg()
|
||||
&& let Some(b) = best.as_mut()
|
||||
{
|
||||
b.metadata = meta;
|
||||
b.cached = Some(xl.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
to_merge.clear();
|
||||
}
|
||||
|
||||
if let Some(best_entry) = &best
|
||||
&& best_entry.name > last
|
||||
{
|
||||
if !send_or_cancel(&rx, &out_channel, best_entry.clone()).await? {
|
||||
return Ok(());
|
||||
}
|
||||
last = best_entry.name.clone();
|
||||
}
|
||||
|
||||
if !select_from(&rx, &mut in_channels, best_idx, &mut top, &mut n_done).await? {
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
impl Sets {
|
||||
@@ -9230,6 +9225,178 @@ mod test {
|
||||
assert_eq!(results, vec!["obj-a", "obj-b"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn walk_ascending_versions_contract_reverses_newest_first_metadata() {
|
||||
// Documents the invariant the walk `versions_sort` handling relies on:
|
||||
// `FileMeta` keeps versions newest-first, so ascending emission is the
|
||||
// exact reverse of `file_info_versions` output.
|
||||
let older = time::OffsetDateTime::from_unix_timestamp(1_705_312_300).expect("valid timestamp");
|
||||
let newer = time::OffsetDateTime::from_unix_timestamp(1_705_312_500).expect("valid timestamp");
|
||||
let entry =
|
||||
test_object_meta_entry_with_erasure_versions("obj-a", &[(older, "etag-old", 4, 2), (newer, "etag-new", 4, 2)]);
|
||||
|
||||
let fvs = entry.file_info_versions("bucket").expect("versions should parse");
|
||||
assert_eq!(fvs.versions.len(), 2, "both versions should be visible");
|
||||
assert!(
|
||||
fvs.versions[0].mod_time >= fvs.versions[1].mod_time,
|
||||
"file_info_versions must yield newest-first"
|
||||
);
|
||||
assert!(fvs.versions[0].is_latest, "the first (newest) version carries is_latest");
|
||||
|
||||
let mut ascending = fvs.versions;
|
||||
ascending.reverse();
|
||||
assert!(
|
||||
ascending[0].mod_time <= ascending[1].mod_time,
|
||||
"reversing newest-first output must produce ascending mod_time order"
|
||||
);
|
||||
assert!(
|
||||
ascending.last().expect("versions present").is_latest,
|
||||
"ascending emission ends with the latest version"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn merge_entry_channels_merges_interleaved_channels_in_global_order() {
|
||||
let mut txs = Vec::new();
|
||||
let mut rxs = Vec::new();
|
||||
for _ in 0..4 {
|
||||
let (tx, rx) = mpsc::channel(16);
|
||||
txs.push(tx);
|
||||
rxs.push(rx);
|
||||
}
|
||||
|
||||
// Round-robin keys so every channel stays sorted while the global
|
||||
// stream interleaves across all four channels.
|
||||
let names: Vec<String> = (0..24).map(|i| format!("obj-{i:02}")).collect();
|
||||
for (i, name) in names.iter().enumerate() {
|
||||
txs[i % 4].send(test_meta_entry(name)).await.unwrap();
|
||||
}
|
||||
// The same trailing key on three channels must be emitted once.
|
||||
for tx in txs.iter().take(3) {
|
||||
tx.send(test_meta_entry("obj-99")).await.unwrap();
|
||||
}
|
||||
drop(txs);
|
||||
|
||||
let (out_tx, mut out_rx) = mpsc::channel(64);
|
||||
let cancel = CancellationToken::new();
|
||||
let handle = tokio::spawn(merge_entry_channels(cancel, rxs, out_tx, 1));
|
||||
|
||||
let mut results = Vec::new();
|
||||
while let Some(entry) = out_rx.recv().await {
|
||||
results.push(entry.name.clone());
|
||||
}
|
||||
handle.await.unwrap().unwrap();
|
||||
|
||||
let mut expected = names;
|
||||
expected.push("obj-99".to_owned());
|
||||
assert_eq!(results, expected);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn merge_entry_channels_emits_when_cleaned_order_differs_from_raw_order() {
|
||||
// Regression: `a//c` cleans to `a/c` and `a/b` is already clean. In
|
||||
// cleaned order `a/b` < `a/c`, but raw byte order is the opposite
|
||||
// (`a//c` < `a/b` because '/' < 'b'). The heap pops in cleaned order,
|
||||
// so gating emission on the raw name would drop `a//c` after `a/b` is
|
||||
// emitted. Both distinct keys must survive.
|
||||
let (tx_a, rx_a) = mpsc::channel(4);
|
||||
let (tx_b, rx_b) = mpsc::channel(4);
|
||||
let (out_tx, mut out_rx) = mpsc::channel(8);
|
||||
|
||||
tx_a.send(test_meta_entry("a/b")).await.unwrap();
|
||||
drop(tx_a);
|
||||
tx_b.send(test_meta_entry("a//c")).await.unwrap();
|
||||
drop(tx_b);
|
||||
|
||||
let cancel = CancellationToken::new();
|
||||
let handle = tokio::spawn(merge_entry_channels(cancel, vec![rx_a, rx_b], out_tx, 1));
|
||||
|
||||
let mut results = Vec::new();
|
||||
while let Some(entry) = out_rx.recv().await {
|
||||
results.push(entry.name.clone());
|
||||
}
|
||||
handle.await.unwrap().unwrap();
|
||||
|
||||
assert_eq!(
|
||||
results,
|
||||
vec!["a/b", "a//c"],
|
||||
"distinct cleaned keys must both be emitted in cleaned order"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn merge_entry_channels_object_wins_over_same_named_prefix_dir() {
|
||||
let (tx_a, rx_a) = mpsc::channel(4);
|
||||
let (tx_b, rx_b) = mpsc::channel(4);
|
||||
let (out_tx, mut out_rx) = mpsc::channel(8);
|
||||
|
||||
// `path::clean("a/") == "a"`, so the prefix dir and the object group
|
||||
// under the same key and the object must win.
|
||||
tx_a.send(test_dir_meta_entry("a/")).await.unwrap();
|
||||
drop(tx_a);
|
||||
tx_b.send(test_object_meta_entry("a")).await.unwrap();
|
||||
drop(tx_b);
|
||||
|
||||
let cancel = CancellationToken::new();
|
||||
let handle = tokio::spawn(merge_entry_channels(cancel, vec![rx_a, rx_b], out_tx, 1));
|
||||
|
||||
let mut results = Vec::new();
|
||||
while let Some(entry) = out_rx.recv().await {
|
||||
assert!(entry.is_object(), "the object entry must shadow the same-named prefix dir");
|
||||
results.push(entry.name.clone());
|
||||
}
|
||||
handle.await.unwrap().unwrap();
|
||||
|
||||
assert_eq!(results, vec!["a"]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn merge_entry_channels_collapses_equivalent_prefix_dirs() {
|
||||
let (tx_a, rx_a) = mpsc::channel(4);
|
||||
let (tx_b, rx_b) = mpsc::channel(4);
|
||||
let (out_tx, mut out_rx) = mpsc::channel(8);
|
||||
|
||||
tx_a.send(test_dir_meta_entry("x/")).await.unwrap();
|
||||
drop(tx_a);
|
||||
tx_b.send(test_dir_meta_entry("x/")).await.unwrap();
|
||||
drop(tx_b);
|
||||
|
||||
let cancel = CancellationToken::new();
|
||||
let handle = tokio::spawn(merge_entry_channels(cancel, vec![rx_a, rx_b], out_tx, 1));
|
||||
|
||||
let mut results = Vec::new();
|
||||
while let Some(entry) = out_rx.recv().await {
|
||||
results.push(entry.name.clone());
|
||||
}
|
||||
handle.await.unwrap().unwrap();
|
||||
|
||||
assert_eq!(results, vec!["x/"]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn merge_entry_channels_normalizes_uncleaned_names_before_grouping() {
|
||||
let (tx_a, rx_a) = mpsc::channel(4);
|
||||
let (tx_b, rx_b) = mpsc::channel(4);
|
||||
let (out_tx, mut out_rx) = mpsc::channel(8);
|
||||
|
||||
// Both names clean to "a/b"; they must merge into a single emission.
|
||||
tx_a.send(test_meta_entry("a//b")).await.unwrap();
|
||||
drop(tx_a);
|
||||
tx_b.send(test_meta_entry("a/b")).await.unwrap();
|
||||
drop(tx_b);
|
||||
|
||||
let cancel = CancellationToken::new();
|
||||
let handle = tokio::spawn(merge_entry_channels(cancel, vec![rx_a, rx_b], out_tx, 1));
|
||||
|
||||
let mut results = Vec::new();
|
||||
while let Some(entry) = out_rx.recv().await {
|
||||
results.push(entry.name.clone());
|
||||
}
|
||||
handle.await.unwrap().unwrap();
|
||||
|
||||
assert_eq!(results, vec!["a/b"]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn merge_entry_channels_respects_cancellation() {
|
||||
let (tx, rx) = mpsc::channel::<MetaCacheEntry>(4);
|
||||
|
||||
Reference in New Issue
Block a user