mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-16 01:48:21 +00:00
Merge remote-tracking branch 'origin/main' into cxymds/fix-rebalance-multipart-retry
This commit is contained in:
马登山
+64
-20
@@ -1,6 +1,6 @@
|
||||
# ARCHITECTURE.md
|
||||
|
||||
> Last updated: 2026-07-02 · Revision: 2
|
||||
> Last updated: 2026-08-12 · Revision: 3
|
||||
>
|
||||
> This document describes the high-level architecture of RustFS.
|
||||
> If you want to familiarize yourself with the code base, you are in the right place!
|
||||
@@ -119,19 +119,44 @@ module split is tracked under `docs/architecture/`.
|
||||
|
||||
3. **Each type has exactly one definition.** Types shared across crates must be defined
|
||||
in one crate and re-exported or imported by others.
|
||||
- ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
|
||||
`BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
|
||||
- ⚠️ VIOLATED: `ReplicationStats` names three unrelated types
|
||||
(`crates/data-usage/src/data_usage.rs`,
|
||||
`crates/obs/src/metrics/collectors/replication.rs`,
|
||||
`crates/ecstore/src/bucket/replication/replication_state.rs`) — a naming
|
||||
collision, not copies; renaming is tracked in rustfs/backlog#1847.
|
||||
- `LastMinuteLatency` has two deliberately different implementations: the
|
||||
per-second bucketed accumulator in `crates/common/src/last_minute.rs` and
|
||||
the in-memory endpoint-health sample tracker in
|
||||
`crates/ecstore/src/bucket/bucket_target_sys.rs` (its doc comment explains
|
||||
why it stays local).
|
||||
- ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
|
||||
definition (`crates/io-core/src/backpressure.rs`,
|
||||
`crates/data-usage/src/data_usage.rs`). A zero-consumer
|
||||
`BackpressureSettings` copy lingers in `crates/io-metrics/src/config.rs`;
|
||||
its removal is tracked in rustfs/backlog#1833.
|
||||
|
||||
4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
|
||||
storage-level abstractions (objects, buckets, disks, pools).
|
||||
- ⚠️ VIOLATED: 58 files under `crates/ecstore/src` reference `s3s`
|
||||
(`rg -l 's3s' crates/ecstore/src | wc -l`), `crates/ecstore/src/client/`
|
||||
is a ~9.4K-line embedded S3 HTTP client, and `crates/ecstore/Cargo.toml`
|
||||
depends on `s3s`, `http`, `hyper`/`hyper-util`/`hyper-rustls`, and
|
||||
`reqwest`. Target state: the engine's need to act as an S3 client
|
||||
(tiering, replication targets) is served by an extracted client crate,
|
||||
and ecstore holds no wire or DTO types.
|
||||
|
||||
5. **The `rustfs` binary crate is the only place that wires everything together.**
|
||||
Individual crates should be testable in isolation.
|
||||
|
||||
6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
|
||||
not bare `Error`).
|
||||
- ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
|
||||
`heal` use `anyhow` in library code.
|
||||
- ✅ RESOLVED (strategy): `snafu` is gone from source
|
||||
(`rg -l snafu crates/ rustfs/` is empty) and library code no longer uses
|
||||
`anyhow` (remaining hits are test code and the `e2e_test` crate; `heal`
|
||||
uses `thiserror`).
|
||||
- ⚠️ VIOLATED (naming): 6 crates still export a bare `pub enum Error`:
|
||||
`crypto`, `filemeta`, `heal`, `iam`, `policy`, and `replication`
|
||||
(`src/resync.rs`) — all `thiserror`-derived.
|
||||
|
||||
## Known Structural Issues
|
||||
|
||||
@@ -140,13 +165,25 @@ module split is tracked under `docs/architecture/`.
|
||||
|
||||
### Critical
|
||||
|
||||
- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
|
||||
but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
|
||||
types instead of importing them.
|
||||
- **scanner/data-usage duplicate `.usage-cache.bin` serialization types.** The
|
||||
original finding ("common/scanner code duplication, ~3K lines") is resolved:
|
||||
`scanner` imports the shared data-usage types from `rustfs-data-usage` (see
|
||||
the `pub use rustfs_data_usage::…` re-exports at the top of
|
||||
`crates/scanner/src/data_usage_define.rs`). What remains: `scanner` and
|
||||
`data-usage` each hold their own serialization types for the scanner cache
|
||||
file (`DataUsageCacheInfo`/`DataUsageEntryInfo` in
|
||||
`crates/scanner/src/data_usage_define.rs` vs
|
||||
`DataUsageCacheInfo`/`DataUsageEntry` in
|
||||
`crates/data-usage/src/data_usage.rs`); convergence is tracked in
|
||||
rustfs/backlog#1828.
|
||||
|
||||
- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
|
||||
bucket management, erasure coding, replication, lifecycle, RPC, and configuration
|
||||
— all in one crate. It should be decomposed along its existing subdirectories.
|
||||
- **ecstore is a monolith (265 files, ~288K lines — roughly half is inline
|
||||
`#[cfg(test)]` code).** Measured with
|
||||
`find crates/ecstore/src -name '*.rs' | xargs wc -l`. It contains disk
|
||||
management, bucket management, erasure coding, replication, lifecycle, RPC,
|
||||
and configuration — all in one crate. It should be decomposed along its
|
||||
existing subdirectories; the split plan lives in
|
||||
[docs/architecture/ecstore-module-split-plan.md](docs/architecture/ecstore-module-split-plan.md).
|
||||
|
||||
### High
|
||||
|
||||
@@ -154,19 +191,26 @@ module split is tracked under `docs/architecture/`.
|
||||
`common → filemeta/madmin` edges must stay removed so leaf/helper crates do
|
||||
not regain upward dependencies.
|
||||
|
||||
- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
|
||||
concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
|
||||
explicit projections into the concurrency/io-core policy shapes, and workload
|
||||
- **Three-layer backpressure/deadlock policy bridging** across io-core,
|
||||
concurrency, and `rustfs/src/storage`. The config types are no longer
|
||||
duplicated (`BackpressureConfig` and `DeadlockDetectorConfig` are each
|
||||
defined once, in io-core). Storage policies expose and consume explicit
|
||||
projections into the concurrency/io-core policy shapes, and workload
|
||||
admission snapshots are composed through provider registries; later work
|
||||
should use those bridges before deleting compatibility wrappers.
|
||||
|
||||
### Medium
|
||||
|
||||
- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
|
||||
mixed naming (bare `Error` vs descriptive names).
|
||||
- **Bare `Error` naming.** Error-handling strategy has converged on `thiserror`
|
||||
(no `snafu`, no `anyhow` in library code); the remaining inconsistency is the
|
||||
bare `pub enum Error` naming in the 6 crates listed under Invariant 6.
|
||||
|
||||
- **Ambiguous common vs utils boundary.** Both described as "utilities and data
|
||||
structures." Need clear ownership rules.
|
||||
- **`common` is mostly parked domain code, not shared utilities.** Of its
|
||||
6,724 lines, ~83% is scanner/heal domain code stranded there to break
|
||||
dependency cycles (`metrics.rs`, ~4,810 lines of scanner-domain metrics;
|
||||
`heal_channel.rs`, ~776 lines of heal-domain channel types). The
|
||||
"common vs utils" naming ambiguity is secondary to moving that code to its
|
||||
domain owners.
|
||||
|
||||
## Cross-Cutting Concerns
|
||||
|
||||
@@ -232,7 +276,7 @@ The binary (`main.rs`) boots in this order:
|
||||
|
||||
```
|
||||
┌─────────┐
|
||||
│ rustfs │ (binary + lib, 75K lines)
|
||||
│ rustfs │ (binary + lib)
|
||||
│ main │
|
||||
└────┬────┘
|
||||
│
|
||||
@@ -255,7 +299,7 @@ The binary (`main.rs`) boots in this order:
|
||||
│ │ │
|
||||
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
|
||||
│ ecstore │ │ rio │ │ io-core │
|
||||
│ (87K,core) │ │ (readers) │ │ (zero-copy) │
|
||||
│ (core) │ │ (readers) │ │ (zero-copy) │
|
||||
└─────┬──────┘ └─────────────┘ └─────────────┘
|
||||
│
|
||||
┌─────┬──┼──┬─────┬──────┐
|
||||
|
||||
Generated
+74
-35
@@ -104,6 +104,12 @@ dependencies = [
|
||||
"memchr",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "aliasable"
|
||||
version = "0.1.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "250f629c0161ad8107cf89319e990051fae62832fd343083bea452d93e2205fd"
|
||||
|
||||
[[package]]
|
||||
name = "aligned-vec"
|
||||
version = "0.6.4"
|
||||
@@ -266,24 +272,24 @@ checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
|
||||
|
||||
[[package]]
|
||||
name = "apache-avro"
|
||||
version = "0.21.0"
|
||||
version = "0.22.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "36fa98bc79671c7981272d91a8753a928ff6a1cd8e4f20a44c45bd5d313840bf"
|
||||
checksum = "312c1ea69e5fe9966e0029fb95aca8790100b85aff4f0d3b00a9337c74069a9c"
|
||||
dependencies = [
|
||||
"bigdecimal",
|
||||
"bon",
|
||||
"digest 0.10.7",
|
||||
"digest 0.11.3",
|
||||
"log",
|
||||
"miniz_oxide",
|
||||
"miniz_oxide 0.9.1",
|
||||
"num-bigint 0.4.8",
|
||||
"ouroboros",
|
||||
"quad-rand",
|
||||
"rand 0.9.5",
|
||||
"rand 0.10.2",
|
||||
"regex-lite",
|
||||
"serde",
|
||||
"serde_bytes",
|
||||
"serde_json",
|
||||
"strum 0.27.2",
|
||||
"strum_macros 0.27.2",
|
||||
"strum",
|
||||
"thiserror 2.0.20",
|
||||
"uuid",
|
||||
]
|
||||
@@ -1458,7 +1464,7 @@ dependencies = [
|
||||
"addr2line",
|
||||
"cfg-if",
|
||||
"libc",
|
||||
"miniz_oxide",
|
||||
"miniz_oxide 0.8.9",
|
||||
"object 0.37.3",
|
||||
"rustc-demangle",
|
||||
"windows-link",
|
||||
@@ -2003,7 +2009,7 @@ version = "4.6.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061"
|
||||
dependencies = [
|
||||
"heck",
|
||||
"heck 0.5.0",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.3",
|
||||
@@ -4184,7 +4190,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
|
||||
dependencies = [
|
||||
"crc32fast",
|
||||
"miniz_oxide",
|
||||
"miniz_oxide 0.8.9",
|
||||
"zlib-rs",
|
||||
]
|
||||
|
||||
@@ -4854,6 +4860,12 @@ dependencies = [
|
||||
"stable_deref_trait",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "heck"
|
||||
version = "0.4.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8"
|
||||
|
||||
[[package]]
|
||||
name = "heck"
|
||||
version = "0.5.0"
|
||||
@@ -6391,6 +6403,15 @@ dependencies = [
|
||||
"simd-adler32",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "miniz_oxide"
|
||||
version = "0.9.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
|
||||
dependencies = [
|
||||
"adler2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "minlz"
|
||||
version = "1.2.3"
|
||||
@@ -6485,7 +6506,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a4db8a44120571277accfaa3f3d91e7d3989d601d817c2fc01a9391b86135666"
|
||||
dependencies = [
|
||||
"darling 0.23.0",
|
||||
"heck",
|
||||
"heck 0.5.0",
|
||||
"manyhow",
|
||||
"num-bigint 0.4.8",
|
||||
"proc-macro-crate",
|
||||
@@ -7190,6 +7211,30 @@ dependencies = [
|
||||
"num-traits",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ouroboros"
|
||||
version = "0.18.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1e0f050db9c44b97a94723127e6be766ac5c340c48f2c4bb3ffa11713744be59"
|
||||
dependencies = [
|
||||
"aliasable",
|
||||
"ouroboros_macro",
|
||||
"static_assertions",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ouroboros_macro"
|
||||
version = "0.18.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3c7028bdd3d43083f6d8d4d5187680d0d3560d54df4cc9d752005268b41e64d0"
|
||||
dependencies = [
|
||||
"heck 0.4.1",
|
||||
"proc-macro2",
|
||||
"proc-macro2-diagnostics",
|
||||
"quote",
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "outref"
|
||||
version = "0.5.2"
|
||||
@@ -7925,6 +7970,19 @@ dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2-diagnostics"
|
||||
version = "0.10.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "af066a9c399a26e020ada66a034357a868728e72cd426f3adcd35f80d88d88c8"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.119",
|
||||
"version_check",
|
||||
"yansi",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "prometheus"
|
||||
version = "0.14.0"
|
||||
@@ -7984,7 +8042,7 @@ version = "0.13.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
|
||||
dependencies = [
|
||||
"heck",
|
||||
"heck 0.5.0",
|
||||
"itertools 0.14.0",
|
||||
"log",
|
||||
"multimap",
|
||||
@@ -8004,7 +8062,7 @@ version = "0.14.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
|
||||
dependencies = [
|
||||
"heck",
|
||||
"heck 0.5.0",
|
||||
"itertools 0.14.0",
|
||||
"log",
|
||||
"multimap",
|
||||
@@ -9287,7 +9345,6 @@ dependencies = [
|
||||
name = "rustfs-data-usage"
|
||||
version = "1.0.0-rc.1"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"hotpath",
|
||||
"rmp-serde",
|
||||
"rustfs-filemeta",
|
||||
@@ -9924,7 +9981,7 @@ dependencies = [
|
||||
"rustfs-crypto",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"strum 0.28.0",
|
||||
"strum",
|
||||
"temp-env",
|
||||
"test-case",
|
||||
"thiserror 2.0.20",
|
||||
@@ -11513,31 +11570,13 @@ version = "0.11.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
|
||||
|
||||
[[package]]
|
||||
name = "strum"
|
||||
version = "0.27.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf"
|
||||
|
||||
[[package]]
|
||||
name = "strum"
|
||||
version = "0.28.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9628de9b8791db39ceda2b119bbe13134770b56c138ec1d3af810d045c04f9bd"
|
||||
dependencies = [
|
||||
"strum_macros 0.28.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "strum_macros"
|
||||
version = "0.27.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7"
|
||||
dependencies = [
|
||||
"heck",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.119",
|
||||
"strum_macros",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -11546,7 +11585,7 @@ version = "0.28.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664"
|
||||
dependencies = [
|
||||
"heck",
|
||||
"heck 0.5.0",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.119",
|
||||
|
||||
+1
-1
@@ -171,7 +171,7 @@ tower = { version = "0.5.3" }
|
||||
tower-http = { version = "0.7.0" }
|
||||
|
||||
# Serialization and Data Formats
|
||||
apache-avro = "0.21.0"
|
||||
apache-avro = "0.22.0"
|
||||
bytes = { version = "1.12.1" }
|
||||
bytesize = "2.7.0"
|
||||
byteorder = "1.5.0"
|
||||
|
||||
@@ -37,7 +37,6 @@ hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
|
||||
hotpath.workspace = true
|
||||
serde = { workspace = true, features = ["derive"] }
|
||||
rmp-serde = { workspace = true }
|
||||
async-trait = { workspace = true }
|
||||
rustfs-filemeta = { workspace = true }
|
||||
|
||||
[lib]
|
||||
|
||||
@@ -846,8 +846,15 @@ impl DataUsageEntry {
|
||||
}
|
||||
}
|
||||
|
||||
/// Data usage cache info
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
/// Read-only projection of the scanner's `.usage-cache.bin` info block.
|
||||
///
|
||||
/// The canonical wire format is written by the hand-written map-encoded
|
||||
/// `Serialize` on the scanner-side `DataUsageCacheInfo`
|
||||
/// (`crates/scanner/src/data_usage_define.rs`), which carries 16 fields.
|
||||
/// This type decodes only the shared subset and is deliberately not
|
||||
/// `Serialize`: a derived (array) encoding of this 6-field subset would
|
||||
/// corrupt the cache for scanner readers, so no write path may exist here.
|
||||
#[derive(Clone, Debug, Default, Deserialize)]
|
||||
pub struct DataUsageCacheInfo {
|
||||
pub name: String,
|
||||
pub next_cycle: u64,
|
||||
@@ -863,8 +870,12 @@ pub struct DataUsageCacheInfo {
|
||||
pub snapshot_complete: bool,
|
||||
}
|
||||
|
||||
/// Data usage cache
|
||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
||||
/// Read-only projection of a scanner-written `.usage-cache.bin` file.
|
||||
///
|
||||
/// The scanner-side `DataUsageCache` (`crates/scanner/src/data_usage_define.rs`)
|
||||
/// owns the persisted format; this type only decodes it (see
|
||||
/// [`DataUsageCacheInfo`]) and must never grow a serialization path.
|
||||
#[derive(Clone, Debug, Default, Deserialize)]
|
||||
pub struct DataUsageCache {
|
||||
pub info: DataUsageCacheInfo,
|
||||
pub cache: HashMap<String, DataUsageEntry>,
|
||||
@@ -1186,31 +1197,10 @@ impl DataUsageCache {
|
||||
}
|
||||
}
|
||||
|
||||
pub fn marshal_msg(&self) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let mut buf = Vec::new();
|
||||
self.serialize(&mut rmp_serde::Serializer::new(&mut buf))?;
|
||||
Ok(buf)
|
||||
}
|
||||
|
||||
pub fn unmarshal(buf: &[u8]) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let t: Self = rmp_serde::from_slice(buf)?;
|
||||
Ok(t)
|
||||
}
|
||||
|
||||
// Note: load and save methods are storage-specific and should be implemented
|
||||
// in the ecstore crate where storage access is available
|
||||
}
|
||||
|
||||
/// Trait for storage-specific operations on DataUsageCache
|
||||
#[async_trait::async_trait]
|
||||
pub trait DataUsageCacheStorage {
|
||||
/// Load data usage cache from backend storage
|
||||
async fn load(store: &dyn std::any::Any, name: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
|
||||
where
|
||||
Self: Sized;
|
||||
|
||||
/// Save data usage cache to backend storage
|
||||
async fn save(&self, name: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>>;
|
||||
}
|
||||
|
||||
// Helper structs and functions for cache operations
|
||||
@@ -1832,6 +1822,82 @@ mod tests {
|
||||
assert!(decoded.all_tier_stats.is_none());
|
||||
}
|
||||
|
||||
/// Scanner-written `.usage-cache.bin` bytes: a 2-element array of the
|
||||
/// canonical 16-field map-encoded info block and one map-encoded entry.
|
||||
/// Captured from the canonical writer's `marshal_msg` — see
|
||||
/// `usage_cache_wire_format_is_pinned` in
|
||||
/// `crates/scanner/src/data_usage_define.rs`, which pins these exact
|
||||
/// bytes and documents regeneration. Hardcoded here because a
|
||||
/// dev-dependency on rustfs-scanner would pull the whole ecstore tree
|
||||
/// into this crate's test build, and a fixture generated at test runtime
|
||||
/// could not detect writer drift anyway.
|
||||
const SCANNER_USAGE_CACHE_WIRE_FIXTURE: &[u8] = &[
|
||||
0x92, 0xde, 0x00, 0x10, 0xa4, 0x6e, 0x61, 0x6d, 0x65, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||
0x74, 0xaa, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x07, 0xac, 0x6c, 0x65, 0x61, 0x64, 0x65, 0x72,
|
||||
0x5f, 0x65, 0x70, 0x6f, 0x63, 0x68, 0x09, 0xab, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x92,
|
||||
0xce, 0x65, 0x53, 0xf1, 0x00, 0x00, 0xac, 0x73, 0x6b, 0x69, 0x70, 0x5f, 0x68, 0x65, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0xc3,
|
||||
0xa9, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0xc0, 0xab, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
|
||||
0x69, 0x6f, 0x6e, 0xc0, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x81,
|
||||
0xb0, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x6c, 0x6f, 0x73, 0x74, 0x0b, 0xb1, 0x73,
|
||||
0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0x5f, 0x61, 0x66, 0x74, 0x65, 0x72, 0xb2, 0x77, 0x69, 0x72,
|
||||
0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0xaf, 0x73, 0x63, 0x61, 0x6e,
|
||||
0x5f, 0x63, 0x68, 0x65, 0x63, 0x6b, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0xc0, 0xad, 0x70, 0x65, 0x6e, 0x64, 0x69, 0x6e, 0x67,
|
||||
0x5f, 0x68, 0x65, 0x61, 0x6c, 0x73, 0x91, 0x9a, 0xa6, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0xab, 0x77, 0x69, 0x72, 0x65,
|
||||
0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0xa6, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x6e, 0xc0, 0x01, 0x64, 0xcc, 0xc8, 0x03,
|
||||
0xa8, 0x64, 0x65, 0x66, 0x65, 0x72, 0x72, 0x65, 0x64, 0xa6, 0x62, 0x75, 0x64, 0x67, 0x65, 0x74, 0xab, 0x6f, 0x62, 0x6a,
|
||||
0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x63, 0x6b, 0xc0, 0xa6, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x92, 0x01, 0x02, 0xb1,
|
||||
0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0xc3, 0xb0, 0x73,
|
||||
0x63, 0x61, 0x6e, 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x5f, 0x64, 0x69, 0x67, 0x65, 0x73, 0x74, 0xdc, 0x00, 0x20, 0x03, 0x03,
|
||||
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
|
||||
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0xb0, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x6b, 0x65, 0x79,
|
||||
0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x01, 0x81, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||
0x74, 0x8b, 0xa8, 0x63, 0x68, 0x69, 0x6c, 0x64, 0x72, 0x65, 0x6e, 0x90, 0xa4, 0x73, 0x69, 0x7a, 0x65, 0xcd, 0x10, 0x00,
|
||||
0xa7, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x03, 0xa8, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x05, 0xae,
|
||||
0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x5f, 0x6d, 0x61, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x01, 0xa9, 0x6f, 0x62, 0x6a, 0x5f,
|
||||
0x73, 0x69, 0x7a, 0x65, 0x73, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xac, 0x6f, 0x62,
|
||||
0x6a, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x97, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x72,
|
||||
0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0xc0, 0xa9, 0x63, 0x6f,
|
||||
0x6d, 0x70, 0x61, 0x63, 0x74, 0x65, 0x64, 0xc3, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65,
|
||||
0x63, 0x74, 0x73, 0x02, 0xae, 0x61, 0x6c, 0x6c, 0x5f, 0x74, 0x69, 0x65, 0x72, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0x91,
|
||||
0x81, 0xa4, 0x57, 0x41, 0x52, 0x4d, 0x93, 0xcd, 0x08, 0x00, 0x02, 0x01,
|
||||
];
|
||||
|
||||
#[test]
|
||||
fn thin_usage_cache_decodes_scanner_wire_fixture() {
|
||||
let decoded =
|
||||
DataUsageCache::unmarshal(SCANNER_USAGE_CACHE_WIRE_FIXTURE).expect("thin projection decodes a scanner-written cache");
|
||||
|
||||
// The six fields shared with the scanner's 16-field info block; the
|
||||
// remaining ten (lifecycle, replication, checkpoint, heals, ...) must
|
||||
// be skipped, not error.
|
||||
assert_eq!(decoded.info.name, "wire-bucket");
|
||||
assert_eq!(decoded.info.next_cycle, 7);
|
||||
assert_eq!(
|
||||
decoded.info.last_update,
|
||||
Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000))
|
||||
);
|
||||
assert!(decoded.info.skip_healing);
|
||||
assert_eq!(decoded.info.failed_objects.get("wire-bucket/lost"), Some(&11));
|
||||
assert!(decoded.info.snapshot_complete);
|
||||
|
||||
// Entries use the shared canonical map-encoded type end to end.
|
||||
let entry = decoded.cache.get("wire-bucket").expect("fixture entry decodes");
|
||||
assert_eq!(entry.size, 4096);
|
||||
assert_eq!(entry.objects, 3);
|
||||
assert_eq!(entry.versions, 5);
|
||||
assert_eq!(entry.delete_markers, 1);
|
||||
assert!(entry.compacted);
|
||||
assert_eq!(entry.failed_objects, 2);
|
||||
assert_eq!(
|
||||
entry.all_tier_stats.as_ref().and_then(|tiers| tiers.tiers.get("WARM")),
|
||||
Some(&TierStats {
|
||||
total_size: 2048,
|
||||
num_versions: 2,
|
||||
num_objects: 1,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hash_path_uses_portable_slash_semantics() {
|
||||
for (input, expected) in [
|
||||
|
||||
@@ -62,8 +62,8 @@ pub(crate) struct VersionShardCensus {
|
||||
pub data_dir: Option<String>,
|
||||
pub erasure_index: Option<usize>,
|
||||
pub expected_part_numbers: BTreeSet<usize>,
|
||||
pub present_part_numbers: BTreeSet<usize>,
|
||||
pub present_part_fingerprints: BTreeMap<usize, PartShardFingerprint>,
|
||||
pub inline_data_fingerprint: Option<PartShardFingerprint>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
@@ -74,7 +74,12 @@ pub(crate) struct PartShardFingerprint {
|
||||
|
||||
impl VersionShardCensus {
|
||||
pub(crate) fn is_complete(&self) -> bool {
|
||||
self.has_xl_meta && self.expected_part_numbers == self.present_part_numbers
|
||||
self.has_xl_meta
|
||||
&& self.expected_part_numbers.len() == self.present_part_fingerprints.len()
|
||||
&& self
|
||||
.expected_part_numbers
|
||||
.iter()
|
||||
.all(|part_number| self.present_part_fingerprints.contains_key(part_number))
|
||||
}
|
||||
|
||||
pub(crate) fn matches_manifest(&self, manifest: &Self) -> bool {
|
||||
@@ -85,6 +90,7 @@ impl VersionShardCensus {
|
||||
&& self.erasure_index == manifest.erasure_index
|
||||
&& self.expected_part_numbers == manifest.expected_part_numbers
|
||||
&& self.present_part_fingerprints == manifest.present_part_fingerprints
|
||||
&& self.inline_data_fingerprint == manifest.inline_data_fingerprint
|
||||
}
|
||||
}
|
||||
|
||||
@@ -93,6 +99,13 @@ fn sha256_hex(data: &[u8]) -> String {
|
||||
digest.iter().map(|byte| format!("{byte:02x}")).collect()
|
||||
}
|
||||
|
||||
fn shard_fingerprint(data: &[u8]) -> ChaosResult<PartShardFingerprint> {
|
||||
Ok(PartShardFingerprint {
|
||||
size: u64::try_from(data.len())?,
|
||||
sha256: sha256_hex(data),
|
||||
})
|
||||
}
|
||||
|
||||
/// Single-node RustFS server with `disk_count` local volume directories that
|
||||
/// can be faulted individually while the server is running.
|
||||
pub struct DiskFaultHarness {
|
||||
@@ -301,8 +314,8 @@ pub(crate) fn census_object_version_on_disk(
|
||||
data_dir: None,
|
||||
erasure_index: None,
|
||||
expected_part_numbers: BTreeSet::new(),
|
||||
present_part_numbers: BTreeSet::new(),
|
||||
present_part_fingerprints: BTreeMap::new(),
|
||||
inline_data_fingerprint: None,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -315,10 +328,10 @@ pub(crate) fn census_object_version_on_disk(
|
||||
};
|
||||
let data_dir = file_info.data_dir.map(|id| id.to_string());
|
||||
let erasure_index = Some(file_info.erasure.index);
|
||||
let inline_data_fingerprint = file_info.data.as_deref().map(shard_fingerprint).transpose()?;
|
||||
let part_dir = data_dir.as_ref().map_or_else(|| object_dir.clone(), |id| object_dir.join(id));
|
||||
let (present_part_numbers, present_part_fingerprints) = match std::fs::read_dir(&part_dir) {
|
||||
let present_part_fingerprints = match std::fs::read_dir(&part_dir) {
|
||||
Ok(entries) => {
|
||||
let mut numbers = BTreeSet::new();
|
||||
let mut fingerprints = BTreeMap::new();
|
||||
for entry in entries {
|
||||
let entry = entry?;
|
||||
@@ -333,19 +346,12 @@ pub(crate) fn census_object_version_on_disk(
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
numbers.insert(part_number);
|
||||
let data = std::fs::read(entry.path())?;
|
||||
fingerprints.insert(
|
||||
part_number,
|
||||
PartShardFingerprint {
|
||||
size: u64::try_from(data.len())?,
|
||||
sha256: sha256_hex(&data),
|
||||
},
|
||||
);
|
||||
fingerprints.insert(part_number, shard_fingerprint(&data)?);
|
||||
}
|
||||
(numbers, fingerprints)
|
||||
fingerprints
|
||||
}
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => (BTreeSet::new(), BTreeMap::new()),
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => BTreeMap::new(),
|
||||
Err(error) => return Err(error.into()),
|
||||
};
|
||||
|
||||
@@ -355,8 +361,8 @@ pub(crate) fn census_object_version_on_disk(
|
||||
data_dir,
|
||||
erasure_index,
|
||||
expected_part_numbers,
|
||||
present_part_numbers,
|
||||
present_part_fingerprints,
|
||||
inline_data_fingerprint,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -396,3 +402,43 @@ pub async fn signed_admin_post(url: &str, body: Option<&str>, access_key: &str,
|
||||
|
||||
Ok(body)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn complete_census() -> VersionShardCensus {
|
||||
VersionShardCensus {
|
||||
version_id: Some("version".to_string()),
|
||||
has_xl_meta: true,
|
||||
data_dir: Some("data-dir".to_string()),
|
||||
erasure_index: Some(3),
|
||||
expected_part_numbers: BTreeSet::from([1]),
|
||||
present_part_fingerprints: BTreeMap::from([(1, shard_fingerprint(b"part").unwrap())]),
|
||||
inline_data_fingerprint: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shard_fingerprint_uses_physical_length_and_sha256() {
|
||||
assert_eq!(
|
||||
shard_fingerprint(b"abc").unwrap(),
|
||||
PartShardFingerprint {
|
||||
size: 3,
|
||||
sha256: "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad".to_string(),
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn manifest_requires_matching_inline_payload() {
|
||||
let mut expected = complete_census();
|
||||
expected.expected_part_numbers.clear();
|
||||
expected.present_part_fingerprints.clear();
|
||||
expected.inline_data_fingerprint = Some(shard_fingerprint(b"expected").unwrap());
|
||||
let mut changed = expected.clone();
|
||||
changed.inline_data_fingerprint = Some(shard_fingerprint(b"changed").unwrap());
|
||||
assert!(expected.matches_manifest(&expected));
|
||||
assert!(!changed.matches_manifest(&expected));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -349,11 +349,32 @@ mod tests {
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let first_inline = client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("versions/inline.bin")
|
||||
.body(ByteStream::from(payload(8 * 1024, 40)))
|
||||
.send()
|
||||
.await?;
|
||||
let first_inline_version = first_inline
|
||||
.version_id()
|
||||
.ok_or("first inline PUT did not return a version ID")?;
|
||||
let second_inline = client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("versions/inline.bin")
|
||||
.body(ByteStream::from(payload(8 * 1024, 41)))
|
||||
.send()
|
||||
.await?;
|
||||
let second_inline_version = second_inline
|
||||
.version_id()
|
||||
.ok_or("second inline PUT did not return a version ID")?;
|
||||
|
||||
let first = client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.body(ByteStream::from(payload(256 * 1024, 41)))
|
||||
.body(ByteStream::from(payload(128 * 1024, 41)))
|
||||
.send()
|
||||
.await?;
|
||||
let first_version = first.version_id().ok_or("first PUT did not return a version ID")?;
|
||||
@@ -361,16 +382,36 @@ mod tests {
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key(key)
|
||||
.body(ByteStream::from(payload(256 * 1024, 42)))
|
||||
.body(ByteStream::from(payload(3 * 1024 * 1024, 42)))
|
||||
.send()
|
||||
.await?;
|
||||
let second_version = second.version_id().ok_or("second PUT did not return a version ID")?;
|
||||
let delete = client.delete_object().bucket(bucket).key(key).send().await?;
|
||||
let delete_version = delete.version_id().ok_or("delete marker did not return a version ID")?;
|
||||
|
||||
let first_inline_census = harness.census_object_version(0, bucket, "versions/inline.bin", Some(first_inline_version))?;
|
||||
let second_inline_census =
|
||||
harness.census_object_version(0, bucket, "versions/inline.bin", Some(second_inline_version))?;
|
||||
let first_census = harness.census_object_version(0, bucket, key, Some(first_version))?;
|
||||
let first_other_disk_census = harness.census_object_version(1, bucket, key, Some(first_version))?;
|
||||
let second_census = harness.census_object_version(0, bucket, key, Some(second_version))?;
|
||||
let delete_census = harness.census_object_version(0, bucket, key, Some(delete_version))?;
|
||||
assert!(
|
||||
first_inline_census.is_complete() && second_inline_census.is_complete(),
|
||||
"inline version physical census is incomplete: first={first_inline_census:?} second={second_inline_census:?}"
|
||||
);
|
||||
assert!(
|
||||
first_inline_census.present_part_fingerprints.is_empty() && second_inline_census.present_part_fingerprints.is_empty(),
|
||||
"inline versions must not select external shard files: first={first_inline_census:?} second={second_inline_census:?}"
|
||||
);
|
||||
assert!(
|
||||
first_inline_census.inline_data_fingerprint.is_some() && second_inline_census.inline_data_fingerprint.is_some(),
|
||||
"inline versions must fingerprint payload bytes stored in xl.meta"
|
||||
);
|
||||
assert_ne!(
|
||||
first_inline_census.inline_data_fingerprint, second_inline_census.inline_data_fingerprint,
|
||||
"same-size inline versions with different payloads must retain distinct xl.meta fingerprints"
|
||||
);
|
||||
assert!(
|
||||
first_census.is_complete(),
|
||||
"first version physical census is incomplete: {first_census:?}"
|
||||
@@ -379,6 +420,14 @@ mod tests {
|
||||
second_census.is_complete(),
|
||||
"second version physical census is incomplete: {second_census:?}"
|
||||
);
|
||||
assert!(
|
||||
first_other_disk_census.is_complete(),
|
||||
"first version physical census on the second disk is incomplete: {first_other_disk_census:?}"
|
||||
);
|
||||
assert_ne!(
|
||||
first_census.erasure_index, first_other_disk_census.erasure_index,
|
||||
"physical census must preserve each disk's erasure index"
|
||||
);
|
||||
assert_ne!(
|
||||
first_census.data_dir, second_census.data_dir,
|
||||
"distinct object versions must select distinct physical data directories"
|
||||
@@ -387,6 +436,24 @@ mod tests {
|
||||
first_census.expected_part_numbers, second_census.expected_part_numbers,
|
||||
"same single-part shape should expose the same part numbers"
|
||||
);
|
||||
let first_part = first_census
|
||||
.present_part_fingerprints
|
||||
.values()
|
||||
.next()
|
||||
.ok_or("first version did not expose a physical part fingerprint")?;
|
||||
let second_part = second_census
|
||||
.present_part_fingerprints
|
||||
.values()
|
||||
.next()
|
||||
.ok_or("second version did not expose a physical part fingerprint")?;
|
||||
assert_ne!(
|
||||
first_part.size, second_part.size,
|
||||
"different shard lengths must retain their physical sizes"
|
||||
);
|
||||
assert_ne!(
|
||||
first_part.sha256, second_part.sha256,
|
||||
"different shard contents must retain their physical hashes"
|
||||
);
|
||||
assert!(
|
||||
delete_census.is_complete(),
|
||||
"delete marker physical census is incomplete: {delete_census:?}"
|
||||
@@ -396,7 +463,7 @@ mod tests {
|
||||
"delete marker must not declare object shards: {delete_census:?}"
|
||||
);
|
||||
assert!(
|
||||
delete_census.present_part_numbers.is_empty(),
|
||||
delete_census.present_part_fingerprints.is_empty(),
|
||||
"delete marker must not select stale object shards: {delete_census:?}"
|
||||
);
|
||||
Ok(())
|
||||
|
||||
@@ -59,6 +59,13 @@ mod tests {
|
||||
expected: VersionShardCensus,
|
||||
}
|
||||
|
||||
#[derive(Debug, Eq, PartialEq)]
|
||||
enum CompletionSample {
|
||||
Pending,
|
||||
Ready,
|
||||
CompletedWithIncomplete(BTreeSet<String>),
|
||||
}
|
||||
|
||||
struct MountNamespaceGuard {
|
||||
mounts: Vec<PathBuf>,
|
||||
}
|
||||
@@ -75,7 +82,7 @@ mod tests {
|
||||
impl MountNamespaceGuard {
|
||||
fn new() -> Result<Self, Box<dyn Error + Send + Sync>> {
|
||||
verify_isolated_mount_namespace()?;
|
||||
run_command("mount", ["--make-rprivate", "/"])?;
|
||||
run_command("mount", &["--make-rprivate", "/"])?;
|
||||
Ok(Self { mounts: Vec::new() })
|
||||
}
|
||||
|
||||
@@ -108,15 +115,15 @@ mod tests {
|
||||
return Err("losetup --find --show returned an empty loop device".into());
|
||||
}
|
||||
|
||||
run_command_dynamic("mkfs.ext4", &["-F", &loop_device])?;
|
||||
run_command("mkfs.ext4", &["-F", &loop_device])?;
|
||||
let sectors = run_command_stdout("blockdev", &["--getsz", &loop_device])?;
|
||||
let dm_name = format!("rustfs_e2e_{label}_{}", std::process::id());
|
||||
let table = format!("0 {sectors} linear {loop_device} 0");
|
||||
let mapper = format!("/dev/mapper/{dm_name}");
|
||||
run_command_dynamic("dmsetup", &["create", &dm_name, "--table", &table])?;
|
||||
run_command("dmsetup", &["create", &dm_name, "--table", &table])?;
|
||||
|
||||
let target_arg = path_to_string(target, "faultable mount target")?;
|
||||
run_command_dynamic("mount", &[&mapper, &target_arg])?;
|
||||
run_command("mount", &[&mapper, &target_arg])?;
|
||||
|
||||
Ok(Self {
|
||||
target: target.to_path_buf(),
|
||||
@@ -131,17 +138,17 @@ mod tests {
|
||||
fn make_unavailable(&self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let sectors = run_command_stdout("blockdev", &["--getsz", &self.loop_device])?;
|
||||
let error_table = format!("0 {sectors} error");
|
||||
run_command_dynamic("dmsetup", &["suspend", &self.dm_name])?;
|
||||
run_command_dynamic("dmsetup", &["load", &self.dm_name, "--table", &error_table])?;
|
||||
run_command_dynamic("dmsetup", &["resume", &self.dm_name])
|
||||
run_command("dmsetup", &["suspend", &self.dm_name])?;
|
||||
run_command("dmsetup", &["load", &self.dm_name, "--table", &error_table])?;
|
||||
run_command("dmsetup", &["resume", &self.dm_name])
|
||||
}
|
||||
|
||||
fn restore_available(&self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let sectors = run_command_stdout("blockdev", &["--getsz", &self.loop_device])?;
|
||||
let linear_table = format!("0 {sectors} linear {} 0", self.loop_device);
|
||||
run_command_dynamic("dmsetup", &["suspend", &self.dm_name])?;
|
||||
run_command_dynamic("dmsetup", &["load", &self.dm_name, "--table", &linear_table])?;
|
||||
run_command_dynamic("dmsetup", &["resume", &self.dm_name])
|
||||
run_command("dmsetup", &["suspend", &self.dm_name])?;
|
||||
run_command("dmsetup", &["load", &self.dm_name, "--table", &linear_table])?;
|
||||
run_command("dmsetup", &["resume", &self.dm_name])
|
||||
}
|
||||
|
||||
fn cleanup(&mut self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
@@ -157,14 +164,14 @@ mod tests {
|
||||
}
|
||||
}
|
||||
if self.dm_created {
|
||||
if let Err(error) = run_command_dynamic("dmsetup", &["remove", "-f", &self.dm_name]) {
|
||||
if let Err(error) = run_command("dmsetup", &["remove", "-f", &self.dm_name]) {
|
||||
first_error.get_or_insert(error);
|
||||
} else {
|
||||
self.dm_created = false;
|
||||
}
|
||||
}
|
||||
if !self.loop_device.is_empty() {
|
||||
if let Err(error) = run_command_dynamic("losetup", &["-d", &self.loop_device]) {
|
||||
if let Err(error) = run_command("losetup", &["-d", &self.loop_device]) {
|
||||
first_error.get_or_insert(error);
|
||||
} else {
|
||||
self.loop_device.clear();
|
||||
@@ -188,14 +195,10 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn run_command<const N: usize>(program: &str, args: [&str; N]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
run_command_dynamic(program, &args)
|
||||
}
|
||||
|
||||
fn run_command_dynamic(program: &str, args: &[&str]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
fn checked_command_output(program: &str, args: &[&str]) -> Result<std::process::Output, Box<dyn Error + Send + Sync>> {
|
||||
let output = Command::new(program).args(args).output()?;
|
||||
if output.status.success() {
|
||||
return Ok(());
|
||||
return Ok(output);
|
||||
}
|
||||
Err(format!(
|
||||
"{program} {} failed with status {}: stdout={} stderr={}",
|
||||
@@ -207,19 +210,14 @@ mod tests {
|
||||
.into())
|
||||
}
|
||||
|
||||
fn run_command(program: &str, args: &[&str]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
checked_command_output(program, args).map(drop)
|
||||
}
|
||||
|
||||
fn run_command_stdout(program: &str, args: &[&str]) -> Result<String, Box<dyn Error + Send + Sync>> {
|
||||
let output = Command::new(program).args(args).output()?;
|
||||
if output.status.success() {
|
||||
return Ok(String::from_utf8_lossy(&output.stdout).trim().to_string());
|
||||
}
|
||||
Err(format!(
|
||||
"{program} {} failed with status {}: stdout={} stderr={}",
|
||||
args.join(" "),
|
||||
output.status,
|
||||
String::from_utf8_lossy(&output.stdout),
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
)
|
||||
.into())
|
||||
Ok(String::from_utf8(checked_command_output(program, args)?.stdout)?
|
||||
.trim()
|
||||
.to_string())
|
||||
}
|
||||
|
||||
fn path_to_string(path: &Path, label: &str) -> Result<String, Box<dyn Error + Send + Sync>> {
|
||||
@@ -260,14 +258,14 @@ mod tests {
|
||||
let target = target
|
||||
.to_str()
|
||||
.ok_or_else(|| format!("tmpfs target path is not UTF-8: {target:?}"))?;
|
||||
run_command("mount", ["-t", "tmpfs", "-o", MOUNT_SIZE, label, target])
|
||||
run_command("mount", &["-t", "tmpfs", "-o", MOUNT_SIZE, label, target])
|
||||
}
|
||||
|
||||
fn detach_mount(target: &Path) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let target = target
|
||||
.to_str()
|
||||
.ok_or_else(|| format!("umount target path is not UTF-8: {target:?}"))?;
|
||||
run_command("umount", [target])
|
||||
run_command("umount", &[target])
|
||||
}
|
||||
|
||||
fn privileged_run_enabled() -> Result<bool, Box<dyn Error + Send + Sync>> {
|
||||
@@ -423,6 +421,10 @@ mod tests {
|
||||
.await?;
|
||||
versions.push((versioned_bucket, "history/object.bin", deleted.version_id().map(str::to_owned), None));
|
||||
|
||||
let (version_id, body_sha256) =
|
||||
put_object_version(client, versioned_bucket, "history/inline.bin", payload(8 * 1024, 9)).await?;
|
||||
versions.push((versioned_bucket, "history/inline.bin", version_id, Some(body_sha256)));
|
||||
|
||||
let (version_id, body_sha256) = put_multipart_version(
|
||||
client,
|
||||
versioned_bucket,
|
||||
@@ -436,7 +438,7 @@ mod tests {
|
||||
put_object_version(client, null_bucket, "null/current.bin", payload(512 * 1024, 8)).await?;
|
||||
versions.push((null_bucket, "null/current.bin", version_id, Some(body_sha256)));
|
||||
|
||||
versions
|
||||
let versions = versions
|
||||
.into_iter()
|
||||
.map(|(bucket, key, version_id, body_sha256)| {
|
||||
let expected = census_object_version_on_disk(target_disk, bucket, key, version_id.as_deref())?;
|
||||
@@ -451,7 +453,15 @@ mod tests {
|
||||
expected,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
.collect::<Result<Vec<_>, Box<dyn Error + Send + Sync>>>()?;
|
||||
let inline = versions
|
||||
.iter()
|
||||
.find(|version| version.key == "history/inline.bin")
|
||||
.ok_or("inline replacement baseline was not recorded")?;
|
||||
if inline.expected.inline_data_fingerprint.is_none() || !inline.expected.present_part_fingerprints.is_empty() {
|
||||
return Err(format!("inline replacement baseline lacks xl.meta payload evidence: {:?}", inline.expected).into());
|
||||
}
|
||||
Ok(versions)
|
||||
}
|
||||
|
||||
async fn verify_bodies(client: &Client, versions: &[BaselineVersion]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
@@ -597,6 +607,33 @@ mod tests {
|
||||
Ok(String::from_utf8_lossy(&log[start..]).into_owned())
|
||||
}
|
||||
|
||||
fn live_disk_loss_scan_completed(log: &str, target_disk: &Path) -> bool {
|
||||
let target = target_disk.to_string_lossy();
|
||||
let mut saw_live_loss = false;
|
||||
for line in log.lines() {
|
||||
if line.contains("Heal auto-scan disk inspection failed")
|
||||
&& line.contains("check_failed")
|
||||
&& line.contains(target.as_ref())
|
||||
{
|
||||
saw_live_loss = true;
|
||||
continue;
|
||||
}
|
||||
if saw_live_loss && (line.contains("Heal auto disk scanner idle") || line.contains("Heal auto-scan cycle completed"))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn live_disk_loss_scan_completed_from_path(
|
||||
log_path: &Path,
|
||||
start_offset: u64,
|
||||
target_disk: &Path,
|
||||
) -> Result<bool, Box<dyn Error + Send + Sync>> {
|
||||
Ok(live_disk_loss_scan_completed(&log_from_offset(log_path, start_offset)?, target_disk))
|
||||
}
|
||||
|
||||
async fn wait_for_live_disk_loss_observation(
|
||||
log_path: &Path,
|
||||
target_disk: &Path,
|
||||
@@ -605,20 +642,12 @@ mod tests {
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let deadline = Instant::now() + Duration::from_secs(timeout_secs);
|
||||
let mut tick = interval(Duration::from_secs(1));
|
||||
let target = target_disk.to_string_lossy();
|
||||
loop {
|
||||
let log = log_from_offset(log_path, start_offset)?;
|
||||
let mut saw_live_loss = false;
|
||||
for line in log.lines() {
|
||||
if line.contains("check_failed") && line.contains(target.as_ref()) {
|
||||
saw_live_loss = true;
|
||||
continue;
|
||||
}
|
||||
if saw_live_loss && line.contains("Heal auto disk scanner idle") {
|
||||
return Ok(());
|
||||
}
|
||||
if live_disk_loss_scan_completed_from_path(log_path, start_offset, target_disk)? {
|
||||
return Ok(());
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
let log = log_from_offset(log_path, start_offset)?;
|
||||
return Err(format!(
|
||||
"scanner did not finish a live target-loss scan for {target_disk:?} within {timeout_secs}s; log tail:\n{}",
|
||||
log_tail(&log)
|
||||
@@ -629,14 +658,10 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn require_definitive_replacement_status(
|
||||
status: &serde_json::Value,
|
||||
context: &str,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
if status["cluster"]["definitive"].as_bool().unwrap_or(false) {
|
||||
return Ok(());
|
||||
}
|
||||
Err(format!("{context} requires a definitive cluster replacement status: {status}").into())
|
||||
fn cluster_status_is_definitive(status: &serde_json::Value) -> Result<bool, Box<dyn Error + Send + Sync>> {
|
||||
status["cluster"]["definitive"]
|
||||
.as_bool()
|
||||
.ok_or_else(|| format!("replacement recovery status omitted cluster.definitive: {status}").into())
|
||||
}
|
||||
|
||||
async fn assert_no_replacement_status_records(
|
||||
@@ -644,8 +669,17 @@ mod tests {
|
||||
target_disk: &Path,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let status = replacement_status(cluster).await?;
|
||||
require_definitive_replacement_status(&status, "live missing replacement status check")?;
|
||||
let states = target_record_states(&status, target_disk);
|
||||
assert_no_replacement_status_records_in_status(&status, target_disk)
|
||||
}
|
||||
|
||||
fn assert_no_replacement_status_records_in_status(
|
||||
status: &serde_json::Value,
|
||||
target_disk: &Path,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
if !cluster_status_is_definitive(status)? {
|
||||
return Err(format!("live missing replacement status check requires a definitive cluster status: {status}").into());
|
||||
}
|
||||
let states = target_record_states(status, target_disk);
|
||||
if states.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
@@ -655,11 +689,6 @@ mod tests {
|
||||
.into())
|
||||
}
|
||||
|
||||
fn target_record_has_state(status: &serde_json::Value, target_disk: &Path, states: &[&str]) -> bool {
|
||||
let present_states = target_record_states(status, target_disk);
|
||||
states.iter().any(|state| present_states.contains(*state))
|
||||
}
|
||||
|
||||
fn target_record_states(status: &serde_json::Value, target_disk: &Path) -> BTreeSet<String> {
|
||||
let target = target_disk.to_string_lossy();
|
||||
status["cluster"]["records"]
|
||||
@@ -694,6 +723,57 @@ mod tests {
|
||||
Ok(missing)
|
||||
}
|
||||
|
||||
fn replacement_completion_state(
|
||||
status: &serde_json::Value,
|
||||
target_disk: &Path,
|
||||
missing: BTreeSet<String>,
|
||||
) -> Result<CompletionSample, Box<dyn Error + Send + Sync>> {
|
||||
if !cluster_status_is_definitive(status)? {
|
||||
return Ok(CompletionSample::Pending);
|
||||
}
|
||||
if !target_record_states(status, target_disk).contains("completed") {
|
||||
return Ok(CompletionSample::Pending);
|
||||
}
|
||||
if missing.is_empty() {
|
||||
return Ok(CompletionSample::Ready);
|
||||
}
|
||||
Ok(CompletionSample::CompletedWithIncomplete(missing))
|
||||
}
|
||||
|
||||
async fn sample_replacement_completion<C, S, F>(
|
||||
target_disk: &Path,
|
||||
census: C,
|
||||
status: S,
|
||||
) -> Result<CompletionSample, Box<dyn Error + Send + Sync>>
|
||||
where
|
||||
C: FnOnce() -> Result<BTreeSet<String>, Box<dyn Error + Send + Sync>>,
|
||||
S: FnOnce() -> F,
|
||||
F: std::future::Future<Output = Result<serde_json::Value, Box<dyn Error + Send + Sync>>>,
|
||||
{
|
||||
let missing = census()?;
|
||||
let status = status().await?;
|
||||
replacement_completion_state(&status, target_disk, missing)
|
||||
}
|
||||
|
||||
async fn confirm_replacement_completion<C, S, F>(
|
||||
target_disk: &Path,
|
||||
mut census: C,
|
||||
mut status: S,
|
||||
) -> Result<CompletionSample, Box<dyn Error + Send + Sync>>
|
||||
where
|
||||
C: FnMut() -> Result<BTreeSet<String>, Box<dyn Error + Send + Sync>>,
|
||||
S: FnMut() -> F,
|
||||
F: std::future::Future<Output = Result<serde_json::Value, Box<dyn Error + Send + Sync>>>,
|
||||
{
|
||||
let missing = census()?;
|
||||
let status = status().await?;
|
||||
let first = replacement_completion_state(&status, target_disk, missing)?;
|
||||
if matches!(first, CompletionSample::CompletedWithIncomplete(_)) {
|
||||
return replacement_completion_state(&status, target_disk, census()?);
|
||||
}
|
||||
Ok(first)
|
||||
}
|
||||
|
||||
async fn wait_for_completed_replacement_with_census(
|
||||
cluster: &RustFSTestClusterEnvironment,
|
||||
target_disk: &Path,
|
||||
@@ -703,28 +783,25 @@ mod tests {
|
||||
let deadline = Instant::now() + Duration::from_secs(timeout_secs);
|
||||
let mut tick = interval(Duration::from_secs(1));
|
||||
loop {
|
||||
let status = replacement_status(cluster).await?;
|
||||
let missing = incomplete_versions(target_disk, versions)?;
|
||||
if require_definitive_replacement_status(&status, "replacement completion poll").is_err() {
|
||||
if Instant::now() >= deadline {
|
||||
match confirm_replacement_completion(
|
||||
target_disk,
|
||||
|| incomplete_versions(target_disk, versions),
|
||||
|| replacement_status(cluster),
|
||||
)
|
||||
.await?
|
||||
{
|
||||
CompletionSample::Ready => return Ok(()),
|
||||
CompletionSample::CompletedWithIncomplete(confirmed_missing) => {
|
||||
return Err(format!(
|
||||
"replacement recovery status never became definitive within {timeout_secs}s while waiting for physical census; latest status: {status}; missing: {missing:?}"
|
||||
)
|
||||
.into());
|
||||
"replacement status remained completed across two incomplete physical censuses: {confirmed_missing:?}"
|
||||
)
|
||||
.into());
|
||||
}
|
||||
tick.tick().await;
|
||||
continue;
|
||||
}
|
||||
if target_record_has_state(&status, target_disk, &["completed"]) {
|
||||
if !missing.is_empty() {
|
||||
return Err(format!(
|
||||
"replacement status reached completed before target physical census matched baseline: {missing:?}; status: {status}"
|
||||
)
|
||||
.into());
|
||||
}
|
||||
return Ok(());
|
||||
CompletionSample::Pending => {}
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
let missing = incomplete_versions(target_disk, versions)?;
|
||||
let status = replacement_status(cluster).await?;
|
||||
return Err(format!(
|
||||
"replacement target did not reach completed with matching physical census within {timeout_secs}s: missing={missing:?}; status={status}"
|
||||
)
|
||||
@@ -812,6 +889,175 @@ mod tests {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn live_loss_barrier_requires_scanner_failure_after_log_offset() -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
let target = Path::new("/mnt/target");
|
||||
assert!(live_disk_loss_scan_completed(
|
||||
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto-scan cycle completed",
|
||||
target
|
||||
));
|
||||
assert!(live_disk_loss_scan_completed(
|
||||
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle",
|
||||
target
|
||||
));
|
||||
assert!(!live_disk_loss_scan_completed(
|
||||
"Heal auto disk scanner idle\nHeal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed",
|
||||
target
|
||||
));
|
||||
assert!(!live_disk_loss_scan_completed(
|
||||
"event=disk_health_check_failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle",
|
||||
target
|
||||
));
|
||||
assert!(!live_disk_loss_scan_completed(
|
||||
"Heal auto-scan disk inspection failed endpoint=/mnt/other disk_state=check_failed\nHeal auto disk scanner idle",
|
||||
target
|
||||
));
|
||||
let path = std::env::temp_dir().join(format!("rustfs-replacement-scan-{}.log", std::process::id()));
|
||||
let stale =
|
||||
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle\n";
|
||||
fs::write(&path, stale)?;
|
||||
let offset = log_len(&path)?;
|
||||
assert!(!live_disk_loss_scan_completed_from_path(&path, offset, target)?);
|
||||
let fresh =
|
||||
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle\n";
|
||||
fs::write(&path, format!("{stale}{fresh}"))?;
|
||||
assert!(live_disk_loss_scan_completed_from_path(&path, offset, target)?);
|
||||
fs::remove_file(path)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn completion_requires_definitive_status_and_prior_census_match() {
|
||||
let target = Path::new("/mnt/target");
|
||||
let non_definitive = serde_json::json!({
|
||||
"cluster": {"definitive": false, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||
});
|
||||
assert_eq!(
|
||||
replacement_completion_state(&non_definitive, target, BTreeSet::new()).unwrap(),
|
||||
CompletionSample::Pending
|
||||
);
|
||||
|
||||
let omitted = serde_json::json!({
|
||||
"cluster": {"records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||
});
|
||||
assert!(replacement_completion_state(&omitted, target, BTreeSet::new()).is_err());
|
||||
|
||||
let definitive = serde_json::json!({
|
||||
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||
});
|
||||
assert_eq!(
|
||||
replacement_completion_state(&definitive, target, BTreeSet::from(["missing".to_string()])).unwrap(),
|
||||
CompletionSample::CompletedWithIncomplete(BTreeSet::from(["missing".to_string()]))
|
||||
);
|
||||
assert_eq!(
|
||||
replacement_completion_state(&definitive, target, BTreeSet::new()).unwrap(),
|
||||
CompletionSample::Ready
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn completion_poll_samples_census_before_status() {
|
||||
let order = std::rc::Rc::new(std::cell::RefCell::new(Vec::new()));
|
||||
let census_order = order.clone();
|
||||
let status_order = order.clone();
|
||||
let sample = sample_replacement_completion(
|
||||
Path::new("/mnt/target"),
|
||||
move || {
|
||||
census_order.borrow_mut().push("census");
|
||||
Ok::<_, Box<dyn Error + Send + Sync>>(BTreeSet::new())
|
||||
},
|
||||
move || async move {
|
||||
status_order.borrow_mut().push("status");
|
||||
Ok::<_, Box<dyn Error + Send + Sync>>(serde_json::json!({
|
||||
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||
}))
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(sample, CompletionSample::Ready);
|
||||
assert_eq!(*order.borrow(), ["census", "status"]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn completed_status_confirms_a_stale_incomplete_census() {
|
||||
let samples = std::rc::Rc::new(std::cell::RefCell::new(std::collections::VecDeque::from([
|
||||
BTreeSet::from(["missing".to_string()]),
|
||||
BTreeSet::new(),
|
||||
])));
|
||||
let census_samples = samples.clone();
|
||||
let result = confirm_replacement_completion(
|
||||
Path::new("/mnt/target"),
|
||||
move || {
|
||||
census_samples
|
||||
.borrow_mut()
|
||||
.pop_front()
|
||||
.ok_or_else(|| "missing census sample".into())
|
||||
},
|
||||
|| async {
|
||||
Ok(serde_json::json!({
|
||||
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||
}))
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(result, CompletionSample::Ready);
|
||||
assert!(samples.borrow().is_empty());
|
||||
|
||||
let status_samples = std::rc::Rc::new(std::cell::RefCell::new(std::collections::VecDeque::from([
|
||||
serde_json::json!({
|
||||
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||
}),
|
||||
serde_json::json!({
|
||||
"cluster": {"definitive": false, "records": []}
|
||||
}),
|
||||
])));
|
||||
let persistent = std::rc::Rc::new(std::cell::RefCell::new(std::collections::VecDeque::from([
|
||||
BTreeSet::from(["missing".to_string()]),
|
||||
BTreeSet::from(["still-missing".to_string()]),
|
||||
])));
|
||||
let census_samples = persistent.clone();
|
||||
let statuses = status_samples.clone();
|
||||
let result = confirm_replacement_completion(
|
||||
Path::new("/mnt/target"),
|
||||
move || {
|
||||
census_samples
|
||||
.borrow_mut()
|
||||
.pop_front()
|
||||
.ok_or_else(|| "missing census sample".into())
|
||||
},
|
||||
move || {
|
||||
let statuses = statuses.clone();
|
||||
async move {
|
||||
statuses
|
||||
.borrow_mut()
|
||||
.pop_front()
|
||||
.ok_or_else(|| "missing status sample".into())
|
||||
}
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
result,
|
||||
CompletionSample::CompletedWithIncomplete(BTreeSet::from(["still-missing".to_string()]))
|
||||
);
|
||||
assert!(persistent.borrow().is_empty());
|
||||
assert_eq!(status_samples.borrow().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absent_status_requires_definitive_empty_records() {
|
||||
let target = Path::new("/mnt/target");
|
||||
let non_definitive = serde_json::json!({"cluster": {"definitive": false, "records": []}});
|
||||
assert!(assert_no_replacement_status_records_in_status(&non_definitive, target).is_err());
|
||||
let omitted = serde_json::json!({"cluster": {"records": []}});
|
||||
assert!(assert_no_replacement_status_records_in_status(&omitted, target).is_err());
|
||||
let definitive = serde_json::json!({"cluster": {"definitive": true, "records": []}});
|
||||
assert!(assert_no_replacement_status_records_in_status(&definitive, target).is_ok());
|
||||
}
|
||||
|
||||
/// Linux mount namespaces are per-thread; keep mount setup and process
|
||||
/// spawning on one OS thread so child RustFS nodes inherit the test mounts.
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
|
||||
@@ -2150,30 +2150,6 @@ pub async fn load_data_usage_cache(store: &crate::set_disk::SetDisks, name: &str
|
||||
Ok(d)
|
||||
}
|
||||
|
||||
#[instrument(skip(cache))]
|
||||
pub async fn save_data_usage_cache(cache: &DataUsageCache, name: &str) -> crate::error::Result<()> {
|
||||
use crate::config::com::save_config;
|
||||
use crate::disk::BUCKET_META_PREFIX;
|
||||
use std::path::Path;
|
||||
|
||||
let Some(store) = runtime_sources::object_store_handle() else {
|
||||
return Err(Error::other("errServerNotInitialized"));
|
||||
};
|
||||
let buf = cache.marshal_msg().map_err(Error::other)?;
|
||||
let buf_clone = buf.clone();
|
||||
|
||||
let store_clone = store.clone();
|
||||
|
||||
let name = Path::new(BUCKET_META_PREFIX).join(name).to_string_lossy().to_string();
|
||||
|
||||
let name_clone = name.clone();
|
||||
tokio::spawn(async move {
|
||||
let _ = save_config(store_clone, &format!("{}{}", name_clone, ".bkp"), buf_clone).await;
|
||||
});
|
||||
save_config(store, &name, buf).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Persist the current in-memory compression total to the backend.
|
||||
/// Resets the debounce counter so the next auto-persist won't fire
|
||||
/// immediately after this manual flush (intended for shutdown paths).
|
||||
|
||||
@@ -331,7 +331,14 @@ impl From<std::io::Error> for DiskError {
|
||||
}
|
||||
match e.downcast::<DiskError>() {
|
||||
Ok(disk_error) => disk_error,
|
||||
Err(io_error) => DiskError::Io(io_error),
|
||||
// Mirror `From<io::Error> for StorageError`: a StorageError boxed
|
||||
// through `From<StorageError> for io::Error` must recover its typed
|
||||
// classification instead of degrading to `DiskError::Io`, which
|
||||
// quorum aggregation (`reduce_errs`) would count as a distinct error.
|
||||
Err(io_error) => match io_error.downcast::<crate::error::StorageError>() {
|
||||
Ok(storage_error) => storage_error.into(),
|
||||
Err(io_error) => DiskError::Io(io_error),
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -953,6 +960,27 @@ mod tests {
|
||||
assert_eq!(original_disk_error, recovered_disk_error);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_io_error_with_storage_error_inside() {
|
||||
use crate::error::StorageError;
|
||||
|
||||
// An io::Error boxing a disk-representable StorageError (as produced by
|
||||
// `From<StorageError> for io::Error`) must recover the typed DiskError
|
||||
// variant instead of degrading to an opaque DiskError::Io.
|
||||
let io_with_storage_error: std::io::Error = StorageError::FaultyRemoteDisk.into();
|
||||
let recovered: DiskError = io_with_storage_error.into();
|
||||
assert_eq!(recovered, DiskError::FaultyRemoteDisk);
|
||||
|
||||
let io_with_storage_error: std::io::Error = StorageError::FileAccessDenied.into();
|
||||
let recovered: DiskError = io_with_storage_error.into();
|
||||
assert_eq!(recovered, DiskError::FileAccessDenied);
|
||||
|
||||
// A StorageError with no DiskError analog stays an opaque Io error.
|
||||
let io_with_bucket_error: std::io::Error = StorageError::BucketNotFound("bucket".to_string()).into();
|
||||
let recovered: DiskError = io_with_bucket_error.into();
|
||||
assert!(matches!(recovered, DiskError::Io(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_io_error_different_kinds() {
|
||||
use std::io::ErrorKind;
|
||||
|
||||
@@ -2955,11 +2955,31 @@ impl std::fmt::Debug for StdBackend {
|
||||
|
||||
impl StdBackend {
|
||||
pub(crate) fn new(root: PathBuf) -> Self {
|
||||
Self::build(root, true)
|
||||
}
|
||||
|
||||
/// Construct without the descriptor cache.
|
||||
///
|
||||
/// `UringBackend` wraps a `StdBackend` and runs its own `FdCache` over the
|
||||
/// same positioned reads. If the inner `StdBackend` also built a cache, a
|
||||
/// fallback read (`UringBackend::pread_bytes` delegates to the inner backend
|
||||
/// on latch-off / O_DIRECT / buffered errors) would populate a *second*
|
||||
/// cache that `UringBackend`'s invalidation never touches — re-opening the
|
||||
/// stale-inode hazard `FdCache` exists to close (rustfs/backlog#1176/#1801).
|
||||
/// The wrapper therefore owns the only cache for the disk; the inner backend
|
||||
/// opens per read. This also avoids double-counting `FD_CACHE_CAPACITY`
|
||||
/// against `RLIMIT_NOFILE` (rustfs/backlog#1178).
|
||||
#[cfg(target_os = "linux")]
|
||||
pub(crate) fn new_without_fd_cache(root: PathBuf) -> Self {
|
||||
Self::build(root, false)
|
||||
}
|
||||
|
||||
fn build(root: PathBuf, build_fd_cache: bool) -> Self {
|
||||
// Gate the fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178):
|
||||
// 512 fds/disk with a low soft limit and several disks would hit EMFILE.
|
||||
// Fall back to open-per-read when the limit is too small.
|
||||
#[cfg(target_os = "linux")]
|
||||
let fd_cache = if is_local_fd_cache_enabled() {
|
||||
let fd_cache = if build_fd_cache && is_local_fd_cache_enabled() {
|
||||
if rlimit_allows_fd_cache() {
|
||||
Some(FdCache::new())
|
||||
} else {
|
||||
@@ -2973,6 +2993,10 @@ impl StdBackend {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
// `build_fd_cache` is only consulted on Linux (for the fd cache); on
|
||||
// other platforms it has no effect and would trip the unused-variable lint.
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let _ = build_fd_cache;
|
||||
Self {
|
||||
root,
|
||||
#[cfg(target_os = "linux")]
|
||||
@@ -4120,7 +4144,7 @@ impl UringBackend {
|
||||
// struct (rustfs/backlog#1185).
|
||||
let root_label = root.display().to_string();
|
||||
Some(Self {
|
||||
inner: StdBackend::new(root.clone()),
|
||||
inner: StdBackend::new_without_fd_cache(root.clone()),
|
||||
root,
|
||||
root_label,
|
||||
driver: std::mem::ManuallyDrop::new(driver),
|
||||
@@ -19919,6 +19943,23 @@ mod test {
|
||||
assert_eq!(cache.entry_count().await, 0, "prefix invalidation must drop the cached descriptor");
|
||||
}
|
||||
|
||||
/// `StdBackend::new_without_fd_cache` must not build a descriptor cache.
|
||||
/// `UringBackend` wraps a `StdBackend` and owns the only cache for the disk,
|
||||
/// so an inner cache would be populated by fallback reads
|
||||
/// (`UringBackend::pread_bytes` delegates inward) yet never invalidated —
|
||||
/// the stale-inode hazard `FdCache` exists to close (backlog#1176/#1801).
|
||||
/// This pins the contract so a future constructor change cannot regress it.
|
||||
#[cfg(target_os = "linux")]
|
||||
#[test]
|
||||
fn new_without_fd_cache_builds_no_descriptor_cache() {
|
||||
let root_dir = tempfile::tempdir().expect("operation should succeed");
|
||||
let backend = StdBackend::new_without_fd_cache(root_dir.path().to_path_buf());
|
||||
assert!(
|
||||
backend.fd_cache.is_none(),
|
||||
"new_without_fd_cache must not build a descriptor cache — UringBackend owns the only cache for the disk"
|
||||
);
|
||||
}
|
||||
|
||||
/// The mutation paths on `LocalDisk` must actually call
|
||||
/// `invalidate_cached_fds`, not merely have it available (backlog#1145).
|
||||
/// `rename_file` replaces the inode at a path a reader has already cached;
|
||||
|
||||
@@ -358,6 +358,13 @@ impl From<StorageError> for DiskError {
|
||||
StorageError::VolumeNotFound => DiskError::VolumeNotFound,
|
||||
StorageError::VolumeExists => DiskError::VolumeExists,
|
||||
StorageError::FileNameTooLong => DiskError::FileNameTooLong,
|
||||
StorageError::FaultyRemoteDisk => DiskError::FaultyRemoteDisk,
|
||||
StorageError::DiskAccessDenied => DiskError::DiskAccessDenied,
|
||||
StorageError::DriveIsRoot => DiskError::DriveIsRoot,
|
||||
StorageError::IsNotRegular => DiskError::IsNotRegular,
|
||||
StorageError::VolumeNotEmpty => DiskError::VolumeNotEmpty,
|
||||
StorageError::VolumeAccessDenied => DiskError::VolumeAccessDenied,
|
||||
StorageError::FileAccessDenied => DiskError::FileAccessDenied,
|
||||
_ => DiskError::other(val),
|
||||
}
|
||||
}
|
||||
@@ -1492,6 +1499,49 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
// Every DiskError variant must survive DiskError -> StorageError -> DiskError
|
||||
// unchanged. A variant that degrades to `DiskError::Io` on the way back loses
|
||||
// its identity for quorum aggregation (`reduce_errs` classifies by variant
|
||||
// equality), so ignore-list entries such as FaultyRemoteDisk and
|
||||
// DiskAccessDenied would silently stop matching.
|
||||
#[test]
|
||||
fn test_disk_error_storage_error_round_trip_identity_all_variants() {
|
||||
// DiskError codes are contiguous from 0x01, so enumerating via from_u32
|
||||
// covers every variant and picks up newly appended ones automatically.
|
||||
let all_variants: Vec<DiskError> = (1u32..).map_while(DiskError::from_u32).collect();
|
||||
assert!(
|
||||
all_variants.len() >= 42,
|
||||
"DiskError variant enumeration shrank: got {}, expected at least 42",
|
||||
all_variants.len()
|
||||
);
|
||||
|
||||
for original in all_variants {
|
||||
let storage_error: StorageError = original.clone().into();
|
||||
let round_tripped: DiskError = storage_error.into();
|
||||
|
||||
assert_eq!(
|
||||
std::mem::discriminant(&original),
|
||||
std::mem::discriminant(&round_tripped),
|
||||
"round trip changed variant: {original:?} -> {round_tripped:?}"
|
||||
);
|
||||
assert_eq!(original, round_tripped, "round trip not identical for {original:?}");
|
||||
}
|
||||
|
||||
// Io is the only payload-carrying variant: a representative kind and
|
||||
// message must both survive the round trip.
|
||||
let io_original = DiskError::Io(IoError::new(ErrorKind::PermissionDenied, "denied"));
|
||||
let storage_error: StorageError = io_original.clone().into();
|
||||
let io_round_tripped: DiskError = storage_error.into();
|
||||
assert_eq!(io_original, io_round_tripped);
|
||||
match io_round_tripped {
|
||||
DiskError::Io(inner) => {
|
||||
assert_eq!(inner.kind(), ErrorKind::PermissionDenied);
|
||||
assert_eq!(inner.to_string(), "denied");
|
||||
}
|
||||
other => panic!("expected DiskError::Io, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_storage_error_from_io_error() {
|
||||
// Test direct IO error conversion
|
||||
|
||||
@@ -9179,6 +9179,71 @@ mod tests {
|
||||
assert_eq!(body.as_ref(), payload);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn direct_memory_versioned_bucket_uses_inline_data_shards_for_latest() {
|
||||
let tempdir = tempfile::tempdir().expect("tempdir should be created");
|
||||
let endpoint =
|
||||
Endpoint::try_from(tempdir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
|
||||
let disk = new_disk(
|
||||
&endpoint,
|
||||
&DiskOption {
|
||||
cleanup: false,
|
||||
health_check: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("disk should be created");
|
||||
|
||||
let payload = vec![b'v'; 64 * 1024];
|
||||
let payload_size = i64::try_from(payload.len()).expect("test payload size should fit i64");
|
||||
let (erasure, files, _read_length, _checksum_algo) = inline_bitrot_files_for_payload(&payload).await;
|
||||
let mut fi = FileInfo::new("bucket/object", erasure.data_shards, erasure.parity_shards);
|
||||
fi.size = payload_size;
|
||||
fi.data = files[0].data.clone();
|
||||
fi.add_object_part(1, String::new(), payload.len(), None, payload_size, None, None);
|
||||
|
||||
let mut object_info = ObjectInfo {
|
||||
size: payload_size,
|
||||
actual_size: payload_size,
|
||||
parts: Arc::new(vec![ObjectPartInfo {
|
||||
number: 1,
|
||||
size: payload.len(),
|
||||
actual_size: payload_size,
|
||||
..Default::default()
|
||||
}]),
|
||||
..Default::default()
|
||||
};
|
||||
object_info.inlined = true;
|
||||
let opts = ObjectOptions {
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
};
|
||||
let metrics_size_bucket = rustfs_io_metrics::get_object_size_bucket(fi.size);
|
||||
|
||||
assert_eq!(
|
||||
get_small_object_direct_memory_decision_with_threshold(&None, &object_info, &fi, &opts, true, 128 * 1024),
|
||||
GetDirectMemoryDecision::Use {
|
||||
object_size: payload.len()
|
||||
}
|
||||
);
|
||||
|
||||
let body = SetDisks::try_get_object_direct_data_shards_with_fileinfo(
|
||||
"bucket",
|
||||
"object",
|
||||
&fi,
|
||||
&files,
|
||||
&vec![Some(disk); erasure.total_shard_count()],
|
||||
true,
|
||||
GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART,
|
||||
metrics_size_bucket,
|
||||
)
|
||||
.await
|
||||
.expect("versioned latest direct-memory read should not fail")
|
||||
.expect("versioned latest should use inline data shards");
|
||||
|
||||
assert_eq!(body.as_ref(), payload);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn direct_memory_data_shards_direct_read_reassembles_single_block_payload() {
|
||||
use uuid::Uuid;
|
||||
|
||||
@@ -221,9 +221,9 @@ impl SetDisks {
|
||||
let disks = self.disks.read().await.clone();
|
||||
let required_reads = self.default_read_quorum();
|
||||
|
||||
let bucket = bucket.to_string();
|
||||
let object = object.to_string();
|
||||
let version_id = version_id.to_string();
|
||||
let bucket: Arc<str> = Arc::from(bucket);
|
||||
let object: Arc<str> = Arc::from(object);
|
||||
let version_id: Arc<str> = Arc::from(version_id);
|
||||
let opts = *opts;
|
||||
|
||||
let processor = runtime_sources::batch_processors().read_processor();
|
||||
|
||||
@@ -2403,6 +2403,150 @@ mod tests {
|
||||
assert_eq!(decoded.cache.get("bucket").map(|entry| entry.objects), Some(3));
|
||||
}
|
||||
|
||||
/// Deterministic, fully populated cache used to pin the persisted
|
||||
/// `.usage-cache.bin` wire bytes. Every map/set holds at most one element
|
||||
/// so the map-encoded `marshal_msg` output is byte-stable.
|
||||
fn wire_fixture_cache() -> DataUsageCache {
|
||||
let mut entry = DataUsageEntry {
|
||||
size: 4096,
|
||||
objects: 3,
|
||||
versions: 5,
|
||||
delete_markers: 1,
|
||||
compacted: true,
|
||||
failed_objects: 2,
|
||||
..Default::default()
|
||||
};
|
||||
entry.add_tier_sizes(&HashMap::from([(
|
||||
"WARM".to_string(),
|
||||
TierStats {
|
||||
total_size: 2048,
|
||||
num_versions: 2,
|
||||
num_objects: 1,
|
||||
},
|
||||
)]));
|
||||
let mut cache = DataUsageCache {
|
||||
info: DataUsageCacheInfo {
|
||||
name: "wire-bucket".to_string(),
|
||||
next_cycle: 7,
|
||||
leader_epoch: 9,
|
||||
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000)),
|
||||
skip_healing: true,
|
||||
failed_objects: HashMap::from([("wire-bucket/lost".to_string(), 11)]),
|
||||
scan_resume_after: Some("wire-bucket/resume".to_string()),
|
||||
pending_heals: vec![PendingScannerHeal {
|
||||
kind: PendingScannerHealKind::Object,
|
||||
bucket: "wire-bucket".to_string(),
|
||||
object: Some("broken".to_string()),
|
||||
version_id: None,
|
||||
scan_mode: HealScanMode::Normal,
|
||||
first_seen: 100,
|
||||
last_attempt: 200,
|
||||
attempts: 3,
|
||||
last_admission_result: "deferred".to_string(),
|
||||
last_admission_reason: "budget".to_string(),
|
||||
}],
|
||||
source: Some(DataUsageCacheSource::new(1, 2)),
|
||||
snapshot_complete: true,
|
||||
scan_plan_digest: Some(TEST_PLAN_DIGEST),
|
||||
cache_key_format: DATA_USAGE_CACHE_KEY_FORMAT,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
};
|
||||
cache.replace("wire-bucket", "", entry);
|
||||
cache
|
||||
}
|
||||
|
||||
/// Persisted `.usage-cache.bin` bytes produced by [`wire_fixture_cache`]
|
||||
/// via `DataUsageCache::marshal_msg`: a 2-element array of the 16-field
|
||||
/// map-encoded info block and the map of map-encoded entries.
|
||||
///
|
||||
/// The thin read-only projection in `crates/data-usage` decodes a copy of
|
||||
/// this fixture (`thin_usage_cache_decodes_scanner_wire_fixture`); when
|
||||
/// the encoding legitimately changes, regenerate both copies from
|
||||
/// `wire_fixture_cache().marshal_msg()` and re-verify old readers.
|
||||
const USAGE_CACHE_WIRE_FIXTURE: &[u8] = &[
|
||||
0x92, 0xde, 0x00, 0x10, 0xa4, 0x6e, 0x61, 0x6d, 0x65, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||
0x74, 0xaa, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x07, 0xac, 0x6c, 0x65, 0x61, 0x64, 0x65, 0x72,
|
||||
0x5f, 0x65, 0x70, 0x6f, 0x63, 0x68, 0x09, 0xab, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x92,
|
||||
0xce, 0x65, 0x53, 0xf1, 0x00, 0x00, 0xac, 0x73, 0x6b, 0x69, 0x70, 0x5f, 0x68, 0x65, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0xc3,
|
||||
0xa9, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0xc0, 0xab, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
|
||||
0x69, 0x6f, 0x6e, 0xc0, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x81,
|
||||
0xb0, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x6c, 0x6f, 0x73, 0x74, 0x0b, 0xb1, 0x73,
|
||||
0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0x5f, 0x61, 0x66, 0x74, 0x65, 0x72, 0xb2, 0x77, 0x69, 0x72,
|
||||
0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0xaf, 0x73, 0x63, 0x61, 0x6e,
|
||||
0x5f, 0x63, 0x68, 0x65, 0x63, 0x6b, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0xc0, 0xad, 0x70, 0x65, 0x6e, 0x64, 0x69, 0x6e, 0x67,
|
||||
0x5f, 0x68, 0x65, 0x61, 0x6c, 0x73, 0x91, 0x9a, 0xa6, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0xab, 0x77, 0x69, 0x72, 0x65,
|
||||
0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0xa6, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x6e, 0xc0, 0x01, 0x64, 0xcc, 0xc8, 0x03,
|
||||
0xa8, 0x64, 0x65, 0x66, 0x65, 0x72, 0x72, 0x65, 0x64, 0xa6, 0x62, 0x75, 0x64, 0x67, 0x65, 0x74, 0xab, 0x6f, 0x62, 0x6a,
|
||||
0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x63, 0x6b, 0xc0, 0xa6, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x92, 0x01, 0x02, 0xb1,
|
||||
0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0xc3, 0xb0, 0x73,
|
||||
0x63, 0x61, 0x6e, 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x5f, 0x64, 0x69, 0x67, 0x65, 0x73, 0x74, 0xdc, 0x00, 0x20, 0x03, 0x03,
|
||||
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
|
||||
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0xb0, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x6b, 0x65, 0x79,
|
||||
0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x01, 0x81, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||
0x74, 0x8b, 0xa8, 0x63, 0x68, 0x69, 0x6c, 0x64, 0x72, 0x65, 0x6e, 0x90, 0xa4, 0x73, 0x69, 0x7a, 0x65, 0xcd, 0x10, 0x00,
|
||||
0xa7, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x03, 0xa8, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x05, 0xae,
|
||||
0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x5f, 0x6d, 0x61, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x01, 0xa9, 0x6f, 0x62, 0x6a, 0x5f,
|
||||
0x73, 0x69, 0x7a, 0x65, 0x73, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xac, 0x6f, 0x62,
|
||||
0x6a, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x97, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x72,
|
||||
0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0xc0, 0xa9, 0x63, 0x6f,
|
||||
0x6d, 0x70, 0x61, 0x63, 0x74, 0x65, 0x64, 0xc3, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65,
|
||||
0x63, 0x74, 0x73, 0x02, 0xae, 0x61, 0x6c, 0x6c, 0x5f, 0x74, 0x69, 0x65, 0x72, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0x91,
|
||||
0x81, 0xa4, 0x57, 0x41, 0x52, 0x4d, 0x93, 0xcd, 0x08, 0x00, 0x02, 0x01,
|
||||
];
|
||||
|
||||
#[test]
|
||||
fn usage_cache_wire_format_is_pinned() {
|
||||
// Writer: the canonical map-encoded serializer must reproduce the
|
||||
// pinned bytes. Round-trip tests cannot see format drift, so any
|
||||
// encoding change (field rename/reorder, map->array switch) fails
|
||||
// here and forces re-verifying old readers and the thin projection
|
||||
// in crates/data-usage before the fixture is regenerated.
|
||||
let encoded = wire_fixture_cache().marshal_msg().expect("marshal fixture cache");
|
||||
assert_eq!(
|
||||
encoded.as_slice(),
|
||||
USAGE_CACHE_WIRE_FIXTURE,
|
||||
"persisted .usage-cache.bin encoding drifted from the pinned fixture"
|
||||
);
|
||||
|
||||
// Reader: the pinned bytes decode with every field intact.
|
||||
let decoded = DataUsageCache::unmarshal(USAGE_CACHE_WIRE_FIXTURE).expect("decode pinned fixture");
|
||||
assert_eq!(decoded.info.name, "wire-bucket");
|
||||
assert_eq!(decoded.info.next_cycle, 7);
|
||||
assert_eq!(decoded.info.leader_epoch, 9);
|
||||
assert_eq!(
|
||||
decoded.info.last_update,
|
||||
Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000))
|
||||
);
|
||||
assert!(decoded.info.skip_healing);
|
||||
assert_eq!(decoded.info.failed_objects.get("wire-bucket/lost"), Some(&11));
|
||||
assert_eq!(decoded.info.scan_resume_after.as_deref(), Some("wire-bucket/resume"));
|
||||
assert_eq!(decoded.info.pending_heals.len(), 1);
|
||||
assert_eq!(decoded.info.pending_heals[0].kind, PendingScannerHealKind::Object);
|
||||
assert_eq!(decoded.info.pending_heals[0].object.as_deref(), Some("broken"));
|
||||
assert_eq!(decoded.info.source, Some(DataUsageCacheSource::new(1, 2)));
|
||||
assert!(decoded.info.snapshot_complete);
|
||||
assert_eq!(decoded.info.scan_plan_digest, Some(TEST_PLAN_DIGEST));
|
||||
assert_eq!(decoded.info.cache_key_format, DATA_USAGE_CACHE_KEY_FORMAT);
|
||||
|
||||
let entry = decoded.cache.get("wire-bucket").expect("fixture entry decodes");
|
||||
assert_eq!(entry.size, 4096);
|
||||
assert_eq!(entry.objects, 3);
|
||||
assert_eq!(entry.versions, 5);
|
||||
assert_eq!(entry.delete_markers, 1);
|
||||
assert!(entry.compacted);
|
||||
assert_eq!(entry.failed_objects, 2);
|
||||
assert_eq!(
|
||||
entry.all_tier_stats.as_ref().and_then(|tiers| tiers.tiers.get("WARM")),
|
||||
Some(&TierStats {
|
||||
total_size: 2048,
|
||||
num_versions: 2,
|
||||
num_objects: 1,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn data_usage_cache_prepare_for_scan_rejects_unscoped_distributed_cache() {
|
||||
let mut cache = DataUsageCache {
|
||||
|
||||
@@ -13,6 +13,9 @@ for later deletion.
|
||||
## Open Items
|
||||
|
||||
- `table-publication-fence-v1` S3 Tables publication fencing: nodes that predate table and table-bucket publication fences can mutate live files while a new node is publishing a catalog pointer. New nodes retain exact object guards until the operator confirms that every serving node uses the new fences. Fleet confirmation also requires non-overlapping active warehouse prefixes and lifecycle workers that exclude table buckets. Remove the exact live-file fallback and the fleet-confirmation gate after the minimum supported RustFS release acquires table fences for registered-table mutations and table-bucket fences for unresolved-prefix mutations.
|
||||
- `table-catalog-strong-snapshot-v1` durable strong catalog snapshot compatibility: mixed-version deployments continue writing version 1 snapshots until operators confirm that every serving node reads version 2, and version 1 table/view identifier collisions remain available only for cleanup. Remove version 1 writes and collision cleanup after the minimum supported RustFS release reads version 2 and every retained durable strong snapshot is collision-free and has been upgraded to version 2.
|
||||
- `table-catalog-migration-fence-v1` durable strong migration fence compatibility: version 1 "PREPARING" fences did not distinguish a known-absent global strong snapshot from an unknown baseline, so retries read them but fail closed if the global snapshot is missing. Version 2 preserves the same JSON shape and records the pre-migration global snapshot ETag in the existing target_snapshot_etag field while the fence is "PREPARING". Remove version 1 reads after every supported direct-upgrade source writes version 2 fences and operators have completed or cancelled every older in-progress backing migration.
|
||||
- `table-catalog-backing-manifest-v1-wire-labels` durable strong backing manifest labels: version 1 published "STRONG_KV_WAL" and "CUT_OVER_LINEARIZABLE_READS" before the implementation was narrowed to the ETag-CAS durable snapshot backing. Internal names and operator documentation describe the implemented semantics, while version 1 responses retain those labels for existing clients. Replace the labels only in a new manifest version with an explicit client migration contract.
|
||||
- `cross-pool-fence-v1` authenticated unsupported advertisement: predeployment servers recognize the versioned cross-pool fence capability probe but report support version 0, allowing a later all-peer probe to distinguish predeployment nodes without activating a second lock domain. Replace the unsupported advertisement only when composite lock acquisition, a cluster-wide activation fence, complete fleet proof, commit-time proof revalidation, and fail-closed revocation ship together.
|
||||
- `table-catalog-dotted-namespace` Iceberg REST namespace path compatibility: existing RustFS clients use dotted namespace paths, while the standard multi-level contract uses the URL-encoded unit separator `%1F`. New servers accept both forms so a rolling upgrade does not invalidate existing catalog configuration. Remove the dotted fallback after the minimum supported RustFS release advertises `%1F` and all supported clients have refreshed their catalog configuration.
|
||||
- `rustfs-5509` FileInfo positional MessagePack decoding: beta.11 serialized 28 fields, while beta.12 inserted transition-version fields in the middle and serialized an incompatible 30-field array. New releases write named maps and retain readers for both shipped array layouts so direct and rolling upgrades can read either release. Remove the positional-array readers after every supported direct-upgrade release writes named maps and no retained RPC payload can contain a pre-map FileInfo array.
|
||||
|
||||
@@ -10,10 +10,28 @@ and rollback steps.
|
||||
| Area | Current owner | Size | Split status |
|
||||
|---|---|---:|---|
|
||||
| Bucket lifecycle | `crates/lifecycle/` + `crates/ecstore/src/bucket/lifecycle/` | core contracts + ECStore runtime | Core contract extracted |
|
||||
| Bucket replication | `crates/ecstore/src/bucket/replication/` | 8,730 lines | Contracts extracted; runtime move pending |
|
||||
| Bucket replication | `crates/ecstore/src/bucket/replication/` | 15,619 lines | Contracts extracted; runtime move pending |
|
||||
| Set disks | `crates/ecstore/src/set_disk/` | state carrier plus operation modules | Keep in ECStore |
|
||||
| Public ECStore facade | `crates/ecstore/src/api/mod.rs` | broad compatibility surface | Shrink only through guarded PRs |
|
||||
|
||||
Measured 2026-08-12: the whole crate is 265 files / ~288K lines (roughly half
|
||||
is inline `#[cfg(test)]` code). The largest single files are `disk/local.rs`
|
||||
(21,063 lines), `bucket/lifecycle/bucket_lifecycle_ops.rs` (11,961 lines), and
|
||||
`set_disk/mod.rs` (11,151 lines). Reproduce with:
|
||||
|
||||
```bash
|
||||
find crates/ecstore/src -name '*.rs' | xargs wc -l | sort -rn | head
|
||||
find crates/ecstore/src/bucket/replication -name '*.rs' | xargs wc -l | tail -1
|
||||
```
|
||||
|
||||
No split step has landed since the contract-extraction PRs of 2026-07-04,
|
||||
while the `bucket/replication` runtime grew from 8,730 to 15,619 lines (+79%)
|
||||
through feature work (e.g. SSE-C ciphertext passthrough replication #5898,
|
||||
delete-marker purge retry/replay #5864). To keep the gap from widening: in
|
||||
domains that already have a contract crate, new replication runtime logic that
|
||||
does not need ECStore runtime state must land in `rustfs-replication`, not in
|
||||
`crates/ecstore/src/bucket/replication/`.
|
||||
|
||||
The file split inside `set_disk/` is already operation-oriented: read, write,
|
||||
list, multipart, lock, heal, and replication code live in separate modules.
|
||||
The remaining large surface is the shared `SetDisks` state and cross-cutting
|
||||
|
||||
@@ -116,12 +116,13 @@ catalog extension.
|
||||
| Commit publication fencing | Supported with rolling-upgrade gate | Existing deployments retain exact object guards so older writers cannot mutate referenced files during publication. Set `RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED=true` only after every serving node supports table and table-bucket publication fences. In scalable mode, active table warehouse prefixes must not overlap, ordinary lifecycle expiry remains disabled for table buckets, and first enablement, first publication, drop, and warehouse relocation are serialized by the table-bucket fence. |
|
||||
| Post-CAS finalization recovery | Supported | Diagnostics and recovery can repair stale or missing idempotency indexes without changing the current table pointer. |
|
||||
| Catalog export | Supported | Exposes table state, commit recovery state, and backing migration information for operator inspection. |
|
||||
| Strong backing state transfer | Supported | Object-backed table bucket, namespace, table, view, commit-log, and idempotency state can be materialized into the durable strong snapshot. The transfer is deterministic, ETag-CAS protected, idempotent after an interrupted finalization, and fails closed when a table or view has no owning namespace entry. |
|
||||
| Durable backing migration preflight | Supported | `GET /iceberg/v1/{warehouse}/catalog/migration` and the `/_iceberg/v1` alias inspect object-backed catalog inventory, recovery blockers, warehouse prefix index readiness, persistent write-fence state, target snapshot agreement, and whether every table bucket is ready for cutover. |
|
||||
| Durable backing migration execution | Preview / controlled | `POST /iceberg/v1/{warehouse}/catalog/migration` fences table-bucket registry changes, acquires a persistent per-bucket write fence, drains in-flight catalog mutations, materializes the target snapshot, and reports `ready_to_enable_durable_strong`. `DELETE` safely releases the bucket fence only while its target state has not advanced, and releases the registry fence after the last bucket is cancelled. Both mutations require `admin:MigrateTableCatalog`. |
|
||||
| Strong backing state transfer | Supported | Object-backed table bucket, namespace, table, view, commit-log, and idempotency state can be materialized into the durable strong snapshot. The transfer is deterministic, ETag-CAS protected, idempotent after an interrupted finalization, validates candidate state through the restart decoder before publication, preserves resource-backed implicit namespaces, and fails closed when an inactive explicit namespace conflicts with active descendants or resources. Snapshot hydration requires a stable non-empty ETag, caps the encoded snapshot at 64 MiB, shares state and reload serialization across requests in one server context, and rejects disappearance or format-version rollback after observation. Configured durable-strong mode rejects a missing snapshot on its first catalog access after startup; only object-backed migration may initialize an empty target. |
|
||||
| Durable backing migration preflight | Supported | `GET /iceberg/v1/{warehouse}/catalog/migration` and the `/_iceberg/v1` alias inspect object-backed catalog inventory, recovery blockers, warehouse prefix index readiness, active table/view identifier collisions, persistent write-fence state, target snapshot agreement, and whether every table bucket is ready for cutover. |
|
||||
| Durable backing migration execution | Preview / controlled | `POST /iceberg/v1/{warehouse}/catalog/migration` fences table-bucket registry changes, acquires a persistent per-bucket write fence, records whether a global strong snapshot existed before publication, drains in-flight catalog mutations, materializes the target snapshot, and reports `ready_to_enable_durable_strong`. Retries and `DELETE` may restore a known-absent initial target after an ambiguous first write, but fail closed if a previously existing or materialized global snapshot disappears. `DELETE` releases the bucket fence only while its target state has not advanced, and releases the registry fence after the last bucket is cancelled. Both mutations require `admin:MigrateTableCatalog`. |
|
||||
| Strong snapshot rolling compatibility | Supported | Durable strong control-plane reads snapshot versions 1 and 2, writes version 1 by default, and writes version 2 only after both the requested and fleet-confirmed gates are enabled. A running process rejects any lower-format snapshot after observing a higher format. Once version 2 is fleet-confirmed, table data-plane resolution fails closed until the persisted snapshot is version 2; a missing table-bucket entry also fails closed instead of bypassing table-aware authorization. |
|
||||
| Disaster recovery rehearsal | Manual/live harness | `failure_coverage.py --print-disaster-recovery-rehearsal` generates an operator runbook covering catalog export, diagnostics, safe recovery repair, rollback/import, durable backing migration dry-run, post-recovery loadTable, and table data-plane policy probes. |
|
||||
| Scale and fault rehearsal | Manual/live harness | `failure_coverage.py --print-scale-fault-rehearsal` generates an opt-in runbook for concurrent writer stress, maintenance scheduler lease recovery, durable backing cutover preflight, recovery/rollback/import under load, and post-run evidence capture. |
|
||||
| Strong KV/WAL backing cutover | Preview / controlled | Operators can select durable strong backing with `RUSTFS_TABLE_CATALOG_BACKING=durable-strong` only after every table bucket reports `SNAPSHOT_MATERIALIZED` and `ready_to_enable_durable_strong: true`. Object-only advanced operations fail closed in durable strong mode. |
|
||||
| Durable strong snapshot backing cutover | Preview / controlled | Operators can select the ETag-CAS snapshot backing with `RUSTFS_TABLE_CATALOG_BACKING=durable-strong` only after every table bucket reports `SNAPSHOT_MATERIALIZED` and `ready_to_enable_durable_strong: true`. This mode does not claim a separate external KV/WAL service, and object-only advanced operations fail closed. Version 1 backing manifests retain the legacy `STRONG_KV_WAL` and `CUT_OVER_LINEARIZABLE_READS` wire labels for client compatibility; those labels do not expand the implementation claim. |
|
||||
| Single active writer region | Supported policy | Diagnostics publish single-active-writer semantics and read-only replica limits. |
|
||||
| Active-active multi-region writes | Not claimed | A table must not accept independent concurrent writers in multiple active regions. |
|
||||
|
||||
@@ -136,23 +137,60 @@ warehouse:
|
||||
`GetTableCatalogAction` on each table bucket. Treat every `blockers` entry as
|
||||
fail-closed; repair commit recovery state and backfill the warehouse prefix
|
||||
index before continuing.
|
||||
3. Run `POST /iceberg/v1/{warehouse}/catalog/migration` with
|
||||
`admin:MigrateTableCatalog`. This persists the source write fence before it
|
||||
drains in-flight mutations and copies the catalog state.
|
||||
4. Repeat the preflight and materialization for every table bucket. Do not set
|
||||
3. Before the migration `POST`, drain every catalog writer that predates the
|
||||
durable-backing migration fence and restart it on a fence-aware release. An
|
||||
older writer does not recognize the persisted fence and can otherwise
|
||||
mutate the object-backed source after the snapshot inventory is captured.
|
||||
Keep all catalog writers on the fence-aware release until cutover completes.
|
||||
4. Inventory object-only advanced operations, including maintenance workers,
|
||||
catalog recovery, export, diagnostics, and external catalog bridge writes.
|
||||
Quiesce mutating operations before cutover and confirm that each required
|
||||
operation is supported by durable-strong mode; unsupported operations fail
|
||||
closed after cutover rather than continuing against object-backed state.
|
||||
5. Run `POST /iceberg/v1/{warehouse}/catalog/migration` with
|
||||
`admin:MigrateTableCatalog`. This acquires the exclusive migration fence to
|
||||
drain in-flight fence-aware mutations, persists the source fence while
|
||||
exclusivity is held, and then copies the catalog state.
|
||||
6. Repeat the preflight and materialization for every table bucket. Do not set
|
||||
`RUSTFS_TABLE_CATALOG_BACKING=durable-strong` until the preflight reports
|
||||
`SNAPSHOT_MATERIALIZED`, no blockers, and
|
||||
`ready_to_enable_durable_strong: true`.
|
||||
5. Restart with durable strong backing enabled, then verify catalog config,
|
||||
7. Restart with durable strong backing enabled, then verify catalog config,
|
||||
table and view loads, commit idempotency, and table data-plane policy
|
||||
resolution before admitting writers.
|
||||
6. Before restarting into durable-strong mode, `DELETE` on the migration
|
||||
8. Before restarting into durable-strong mode, `DELETE` on the migration
|
||||
endpoint can remove a migration-created target bucket snapshot and release
|
||||
the source fence. After the durable-strong state advances, cancellation
|
||||
fails closed; recovery requires an operator-selected restore or reverse
|
||||
migration instead of restarting against the stale object-backed pointer.
|
||||
7. Preserve the object-backed catalog backup until durable strong backing has
|
||||
9. Preserve the object-backed catalog backup until durable strong backing has
|
||||
passed the operator's retention window.
|
||||
10. Keep strong snapshot writes on version 1 during a rolling binary upgrade.
|
||||
After every catalog writer can read version 2, set both
|
||||
`RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2=true` and
|
||||
`RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED=true`, then restart
|
||||
the catalog writers. Perform a controlled catalog write or migration
|
||||
materialization and confirm that the persisted snapshot is version 2 before
|
||||
serving table data-plane traffic. Setting only one gate does not change the
|
||||
write format.
|
||||
11. After any version 2 snapshot is persisted, do not roll catalog writers back
|
||||
to a binary that only reads version 1. Current binaries preserve version 2
|
||||
even when the gates are later disabled. A running process rejects restored
|
||||
version 1 content after observing version 2, but cannot distinguish an older
|
||||
snapshot with the same format version from a deliberate restore. The format
|
||||
high-water mark is process-local: restoring any older snapshot and restarting
|
||||
every writer is a privileged disaster-recovery rollback that cannot be
|
||||
inferred from the restored object alone. Recovery must restore a compatible
|
||||
binary and a snapshot selected through the operator recovery procedure.
|
||||
12. Migration preflight rejects an active table/view identifier collision before
|
||||
it writes a migration fence. A pre-existing version 1 strong snapshot with
|
||||
such a collision is loaded in cleanup-only quarantine. Ambiguous reads fail
|
||||
closed; each cleanup mutation must reduce the collision set, and unrelated
|
||||
writes remain blocked until all collisions are removed. Drain catalog
|
||||
writers that predate cleanup quarantine before starting this repair, and
|
||||
complete cleanup before the first version 2 write. Restoring any version 1
|
||||
snapshot after a writer has observed version 2 fails closed instead of
|
||||
replacing the in-process catalog state.
|
||||
|
||||
## Production Failure Coverage
|
||||
|
||||
|
||||
@@ -2067,9 +2067,12 @@ fn job_id_from_params(params: &Params<'_, '_>) -> S3Result<String> {
|
||||
fn table_catalog_backend_from_extensions(
|
||||
extensions: &http::Extensions,
|
||||
) -> S3Result<crate::table_catalog::EcStoreTableCatalogObjectBackend<ECStore>> {
|
||||
let store = runtime_sources::object_store_from_extensions(extensions)
|
||||
.ok_or_else(|| table_catalog_internal_error("request object store is not initialized"))?;
|
||||
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new(store))
|
||||
let context = runtime_sources::app_context_from_extensions(extensions)
|
||||
.ok_or_else(|| table_catalog_internal_error("request application context is not initialized"))?;
|
||||
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new_with_strong_runtime(
|
||||
context.object_store(),
|
||||
context.table_catalog_strong_runtime(),
|
||||
))
|
||||
}
|
||||
|
||||
type EcStoreObjectTableCatalogStore =
|
||||
|
||||
@@ -2512,7 +2512,7 @@ async fn commit_publication_replays_historical_standard_commit_across_backings()
|
||||
crate::table_catalog::TableCatalogBackingMode::DurableStrong,
|
||||
] {
|
||||
let metadata_backend = TestTableCatalogObjectBackend::default();
|
||||
let store = crate::table_catalog::ConfiguredTableCatalogStore::new(metadata_backend.clone(), mode);
|
||||
let store = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend.clone(), mode);
|
||||
let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace should parse");
|
||||
create_standard_events_table(&store, &metadata_backend, &namespace).await;
|
||||
let first_request = serde_json::json!({
|
||||
@@ -2559,7 +2559,7 @@ async fn commit_publication_replays_historical_standard_commit_across_backings()
|
||||
}
|
||||
}
|
||||
|
||||
let store = crate::table_catalog::ConfiguredTableCatalogStore::new(metadata_backend.clone(), mode);
|
||||
let store = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend.clone(), mode);
|
||||
let second = standard_commit_table_response(
|
||||
&store,
|
||||
&trusted_table_commit_backend(&metadata_backend),
|
||||
@@ -7313,10 +7313,10 @@ fn test_manifest_list_avro_entries_with_partition_specs(manifests: &[(&str, i32,
|
||||
"#,
|
||||
)
|
||||
.expect("manifest list avro schema should parse");
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest list writer should initialize");
|
||||
for (manifest_path, partition_spec_id, sequence_number, snapshot_id) in manifests {
|
||||
writer
|
||||
.append(apache_avro::types::Value::Record(vec![
|
||||
.append_value(apache_avro::types::Value::Record(vec![
|
||||
(
|
||||
"manifest_path".to_string(),
|
||||
apache_avro::types::Value::String((*manifest_path).to_string()),
|
||||
@@ -7368,10 +7368,10 @@ fn test_manifest_avro_bytes(files: &[(&str, i32, i32, i64, i64)]) -> Vec<u8> {
|
||||
"#,
|
||||
)
|
||||
.expect("manifest avro schema should parse");
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest writer should initialize");
|
||||
for (file_path, content, status, snapshot_id, sequence_number) in files {
|
||||
writer
|
||||
.append(apache_avro::types::Value::Record(vec![
|
||||
.append_value(apache_avro::types::Value::Record(vec![
|
||||
("status".to_string(), apache_avro::types::Value::Int(*status)),
|
||||
("snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
|
||||
("sequence_number".to_string(), apache_avro::types::Value::Long(*sequence_number)),
|
||||
@@ -7427,10 +7427,10 @@ fn test_manifest_avro_bytes_with_nullable_sequences(files: &[(&str, i32, i32, i6
|
||||
"#,
|
||||
)
|
||||
.expect("manifest avro schema should parse");
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest writer should initialize");
|
||||
for (file_path, content, status, snapshot_id, sequence_number) in files {
|
||||
writer
|
||||
.append(apache_avro::types::Value::Record(vec![
|
||||
.append_value(apache_avro::types::Value::Record(vec![
|
||||
("status".to_string(), apache_avro::types::Value::Int(*status)),
|
||||
("snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
|
||||
("sequence_number".to_string(), test_nullable_long(*sequence_number)),
|
||||
|
||||
@@ -76,6 +76,7 @@ pub struct AppContext {
|
||||
buffer_config: Arc<dyn BufferConfigInterface>,
|
||||
object_data_cache: Arc<ObjectDataCacheAdapter>,
|
||||
object_traffic_health: Arc<ObjectTrafficHealth>,
|
||||
table_catalog_strong_runtime: crate::table_catalog::StrongTableCatalogRuntime,
|
||||
}
|
||||
|
||||
impl AppContext {
|
||||
@@ -125,6 +126,7 @@ impl AppContext {
|
||||
buffer_config: default_buffer_config_interface(),
|
||||
object_data_cache,
|
||||
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
|
||||
table_catalog_strong_runtime: crate::table_catalog::StrongTableCatalogRuntime::default(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -144,6 +146,10 @@ impl AppContext {
|
||||
Arc::clone(&self.object_traffic_health)
|
||||
}
|
||||
|
||||
pub(crate) fn table_catalog_strong_runtime(&self) -> crate::table_catalog::StrongTableCatalogRuntime {
|
||||
self.table_catalog_strong_runtime.clone()
|
||||
}
|
||||
|
||||
pub fn iam(&self) -> Arc<dyn IamInterface> {
|
||||
self.iam.clone()
|
||||
}
|
||||
@@ -350,6 +356,7 @@ impl AppContext {
|
||||
buffer_config: interfaces.buffer_config,
|
||||
object_data_cache: ObjectDataCacheAdapter::disabled_arc(),
|
||||
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
|
||||
table_catalog_strong_runtime: crate::table_catalog::StrongTableCatalogRuntime::default(),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2355,7 +2355,7 @@ mod tests {
|
||||
let req = build_request(input, Method::GET);
|
||||
|
||||
let err = make_usecase().execute_list_multipart_uploads(req).await.unwrap_err();
|
||||
assert_eq!(err.code(), &S3ErrorCode::NotImplemented);
|
||||
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
|
||||
assert_eq!(err.message(), Some("Invalid key marker"));
|
||||
}
|
||||
|
||||
|
||||
@@ -1438,8 +1438,15 @@ fn table_data_plane_content_mutation(action: Action) -> bool {
|
||||
fn table_catalog_backend_for_data_plane<T>(
|
||||
req: &S3Request<T>,
|
||||
) -> S3Result<crate::table_catalog::EcStoreTableCatalogObjectBackend<ECStore>> {
|
||||
let store = request_object_store(req)?;
|
||||
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new(store))
|
||||
let context = match req.extensions.get::<Arc<ServerContextSlot>>() {
|
||||
Some(server_ctx) => server_ctx.installed_app_context(),
|
||||
None => runtime_sources::current_app_context(),
|
||||
}
|
||||
.ok_or_else(object_store_not_initialized_error)?;
|
||||
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new_with_strong_runtime(
|
||||
context.object_store(),
|
||||
context.table_catalog_strong_runtime(),
|
||||
))
|
||||
}
|
||||
|
||||
fn table_catalog_store_for_data_plane<T>(
|
||||
|
||||
@@ -39,7 +39,6 @@ use rustfs_io_metrics::internode_metrics::{
|
||||
INTERNODE_OPERATION_NS_SCANNER, INTERNODE_OPERATION_PUT_FILE_CAPABILITY, INTERNODE_OPERATION_PUT_FILE_STREAM,
|
||||
INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_OPERATION_WALK_DIR, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
|
||||
};
|
||||
use rustfs_utils::net::bytes_stream;
|
||||
use s3s::Body;
|
||||
use s3s::dto::StreamingBlob;
|
||||
use serde::de::DeserializeOwned;
|
||||
@@ -51,7 +50,7 @@ use std::pin::Pin;
|
||||
use std::sync::{Arc, LazyLock, Weak};
|
||||
use std::task::{Context, Poll};
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::io::{self, AsyncWriteExt};
|
||||
use tokio::io::{self, AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::sync::{Mutex, oneshot};
|
||||
use tokio_util::{io::ReaderStream, sync::CancellationToken};
|
||||
use tower::Service;
|
||||
@@ -663,15 +662,24 @@ where
|
||||
R: tokio::io::AsyncRead + Unpin + Send + Sync + 'static,
|
||||
{
|
||||
let metrics = runtime_sources::current_internode_metrics();
|
||||
let stream = ReaderStream::with_capacity(reader, DEFAULT_READ_BUFFER_SIZE).map_ok(move |bytes| {
|
||||
let read_buffer_size = read_file_stream_buffer_size(length);
|
||||
let read_limit = if length == 0 {
|
||||
u64::MAX
|
||||
} else {
|
||||
u64::try_from(length).unwrap_or(u64::MAX)
|
||||
};
|
||||
let stream = ReaderStream::with_capacity(reader.take(read_limit), read_buffer_size).map_ok(move |bytes| {
|
||||
metrics.record_sent_bytes_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, bytes.len());
|
||||
bytes
|
||||
});
|
||||
Box::pin(stream)
|
||||
}
|
||||
|
||||
fn read_file_stream_buffer_size(length: usize) -> usize {
|
||||
if length == 0 {
|
||||
Box::pin(stream)
|
||||
DEFAULT_READ_BUFFER_SIZE
|
||||
} else {
|
||||
Box::pin(bytes_stream(stream, length))
|
||||
length.min(DEFAULT_READ_BUFFER_SIZE)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1664,7 +1672,7 @@ fn put_file_stage_error_message(stage: &str, query: &PutFileQuery, err: &dyn std
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
DiskError, InternodeRpcService, LOG_SUBSYSTEM_DIRECTORY_WALK, LOG_SUBSYSTEM_FILE_TRANSFER,
|
||||
DEFAULT_READ_BUFFER_SIZE, DiskError, InternodeRpcService, LOG_SUBSYSTEM_DIRECTORY_WALK, LOG_SUBSYSTEM_FILE_TRANSFER,
|
||||
LOG_SUBSYSTEM_NAMESPACE_SCANNER, LOG_SUBSYSTEM_ROUTING, NS_SCANNER_BODY_SHA256_QUERY,
|
||||
NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CYCLE_QUERY, NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PATH,
|
||||
NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY, NS_SCANNER_SESSION_ID_QUERY,
|
||||
@@ -1673,10 +1681,10 @@ mod tests {
|
||||
append_walk_dir_completion, internode_http_operation, internode_rpc_subsystem, is_internode_rpc_path,
|
||||
ns_scanner_response_body, ns_scanner_server_epoch_matches, put_body_size_mismatch, put_file_auth_nonce,
|
||||
put_file_capability_response, put_file_server_epoch_matches, put_file_stage_error_message, put_file_target_lock,
|
||||
read_file_body_stream, remote_scanner_claim_rejection, response_with_disk_error, supports_walk_dir_stream_completion,
|
||||
validate_walk_dir_completion_request, verify_internode_rpc_signature, verify_ns_scanner_body_digest,
|
||||
verify_walk_dir_body_digest, walk_dir_response_body, write_authenticated_put_file, write_body_chunks_to_writer,
|
||||
write_put_file_body_chunks_to_writer,
|
||||
read_file_body_stream, read_file_stream_buffer_size, remote_scanner_claim_rejection, response_with_disk_error,
|
||||
supports_walk_dir_stream_completion, validate_walk_dir_completion_request, verify_internode_rpc_signature,
|
||||
verify_ns_scanner_body_digest, verify_walk_dir_body_digest, walk_dir_response_body, write_authenticated_put_file,
|
||||
write_body_chunks_to_writer, write_put_file_body_chunks_to_writer,
|
||||
};
|
||||
use crate::storage::storage_api::ecstore_rpc::{build_put_file_auth_trailer, gen_signature_headers};
|
||||
use crate::storage::storage_api::rpc_consumer::http_service::{DiskAPI as _, DiskOption, DiskStore, Endpoint, new_disk};
|
||||
@@ -1704,6 +1712,23 @@ mod tests {
|
||||
use tokio_stream::StreamExt;
|
||||
use tokio_stream::iter;
|
||||
|
||||
struct RejectExtraPollReader {
|
||||
emitted: bool,
|
||||
}
|
||||
|
||||
impl io::AsyncRead for RejectExtraPollReader {
|
||||
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut io::ReadBuf<'_>) -> Poll<io::Result<()>> {
|
||||
if self.emitted {
|
||||
return Poll::Ready(Err(io::Error::other("reader polled past the requested length")));
|
||||
}
|
||||
|
||||
let bytes = b"hello world";
|
||||
buf.put_slice(&bytes[..buf.remaining().min(bytes.len())]);
|
||||
self.emitted = true;
|
||||
Poll::Ready(Ok(()))
|
||||
}
|
||||
}
|
||||
|
||||
async fn new_put_file_test_disk() -> (DiskStore, tempfile::TempDir) {
|
||||
let dir = tempfile::tempdir().expect("temp directory should be created");
|
||||
let endpoint =
|
||||
@@ -2842,11 +2867,7 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn read_file_body_stream_truncates_to_requested_length() {
|
||||
let (reader, mut writer) = tokio::io::duplex(64);
|
||||
tokio::spawn(async move {
|
||||
writer.write_all(b"hello world").await.expect("write succeeds");
|
||||
});
|
||||
|
||||
let reader = RejectExtraPollReader { emitted: false };
|
||||
let mut stream = read_file_body_stream(reader, 5, INTERNODE_OPERATION_READ_FILE_STREAM);
|
||||
let mut out = Vec::new();
|
||||
while let Some(chunk) = stream.next().await {
|
||||
@@ -2856,6 +2877,18 @@ mod tests {
|
||||
assert_eq!(out, b"hello");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_file_body_stream_sizes_buffer_to_requested_length() {
|
||||
for (length, expected_capacity) in [
|
||||
(0, DEFAULT_READ_BUFFER_SIZE),
|
||||
(40 * 1024, 40 * 1024),
|
||||
(DEFAULT_READ_BUFFER_SIZE, DEFAULT_READ_BUFFER_SIZE),
|
||||
(DEFAULT_READ_BUFFER_SIZE + 1, DEFAULT_READ_BUFFER_SIZE),
|
||||
] {
|
||||
assert_eq!(read_file_stream_buffer_size(length), expected_capacity);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_file_error_response_marks_only_missing_disk_errors() {
|
||||
for (error, expected) in [
|
||||
|
||||
@@ -140,7 +140,7 @@ pub(crate) fn parse_list_multipart_uploads_params(
|
||||
if let Some(key_marker) = &key_marker
|
||||
&& !key_marker.starts_with(prefix.as_str())
|
||||
{
|
||||
return Err(S3Error::with_message(S3ErrorCode::NotImplemented, "Invalid key marker".to_string()));
|
||||
return Err(S3Error::with_message(S3ErrorCode::InvalidArgument, "Invalid key marker".to_string()));
|
||||
}
|
||||
|
||||
Ok(ListMultipartUploadsParams {
|
||||
@@ -390,7 +390,7 @@ mod tests {
|
||||
let err = parse_list_multipart_uploads_params(Some("prefix/".to_string()), Some("other/key-marker".to_string()), None)
|
||||
.expect_err("expected invalid key marker");
|
||||
|
||||
assert_eq!(*err.code(), S3ErrorCode::NotImplemented);
|
||||
assert_eq!(*err.code(), S3ErrorCode::InvalidArgument);
|
||||
assert_eq!(err.message(), Some("Invalid key marker"));
|
||||
}
|
||||
|
||||
|
||||
@@ -44,11 +44,14 @@ pub(crate) fn table_matches_staged_base(table: &TableEntry, commit_log: &CommitL
|
||||
pub(crate) struct TableCommitHistoryIndex<'a> {
|
||||
table_id: &'a str,
|
||||
reachable_states: BTreeSet<(&'a str, &'a str)>,
|
||||
ambiguous_states: BTreeSet<(&'a str, &'a str)>,
|
||||
cycle_detected: bool,
|
||||
}
|
||||
|
||||
impl<'a> TableCommitHistoryIndex<'a> {
|
||||
pub(crate) fn new(table: &'a TableEntry, commits: impl IntoIterator<Item = &'a CommitLogEntry>) -> Self {
|
||||
let mut by_new_state = BTreeMap::<(&str, &str), Option<(&str, &str)>>::new();
|
||||
let mut ambiguous_states = BTreeSet::new();
|
||||
for commit in commits
|
||||
.into_iter()
|
||||
.filter(|commit| commit.table_id == table.table_id && !matches!(commit.status, CommitLogStatus::Failed))
|
||||
@@ -57,27 +60,39 @@ impl<'a> TableCommitHistoryIndex<'a> {
|
||||
let previous = (commit.previous_metadata_location.as_str(), commit.expected_version_token.as_str());
|
||||
by_new_state
|
||||
.entry(key)
|
||||
.and_modify(|candidate| *candidate = None)
|
||||
.and_modify(|candidate| {
|
||||
*candidate = None;
|
||||
ambiguous_states.insert(key);
|
||||
})
|
||||
.or_insert(Some(previous));
|
||||
}
|
||||
|
||||
let mut reachable_states = BTreeSet::new();
|
||||
let mut state = (table.metadata_location.as_str(), table.version_token.as_str());
|
||||
while reachable_states.insert(state) {
|
||||
let cycle_detected = loop {
|
||||
if !reachable_states.insert(state) {
|
||||
break true;
|
||||
}
|
||||
let Some(Some(previous)) = by_new_state.get(&state) else {
|
||||
break;
|
||||
break false;
|
||||
};
|
||||
state = *previous;
|
||||
}
|
||||
};
|
||||
Self {
|
||||
table_id: &table.table_id,
|
||||
reachable_states,
|
||||
ambiguous_states,
|
||||
cycle_detected,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn proves_committed(&self, target: &CommitLogEntry) -> bool {
|
||||
self.table_id == target.table_id.as_str()
|
||||
!self.cycle_detected
|
||||
&& self.table_id == target.table_id.as_str()
|
||||
&& !matches!(target.status, CommitLogStatus::Failed)
|
||||
&& !self
|
||||
.ambiguous_states
|
||||
.contains(&(target.new_metadata_location.as_str(), target.new_version_token.as_str()))
|
||||
&& self
|
||||
.reachable_states
|
||||
.contains(&(target.new_metadata_location.as_str(), target.new_version_token.as_str()))
|
||||
@@ -202,7 +217,7 @@ pub(crate) fn table_commit_recovery_entry(
|
||||
TableCommitRecoveryState::FinalizationRequired,
|
||||
"a later committed pointer proves this staged commit is part of table history".to_string(),
|
||||
)
|
||||
} else if matches!(commit_log.status, CommitLogStatus::Committed) {
|
||||
} else if matches!(commit_log.status, CommitLogStatus::Committed) && historically_committed {
|
||||
if idempotency_index_repair_required {
|
||||
(
|
||||
TableCommitRecoveryState::IdempotencyIndexRepairRequired,
|
||||
@@ -214,6 +229,11 @@ pub(crate) fn table_commit_recovery_entry(
|
||||
"commit is finalized and may be older than the current table pointer".to_string(),
|
||||
)
|
||||
}
|
||||
} else if matches!(commit_log.status, CommitLogStatus::Committed) {
|
||||
(
|
||||
TableCommitRecoveryState::ManualReview,
|
||||
"committed log is not reachable from the current table pointer".to_string(),
|
||||
)
|
||||
} else if table_matches_staged_base(table, commit_log) {
|
||||
(
|
||||
TableCommitRecoveryState::StagedBeforeTableUpdate,
|
||||
|
||||
@@ -286,17 +286,15 @@ where
|
||||
if table.state != TableCatalogEntryState::Active {
|
||||
continue;
|
||||
}
|
||||
let Ok(warehouse_object_prefix) = table_warehouse_object_prefix(&table) else {
|
||||
continue;
|
||||
};
|
||||
let warehouse_object_prefix = table_warehouse_object_prefix(&table)?;
|
||||
if !object.starts_with(&warehouse_object_prefix) {
|
||||
continue;
|
||||
}
|
||||
if matched
|
||||
.as_ref()
|
||||
.is_some_and(|current| current.warehouse_object_prefix.len() >= warehouse_object_prefix.len())
|
||||
{
|
||||
continue;
|
||||
if let Some(current) = matched.as_ref() {
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"object {object} matches overlapping active table warehouse prefixes {} and {warehouse_object_prefix}",
|
||||
current.warehouse_object_prefix
|
||||
)));
|
||||
}
|
||||
matched = Some(table_data_plane_resource_from_entry(table, warehouse_object_prefix));
|
||||
}
|
||||
|
||||
@@ -921,9 +921,11 @@ pub(crate) fn compacted_manifest_list_avro_bytes(summary: CompactionManifestList
|
||||
"#,
|
||||
)
|
||||
.map_err(|err| TableCatalogStoreError::Internal(format!("failed to build compaction manifest list schema: {err}")))?;
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).map_err(|err| {
|
||||
TableCatalogStoreError::Internal(format!("failed to initialize compaction manifest list writer: {err}"))
|
||||
})?;
|
||||
writer
|
||||
.append(apache_avro::types::Value::Record(vec![
|
||||
.append_value(apache_avro::types::Value::Record(vec![
|
||||
(
|
||||
"manifest_path".to_string(),
|
||||
apache_avro::types::Value::String(summary.manifest_path.to_string()),
|
||||
@@ -1095,14 +1097,15 @@ fn compaction_partition_field_schema(value: &apache_avro::types::Value) -> Optio
|
||||
|
||||
pub(crate) fn compacted_manifest_avro_bytes(data_files: &[CompactedDataFile]) -> TableCatalogStoreResult<Vec<u8>> {
|
||||
let schema = compacted_manifest_avro_schema(data_files)?;
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new())
|
||||
.map_err(|err| TableCatalogStoreError::Internal(format!("failed to initialize compaction manifest writer: {err}")))?;
|
||||
for data_file in data_files {
|
||||
let sort_order_id = match data_file.sort_order_id {
|
||||
Some(sort_order_id) => apache_avro::types::Value::Union(1, Box::new(apache_avro::types::Value::Int(sort_order_id))),
|
||||
None => apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
|
||||
};
|
||||
writer
|
||||
.append(apache_avro::types::Value::Record(vec![
|
||||
.append_value(apache_avro::types::Value::Record(vec![
|
||||
("status".to_string(), apache_avro::types::Value::Int(data_file.status)),
|
||||
("snapshot_id".to_string(), apache_avro::types::Value::Long(data_file.snapshot_id)),
|
||||
("sequence_number".to_string(), apache_avro::types::Value::Long(data_file.sequence_number)),
|
||||
|
||||
@@ -97,6 +97,9 @@ pub(crate) const TABLE_CATALOG_BACKING_MANIFEST_VERSION: u16 = 1;
|
||||
pub(crate) const ENV_TABLE_CATALOG_BACKING: &str = "RUSTFS_TABLE_CATALOG_BACKING";
|
||||
pub(crate) const ENV_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED: &str =
|
||||
"RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED";
|
||||
pub(crate) const ENV_TABLE_CATALOG_STRONG_SNAPSHOT_V2: &str = "RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2";
|
||||
pub(crate) const ENV_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED: &str =
|
||||
"RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED";
|
||||
pub(crate) const TABLE_CATALOG_BACKING_OBJECT: &str = "object";
|
||||
pub(crate) const TABLE_CATALOG_BACKING_DURABLE_STRONG: &str = "durable-strong";
|
||||
pub(crate) const TABLE_METADATA_DIGEST_REQUIREMENT_TYPE: &str = "assert-rustfs-metadata-sha256";
|
||||
@@ -159,10 +162,12 @@ const ICEBERG_MAX_REF_AGE_MS_PROPERTY: &str = "history.expire.max-ref-age-ms";
|
||||
const ICEBERG_REF_MIN_SNAPSHOTS_TO_KEEP_FIELD: &str = "min-snapshots-to-keep";
|
||||
const ICEBERG_REF_MAX_SNAPSHOT_AGE_MS_FIELD: &str = "max-snapshot-age-ms";
|
||||
const ICEBERG_REF_MAX_REF_AGE_MS_FIELD: &str = "max-ref-age-ms";
|
||||
const STRONG_TABLE_CATALOG_SNAPSHOT_VERSION: u16 = 1;
|
||||
const STRONG_TABLE_CATALOG_SNAPSHOT_MIN_READ_VERSION: u16 = 1;
|
||||
const STRONG_TABLE_CATALOG_SNAPSHOT_VERSION: u16 = 2;
|
||||
const STRONG_TABLE_CATALOG_BACKING_ROOT: &str = "strong-backing";
|
||||
const STRONG_TABLE_CATALOG_SNAPSHOT_FILE: &str = "snapshot.json";
|
||||
const TABLE_CATALOG_MIGRATION_VERSION: u16 = 1;
|
||||
const TABLE_CATALOG_MIGRATION_MIN_READ_VERSION: u16 = 1;
|
||||
const TABLE_CATALOG_MIGRATION_VERSION: u16 = 2;
|
||||
const TABLE_CATALOG_MIGRATION_ROOT: &str = "backing-migration";
|
||||
const TABLE_CATALOG_MIGRATION_FENCE_FILE: &str = "durable-strong-fence.json";
|
||||
const TABLE_CATALOG_MIGRATION_FENCE_LOCK: &str = "durable-strong-fence.lock";
|
||||
|
||||
@@ -1067,7 +1067,9 @@ pub(crate) struct TableCatalogBackingProfile {
|
||||
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
|
||||
pub(crate) enum TableCatalogBackingKind {
|
||||
ObjectBacked,
|
||||
StrongKvWal,
|
||||
// RUSTFS_COMPAT_TODO(table-catalog-backing-manifest-v1-wire-labels): Keep the version 1 wire label for existing clients. Remove after a versioned manifest with an explicit client migration contract replaces it.
|
||||
#[serde(rename = "STRONG_KV_WAL")]
|
||||
DurableStrongSnapshot,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||
@@ -1203,7 +1205,9 @@ pub(crate) enum TableCatalogBackingMigrationStep {
|
||||
ReplayCommitLog,
|
||||
VerifyCurrentPointer,
|
||||
EnableSingleWriterFencing,
|
||||
CutOverLinearizableReads,
|
||||
// RUSTFS_COMPAT_TODO(table-catalog-backing-manifest-v1-wire-labels): Keep the version 1 wire label for existing clients. Remove after a versioned manifest with an explicit client migration contract replaces it.
|
||||
#[serde(rename = "CUT_OVER_LINEARIZABLE_READS")]
|
||||
CutOverDurableSnapshotReads,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||
@@ -1213,6 +1217,8 @@ pub(crate) enum TableCatalogBackingMigrationBlocker {
|
||||
CommitManualReviewRequired,
|
||||
WarehouseIndexBackfillRequired,
|
||||
DuplicateWarehousePrefix,
|
||||
DuplicateTableIdentity,
|
||||
TableViewIdentifierCollision,
|
||||
DurableStrongSnapshotChanged,
|
||||
}
|
||||
|
||||
@@ -1222,6 +1228,8 @@ pub(crate) enum TableCatalogBackingMigrationAction {
|
||||
RunCatalogRecovery,
|
||||
BackfillWarehouseIndex,
|
||||
ReviewDuplicateWarehousePrefixes,
|
||||
ReviewDuplicateTableIdentities,
|
||||
ReviewTableViewIdentifierCollisions,
|
||||
SnapshotObjectBackedCatalog,
|
||||
EnableDurableStrongBacking,
|
||||
VerifyDurableStrongSnapshot,
|
||||
|
||||
@@ -13,11 +13,13 @@
|
||||
// limitations under the License.
|
||||
|
||||
use super::object::{
|
||||
ObjectTableCatalogStore, validate_namespace_entry_object, validate_table_entry_object, validate_view_entry_object,
|
||||
ObjectTableCatalogStore, validate_commit_idempotency_entry_object, validate_commit_log_entry_object,
|
||||
validate_namespace_entry_object, validate_table_bucket_entry_object, validate_table_entry_object, validate_view_entry_object,
|
||||
};
|
||||
use super::strong::{
|
||||
StrongCommitSnapshotRecord, StrongTableCatalogBucketSnapshot, StrongTableCatalogState, TableCatalogBackingMigrationFence,
|
||||
TableCatalogBackingMigrationFenceStatus, TableCatalogBackingMigrationGlobalFence, table_catalog_bucket_snapshot_fingerprint,
|
||||
TableCatalogBackingMigrationFenceStatus, TableCatalogBackingMigrationGlobalFence,
|
||||
TableCatalogBackingMigrationTargetSnapshotState, table_catalog_bucket_snapshot_fingerprint,
|
||||
};
|
||||
use super::*;
|
||||
|
||||
@@ -82,14 +84,14 @@ pub(super) fn table_catalog_backing_manifest(
|
||||
},
|
||||
migration: TableCatalogBackingMigrationPlan {
|
||||
source_kind: TableCatalogBackingKind::ObjectBacked,
|
||||
target_kind: TableCatalogBackingKind::StrongKvWal,
|
||||
target_kind: TableCatalogBackingKind::DurableStrongSnapshot,
|
||||
status: migration_status,
|
||||
required_steps: vec![
|
||||
TableCatalogBackingMigrationStep::SnapshotCatalogExport,
|
||||
TableCatalogBackingMigrationStep::ReplayCommitLog,
|
||||
TableCatalogBackingMigrationStep::VerifyCurrentPointer,
|
||||
TableCatalogBackingMigrationStep::EnableSingleWriterFencing,
|
||||
TableCatalogBackingMigrationStep::CutOverLinearizableReads,
|
||||
TableCatalogBackingMigrationStep::CutOverDurableSnapshotReads,
|
||||
],
|
||||
blockers,
|
||||
},
|
||||
@@ -118,12 +120,100 @@ impl<B> ObjectTableCatalogStore<B>
|
||||
where
|
||||
B: TableCatalogObjectBackend,
|
||||
{
|
||||
fn migration_target_snapshot_state(
|
||||
fence: &TableCatalogBackingMigrationFence,
|
||||
) -> TableCatalogBackingMigrationTargetSnapshotState {
|
||||
// RUSTFS_COMPAT_TODO(table-catalog-migration-fence-v1): Version 1 PREPARING fences have no durable baseline. Remove after all supported upgrade sources write version 2 fences and all version 1 migrations are completed or cancelled.
|
||||
if fence.version < TABLE_CATALOG_MIGRATION_VERSION {
|
||||
TableCatalogBackingMigrationTargetSnapshotState::Unknown
|
||||
} else if fence.target_snapshot_etag.is_some() {
|
||||
TableCatalogBackingMigrationTargetSnapshotState::Present
|
||||
} else {
|
||||
TableCatalogBackingMigrationTargetSnapshotState::Absent
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_backing_migration_fence(
|
||||
table_bucket: &str,
|
||||
fence: &TableCatalogBackingMigrationFence,
|
||||
) -> TableCatalogStoreResult<()> {
|
||||
if !(TABLE_CATALOG_MIGRATION_MIN_READ_VERSION..=TABLE_CATALOG_MIGRATION_VERSION).contains(&fence.version)
|
||||
|| fence.table_bucket != table_bucket
|
||||
|| fence.migration_id.is_empty()
|
||||
{
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"invalid durable strong migration fence for table bucket {table_bucket}"
|
||||
)));
|
||||
}
|
||||
let target_snapshot_state = Self::migration_target_snapshot_state(fence);
|
||||
if fence.target_bucket_existed && target_snapshot_state == TableCatalogBackingMigrationTargetSnapshotState::Absent {
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"durable strong migration fence for table bucket {table_bucket} has an inconsistent target snapshot baseline"
|
||||
)));
|
||||
}
|
||||
match fence.status {
|
||||
TableCatalogBackingMigrationFenceStatus::Preparing if fence.source_fingerprint.is_some() => {
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"preparing durable strong migration fence for table bucket {table_bucket} has materialized state"
|
||||
)));
|
||||
}
|
||||
TableCatalogBackingMigrationFenceStatus::Materialized
|
||||
if fence.source_fingerprint.is_none() || fence.target_snapshot_etag.is_none() =>
|
||||
{
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"materialized durable strong migration fence for table bucket {table_bucket} is incomplete"
|
||||
)));
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_global_backing_migration_fence(fence: &TableCatalogBackingMigrationGlobalFence) -> TableCatalogStoreResult<()> {
|
||||
if !(TABLE_CATALOG_MIGRATION_MIN_READ_VERSION..=TABLE_CATALOG_MIGRATION_VERSION).contains(&fence.version)
|
||||
|| fence.migration_id.is_empty()
|
||||
{
|
||||
return Err(TableCatalogStoreError::Invalid(
|
||||
"invalid durable strong global migration fence".to_string(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn observe_durable_strong_migration_target(
|
||||
strong_store: &StrongTableCatalogStore<B>,
|
||||
table_bucket: &str,
|
||||
fence: Option<&TableCatalogBackingMigrationFence>,
|
||||
) -> TableCatalogStoreResult<(Option<String>, Option<String>)> {
|
||||
let target_snapshot_state = fence.map(Self::migration_target_snapshot_state);
|
||||
let permits_absent_snapshot = fence.is_some_and(|fence| {
|
||||
fence.status == TableCatalogBackingMigrationFenceStatus::Preparing
|
||||
&& !fence.target_bucket_existed
|
||||
&& target_snapshot_state == Some(TableCatalogBackingMigrationTargetSnapshotState::Absent)
|
||||
});
|
||||
if permits_absent_snapshot {
|
||||
strong_store.restore_absent_migration_snapshot_baseline().await?;
|
||||
}
|
||||
let observation = strong_store.bucket_snapshot_observation(table_bucket).await?;
|
||||
if fence.is_some() && observation.1.is_none() && !permits_absent_snapshot {
|
||||
return Err(TableCatalogStoreError::Conflict(
|
||||
"durable strong catalog snapshot is missing for an in-progress backing migration".to_string(),
|
||||
));
|
||||
}
|
||||
Ok(observation)
|
||||
}
|
||||
|
||||
async fn read_backing_migration_fence(
|
||||
&self,
|
||||
table_bucket: &str,
|
||||
) -> TableCatalogStoreResult<Option<(TableCatalogBackingMigrationFence, Option<String>)>> {
|
||||
self.read_entry(self.catalog_bucket(), &self.paths.backing_migration_fence_path(table_bucket))
|
||||
.await
|
||||
let fence = self
|
||||
.read_entry(self.catalog_bucket(), &self.paths.backing_migration_fence_path(table_bucket))
|
||||
.await?;
|
||||
if let Some((fence, _)) = fence.as_ref() {
|
||||
Self::validate_backing_migration_fence(table_bucket, fence)?;
|
||||
}
|
||||
Ok(fence)
|
||||
}
|
||||
|
||||
pub(super) async fn acquire_table_bucket_registry_write_permit(&self) -> TableCatalogStoreResult<Box<dyn Send>> {
|
||||
@@ -164,11 +254,7 @@ where
|
||||
.read_entry::<TableCatalogBackingMigrationGlobalFence>(self.catalog_bucket(), fence_path)
|
||||
.await?
|
||||
{
|
||||
if fence.version != TABLE_CATALOG_MIGRATION_VERSION {
|
||||
return Err(TableCatalogStoreError::Invalid(
|
||||
"invalid durable strong global migration fence".to_string(),
|
||||
));
|
||||
}
|
||||
Self::validate_global_backing_migration_fence(&fence)?;
|
||||
return Ok(fence);
|
||||
}
|
||||
let fence = TableCatalogBackingMigrationGlobalFence {
|
||||
@@ -181,6 +267,13 @@ where
|
||||
}
|
||||
|
||||
async fn clear_global_backing_migration_fence_if_unused(&self, fence_path: &str) -> TableCatalogStoreResult<()> {
|
||||
let Some((fence, _)) = self
|
||||
.read_entry::<TableCatalogBackingMigrationGlobalFence>(self.catalog_bucket(), fence_path)
|
||||
.await?
|
||||
else {
|
||||
return Ok(());
|
||||
};
|
||||
Self::validate_global_backing_migration_fence(&fence)?;
|
||||
let bucket_objects = self
|
||||
.backend
|
||||
.list_objects(self.catalog_bucket(), &self.paths.table_bucket_entries_prefix())
|
||||
@@ -194,19 +287,6 @@ where
|
||||
self.backend.delete_object(self.catalog_bucket(), fence_path).await
|
||||
}
|
||||
|
||||
pub(super) async fn ensure_object_backed_writes_allowed(&self, table_bucket: &str) -> TableCatalogStoreResult<()> {
|
||||
if self
|
||||
.backend
|
||||
.object_exists(self.catalog_bucket(), &self.paths.backing_migration_fence_path(table_bucket))
|
||||
.await?
|
||||
{
|
||||
return Err(TableCatalogStoreError::Conflict(format!(
|
||||
"object-backed catalog writes are fenced while table bucket {table_bucket} is prepared for durable strong cutover"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn collect_bucket_snapshot_with_locks(
|
||||
&self,
|
||||
table_bucket: &str,
|
||||
@@ -220,11 +300,7 @@ where
|
||||
else {
|
||||
return Err(TableCatalogStoreError::NotFound(format!("table bucket {table_bucket}")));
|
||||
};
|
||||
if table_bucket_entry.table_bucket != table_bucket {
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"table bucket entry does not match migration target {table_bucket}"
|
||||
)));
|
||||
}
|
||||
validate_table_bucket_entry_object(&self.paths, &bucket_path, &table_bucket_entry)?;
|
||||
|
||||
let mut namespaces = Vec::new();
|
||||
let mut tables = Vec::new();
|
||||
@@ -286,6 +362,7 @@ where
|
||||
"commit log changed while preparing durable strong snapshot: {commit_object}"
|
||||
)));
|
||||
};
|
||||
validate_commit_log_entry_object(&self.paths, &commit_object, table_bucket, &table_entry.table_id, &commit)?;
|
||||
commits.push(StrongCommitSnapshotRecord {
|
||||
table_bucket: table_bucket.to_string(),
|
||||
table_id: table_entry.table_id.clone(),
|
||||
@@ -313,6 +390,13 @@ where
|
||||
"idempotency index changed while preparing durable strong snapshot: {idempotency_object}"
|
||||
)));
|
||||
};
|
||||
validate_commit_idempotency_entry_object(
|
||||
&self.paths,
|
||||
&idempotency_object,
|
||||
table_bucket,
|
||||
&table_entry.table_id,
|
||||
&commit,
|
||||
)?;
|
||||
let lookup_key = commit.idempotency_key.clone().ok_or_else(|| {
|
||||
TableCatalogStoreError::Invalid(format!("idempotency index {idempotency_object} has no idempotency key"))
|
||||
})?;
|
||||
@@ -360,6 +444,22 @@ where
|
||||
|
||||
fn validate_bucket_snapshot_for_migration(&self, snapshot: &StrongTableCatalogBucketSnapshot) -> TableCatalogStoreResult<()> {
|
||||
let table_bucket = &snapshot.table_bucket.table_bucket;
|
||||
let active_table_identifiers = snapshot
|
||||
.tables
|
||||
.iter()
|
||||
.filter(|table| table.state == TableCatalogEntryState::Active)
|
||||
.map(|table| (&table.namespace, &table.table))
|
||||
.collect::<BTreeSet<_>>();
|
||||
if snapshot
|
||||
.views
|
||||
.iter()
|
||||
.filter(|view| view.state == TableCatalogEntryState::Active)
|
||||
.any(|view| active_table_identifiers.contains(&(&view.namespace, &view.view)))
|
||||
{
|
||||
return Err(TableCatalogStoreError::Conflict(format!(
|
||||
"table bucket {table_bucket} contains an active table/view identifier collision"
|
||||
)));
|
||||
}
|
||||
let tables_by_id = snapshot
|
||||
.tables
|
||||
.iter()
|
||||
@@ -498,6 +598,15 @@ where
|
||||
if self.get_table_bucket(table_bucket).await?.is_none() {
|
||||
return Err(TableCatalogStoreError::NotFound(format!("table bucket {table_bucket}")));
|
||||
}
|
||||
if let Some((global_fence, _)) = self
|
||||
.read_entry::<TableCatalogBackingMigrationGlobalFence>(
|
||||
self.catalog_bucket(),
|
||||
&self.paths.backing_migration_global_fence_path(),
|
||||
)
|
||||
.await?
|
||||
{
|
||||
Self::validate_global_backing_migration_fence(&global_fence)?;
|
||||
}
|
||||
|
||||
let namespace_objects = self
|
||||
.backend
|
||||
@@ -511,6 +620,10 @@ where
|
||||
let mut recovery_required_count: usize = 0;
|
||||
let mut manual_review_count: usize = 0;
|
||||
let mut warehouse_prefix_owners = BTreeMap::<String, usize>::new();
|
||||
let mut table_ids = BTreeSet::<String>::new();
|
||||
let mut duplicate_table_identity = false;
|
||||
let mut active_table_identifiers = BTreeSet::<(String, String)>::new();
|
||||
let mut active_view_identifiers = BTreeSet::<(String, String)>::new();
|
||||
|
||||
for object in namespace_objects {
|
||||
if object.ends_with(NAMESPACE_ENTRY_FILE) {
|
||||
@@ -526,6 +639,9 @@ where
|
||||
continue;
|
||||
};
|
||||
validate_view_entry_object(&self.paths, &object, &entry)?;
|
||||
if entry.state == TableCatalogEntryState::Active {
|
||||
active_view_identifiers.insert((entry.namespace.clone(), entry.view.clone()));
|
||||
}
|
||||
view_count = view_count.saturating_add(1);
|
||||
continue;
|
||||
}
|
||||
@@ -538,7 +654,11 @@ where
|
||||
};
|
||||
validate_table_entry_object(&self.paths, &object, &table)?;
|
||||
table_count = table_count.saturating_add(1);
|
||||
if !table_ids.insert(table.table_id.clone()) {
|
||||
duplicate_table_identity = true;
|
||||
}
|
||||
if table.state == TableCatalogEntryState::Active {
|
||||
active_table_identifiers.insert((table.namespace.clone(), table.table.clone()));
|
||||
let warehouse_prefix = table_warehouse_object_prefix(&table)?;
|
||||
warehouse_prefix_owners
|
||||
.entry(warehouse_prefix)
|
||||
@@ -548,17 +668,31 @@ where
|
||||
|
||||
let recovery = self.table_commit_recovery_report_for_entry(&table, 0).await?;
|
||||
commit_log_count = commit_log_count.saturating_add(recovery.commits.len());
|
||||
idempotency_index_count = idempotency_index_count.saturating_add(
|
||||
self.backend
|
||||
.list_objects(
|
||||
self.catalog_bucket(),
|
||||
&self.paths.commit_idempotency_entries_prefix(table_bucket, &table.table_id),
|
||||
)
|
||||
for idempotency_object in self
|
||||
.backend
|
||||
.list_objects(
|
||||
self.catalog_bucket(),
|
||||
&self.paths.commit_idempotency_entries_prefix(table_bucket, &table.table_id),
|
||||
)
|
||||
.await?
|
||||
.into_iter()
|
||||
.filter(|object| object.ends_with(".json"))
|
||||
{
|
||||
let Some((commit, _)) = self
|
||||
.read_entry::<CommitLogEntry>(self.catalog_bucket(), &idempotency_object)
|
||||
.await?
|
||||
.into_iter()
|
||||
.filter(|object| object.ends_with(".json"))
|
||||
.count(),
|
||||
);
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
validate_commit_idempotency_entry_object(
|
||||
&self.paths,
|
||||
&idempotency_object,
|
||||
table_bucket,
|
||||
&table.table_id,
|
||||
&commit,
|
||||
)?;
|
||||
idempotency_index_count = idempotency_index_count.saturating_add(1);
|
||||
}
|
||||
recovery_required_count = recovery_required_count
|
||||
.saturating_add(recovery.staged_before_table_update_count)
|
||||
.saturating_add(recovery.finalization_required_count)
|
||||
@@ -568,6 +702,13 @@ where
|
||||
|
||||
let warehouse_index_ready = self.warehouse_index_ready(table_bucket).await?;
|
||||
let duplicate_warehouse_prefix_count = warehouse_prefix_owners.values().filter(|count| **count > 1).count();
|
||||
let overlapping_warehouse_prefix = warehouse_prefix_owners
|
||||
.keys()
|
||||
.collect::<Vec<_>>()
|
||||
.windows(2)
|
||||
.any(|window| warehouse_object_prefixes_overlap(window[0], window[1]));
|
||||
let conflicting_warehouse_prefix = duplicate_warehouse_prefix_count > 0 || overlapping_warehouse_prefix;
|
||||
let table_view_identifier_collision_count = active_table_identifiers.intersection(&active_view_identifiers).count();
|
||||
let mut blockers = Vec::new();
|
||||
let mut recommended_actions = Vec::new();
|
||||
if recovery_required_count > 0 {
|
||||
@@ -583,12 +724,24 @@ where
|
||||
blockers.push(TableCatalogBackingMigrationBlocker::WarehouseIndexBackfillRequired);
|
||||
recommended_actions.push(TableCatalogBackingMigrationAction::BackfillWarehouseIndex);
|
||||
}
|
||||
if duplicate_warehouse_prefix_count > 0 {
|
||||
if conflicting_warehouse_prefix {
|
||||
blockers.push(TableCatalogBackingMigrationBlocker::DuplicateWarehousePrefix);
|
||||
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewDuplicateWarehousePrefixes);
|
||||
}
|
||||
if duplicate_table_identity {
|
||||
blockers.push(TableCatalogBackingMigrationBlocker::DuplicateTableIdentity);
|
||||
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewDuplicateTableIdentities);
|
||||
}
|
||||
if table_view_identifier_collision_count > 0 {
|
||||
blockers.push(TableCatalogBackingMigrationBlocker::TableViewIdentifierCollision);
|
||||
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewTableViewIdentifierCollisions);
|
||||
}
|
||||
|
||||
let mut status = if manual_review_count > 0 || duplicate_warehouse_prefix_count > 0 {
|
||||
let mut status = if manual_review_count > 0
|
||||
|| conflicting_warehouse_prefix
|
||||
|| duplicate_table_identity
|
||||
|| table_view_identifier_collision_count > 0
|
||||
{
|
||||
TableCatalogBackingMigrationStatus::ManualReviewRequired
|
||||
} else if recovery_required_count > 0 || !warehouse_index_ready {
|
||||
TableCatalogBackingMigrationStatus::RecoveryRequired
|
||||
@@ -597,6 +750,14 @@ where
|
||||
};
|
||||
|
||||
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
||||
let source_table_buckets = self.object_backed_table_buckets().await?;
|
||||
let source_table_bucket_names = source_table_buckets.keys().cloned().collect::<BTreeSet<_>>();
|
||||
let target_table_buckets = strong_store.table_bucket_names().await?;
|
||||
if !target_table_buckets.is_subset(&source_table_bucket_names) {
|
||||
status = TableCatalogBackingMigrationStatus::ManualReviewRequired;
|
||||
blockers.push(TableCatalogBackingMigrationBlocker::DurableStrongSnapshotChanged);
|
||||
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewDurableStrongSnapshot);
|
||||
}
|
||||
let migration_fence = self.read_backing_migration_fence(table_bucket).await?.map(|(fence, _)| fence);
|
||||
let object_backed_writes_fenced = migration_fence.is_some();
|
||||
if status == TableCatalogBackingMigrationStatus::ReadyToSnapshot
|
||||
@@ -633,7 +794,7 @@ where
|
||||
Ok(TableCatalogBackingMigrationDryRunReport {
|
||||
table_bucket: table_bucket.to_string(),
|
||||
source_kind: TableCatalogBackingKind::ObjectBacked,
|
||||
target_kind: TableCatalogBackingKind::StrongKvWal,
|
||||
target_kind: TableCatalogBackingKind::DurableStrongSnapshot,
|
||||
status,
|
||||
namespace_count,
|
||||
table_count,
|
||||
@@ -684,11 +845,17 @@ where
|
||||
let existing_fence = self.read_backing_migration_fence(table_bucket).await?;
|
||||
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
||||
if let Some((fence, _)) = existing_fence.as_ref()
|
||||
&& (fence.version != TABLE_CATALOG_MIGRATION_VERSION || fence.table_bucket != table_bucket)
|
||||
&& fence.status == TableCatalogBackingMigrationFenceStatus::Preparing
|
||||
&& !fence.target_bucket_existed
|
||||
&& Self::migration_target_snapshot_state(fence) == TableCatalogBackingMigrationTargetSnapshotState::Absent
|
||||
{
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"invalid durable strong migration fence for table bucket {table_bucket}"
|
||||
)));
|
||||
strong_store.restore_absent_migration_snapshot_baseline().await?;
|
||||
}
|
||||
let source_table_bucket_names = self.object_backed_table_buckets().await?.into_keys().collect::<BTreeSet<_>>();
|
||||
if !strong_store.table_bucket_names().await?.is_subset(&source_table_bucket_names) {
|
||||
return Err(TableCatalogStoreError::Conflict(
|
||||
"durable strong snapshot contains table buckets outside the object-backed catalog inventory".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
if !self.warehouse_index_ready(table_bucket).await? {
|
||||
@@ -712,10 +879,16 @@ where
|
||||
}
|
||||
|
||||
self.ensure_global_backing_migration_fence(&global_fence_path).await?;
|
||||
let (target_fingerprint, target_snapshot_etag) = Self::observe_durable_strong_migration_target(
|
||||
&strong_store,
|
||||
table_bucket,
|
||||
existing_fence.as_ref().map(|(fence, _)| fence),
|
||||
)
|
||||
.await?;
|
||||
let (migration_id, target_bucket_existed) = if let Some((fence, _)) = existing_fence.as_ref() {
|
||||
(fence.migration_id.clone(), fence.target_bucket_existed)
|
||||
} else {
|
||||
let target_bucket_existed = strong_store.bucket_snapshot_fingerprint(table_bucket).await?.is_some();
|
||||
let target_bucket_existed = target_fingerprint.is_some();
|
||||
let fence = TableCatalogBackingMigrationFence {
|
||||
version: TABLE_CATALOG_MIGRATION_VERSION,
|
||||
table_bucket: table_bucket.to_string(),
|
||||
@@ -723,7 +896,7 @@ where
|
||||
status: TableCatalogBackingMigrationFenceStatus::Preparing,
|
||||
target_bucket_existed,
|
||||
source_fingerprint: None,
|
||||
target_snapshot_etag: None,
|
||||
target_snapshot_etag,
|
||||
};
|
||||
self.write_entry(self.catalog_bucket(), &fence_path, &fence, TableCatalogPutPrecondition::IfAbsent)
|
||||
.await?;
|
||||
@@ -749,7 +922,7 @@ where
|
||||
Ok(TableCatalogBackingMigrationExecutionReport {
|
||||
table_bucket: table_bucket.to_string(),
|
||||
source_kind: TableCatalogBackingKind::ObjectBacked,
|
||||
target_kind: TableCatalogBackingKind::StrongKvWal,
|
||||
target_kind: TableCatalogBackingKind::DurableStrongSnapshot,
|
||||
status: if created {
|
||||
TableCatalogBackingMigrationExecutionStatus::SnapshotMaterialized
|
||||
} else {
|
||||
@@ -793,11 +966,6 @@ where
|
||||
});
|
||||
};
|
||||
self.ensure_global_backing_migration_fence(&global_fence_path).await?;
|
||||
if fence.version != TABLE_CATALOG_MIGRATION_VERSION || fence.table_bucket != table_bucket {
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"invalid durable strong migration fence for table bucket {table_bucket}"
|
||||
)));
|
||||
}
|
||||
|
||||
let mut source_guards = Vec::new();
|
||||
let source = self
|
||||
@@ -813,12 +981,21 @@ where
|
||||
}
|
||||
|
||||
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
||||
if fence.status == TableCatalogBackingMigrationFenceStatus::Materialized
|
||||
&& strong_store.bucket_snapshot_fingerprint(table_bucket).await?.as_deref() != Some(&source_fingerprint)
|
||||
{
|
||||
return Err(TableCatalogStoreError::Conflict(format!(
|
||||
"durable strong catalog state changed after materializing table bucket {table_bucket}"
|
||||
)));
|
||||
let (target_fingerprint, target_snapshot_etag) =
|
||||
Self::observe_durable_strong_migration_target(&strong_store, table_bucket, Some(&fence)).await?;
|
||||
if fence.status == TableCatalogBackingMigrationFenceStatus::Materialized {
|
||||
let target_matches_source = target_fingerprint.as_deref() == Some(&source_fingerprint);
|
||||
let target_was_already_removed = !fence.target_bucket_existed && target_fingerprint.is_none();
|
||||
if !target_matches_source && !target_was_already_removed {
|
||||
return Err(TableCatalogStoreError::Conflict(format!(
|
||||
"durable strong catalog state changed after materializing table bucket {table_bucket}"
|
||||
)));
|
||||
}
|
||||
if target_matches_source && target_snapshot_etag != fence.target_snapshot_etag {
|
||||
return Err(TableCatalogStoreError::Conflict(
|
||||
"durable strong catalog snapshot advanced after materialization".to_string(),
|
||||
));
|
||||
}
|
||||
}
|
||||
if !fence.target_bucket_existed {
|
||||
strong_store
|
||||
@@ -836,30 +1013,19 @@ where
|
||||
}
|
||||
|
||||
async fn all_table_buckets_materialized(&self, strong_store: &StrongTableCatalogStore<B>) -> TableCatalogStoreResult<bool> {
|
||||
if self
|
||||
let Some((global_fence, _)) = self
|
||||
.read_entry::<TableCatalogBackingMigrationGlobalFence>(
|
||||
self.catalog_bucket(),
|
||||
&self.paths.backing_migration_global_fence_path(),
|
||||
)
|
||||
.await?
|
||||
.is_none()
|
||||
{
|
||||
else {
|
||||
return Ok(false);
|
||||
}
|
||||
let table_bucket_objects = self
|
||||
.backend
|
||||
.list_objects(self.catalog_bucket(), &self.paths.table_bucket_entries_prefix())
|
||||
.await?;
|
||||
for table_bucket_object in table_bucket_objects
|
||||
.iter()
|
||||
.filter(|object| object.ends_with(TABLE_BUCKET_ENTRY_FILE))
|
||||
{
|
||||
let Some((entry, _)) = self
|
||||
.read_entry::<TableBucketEntry>(self.catalog_bucket(), table_bucket_object)
|
||||
.await?
|
||||
else {
|
||||
return Ok(false);
|
||||
};
|
||||
};
|
||||
Self::validate_global_backing_migration_fence(&global_fence)?;
|
||||
let source_table_buckets = self.object_backed_table_buckets().await?;
|
||||
let source_table_bucket_names = source_table_buckets.keys().cloned().collect::<BTreeSet<_>>();
|
||||
for entry in source_table_buckets.values() {
|
||||
let Some((fence, _)) = self.read_backing_migration_fence(&entry.table_bucket).await? else {
|
||||
return Ok(false);
|
||||
};
|
||||
@@ -878,6 +1044,26 @@ where
|
||||
return Ok(false);
|
||||
}
|
||||
}
|
||||
Ok(true)
|
||||
Ok(strong_store.table_bucket_names().await? == source_table_bucket_names)
|
||||
}
|
||||
|
||||
async fn object_backed_table_buckets(&self) -> TableCatalogStoreResult<BTreeMap<String, TableBucketEntry>> {
|
||||
let mut table_buckets = BTreeMap::new();
|
||||
for object in self
|
||||
.backend
|
||||
.list_objects(self.catalog_bucket(), &self.paths.table_bucket_entries_prefix())
|
||||
.await?
|
||||
.into_iter()
|
||||
.filter(|object| object.ends_with(TABLE_BUCKET_ENTRY_FILE))
|
||||
{
|
||||
let Some((entry, _)) = self.read_entry::<TableBucketEntry>(self.catalog_bucket(), &object).await? else {
|
||||
return Err(TableCatalogStoreError::Conflict(format!(
|
||||
"table bucket changed while reading durable strong migration inventory: {object}"
|
||||
)));
|
||||
};
|
||||
validate_table_bucket_entry_object(&self.paths, &object, &entry)?;
|
||||
table_buckets.insert(entry.table_bucket.clone(), entry);
|
||||
}
|
||||
Ok(table_buckets)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,8 +21,39 @@ mod strong;
|
||||
use migration::table_catalog_backing_manifest;
|
||||
pub(crate) use object::ObjectTableCatalogStore;
|
||||
#[cfg(test)]
|
||||
pub(super) use strong::StrongTableCatalogSnapshot;
|
||||
pub(crate) use strong::StrongTableCatalogStore;
|
||||
pub(super) use strong::{
|
||||
STRONG_TABLE_CATALOG_RELOAD_MAX_ATTEMPTS, STRONG_TABLE_CATALOG_SNAPSHOT_MAX_SIZE, StrongCommitSnapshotRecord,
|
||||
StrongTableCatalogBucketSnapshot, StrongTableCatalogSnapshot, strong_snapshot_write_version,
|
||||
table_catalog_bucket_snapshot_fingerprint,
|
||||
};
|
||||
pub(crate) use strong::{StrongTableCatalogRuntime, StrongTableCatalogStore};
|
||||
|
||||
fn validate_table_bucket_entry(entry: &TableBucketEntry) -> TableCatalogStoreResult<()> {
|
||||
validate_catalog_entry_version("table bucket", entry.version)?;
|
||||
if entry.table_bucket.is_empty() {
|
||||
return Err(TableCatalogStoreError::Invalid("table bucket name cannot be empty".to_string()));
|
||||
}
|
||||
if entry.catalog_type != TABLE_BUCKET_CATALOG_TYPE {
|
||||
return Err(TableCatalogStoreError::Invalid("unsupported table bucket catalog type".to_string()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_table_entry_version_and_id(entry: &TableEntry) -> TableCatalogStoreResult<()> {
|
||||
validate_catalog_entry_version("table", entry.version)?;
|
||||
if entry.table_id.is_empty() {
|
||||
return Err(TableCatalogStoreError::Invalid("table id cannot be empty".to_string()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_view_entry_version_and_id(entry: &ViewEntry) -> TableCatalogStoreResult<()> {
|
||||
validate_catalog_entry_version("view", entry.version)?;
|
||||
if entry.view_id.is_empty() {
|
||||
return Err(TableCatalogStoreError::Invalid("view id cannot be empty".to_string()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_namespace_entry_identity(entry: &NamespaceEntry) -> TableCatalogStoreResult<Namespace> {
|
||||
validate_catalog_entry_version("namespace", entry.version)?;
|
||||
@@ -406,8 +437,31 @@ pub(crate) enum TableCatalogPutPrecondition {
|
||||
IfMatch(String),
|
||||
}
|
||||
|
||||
pub(in crate::table_catalog) fn catalog_list_next_continuation(
|
||||
seen: &mut BTreeSet<String>,
|
||||
is_truncated: bool,
|
||||
next: Option<String>,
|
||||
) -> TableCatalogStoreResult<Option<String>> {
|
||||
if !is_truncated {
|
||||
return Ok(None);
|
||||
}
|
||||
let next = next.filter(|next| !next.is_empty()).ok_or_else(|| {
|
||||
TableCatalogStoreError::Internal("truncated catalog object listing has no continuation token".to_string())
|
||||
})?;
|
||||
if !seen.insert(next.clone()) {
|
||||
return Err(TableCatalogStoreError::Internal(
|
||||
"catalog object listing continuation token did not advance".to_string(),
|
||||
));
|
||||
}
|
||||
Ok(Some(next))
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
pub(crate) trait TableCatalogObjectBackend: Clone + Send + Sync + 'static {
|
||||
fn strong_catalog_runtime(&self) -> Option<StrongTableCatalogRuntime> {
|
||||
None
|
||||
}
|
||||
|
||||
async fn read_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Option<TableCatalogObject>>;
|
||||
|
||||
async fn read_object_limited(
|
||||
@@ -845,10 +899,16 @@ where
|
||||
B: TableCatalogObjectBackend,
|
||||
{
|
||||
pub(crate) fn from_env(backend: B) -> TableCatalogStoreResult<Self> {
|
||||
Ok(Self::new(backend, TableCatalogBackingMode::from_env()?))
|
||||
Ok(match TableCatalogBackingMode::from_env()? {
|
||||
TableCatalogBackingMode::ObjectBacked => Self::ObjectBacked(ObjectTableCatalogStore::new(backend)),
|
||||
TableCatalogBackingMode::DurableStrong => {
|
||||
Self::DurableStrong(StrongTableCatalogStore::new_requiring_snapshot(backend))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) fn new(backend: B, mode: TableCatalogBackingMode) -> Self {
|
||||
#[cfg(test)]
|
||||
pub(crate) fn new_for_test(backend: B, mode: TableCatalogBackingMode) -> Self {
|
||||
match mode {
|
||||
TableCatalogBackingMode::ObjectBacked => Self::ObjectBacked(ObjectTableCatalogStore::new(backend)),
|
||||
TableCatalogBackingMode::DurableStrong => Self::DurableStrong(StrongTableCatalogStore::new(backend)),
|
||||
@@ -1352,12 +1412,14 @@ where
|
||||
|
||||
pub(crate) struct EcStoreTableCatalogObjectBackend<S> {
|
||||
store: Arc<S>,
|
||||
strong_runtime: StrongTableCatalogRuntime,
|
||||
}
|
||||
|
||||
impl<S> Clone for EcStoreTableCatalogObjectBackend<S> {
|
||||
fn clone(&self) -> Self {
|
||||
Self {
|
||||
store: self.store.clone(),
|
||||
strong_runtime: self.strong_runtime.clone(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1366,8 +1428,8 @@ impl<S> EcStoreTableCatalogObjectBackend<S>
|
||||
where
|
||||
S: TableCatalogStorage,
|
||||
{
|
||||
pub fn new(store: Arc<S>) -> Self {
|
||||
Self { store }
|
||||
pub fn new_with_strong_runtime(store: Arc<S>, strong_runtime: StrongTableCatalogRuntime) -> Self {
|
||||
Self { store, strong_runtime }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1378,6 +1440,10 @@ impl<S> TableCatalogObjectBackend for EcStoreTableCatalogObjectBackend<S>
|
||||
where
|
||||
S: TableCatalogStorage,
|
||||
{
|
||||
fn strong_catalog_runtime(&self) -> Option<StrongTableCatalogRuntime> {
|
||||
Some(self.strong_runtime.clone())
|
||||
}
|
||||
|
||||
async fn read_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Option<TableCatalogObject>> {
|
||||
self.read_object_with_options(bucket, object, ObjectOptions::default(), None)
|
||||
.await
|
||||
@@ -1537,6 +1603,7 @@ where
|
||||
|
||||
async fn list_objects(&self, bucket: &str, prefix: &str) -> TableCatalogStoreResult<Vec<String>> {
|
||||
let mut continuation = None;
|
||||
let mut seen_continuations = BTreeSet::new();
|
||||
let mut objects = BTreeSet::new();
|
||||
let max_keys = i32::try_from(TABLE_CATALOG_LIST_MAX_KEYS)
|
||||
.map_err(|_| TableCatalogStoreError::Internal("catalog list limit exceeds storage API range".to_string()))?;
|
||||
@@ -1553,14 +1620,10 @@ where
|
||||
objects.insert(object.name);
|
||||
}
|
||||
|
||||
if !result.is_truncated {
|
||||
break;
|
||||
}
|
||||
|
||||
let Some(next) = result.next_continuation_token else {
|
||||
break;
|
||||
match catalog_list_next_continuation(&mut seen_continuations, result.is_truncated, result.next_continuation_token)? {
|
||||
Some(next) => continuation = Some(next),
|
||||
None => break,
|
||||
};
|
||||
continuation = Some(next);
|
||||
}
|
||||
|
||||
Ok(objects.into_iter().collect())
|
||||
@@ -1627,20 +1690,6 @@ where
|
||||
opts: ObjectOptions,
|
||||
max_size: Option<usize>,
|
||||
) -> TableCatalogStoreResult<Option<TableCatalogObject>> {
|
||||
let info = match self.store.get_object_info(bucket, object, &opts).await {
|
||||
Ok(info) => info,
|
||||
Err(err) if is_missing_storage_error(&err) => return Ok(None),
|
||||
Err(err) => return Err(storage_error_to_catalog("read catalog object info", err)),
|
||||
};
|
||||
if let Some(max_size) = max_size {
|
||||
let object_size = usize::try_from(info.size)
|
||||
.map_err(|_| TableCatalogStoreError::Invalid(format!("catalog object {bucket}/{object} has an invalid size")))?;
|
||||
if object_size > max_size {
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"catalog object {bucket}/{object} exceeds the maximum size of {max_size} bytes"
|
||||
)));
|
||||
}
|
||||
}
|
||||
let mut reader = match self
|
||||
.store
|
||||
.get_object_reader(bucket, object, None, HeaderMap::new(), &opts)
|
||||
@@ -1650,6 +1699,17 @@ where
|
||||
Err(err) if is_missing_storage_error(&err) => return Ok(None),
|
||||
Err(err) => return Err(storage_error_to_catalog("read catalog object", err)),
|
||||
};
|
||||
if let Some(max_size) = max_size {
|
||||
let object_size = usize::try_from(reader.object_info.size)
|
||||
.map_err(|_| TableCatalogStoreError::Invalid(format!("catalog object {bucket}/{object} has an invalid size")))?;
|
||||
if object_size > max_size {
|
||||
return Err(TableCatalogStoreError::Invalid(format!(
|
||||
"catalog object {bucket}/{object} exceeds the maximum size of {max_size} bytes"
|
||||
)));
|
||||
}
|
||||
}
|
||||
let etag = reader.object_info.etag.clone();
|
||||
let mod_time = reader.object_info.mod_time;
|
||||
let mut data = Vec::new();
|
||||
if let Some(max_size) = max_size {
|
||||
let read_limit = u64::try_from(max_size.saturating_add(1)).unwrap_or(u64::MAX);
|
||||
@@ -1666,11 +1726,7 @@ where
|
||||
TableCatalogStoreError::Internal(format!("failed to read catalog object {bucket}/{object}: {err}"))
|
||||
})?;
|
||||
}
|
||||
Ok(Some(TableCatalogObject {
|
||||
data,
|
||||
etag: info.etag,
|
||||
mod_time: info.mod_time,
|
||||
}))
|
||||
Ok(Some(TableCatalogObject { data, etag, mod_time }))
|
||||
}
|
||||
|
||||
async fn put_object_with_options(
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+5613
-109
File diff suppressed because it is too large
Load Diff
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user