mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-28 07:57:01 +00:00
Merge remote-tracking branch 'origin/main' into cxymds/fix-rebalance-multipart-retry
This commit is contained in:
马登山
+64
-20
@@ -1,6 +1,6 @@
|
|||||||
# ARCHITECTURE.md
|
# ARCHITECTURE.md
|
||||||
|
|
||||||
> Last updated: 2026-07-02 · Revision: 2
|
> Last updated: 2026-08-12 · Revision: 3
|
||||||
>
|
>
|
||||||
> This document describes the high-level architecture of RustFS.
|
> This document describes the high-level architecture of RustFS.
|
||||||
> If you want to familiarize yourself with the code base, you are in the right place!
|
> If you want to familiarize yourself with the code base, you are in the right place!
|
||||||
@@ -119,19 +119,44 @@ module split is tracked under `docs/architecture/`.
|
|||||||
|
|
||||||
3. **Each type has exactly one definition.** Types shared across crates must be defined
|
3. **Each type has exactly one definition.** Types shared across crates must be defined
|
||||||
in one crate and re-exported or imported by others.
|
in one crate and re-exported or imported by others.
|
||||||
- ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
|
- ⚠️ VIOLATED: `ReplicationStats` names three unrelated types
|
||||||
`BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
|
(`crates/data-usage/src/data_usage.rs`,
|
||||||
|
`crates/obs/src/metrics/collectors/replication.rs`,
|
||||||
|
`crates/ecstore/src/bucket/replication/replication_state.rs`) — a naming
|
||||||
|
collision, not copies; renaming is tracked in rustfs/backlog#1847.
|
||||||
|
- `LastMinuteLatency` has two deliberately different implementations: the
|
||||||
|
per-second bucketed accumulator in `crates/common/src/last_minute.rs` and
|
||||||
|
the in-memory endpoint-health sample tracker in
|
||||||
|
`crates/ecstore/src/bucket/bucket_target_sys.rs` (its doc comment explains
|
||||||
|
why it stays local).
|
||||||
|
- ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
|
||||||
|
definition (`crates/io-core/src/backpressure.rs`,
|
||||||
|
`crates/data-usage/src/data_usage.rs`). A zero-consumer
|
||||||
|
`BackpressureSettings` copy lingers in `crates/io-metrics/src/config.rs`;
|
||||||
|
its removal is tracked in rustfs/backlog#1833.
|
||||||
|
|
||||||
4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
|
4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
|
||||||
storage-level abstractions (objects, buckets, disks, pools).
|
storage-level abstractions (objects, buckets, disks, pools).
|
||||||
|
- ⚠️ VIOLATED: 58 files under `crates/ecstore/src` reference `s3s`
|
||||||
|
(`rg -l 's3s' crates/ecstore/src | wc -l`), `crates/ecstore/src/client/`
|
||||||
|
is a ~9.4K-line embedded S3 HTTP client, and `crates/ecstore/Cargo.toml`
|
||||||
|
depends on `s3s`, `http`, `hyper`/`hyper-util`/`hyper-rustls`, and
|
||||||
|
`reqwest`. Target state: the engine's need to act as an S3 client
|
||||||
|
(tiering, replication targets) is served by an extracted client crate,
|
||||||
|
and ecstore holds no wire or DTO types.
|
||||||
|
|
||||||
5. **The `rustfs` binary crate is the only place that wires everything together.**
|
5. **The `rustfs` binary crate is the only place that wires everything together.**
|
||||||
Individual crates should be testable in isolation.
|
Individual crates should be testable in isolation.
|
||||||
|
|
||||||
6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
|
6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
|
||||||
not bare `Error`).
|
not bare `Error`).
|
||||||
- ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
|
- ✅ RESOLVED (strategy): `snafu` is gone from source
|
||||||
`heal` use `anyhow` in library code.
|
(`rg -l snafu crates/ rustfs/` is empty) and library code no longer uses
|
||||||
|
`anyhow` (remaining hits are test code and the `e2e_test` crate; `heal`
|
||||||
|
uses `thiserror`).
|
||||||
|
- ⚠️ VIOLATED (naming): 6 crates still export a bare `pub enum Error`:
|
||||||
|
`crypto`, `filemeta`, `heal`, `iam`, `policy`, and `replication`
|
||||||
|
(`src/resync.rs`) — all `thiserror`-derived.
|
||||||
|
|
||||||
## Known Structural Issues
|
## Known Structural Issues
|
||||||
|
|
||||||
@@ -140,13 +165,25 @@ module split is tracked under `docs/architecture/`.
|
|||||||
|
|
||||||
### Critical
|
### Critical
|
||||||
|
|
||||||
- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
|
- **scanner/data-usage duplicate `.usage-cache.bin` serialization types.** The
|
||||||
but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
|
original finding ("common/scanner code duplication, ~3K lines") is resolved:
|
||||||
types instead of importing them.
|
`scanner` imports the shared data-usage types from `rustfs-data-usage` (see
|
||||||
|
the `pub use rustfs_data_usage::…` re-exports at the top of
|
||||||
|
`crates/scanner/src/data_usage_define.rs`). What remains: `scanner` and
|
||||||
|
`data-usage` each hold their own serialization types for the scanner cache
|
||||||
|
file (`DataUsageCacheInfo`/`DataUsageEntryInfo` in
|
||||||
|
`crates/scanner/src/data_usage_define.rs` vs
|
||||||
|
`DataUsageCacheInfo`/`DataUsageEntry` in
|
||||||
|
`crates/data-usage/src/data_usage.rs`); convergence is tracked in
|
||||||
|
rustfs/backlog#1828.
|
||||||
|
|
||||||
- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
|
- **ecstore is a monolith (265 files, ~288K lines — roughly half is inline
|
||||||
bucket management, erasure coding, replication, lifecycle, RPC, and configuration
|
`#[cfg(test)]` code).** Measured with
|
||||||
— all in one crate. It should be decomposed along its existing subdirectories.
|
`find crates/ecstore/src -name '*.rs' | xargs wc -l`. It contains disk
|
||||||
|
management, bucket management, erasure coding, replication, lifecycle, RPC,
|
||||||
|
and configuration — all in one crate. It should be decomposed along its
|
||||||
|
existing subdirectories; the split plan lives in
|
||||||
|
[docs/architecture/ecstore-module-split-plan.md](docs/architecture/ecstore-module-split-plan.md).
|
||||||
|
|
||||||
### High
|
### High
|
||||||
|
|
||||||
@@ -154,19 +191,26 @@ module split is tracked under `docs/architecture/`.
|
|||||||
`common → filemeta/madmin` edges must stay removed so leaf/helper crates do
|
`common → filemeta/madmin` edges must stay removed so leaf/helper crates do
|
||||||
not regain upward dependencies.
|
not regain upward dependencies.
|
||||||
|
|
||||||
- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
|
- **Three-layer backpressure/deadlock policy bridging** across io-core,
|
||||||
concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
|
concurrency, and `rustfs/src/storage`. The config types are no longer
|
||||||
explicit projections into the concurrency/io-core policy shapes, and workload
|
duplicated (`BackpressureConfig` and `DeadlockDetectorConfig` are each
|
||||||
|
defined once, in io-core). Storage policies expose and consume explicit
|
||||||
|
projections into the concurrency/io-core policy shapes, and workload
|
||||||
admission snapshots are composed through provider registries; later work
|
admission snapshots are composed through provider registries; later work
|
||||||
should use those bridges before deleting compatibility wrappers.
|
should use those bridges before deleting compatibility wrappers.
|
||||||
|
|
||||||
### Medium
|
### Medium
|
||||||
|
|
||||||
- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
|
- **Bare `Error` naming.** Error-handling strategy has converged on `thiserror`
|
||||||
mixed naming (bare `Error` vs descriptive names).
|
(no `snafu`, no `anyhow` in library code); the remaining inconsistency is the
|
||||||
|
bare `pub enum Error` naming in the 6 crates listed under Invariant 6.
|
||||||
|
|
||||||
- **Ambiguous common vs utils boundary.** Both described as "utilities and data
|
- **`common` is mostly parked domain code, not shared utilities.** Of its
|
||||||
structures." Need clear ownership rules.
|
6,724 lines, ~83% is scanner/heal domain code stranded there to break
|
||||||
|
dependency cycles (`metrics.rs`, ~4,810 lines of scanner-domain metrics;
|
||||||
|
`heal_channel.rs`, ~776 lines of heal-domain channel types). The
|
||||||
|
"common vs utils" naming ambiguity is secondary to moving that code to its
|
||||||
|
domain owners.
|
||||||
|
|
||||||
## Cross-Cutting Concerns
|
## Cross-Cutting Concerns
|
||||||
|
|
||||||
@@ -232,7 +276,7 @@ The binary (`main.rs`) boots in this order:
|
|||||||
|
|
||||||
```
|
```
|
||||||
┌─────────┐
|
┌─────────┐
|
||||||
│ rustfs │ (binary + lib, 75K lines)
|
│ rustfs │ (binary + lib)
|
||||||
│ main │
|
│ main │
|
||||||
└────┬────┘
|
└────┬────┘
|
||||||
│
|
│
|
||||||
@@ -255,7 +299,7 @@ The binary (`main.rs`) boots in this order:
|
|||||||
│ │ │
|
│ │ │
|
||||||
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
|
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
|
||||||
│ ecstore │ │ rio │ │ io-core │
|
│ ecstore │ │ rio │ │ io-core │
|
||||||
│ (87K,core) │ │ (readers) │ │ (zero-copy) │
|
│ (core) │ │ (readers) │ │ (zero-copy) │
|
||||||
└─────┬──────┘ └─────────────┘ └─────────────┘
|
└─────┬──────┘ └─────────────┘ └─────────────┘
|
||||||
│
|
│
|
||||||
┌─────┬──┼──┬─────┬──────┐
|
┌─────┬──┼──┬─────┬──────┐
|
||||||
|
|||||||
Generated
+74
-35
@@ -104,6 +104,12 @@ dependencies = [
|
|||||||
"memchr",
|
"memchr",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "aliasable"
|
||||||
|
version = "0.1.3"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "250f629c0161ad8107cf89319e990051fae62832fd343083bea452d93e2205fd"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "aligned-vec"
|
name = "aligned-vec"
|
||||||
version = "0.6.4"
|
version = "0.6.4"
|
||||||
@@ -266,24 +272,24 @@ checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "apache-avro"
|
name = "apache-avro"
|
||||||
version = "0.21.0"
|
version = "0.22.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "36fa98bc79671c7981272d91a8753a928ff6a1cd8e4f20a44c45bd5d313840bf"
|
checksum = "312c1ea69e5fe9966e0029fb95aca8790100b85aff4f0d3b00a9337c74069a9c"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"bigdecimal",
|
"bigdecimal",
|
||||||
"bon",
|
"bon",
|
||||||
"digest 0.10.7",
|
"digest 0.11.3",
|
||||||
"log",
|
"log",
|
||||||
"miniz_oxide",
|
"miniz_oxide 0.9.1",
|
||||||
"num-bigint 0.4.8",
|
"num-bigint 0.4.8",
|
||||||
|
"ouroboros",
|
||||||
"quad-rand",
|
"quad-rand",
|
||||||
"rand 0.9.5",
|
"rand 0.10.2",
|
||||||
"regex-lite",
|
"regex-lite",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_bytes",
|
"serde_bytes",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"strum 0.27.2",
|
"strum",
|
||||||
"strum_macros 0.27.2",
|
|
||||||
"thiserror 2.0.20",
|
"thiserror 2.0.20",
|
||||||
"uuid",
|
"uuid",
|
||||||
]
|
]
|
||||||
@@ -1458,7 +1464,7 @@ dependencies = [
|
|||||||
"addr2line",
|
"addr2line",
|
||||||
"cfg-if",
|
"cfg-if",
|
||||||
"libc",
|
"libc",
|
||||||
"miniz_oxide",
|
"miniz_oxide 0.8.9",
|
||||||
"object 0.37.3",
|
"object 0.37.3",
|
||||||
"rustc-demangle",
|
"rustc-demangle",
|
||||||
"windows-link",
|
"windows-link",
|
||||||
@@ -2003,7 +2009,7 @@ version = "4.6.4"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061"
|
checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"heck",
|
"heck 0.5.0",
|
||||||
"proc-macro2",
|
"proc-macro2",
|
||||||
"quote",
|
"quote",
|
||||||
"syn 3.0.3",
|
"syn 3.0.3",
|
||||||
@@ -4184,7 +4190,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
|
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"crc32fast",
|
"crc32fast",
|
||||||
"miniz_oxide",
|
"miniz_oxide 0.8.9",
|
||||||
"zlib-rs",
|
"zlib-rs",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -4854,6 +4860,12 @@ dependencies = [
|
|||||||
"stable_deref_trait",
|
"stable_deref_trait",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "heck"
|
||||||
|
version = "0.4.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8"
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "heck"
|
name = "heck"
|
||||||
version = "0.5.0"
|
version = "0.5.0"
|
||||||
@@ -6391,6 +6403,15 @@ dependencies = [
|
|||||||
"simd-adler32",
|
"simd-adler32",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "miniz_oxide"
|
||||||
|
version = "0.9.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
|
||||||
|
dependencies = [
|
||||||
|
"adler2",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "minlz"
|
name = "minlz"
|
||||||
version = "1.2.3"
|
version = "1.2.3"
|
||||||
@@ -6485,7 +6506,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||||||
checksum = "a4db8a44120571277accfaa3f3d91e7d3989d601d817c2fc01a9391b86135666"
|
checksum = "a4db8a44120571277accfaa3f3d91e7d3989d601d817c2fc01a9391b86135666"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"darling 0.23.0",
|
"darling 0.23.0",
|
||||||
"heck",
|
"heck 0.5.0",
|
||||||
"manyhow",
|
"manyhow",
|
||||||
"num-bigint 0.4.8",
|
"num-bigint 0.4.8",
|
||||||
"proc-macro-crate",
|
"proc-macro-crate",
|
||||||
@@ -7190,6 +7211,30 @@ dependencies = [
|
|||||||
"num-traits",
|
"num-traits",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "ouroboros"
|
||||||
|
version = "0.18.5"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "1e0f050db9c44b97a94723127e6be766ac5c340c48f2c4bb3ffa11713744be59"
|
||||||
|
dependencies = [
|
||||||
|
"aliasable",
|
||||||
|
"ouroboros_macro",
|
||||||
|
"static_assertions",
|
||||||
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "ouroboros_macro"
|
||||||
|
version = "0.18.5"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "3c7028bdd3d43083f6d8d4d5187680d0d3560d54df4cc9d752005268b41e64d0"
|
||||||
|
dependencies = [
|
||||||
|
"heck 0.4.1",
|
||||||
|
"proc-macro2",
|
||||||
|
"proc-macro2-diagnostics",
|
||||||
|
"quote",
|
||||||
|
"syn 2.0.119",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "outref"
|
name = "outref"
|
||||||
version = "0.5.2"
|
version = "0.5.2"
|
||||||
@@ -7925,6 +7970,19 @@ dependencies = [
|
|||||||
"unicode-ident",
|
"unicode-ident",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "proc-macro2-diagnostics"
|
||||||
|
version = "0.10.1"
|
||||||
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
|
checksum = "af066a9c399a26e020ada66a034357a868728e72cd426f3adcd35f80d88d88c8"
|
||||||
|
dependencies = [
|
||||||
|
"proc-macro2",
|
||||||
|
"quote",
|
||||||
|
"syn 2.0.119",
|
||||||
|
"version_check",
|
||||||
|
"yansi",
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "prometheus"
|
name = "prometheus"
|
||||||
version = "0.14.0"
|
version = "0.14.0"
|
||||||
@@ -7984,7 +8042,7 @@ version = "0.13.5"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
|
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"heck",
|
"heck 0.5.0",
|
||||||
"itertools 0.14.0",
|
"itertools 0.14.0",
|
||||||
"log",
|
"log",
|
||||||
"multimap",
|
"multimap",
|
||||||
@@ -8004,7 +8062,7 @@ version = "0.14.4"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
|
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"heck",
|
"heck 0.5.0",
|
||||||
"itertools 0.14.0",
|
"itertools 0.14.0",
|
||||||
"log",
|
"log",
|
||||||
"multimap",
|
"multimap",
|
||||||
@@ -9287,7 +9345,6 @@ dependencies = [
|
|||||||
name = "rustfs-data-usage"
|
name = "rustfs-data-usage"
|
||||||
version = "1.0.0-rc.1"
|
version = "1.0.0-rc.1"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"async-trait",
|
|
||||||
"hotpath",
|
"hotpath",
|
||||||
"rmp-serde",
|
"rmp-serde",
|
||||||
"rustfs-filemeta",
|
"rustfs-filemeta",
|
||||||
@@ -9924,7 +9981,7 @@ dependencies = [
|
|||||||
"rustfs-crypto",
|
"rustfs-crypto",
|
||||||
"serde",
|
"serde",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"strum 0.28.0",
|
"strum",
|
||||||
"temp-env",
|
"temp-env",
|
||||||
"test-case",
|
"test-case",
|
||||||
"thiserror 2.0.20",
|
"thiserror 2.0.20",
|
||||||
@@ -11513,31 +11570,13 @@ version = "0.11.1"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
|
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "strum"
|
|
||||||
version = "0.27.2"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf"
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "strum"
|
name = "strum"
|
||||||
version = "0.28.0"
|
version = "0.28.0"
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "9628de9b8791db39ceda2b119bbe13134770b56c138ec1d3af810d045c04f9bd"
|
checksum = "9628de9b8791db39ceda2b119bbe13134770b56c138ec1d3af810d045c04f9bd"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"strum_macros 0.28.0",
|
"strum_macros",
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "strum_macros"
|
|
||||||
version = "0.27.2"
|
|
||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
|
||||||
checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7"
|
|
||||||
dependencies = [
|
|
||||||
"heck",
|
|
||||||
"proc-macro2",
|
|
||||||
"quote",
|
|
||||||
"syn 2.0.119",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -11546,7 +11585,7 @@ version = "0.28.0"
|
|||||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||||
checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664"
|
checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"heck",
|
"heck 0.5.0",
|
||||||
"proc-macro2",
|
"proc-macro2",
|
||||||
"quote",
|
"quote",
|
||||||
"syn 2.0.119",
|
"syn 2.0.119",
|
||||||
|
|||||||
+1
-1
@@ -171,7 +171,7 @@ tower = { version = "0.5.3" }
|
|||||||
tower-http = { version = "0.7.0" }
|
tower-http = { version = "0.7.0" }
|
||||||
|
|
||||||
# Serialization and Data Formats
|
# Serialization and Data Formats
|
||||||
apache-avro = "0.21.0"
|
apache-avro = "0.22.0"
|
||||||
bytes = { version = "1.12.1" }
|
bytes = { version = "1.12.1" }
|
||||||
bytesize = "2.7.0"
|
bytesize = "2.7.0"
|
||||||
byteorder = "1.5.0"
|
byteorder = "1.5.0"
|
||||||
|
|||||||
@@ -37,7 +37,6 @@ hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
|
|||||||
hotpath.workspace = true
|
hotpath.workspace = true
|
||||||
serde = { workspace = true, features = ["derive"] }
|
serde = { workspace = true, features = ["derive"] }
|
||||||
rmp-serde = { workspace = true }
|
rmp-serde = { workspace = true }
|
||||||
async-trait = { workspace = true }
|
|
||||||
rustfs-filemeta = { workspace = true }
|
rustfs-filemeta = { workspace = true }
|
||||||
|
|
||||||
[lib]
|
[lib]
|
||||||
|
|||||||
@@ -846,8 +846,15 @@ impl DataUsageEntry {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Data usage cache info
|
/// Read-only projection of the scanner's `.usage-cache.bin` info block.
|
||||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
///
|
||||||
|
/// The canonical wire format is written by the hand-written map-encoded
|
||||||
|
/// `Serialize` on the scanner-side `DataUsageCacheInfo`
|
||||||
|
/// (`crates/scanner/src/data_usage_define.rs`), which carries 16 fields.
|
||||||
|
/// This type decodes only the shared subset and is deliberately not
|
||||||
|
/// `Serialize`: a derived (array) encoding of this 6-field subset would
|
||||||
|
/// corrupt the cache for scanner readers, so no write path may exist here.
|
||||||
|
#[derive(Clone, Debug, Default, Deserialize)]
|
||||||
pub struct DataUsageCacheInfo {
|
pub struct DataUsageCacheInfo {
|
||||||
pub name: String,
|
pub name: String,
|
||||||
pub next_cycle: u64,
|
pub next_cycle: u64,
|
||||||
@@ -863,8 +870,12 @@ pub struct DataUsageCacheInfo {
|
|||||||
pub snapshot_complete: bool,
|
pub snapshot_complete: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Data usage cache
|
/// Read-only projection of a scanner-written `.usage-cache.bin` file.
|
||||||
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
|
///
|
||||||
|
/// The scanner-side `DataUsageCache` (`crates/scanner/src/data_usage_define.rs`)
|
||||||
|
/// owns the persisted format; this type only decodes it (see
|
||||||
|
/// [`DataUsageCacheInfo`]) and must never grow a serialization path.
|
||||||
|
#[derive(Clone, Debug, Default, Deserialize)]
|
||||||
pub struct DataUsageCache {
|
pub struct DataUsageCache {
|
||||||
pub info: DataUsageCacheInfo,
|
pub info: DataUsageCacheInfo,
|
||||||
pub cache: HashMap<String, DataUsageEntry>,
|
pub cache: HashMap<String, DataUsageEntry>,
|
||||||
@@ -1186,31 +1197,10 @@ impl DataUsageCache {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn marshal_msg(&self) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
|
|
||||||
let mut buf = Vec::new();
|
|
||||||
self.serialize(&mut rmp_serde::Serializer::new(&mut buf))?;
|
|
||||||
Ok(buf)
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn unmarshal(buf: &[u8]) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
|
pub fn unmarshal(buf: &[u8]) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
|
||||||
let t: Self = rmp_serde::from_slice(buf)?;
|
let t: Self = rmp_serde::from_slice(buf)?;
|
||||||
Ok(t)
|
Ok(t)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Note: load and save methods are storage-specific and should be implemented
|
|
||||||
// in the ecstore crate where storage access is available
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Trait for storage-specific operations on DataUsageCache
|
|
||||||
#[async_trait::async_trait]
|
|
||||||
pub trait DataUsageCacheStorage {
|
|
||||||
/// Load data usage cache from backend storage
|
|
||||||
async fn load(store: &dyn std::any::Any, name: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
|
|
||||||
where
|
|
||||||
Self: Sized;
|
|
||||||
|
|
||||||
/// Save data usage cache to backend storage
|
|
||||||
async fn save(&self, name: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>>;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Helper structs and functions for cache operations
|
// Helper structs and functions for cache operations
|
||||||
@@ -1832,6 +1822,82 @@ mod tests {
|
|||||||
assert!(decoded.all_tier_stats.is_none());
|
assert!(decoded.all_tier_stats.is_none());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Scanner-written `.usage-cache.bin` bytes: a 2-element array of the
|
||||||
|
/// canonical 16-field map-encoded info block and one map-encoded entry.
|
||||||
|
/// Captured from the canonical writer's `marshal_msg` — see
|
||||||
|
/// `usage_cache_wire_format_is_pinned` in
|
||||||
|
/// `crates/scanner/src/data_usage_define.rs`, which pins these exact
|
||||||
|
/// bytes and documents regeneration. Hardcoded here because a
|
||||||
|
/// dev-dependency on rustfs-scanner would pull the whole ecstore tree
|
||||||
|
/// into this crate's test build, and a fixture generated at test runtime
|
||||||
|
/// could not detect writer drift anyway.
|
||||||
|
const SCANNER_USAGE_CACHE_WIRE_FIXTURE: &[u8] = &[
|
||||||
|
0x92, 0xde, 0x00, 0x10, 0xa4, 0x6e, 0x61, 0x6d, 0x65, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||||
|
0x74, 0xaa, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x07, 0xac, 0x6c, 0x65, 0x61, 0x64, 0x65, 0x72,
|
||||||
|
0x5f, 0x65, 0x70, 0x6f, 0x63, 0x68, 0x09, 0xab, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x92,
|
||||||
|
0xce, 0x65, 0x53, 0xf1, 0x00, 0x00, 0xac, 0x73, 0x6b, 0x69, 0x70, 0x5f, 0x68, 0x65, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0xc3,
|
||||||
|
0xa9, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0xc0, 0xab, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
|
||||||
|
0x69, 0x6f, 0x6e, 0xc0, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x81,
|
||||||
|
0xb0, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x6c, 0x6f, 0x73, 0x74, 0x0b, 0xb1, 0x73,
|
||||||
|
0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0x5f, 0x61, 0x66, 0x74, 0x65, 0x72, 0xb2, 0x77, 0x69, 0x72,
|
||||||
|
0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0xaf, 0x73, 0x63, 0x61, 0x6e,
|
||||||
|
0x5f, 0x63, 0x68, 0x65, 0x63, 0x6b, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0xc0, 0xad, 0x70, 0x65, 0x6e, 0x64, 0x69, 0x6e, 0x67,
|
||||||
|
0x5f, 0x68, 0x65, 0x61, 0x6c, 0x73, 0x91, 0x9a, 0xa6, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0xab, 0x77, 0x69, 0x72, 0x65,
|
||||||
|
0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0xa6, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x6e, 0xc0, 0x01, 0x64, 0xcc, 0xc8, 0x03,
|
||||||
|
0xa8, 0x64, 0x65, 0x66, 0x65, 0x72, 0x72, 0x65, 0x64, 0xa6, 0x62, 0x75, 0x64, 0x67, 0x65, 0x74, 0xab, 0x6f, 0x62, 0x6a,
|
||||||
|
0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x63, 0x6b, 0xc0, 0xa6, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x92, 0x01, 0x02, 0xb1,
|
||||||
|
0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0xc3, 0xb0, 0x73,
|
||||||
|
0x63, 0x61, 0x6e, 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x5f, 0x64, 0x69, 0x67, 0x65, 0x73, 0x74, 0xdc, 0x00, 0x20, 0x03, 0x03,
|
||||||
|
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
|
||||||
|
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0xb0, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x6b, 0x65, 0x79,
|
||||||
|
0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x01, 0x81, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||||
|
0x74, 0x8b, 0xa8, 0x63, 0x68, 0x69, 0x6c, 0x64, 0x72, 0x65, 0x6e, 0x90, 0xa4, 0x73, 0x69, 0x7a, 0x65, 0xcd, 0x10, 0x00,
|
||||||
|
0xa7, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x03, 0xa8, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x05, 0xae,
|
||||||
|
0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x5f, 0x6d, 0x61, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x01, 0xa9, 0x6f, 0x62, 0x6a, 0x5f,
|
||||||
|
0x73, 0x69, 0x7a, 0x65, 0x73, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xac, 0x6f, 0x62,
|
||||||
|
0x6a, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x97, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x72,
|
||||||
|
0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0xc0, 0xa9, 0x63, 0x6f,
|
||||||
|
0x6d, 0x70, 0x61, 0x63, 0x74, 0x65, 0x64, 0xc3, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65,
|
||||||
|
0x63, 0x74, 0x73, 0x02, 0xae, 0x61, 0x6c, 0x6c, 0x5f, 0x74, 0x69, 0x65, 0x72, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0x91,
|
||||||
|
0x81, 0xa4, 0x57, 0x41, 0x52, 0x4d, 0x93, 0xcd, 0x08, 0x00, 0x02, 0x01,
|
||||||
|
];
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn thin_usage_cache_decodes_scanner_wire_fixture() {
|
||||||
|
let decoded =
|
||||||
|
DataUsageCache::unmarshal(SCANNER_USAGE_CACHE_WIRE_FIXTURE).expect("thin projection decodes a scanner-written cache");
|
||||||
|
|
||||||
|
// The six fields shared with the scanner's 16-field info block; the
|
||||||
|
// remaining ten (lifecycle, replication, checkpoint, heals, ...) must
|
||||||
|
// be skipped, not error.
|
||||||
|
assert_eq!(decoded.info.name, "wire-bucket");
|
||||||
|
assert_eq!(decoded.info.next_cycle, 7);
|
||||||
|
assert_eq!(
|
||||||
|
decoded.info.last_update,
|
||||||
|
Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000))
|
||||||
|
);
|
||||||
|
assert!(decoded.info.skip_healing);
|
||||||
|
assert_eq!(decoded.info.failed_objects.get("wire-bucket/lost"), Some(&11));
|
||||||
|
assert!(decoded.info.snapshot_complete);
|
||||||
|
|
||||||
|
// Entries use the shared canonical map-encoded type end to end.
|
||||||
|
let entry = decoded.cache.get("wire-bucket").expect("fixture entry decodes");
|
||||||
|
assert_eq!(entry.size, 4096);
|
||||||
|
assert_eq!(entry.objects, 3);
|
||||||
|
assert_eq!(entry.versions, 5);
|
||||||
|
assert_eq!(entry.delete_markers, 1);
|
||||||
|
assert!(entry.compacted);
|
||||||
|
assert_eq!(entry.failed_objects, 2);
|
||||||
|
assert_eq!(
|
||||||
|
entry.all_tier_stats.as_ref().and_then(|tiers| tiers.tiers.get("WARM")),
|
||||||
|
Some(&TierStats {
|
||||||
|
total_size: 2048,
|
||||||
|
num_versions: 2,
|
||||||
|
num_objects: 1,
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn hash_path_uses_portable_slash_semantics() {
|
fn hash_path_uses_portable_slash_semantics() {
|
||||||
for (input, expected) in [
|
for (input, expected) in [
|
||||||
|
|||||||
@@ -62,8 +62,8 @@ pub(crate) struct VersionShardCensus {
|
|||||||
pub data_dir: Option<String>,
|
pub data_dir: Option<String>,
|
||||||
pub erasure_index: Option<usize>,
|
pub erasure_index: Option<usize>,
|
||||||
pub expected_part_numbers: BTreeSet<usize>,
|
pub expected_part_numbers: BTreeSet<usize>,
|
||||||
pub present_part_numbers: BTreeSet<usize>,
|
|
||||||
pub present_part_fingerprints: BTreeMap<usize, PartShardFingerprint>,
|
pub present_part_fingerprints: BTreeMap<usize, PartShardFingerprint>,
|
||||||
|
pub inline_data_fingerprint: Option<PartShardFingerprint>,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
@@ -74,7 +74,12 @@ pub(crate) struct PartShardFingerprint {
|
|||||||
|
|
||||||
impl VersionShardCensus {
|
impl VersionShardCensus {
|
||||||
pub(crate) fn is_complete(&self) -> bool {
|
pub(crate) fn is_complete(&self) -> bool {
|
||||||
self.has_xl_meta && self.expected_part_numbers == self.present_part_numbers
|
self.has_xl_meta
|
||||||
|
&& self.expected_part_numbers.len() == self.present_part_fingerprints.len()
|
||||||
|
&& self
|
||||||
|
.expected_part_numbers
|
||||||
|
.iter()
|
||||||
|
.all(|part_number| self.present_part_fingerprints.contains_key(part_number))
|
||||||
}
|
}
|
||||||
|
|
||||||
pub(crate) fn matches_manifest(&self, manifest: &Self) -> bool {
|
pub(crate) fn matches_manifest(&self, manifest: &Self) -> bool {
|
||||||
@@ -85,6 +90,7 @@ impl VersionShardCensus {
|
|||||||
&& self.erasure_index == manifest.erasure_index
|
&& self.erasure_index == manifest.erasure_index
|
||||||
&& self.expected_part_numbers == manifest.expected_part_numbers
|
&& self.expected_part_numbers == manifest.expected_part_numbers
|
||||||
&& self.present_part_fingerprints == manifest.present_part_fingerprints
|
&& self.present_part_fingerprints == manifest.present_part_fingerprints
|
||||||
|
&& self.inline_data_fingerprint == manifest.inline_data_fingerprint
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -93,6 +99,13 @@ fn sha256_hex(data: &[u8]) -> String {
|
|||||||
digest.iter().map(|byte| format!("{byte:02x}")).collect()
|
digest.iter().map(|byte| format!("{byte:02x}")).collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn shard_fingerprint(data: &[u8]) -> ChaosResult<PartShardFingerprint> {
|
||||||
|
Ok(PartShardFingerprint {
|
||||||
|
size: u64::try_from(data.len())?,
|
||||||
|
sha256: sha256_hex(data),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// Single-node RustFS server with `disk_count` local volume directories that
|
/// Single-node RustFS server with `disk_count` local volume directories that
|
||||||
/// can be faulted individually while the server is running.
|
/// can be faulted individually while the server is running.
|
||||||
pub struct DiskFaultHarness {
|
pub struct DiskFaultHarness {
|
||||||
@@ -301,8 +314,8 @@ pub(crate) fn census_object_version_on_disk(
|
|||||||
data_dir: None,
|
data_dir: None,
|
||||||
erasure_index: None,
|
erasure_index: None,
|
||||||
expected_part_numbers: BTreeSet::new(),
|
expected_part_numbers: BTreeSet::new(),
|
||||||
present_part_numbers: BTreeSet::new(),
|
|
||||||
present_part_fingerprints: BTreeMap::new(),
|
present_part_fingerprints: BTreeMap::new(),
|
||||||
|
inline_data_fingerprint: None,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -315,10 +328,10 @@ pub(crate) fn census_object_version_on_disk(
|
|||||||
};
|
};
|
||||||
let data_dir = file_info.data_dir.map(|id| id.to_string());
|
let data_dir = file_info.data_dir.map(|id| id.to_string());
|
||||||
let erasure_index = Some(file_info.erasure.index);
|
let erasure_index = Some(file_info.erasure.index);
|
||||||
|
let inline_data_fingerprint = file_info.data.as_deref().map(shard_fingerprint).transpose()?;
|
||||||
let part_dir = data_dir.as_ref().map_or_else(|| object_dir.clone(), |id| object_dir.join(id));
|
let part_dir = data_dir.as_ref().map_or_else(|| object_dir.clone(), |id| object_dir.join(id));
|
||||||
let (present_part_numbers, present_part_fingerprints) = match std::fs::read_dir(&part_dir) {
|
let present_part_fingerprints = match std::fs::read_dir(&part_dir) {
|
||||||
Ok(entries) => {
|
Ok(entries) => {
|
||||||
let mut numbers = BTreeSet::new();
|
|
||||||
let mut fingerprints = BTreeMap::new();
|
let mut fingerprints = BTreeMap::new();
|
||||||
for entry in entries {
|
for entry in entries {
|
||||||
let entry = entry?;
|
let entry = entry?;
|
||||||
@@ -333,19 +346,12 @@ pub(crate) fn census_object_version_on_disk(
|
|||||||
else {
|
else {
|
||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
numbers.insert(part_number);
|
|
||||||
let data = std::fs::read(entry.path())?;
|
let data = std::fs::read(entry.path())?;
|
||||||
fingerprints.insert(
|
fingerprints.insert(part_number, shard_fingerprint(&data)?);
|
||||||
part_number,
|
|
||||||
PartShardFingerprint {
|
|
||||||
size: u64::try_from(data.len())?,
|
|
||||||
sha256: sha256_hex(&data),
|
|
||||||
},
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
(numbers, fingerprints)
|
fingerprints
|
||||||
}
|
}
|
||||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => (BTreeSet::new(), BTreeMap::new()),
|
Err(error) if error.kind() == std::io::ErrorKind::NotFound => BTreeMap::new(),
|
||||||
Err(error) => return Err(error.into()),
|
Err(error) => return Err(error.into()),
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -355,8 +361,8 @@ pub(crate) fn census_object_version_on_disk(
|
|||||||
data_dir,
|
data_dir,
|
||||||
erasure_index,
|
erasure_index,
|
||||||
expected_part_numbers,
|
expected_part_numbers,
|
||||||
present_part_numbers,
|
|
||||||
present_part_fingerprints,
|
present_part_fingerprints,
|
||||||
|
inline_data_fingerprint,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -396,3 +402,43 @@ pub async fn signed_admin_post(url: &str, body: Option<&str>, access_key: &str,
|
|||||||
|
|
||||||
Ok(body)
|
Ok(body)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn complete_census() -> VersionShardCensus {
|
||||||
|
VersionShardCensus {
|
||||||
|
version_id: Some("version".to_string()),
|
||||||
|
has_xl_meta: true,
|
||||||
|
data_dir: Some("data-dir".to_string()),
|
||||||
|
erasure_index: Some(3),
|
||||||
|
expected_part_numbers: BTreeSet::from([1]),
|
||||||
|
present_part_fingerprints: BTreeMap::from([(1, shard_fingerprint(b"part").unwrap())]),
|
||||||
|
inline_data_fingerprint: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn shard_fingerprint_uses_physical_length_and_sha256() {
|
||||||
|
assert_eq!(
|
||||||
|
shard_fingerprint(b"abc").unwrap(),
|
||||||
|
PartShardFingerprint {
|
||||||
|
size: 3,
|
||||||
|
sha256: "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad".to_string(),
|
||||||
|
}
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn manifest_requires_matching_inline_payload() {
|
||||||
|
let mut expected = complete_census();
|
||||||
|
expected.expected_part_numbers.clear();
|
||||||
|
expected.present_part_fingerprints.clear();
|
||||||
|
expected.inline_data_fingerprint = Some(shard_fingerprint(b"expected").unwrap());
|
||||||
|
let mut changed = expected.clone();
|
||||||
|
changed.inline_data_fingerprint = Some(shard_fingerprint(b"changed").unwrap());
|
||||||
|
assert!(expected.matches_manifest(&expected));
|
||||||
|
assert!(!changed.matches_manifest(&expected));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -349,11 +349,32 @@ mod tests {
|
|||||||
.send()
|
.send()
|
||||||
.await?;
|
.await?;
|
||||||
|
|
||||||
|
let first_inline = client
|
||||||
|
.put_object()
|
||||||
|
.bucket(bucket)
|
||||||
|
.key("versions/inline.bin")
|
||||||
|
.body(ByteStream::from(payload(8 * 1024, 40)))
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
let first_inline_version = first_inline
|
||||||
|
.version_id()
|
||||||
|
.ok_or("first inline PUT did not return a version ID")?;
|
||||||
|
let second_inline = client
|
||||||
|
.put_object()
|
||||||
|
.bucket(bucket)
|
||||||
|
.key("versions/inline.bin")
|
||||||
|
.body(ByteStream::from(payload(8 * 1024, 41)))
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
let second_inline_version = second_inline
|
||||||
|
.version_id()
|
||||||
|
.ok_or("second inline PUT did not return a version ID")?;
|
||||||
|
|
||||||
let first = client
|
let first = client
|
||||||
.put_object()
|
.put_object()
|
||||||
.bucket(bucket)
|
.bucket(bucket)
|
||||||
.key(key)
|
.key(key)
|
||||||
.body(ByteStream::from(payload(256 * 1024, 41)))
|
.body(ByteStream::from(payload(128 * 1024, 41)))
|
||||||
.send()
|
.send()
|
||||||
.await?;
|
.await?;
|
||||||
let first_version = first.version_id().ok_or("first PUT did not return a version ID")?;
|
let first_version = first.version_id().ok_or("first PUT did not return a version ID")?;
|
||||||
@@ -361,16 +382,36 @@ mod tests {
|
|||||||
.put_object()
|
.put_object()
|
||||||
.bucket(bucket)
|
.bucket(bucket)
|
||||||
.key(key)
|
.key(key)
|
||||||
.body(ByteStream::from(payload(256 * 1024, 42)))
|
.body(ByteStream::from(payload(3 * 1024 * 1024, 42)))
|
||||||
.send()
|
.send()
|
||||||
.await?;
|
.await?;
|
||||||
let second_version = second.version_id().ok_or("second PUT did not return a version ID")?;
|
let second_version = second.version_id().ok_or("second PUT did not return a version ID")?;
|
||||||
let delete = client.delete_object().bucket(bucket).key(key).send().await?;
|
let delete = client.delete_object().bucket(bucket).key(key).send().await?;
|
||||||
let delete_version = delete.version_id().ok_or("delete marker did not return a version ID")?;
|
let delete_version = delete.version_id().ok_or("delete marker did not return a version ID")?;
|
||||||
|
|
||||||
|
let first_inline_census = harness.census_object_version(0, bucket, "versions/inline.bin", Some(first_inline_version))?;
|
||||||
|
let second_inline_census =
|
||||||
|
harness.census_object_version(0, bucket, "versions/inline.bin", Some(second_inline_version))?;
|
||||||
let first_census = harness.census_object_version(0, bucket, key, Some(first_version))?;
|
let first_census = harness.census_object_version(0, bucket, key, Some(first_version))?;
|
||||||
|
let first_other_disk_census = harness.census_object_version(1, bucket, key, Some(first_version))?;
|
||||||
let second_census = harness.census_object_version(0, bucket, key, Some(second_version))?;
|
let second_census = harness.census_object_version(0, bucket, key, Some(second_version))?;
|
||||||
let delete_census = harness.census_object_version(0, bucket, key, Some(delete_version))?;
|
let delete_census = harness.census_object_version(0, bucket, key, Some(delete_version))?;
|
||||||
|
assert!(
|
||||||
|
first_inline_census.is_complete() && second_inline_census.is_complete(),
|
||||||
|
"inline version physical census is incomplete: first={first_inline_census:?} second={second_inline_census:?}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
first_inline_census.present_part_fingerprints.is_empty() && second_inline_census.present_part_fingerprints.is_empty(),
|
||||||
|
"inline versions must not select external shard files: first={first_inline_census:?} second={second_inline_census:?}"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
first_inline_census.inline_data_fingerprint.is_some() && second_inline_census.inline_data_fingerprint.is_some(),
|
||||||
|
"inline versions must fingerprint payload bytes stored in xl.meta"
|
||||||
|
);
|
||||||
|
assert_ne!(
|
||||||
|
first_inline_census.inline_data_fingerprint, second_inline_census.inline_data_fingerprint,
|
||||||
|
"same-size inline versions with different payloads must retain distinct xl.meta fingerprints"
|
||||||
|
);
|
||||||
assert!(
|
assert!(
|
||||||
first_census.is_complete(),
|
first_census.is_complete(),
|
||||||
"first version physical census is incomplete: {first_census:?}"
|
"first version physical census is incomplete: {first_census:?}"
|
||||||
@@ -379,6 +420,14 @@ mod tests {
|
|||||||
second_census.is_complete(),
|
second_census.is_complete(),
|
||||||
"second version physical census is incomplete: {second_census:?}"
|
"second version physical census is incomplete: {second_census:?}"
|
||||||
);
|
);
|
||||||
|
assert!(
|
||||||
|
first_other_disk_census.is_complete(),
|
||||||
|
"first version physical census on the second disk is incomplete: {first_other_disk_census:?}"
|
||||||
|
);
|
||||||
|
assert_ne!(
|
||||||
|
first_census.erasure_index, first_other_disk_census.erasure_index,
|
||||||
|
"physical census must preserve each disk's erasure index"
|
||||||
|
);
|
||||||
assert_ne!(
|
assert_ne!(
|
||||||
first_census.data_dir, second_census.data_dir,
|
first_census.data_dir, second_census.data_dir,
|
||||||
"distinct object versions must select distinct physical data directories"
|
"distinct object versions must select distinct physical data directories"
|
||||||
@@ -387,6 +436,24 @@ mod tests {
|
|||||||
first_census.expected_part_numbers, second_census.expected_part_numbers,
|
first_census.expected_part_numbers, second_census.expected_part_numbers,
|
||||||
"same single-part shape should expose the same part numbers"
|
"same single-part shape should expose the same part numbers"
|
||||||
);
|
);
|
||||||
|
let first_part = first_census
|
||||||
|
.present_part_fingerprints
|
||||||
|
.values()
|
||||||
|
.next()
|
||||||
|
.ok_or("first version did not expose a physical part fingerprint")?;
|
||||||
|
let second_part = second_census
|
||||||
|
.present_part_fingerprints
|
||||||
|
.values()
|
||||||
|
.next()
|
||||||
|
.ok_or("second version did not expose a physical part fingerprint")?;
|
||||||
|
assert_ne!(
|
||||||
|
first_part.size, second_part.size,
|
||||||
|
"different shard lengths must retain their physical sizes"
|
||||||
|
);
|
||||||
|
assert_ne!(
|
||||||
|
first_part.sha256, second_part.sha256,
|
||||||
|
"different shard contents must retain their physical hashes"
|
||||||
|
);
|
||||||
assert!(
|
assert!(
|
||||||
delete_census.is_complete(),
|
delete_census.is_complete(),
|
||||||
"delete marker physical census is incomplete: {delete_census:?}"
|
"delete marker physical census is incomplete: {delete_census:?}"
|
||||||
@@ -396,7 +463,7 @@ mod tests {
|
|||||||
"delete marker must not declare object shards: {delete_census:?}"
|
"delete marker must not declare object shards: {delete_census:?}"
|
||||||
);
|
);
|
||||||
assert!(
|
assert!(
|
||||||
delete_census.present_part_numbers.is_empty(),
|
delete_census.present_part_fingerprints.is_empty(),
|
||||||
"delete marker must not select stale object shards: {delete_census:?}"
|
"delete marker must not select stale object shards: {delete_census:?}"
|
||||||
);
|
);
|
||||||
Ok(())
|
Ok(())
|
||||||
|
|||||||
@@ -59,6 +59,13 @@ mod tests {
|
|||||||
expected: VersionShardCensus,
|
expected: VersionShardCensus,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Eq, PartialEq)]
|
||||||
|
enum CompletionSample {
|
||||||
|
Pending,
|
||||||
|
Ready,
|
||||||
|
CompletedWithIncomplete(BTreeSet<String>),
|
||||||
|
}
|
||||||
|
|
||||||
struct MountNamespaceGuard {
|
struct MountNamespaceGuard {
|
||||||
mounts: Vec<PathBuf>,
|
mounts: Vec<PathBuf>,
|
||||||
}
|
}
|
||||||
@@ -75,7 +82,7 @@ mod tests {
|
|||||||
impl MountNamespaceGuard {
|
impl MountNamespaceGuard {
|
||||||
fn new() -> Result<Self, Box<dyn Error + Send + Sync>> {
|
fn new() -> Result<Self, Box<dyn Error + Send + Sync>> {
|
||||||
verify_isolated_mount_namespace()?;
|
verify_isolated_mount_namespace()?;
|
||||||
run_command("mount", ["--make-rprivate", "/"])?;
|
run_command("mount", &["--make-rprivate", "/"])?;
|
||||||
Ok(Self { mounts: Vec::new() })
|
Ok(Self { mounts: Vec::new() })
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -108,15 +115,15 @@ mod tests {
|
|||||||
return Err("losetup --find --show returned an empty loop device".into());
|
return Err("losetup --find --show returned an empty loop device".into());
|
||||||
}
|
}
|
||||||
|
|
||||||
run_command_dynamic("mkfs.ext4", &["-F", &loop_device])?;
|
run_command("mkfs.ext4", &["-F", &loop_device])?;
|
||||||
let sectors = run_command_stdout("blockdev", &["--getsz", &loop_device])?;
|
let sectors = run_command_stdout("blockdev", &["--getsz", &loop_device])?;
|
||||||
let dm_name = format!("rustfs_e2e_{label}_{}", std::process::id());
|
let dm_name = format!("rustfs_e2e_{label}_{}", std::process::id());
|
||||||
let table = format!("0 {sectors} linear {loop_device} 0");
|
let table = format!("0 {sectors} linear {loop_device} 0");
|
||||||
let mapper = format!("/dev/mapper/{dm_name}");
|
let mapper = format!("/dev/mapper/{dm_name}");
|
||||||
run_command_dynamic("dmsetup", &["create", &dm_name, "--table", &table])?;
|
run_command("dmsetup", &["create", &dm_name, "--table", &table])?;
|
||||||
|
|
||||||
let target_arg = path_to_string(target, "faultable mount target")?;
|
let target_arg = path_to_string(target, "faultable mount target")?;
|
||||||
run_command_dynamic("mount", &[&mapper, &target_arg])?;
|
run_command("mount", &[&mapper, &target_arg])?;
|
||||||
|
|
||||||
Ok(Self {
|
Ok(Self {
|
||||||
target: target.to_path_buf(),
|
target: target.to_path_buf(),
|
||||||
@@ -131,17 +138,17 @@ mod tests {
|
|||||||
fn make_unavailable(&self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
fn make_unavailable(&self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
let sectors = run_command_stdout("blockdev", &["--getsz", &self.loop_device])?;
|
let sectors = run_command_stdout("blockdev", &["--getsz", &self.loop_device])?;
|
||||||
let error_table = format!("0 {sectors} error");
|
let error_table = format!("0 {sectors} error");
|
||||||
run_command_dynamic("dmsetup", &["suspend", &self.dm_name])?;
|
run_command("dmsetup", &["suspend", &self.dm_name])?;
|
||||||
run_command_dynamic("dmsetup", &["load", &self.dm_name, "--table", &error_table])?;
|
run_command("dmsetup", &["load", &self.dm_name, "--table", &error_table])?;
|
||||||
run_command_dynamic("dmsetup", &["resume", &self.dm_name])
|
run_command("dmsetup", &["resume", &self.dm_name])
|
||||||
}
|
}
|
||||||
|
|
||||||
fn restore_available(&self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
fn restore_available(&self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
let sectors = run_command_stdout("blockdev", &["--getsz", &self.loop_device])?;
|
let sectors = run_command_stdout("blockdev", &["--getsz", &self.loop_device])?;
|
||||||
let linear_table = format!("0 {sectors} linear {} 0", self.loop_device);
|
let linear_table = format!("0 {sectors} linear {} 0", self.loop_device);
|
||||||
run_command_dynamic("dmsetup", &["suspend", &self.dm_name])?;
|
run_command("dmsetup", &["suspend", &self.dm_name])?;
|
||||||
run_command_dynamic("dmsetup", &["load", &self.dm_name, "--table", &linear_table])?;
|
run_command("dmsetup", &["load", &self.dm_name, "--table", &linear_table])?;
|
||||||
run_command_dynamic("dmsetup", &["resume", &self.dm_name])
|
run_command("dmsetup", &["resume", &self.dm_name])
|
||||||
}
|
}
|
||||||
|
|
||||||
fn cleanup(&mut self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
fn cleanup(&mut self) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
@@ -157,14 +164,14 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if self.dm_created {
|
if self.dm_created {
|
||||||
if let Err(error) = run_command_dynamic("dmsetup", &["remove", "-f", &self.dm_name]) {
|
if let Err(error) = run_command("dmsetup", &["remove", "-f", &self.dm_name]) {
|
||||||
first_error.get_or_insert(error);
|
first_error.get_or_insert(error);
|
||||||
} else {
|
} else {
|
||||||
self.dm_created = false;
|
self.dm_created = false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !self.loop_device.is_empty() {
|
if !self.loop_device.is_empty() {
|
||||||
if let Err(error) = run_command_dynamic("losetup", &["-d", &self.loop_device]) {
|
if let Err(error) = run_command("losetup", &["-d", &self.loop_device]) {
|
||||||
first_error.get_or_insert(error);
|
first_error.get_or_insert(error);
|
||||||
} else {
|
} else {
|
||||||
self.loop_device.clear();
|
self.loop_device.clear();
|
||||||
@@ -188,14 +195,10 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn run_command<const N: usize>(program: &str, args: [&str; N]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
fn checked_command_output(program: &str, args: &[&str]) -> Result<std::process::Output, Box<dyn Error + Send + Sync>> {
|
||||||
run_command_dynamic(program, &args)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn run_command_dynamic(program: &str, args: &[&str]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
|
||||||
let output = Command::new(program).args(args).output()?;
|
let output = Command::new(program).args(args).output()?;
|
||||||
if output.status.success() {
|
if output.status.success() {
|
||||||
return Ok(());
|
return Ok(output);
|
||||||
}
|
}
|
||||||
Err(format!(
|
Err(format!(
|
||||||
"{program} {} failed with status {}: stdout={} stderr={}",
|
"{program} {} failed with status {}: stdout={} stderr={}",
|
||||||
@@ -207,19 +210,14 @@ mod tests {
|
|||||||
.into())
|
.into())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn run_command(program: &str, args: &[&str]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
|
checked_command_output(program, args).map(drop)
|
||||||
|
}
|
||||||
|
|
||||||
fn run_command_stdout(program: &str, args: &[&str]) -> Result<String, Box<dyn Error + Send + Sync>> {
|
fn run_command_stdout(program: &str, args: &[&str]) -> Result<String, Box<dyn Error + Send + Sync>> {
|
||||||
let output = Command::new(program).args(args).output()?;
|
Ok(String::from_utf8(checked_command_output(program, args)?.stdout)?
|
||||||
if output.status.success() {
|
.trim()
|
||||||
return Ok(String::from_utf8_lossy(&output.stdout).trim().to_string());
|
.to_string())
|
||||||
}
|
|
||||||
Err(format!(
|
|
||||||
"{program} {} failed with status {}: stdout={} stderr={}",
|
|
||||||
args.join(" "),
|
|
||||||
output.status,
|
|
||||||
String::from_utf8_lossy(&output.stdout),
|
|
||||||
String::from_utf8_lossy(&output.stderr)
|
|
||||||
)
|
|
||||||
.into())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn path_to_string(path: &Path, label: &str) -> Result<String, Box<dyn Error + Send + Sync>> {
|
fn path_to_string(path: &Path, label: &str) -> Result<String, Box<dyn Error + Send + Sync>> {
|
||||||
@@ -260,14 +258,14 @@ mod tests {
|
|||||||
let target = target
|
let target = target
|
||||||
.to_str()
|
.to_str()
|
||||||
.ok_or_else(|| format!("tmpfs target path is not UTF-8: {target:?}"))?;
|
.ok_or_else(|| format!("tmpfs target path is not UTF-8: {target:?}"))?;
|
||||||
run_command("mount", ["-t", "tmpfs", "-o", MOUNT_SIZE, label, target])
|
run_command("mount", &["-t", "tmpfs", "-o", MOUNT_SIZE, label, target])
|
||||||
}
|
}
|
||||||
|
|
||||||
fn detach_mount(target: &Path) -> Result<(), Box<dyn Error + Send + Sync>> {
|
fn detach_mount(target: &Path) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
let target = target
|
let target = target
|
||||||
.to_str()
|
.to_str()
|
||||||
.ok_or_else(|| format!("umount target path is not UTF-8: {target:?}"))?;
|
.ok_or_else(|| format!("umount target path is not UTF-8: {target:?}"))?;
|
||||||
run_command("umount", [target])
|
run_command("umount", &[target])
|
||||||
}
|
}
|
||||||
|
|
||||||
fn privileged_run_enabled() -> Result<bool, Box<dyn Error + Send + Sync>> {
|
fn privileged_run_enabled() -> Result<bool, Box<dyn Error + Send + Sync>> {
|
||||||
@@ -423,6 +421,10 @@ mod tests {
|
|||||||
.await?;
|
.await?;
|
||||||
versions.push((versioned_bucket, "history/object.bin", deleted.version_id().map(str::to_owned), None));
|
versions.push((versioned_bucket, "history/object.bin", deleted.version_id().map(str::to_owned), None));
|
||||||
|
|
||||||
|
let (version_id, body_sha256) =
|
||||||
|
put_object_version(client, versioned_bucket, "history/inline.bin", payload(8 * 1024, 9)).await?;
|
||||||
|
versions.push((versioned_bucket, "history/inline.bin", version_id, Some(body_sha256)));
|
||||||
|
|
||||||
let (version_id, body_sha256) = put_multipart_version(
|
let (version_id, body_sha256) = put_multipart_version(
|
||||||
client,
|
client,
|
||||||
versioned_bucket,
|
versioned_bucket,
|
||||||
@@ -436,7 +438,7 @@ mod tests {
|
|||||||
put_object_version(client, null_bucket, "null/current.bin", payload(512 * 1024, 8)).await?;
|
put_object_version(client, null_bucket, "null/current.bin", payload(512 * 1024, 8)).await?;
|
||||||
versions.push((null_bucket, "null/current.bin", version_id, Some(body_sha256)));
|
versions.push((null_bucket, "null/current.bin", version_id, Some(body_sha256)));
|
||||||
|
|
||||||
versions
|
let versions = versions
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|(bucket, key, version_id, body_sha256)| {
|
.map(|(bucket, key, version_id, body_sha256)| {
|
||||||
let expected = census_object_version_on_disk(target_disk, bucket, key, version_id.as_deref())?;
|
let expected = census_object_version_on_disk(target_disk, bucket, key, version_id.as_deref())?;
|
||||||
@@ -451,7 +453,15 @@ mod tests {
|
|||||||
expected,
|
expected,
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
.collect()
|
.collect::<Result<Vec<_>, Box<dyn Error + Send + Sync>>>()?;
|
||||||
|
let inline = versions
|
||||||
|
.iter()
|
||||||
|
.find(|version| version.key == "history/inline.bin")
|
||||||
|
.ok_or("inline replacement baseline was not recorded")?;
|
||||||
|
if inline.expected.inline_data_fingerprint.is_none() || !inline.expected.present_part_fingerprints.is_empty() {
|
||||||
|
return Err(format!("inline replacement baseline lacks xl.meta payload evidence: {:?}", inline.expected).into());
|
||||||
|
}
|
||||||
|
Ok(versions)
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn verify_bodies(client: &Client, versions: &[BaselineVersion]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
async fn verify_bodies(client: &Client, versions: &[BaselineVersion]) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
@@ -597,6 +607,33 @@ mod tests {
|
|||||||
Ok(String::from_utf8_lossy(&log[start..]).into_owned())
|
Ok(String::from_utf8_lossy(&log[start..]).into_owned())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn live_disk_loss_scan_completed(log: &str, target_disk: &Path) -> bool {
|
||||||
|
let target = target_disk.to_string_lossy();
|
||||||
|
let mut saw_live_loss = false;
|
||||||
|
for line in log.lines() {
|
||||||
|
if line.contains("Heal auto-scan disk inspection failed")
|
||||||
|
&& line.contains("check_failed")
|
||||||
|
&& line.contains(target.as_ref())
|
||||||
|
{
|
||||||
|
saw_live_loss = true;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if saw_live_loss && (line.contains("Heal auto disk scanner idle") || line.contains("Heal auto-scan cycle completed"))
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
false
|
||||||
|
}
|
||||||
|
|
||||||
|
fn live_disk_loss_scan_completed_from_path(
|
||||||
|
log_path: &Path,
|
||||||
|
start_offset: u64,
|
||||||
|
target_disk: &Path,
|
||||||
|
) -> Result<bool, Box<dyn Error + Send + Sync>> {
|
||||||
|
Ok(live_disk_loss_scan_completed(&log_from_offset(log_path, start_offset)?, target_disk))
|
||||||
|
}
|
||||||
|
|
||||||
async fn wait_for_live_disk_loss_observation(
|
async fn wait_for_live_disk_loss_observation(
|
||||||
log_path: &Path,
|
log_path: &Path,
|
||||||
target_disk: &Path,
|
target_disk: &Path,
|
||||||
@@ -605,20 +642,12 @@ mod tests {
|
|||||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
let deadline = Instant::now() + Duration::from_secs(timeout_secs);
|
let deadline = Instant::now() + Duration::from_secs(timeout_secs);
|
||||||
let mut tick = interval(Duration::from_secs(1));
|
let mut tick = interval(Duration::from_secs(1));
|
||||||
let target = target_disk.to_string_lossy();
|
|
||||||
loop {
|
loop {
|
||||||
let log = log_from_offset(log_path, start_offset)?;
|
if live_disk_loss_scan_completed_from_path(log_path, start_offset, target_disk)? {
|
||||||
let mut saw_live_loss = false;
|
return Ok(());
|
||||||
for line in log.lines() {
|
|
||||||
if line.contains("check_failed") && line.contains(target.as_ref()) {
|
|
||||||
saw_live_loss = true;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if saw_live_loss && line.contains("Heal auto disk scanner idle") {
|
|
||||||
return Ok(());
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if Instant::now() >= deadline {
|
if Instant::now() >= deadline {
|
||||||
|
let log = log_from_offset(log_path, start_offset)?;
|
||||||
return Err(format!(
|
return Err(format!(
|
||||||
"scanner did not finish a live target-loss scan for {target_disk:?} within {timeout_secs}s; log tail:\n{}",
|
"scanner did not finish a live target-loss scan for {target_disk:?} within {timeout_secs}s; log tail:\n{}",
|
||||||
log_tail(&log)
|
log_tail(&log)
|
||||||
@@ -629,14 +658,10 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn require_definitive_replacement_status(
|
fn cluster_status_is_definitive(status: &serde_json::Value) -> Result<bool, Box<dyn Error + Send + Sync>> {
|
||||||
status: &serde_json::Value,
|
status["cluster"]["definitive"]
|
||||||
context: &str,
|
.as_bool()
|
||||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
.ok_or_else(|| format!("replacement recovery status omitted cluster.definitive: {status}").into())
|
||||||
if status["cluster"]["definitive"].as_bool().unwrap_or(false) {
|
|
||||||
return Ok(());
|
|
||||||
}
|
|
||||||
Err(format!("{context} requires a definitive cluster replacement status: {status}").into())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn assert_no_replacement_status_records(
|
async fn assert_no_replacement_status_records(
|
||||||
@@ -644,8 +669,17 @@ mod tests {
|
|||||||
target_disk: &Path,
|
target_disk: &Path,
|
||||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
let status = replacement_status(cluster).await?;
|
let status = replacement_status(cluster).await?;
|
||||||
require_definitive_replacement_status(&status, "live missing replacement status check")?;
|
assert_no_replacement_status_records_in_status(&status, target_disk)
|
||||||
let states = target_record_states(&status, target_disk);
|
}
|
||||||
|
|
||||||
|
fn assert_no_replacement_status_records_in_status(
|
||||||
|
status: &serde_json::Value,
|
||||||
|
target_disk: &Path,
|
||||||
|
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
|
if !cluster_status_is_definitive(status)? {
|
||||||
|
return Err(format!("live missing replacement status check requires a definitive cluster status: {status}").into());
|
||||||
|
}
|
||||||
|
let states = target_record_states(status, target_disk);
|
||||||
if states.is_empty() {
|
if states.is_empty() {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
@@ -655,11 +689,6 @@ mod tests {
|
|||||||
.into())
|
.into())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn target_record_has_state(status: &serde_json::Value, target_disk: &Path, states: &[&str]) -> bool {
|
|
||||||
let present_states = target_record_states(status, target_disk);
|
|
||||||
states.iter().any(|state| present_states.contains(*state))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn target_record_states(status: &serde_json::Value, target_disk: &Path) -> BTreeSet<String> {
|
fn target_record_states(status: &serde_json::Value, target_disk: &Path) -> BTreeSet<String> {
|
||||||
let target = target_disk.to_string_lossy();
|
let target = target_disk.to_string_lossy();
|
||||||
status["cluster"]["records"]
|
status["cluster"]["records"]
|
||||||
@@ -694,6 +723,57 @@ mod tests {
|
|||||||
Ok(missing)
|
Ok(missing)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn replacement_completion_state(
|
||||||
|
status: &serde_json::Value,
|
||||||
|
target_disk: &Path,
|
||||||
|
missing: BTreeSet<String>,
|
||||||
|
) -> Result<CompletionSample, Box<dyn Error + Send + Sync>> {
|
||||||
|
if !cluster_status_is_definitive(status)? {
|
||||||
|
return Ok(CompletionSample::Pending);
|
||||||
|
}
|
||||||
|
if !target_record_states(status, target_disk).contains("completed") {
|
||||||
|
return Ok(CompletionSample::Pending);
|
||||||
|
}
|
||||||
|
if missing.is_empty() {
|
||||||
|
return Ok(CompletionSample::Ready);
|
||||||
|
}
|
||||||
|
Ok(CompletionSample::CompletedWithIncomplete(missing))
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn sample_replacement_completion<C, S, F>(
|
||||||
|
target_disk: &Path,
|
||||||
|
census: C,
|
||||||
|
status: S,
|
||||||
|
) -> Result<CompletionSample, Box<dyn Error + Send + Sync>>
|
||||||
|
where
|
||||||
|
C: FnOnce() -> Result<BTreeSet<String>, Box<dyn Error + Send + Sync>>,
|
||||||
|
S: FnOnce() -> F,
|
||||||
|
F: std::future::Future<Output = Result<serde_json::Value, Box<dyn Error + Send + Sync>>>,
|
||||||
|
{
|
||||||
|
let missing = census()?;
|
||||||
|
let status = status().await?;
|
||||||
|
replacement_completion_state(&status, target_disk, missing)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn confirm_replacement_completion<C, S, F>(
|
||||||
|
target_disk: &Path,
|
||||||
|
mut census: C,
|
||||||
|
mut status: S,
|
||||||
|
) -> Result<CompletionSample, Box<dyn Error + Send + Sync>>
|
||||||
|
where
|
||||||
|
C: FnMut() -> Result<BTreeSet<String>, Box<dyn Error + Send + Sync>>,
|
||||||
|
S: FnMut() -> F,
|
||||||
|
F: std::future::Future<Output = Result<serde_json::Value, Box<dyn Error + Send + Sync>>>,
|
||||||
|
{
|
||||||
|
let missing = census()?;
|
||||||
|
let status = status().await?;
|
||||||
|
let first = replacement_completion_state(&status, target_disk, missing)?;
|
||||||
|
if matches!(first, CompletionSample::CompletedWithIncomplete(_)) {
|
||||||
|
return replacement_completion_state(&status, target_disk, census()?);
|
||||||
|
}
|
||||||
|
Ok(first)
|
||||||
|
}
|
||||||
|
|
||||||
async fn wait_for_completed_replacement_with_census(
|
async fn wait_for_completed_replacement_with_census(
|
||||||
cluster: &RustFSTestClusterEnvironment,
|
cluster: &RustFSTestClusterEnvironment,
|
||||||
target_disk: &Path,
|
target_disk: &Path,
|
||||||
@@ -703,28 +783,25 @@ mod tests {
|
|||||||
let deadline = Instant::now() + Duration::from_secs(timeout_secs);
|
let deadline = Instant::now() + Duration::from_secs(timeout_secs);
|
||||||
let mut tick = interval(Duration::from_secs(1));
|
let mut tick = interval(Duration::from_secs(1));
|
||||||
loop {
|
loop {
|
||||||
let status = replacement_status(cluster).await?;
|
match confirm_replacement_completion(
|
||||||
let missing = incomplete_versions(target_disk, versions)?;
|
target_disk,
|
||||||
if require_definitive_replacement_status(&status, "replacement completion poll").is_err() {
|
|| incomplete_versions(target_disk, versions),
|
||||||
if Instant::now() >= deadline {
|
|| replacement_status(cluster),
|
||||||
|
)
|
||||||
|
.await?
|
||||||
|
{
|
||||||
|
CompletionSample::Ready => return Ok(()),
|
||||||
|
CompletionSample::CompletedWithIncomplete(confirmed_missing) => {
|
||||||
return Err(format!(
|
return Err(format!(
|
||||||
"replacement recovery status never became definitive within {timeout_secs}s while waiting for physical census; latest status: {status}; missing: {missing:?}"
|
"replacement status remained completed across two incomplete physical censuses: {confirmed_missing:?}"
|
||||||
)
|
)
|
||||||
.into());
|
.into());
|
||||||
}
|
}
|
||||||
tick.tick().await;
|
CompletionSample::Pending => {}
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if target_record_has_state(&status, target_disk, &["completed"]) {
|
|
||||||
if !missing.is_empty() {
|
|
||||||
return Err(format!(
|
|
||||||
"replacement status reached completed before target physical census matched baseline: {missing:?}; status: {status}"
|
|
||||||
)
|
|
||||||
.into());
|
|
||||||
}
|
|
||||||
return Ok(());
|
|
||||||
}
|
}
|
||||||
if Instant::now() >= deadline {
|
if Instant::now() >= deadline {
|
||||||
|
let missing = incomplete_versions(target_disk, versions)?;
|
||||||
|
let status = replacement_status(cluster).await?;
|
||||||
return Err(format!(
|
return Err(format!(
|
||||||
"replacement target did not reach completed with matching physical census within {timeout_secs}s: missing={missing:?}; status={status}"
|
"replacement target did not reach completed with matching physical census within {timeout_secs}s: missing={missing:?}; status={status}"
|
||||||
)
|
)
|
||||||
@@ -812,6 +889,175 @@ mod tests {
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn live_loss_barrier_requires_scanner_failure_after_log_offset() -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||||
|
let target = Path::new("/mnt/target");
|
||||||
|
assert!(live_disk_loss_scan_completed(
|
||||||
|
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto-scan cycle completed",
|
||||||
|
target
|
||||||
|
));
|
||||||
|
assert!(live_disk_loss_scan_completed(
|
||||||
|
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle",
|
||||||
|
target
|
||||||
|
));
|
||||||
|
assert!(!live_disk_loss_scan_completed(
|
||||||
|
"Heal auto disk scanner idle\nHeal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed",
|
||||||
|
target
|
||||||
|
));
|
||||||
|
assert!(!live_disk_loss_scan_completed(
|
||||||
|
"event=disk_health_check_failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle",
|
||||||
|
target
|
||||||
|
));
|
||||||
|
assert!(!live_disk_loss_scan_completed(
|
||||||
|
"Heal auto-scan disk inspection failed endpoint=/mnt/other disk_state=check_failed\nHeal auto disk scanner idle",
|
||||||
|
target
|
||||||
|
));
|
||||||
|
let path = std::env::temp_dir().join(format!("rustfs-replacement-scan-{}.log", std::process::id()));
|
||||||
|
let stale =
|
||||||
|
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle\n";
|
||||||
|
fs::write(&path, stale)?;
|
||||||
|
let offset = log_len(&path)?;
|
||||||
|
assert!(!live_disk_loss_scan_completed_from_path(&path, offset, target)?);
|
||||||
|
let fresh =
|
||||||
|
"Heal auto-scan disk inspection failed endpoint=/mnt/target disk_state=check_failed\nHeal auto disk scanner idle\n";
|
||||||
|
fs::write(&path, format!("{stale}{fresh}"))?;
|
||||||
|
assert!(live_disk_loss_scan_completed_from_path(&path, offset, target)?);
|
||||||
|
fs::remove_file(path)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn completion_requires_definitive_status_and_prior_census_match() {
|
||||||
|
let target = Path::new("/mnt/target");
|
||||||
|
let non_definitive = serde_json::json!({
|
||||||
|
"cluster": {"definitive": false, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
replacement_completion_state(&non_definitive, target, BTreeSet::new()).unwrap(),
|
||||||
|
CompletionSample::Pending
|
||||||
|
);
|
||||||
|
|
||||||
|
let omitted = serde_json::json!({
|
||||||
|
"cluster": {"records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||||
|
});
|
||||||
|
assert!(replacement_completion_state(&omitted, target, BTreeSet::new()).is_err());
|
||||||
|
|
||||||
|
let definitive = serde_json::json!({
|
||||||
|
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||||
|
});
|
||||||
|
assert_eq!(
|
||||||
|
replacement_completion_state(&definitive, target, BTreeSet::from(["missing".to_string()])).unwrap(),
|
||||||
|
CompletionSample::CompletedWithIncomplete(BTreeSet::from(["missing".to_string()]))
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
replacement_completion_state(&definitive, target, BTreeSet::new()).unwrap(),
|
||||||
|
CompletionSample::Ready
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn completion_poll_samples_census_before_status() {
|
||||||
|
let order = std::rc::Rc::new(std::cell::RefCell::new(Vec::new()));
|
||||||
|
let census_order = order.clone();
|
||||||
|
let status_order = order.clone();
|
||||||
|
let sample = sample_replacement_completion(
|
||||||
|
Path::new("/mnt/target"),
|
||||||
|
move || {
|
||||||
|
census_order.borrow_mut().push("census");
|
||||||
|
Ok::<_, Box<dyn Error + Send + Sync>>(BTreeSet::new())
|
||||||
|
},
|
||||||
|
move || async move {
|
||||||
|
status_order.borrow_mut().push("status");
|
||||||
|
Ok::<_, Box<dyn Error + Send + Sync>>(serde_json::json!({
|
||||||
|
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||||
|
}))
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(sample, CompletionSample::Ready);
|
||||||
|
assert_eq!(*order.borrow(), ["census", "status"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn completed_status_confirms_a_stale_incomplete_census() {
|
||||||
|
let samples = std::rc::Rc::new(std::cell::RefCell::new(std::collections::VecDeque::from([
|
||||||
|
BTreeSet::from(["missing".to_string()]),
|
||||||
|
BTreeSet::new(),
|
||||||
|
])));
|
||||||
|
let census_samples = samples.clone();
|
||||||
|
let result = confirm_replacement_completion(
|
||||||
|
Path::new("/mnt/target"),
|
||||||
|
move || {
|
||||||
|
census_samples
|
||||||
|
.borrow_mut()
|
||||||
|
.pop_front()
|
||||||
|
.ok_or_else(|| "missing census sample".into())
|
||||||
|
},
|
||||||
|
|| async {
|
||||||
|
Ok(serde_json::json!({
|
||||||
|
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||||
|
}))
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(result, CompletionSample::Ready);
|
||||||
|
assert!(samples.borrow().is_empty());
|
||||||
|
|
||||||
|
let status_samples = std::rc::Rc::new(std::cell::RefCell::new(std::collections::VecDeque::from([
|
||||||
|
serde_json::json!({
|
||||||
|
"cluster": {"definitive": true, "records": [{"state": "completed", "targetSlots": ["/mnt/target"]}]}
|
||||||
|
}),
|
||||||
|
serde_json::json!({
|
||||||
|
"cluster": {"definitive": false, "records": []}
|
||||||
|
}),
|
||||||
|
])));
|
||||||
|
let persistent = std::rc::Rc::new(std::cell::RefCell::new(std::collections::VecDeque::from([
|
||||||
|
BTreeSet::from(["missing".to_string()]),
|
||||||
|
BTreeSet::from(["still-missing".to_string()]),
|
||||||
|
])));
|
||||||
|
let census_samples = persistent.clone();
|
||||||
|
let statuses = status_samples.clone();
|
||||||
|
let result = confirm_replacement_completion(
|
||||||
|
Path::new("/mnt/target"),
|
||||||
|
move || {
|
||||||
|
census_samples
|
||||||
|
.borrow_mut()
|
||||||
|
.pop_front()
|
||||||
|
.ok_or_else(|| "missing census sample".into())
|
||||||
|
},
|
||||||
|
move || {
|
||||||
|
let statuses = statuses.clone();
|
||||||
|
async move {
|
||||||
|
statuses
|
||||||
|
.borrow_mut()
|
||||||
|
.pop_front()
|
||||||
|
.ok_or_else(|| "missing status sample".into())
|
||||||
|
}
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
result,
|
||||||
|
CompletionSample::CompletedWithIncomplete(BTreeSet::from(["still-missing".to_string()]))
|
||||||
|
);
|
||||||
|
assert!(persistent.borrow().is_empty());
|
||||||
|
assert_eq!(status_samples.borrow().len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn absent_status_requires_definitive_empty_records() {
|
||||||
|
let target = Path::new("/mnt/target");
|
||||||
|
let non_definitive = serde_json::json!({"cluster": {"definitive": false, "records": []}});
|
||||||
|
assert!(assert_no_replacement_status_records_in_status(&non_definitive, target).is_err());
|
||||||
|
let omitted = serde_json::json!({"cluster": {"records": []}});
|
||||||
|
assert!(assert_no_replacement_status_records_in_status(&omitted, target).is_err());
|
||||||
|
let definitive = serde_json::json!({"cluster": {"definitive": true, "records": []}});
|
||||||
|
assert!(assert_no_replacement_status_records_in_status(&definitive, target).is_ok());
|
||||||
|
}
|
||||||
|
|
||||||
/// Linux mount namespaces are per-thread; keep mount setup and process
|
/// Linux mount namespaces are per-thread; keep mount setup and process
|
||||||
/// spawning on one OS thread so child RustFS nodes inherit the test mounts.
|
/// spawning on one OS thread so child RustFS nodes inherit the test mounts.
|
||||||
#[tokio::test(flavor = "current_thread")]
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
|||||||
@@ -2150,30 +2150,6 @@ pub async fn load_data_usage_cache(store: &crate::set_disk::SetDisks, name: &str
|
|||||||
Ok(d)
|
Ok(d)
|
||||||
}
|
}
|
||||||
|
|
||||||
#[instrument(skip(cache))]
|
|
||||||
pub async fn save_data_usage_cache(cache: &DataUsageCache, name: &str) -> crate::error::Result<()> {
|
|
||||||
use crate::config::com::save_config;
|
|
||||||
use crate::disk::BUCKET_META_PREFIX;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
let Some(store) = runtime_sources::object_store_handle() else {
|
|
||||||
return Err(Error::other("errServerNotInitialized"));
|
|
||||||
};
|
|
||||||
let buf = cache.marshal_msg().map_err(Error::other)?;
|
|
||||||
let buf_clone = buf.clone();
|
|
||||||
|
|
||||||
let store_clone = store.clone();
|
|
||||||
|
|
||||||
let name = Path::new(BUCKET_META_PREFIX).join(name).to_string_lossy().to_string();
|
|
||||||
|
|
||||||
let name_clone = name.clone();
|
|
||||||
tokio::spawn(async move {
|
|
||||||
let _ = save_config(store_clone, &format!("{}{}", name_clone, ".bkp"), buf_clone).await;
|
|
||||||
});
|
|
||||||
save_config(store, &name, buf).await?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Persist the current in-memory compression total to the backend.
|
/// Persist the current in-memory compression total to the backend.
|
||||||
/// Resets the debounce counter so the next auto-persist won't fire
|
/// Resets the debounce counter so the next auto-persist won't fire
|
||||||
/// immediately after this manual flush (intended for shutdown paths).
|
/// immediately after this manual flush (intended for shutdown paths).
|
||||||
|
|||||||
@@ -331,7 +331,14 @@ impl From<std::io::Error> for DiskError {
|
|||||||
}
|
}
|
||||||
match e.downcast::<DiskError>() {
|
match e.downcast::<DiskError>() {
|
||||||
Ok(disk_error) => disk_error,
|
Ok(disk_error) => disk_error,
|
||||||
Err(io_error) => DiskError::Io(io_error),
|
// Mirror `From<io::Error> for StorageError`: a StorageError boxed
|
||||||
|
// through `From<StorageError> for io::Error` must recover its typed
|
||||||
|
// classification instead of degrading to `DiskError::Io`, which
|
||||||
|
// quorum aggregation (`reduce_errs`) would count as a distinct error.
|
||||||
|
Err(io_error) => match io_error.downcast::<crate::error::StorageError>() {
|
||||||
|
Ok(storage_error) => storage_error.into(),
|
||||||
|
Err(io_error) => DiskError::Io(io_error),
|
||||||
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -953,6 +960,27 @@ mod tests {
|
|||||||
assert_eq!(original_disk_error, recovered_disk_error);
|
assert_eq!(original_disk_error, recovered_disk_error);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_io_error_with_storage_error_inside() {
|
||||||
|
use crate::error::StorageError;
|
||||||
|
|
||||||
|
// An io::Error boxing a disk-representable StorageError (as produced by
|
||||||
|
// `From<StorageError> for io::Error`) must recover the typed DiskError
|
||||||
|
// variant instead of degrading to an opaque DiskError::Io.
|
||||||
|
let io_with_storage_error: std::io::Error = StorageError::FaultyRemoteDisk.into();
|
||||||
|
let recovered: DiskError = io_with_storage_error.into();
|
||||||
|
assert_eq!(recovered, DiskError::FaultyRemoteDisk);
|
||||||
|
|
||||||
|
let io_with_storage_error: std::io::Error = StorageError::FileAccessDenied.into();
|
||||||
|
let recovered: DiskError = io_with_storage_error.into();
|
||||||
|
assert_eq!(recovered, DiskError::FileAccessDenied);
|
||||||
|
|
||||||
|
// A StorageError with no DiskError analog stays an opaque Io error.
|
||||||
|
let io_with_bucket_error: std::io::Error = StorageError::BucketNotFound("bucket".to_string()).into();
|
||||||
|
let recovered: DiskError = io_with_bucket_error.into();
|
||||||
|
assert!(matches!(recovered, DiskError::Io(_)));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_io_error_different_kinds() {
|
fn test_io_error_different_kinds() {
|
||||||
use std::io::ErrorKind;
|
use std::io::ErrorKind;
|
||||||
|
|||||||
@@ -2955,11 +2955,31 @@ impl std::fmt::Debug for StdBackend {
|
|||||||
|
|
||||||
impl StdBackend {
|
impl StdBackend {
|
||||||
pub(crate) fn new(root: PathBuf) -> Self {
|
pub(crate) fn new(root: PathBuf) -> Self {
|
||||||
|
Self::build(root, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Construct without the descriptor cache.
|
||||||
|
///
|
||||||
|
/// `UringBackend` wraps a `StdBackend` and runs its own `FdCache` over the
|
||||||
|
/// same positioned reads. If the inner `StdBackend` also built a cache, a
|
||||||
|
/// fallback read (`UringBackend::pread_bytes` delegates to the inner backend
|
||||||
|
/// on latch-off / O_DIRECT / buffered errors) would populate a *second*
|
||||||
|
/// cache that `UringBackend`'s invalidation never touches — re-opening the
|
||||||
|
/// stale-inode hazard `FdCache` exists to close (rustfs/backlog#1176/#1801).
|
||||||
|
/// The wrapper therefore owns the only cache for the disk; the inner backend
|
||||||
|
/// opens per read. This also avoids double-counting `FD_CACHE_CAPACITY`
|
||||||
|
/// against `RLIMIT_NOFILE` (rustfs/backlog#1178).
|
||||||
|
#[cfg(target_os = "linux")]
|
||||||
|
pub(crate) fn new_without_fd_cache(root: PathBuf) -> Self {
|
||||||
|
Self::build(root, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build(root: PathBuf, build_fd_cache: bool) -> Self {
|
||||||
// Gate the fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178):
|
// Gate the fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178):
|
||||||
// 512 fds/disk with a low soft limit and several disks would hit EMFILE.
|
// 512 fds/disk with a low soft limit and several disks would hit EMFILE.
|
||||||
// Fall back to open-per-read when the limit is too small.
|
// Fall back to open-per-read when the limit is too small.
|
||||||
#[cfg(target_os = "linux")]
|
#[cfg(target_os = "linux")]
|
||||||
let fd_cache = if is_local_fd_cache_enabled() {
|
let fd_cache = if build_fd_cache && is_local_fd_cache_enabled() {
|
||||||
if rlimit_allows_fd_cache() {
|
if rlimit_allows_fd_cache() {
|
||||||
Some(FdCache::new())
|
Some(FdCache::new())
|
||||||
} else {
|
} else {
|
||||||
@@ -2973,6 +2993,10 @@ impl StdBackend {
|
|||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
};
|
};
|
||||||
|
// `build_fd_cache` is only consulted on Linux (for the fd cache); on
|
||||||
|
// other platforms it has no effect and would trip the unused-variable lint.
|
||||||
|
#[cfg(not(target_os = "linux"))]
|
||||||
|
let _ = build_fd_cache;
|
||||||
Self {
|
Self {
|
||||||
root,
|
root,
|
||||||
#[cfg(target_os = "linux")]
|
#[cfg(target_os = "linux")]
|
||||||
@@ -4120,7 +4144,7 @@ impl UringBackend {
|
|||||||
// struct (rustfs/backlog#1185).
|
// struct (rustfs/backlog#1185).
|
||||||
let root_label = root.display().to_string();
|
let root_label = root.display().to_string();
|
||||||
Some(Self {
|
Some(Self {
|
||||||
inner: StdBackend::new(root.clone()),
|
inner: StdBackend::new_without_fd_cache(root.clone()),
|
||||||
root,
|
root,
|
||||||
root_label,
|
root_label,
|
||||||
driver: std::mem::ManuallyDrop::new(driver),
|
driver: std::mem::ManuallyDrop::new(driver),
|
||||||
@@ -19919,6 +19943,23 @@ mod test {
|
|||||||
assert_eq!(cache.entry_count().await, 0, "prefix invalidation must drop the cached descriptor");
|
assert_eq!(cache.entry_count().await, 0, "prefix invalidation must drop the cached descriptor");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `StdBackend::new_without_fd_cache` must not build a descriptor cache.
|
||||||
|
/// `UringBackend` wraps a `StdBackend` and owns the only cache for the disk,
|
||||||
|
/// so an inner cache would be populated by fallback reads
|
||||||
|
/// (`UringBackend::pread_bytes` delegates inward) yet never invalidated —
|
||||||
|
/// the stale-inode hazard `FdCache` exists to close (backlog#1176/#1801).
|
||||||
|
/// This pins the contract so a future constructor change cannot regress it.
|
||||||
|
#[cfg(target_os = "linux")]
|
||||||
|
#[test]
|
||||||
|
fn new_without_fd_cache_builds_no_descriptor_cache() {
|
||||||
|
let root_dir = tempfile::tempdir().expect("operation should succeed");
|
||||||
|
let backend = StdBackend::new_without_fd_cache(root_dir.path().to_path_buf());
|
||||||
|
assert!(
|
||||||
|
backend.fd_cache.is_none(),
|
||||||
|
"new_without_fd_cache must not build a descriptor cache — UringBackend owns the only cache for the disk"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/// The mutation paths on `LocalDisk` must actually call
|
/// The mutation paths on `LocalDisk` must actually call
|
||||||
/// `invalidate_cached_fds`, not merely have it available (backlog#1145).
|
/// `invalidate_cached_fds`, not merely have it available (backlog#1145).
|
||||||
/// `rename_file` replaces the inode at a path a reader has already cached;
|
/// `rename_file` replaces the inode at a path a reader has already cached;
|
||||||
|
|||||||
@@ -358,6 +358,13 @@ impl From<StorageError> for DiskError {
|
|||||||
StorageError::VolumeNotFound => DiskError::VolumeNotFound,
|
StorageError::VolumeNotFound => DiskError::VolumeNotFound,
|
||||||
StorageError::VolumeExists => DiskError::VolumeExists,
|
StorageError::VolumeExists => DiskError::VolumeExists,
|
||||||
StorageError::FileNameTooLong => DiskError::FileNameTooLong,
|
StorageError::FileNameTooLong => DiskError::FileNameTooLong,
|
||||||
|
StorageError::FaultyRemoteDisk => DiskError::FaultyRemoteDisk,
|
||||||
|
StorageError::DiskAccessDenied => DiskError::DiskAccessDenied,
|
||||||
|
StorageError::DriveIsRoot => DiskError::DriveIsRoot,
|
||||||
|
StorageError::IsNotRegular => DiskError::IsNotRegular,
|
||||||
|
StorageError::VolumeNotEmpty => DiskError::VolumeNotEmpty,
|
||||||
|
StorageError::VolumeAccessDenied => DiskError::VolumeAccessDenied,
|
||||||
|
StorageError::FileAccessDenied => DiskError::FileAccessDenied,
|
||||||
_ => DiskError::other(val),
|
_ => DiskError::other(val),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1492,6 +1499,49 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Every DiskError variant must survive DiskError -> StorageError -> DiskError
|
||||||
|
// unchanged. A variant that degrades to `DiskError::Io` on the way back loses
|
||||||
|
// its identity for quorum aggregation (`reduce_errs` classifies by variant
|
||||||
|
// equality), so ignore-list entries such as FaultyRemoteDisk and
|
||||||
|
// DiskAccessDenied would silently stop matching.
|
||||||
|
#[test]
|
||||||
|
fn test_disk_error_storage_error_round_trip_identity_all_variants() {
|
||||||
|
// DiskError codes are contiguous from 0x01, so enumerating via from_u32
|
||||||
|
// covers every variant and picks up newly appended ones automatically.
|
||||||
|
let all_variants: Vec<DiskError> = (1u32..).map_while(DiskError::from_u32).collect();
|
||||||
|
assert!(
|
||||||
|
all_variants.len() >= 42,
|
||||||
|
"DiskError variant enumeration shrank: got {}, expected at least 42",
|
||||||
|
all_variants.len()
|
||||||
|
);
|
||||||
|
|
||||||
|
for original in all_variants {
|
||||||
|
let storage_error: StorageError = original.clone().into();
|
||||||
|
let round_tripped: DiskError = storage_error.into();
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
std::mem::discriminant(&original),
|
||||||
|
std::mem::discriminant(&round_tripped),
|
||||||
|
"round trip changed variant: {original:?} -> {round_tripped:?}"
|
||||||
|
);
|
||||||
|
assert_eq!(original, round_tripped, "round trip not identical for {original:?}");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Io is the only payload-carrying variant: a representative kind and
|
||||||
|
// message must both survive the round trip.
|
||||||
|
let io_original = DiskError::Io(IoError::new(ErrorKind::PermissionDenied, "denied"));
|
||||||
|
let storage_error: StorageError = io_original.clone().into();
|
||||||
|
let io_round_tripped: DiskError = storage_error.into();
|
||||||
|
assert_eq!(io_original, io_round_tripped);
|
||||||
|
match io_round_tripped {
|
||||||
|
DiskError::Io(inner) => {
|
||||||
|
assert_eq!(inner.kind(), ErrorKind::PermissionDenied);
|
||||||
|
assert_eq!(inner.to_string(), "denied");
|
||||||
|
}
|
||||||
|
other => panic!("expected DiskError::Io, got {other:?}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_storage_error_from_io_error() {
|
fn test_storage_error_from_io_error() {
|
||||||
// Test direct IO error conversion
|
// Test direct IO error conversion
|
||||||
|
|||||||
@@ -9179,6 +9179,71 @@ mod tests {
|
|||||||
assert_eq!(body.as_ref(), payload);
|
assert_eq!(body.as_ref(), payload);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn direct_memory_versioned_bucket_uses_inline_data_shards_for_latest() {
|
||||||
|
let tempdir = tempfile::tempdir().expect("tempdir should be created");
|
||||||
|
let endpoint =
|
||||||
|
Endpoint::try_from(tempdir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
|
||||||
|
let disk = new_disk(
|
||||||
|
&endpoint,
|
||||||
|
&DiskOption {
|
||||||
|
cleanup: false,
|
||||||
|
health_check: false,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("disk should be created");
|
||||||
|
|
||||||
|
let payload = vec![b'v'; 64 * 1024];
|
||||||
|
let payload_size = i64::try_from(payload.len()).expect("test payload size should fit i64");
|
||||||
|
let (erasure, files, _read_length, _checksum_algo) = inline_bitrot_files_for_payload(&payload).await;
|
||||||
|
let mut fi = FileInfo::new("bucket/object", erasure.data_shards, erasure.parity_shards);
|
||||||
|
fi.size = payload_size;
|
||||||
|
fi.data = files[0].data.clone();
|
||||||
|
fi.add_object_part(1, String::new(), payload.len(), None, payload_size, None, None);
|
||||||
|
|
||||||
|
let mut object_info = ObjectInfo {
|
||||||
|
size: payload_size,
|
||||||
|
actual_size: payload_size,
|
||||||
|
parts: Arc::new(vec![ObjectPartInfo {
|
||||||
|
number: 1,
|
||||||
|
size: payload.len(),
|
||||||
|
actual_size: payload_size,
|
||||||
|
..Default::default()
|
||||||
|
}]),
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
object_info.inlined = true;
|
||||||
|
let opts = ObjectOptions {
|
||||||
|
versioned: true,
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
let metrics_size_bucket = rustfs_io_metrics::get_object_size_bucket(fi.size);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
get_small_object_direct_memory_decision_with_threshold(&None, &object_info, &fi, &opts, true, 128 * 1024),
|
||||||
|
GetDirectMemoryDecision::Use {
|
||||||
|
object_size: payload.len()
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
let body = SetDisks::try_get_object_direct_data_shards_with_fileinfo(
|
||||||
|
"bucket",
|
||||||
|
"object",
|
||||||
|
&fi,
|
||||||
|
&files,
|
||||||
|
&vec![Some(disk); erasure.total_shard_count()],
|
||||||
|
true,
|
||||||
|
GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART,
|
||||||
|
metrics_size_bucket,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("versioned latest direct-memory read should not fail")
|
||||||
|
.expect("versioned latest should use inline data shards");
|
||||||
|
|
||||||
|
assert_eq!(body.as_ref(), payload);
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn direct_memory_data_shards_direct_read_reassembles_single_block_payload() {
|
async fn direct_memory_data_shards_direct_read_reassembles_single_block_payload() {
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
|
|||||||
@@ -221,9 +221,9 @@ impl SetDisks {
|
|||||||
let disks = self.disks.read().await.clone();
|
let disks = self.disks.read().await.clone();
|
||||||
let required_reads = self.default_read_quorum();
|
let required_reads = self.default_read_quorum();
|
||||||
|
|
||||||
let bucket = bucket.to_string();
|
let bucket: Arc<str> = Arc::from(bucket);
|
||||||
let object = object.to_string();
|
let object: Arc<str> = Arc::from(object);
|
||||||
let version_id = version_id.to_string();
|
let version_id: Arc<str> = Arc::from(version_id);
|
||||||
let opts = *opts;
|
let opts = *opts;
|
||||||
|
|
||||||
let processor = runtime_sources::batch_processors().read_processor();
|
let processor = runtime_sources::batch_processors().read_processor();
|
||||||
|
|||||||
@@ -2403,6 +2403,150 @@ mod tests {
|
|||||||
assert_eq!(decoded.cache.get("bucket").map(|entry| entry.objects), Some(3));
|
assert_eq!(decoded.cache.get("bucket").map(|entry| entry.objects), Some(3));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Deterministic, fully populated cache used to pin the persisted
|
||||||
|
/// `.usage-cache.bin` wire bytes. Every map/set holds at most one element
|
||||||
|
/// so the map-encoded `marshal_msg` output is byte-stable.
|
||||||
|
fn wire_fixture_cache() -> DataUsageCache {
|
||||||
|
let mut entry = DataUsageEntry {
|
||||||
|
size: 4096,
|
||||||
|
objects: 3,
|
||||||
|
versions: 5,
|
||||||
|
delete_markers: 1,
|
||||||
|
compacted: true,
|
||||||
|
failed_objects: 2,
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
entry.add_tier_sizes(&HashMap::from([(
|
||||||
|
"WARM".to_string(),
|
||||||
|
TierStats {
|
||||||
|
total_size: 2048,
|
||||||
|
num_versions: 2,
|
||||||
|
num_objects: 1,
|
||||||
|
},
|
||||||
|
)]));
|
||||||
|
let mut cache = DataUsageCache {
|
||||||
|
info: DataUsageCacheInfo {
|
||||||
|
name: "wire-bucket".to_string(),
|
||||||
|
next_cycle: 7,
|
||||||
|
leader_epoch: 9,
|
||||||
|
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000)),
|
||||||
|
skip_healing: true,
|
||||||
|
failed_objects: HashMap::from([("wire-bucket/lost".to_string(), 11)]),
|
||||||
|
scan_resume_after: Some("wire-bucket/resume".to_string()),
|
||||||
|
pending_heals: vec![PendingScannerHeal {
|
||||||
|
kind: PendingScannerHealKind::Object,
|
||||||
|
bucket: "wire-bucket".to_string(),
|
||||||
|
object: Some("broken".to_string()),
|
||||||
|
version_id: None,
|
||||||
|
scan_mode: HealScanMode::Normal,
|
||||||
|
first_seen: 100,
|
||||||
|
last_attempt: 200,
|
||||||
|
attempts: 3,
|
||||||
|
last_admission_result: "deferred".to_string(),
|
||||||
|
last_admission_reason: "budget".to_string(),
|
||||||
|
}],
|
||||||
|
source: Some(DataUsageCacheSource::new(1, 2)),
|
||||||
|
snapshot_complete: true,
|
||||||
|
scan_plan_digest: Some(TEST_PLAN_DIGEST),
|
||||||
|
cache_key_format: DATA_USAGE_CACHE_KEY_FORMAT,
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
cache.replace("wire-bucket", "", entry);
|
||||||
|
cache
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Persisted `.usage-cache.bin` bytes produced by [`wire_fixture_cache`]
|
||||||
|
/// via `DataUsageCache::marshal_msg`: a 2-element array of the 16-field
|
||||||
|
/// map-encoded info block and the map of map-encoded entries.
|
||||||
|
///
|
||||||
|
/// The thin read-only projection in `crates/data-usage` decodes a copy of
|
||||||
|
/// this fixture (`thin_usage_cache_decodes_scanner_wire_fixture`); when
|
||||||
|
/// the encoding legitimately changes, regenerate both copies from
|
||||||
|
/// `wire_fixture_cache().marshal_msg()` and re-verify old readers.
|
||||||
|
const USAGE_CACHE_WIRE_FIXTURE: &[u8] = &[
|
||||||
|
0x92, 0xde, 0x00, 0x10, 0xa4, 0x6e, 0x61, 0x6d, 0x65, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||||
|
0x74, 0xaa, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x07, 0xac, 0x6c, 0x65, 0x61, 0x64, 0x65, 0x72,
|
||||||
|
0x5f, 0x65, 0x70, 0x6f, 0x63, 0x68, 0x09, 0xab, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x92,
|
||||||
|
0xce, 0x65, 0x53, 0xf1, 0x00, 0x00, 0xac, 0x73, 0x6b, 0x69, 0x70, 0x5f, 0x68, 0x65, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0xc3,
|
||||||
|
0xa9, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0xc0, 0xab, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
|
||||||
|
0x69, 0x6f, 0x6e, 0xc0, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x81,
|
||||||
|
0xb0, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x6c, 0x6f, 0x73, 0x74, 0x0b, 0xb1, 0x73,
|
||||||
|
0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0x5f, 0x61, 0x66, 0x74, 0x65, 0x72, 0xb2, 0x77, 0x69, 0x72,
|
||||||
|
0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0xaf, 0x73, 0x63, 0x61, 0x6e,
|
||||||
|
0x5f, 0x63, 0x68, 0x65, 0x63, 0x6b, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0xc0, 0xad, 0x70, 0x65, 0x6e, 0x64, 0x69, 0x6e, 0x67,
|
||||||
|
0x5f, 0x68, 0x65, 0x61, 0x6c, 0x73, 0x91, 0x9a, 0xa6, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0xab, 0x77, 0x69, 0x72, 0x65,
|
||||||
|
0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0xa6, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x6e, 0xc0, 0x01, 0x64, 0xcc, 0xc8, 0x03,
|
||||||
|
0xa8, 0x64, 0x65, 0x66, 0x65, 0x72, 0x72, 0x65, 0x64, 0xa6, 0x62, 0x75, 0x64, 0x67, 0x65, 0x74, 0xab, 0x6f, 0x62, 0x6a,
|
||||||
|
0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x63, 0x6b, 0xc0, 0xa6, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x92, 0x01, 0x02, 0xb1,
|
||||||
|
0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0xc3, 0xb0, 0x73,
|
||||||
|
0x63, 0x61, 0x6e, 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x5f, 0x64, 0x69, 0x67, 0x65, 0x73, 0x74, 0xdc, 0x00, 0x20, 0x03, 0x03,
|
||||||
|
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
|
||||||
|
0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0xb0, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x6b, 0x65, 0x79,
|
||||||
|
0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x01, 0x81, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
|
||||||
|
0x74, 0x8b, 0xa8, 0x63, 0x68, 0x69, 0x6c, 0x64, 0x72, 0x65, 0x6e, 0x90, 0xa4, 0x73, 0x69, 0x7a, 0x65, 0xcd, 0x10, 0x00,
|
||||||
|
0xa7, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x03, 0xa8, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x05, 0xae,
|
||||||
|
0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x5f, 0x6d, 0x61, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x01, 0xa9, 0x6f, 0x62, 0x6a, 0x5f,
|
||||||
|
0x73, 0x69, 0x7a, 0x65, 0x73, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xac, 0x6f, 0x62,
|
||||||
|
0x6a, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x97, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x72,
|
||||||
|
0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0xc0, 0xa9, 0x63, 0x6f,
|
||||||
|
0x6d, 0x70, 0x61, 0x63, 0x74, 0x65, 0x64, 0xc3, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65,
|
||||||
|
0x63, 0x74, 0x73, 0x02, 0xae, 0x61, 0x6c, 0x6c, 0x5f, 0x74, 0x69, 0x65, 0x72, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0x91,
|
||||||
|
0x81, 0xa4, 0x57, 0x41, 0x52, 0x4d, 0x93, 0xcd, 0x08, 0x00, 0x02, 0x01,
|
||||||
|
];
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn usage_cache_wire_format_is_pinned() {
|
||||||
|
// Writer: the canonical map-encoded serializer must reproduce the
|
||||||
|
// pinned bytes. Round-trip tests cannot see format drift, so any
|
||||||
|
// encoding change (field rename/reorder, map->array switch) fails
|
||||||
|
// here and forces re-verifying old readers and the thin projection
|
||||||
|
// in crates/data-usage before the fixture is regenerated.
|
||||||
|
let encoded = wire_fixture_cache().marshal_msg().expect("marshal fixture cache");
|
||||||
|
assert_eq!(
|
||||||
|
encoded.as_slice(),
|
||||||
|
USAGE_CACHE_WIRE_FIXTURE,
|
||||||
|
"persisted .usage-cache.bin encoding drifted from the pinned fixture"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Reader: the pinned bytes decode with every field intact.
|
||||||
|
let decoded = DataUsageCache::unmarshal(USAGE_CACHE_WIRE_FIXTURE).expect("decode pinned fixture");
|
||||||
|
assert_eq!(decoded.info.name, "wire-bucket");
|
||||||
|
assert_eq!(decoded.info.next_cycle, 7);
|
||||||
|
assert_eq!(decoded.info.leader_epoch, 9);
|
||||||
|
assert_eq!(
|
||||||
|
decoded.info.last_update,
|
||||||
|
Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000))
|
||||||
|
);
|
||||||
|
assert!(decoded.info.skip_healing);
|
||||||
|
assert_eq!(decoded.info.failed_objects.get("wire-bucket/lost"), Some(&11));
|
||||||
|
assert_eq!(decoded.info.scan_resume_after.as_deref(), Some("wire-bucket/resume"));
|
||||||
|
assert_eq!(decoded.info.pending_heals.len(), 1);
|
||||||
|
assert_eq!(decoded.info.pending_heals[0].kind, PendingScannerHealKind::Object);
|
||||||
|
assert_eq!(decoded.info.pending_heals[0].object.as_deref(), Some("broken"));
|
||||||
|
assert_eq!(decoded.info.source, Some(DataUsageCacheSource::new(1, 2)));
|
||||||
|
assert!(decoded.info.snapshot_complete);
|
||||||
|
assert_eq!(decoded.info.scan_plan_digest, Some(TEST_PLAN_DIGEST));
|
||||||
|
assert_eq!(decoded.info.cache_key_format, DATA_USAGE_CACHE_KEY_FORMAT);
|
||||||
|
|
||||||
|
let entry = decoded.cache.get("wire-bucket").expect("fixture entry decodes");
|
||||||
|
assert_eq!(entry.size, 4096);
|
||||||
|
assert_eq!(entry.objects, 3);
|
||||||
|
assert_eq!(entry.versions, 5);
|
||||||
|
assert_eq!(entry.delete_markers, 1);
|
||||||
|
assert!(entry.compacted);
|
||||||
|
assert_eq!(entry.failed_objects, 2);
|
||||||
|
assert_eq!(
|
||||||
|
entry.all_tier_stats.as_ref().and_then(|tiers| tiers.tiers.get("WARM")),
|
||||||
|
Some(&TierStats {
|
||||||
|
total_size: 2048,
|
||||||
|
num_versions: 2,
|
||||||
|
num_objects: 1,
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn data_usage_cache_prepare_for_scan_rejects_unscoped_distributed_cache() {
|
fn data_usage_cache_prepare_for_scan_rejects_unscoped_distributed_cache() {
|
||||||
let mut cache = DataUsageCache {
|
let mut cache = DataUsageCache {
|
||||||
|
|||||||
@@ -13,6 +13,9 @@ for later deletion.
|
|||||||
## Open Items
|
## Open Items
|
||||||
|
|
||||||
- `table-publication-fence-v1` S3 Tables publication fencing: nodes that predate table and table-bucket publication fences can mutate live files while a new node is publishing a catalog pointer. New nodes retain exact object guards until the operator confirms that every serving node uses the new fences. Fleet confirmation also requires non-overlapping active warehouse prefixes and lifecycle workers that exclude table buckets. Remove the exact live-file fallback and the fleet-confirmation gate after the minimum supported RustFS release acquires table fences for registered-table mutations and table-bucket fences for unresolved-prefix mutations.
|
- `table-publication-fence-v1` S3 Tables publication fencing: nodes that predate table and table-bucket publication fences can mutate live files while a new node is publishing a catalog pointer. New nodes retain exact object guards until the operator confirms that every serving node uses the new fences. Fleet confirmation also requires non-overlapping active warehouse prefixes and lifecycle workers that exclude table buckets. Remove the exact live-file fallback and the fleet-confirmation gate after the minimum supported RustFS release acquires table fences for registered-table mutations and table-bucket fences for unresolved-prefix mutations.
|
||||||
|
- `table-catalog-strong-snapshot-v1` durable strong catalog snapshot compatibility: mixed-version deployments continue writing version 1 snapshots until operators confirm that every serving node reads version 2, and version 1 table/view identifier collisions remain available only for cleanup. Remove version 1 writes and collision cleanup after the minimum supported RustFS release reads version 2 and every retained durable strong snapshot is collision-free and has been upgraded to version 2.
|
||||||
|
- `table-catalog-migration-fence-v1` durable strong migration fence compatibility: version 1 "PREPARING" fences did not distinguish a known-absent global strong snapshot from an unknown baseline, so retries read them but fail closed if the global snapshot is missing. Version 2 preserves the same JSON shape and records the pre-migration global snapshot ETag in the existing target_snapshot_etag field while the fence is "PREPARING". Remove version 1 reads after every supported direct-upgrade source writes version 2 fences and operators have completed or cancelled every older in-progress backing migration.
|
||||||
|
- `table-catalog-backing-manifest-v1-wire-labels` durable strong backing manifest labels: version 1 published "STRONG_KV_WAL" and "CUT_OVER_LINEARIZABLE_READS" before the implementation was narrowed to the ETag-CAS durable snapshot backing. Internal names and operator documentation describe the implemented semantics, while version 1 responses retain those labels for existing clients. Replace the labels only in a new manifest version with an explicit client migration contract.
|
||||||
- `cross-pool-fence-v1` authenticated unsupported advertisement: predeployment servers recognize the versioned cross-pool fence capability probe but report support version 0, allowing a later all-peer probe to distinguish predeployment nodes without activating a second lock domain. Replace the unsupported advertisement only when composite lock acquisition, a cluster-wide activation fence, complete fleet proof, commit-time proof revalidation, and fail-closed revocation ship together.
|
- `cross-pool-fence-v1` authenticated unsupported advertisement: predeployment servers recognize the versioned cross-pool fence capability probe but report support version 0, allowing a later all-peer probe to distinguish predeployment nodes without activating a second lock domain. Replace the unsupported advertisement only when composite lock acquisition, a cluster-wide activation fence, complete fleet proof, commit-time proof revalidation, and fail-closed revocation ship together.
|
||||||
- `table-catalog-dotted-namespace` Iceberg REST namespace path compatibility: existing RustFS clients use dotted namespace paths, while the standard multi-level contract uses the URL-encoded unit separator `%1F`. New servers accept both forms so a rolling upgrade does not invalidate existing catalog configuration. Remove the dotted fallback after the minimum supported RustFS release advertises `%1F` and all supported clients have refreshed their catalog configuration.
|
- `table-catalog-dotted-namespace` Iceberg REST namespace path compatibility: existing RustFS clients use dotted namespace paths, while the standard multi-level contract uses the URL-encoded unit separator `%1F`. New servers accept both forms so a rolling upgrade does not invalidate existing catalog configuration. Remove the dotted fallback after the minimum supported RustFS release advertises `%1F` and all supported clients have refreshed their catalog configuration.
|
||||||
- `rustfs-5509` FileInfo positional MessagePack decoding: beta.11 serialized 28 fields, while beta.12 inserted transition-version fields in the middle and serialized an incompatible 30-field array. New releases write named maps and retain readers for both shipped array layouts so direct and rolling upgrades can read either release. Remove the positional-array readers after every supported direct-upgrade release writes named maps and no retained RPC payload can contain a pre-map FileInfo array.
|
- `rustfs-5509` FileInfo positional MessagePack decoding: beta.11 serialized 28 fields, while beta.12 inserted transition-version fields in the middle and serialized an incompatible 30-field array. New releases write named maps and retain readers for both shipped array layouts so direct and rolling upgrades can read either release. Remove the positional-array readers after every supported direct-upgrade release writes named maps and no retained RPC payload can contain a pre-map FileInfo array.
|
||||||
|
|||||||
@@ -10,10 +10,28 @@ and rollback steps.
|
|||||||
| Area | Current owner | Size | Split status |
|
| Area | Current owner | Size | Split status |
|
||||||
|---|---|---:|---|
|
|---|---|---:|---|
|
||||||
| Bucket lifecycle | `crates/lifecycle/` + `crates/ecstore/src/bucket/lifecycle/` | core contracts + ECStore runtime | Core contract extracted |
|
| Bucket lifecycle | `crates/lifecycle/` + `crates/ecstore/src/bucket/lifecycle/` | core contracts + ECStore runtime | Core contract extracted |
|
||||||
| Bucket replication | `crates/ecstore/src/bucket/replication/` | 8,730 lines | Contracts extracted; runtime move pending |
|
| Bucket replication | `crates/ecstore/src/bucket/replication/` | 15,619 lines | Contracts extracted; runtime move pending |
|
||||||
| Set disks | `crates/ecstore/src/set_disk/` | state carrier plus operation modules | Keep in ECStore |
|
| Set disks | `crates/ecstore/src/set_disk/` | state carrier plus operation modules | Keep in ECStore |
|
||||||
| Public ECStore facade | `crates/ecstore/src/api/mod.rs` | broad compatibility surface | Shrink only through guarded PRs |
|
| Public ECStore facade | `crates/ecstore/src/api/mod.rs` | broad compatibility surface | Shrink only through guarded PRs |
|
||||||
|
|
||||||
|
Measured 2026-08-12: the whole crate is 265 files / ~288K lines (roughly half
|
||||||
|
is inline `#[cfg(test)]` code). The largest single files are `disk/local.rs`
|
||||||
|
(21,063 lines), `bucket/lifecycle/bucket_lifecycle_ops.rs` (11,961 lines), and
|
||||||
|
`set_disk/mod.rs` (11,151 lines). Reproduce with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
find crates/ecstore/src -name '*.rs' | xargs wc -l | sort -rn | head
|
||||||
|
find crates/ecstore/src/bucket/replication -name '*.rs' | xargs wc -l | tail -1
|
||||||
|
```
|
||||||
|
|
||||||
|
No split step has landed since the contract-extraction PRs of 2026-07-04,
|
||||||
|
while the `bucket/replication` runtime grew from 8,730 to 15,619 lines (+79%)
|
||||||
|
through feature work (e.g. SSE-C ciphertext passthrough replication #5898,
|
||||||
|
delete-marker purge retry/replay #5864). To keep the gap from widening: in
|
||||||
|
domains that already have a contract crate, new replication runtime logic that
|
||||||
|
does not need ECStore runtime state must land in `rustfs-replication`, not in
|
||||||
|
`crates/ecstore/src/bucket/replication/`.
|
||||||
|
|
||||||
The file split inside `set_disk/` is already operation-oriented: read, write,
|
The file split inside `set_disk/` is already operation-oriented: read, write,
|
||||||
list, multipart, lock, heal, and replication code live in separate modules.
|
list, multipart, lock, heal, and replication code live in separate modules.
|
||||||
The remaining large surface is the shared `SetDisks` state and cross-cutting
|
The remaining large surface is the shared `SetDisks` state and cross-cutting
|
||||||
|
|||||||
@@ -116,12 +116,13 @@ catalog extension.
|
|||||||
| Commit publication fencing | Supported with rolling-upgrade gate | Existing deployments retain exact object guards so older writers cannot mutate referenced files during publication. Set `RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED=true` only after every serving node supports table and table-bucket publication fences. In scalable mode, active table warehouse prefixes must not overlap, ordinary lifecycle expiry remains disabled for table buckets, and first enablement, first publication, drop, and warehouse relocation are serialized by the table-bucket fence. |
|
| Commit publication fencing | Supported with rolling-upgrade gate | Existing deployments retain exact object guards so older writers cannot mutate referenced files during publication. Set `RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED=true` only after every serving node supports table and table-bucket publication fences. In scalable mode, active table warehouse prefixes must not overlap, ordinary lifecycle expiry remains disabled for table buckets, and first enablement, first publication, drop, and warehouse relocation are serialized by the table-bucket fence. |
|
||||||
| Post-CAS finalization recovery | Supported | Diagnostics and recovery can repair stale or missing idempotency indexes without changing the current table pointer. |
|
| Post-CAS finalization recovery | Supported | Diagnostics and recovery can repair stale or missing idempotency indexes without changing the current table pointer. |
|
||||||
| Catalog export | Supported | Exposes table state, commit recovery state, and backing migration information for operator inspection. |
|
| Catalog export | Supported | Exposes table state, commit recovery state, and backing migration information for operator inspection. |
|
||||||
| Strong backing state transfer | Supported | Object-backed table bucket, namespace, table, view, commit-log, and idempotency state can be materialized into the durable strong snapshot. The transfer is deterministic, ETag-CAS protected, idempotent after an interrupted finalization, and fails closed when a table or view has no owning namespace entry. |
|
| Strong backing state transfer | Supported | Object-backed table bucket, namespace, table, view, commit-log, and idempotency state can be materialized into the durable strong snapshot. The transfer is deterministic, ETag-CAS protected, idempotent after an interrupted finalization, validates candidate state through the restart decoder before publication, preserves resource-backed implicit namespaces, and fails closed when an inactive explicit namespace conflicts with active descendants or resources. Snapshot hydration requires a stable non-empty ETag, caps the encoded snapshot at 64 MiB, shares state and reload serialization across requests in one server context, and rejects disappearance or format-version rollback after observation. Configured durable-strong mode rejects a missing snapshot on its first catalog access after startup; only object-backed migration may initialize an empty target. |
|
||||||
| Durable backing migration preflight | Supported | `GET /iceberg/v1/{warehouse}/catalog/migration` and the `/_iceberg/v1` alias inspect object-backed catalog inventory, recovery blockers, warehouse prefix index readiness, persistent write-fence state, target snapshot agreement, and whether every table bucket is ready for cutover. |
|
| Durable backing migration preflight | Supported | `GET /iceberg/v1/{warehouse}/catalog/migration` and the `/_iceberg/v1` alias inspect object-backed catalog inventory, recovery blockers, warehouse prefix index readiness, active table/view identifier collisions, persistent write-fence state, target snapshot agreement, and whether every table bucket is ready for cutover. |
|
||||||
| Durable backing migration execution | Preview / controlled | `POST /iceberg/v1/{warehouse}/catalog/migration` fences table-bucket registry changes, acquires a persistent per-bucket write fence, drains in-flight catalog mutations, materializes the target snapshot, and reports `ready_to_enable_durable_strong`. `DELETE` safely releases the bucket fence only while its target state has not advanced, and releases the registry fence after the last bucket is cancelled. Both mutations require `admin:MigrateTableCatalog`. |
|
| Durable backing migration execution | Preview / controlled | `POST /iceberg/v1/{warehouse}/catalog/migration` fences table-bucket registry changes, acquires a persistent per-bucket write fence, records whether a global strong snapshot existed before publication, drains in-flight catalog mutations, materializes the target snapshot, and reports `ready_to_enable_durable_strong`. Retries and `DELETE` may restore a known-absent initial target after an ambiguous first write, but fail closed if a previously existing or materialized global snapshot disappears. `DELETE` releases the bucket fence only while its target state has not advanced, and releases the registry fence after the last bucket is cancelled. Both mutations require `admin:MigrateTableCatalog`. |
|
||||||
|
| Strong snapshot rolling compatibility | Supported | Durable strong control-plane reads snapshot versions 1 and 2, writes version 1 by default, and writes version 2 only after both the requested and fleet-confirmed gates are enabled. A running process rejects any lower-format snapshot after observing a higher format. Once version 2 is fleet-confirmed, table data-plane resolution fails closed until the persisted snapshot is version 2; a missing table-bucket entry also fails closed instead of bypassing table-aware authorization. |
|
||||||
| Disaster recovery rehearsal | Manual/live harness | `failure_coverage.py --print-disaster-recovery-rehearsal` generates an operator runbook covering catalog export, diagnostics, safe recovery repair, rollback/import, durable backing migration dry-run, post-recovery loadTable, and table data-plane policy probes. |
|
| Disaster recovery rehearsal | Manual/live harness | `failure_coverage.py --print-disaster-recovery-rehearsal` generates an operator runbook covering catalog export, diagnostics, safe recovery repair, rollback/import, durable backing migration dry-run, post-recovery loadTable, and table data-plane policy probes. |
|
||||||
| Scale and fault rehearsal | Manual/live harness | `failure_coverage.py --print-scale-fault-rehearsal` generates an opt-in runbook for concurrent writer stress, maintenance scheduler lease recovery, durable backing cutover preflight, recovery/rollback/import under load, and post-run evidence capture. |
|
| Scale and fault rehearsal | Manual/live harness | `failure_coverage.py --print-scale-fault-rehearsal` generates an opt-in runbook for concurrent writer stress, maintenance scheduler lease recovery, durable backing cutover preflight, recovery/rollback/import under load, and post-run evidence capture. |
|
||||||
| Strong KV/WAL backing cutover | Preview / controlled | Operators can select durable strong backing with `RUSTFS_TABLE_CATALOG_BACKING=durable-strong` only after every table bucket reports `SNAPSHOT_MATERIALIZED` and `ready_to_enable_durable_strong: true`. Object-only advanced operations fail closed in durable strong mode. |
|
| Durable strong snapshot backing cutover | Preview / controlled | Operators can select the ETag-CAS snapshot backing with `RUSTFS_TABLE_CATALOG_BACKING=durable-strong` only after every table bucket reports `SNAPSHOT_MATERIALIZED` and `ready_to_enable_durable_strong: true`. This mode does not claim a separate external KV/WAL service, and object-only advanced operations fail closed. Version 1 backing manifests retain the legacy `STRONG_KV_WAL` and `CUT_OVER_LINEARIZABLE_READS` wire labels for client compatibility; those labels do not expand the implementation claim. |
|
||||||
| Single active writer region | Supported policy | Diagnostics publish single-active-writer semantics and read-only replica limits. |
|
| Single active writer region | Supported policy | Diagnostics publish single-active-writer semantics and read-only replica limits. |
|
||||||
| Active-active multi-region writes | Not claimed | A table must not accept independent concurrent writers in multiple active regions. |
|
| Active-active multi-region writes | Not claimed | A table must not accept independent concurrent writers in multiple active regions. |
|
||||||
|
|
||||||
@@ -136,23 +137,60 @@ warehouse:
|
|||||||
`GetTableCatalogAction` on each table bucket. Treat every `blockers` entry as
|
`GetTableCatalogAction` on each table bucket. Treat every `blockers` entry as
|
||||||
fail-closed; repair commit recovery state and backfill the warehouse prefix
|
fail-closed; repair commit recovery state and backfill the warehouse prefix
|
||||||
index before continuing.
|
index before continuing.
|
||||||
3. Run `POST /iceberg/v1/{warehouse}/catalog/migration` with
|
3. Before the migration `POST`, drain every catalog writer that predates the
|
||||||
`admin:MigrateTableCatalog`. This persists the source write fence before it
|
durable-backing migration fence and restart it on a fence-aware release. An
|
||||||
drains in-flight mutations and copies the catalog state.
|
older writer does not recognize the persisted fence and can otherwise
|
||||||
4. Repeat the preflight and materialization for every table bucket. Do not set
|
mutate the object-backed source after the snapshot inventory is captured.
|
||||||
|
Keep all catalog writers on the fence-aware release until cutover completes.
|
||||||
|
4. Inventory object-only advanced operations, including maintenance workers,
|
||||||
|
catalog recovery, export, diagnostics, and external catalog bridge writes.
|
||||||
|
Quiesce mutating operations before cutover and confirm that each required
|
||||||
|
operation is supported by durable-strong mode; unsupported operations fail
|
||||||
|
closed after cutover rather than continuing against object-backed state.
|
||||||
|
5. Run `POST /iceberg/v1/{warehouse}/catalog/migration` with
|
||||||
|
`admin:MigrateTableCatalog`. This acquires the exclusive migration fence to
|
||||||
|
drain in-flight fence-aware mutations, persists the source fence while
|
||||||
|
exclusivity is held, and then copies the catalog state.
|
||||||
|
6. Repeat the preflight and materialization for every table bucket. Do not set
|
||||||
`RUSTFS_TABLE_CATALOG_BACKING=durable-strong` until the preflight reports
|
`RUSTFS_TABLE_CATALOG_BACKING=durable-strong` until the preflight reports
|
||||||
`SNAPSHOT_MATERIALIZED`, no blockers, and
|
`SNAPSHOT_MATERIALIZED`, no blockers, and
|
||||||
`ready_to_enable_durable_strong: true`.
|
`ready_to_enable_durable_strong: true`.
|
||||||
5. Restart with durable strong backing enabled, then verify catalog config,
|
7. Restart with durable strong backing enabled, then verify catalog config,
|
||||||
table and view loads, commit idempotency, and table data-plane policy
|
table and view loads, commit idempotency, and table data-plane policy
|
||||||
resolution before admitting writers.
|
resolution before admitting writers.
|
||||||
6. Before restarting into durable-strong mode, `DELETE` on the migration
|
8. Before restarting into durable-strong mode, `DELETE` on the migration
|
||||||
endpoint can remove a migration-created target bucket snapshot and release
|
endpoint can remove a migration-created target bucket snapshot and release
|
||||||
the source fence. After the durable-strong state advances, cancellation
|
the source fence. After the durable-strong state advances, cancellation
|
||||||
fails closed; recovery requires an operator-selected restore or reverse
|
fails closed; recovery requires an operator-selected restore or reverse
|
||||||
migration instead of restarting against the stale object-backed pointer.
|
migration instead of restarting against the stale object-backed pointer.
|
||||||
7. Preserve the object-backed catalog backup until durable strong backing has
|
9. Preserve the object-backed catalog backup until durable strong backing has
|
||||||
passed the operator's retention window.
|
passed the operator's retention window.
|
||||||
|
10. Keep strong snapshot writes on version 1 during a rolling binary upgrade.
|
||||||
|
After every catalog writer can read version 2, set both
|
||||||
|
`RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2=true` and
|
||||||
|
`RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED=true`, then restart
|
||||||
|
the catalog writers. Perform a controlled catalog write or migration
|
||||||
|
materialization and confirm that the persisted snapshot is version 2 before
|
||||||
|
serving table data-plane traffic. Setting only one gate does not change the
|
||||||
|
write format.
|
||||||
|
11. After any version 2 snapshot is persisted, do not roll catalog writers back
|
||||||
|
to a binary that only reads version 1. Current binaries preserve version 2
|
||||||
|
even when the gates are later disabled. A running process rejects restored
|
||||||
|
version 1 content after observing version 2, but cannot distinguish an older
|
||||||
|
snapshot with the same format version from a deliberate restore. The format
|
||||||
|
high-water mark is process-local: restoring any older snapshot and restarting
|
||||||
|
every writer is a privileged disaster-recovery rollback that cannot be
|
||||||
|
inferred from the restored object alone. Recovery must restore a compatible
|
||||||
|
binary and a snapshot selected through the operator recovery procedure.
|
||||||
|
12. Migration preflight rejects an active table/view identifier collision before
|
||||||
|
it writes a migration fence. A pre-existing version 1 strong snapshot with
|
||||||
|
such a collision is loaded in cleanup-only quarantine. Ambiguous reads fail
|
||||||
|
closed; each cleanup mutation must reduce the collision set, and unrelated
|
||||||
|
writes remain blocked until all collisions are removed. Drain catalog
|
||||||
|
writers that predate cleanup quarantine before starting this repair, and
|
||||||
|
complete cleanup before the first version 2 write. Restoring any version 1
|
||||||
|
snapshot after a writer has observed version 2 fails closed instead of
|
||||||
|
replacing the in-process catalog state.
|
||||||
|
|
||||||
## Production Failure Coverage
|
## Production Failure Coverage
|
||||||
|
|
||||||
|
|||||||
@@ -2067,9 +2067,12 @@ fn job_id_from_params(params: &Params<'_, '_>) -> S3Result<String> {
|
|||||||
fn table_catalog_backend_from_extensions(
|
fn table_catalog_backend_from_extensions(
|
||||||
extensions: &http::Extensions,
|
extensions: &http::Extensions,
|
||||||
) -> S3Result<crate::table_catalog::EcStoreTableCatalogObjectBackend<ECStore>> {
|
) -> S3Result<crate::table_catalog::EcStoreTableCatalogObjectBackend<ECStore>> {
|
||||||
let store = runtime_sources::object_store_from_extensions(extensions)
|
let context = runtime_sources::app_context_from_extensions(extensions)
|
||||||
.ok_or_else(|| table_catalog_internal_error("request object store is not initialized"))?;
|
.ok_or_else(|| table_catalog_internal_error("request application context is not initialized"))?;
|
||||||
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new(store))
|
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new_with_strong_runtime(
|
||||||
|
context.object_store(),
|
||||||
|
context.table_catalog_strong_runtime(),
|
||||||
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
type EcStoreObjectTableCatalogStore =
|
type EcStoreObjectTableCatalogStore =
|
||||||
|
|||||||
@@ -2512,7 +2512,7 @@ async fn commit_publication_replays_historical_standard_commit_across_backings()
|
|||||||
crate::table_catalog::TableCatalogBackingMode::DurableStrong,
|
crate::table_catalog::TableCatalogBackingMode::DurableStrong,
|
||||||
] {
|
] {
|
||||||
let metadata_backend = TestTableCatalogObjectBackend::default();
|
let metadata_backend = TestTableCatalogObjectBackend::default();
|
||||||
let store = crate::table_catalog::ConfiguredTableCatalogStore::new(metadata_backend.clone(), mode);
|
let store = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend.clone(), mode);
|
||||||
let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace should parse");
|
let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace should parse");
|
||||||
create_standard_events_table(&store, &metadata_backend, &namespace).await;
|
create_standard_events_table(&store, &metadata_backend, &namespace).await;
|
||||||
let first_request = serde_json::json!({
|
let first_request = serde_json::json!({
|
||||||
@@ -2559,7 +2559,7 @@ async fn commit_publication_replays_historical_standard_commit_across_backings()
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let store = crate::table_catalog::ConfiguredTableCatalogStore::new(metadata_backend.clone(), mode);
|
let store = crate::table_catalog::ConfiguredTableCatalogStore::new_for_test(metadata_backend.clone(), mode);
|
||||||
let second = standard_commit_table_response(
|
let second = standard_commit_table_response(
|
||||||
&store,
|
&store,
|
||||||
&trusted_table_commit_backend(&metadata_backend),
|
&trusted_table_commit_backend(&metadata_backend),
|
||||||
@@ -7313,10 +7313,10 @@ fn test_manifest_list_avro_entries_with_partition_specs(manifests: &[(&str, i32,
|
|||||||
"#,
|
"#,
|
||||||
)
|
)
|
||||||
.expect("manifest list avro schema should parse");
|
.expect("manifest list avro schema should parse");
|
||||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest list writer should initialize");
|
||||||
for (manifest_path, partition_spec_id, sequence_number, snapshot_id) in manifests {
|
for (manifest_path, partition_spec_id, sequence_number, snapshot_id) in manifests {
|
||||||
writer
|
writer
|
||||||
.append(apache_avro::types::Value::Record(vec![
|
.append_value(apache_avro::types::Value::Record(vec![
|
||||||
(
|
(
|
||||||
"manifest_path".to_string(),
|
"manifest_path".to_string(),
|
||||||
apache_avro::types::Value::String((*manifest_path).to_string()),
|
apache_avro::types::Value::String((*manifest_path).to_string()),
|
||||||
@@ -7368,10 +7368,10 @@ fn test_manifest_avro_bytes(files: &[(&str, i32, i32, i64, i64)]) -> Vec<u8> {
|
|||||||
"#,
|
"#,
|
||||||
)
|
)
|
||||||
.expect("manifest avro schema should parse");
|
.expect("manifest avro schema should parse");
|
||||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest writer should initialize");
|
||||||
for (file_path, content, status, snapshot_id, sequence_number) in files {
|
for (file_path, content, status, snapshot_id, sequence_number) in files {
|
||||||
writer
|
writer
|
||||||
.append(apache_avro::types::Value::Record(vec![
|
.append_value(apache_avro::types::Value::Record(vec![
|
||||||
("status".to_string(), apache_avro::types::Value::Int(*status)),
|
("status".to_string(), apache_avro::types::Value::Int(*status)),
|
||||||
("snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
|
("snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
|
||||||
("sequence_number".to_string(), apache_avro::types::Value::Long(*sequence_number)),
|
("sequence_number".to_string(), apache_avro::types::Value::Long(*sequence_number)),
|
||||||
@@ -7427,10 +7427,10 @@ fn test_manifest_avro_bytes_with_nullable_sequences(files: &[(&str, i32, i32, i6
|
|||||||
"#,
|
"#,
|
||||||
)
|
)
|
||||||
.expect("manifest avro schema should parse");
|
.expect("manifest avro schema should parse");
|
||||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).expect("manifest writer should initialize");
|
||||||
for (file_path, content, status, snapshot_id, sequence_number) in files {
|
for (file_path, content, status, snapshot_id, sequence_number) in files {
|
||||||
writer
|
writer
|
||||||
.append(apache_avro::types::Value::Record(vec![
|
.append_value(apache_avro::types::Value::Record(vec![
|
||||||
("status".to_string(), apache_avro::types::Value::Int(*status)),
|
("status".to_string(), apache_avro::types::Value::Int(*status)),
|
||||||
("snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
|
("snapshot_id".to_string(), apache_avro::types::Value::Long(*snapshot_id)),
|
||||||
("sequence_number".to_string(), test_nullable_long(*sequence_number)),
|
("sequence_number".to_string(), test_nullable_long(*sequence_number)),
|
||||||
|
|||||||
@@ -76,6 +76,7 @@ pub struct AppContext {
|
|||||||
buffer_config: Arc<dyn BufferConfigInterface>,
|
buffer_config: Arc<dyn BufferConfigInterface>,
|
||||||
object_data_cache: Arc<ObjectDataCacheAdapter>,
|
object_data_cache: Arc<ObjectDataCacheAdapter>,
|
||||||
object_traffic_health: Arc<ObjectTrafficHealth>,
|
object_traffic_health: Arc<ObjectTrafficHealth>,
|
||||||
|
table_catalog_strong_runtime: crate::table_catalog::StrongTableCatalogRuntime,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AppContext {
|
impl AppContext {
|
||||||
@@ -125,6 +126,7 @@ impl AppContext {
|
|||||||
buffer_config: default_buffer_config_interface(),
|
buffer_config: default_buffer_config_interface(),
|
||||||
object_data_cache,
|
object_data_cache,
|
||||||
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
|
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
|
||||||
|
table_catalog_strong_runtime: crate::table_catalog::StrongTableCatalogRuntime::default(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -144,6 +146,10 @@ impl AppContext {
|
|||||||
Arc::clone(&self.object_traffic_health)
|
Arc::clone(&self.object_traffic_health)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub(crate) fn table_catalog_strong_runtime(&self) -> crate::table_catalog::StrongTableCatalogRuntime {
|
||||||
|
self.table_catalog_strong_runtime.clone()
|
||||||
|
}
|
||||||
|
|
||||||
pub fn iam(&self) -> Arc<dyn IamInterface> {
|
pub fn iam(&self) -> Arc<dyn IamInterface> {
|
||||||
self.iam.clone()
|
self.iam.clone()
|
||||||
}
|
}
|
||||||
@@ -350,6 +356,7 @@ impl AppContext {
|
|||||||
buffer_config: interfaces.buffer_config,
|
buffer_config: interfaces.buffer_config,
|
||||||
object_data_cache: ObjectDataCacheAdapter::disabled_arc(),
|
object_data_cache: ObjectDataCacheAdapter::disabled_arc(),
|
||||||
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
|
object_traffic_health: Arc::new(ObjectTrafficHealth::from_env()),
|
||||||
|
table_catalog_strong_runtime: crate::table_catalog::StrongTableCatalogRuntime::default(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2355,7 +2355,7 @@ mod tests {
|
|||||||
let req = build_request(input, Method::GET);
|
let req = build_request(input, Method::GET);
|
||||||
|
|
||||||
let err = make_usecase().execute_list_multipart_uploads(req).await.unwrap_err();
|
let err = make_usecase().execute_list_multipart_uploads(req).await.unwrap_err();
|
||||||
assert_eq!(err.code(), &S3ErrorCode::NotImplemented);
|
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
|
||||||
assert_eq!(err.message(), Some("Invalid key marker"));
|
assert_eq!(err.message(), Some("Invalid key marker"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1438,8 +1438,15 @@ fn table_data_plane_content_mutation(action: Action) -> bool {
|
|||||||
fn table_catalog_backend_for_data_plane<T>(
|
fn table_catalog_backend_for_data_plane<T>(
|
||||||
req: &S3Request<T>,
|
req: &S3Request<T>,
|
||||||
) -> S3Result<crate::table_catalog::EcStoreTableCatalogObjectBackend<ECStore>> {
|
) -> S3Result<crate::table_catalog::EcStoreTableCatalogObjectBackend<ECStore>> {
|
||||||
let store = request_object_store(req)?;
|
let context = match req.extensions.get::<Arc<ServerContextSlot>>() {
|
||||||
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new(store))
|
Some(server_ctx) => server_ctx.installed_app_context(),
|
||||||
|
None => runtime_sources::current_app_context(),
|
||||||
|
}
|
||||||
|
.ok_or_else(object_store_not_initialized_error)?;
|
||||||
|
Ok(crate::table_catalog::EcStoreTableCatalogObjectBackend::new_with_strong_runtime(
|
||||||
|
context.object_store(),
|
||||||
|
context.table_catalog_strong_runtime(),
|
||||||
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
fn table_catalog_store_for_data_plane<T>(
|
fn table_catalog_store_for_data_plane<T>(
|
||||||
|
|||||||
@@ -39,7 +39,6 @@ use rustfs_io_metrics::internode_metrics::{
|
|||||||
INTERNODE_OPERATION_NS_SCANNER, INTERNODE_OPERATION_PUT_FILE_CAPABILITY, INTERNODE_OPERATION_PUT_FILE_STREAM,
|
INTERNODE_OPERATION_NS_SCANNER, INTERNODE_OPERATION_PUT_FILE_CAPABILITY, INTERNODE_OPERATION_PUT_FILE_STREAM,
|
||||||
INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_OPERATION_WALK_DIR, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
|
INTERNODE_OPERATION_READ_FILE_STREAM, INTERNODE_OPERATION_WALK_DIR, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
|
||||||
};
|
};
|
||||||
use rustfs_utils::net::bytes_stream;
|
|
||||||
use s3s::Body;
|
use s3s::Body;
|
||||||
use s3s::dto::StreamingBlob;
|
use s3s::dto::StreamingBlob;
|
||||||
use serde::de::DeserializeOwned;
|
use serde::de::DeserializeOwned;
|
||||||
@@ -51,7 +50,7 @@ use std::pin::Pin;
|
|||||||
use std::sync::{Arc, LazyLock, Weak};
|
use std::sync::{Arc, LazyLock, Weak};
|
||||||
use std::task::{Context, Poll};
|
use std::task::{Context, Poll};
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
use tokio::io::{self, AsyncWriteExt};
|
use tokio::io::{self, AsyncReadExt, AsyncWriteExt};
|
||||||
use tokio::sync::{Mutex, oneshot};
|
use tokio::sync::{Mutex, oneshot};
|
||||||
use tokio_util::{io::ReaderStream, sync::CancellationToken};
|
use tokio_util::{io::ReaderStream, sync::CancellationToken};
|
||||||
use tower::Service;
|
use tower::Service;
|
||||||
@@ -663,15 +662,24 @@ where
|
|||||||
R: tokio::io::AsyncRead + Unpin + Send + Sync + 'static,
|
R: tokio::io::AsyncRead + Unpin + Send + Sync + 'static,
|
||||||
{
|
{
|
||||||
let metrics = runtime_sources::current_internode_metrics();
|
let metrics = runtime_sources::current_internode_metrics();
|
||||||
let stream = ReaderStream::with_capacity(reader, DEFAULT_READ_BUFFER_SIZE).map_ok(move |bytes| {
|
let read_buffer_size = read_file_stream_buffer_size(length);
|
||||||
|
let read_limit = if length == 0 {
|
||||||
|
u64::MAX
|
||||||
|
} else {
|
||||||
|
u64::try_from(length).unwrap_or(u64::MAX)
|
||||||
|
};
|
||||||
|
let stream = ReaderStream::with_capacity(reader.take(read_limit), read_buffer_size).map_ok(move |bytes| {
|
||||||
metrics.record_sent_bytes_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, bytes.len());
|
metrics.record_sent_bytes_for_operation_and_backend(operation, INTERNODE_TRANSPORT_BACKEND_TCP_HTTP, bytes.len());
|
||||||
bytes
|
bytes
|
||||||
});
|
});
|
||||||
|
Box::pin(stream)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_file_stream_buffer_size(length: usize) -> usize {
|
||||||
if length == 0 {
|
if length == 0 {
|
||||||
Box::pin(stream)
|
DEFAULT_READ_BUFFER_SIZE
|
||||||
} else {
|
} else {
|
||||||
Box::pin(bytes_stream(stream, length))
|
length.min(DEFAULT_READ_BUFFER_SIZE)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1664,7 +1672,7 @@ fn put_file_stage_error_message(stage: &str, query: &PutFileQuery, err: &dyn std
|
|||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::{
|
use super::{
|
||||||
DiskError, InternodeRpcService, LOG_SUBSYSTEM_DIRECTORY_WALK, LOG_SUBSYSTEM_FILE_TRANSFER,
|
DEFAULT_READ_BUFFER_SIZE, DiskError, InternodeRpcService, LOG_SUBSYSTEM_DIRECTORY_WALK, LOG_SUBSYSTEM_FILE_TRANSFER,
|
||||||
LOG_SUBSYSTEM_NAMESPACE_SCANNER, LOG_SUBSYSTEM_ROUTING, NS_SCANNER_BODY_SHA256_QUERY,
|
LOG_SUBSYSTEM_NAMESPACE_SCANNER, LOG_SUBSYSTEM_ROUTING, NS_SCANNER_BODY_SHA256_QUERY,
|
||||||
NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CYCLE_QUERY, NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PATH,
|
NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CYCLE_QUERY, NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PATH,
|
||||||
NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY, NS_SCANNER_SESSION_ID_QUERY,
|
NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY, NS_SCANNER_SESSION_ID_QUERY,
|
||||||
@@ -1673,10 +1681,10 @@ mod tests {
|
|||||||
append_walk_dir_completion, internode_http_operation, internode_rpc_subsystem, is_internode_rpc_path,
|
append_walk_dir_completion, internode_http_operation, internode_rpc_subsystem, is_internode_rpc_path,
|
||||||
ns_scanner_response_body, ns_scanner_server_epoch_matches, put_body_size_mismatch, put_file_auth_nonce,
|
ns_scanner_response_body, ns_scanner_server_epoch_matches, put_body_size_mismatch, put_file_auth_nonce,
|
||||||
put_file_capability_response, put_file_server_epoch_matches, put_file_stage_error_message, put_file_target_lock,
|
put_file_capability_response, put_file_server_epoch_matches, put_file_stage_error_message, put_file_target_lock,
|
||||||
read_file_body_stream, remote_scanner_claim_rejection, response_with_disk_error, supports_walk_dir_stream_completion,
|
read_file_body_stream, read_file_stream_buffer_size, remote_scanner_claim_rejection, response_with_disk_error,
|
||||||
validate_walk_dir_completion_request, verify_internode_rpc_signature, verify_ns_scanner_body_digest,
|
supports_walk_dir_stream_completion, validate_walk_dir_completion_request, verify_internode_rpc_signature,
|
||||||
verify_walk_dir_body_digest, walk_dir_response_body, write_authenticated_put_file, write_body_chunks_to_writer,
|
verify_ns_scanner_body_digest, verify_walk_dir_body_digest, walk_dir_response_body, write_authenticated_put_file,
|
||||||
write_put_file_body_chunks_to_writer,
|
write_body_chunks_to_writer, write_put_file_body_chunks_to_writer,
|
||||||
};
|
};
|
||||||
use crate::storage::storage_api::ecstore_rpc::{build_put_file_auth_trailer, gen_signature_headers};
|
use crate::storage::storage_api::ecstore_rpc::{build_put_file_auth_trailer, gen_signature_headers};
|
||||||
use crate::storage::storage_api::rpc_consumer::http_service::{DiskAPI as _, DiskOption, DiskStore, Endpoint, new_disk};
|
use crate::storage::storage_api::rpc_consumer::http_service::{DiskAPI as _, DiskOption, DiskStore, Endpoint, new_disk};
|
||||||
@@ -1704,6 +1712,23 @@ mod tests {
|
|||||||
use tokio_stream::StreamExt;
|
use tokio_stream::StreamExt;
|
||||||
use tokio_stream::iter;
|
use tokio_stream::iter;
|
||||||
|
|
||||||
|
struct RejectExtraPollReader {
|
||||||
|
emitted: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl io::AsyncRead for RejectExtraPollReader {
|
||||||
|
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut io::ReadBuf<'_>) -> Poll<io::Result<()>> {
|
||||||
|
if self.emitted {
|
||||||
|
return Poll::Ready(Err(io::Error::other("reader polled past the requested length")));
|
||||||
|
}
|
||||||
|
|
||||||
|
let bytes = b"hello world";
|
||||||
|
buf.put_slice(&bytes[..buf.remaining().min(bytes.len())]);
|
||||||
|
self.emitted = true;
|
||||||
|
Poll::Ready(Ok(()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async fn new_put_file_test_disk() -> (DiskStore, tempfile::TempDir) {
|
async fn new_put_file_test_disk() -> (DiskStore, tempfile::TempDir) {
|
||||||
let dir = tempfile::tempdir().expect("temp directory should be created");
|
let dir = tempfile::tempdir().expect("temp directory should be created");
|
||||||
let endpoint =
|
let endpoint =
|
||||||
@@ -2842,11 +2867,7 @@ mod tests {
|
|||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn read_file_body_stream_truncates_to_requested_length() {
|
async fn read_file_body_stream_truncates_to_requested_length() {
|
||||||
let (reader, mut writer) = tokio::io::duplex(64);
|
let reader = RejectExtraPollReader { emitted: false };
|
||||||
tokio::spawn(async move {
|
|
||||||
writer.write_all(b"hello world").await.expect("write succeeds");
|
|
||||||
});
|
|
||||||
|
|
||||||
let mut stream = read_file_body_stream(reader, 5, INTERNODE_OPERATION_READ_FILE_STREAM);
|
let mut stream = read_file_body_stream(reader, 5, INTERNODE_OPERATION_READ_FILE_STREAM);
|
||||||
let mut out = Vec::new();
|
let mut out = Vec::new();
|
||||||
while let Some(chunk) = stream.next().await {
|
while let Some(chunk) = stream.next().await {
|
||||||
@@ -2856,6 +2877,18 @@ mod tests {
|
|||||||
assert_eq!(out, b"hello");
|
assert_eq!(out, b"hello");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn read_file_body_stream_sizes_buffer_to_requested_length() {
|
||||||
|
for (length, expected_capacity) in [
|
||||||
|
(0, DEFAULT_READ_BUFFER_SIZE),
|
||||||
|
(40 * 1024, 40 * 1024),
|
||||||
|
(DEFAULT_READ_BUFFER_SIZE, DEFAULT_READ_BUFFER_SIZE),
|
||||||
|
(DEFAULT_READ_BUFFER_SIZE + 1, DEFAULT_READ_BUFFER_SIZE),
|
||||||
|
] {
|
||||||
|
assert_eq!(read_file_stream_buffer_size(length), expected_capacity);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn read_file_error_response_marks_only_missing_disk_errors() {
|
fn read_file_error_response_marks_only_missing_disk_errors() {
|
||||||
for (error, expected) in [
|
for (error, expected) in [
|
||||||
|
|||||||
@@ -140,7 +140,7 @@ pub(crate) fn parse_list_multipart_uploads_params(
|
|||||||
if let Some(key_marker) = &key_marker
|
if let Some(key_marker) = &key_marker
|
||||||
&& !key_marker.starts_with(prefix.as_str())
|
&& !key_marker.starts_with(prefix.as_str())
|
||||||
{
|
{
|
||||||
return Err(S3Error::with_message(S3ErrorCode::NotImplemented, "Invalid key marker".to_string()));
|
return Err(S3Error::with_message(S3ErrorCode::InvalidArgument, "Invalid key marker".to_string()));
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(ListMultipartUploadsParams {
|
Ok(ListMultipartUploadsParams {
|
||||||
@@ -390,7 +390,7 @@ mod tests {
|
|||||||
let err = parse_list_multipart_uploads_params(Some("prefix/".to_string()), Some("other/key-marker".to_string()), None)
|
let err = parse_list_multipart_uploads_params(Some("prefix/".to_string()), Some("other/key-marker".to_string()), None)
|
||||||
.expect_err("expected invalid key marker");
|
.expect_err("expected invalid key marker");
|
||||||
|
|
||||||
assert_eq!(*err.code(), S3ErrorCode::NotImplemented);
|
assert_eq!(*err.code(), S3ErrorCode::InvalidArgument);
|
||||||
assert_eq!(err.message(), Some("Invalid key marker"));
|
assert_eq!(err.message(), Some("Invalid key marker"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -44,11 +44,14 @@ pub(crate) fn table_matches_staged_base(table: &TableEntry, commit_log: &CommitL
|
|||||||
pub(crate) struct TableCommitHistoryIndex<'a> {
|
pub(crate) struct TableCommitHistoryIndex<'a> {
|
||||||
table_id: &'a str,
|
table_id: &'a str,
|
||||||
reachable_states: BTreeSet<(&'a str, &'a str)>,
|
reachable_states: BTreeSet<(&'a str, &'a str)>,
|
||||||
|
ambiguous_states: BTreeSet<(&'a str, &'a str)>,
|
||||||
|
cycle_detected: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl<'a> TableCommitHistoryIndex<'a> {
|
impl<'a> TableCommitHistoryIndex<'a> {
|
||||||
pub(crate) fn new(table: &'a TableEntry, commits: impl IntoIterator<Item = &'a CommitLogEntry>) -> Self {
|
pub(crate) fn new(table: &'a TableEntry, commits: impl IntoIterator<Item = &'a CommitLogEntry>) -> Self {
|
||||||
let mut by_new_state = BTreeMap::<(&str, &str), Option<(&str, &str)>>::new();
|
let mut by_new_state = BTreeMap::<(&str, &str), Option<(&str, &str)>>::new();
|
||||||
|
let mut ambiguous_states = BTreeSet::new();
|
||||||
for commit in commits
|
for commit in commits
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter(|commit| commit.table_id == table.table_id && !matches!(commit.status, CommitLogStatus::Failed))
|
.filter(|commit| commit.table_id == table.table_id && !matches!(commit.status, CommitLogStatus::Failed))
|
||||||
@@ -57,27 +60,39 @@ impl<'a> TableCommitHistoryIndex<'a> {
|
|||||||
let previous = (commit.previous_metadata_location.as_str(), commit.expected_version_token.as_str());
|
let previous = (commit.previous_metadata_location.as_str(), commit.expected_version_token.as_str());
|
||||||
by_new_state
|
by_new_state
|
||||||
.entry(key)
|
.entry(key)
|
||||||
.and_modify(|candidate| *candidate = None)
|
.and_modify(|candidate| {
|
||||||
|
*candidate = None;
|
||||||
|
ambiguous_states.insert(key);
|
||||||
|
})
|
||||||
.or_insert(Some(previous));
|
.or_insert(Some(previous));
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut reachable_states = BTreeSet::new();
|
let mut reachable_states = BTreeSet::new();
|
||||||
let mut state = (table.metadata_location.as_str(), table.version_token.as_str());
|
let mut state = (table.metadata_location.as_str(), table.version_token.as_str());
|
||||||
while reachable_states.insert(state) {
|
let cycle_detected = loop {
|
||||||
|
if !reachable_states.insert(state) {
|
||||||
|
break true;
|
||||||
|
}
|
||||||
let Some(Some(previous)) = by_new_state.get(&state) else {
|
let Some(Some(previous)) = by_new_state.get(&state) else {
|
||||||
break;
|
break false;
|
||||||
};
|
};
|
||||||
state = *previous;
|
state = *previous;
|
||||||
}
|
};
|
||||||
Self {
|
Self {
|
||||||
table_id: &table.table_id,
|
table_id: &table.table_id,
|
||||||
reachable_states,
|
reachable_states,
|
||||||
|
ambiguous_states,
|
||||||
|
cycle_detected,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub(crate) fn proves_committed(&self, target: &CommitLogEntry) -> bool {
|
pub(crate) fn proves_committed(&self, target: &CommitLogEntry) -> bool {
|
||||||
self.table_id == target.table_id.as_str()
|
!self.cycle_detected
|
||||||
|
&& self.table_id == target.table_id.as_str()
|
||||||
&& !matches!(target.status, CommitLogStatus::Failed)
|
&& !matches!(target.status, CommitLogStatus::Failed)
|
||||||
|
&& !self
|
||||||
|
.ambiguous_states
|
||||||
|
.contains(&(target.new_metadata_location.as_str(), target.new_version_token.as_str()))
|
||||||
&& self
|
&& self
|
||||||
.reachable_states
|
.reachable_states
|
||||||
.contains(&(target.new_metadata_location.as_str(), target.new_version_token.as_str()))
|
.contains(&(target.new_metadata_location.as_str(), target.new_version_token.as_str()))
|
||||||
@@ -202,7 +217,7 @@ pub(crate) fn table_commit_recovery_entry(
|
|||||||
TableCommitRecoveryState::FinalizationRequired,
|
TableCommitRecoveryState::FinalizationRequired,
|
||||||
"a later committed pointer proves this staged commit is part of table history".to_string(),
|
"a later committed pointer proves this staged commit is part of table history".to_string(),
|
||||||
)
|
)
|
||||||
} else if matches!(commit_log.status, CommitLogStatus::Committed) {
|
} else if matches!(commit_log.status, CommitLogStatus::Committed) && historically_committed {
|
||||||
if idempotency_index_repair_required {
|
if idempotency_index_repair_required {
|
||||||
(
|
(
|
||||||
TableCommitRecoveryState::IdempotencyIndexRepairRequired,
|
TableCommitRecoveryState::IdempotencyIndexRepairRequired,
|
||||||
@@ -214,6 +229,11 @@ pub(crate) fn table_commit_recovery_entry(
|
|||||||
"commit is finalized and may be older than the current table pointer".to_string(),
|
"commit is finalized and may be older than the current table pointer".to_string(),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
} else if matches!(commit_log.status, CommitLogStatus::Committed) {
|
||||||
|
(
|
||||||
|
TableCommitRecoveryState::ManualReview,
|
||||||
|
"committed log is not reachable from the current table pointer".to_string(),
|
||||||
|
)
|
||||||
} else if table_matches_staged_base(table, commit_log) {
|
} else if table_matches_staged_base(table, commit_log) {
|
||||||
(
|
(
|
||||||
TableCommitRecoveryState::StagedBeforeTableUpdate,
|
TableCommitRecoveryState::StagedBeforeTableUpdate,
|
||||||
|
|||||||
@@ -286,17 +286,15 @@ where
|
|||||||
if table.state != TableCatalogEntryState::Active {
|
if table.state != TableCatalogEntryState::Active {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let Ok(warehouse_object_prefix) = table_warehouse_object_prefix(&table) else {
|
let warehouse_object_prefix = table_warehouse_object_prefix(&table)?;
|
||||||
continue;
|
|
||||||
};
|
|
||||||
if !object.starts_with(&warehouse_object_prefix) {
|
if !object.starts_with(&warehouse_object_prefix) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if matched
|
if let Some(current) = matched.as_ref() {
|
||||||
.as_ref()
|
return Err(TableCatalogStoreError::Invalid(format!(
|
||||||
.is_some_and(|current| current.warehouse_object_prefix.len() >= warehouse_object_prefix.len())
|
"object {object} matches overlapping active table warehouse prefixes {} and {warehouse_object_prefix}",
|
||||||
{
|
current.warehouse_object_prefix
|
||||||
continue;
|
)));
|
||||||
}
|
}
|
||||||
matched = Some(table_data_plane_resource_from_entry(table, warehouse_object_prefix));
|
matched = Some(table_data_plane_resource_from_entry(table, warehouse_object_prefix));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -921,9 +921,11 @@ pub(crate) fn compacted_manifest_list_avro_bytes(summary: CompactionManifestList
|
|||||||
"#,
|
"#,
|
||||||
)
|
)
|
||||||
.map_err(|err| TableCatalogStoreError::Internal(format!("failed to build compaction manifest list schema: {err}")))?;
|
.map_err(|err| TableCatalogStoreError::Internal(format!("failed to build compaction manifest list schema: {err}")))?;
|
||||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
let mut writer = apache_avro::Writer::new(&schema, Vec::new()).map_err(|err| {
|
||||||
|
TableCatalogStoreError::Internal(format!("failed to initialize compaction manifest list writer: {err}"))
|
||||||
|
})?;
|
||||||
writer
|
writer
|
||||||
.append(apache_avro::types::Value::Record(vec![
|
.append_value(apache_avro::types::Value::Record(vec![
|
||||||
(
|
(
|
||||||
"manifest_path".to_string(),
|
"manifest_path".to_string(),
|
||||||
apache_avro::types::Value::String(summary.manifest_path.to_string()),
|
apache_avro::types::Value::String(summary.manifest_path.to_string()),
|
||||||
@@ -1095,14 +1097,15 @@ fn compaction_partition_field_schema(value: &apache_avro::types::Value) -> Optio
|
|||||||
|
|
||||||
pub(crate) fn compacted_manifest_avro_bytes(data_files: &[CompactedDataFile]) -> TableCatalogStoreResult<Vec<u8>> {
|
pub(crate) fn compacted_manifest_avro_bytes(data_files: &[CompactedDataFile]) -> TableCatalogStoreResult<Vec<u8>> {
|
||||||
let schema = compacted_manifest_avro_schema(data_files)?;
|
let schema = compacted_manifest_avro_schema(data_files)?;
|
||||||
let mut writer = apache_avro::Writer::new(&schema, Vec::new());
|
let mut writer = apache_avro::Writer::new(&schema, Vec::new())
|
||||||
|
.map_err(|err| TableCatalogStoreError::Internal(format!("failed to initialize compaction manifest writer: {err}")))?;
|
||||||
for data_file in data_files {
|
for data_file in data_files {
|
||||||
let sort_order_id = match data_file.sort_order_id {
|
let sort_order_id = match data_file.sort_order_id {
|
||||||
Some(sort_order_id) => apache_avro::types::Value::Union(1, Box::new(apache_avro::types::Value::Int(sort_order_id))),
|
Some(sort_order_id) => apache_avro::types::Value::Union(1, Box::new(apache_avro::types::Value::Int(sort_order_id))),
|
||||||
None => apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
|
None => apache_avro::types::Value::Union(0, Box::new(apache_avro::types::Value::Null)),
|
||||||
};
|
};
|
||||||
writer
|
writer
|
||||||
.append(apache_avro::types::Value::Record(vec![
|
.append_value(apache_avro::types::Value::Record(vec![
|
||||||
("status".to_string(), apache_avro::types::Value::Int(data_file.status)),
|
("status".to_string(), apache_avro::types::Value::Int(data_file.status)),
|
||||||
("snapshot_id".to_string(), apache_avro::types::Value::Long(data_file.snapshot_id)),
|
("snapshot_id".to_string(), apache_avro::types::Value::Long(data_file.snapshot_id)),
|
||||||
("sequence_number".to_string(), apache_avro::types::Value::Long(data_file.sequence_number)),
|
("sequence_number".to_string(), apache_avro::types::Value::Long(data_file.sequence_number)),
|
||||||
|
|||||||
@@ -97,6 +97,9 @@ pub(crate) const TABLE_CATALOG_BACKING_MANIFEST_VERSION: u16 = 1;
|
|||||||
pub(crate) const ENV_TABLE_CATALOG_BACKING: &str = "RUSTFS_TABLE_CATALOG_BACKING";
|
pub(crate) const ENV_TABLE_CATALOG_BACKING: &str = "RUSTFS_TABLE_CATALOG_BACKING";
|
||||||
pub(crate) const ENV_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED: &str =
|
pub(crate) const ENV_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED: &str =
|
||||||
"RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED";
|
"RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED";
|
||||||
|
pub(crate) const ENV_TABLE_CATALOG_STRONG_SNAPSHOT_V2: &str = "RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2";
|
||||||
|
pub(crate) const ENV_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED: &str =
|
||||||
|
"RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED";
|
||||||
pub(crate) const TABLE_CATALOG_BACKING_OBJECT: &str = "object";
|
pub(crate) const TABLE_CATALOG_BACKING_OBJECT: &str = "object";
|
||||||
pub(crate) const TABLE_CATALOG_BACKING_DURABLE_STRONG: &str = "durable-strong";
|
pub(crate) const TABLE_CATALOG_BACKING_DURABLE_STRONG: &str = "durable-strong";
|
||||||
pub(crate) const TABLE_METADATA_DIGEST_REQUIREMENT_TYPE: &str = "assert-rustfs-metadata-sha256";
|
pub(crate) const TABLE_METADATA_DIGEST_REQUIREMENT_TYPE: &str = "assert-rustfs-metadata-sha256";
|
||||||
@@ -159,10 +162,12 @@ const ICEBERG_MAX_REF_AGE_MS_PROPERTY: &str = "history.expire.max-ref-age-ms";
|
|||||||
const ICEBERG_REF_MIN_SNAPSHOTS_TO_KEEP_FIELD: &str = "min-snapshots-to-keep";
|
const ICEBERG_REF_MIN_SNAPSHOTS_TO_KEEP_FIELD: &str = "min-snapshots-to-keep";
|
||||||
const ICEBERG_REF_MAX_SNAPSHOT_AGE_MS_FIELD: &str = "max-snapshot-age-ms";
|
const ICEBERG_REF_MAX_SNAPSHOT_AGE_MS_FIELD: &str = "max-snapshot-age-ms";
|
||||||
const ICEBERG_REF_MAX_REF_AGE_MS_FIELD: &str = "max-ref-age-ms";
|
const ICEBERG_REF_MAX_REF_AGE_MS_FIELD: &str = "max-ref-age-ms";
|
||||||
const STRONG_TABLE_CATALOG_SNAPSHOT_VERSION: u16 = 1;
|
const STRONG_TABLE_CATALOG_SNAPSHOT_MIN_READ_VERSION: u16 = 1;
|
||||||
|
const STRONG_TABLE_CATALOG_SNAPSHOT_VERSION: u16 = 2;
|
||||||
const STRONG_TABLE_CATALOG_BACKING_ROOT: &str = "strong-backing";
|
const STRONG_TABLE_CATALOG_BACKING_ROOT: &str = "strong-backing";
|
||||||
const STRONG_TABLE_CATALOG_SNAPSHOT_FILE: &str = "snapshot.json";
|
const STRONG_TABLE_CATALOG_SNAPSHOT_FILE: &str = "snapshot.json";
|
||||||
const TABLE_CATALOG_MIGRATION_VERSION: u16 = 1;
|
const TABLE_CATALOG_MIGRATION_MIN_READ_VERSION: u16 = 1;
|
||||||
|
const TABLE_CATALOG_MIGRATION_VERSION: u16 = 2;
|
||||||
const TABLE_CATALOG_MIGRATION_ROOT: &str = "backing-migration";
|
const TABLE_CATALOG_MIGRATION_ROOT: &str = "backing-migration";
|
||||||
const TABLE_CATALOG_MIGRATION_FENCE_FILE: &str = "durable-strong-fence.json";
|
const TABLE_CATALOG_MIGRATION_FENCE_FILE: &str = "durable-strong-fence.json";
|
||||||
const TABLE_CATALOG_MIGRATION_FENCE_LOCK: &str = "durable-strong-fence.lock";
|
const TABLE_CATALOG_MIGRATION_FENCE_LOCK: &str = "durable-strong-fence.lock";
|
||||||
|
|||||||
@@ -1067,7 +1067,9 @@ pub(crate) struct TableCatalogBackingProfile {
|
|||||||
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
|
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
|
||||||
pub(crate) enum TableCatalogBackingKind {
|
pub(crate) enum TableCatalogBackingKind {
|
||||||
ObjectBacked,
|
ObjectBacked,
|
||||||
StrongKvWal,
|
// RUSTFS_COMPAT_TODO(table-catalog-backing-manifest-v1-wire-labels): Keep the version 1 wire label for existing clients. Remove after a versioned manifest with an explicit client migration contract replaces it.
|
||||||
|
#[serde(rename = "STRONG_KV_WAL")]
|
||||||
|
DurableStrongSnapshot,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||||
@@ -1203,7 +1205,9 @@ pub(crate) enum TableCatalogBackingMigrationStep {
|
|||||||
ReplayCommitLog,
|
ReplayCommitLog,
|
||||||
VerifyCurrentPointer,
|
VerifyCurrentPointer,
|
||||||
EnableSingleWriterFencing,
|
EnableSingleWriterFencing,
|
||||||
CutOverLinearizableReads,
|
// RUSTFS_COMPAT_TODO(table-catalog-backing-manifest-v1-wire-labels): Keep the version 1 wire label for existing clients. Remove after a versioned manifest with an explicit client migration contract replaces it.
|
||||||
|
#[serde(rename = "CUT_OVER_LINEARIZABLE_READS")]
|
||||||
|
CutOverDurableSnapshotReads,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
|
||||||
@@ -1213,6 +1217,8 @@ pub(crate) enum TableCatalogBackingMigrationBlocker {
|
|||||||
CommitManualReviewRequired,
|
CommitManualReviewRequired,
|
||||||
WarehouseIndexBackfillRequired,
|
WarehouseIndexBackfillRequired,
|
||||||
DuplicateWarehousePrefix,
|
DuplicateWarehousePrefix,
|
||||||
|
DuplicateTableIdentity,
|
||||||
|
TableViewIdentifierCollision,
|
||||||
DurableStrongSnapshotChanged,
|
DurableStrongSnapshotChanged,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1222,6 +1228,8 @@ pub(crate) enum TableCatalogBackingMigrationAction {
|
|||||||
RunCatalogRecovery,
|
RunCatalogRecovery,
|
||||||
BackfillWarehouseIndex,
|
BackfillWarehouseIndex,
|
||||||
ReviewDuplicateWarehousePrefixes,
|
ReviewDuplicateWarehousePrefixes,
|
||||||
|
ReviewDuplicateTableIdentities,
|
||||||
|
ReviewTableViewIdentifierCollisions,
|
||||||
SnapshotObjectBackedCatalog,
|
SnapshotObjectBackedCatalog,
|
||||||
EnableDurableStrongBacking,
|
EnableDurableStrongBacking,
|
||||||
VerifyDurableStrongSnapshot,
|
VerifyDurableStrongSnapshot,
|
||||||
|
|||||||
@@ -13,11 +13,13 @@
|
|||||||
// limitations under the License.
|
// limitations under the License.
|
||||||
|
|
||||||
use super::object::{
|
use super::object::{
|
||||||
ObjectTableCatalogStore, validate_namespace_entry_object, validate_table_entry_object, validate_view_entry_object,
|
ObjectTableCatalogStore, validate_commit_idempotency_entry_object, validate_commit_log_entry_object,
|
||||||
|
validate_namespace_entry_object, validate_table_bucket_entry_object, validate_table_entry_object, validate_view_entry_object,
|
||||||
};
|
};
|
||||||
use super::strong::{
|
use super::strong::{
|
||||||
StrongCommitSnapshotRecord, StrongTableCatalogBucketSnapshot, StrongTableCatalogState, TableCatalogBackingMigrationFence,
|
StrongCommitSnapshotRecord, StrongTableCatalogBucketSnapshot, StrongTableCatalogState, TableCatalogBackingMigrationFence,
|
||||||
TableCatalogBackingMigrationFenceStatus, TableCatalogBackingMigrationGlobalFence, table_catalog_bucket_snapshot_fingerprint,
|
TableCatalogBackingMigrationFenceStatus, TableCatalogBackingMigrationGlobalFence,
|
||||||
|
TableCatalogBackingMigrationTargetSnapshotState, table_catalog_bucket_snapshot_fingerprint,
|
||||||
};
|
};
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
@@ -82,14 +84,14 @@ pub(super) fn table_catalog_backing_manifest(
|
|||||||
},
|
},
|
||||||
migration: TableCatalogBackingMigrationPlan {
|
migration: TableCatalogBackingMigrationPlan {
|
||||||
source_kind: TableCatalogBackingKind::ObjectBacked,
|
source_kind: TableCatalogBackingKind::ObjectBacked,
|
||||||
target_kind: TableCatalogBackingKind::StrongKvWal,
|
target_kind: TableCatalogBackingKind::DurableStrongSnapshot,
|
||||||
status: migration_status,
|
status: migration_status,
|
||||||
required_steps: vec![
|
required_steps: vec![
|
||||||
TableCatalogBackingMigrationStep::SnapshotCatalogExport,
|
TableCatalogBackingMigrationStep::SnapshotCatalogExport,
|
||||||
TableCatalogBackingMigrationStep::ReplayCommitLog,
|
TableCatalogBackingMigrationStep::ReplayCommitLog,
|
||||||
TableCatalogBackingMigrationStep::VerifyCurrentPointer,
|
TableCatalogBackingMigrationStep::VerifyCurrentPointer,
|
||||||
TableCatalogBackingMigrationStep::EnableSingleWriterFencing,
|
TableCatalogBackingMigrationStep::EnableSingleWriterFencing,
|
||||||
TableCatalogBackingMigrationStep::CutOverLinearizableReads,
|
TableCatalogBackingMigrationStep::CutOverDurableSnapshotReads,
|
||||||
],
|
],
|
||||||
blockers,
|
blockers,
|
||||||
},
|
},
|
||||||
@@ -118,12 +120,100 @@ impl<B> ObjectTableCatalogStore<B>
|
|||||||
where
|
where
|
||||||
B: TableCatalogObjectBackend,
|
B: TableCatalogObjectBackend,
|
||||||
{
|
{
|
||||||
|
fn migration_target_snapshot_state(
|
||||||
|
fence: &TableCatalogBackingMigrationFence,
|
||||||
|
) -> TableCatalogBackingMigrationTargetSnapshotState {
|
||||||
|
// RUSTFS_COMPAT_TODO(table-catalog-migration-fence-v1): Version 1 PREPARING fences have no durable baseline. Remove after all supported upgrade sources write version 2 fences and all version 1 migrations are completed or cancelled.
|
||||||
|
if fence.version < TABLE_CATALOG_MIGRATION_VERSION {
|
||||||
|
TableCatalogBackingMigrationTargetSnapshotState::Unknown
|
||||||
|
} else if fence.target_snapshot_etag.is_some() {
|
||||||
|
TableCatalogBackingMigrationTargetSnapshotState::Present
|
||||||
|
} else {
|
||||||
|
TableCatalogBackingMigrationTargetSnapshotState::Absent
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_backing_migration_fence(
|
||||||
|
table_bucket: &str,
|
||||||
|
fence: &TableCatalogBackingMigrationFence,
|
||||||
|
) -> TableCatalogStoreResult<()> {
|
||||||
|
if !(TABLE_CATALOG_MIGRATION_MIN_READ_VERSION..=TABLE_CATALOG_MIGRATION_VERSION).contains(&fence.version)
|
||||||
|
|| fence.table_bucket != table_bucket
|
||||||
|
|| fence.migration_id.is_empty()
|
||||||
|
{
|
||||||
|
return Err(TableCatalogStoreError::Invalid(format!(
|
||||||
|
"invalid durable strong migration fence for table bucket {table_bucket}"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
let target_snapshot_state = Self::migration_target_snapshot_state(fence);
|
||||||
|
if fence.target_bucket_existed && target_snapshot_state == TableCatalogBackingMigrationTargetSnapshotState::Absent {
|
||||||
|
return Err(TableCatalogStoreError::Invalid(format!(
|
||||||
|
"durable strong migration fence for table bucket {table_bucket} has an inconsistent target snapshot baseline"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
match fence.status {
|
||||||
|
TableCatalogBackingMigrationFenceStatus::Preparing if fence.source_fingerprint.is_some() => {
|
||||||
|
return Err(TableCatalogStoreError::Invalid(format!(
|
||||||
|
"preparing durable strong migration fence for table bucket {table_bucket} has materialized state"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
TableCatalogBackingMigrationFenceStatus::Materialized
|
||||||
|
if fence.source_fingerprint.is_none() || fence.target_snapshot_etag.is_none() =>
|
||||||
|
{
|
||||||
|
return Err(TableCatalogStoreError::Invalid(format!(
|
||||||
|
"materialized durable strong migration fence for table bucket {table_bucket} is incomplete"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_global_backing_migration_fence(fence: &TableCatalogBackingMigrationGlobalFence) -> TableCatalogStoreResult<()> {
|
||||||
|
if !(TABLE_CATALOG_MIGRATION_MIN_READ_VERSION..=TABLE_CATALOG_MIGRATION_VERSION).contains(&fence.version)
|
||||||
|
|| fence.migration_id.is_empty()
|
||||||
|
{
|
||||||
|
return Err(TableCatalogStoreError::Invalid(
|
||||||
|
"invalid durable strong global migration fence".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn observe_durable_strong_migration_target(
|
||||||
|
strong_store: &StrongTableCatalogStore<B>,
|
||||||
|
table_bucket: &str,
|
||||||
|
fence: Option<&TableCatalogBackingMigrationFence>,
|
||||||
|
) -> TableCatalogStoreResult<(Option<String>, Option<String>)> {
|
||||||
|
let target_snapshot_state = fence.map(Self::migration_target_snapshot_state);
|
||||||
|
let permits_absent_snapshot = fence.is_some_and(|fence| {
|
||||||
|
fence.status == TableCatalogBackingMigrationFenceStatus::Preparing
|
||||||
|
&& !fence.target_bucket_existed
|
||||||
|
&& target_snapshot_state == Some(TableCatalogBackingMigrationTargetSnapshotState::Absent)
|
||||||
|
});
|
||||||
|
if permits_absent_snapshot {
|
||||||
|
strong_store.restore_absent_migration_snapshot_baseline().await?;
|
||||||
|
}
|
||||||
|
let observation = strong_store.bucket_snapshot_observation(table_bucket).await?;
|
||||||
|
if fence.is_some() && observation.1.is_none() && !permits_absent_snapshot {
|
||||||
|
return Err(TableCatalogStoreError::Conflict(
|
||||||
|
"durable strong catalog snapshot is missing for an in-progress backing migration".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(observation)
|
||||||
|
}
|
||||||
|
|
||||||
async fn read_backing_migration_fence(
|
async fn read_backing_migration_fence(
|
||||||
&self,
|
&self,
|
||||||
table_bucket: &str,
|
table_bucket: &str,
|
||||||
) -> TableCatalogStoreResult<Option<(TableCatalogBackingMigrationFence, Option<String>)>> {
|
) -> TableCatalogStoreResult<Option<(TableCatalogBackingMigrationFence, Option<String>)>> {
|
||||||
self.read_entry(self.catalog_bucket(), &self.paths.backing_migration_fence_path(table_bucket))
|
let fence = self
|
||||||
.await
|
.read_entry(self.catalog_bucket(), &self.paths.backing_migration_fence_path(table_bucket))
|
||||||
|
.await?;
|
||||||
|
if let Some((fence, _)) = fence.as_ref() {
|
||||||
|
Self::validate_backing_migration_fence(table_bucket, fence)?;
|
||||||
|
}
|
||||||
|
Ok(fence)
|
||||||
}
|
}
|
||||||
|
|
||||||
pub(super) async fn acquire_table_bucket_registry_write_permit(&self) -> TableCatalogStoreResult<Box<dyn Send>> {
|
pub(super) async fn acquire_table_bucket_registry_write_permit(&self) -> TableCatalogStoreResult<Box<dyn Send>> {
|
||||||
@@ -164,11 +254,7 @@ where
|
|||||||
.read_entry::<TableCatalogBackingMigrationGlobalFence>(self.catalog_bucket(), fence_path)
|
.read_entry::<TableCatalogBackingMigrationGlobalFence>(self.catalog_bucket(), fence_path)
|
||||||
.await?
|
.await?
|
||||||
{
|
{
|
||||||
if fence.version != TABLE_CATALOG_MIGRATION_VERSION {
|
Self::validate_global_backing_migration_fence(&fence)?;
|
||||||
return Err(TableCatalogStoreError::Invalid(
|
|
||||||
"invalid durable strong global migration fence".to_string(),
|
|
||||||
));
|
|
||||||
}
|
|
||||||
return Ok(fence);
|
return Ok(fence);
|
||||||
}
|
}
|
||||||
let fence = TableCatalogBackingMigrationGlobalFence {
|
let fence = TableCatalogBackingMigrationGlobalFence {
|
||||||
@@ -181,6 +267,13 @@ where
|
|||||||
}
|
}
|
||||||
|
|
||||||
async fn clear_global_backing_migration_fence_if_unused(&self, fence_path: &str) -> TableCatalogStoreResult<()> {
|
async fn clear_global_backing_migration_fence_if_unused(&self, fence_path: &str) -> TableCatalogStoreResult<()> {
|
||||||
|
let Some((fence, _)) = self
|
||||||
|
.read_entry::<TableCatalogBackingMigrationGlobalFence>(self.catalog_bucket(), fence_path)
|
||||||
|
.await?
|
||||||
|
else {
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
Self::validate_global_backing_migration_fence(&fence)?;
|
||||||
let bucket_objects = self
|
let bucket_objects = self
|
||||||
.backend
|
.backend
|
||||||
.list_objects(self.catalog_bucket(), &self.paths.table_bucket_entries_prefix())
|
.list_objects(self.catalog_bucket(), &self.paths.table_bucket_entries_prefix())
|
||||||
@@ -194,19 +287,6 @@ where
|
|||||||
self.backend.delete_object(self.catalog_bucket(), fence_path).await
|
self.backend.delete_object(self.catalog_bucket(), fence_path).await
|
||||||
}
|
}
|
||||||
|
|
||||||
pub(super) async fn ensure_object_backed_writes_allowed(&self, table_bucket: &str) -> TableCatalogStoreResult<()> {
|
|
||||||
if self
|
|
||||||
.backend
|
|
||||||
.object_exists(self.catalog_bucket(), &self.paths.backing_migration_fence_path(table_bucket))
|
|
||||||
.await?
|
|
||||||
{
|
|
||||||
return Err(TableCatalogStoreError::Conflict(format!(
|
|
||||||
"object-backed catalog writes are fenced while table bucket {table_bucket} is prepared for durable strong cutover"
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
async fn collect_bucket_snapshot_with_locks(
|
async fn collect_bucket_snapshot_with_locks(
|
||||||
&self,
|
&self,
|
||||||
table_bucket: &str,
|
table_bucket: &str,
|
||||||
@@ -220,11 +300,7 @@ where
|
|||||||
else {
|
else {
|
||||||
return Err(TableCatalogStoreError::NotFound(format!("table bucket {table_bucket}")));
|
return Err(TableCatalogStoreError::NotFound(format!("table bucket {table_bucket}")));
|
||||||
};
|
};
|
||||||
if table_bucket_entry.table_bucket != table_bucket {
|
validate_table_bucket_entry_object(&self.paths, &bucket_path, &table_bucket_entry)?;
|
||||||
return Err(TableCatalogStoreError::Invalid(format!(
|
|
||||||
"table bucket entry does not match migration target {table_bucket}"
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut namespaces = Vec::new();
|
let mut namespaces = Vec::new();
|
||||||
let mut tables = Vec::new();
|
let mut tables = Vec::new();
|
||||||
@@ -286,6 +362,7 @@ where
|
|||||||
"commit log changed while preparing durable strong snapshot: {commit_object}"
|
"commit log changed while preparing durable strong snapshot: {commit_object}"
|
||||||
)));
|
)));
|
||||||
};
|
};
|
||||||
|
validate_commit_log_entry_object(&self.paths, &commit_object, table_bucket, &table_entry.table_id, &commit)?;
|
||||||
commits.push(StrongCommitSnapshotRecord {
|
commits.push(StrongCommitSnapshotRecord {
|
||||||
table_bucket: table_bucket.to_string(),
|
table_bucket: table_bucket.to_string(),
|
||||||
table_id: table_entry.table_id.clone(),
|
table_id: table_entry.table_id.clone(),
|
||||||
@@ -313,6 +390,13 @@ where
|
|||||||
"idempotency index changed while preparing durable strong snapshot: {idempotency_object}"
|
"idempotency index changed while preparing durable strong snapshot: {idempotency_object}"
|
||||||
)));
|
)));
|
||||||
};
|
};
|
||||||
|
validate_commit_idempotency_entry_object(
|
||||||
|
&self.paths,
|
||||||
|
&idempotency_object,
|
||||||
|
table_bucket,
|
||||||
|
&table_entry.table_id,
|
||||||
|
&commit,
|
||||||
|
)?;
|
||||||
let lookup_key = commit.idempotency_key.clone().ok_or_else(|| {
|
let lookup_key = commit.idempotency_key.clone().ok_or_else(|| {
|
||||||
TableCatalogStoreError::Invalid(format!("idempotency index {idempotency_object} has no idempotency key"))
|
TableCatalogStoreError::Invalid(format!("idempotency index {idempotency_object} has no idempotency key"))
|
||||||
})?;
|
})?;
|
||||||
@@ -360,6 +444,22 @@ where
|
|||||||
|
|
||||||
fn validate_bucket_snapshot_for_migration(&self, snapshot: &StrongTableCatalogBucketSnapshot) -> TableCatalogStoreResult<()> {
|
fn validate_bucket_snapshot_for_migration(&self, snapshot: &StrongTableCatalogBucketSnapshot) -> TableCatalogStoreResult<()> {
|
||||||
let table_bucket = &snapshot.table_bucket.table_bucket;
|
let table_bucket = &snapshot.table_bucket.table_bucket;
|
||||||
|
let active_table_identifiers = snapshot
|
||||||
|
.tables
|
||||||
|
.iter()
|
||||||
|
.filter(|table| table.state == TableCatalogEntryState::Active)
|
||||||
|
.map(|table| (&table.namespace, &table.table))
|
||||||
|
.collect::<BTreeSet<_>>();
|
||||||
|
if snapshot
|
||||||
|
.views
|
||||||
|
.iter()
|
||||||
|
.filter(|view| view.state == TableCatalogEntryState::Active)
|
||||||
|
.any(|view| active_table_identifiers.contains(&(&view.namespace, &view.view)))
|
||||||
|
{
|
||||||
|
return Err(TableCatalogStoreError::Conflict(format!(
|
||||||
|
"table bucket {table_bucket} contains an active table/view identifier collision"
|
||||||
|
)));
|
||||||
|
}
|
||||||
let tables_by_id = snapshot
|
let tables_by_id = snapshot
|
||||||
.tables
|
.tables
|
||||||
.iter()
|
.iter()
|
||||||
@@ -498,6 +598,15 @@ where
|
|||||||
if self.get_table_bucket(table_bucket).await?.is_none() {
|
if self.get_table_bucket(table_bucket).await?.is_none() {
|
||||||
return Err(TableCatalogStoreError::NotFound(format!("table bucket {table_bucket}")));
|
return Err(TableCatalogStoreError::NotFound(format!("table bucket {table_bucket}")));
|
||||||
}
|
}
|
||||||
|
if let Some((global_fence, _)) = self
|
||||||
|
.read_entry::<TableCatalogBackingMigrationGlobalFence>(
|
||||||
|
self.catalog_bucket(),
|
||||||
|
&self.paths.backing_migration_global_fence_path(),
|
||||||
|
)
|
||||||
|
.await?
|
||||||
|
{
|
||||||
|
Self::validate_global_backing_migration_fence(&global_fence)?;
|
||||||
|
}
|
||||||
|
|
||||||
let namespace_objects = self
|
let namespace_objects = self
|
||||||
.backend
|
.backend
|
||||||
@@ -511,6 +620,10 @@ where
|
|||||||
let mut recovery_required_count: usize = 0;
|
let mut recovery_required_count: usize = 0;
|
||||||
let mut manual_review_count: usize = 0;
|
let mut manual_review_count: usize = 0;
|
||||||
let mut warehouse_prefix_owners = BTreeMap::<String, usize>::new();
|
let mut warehouse_prefix_owners = BTreeMap::<String, usize>::new();
|
||||||
|
let mut table_ids = BTreeSet::<String>::new();
|
||||||
|
let mut duplicate_table_identity = false;
|
||||||
|
let mut active_table_identifiers = BTreeSet::<(String, String)>::new();
|
||||||
|
let mut active_view_identifiers = BTreeSet::<(String, String)>::new();
|
||||||
|
|
||||||
for object in namespace_objects {
|
for object in namespace_objects {
|
||||||
if object.ends_with(NAMESPACE_ENTRY_FILE) {
|
if object.ends_with(NAMESPACE_ENTRY_FILE) {
|
||||||
@@ -526,6 +639,9 @@ where
|
|||||||
continue;
|
continue;
|
||||||
};
|
};
|
||||||
validate_view_entry_object(&self.paths, &object, &entry)?;
|
validate_view_entry_object(&self.paths, &object, &entry)?;
|
||||||
|
if entry.state == TableCatalogEntryState::Active {
|
||||||
|
active_view_identifiers.insert((entry.namespace.clone(), entry.view.clone()));
|
||||||
|
}
|
||||||
view_count = view_count.saturating_add(1);
|
view_count = view_count.saturating_add(1);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -538,7 +654,11 @@ where
|
|||||||
};
|
};
|
||||||
validate_table_entry_object(&self.paths, &object, &table)?;
|
validate_table_entry_object(&self.paths, &object, &table)?;
|
||||||
table_count = table_count.saturating_add(1);
|
table_count = table_count.saturating_add(1);
|
||||||
|
if !table_ids.insert(table.table_id.clone()) {
|
||||||
|
duplicate_table_identity = true;
|
||||||
|
}
|
||||||
if table.state == TableCatalogEntryState::Active {
|
if table.state == TableCatalogEntryState::Active {
|
||||||
|
active_table_identifiers.insert((table.namespace.clone(), table.table.clone()));
|
||||||
let warehouse_prefix = table_warehouse_object_prefix(&table)?;
|
let warehouse_prefix = table_warehouse_object_prefix(&table)?;
|
||||||
warehouse_prefix_owners
|
warehouse_prefix_owners
|
||||||
.entry(warehouse_prefix)
|
.entry(warehouse_prefix)
|
||||||
@@ -548,17 +668,31 @@ where
|
|||||||
|
|
||||||
let recovery = self.table_commit_recovery_report_for_entry(&table, 0).await?;
|
let recovery = self.table_commit_recovery_report_for_entry(&table, 0).await?;
|
||||||
commit_log_count = commit_log_count.saturating_add(recovery.commits.len());
|
commit_log_count = commit_log_count.saturating_add(recovery.commits.len());
|
||||||
idempotency_index_count = idempotency_index_count.saturating_add(
|
for idempotency_object in self
|
||||||
self.backend
|
.backend
|
||||||
.list_objects(
|
.list_objects(
|
||||||
self.catalog_bucket(),
|
self.catalog_bucket(),
|
||||||
&self.paths.commit_idempotency_entries_prefix(table_bucket, &table.table_id),
|
&self.paths.commit_idempotency_entries_prefix(table_bucket, &table.table_id),
|
||||||
)
|
)
|
||||||
|
.await?
|
||||||
|
.into_iter()
|
||||||
|
.filter(|object| object.ends_with(".json"))
|
||||||
|
{
|
||||||
|
let Some((commit, _)) = self
|
||||||
|
.read_entry::<CommitLogEntry>(self.catalog_bucket(), &idempotency_object)
|
||||||
.await?
|
.await?
|
||||||
.into_iter()
|
else {
|
||||||
.filter(|object| object.ends_with(".json"))
|
continue;
|
||||||
.count(),
|
};
|
||||||
);
|
validate_commit_idempotency_entry_object(
|
||||||
|
&self.paths,
|
||||||
|
&idempotency_object,
|
||||||
|
table_bucket,
|
||||||
|
&table.table_id,
|
||||||
|
&commit,
|
||||||
|
)?;
|
||||||
|
idempotency_index_count = idempotency_index_count.saturating_add(1);
|
||||||
|
}
|
||||||
recovery_required_count = recovery_required_count
|
recovery_required_count = recovery_required_count
|
||||||
.saturating_add(recovery.staged_before_table_update_count)
|
.saturating_add(recovery.staged_before_table_update_count)
|
||||||
.saturating_add(recovery.finalization_required_count)
|
.saturating_add(recovery.finalization_required_count)
|
||||||
@@ -568,6 +702,13 @@ where
|
|||||||
|
|
||||||
let warehouse_index_ready = self.warehouse_index_ready(table_bucket).await?;
|
let warehouse_index_ready = self.warehouse_index_ready(table_bucket).await?;
|
||||||
let duplicate_warehouse_prefix_count = warehouse_prefix_owners.values().filter(|count| **count > 1).count();
|
let duplicate_warehouse_prefix_count = warehouse_prefix_owners.values().filter(|count| **count > 1).count();
|
||||||
|
let overlapping_warehouse_prefix = warehouse_prefix_owners
|
||||||
|
.keys()
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.windows(2)
|
||||||
|
.any(|window| warehouse_object_prefixes_overlap(window[0], window[1]));
|
||||||
|
let conflicting_warehouse_prefix = duplicate_warehouse_prefix_count > 0 || overlapping_warehouse_prefix;
|
||||||
|
let table_view_identifier_collision_count = active_table_identifiers.intersection(&active_view_identifiers).count();
|
||||||
let mut blockers = Vec::new();
|
let mut blockers = Vec::new();
|
||||||
let mut recommended_actions = Vec::new();
|
let mut recommended_actions = Vec::new();
|
||||||
if recovery_required_count > 0 {
|
if recovery_required_count > 0 {
|
||||||
@@ -583,12 +724,24 @@ where
|
|||||||
blockers.push(TableCatalogBackingMigrationBlocker::WarehouseIndexBackfillRequired);
|
blockers.push(TableCatalogBackingMigrationBlocker::WarehouseIndexBackfillRequired);
|
||||||
recommended_actions.push(TableCatalogBackingMigrationAction::BackfillWarehouseIndex);
|
recommended_actions.push(TableCatalogBackingMigrationAction::BackfillWarehouseIndex);
|
||||||
}
|
}
|
||||||
if duplicate_warehouse_prefix_count > 0 {
|
if conflicting_warehouse_prefix {
|
||||||
blockers.push(TableCatalogBackingMigrationBlocker::DuplicateWarehousePrefix);
|
blockers.push(TableCatalogBackingMigrationBlocker::DuplicateWarehousePrefix);
|
||||||
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewDuplicateWarehousePrefixes);
|
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewDuplicateWarehousePrefixes);
|
||||||
}
|
}
|
||||||
|
if duplicate_table_identity {
|
||||||
|
blockers.push(TableCatalogBackingMigrationBlocker::DuplicateTableIdentity);
|
||||||
|
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewDuplicateTableIdentities);
|
||||||
|
}
|
||||||
|
if table_view_identifier_collision_count > 0 {
|
||||||
|
blockers.push(TableCatalogBackingMigrationBlocker::TableViewIdentifierCollision);
|
||||||
|
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewTableViewIdentifierCollisions);
|
||||||
|
}
|
||||||
|
|
||||||
let mut status = if manual_review_count > 0 || duplicate_warehouse_prefix_count > 0 {
|
let mut status = if manual_review_count > 0
|
||||||
|
|| conflicting_warehouse_prefix
|
||||||
|
|| duplicate_table_identity
|
||||||
|
|| table_view_identifier_collision_count > 0
|
||||||
|
{
|
||||||
TableCatalogBackingMigrationStatus::ManualReviewRequired
|
TableCatalogBackingMigrationStatus::ManualReviewRequired
|
||||||
} else if recovery_required_count > 0 || !warehouse_index_ready {
|
} else if recovery_required_count > 0 || !warehouse_index_ready {
|
||||||
TableCatalogBackingMigrationStatus::RecoveryRequired
|
TableCatalogBackingMigrationStatus::RecoveryRequired
|
||||||
@@ -597,6 +750,14 @@ where
|
|||||||
};
|
};
|
||||||
|
|
||||||
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
||||||
|
let source_table_buckets = self.object_backed_table_buckets().await?;
|
||||||
|
let source_table_bucket_names = source_table_buckets.keys().cloned().collect::<BTreeSet<_>>();
|
||||||
|
let target_table_buckets = strong_store.table_bucket_names().await?;
|
||||||
|
if !target_table_buckets.is_subset(&source_table_bucket_names) {
|
||||||
|
status = TableCatalogBackingMigrationStatus::ManualReviewRequired;
|
||||||
|
blockers.push(TableCatalogBackingMigrationBlocker::DurableStrongSnapshotChanged);
|
||||||
|
recommended_actions.push(TableCatalogBackingMigrationAction::ReviewDurableStrongSnapshot);
|
||||||
|
}
|
||||||
let migration_fence = self.read_backing_migration_fence(table_bucket).await?.map(|(fence, _)| fence);
|
let migration_fence = self.read_backing_migration_fence(table_bucket).await?.map(|(fence, _)| fence);
|
||||||
let object_backed_writes_fenced = migration_fence.is_some();
|
let object_backed_writes_fenced = migration_fence.is_some();
|
||||||
if status == TableCatalogBackingMigrationStatus::ReadyToSnapshot
|
if status == TableCatalogBackingMigrationStatus::ReadyToSnapshot
|
||||||
@@ -633,7 +794,7 @@ where
|
|||||||
Ok(TableCatalogBackingMigrationDryRunReport {
|
Ok(TableCatalogBackingMigrationDryRunReport {
|
||||||
table_bucket: table_bucket.to_string(),
|
table_bucket: table_bucket.to_string(),
|
||||||
source_kind: TableCatalogBackingKind::ObjectBacked,
|
source_kind: TableCatalogBackingKind::ObjectBacked,
|
||||||
target_kind: TableCatalogBackingKind::StrongKvWal,
|
target_kind: TableCatalogBackingKind::DurableStrongSnapshot,
|
||||||
status,
|
status,
|
||||||
namespace_count,
|
namespace_count,
|
||||||
table_count,
|
table_count,
|
||||||
@@ -684,11 +845,17 @@ where
|
|||||||
let existing_fence = self.read_backing_migration_fence(table_bucket).await?;
|
let existing_fence = self.read_backing_migration_fence(table_bucket).await?;
|
||||||
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
||||||
if let Some((fence, _)) = existing_fence.as_ref()
|
if let Some((fence, _)) = existing_fence.as_ref()
|
||||||
&& (fence.version != TABLE_CATALOG_MIGRATION_VERSION || fence.table_bucket != table_bucket)
|
&& fence.status == TableCatalogBackingMigrationFenceStatus::Preparing
|
||||||
|
&& !fence.target_bucket_existed
|
||||||
|
&& Self::migration_target_snapshot_state(fence) == TableCatalogBackingMigrationTargetSnapshotState::Absent
|
||||||
{
|
{
|
||||||
return Err(TableCatalogStoreError::Invalid(format!(
|
strong_store.restore_absent_migration_snapshot_baseline().await?;
|
||||||
"invalid durable strong migration fence for table bucket {table_bucket}"
|
}
|
||||||
)));
|
let source_table_bucket_names = self.object_backed_table_buckets().await?.into_keys().collect::<BTreeSet<_>>();
|
||||||
|
if !strong_store.table_bucket_names().await?.is_subset(&source_table_bucket_names) {
|
||||||
|
return Err(TableCatalogStoreError::Conflict(
|
||||||
|
"durable strong snapshot contains table buckets outside the object-backed catalog inventory".to_string(),
|
||||||
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
if !self.warehouse_index_ready(table_bucket).await? {
|
if !self.warehouse_index_ready(table_bucket).await? {
|
||||||
@@ -712,10 +879,16 @@ where
|
|||||||
}
|
}
|
||||||
|
|
||||||
self.ensure_global_backing_migration_fence(&global_fence_path).await?;
|
self.ensure_global_backing_migration_fence(&global_fence_path).await?;
|
||||||
|
let (target_fingerprint, target_snapshot_etag) = Self::observe_durable_strong_migration_target(
|
||||||
|
&strong_store,
|
||||||
|
table_bucket,
|
||||||
|
existing_fence.as_ref().map(|(fence, _)| fence),
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
let (migration_id, target_bucket_existed) = if let Some((fence, _)) = existing_fence.as_ref() {
|
let (migration_id, target_bucket_existed) = if let Some((fence, _)) = existing_fence.as_ref() {
|
||||||
(fence.migration_id.clone(), fence.target_bucket_existed)
|
(fence.migration_id.clone(), fence.target_bucket_existed)
|
||||||
} else {
|
} else {
|
||||||
let target_bucket_existed = strong_store.bucket_snapshot_fingerprint(table_bucket).await?.is_some();
|
let target_bucket_existed = target_fingerprint.is_some();
|
||||||
let fence = TableCatalogBackingMigrationFence {
|
let fence = TableCatalogBackingMigrationFence {
|
||||||
version: TABLE_CATALOG_MIGRATION_VERSION,
|
version: TABLE_CATALOG_MIGRATION_VERSION,
|
||||||
table_bucket: table_bucket.to_string(),
|
table_bucket: table_bucket.to_string(),
|
||||||
@@ -723,7 +896,7 @@ where
|
|||||||
status: TableCatalogBackingMigrationFenceStatus::Preparing,
|
status: TableCatalogBackingMigrationFenceStatus::Preparing,
|
||||||
target_bucket_existed,
|
target_bucket_existed,
|
||||||
source_fingerprint: None,
|
source_fingerprint: None,
|
||||||
target_snapshot_etag: None,
|
target_snapshot_etag,
|
||||||
};
|
};
|
||||||
self.write_entry(self.catalog_bucket(), &fence_path, &fence, TableCatalogPutPrecondition::IfAbsent)
|
self.write_entry(self.catalog_bucket(), &fence_path, &fence, TableCatalogPutPrecondition::IfAbsent)
|
||||||
.await?;
|
.await?;
|
||||||
@@ -749,7 +922,7 @@ where
|
|||||||
Ok(TableCatalogBackingMigrationExecutionReport {
|
Ok(TableCatalogBackingMigrationExecutionReport {
|
||||||
table_bucket: table_bucket.to_string(),
|
table_bucket: table_bucket.to_string(),
|
||||||
source_kind: TableCatalogBackingKind::ObjectBacked,
|
source_kind: TableCatalogBackingKind::ObjectBacked,
|
||||||
target_kind: TableCatalogBackingKind::StrongKvWal,
|
target_kind: TableCatalogBackingKind::DurableStrongSnapshot,
|
||||||
status: if created {
|
status: if created {
|
||||||
TableCatalogBackingMigrationExecutionStatus::SnapshotMaterialized
|
TableCatalogBackingMigrationExecutionStatus::SnapshotMaterialized
|
||||||
} else {
|
} else {
|
||||||
@@ -793,11 +966,6 @@ where
|
|||||||
});
|
});
|
||||||
};
|
};
|
||||||
self.ensure_global_backing_migration_fence(&global_fence_path).await?;
|
self.ensure_global_backing_migration_fence(&global_fence_path).await?;
|
||||||
if fence.version != TABLE_CATALOG_MIGRATION_VERSION || fence.table_bucket != table_bucket {
|
|
||||||
return Err(TableCatalogStoreError::Invalid(format!(
|
|
||||||
"invalid durable strong migration fence for table bucket {table_bucket}"
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut source_guards = Vec::new();
|
let mut source_guards = Vec::new();
|
||||||
let source = self
|
let source = self
|
||||||
@@ -813,12 +981,21 @@ where
|
|||||||
}
|
}
|
||||||
|
|
||||||
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
let strong_store = StrongTableCatalogStore::new(self.backend.clone());
|
||||||
if fence.status == TableCatalogBackingMigrationFenceStatus::Materialized
|
let (target_fingerprint, target_snapshot_etag) =
|
||||||
&& strong_store.bucket_snapshot_fingerprint(table_bucket).await?.as_deref() != Some(&source_fingerprint)
|
Self::observe_durable_strong_migration_target(&strong_store, table_bucket, Some(&fence)).await?;
|
||||||
{
|
if fence.status == TableCatalogBackingMigrationFenceStatus::Materialized {
|
||||||
return Err(TableCatalogStoreError::Conflict(format!(
|
let target_matches_source = target_fingerprint.as_deref() == Some(&source_fingerprint);
|
||||||
"durable strong catalog state changed after materializing table bucket {table_bucket}"
|
let target_was_already_removed = !fence.target_bucket_existed && target_fingerprint.is_none();
|
||||||
)));
|
if !target_matches_source && !target_was_already_removed {
|
||||||
|
return Err(TableCatalogStoreError::Conflict(format!(
|
||||||
|
"durable strong catalog state changed after materializing table bucket {table_bucket}"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
if target_matches_source && target_snapshot_etag != fence.target_snapshot_etag {
|
||||||
|
return Err(TableCatalogStoreError::Conflict(
|
||||||
|
"durable strong catalog snapshot advanced after materialization".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if !fence.target_bucket_existed {
|
if !fence.target_bucket_existed {
|
||||||
strong_store
|
strong_store
|
||||||
@@ -836,30 +1013,19 @@ where
|
|||||||
}
|
}
|
||||||
|
|
||||||
async fn all_table_buckets_materialized(&self, strong_store: &StrongTableCatalogStore<B>) -> TableCatalogStoreResult<bool> {
|
async fn all_table_buckets_materialized(&self, strong_store: &StrongTableCatalogStore<B>) -> TableCatalogStoreResult<bool> {
|
||||||
if self
|
let Some((global_fence, _)) = self
|
||||||
.read_entry::<TableCatalogBackingMigrationGlobalFence>(
|
.read_entry::<TableCatalogBackingMigrationGlobalFence>(
|
||||||
self.catalog_bucket(),
|
self.catalog_bucket(),
|
||||||
&self.paths.backing_migration_global_fence_path(),
|
&self.paths.backing_migration_global_fence_path(),
|
||||||
)
|
)
|
||||||
.await?
|
.await?
|
||||||
.is_none()
|
else {
|
||||||
{
|
|
||||||
return Ok(false);
|
return Ok(false);
|
||||||
}
|
};
|
||||||
let table_bucket_objects = self
|
Self::validate_global_backing_migration_fence(&global_fence)?;
|
||||||
.backend
|
let source_table_buckets = self.object_backed_table_buckets().await?;
|
||||||
.list_objects(self.catalog_bucket(), &self.paths.table_bucket_entries_prefix())
|
let source_table_bucket_names = source_table_buckets.keys().cloned().collect::<BTreeSet<_>>();
|
||||||
.await?;
|
for entry in source_table_buckets.values() {
|
||||||
for table_bucket_object in table_bucket_objects
|
|
||||||
.iter()
|
|
||||||
.filter(|object| object.ends_with(TABLE_BUCKET_ENTRY_FILE))
|
|
||||||
{
|
|
||||||
let Some((entry, _)) = self
|
|
||||||
.read_entry::<TableBucketEntry>(self.catalog_bucket(), table_bucket_object)
|
|
||||||
.await?
|
|
||||||
else {
|
|
||||||
return Ok(false);
|
|
||||||
};
|
|
||||||
let Some((fence, _)) = self.read_backing_migration_fence(&entry.table_bucket).await? else {
|
let Some((fence, _)) = self.read_backing_migration_fence(&entry.table_bucket).await? else {
|
||||||
return Ok(false);
|
return Ok(false);
|
||||||
};
|
};
|
||||||
@@ -878,6 +1044,26 @@ where
|
|||||||
return Ok(false);
|
return Ok(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Ok(true)
|
Ok(strong_store.table_bucket_names().await? == source_table_bucket_names)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn object_backed_table_buckets(&self) -> TableCatalogStoreResult<BTreeMap<String, TableBucketEntry>> {
|
||||||
|
let mut table_buckets = BTreeMap::new();
|
||||||
|
for object in self
|
||||||
|
.backend
|
||||||
|
.list_objects(self.catalog_bucket(), &self.paths.table_bucket_entries_prefix())
|
||||||
|
.await?
|
||||||
|
.into_iter()
|
||||||
|
.filter(|object| object.ends_with(TABLE_BUCKET_ENTRY_FILE))
|
||||||
|
{
|
||||||
|
let Some((entry, _)) = self.read_entry::<TableBucketEntry>(self.catalog_bucket(), &object).await? else {
|
||||||
|
return Err(TableCatalogStoreError::Conflict(format!(
|
||||||
|
"table bucket changed while reading durable strong migration inventory: {object}"
|
||||||
|
)));
|
||||||
|
};
|
||||||
|
validate_table_bucket_entry_object(&self.paths, &object, &entry)?;
|
||||||
|
table_buckets.insert(entry.table_bucket.clone(), entry);
|
||||||
|
}
|
||||||
|
Ok(table_buckets)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,8 +21,39 @@ mod strong;
|
|||||||
use migration::table_catalog_backing_manifest;
|
use migration::table_catalog_backing_manifest;
|
||||||
pub(crate) use object::ObjectTableCatalogStore;
|
pub(crate) use object::ObjectTableCatalogStore;
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
pub(super) use strong::StrongTableCatalogSnapshot;
|
pub(super) use strong::{
|
||||||
pub(crate) use strong::StrongTableCatalogStore;
|
STRONG_TABLE_CATALOG_RELOAD_MAX_ATTEMPTS, STRONG_TABLE_CATALOG_SNAPSHOT_MAX_SIZE, StrongCommitSnapshotRecord,
|
||||||
|
StrongTableCatalogBucketSnapshot, StrongTableCatalogSnapshot, strong_snapshot_write_version,
|
||||||
|
table_catalog_bucket_snapshot_fingerprint,
|
||||||
|
};
|
||||||
|
pub(crate) use strong::{StrongTableCatalogRuntime, StrongTableCatalogStore};
|
||||||
|
|
||||||
|
fn validate_table_bucket_entry(entry: &TableBucketEntry) -> TableCatalogStoreResult<()> {
|
||||||
|
validate_catalog_entry_version("table bucket", entry.version)?;
|
||||||
|
if entry.table_bucket.is_empty() {
|
||||||
|
return Err(TableCatalogStoreError::Invalid("table bucket name cannot be empty".to_string()));
|
||||||
|
}
|
||||||
|
if entry.catalog_type != TABLE_BUCKET_CATALOG_TYPE {
|
||||||
|
return Err(TableCatalogStoreError::Invalid("unsupported table bucket catalog type".to_string()));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_table_entry_version_and_id(entry: &TableEntry) -> TableCatalogStoreResult<()> {
|
||||||
|
validate_catalog_entry_version("table", entry.version)?;
|
||||||
|
if entry.table_id.is_empty() {
|
||||||
|
return Err(TableCatalogStoreError::Invalid("table id cannot be empty".to_string()));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_view_entry_version_and_id(entry: &ViewEntry) -> TableCatalogStoreResult<()> {
|
||||||
|
validate_catalog_entry_version("view", entry.version)?;
|
||||||
|
if entry.view_id.is_empty() {
|
||||||
|
return Err(TableCatalogStoreError::Invalid("view id cannot be empty".to_string()));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
fn validate_namespace_entry_identity(entry: &NamespaceEntry) -> TableCatalogStoreResult<Namespace> {
|
fn validate_namespace_entry_identity(entry: &NamespaceEntry) -> TableCatalogStoreResult<Namespace> {
|
||||||
validate_catalog_entry_version("namespace", entry.version)?;
|
validate_catalog_entry_version("namespace", entry.version)?;
|
||||||
@@ -406,8 +437,31 @@ pub(crate) enum TableCatalogPutPrecondition {
|
|||||||
IfMatch(String),
|
IfMatch(String),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub(in crate::table_catalog) fn catalog_list_next_continuation(
|
||||||
|
seen: &mut BTreeSet<String>,
|
||||||
|
is_truncated: bool,
|
||||||
|
next: Option<String>,
|
||||||
|
) -> TableCatalogStoreResult<Option<String>> {
|
||||||
|
if !is_truncated {
|
||||||
|
return Ok(None);
|
||||||
|
}
|
||||||
|
let next = next.filter(|next| !next.is_empty()).ok_or_else(|| {
|
||||||
|
TableCatalogStoreError::Internal("truncated catalog object listing has no continuation token".to_string())
|
||||||
|
})?;
|
||||||
|
if !seen.insert(next.clone()) {
|
||||||
|
return Err(TableCatalogStoreError::Internal(
|
||||||
|
"catalog object listing continuation token did not advance".to_string(),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(Some(next))
|
||||||
|
}
|
||||||
|
|
||||||
#[async_trait::async_trait]
|
#[async_trait::async_trait]
|
||||||
pub(crate) trait TableCatalogObjectBackend: Clone + Send + Sync + 'static {
|
pub(crate) trait TableCatalogObjectBackend: Clone + Send + Sync + 'static {
|
||||||
|
fn strong_catalog_runtime(&self) -> Option<StrongTableCatalogRuntime> {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
async fn read_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Option<TableCatalogObject>>;
|
async fn read_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Option<TableCatalogObject>>;
|
||||||
|
|
||||||
async fn read_object_limited(
|
async fn read_object_limited(
|
||||||
@@ -845,10 +899,16 @@ where
|
|||||||
B: TableCatalogObjectBackend,
|
B: TableCatalogObjectBackend,
|
||||||
{
|
{
|
||||||
pub(crate) fn from_env(backend: B) -> TableCatalogStoreResult<Self> {
|
pub(crate) fn from_env(backend: B) -> TableCatalogStoreResult<Self> {
|
||||||
Ok(Self::new(backend, TableCatalogBackingMode::from_env()?))
|
Ok(match TableCatalogBackingMode::from_env()? {
|
||||||
|
TableCatalogBackingMode::ObjectBacked => Self::ObjectBacked(ObjectTableCatalogStore::new(backend)),
|
||||||
|
TableCatalogBackingMode::DurableStrong => {
|
||||||
|
Self::DurableStrong(StrongTableCatalogStore::new_requiring_snapshot(backend))
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
pub(crate) fn new(backend: B, mode: TableCatalogBackingMode) -> Self {
|
#[cfg(test)]
|
||||||
|
pub(crate) fn new_for_test(backend: B, mode: TableCatalogBackingMode) -> Self {
|
||||||
match mode {
|
match mode {
|
||||||
TableCatalogBackingMode::ObjectBacked => Self::ObjectBacked(ObjectTableCatalogStore::new(backend)),
|
TableCatalogBackingMode::ObjectBacked => Self::ObjectBacked(ObjectTableCatalogStore::new(backend)),
|
||||||
TableCatalogBackingMode::DurableStrong => Self::DurableStrong(StrongTableCatalogStore::new(backend)),
|
TableCatalogBackingMode::DurableStrong => Self::DurableStrong(StrongTableCatalogStore::new(backend)),
|
||||||
@@ -1352,12 +1412,14 @@ where
|
|||||||
|
|
||||||
pub(crate) struct EcStoreTableCatalogObjectBackend<S> {
|
pub(crate) struct EcStoreTableCatalogObjectBackend<S> {
|
||||||
store: Arc<S>,
|
store: Arc<S>,
|
||||||
|
strong_runtime: StrongTableCatalogRuntime,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl<S> Clone for EcStoreTableCatalogObjectBackend<S> {
|
impl<S> Clone for EcStoreTableCatalogObjectBackend<S> {
|
||||||
fn clone(&self) -> Self {
|
fn clone(&self) -> Self {
|
||||||
Self {
|
Self {
|
||||||
store: self.store.clone(),
|
store: self.store.clone(),
|
||||||
|
strong_runtime: self.strong_runtime.clone(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1366,8 +1428,8 @@ impl<S> EcStoreTableCatalogObjectBackend<S>
|
|||||||
where
|
where
|
||||||
S: TableCatalogStorage,
|
S: TableCatalogStorage,
|
||||||
{
|
{
|
||||||
pub fn new(store: Arc<S>) -> Self {
|
pub fn new_with_strong_runtime(store: Arc<S>, strong_runtime: StrongTableCatalogRuntime) -> Self {
|
||||||
Self { store }
|
Self { store, strong_runtime }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1378,6 +1440,10 @@ impl<S> TableCatalogObjectBackend for EcStoreTableCatalogObjectBackend<S>
|
|||||||
where
|
where
|
||||||
S: TableCatalogStorage,
|
S: TableCatalogStorage,
|
||||||
{
|
{
|
||||||
|
fn strong_catalog_runtime(&self) -> Option<StrongTableCatalogRuntime> {
|
||||||
|
Some(self.strong_runtime.clone())
|
||||||
|
}
|
||||||
|
|
||||||
async fn read_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Option<TableCatalogObject>> {
|
async fn read_object(&self, bucket: &str, object: &str) -> TableCatalogStoreResult<Option<TableCatalogObject>> {
|
||||||
self.read_object_with_options(bucket, object, ObjectOptions::default(), None)
|
self.read_object_with_options(bucket, object, ObjectOptions::default(), None)
|
||||||
.await
|
.await
|
||||||
@@ -1537,6 +1603,7 @@ where
|
|||||||
|
|
||||||
async fn list_objects(&self, bucket: &str, prefix: &str) -> TableCatalogStoreResult<Vec<String>> {
|
async fn list_objects(&self, bucket: &str, prefix: &str) -> TableCatalogStoreResult<Vec<String>> {
|
||||||
let mut continuation = None;
|
let mut continuation = None;
|
||||||
|
let mut seen_continuations = BTreeSet::new();
|
||||||
let mut objects = BTreeSet::new();
|
let mut objects = BTreeSet::new();
|
||||||
let max_keys = i32::try_from(TABLE_CATALOG_LIST_MAX_KEYS)
|
let max_keys = i32::try_from(TABLE_CATALOG_LIST_MAX_KEYS)
|
||||||
.map_err(|_| TableCatalogStoreError::Internal("catalog list limit exceeds storage API range".to_string()))?;
|
.map_err(|_| TableCatalogStoreError::Internal("catalog list limit exceeds storage API range".to_string()))?;
|
||||||
@@ -1553,14 +1620,10 @@ where
|
|||||||
objects.insert(object.name);
|
objects.insert(object.name);
|
||||||
}
|
}
|
||||||
|
|
||||||
if !result.is_truncated {
|
match catalog_list_next_continuation(&mut seen_continuations, result.is_truncated, result.next_continuation_token)? {
|
||||||
break;
|
Some(next) => continuation = Some(next),
|
||||||
}
|
None => break,
|
||||||
|
|
||||||
let Some(next) = result.next_continuation_token else {
|
|
||||||
break;
|
|
||||||
};
|
};
|
||||||
continuation = Some(next);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(objects.into_iter().collect())
|
Ok(objects.into_iter().collect())
|
||||||
@@ -1627,20 +1690,6 @@ where
|
|||||||
opts: ObjectOptions,
|
opts: ObjectOptions,
|
||||||
max_size: Option<usize>,
|
max_size: Option<usize>,
|
||||||
) -> TableCatalogStoreResult<Option<TableCatalogObject>> {
|
) -> TableCatalogStoreResult<Option<TableCatalogObject>> {
|
||||||
let info = match self.store.get_object_info(bucket, object, &opts).await {
|
|
||||||
Ok(info) => info,
|
|
||||||
Err(err) if is_missing_storage_error(&err) => return Ok(None),
|
|
||||||
Err(err) => return Err(storage_error_to_catalog("read catalog object info", err)),
|
|
||||||
};
|
|
||||||
if let Some(max_size) = max_size {
|
|
||||||
let object_size = usize::try_from(info.size)
|
|
||||||
.map_err(|_| TableCatalogStoreError::Invalid(format!("catalog object {bucket}/{object} has an invalid size")))?;
|
|
||||||
if object_size > max_size {
|
|
||||||
return Err(TableCatalogStoreError::Invalid(format!(
|
|
||||||
"catalog object {bucket}/{object} exceeds the maximum size of {max_size} bytes"
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
let mut reader = match self
|
let mut reader = match self
|
||||||
.store
|
.store
|
||||||
.get_object_reader(bucket, object, None, HeaderMap::new(), &opts)
|
.get_object_reader(bucket, object, None, HeaderMap::new(), &opts)
|
||||||
@@ -1650,6 +1699,17 @@ where
|
|||||||
Err(err) if is_missing_storage_error(&err) => return Ok(None),
|
Err(err) if is_missing_storage_error(&err) => return Ok(None),
|
||||||
Err(err) => return Err(storage_error_to_catalog("read catalog object", err)),
|
Err(err) => return Err(storage_error_to_catalog("read catalog object", err)),
|
||||||
};
|
};
|
||||||
|
if let Some(max_size) = max_size {
|
||||||
|
let object_size = usize::try_from(reader.object_info.size)
|
||||||
|
.map_err(|_| TableCatalogStoreError::Invalid(format!("catalog object {bucket}/{object} has an invalid size")))?;
|
||||||
|
if object_size > max_size {
|
||||||
|
return Err(TableCatalogStoreError::Invalid(format!(
|
||||||
|
"catalog object {bucket}/{object} exceeds the maximum size of {max_size} bytes"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let etag = reader.object_info.etag.clone();
|
||||||
|
let mod_time = reader.object_info.mod_time;
|
||||||
let mut data = Vec::new();
|
let mut data = Vec::new();
|
||||||
if let Some(max_size) = max_size {
|
if let Some(max_size) = max_size {
|
||||||
let read_limit = u64::try_from(max_size.saturating_add(1)).unwrap_or(u64::MAX);
|
let read_limit = u64::try_from(max_size.saturating_add(1)).unwrap_or(u64::MAX);
|
||||||
@@ -1666,11 +1726,7 @@ where
|
|||||||
TableCatalogStoreError::Internal(format!("failed to read catalog object {bucket}/{object}: {err}"))
|
TableCatalogStoreError::Internal(format!("failed to read catalog object {bucket}/{object}: {err}"))
|
||||||
})?;
|
})?;
|
||||||
}
|
}
|
||||||
Ok(Some(TableCatalogObject {
|
Ok(Some(TableCatalogObject { data, etag, mod_time }))
|
||||||
data,
|
|
||||||
etag: info.etag,
|
|
||||||
mod_time: info.mod_time,
|
|
||||||
}))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn put_object_with_options(
|
async fn put_object_with_options(
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+5613
-109
File diff suppressed because it is too large
Load Diff
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user