mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-25 21:46:50 +00:00
fix: 12 P1 reliability/security defects from the full-repo audit (backlog#806) (#4256)
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking DecompressReader::poll_read and DecryptReader::poll_read sliced the block body with a fixed `[0..16]` index to read the length varint. The body length comes from an untrusted 24-bit header field, so a corrupted/truncated block shorter than 16 bytes made the slice panic and crash the request task — a read-path DoS on GET of tiered/corrupted data. Pass the whole (arbitrary-length-safe) slice to uvarint and reject a non-positive or out-of-range length prefix with InvalidData. Adds a repro test for each reader; all existing round-trip tests still pass. Refs rustfs/backlog#812 * fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses validate_outbound_ip branched on the IpAddr variant, and the V6 branch's is_loopback/is_unicast_link_local/is_unique_local checks never inspect the embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1, ::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard, letting an attacker reach loopback/private/metadata endpoints. Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which matches only the true mapped form) before classification. Adds reject tests for mapped loopback/private/metadata and an allow test for public IPv6. Refs rustfs/backlog#813 * fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment Four confirmed data-reliability defects: - put_object_multipart_stream: the CompleteMultipartUpload part-collection loop used exclusive `1..total_parts_count`, dropping the final part (and collecting zero parts for a single-part object) — silently truncating the completed object. Extracted collect_complete_parts (1..=total_parts_count) with unit tests. - GCS warm backend get() ignored the requested byte range, returning the whole object for a Range GET; now applies ReadRange::segment like the other backends. - GCS warm backend remove() was an empty stub, so deleting a tiered object left it on GCS forever; now deletes via StorageControl (added a control-plane client), and in_use() actually lists (prefix-scoped) instead of always returning false. - stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a compressed, misaligned error vector; heal_object_dir then zipped it against the full disks array and could make_volume on the WRONG disk. Now returns one index-aligned entry per slot (None -> DiskNotFound), and heal no longer pre-fills the drive report (which would double it). Added an alignment test. Refs rustfs/backlog#807 * fix(kms): stop Vault backend from destroying/reviving keys on failure Two confirmed key-safety defects in the Vault KV2 backend: - get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a fresh random master key and overwriting the stored value. That destroys the original key material, making every DEK ever wrapped by it permanently undecryptable. Decryption must never mutate the stored key: both branches now return a cryptographic_error instead. (The empty-material bootstrap path, which only fills a never-initialized key, is intentionally left intact.) - cancel_key_deletion() reset key_state to Enabled only in the returned response and never persisted it, so the key stayed PendingDeletion in storage and would still be reaped. It now writes the state back via update_key_metadata_in_storage and fails the request if the write fails. Adds ignored (Vault-requiring) integration tests documenting both behaviours. The third item (VaultTransit key state only in memory -> revived as Enabled after restart) is deferred: a fail-closed guard would break restart availability for all transit keys; the correct fix needs a persistent metadata store + Vault integration testing. Tracked in rustfs/backlog#808. Refs rustfs/backlog#808 * fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk Two confirmed admin-API defects: - Standard AssumeRole used the raw client-supplied DurationSeconds with no upper bound, so a caller could mint near-permanent temporary credentials. Clamp it to the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared clamp_assume_role_duration helper, and build the exp claim with saturating_add. This matches the existing AssumeRoleWithWebIdentity path. - ImportBucketMetadata only mutated an in-memory map and returned 200, silently dropping every imported config. It now persists each non-empty config via metadata_sys::update (which merges onto existing on-disk metadata) and returns InternalError if a write fails. Mapping extracted to imported_configs_to_persist with unit tests. Refs rustfs/backlog#809 * fix(heal): enqueue displacing request in release builds push_displacing_lower_priority folded the real enqueue call into debug_assert_eq!(self.push(request), Accepted). In release builds (debug_assertions off) the whole macro — including its argument — is compiled out, so after evicting a lower-priority queued item the new high-priority request was silently dropped and never healed. Hoist self.push(request) out of the assertion so the side effect runs in all builds. Adds a --release regression test. Refs rustfs/backlog#811 * fix(iam): propagate real delete_policy backend errors instead of swallowing them delete_policy's is_from_notify path had its error handling inverted: a real backend failure (disk IO / insufficient quorum) evicted the cache and returned Ok(()), reporting a phantom success while policy.json survived on disk (to be reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent success — returned Err. Propagate real errors and let NoSuchPolicy fall through to the idempotent cache-evict + Ok, matching delete_user / the notification handler in the same file. Adds a backend-error-injection regression test. Refs rustfs/backlog#810 * fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254) still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::) first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests for compatible-form loopback/metadata and confirms ::1 / :: stay rejected. Found by adversarial review of the initial fix. Refs rustfs/backlog#813 * fix(ecstore): fix the same last-part loss in the parallel streaming path put_object_multipart_stream_parallel had the identical off-by-one (1..total_parts_count) that truncated the last part / produced zero parts for a single-part upload — reachable when concurrent stream parts are enabled. Reuse collect_complete_parts, which now returns an error instead of panicking on a gap in the parts map. Adds a missing-part error test. Found by adversarial review of the initial fix. Refs rustfs/backlog#807 * fix(kms): local backend must preserve key material on status change LocalKmsClient (the default KMS backend) regenerated the master key material on enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status change. A single disable+enable cycle therefore destroyed the original key, making every DEK ever wrapped by it permanently undecryptable (silent data loss, no network needed). Preserve the existing material via get_key_material and re-save with only the status changed. Adds a hermetic regression test that wraps a DEK, cycles all four status methods, and asserts the DEK still decrypts. Found by adversarial review of the Vault fix. Refs rustfs/backlog#808 * test(rio): cover the length-prefix guard; correct its comment Add a DecompressReader test that feeds an unterminated length varint so uvarint returns 0 and the new guard (not the downstream codec) produces the InvalidData error, and reword the guard comment which overclaimed that the > len bound prevents a reachable panic (it is belt-and-suspenders). No behavior change. Found by adversarial review. Refs rustfs/backlog#812 * test(rio): build test block headers via vec! to satisfy clippy The new corrupted-block tests built the header with Vec::new() + repeated push, tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed header bytes with vec![] instead. No behavior change. --------- Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Zhengchao An
@@ -208,20 +208,20 @@ impl TransitionClient {
|
||||
|
||||
let mut compl_multipart_upload = CompleteMultipartUpload::default();
|
||||
|
||||
let mut all_parts = Vec::<ObjectPart>::with_capacity(parts_info.len());
|
||||
let part_number = total_parts_count;
|
||||
for i in 1..part_number {
|
||||
let part = parts_info[&i].clone();
|
||||
|
||||
all_parts.push(part.clone());
|
||||
// Parts are keyed 1..=total_parts_count during upload; every one — including the last —
|
||||
// must be collected. The previous exclusive `1..total_parts_count` bound dropped the final
|
||||
// part, silently truncating the completed object (and produced zero parts for a single-part
|
||||
// upload).
|
||||
let mut all_parts = collect_complete_parts(&parts_info, total_parts_count)?;
|
||||
for part in &all_parts {
|
||||
compl_multipart_upload.parts.push(CompletePart {
|
||||
etag: part.etag,
|
||||
etag: part.etag.clone(),
|
||||
part_num: part.part_num,
|
||||
checksum_crc32: part.checksum_crc32,
|
||||
checksum_crc32c: part.checksum_crc32c,
|
||||
checksum_sha1: part.checksum_sha1,
|
||||
checksum_sha256: part.checksum_sha256,
|
||||
checksum_crc64nvme: part.checksum_crc64nvme,
|
||||
checksum_crc32: part.checksum_crc32.clone(),
|
||||
checksum_crc32c: part.checksum_crc32c.clone(),
|
||||
checksum_sha1: part.checksum_sha1.clone(),
|
||||
checksum_sha256: part.checksum_sha256.clone(),
|
||||
checksum_crc64nvme: part.checksum_crc64nvme.clone(),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -397,20 +397,20 @@ impl TransitionClient {
|
||||
|
||||
let mut compl_multipart_upload = CompleteMultipartUpload::default();
|
||||
|
||||
let part_number: i64 = total_parts_count;
|
||||
let mut all_parts = Vec::<ObjectPart>::with_capacity(parts_info.read().unwrap().len());
|
||||
for i in 1..part_number {
|
||||
let part = parts_info.read().unwrap()[&i].clone();
|
||||
|
||||
all_parts.push(part.clone());
|
||||
// Same inclusive collection as the serial path: parts are keyed 1..=total_parts_count, so
|
||||
// the exclusive `1..total_parts_count` bound dropped the final part (and produced zero
|
||||
// parts for a single-part upload), silently truncating the object.
|
||||
let parts_snapshot = parts_info.read().unwrap().clone();
|
||||
let mut all_parts = collect_complete_parts(&parts_snapshot, total_parts_count)?;
|
||||
for part in &all_parts {
|
||||
compl_multipart_upload.parts.push(CompletePart {
|
||||
etag: part.etag,
|
||||
etag: part.etag.clone(),
|
||||
part_num: part.part_num,
|
||||
checksum_crc32: part.checksum_crc32,
|
||||
checksum_crc32c: part.checksum_crc32c,
|
||||
checksum_sha1: part.checksum_sha1,
|
||||
checksum_sha256: part.checksum_sha256,
|
||||
checksum_crc64nvme: part.checksum_crc64nvme,
|
||||
checksum_crc32: part.checksum_crc32.clone(),
|
||||
checksum_crc32c: part.checksum_crc32c.clone(),
|
||||
checksum_sha1: part.checksum_sha1.clone(),
|
||||
checksum_sha256: part.checksum_sha256.clone(),
|
||||
checksum_crc64nvme: part.checksum_crc64nvme.clone(),
|
||||
..Default::default()
|
||||
});
|
||||
}
|
||||
@@ -573,3 +573,67 @@ impl TransitionClient {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Collect the uploaded parts for CompleteMultipartUpload in ascending part order.
|
||||
///
|
||||
/// Parts are keyed `1..=total_parts_count` during upload (see the upload loop that inserts each
|
||||
/// part), so every one — including the final part — must be collected. The previous exclusive
|
||||
/// `1..total_parts_count` bound dropped the last part, silently truncating the completed object,
|
||||
/// and collected zero parts for a single-part upload.
|
||||
fn collect_complete_parts(parts_info: &HashMap<i64, ObjectPart>, total_parts_count: i64) -> Result<Vec<ObjectPart>, Error> {
|
||||
let mut all_parts = Vec::with_capacity(parts_info.len());
|
||||
for i in 1..=total_parts_count {
|
||||
let part = parts_info
|
||||
.get(&i)
|
||||
.ok_or_else(|| Error::other(format!("missing uploaded part {i} of {total_parts_count}")))?;
|
||||
all_parts.push(part.clone());
|
||||
}
|
||||
Ok(all_parts)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ObjectPart, collect_complete_parts};
|
||||
use std::collections::HashMap;
|
||||
|
||||
fn parts_map(n: i64) -> HashMap<i64, ObjectPart> {
|
||||
let mut m = HashMap::new();
|
||||
for i in 1..=n {
|
||||
m.insert(
|
||||
i,
|
||||
ObjectPart {
|
||||
part_num: i,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
}
|
||||
m
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn collects_every_part_including_the_last() {
|
||||
let collected: Vec<i64> = collect_complete_parts(&parts_map(3), 3)
|
||||
.expect("all parts present")
|
||||
.iter()
|
||||
.map(|p| p.part_num)
|
||||
.collect();
|
||||
assert_eq!(collected, vec![1, 2, 3], "CompleteMultipartUpload must include the final part");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn single_part_upload_submits_one_part() {
|
||||
let collected = collect_complete_parts(&parts_map(1), 1).expect("single part present");
|
||||
assert_eq!(collected.len(), 1, "a single-part object must submit exactly one part, not zero");
|
||||
assert_eq!(collected[0].part_num, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_part_is_an_error_not_a_panic() {
|
||||
let mut m = parts_map(3);
|
||||
m.remove(&2);
|
||||
assert!(
|
||||
collect_complete_parts(&m, 3).is_err(),
|
||||
"a gap in the parts map must be an error, not a panic"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,6 +26,7 @@ use google_cloud_auth::credentials::Credentials;
|
||||
use google_cloud_auth::credentials::user_account::Builder;
|
||||
use google_cloud_storage as gcs;
|
||||
use google_cloud_storage::client::Storage;
|
||||
use google_cloud_storage::client::StorageControl;
|
||||
use std::convert::TryFrom;
|
||||
|
||||
use crate::client::{
|
||||
@@ -46,6 +47,7 @@ const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendGCS {
|
||||
pub client: Arc<Storage>,
|
||||
pub control: Arc<StorageControl>,
|
||||
pub bucket: String,
|
||||
pub prefix: String,
|
||||
pub storage_class: String,
|
||||
@@ -70,15 +72,22 @@ impl WarmBackendGCS {
|
||||
|
||||
let Ok(client) = Storage::builder()
|
||||
.with_endpoint(conf.endpoint.clone())
|
||||
.with_credentials(credentials)
|
||||
.with_credentials(credentials.clone())
|
||||
.build()
|
||||
.await
|
||||
else {
|
||||
return Err(std::io::Error::other("Storage::builder error"));
|
||||
};
|
||||
let client = Arc::new(client);
|
||||
// Control-plane client: the data-plane `Storage` client cannot delete or list objects;
|
||||
// delete_object/list_objects live on StorageControl.
|
||||
let Ok(control) = StorageControl::builder().with_credentials(credentials).build().await else {
|
||||
return Err(std::io::Error::other("StorageControl::builder error"));
|
||||
};
|
||||
let control = Arc::new(control);
|
||||
Ok(Self {
|
||||
client,
|
||||
control,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
@@ -125,7 +134,23 @@ impl WarmBackend for WarmBackendGCS {
|
||||
}
|
||||
|
||||
async fn get(&self, object: &str, rv: &str, opts: WarmBackendGetOpts) -> Result<ReadCloser, std::io::Error> {
|
||||
let Ok(mut reader) = self.client.read_object(&self.bucket, &self.get_dest(object)).send().await else {
|
||||
let mut req = self.client.read_object(&self.bucket, &self.get_dest(object));
|
||||
|
||||
// Honor the requested byte range so Range GETs on tiered objects return the exact
|
||||
// interval instead of the whole object (matches the s3/s3sdk/rustfs warm backends).
|
||||
if opts.start_offset >= 0 && opts.length > 0 {
|
||||
let offset: u64 = opts
|
||||
.start_offset
|
||||
.try_into()
|
||||
.map_err(|_| std::io::Error::other("invalid range: negative start_offset"))?;
|
||||
let count: u64 = opts
|
||||
.length
|
||||
.try_into()
|
||||
.map_err(|_| std::io::Error::other("invalid range: negative length"))?;
|
||||
req = req.set_read_range(google_cloud_storage::model_ext::ReadRange::segment(offset, count));
|
||||
}
|
||||
|
||||
let Ok(mut reader) = req.send().await else {
|
||||
return Err(std::io::Error::other("read_object error"));
|
||||
};
|
||||
let mut contents = Vec::new();
|
||||
@@ -136,23 +161,33 @@ impl WarmBackend for WarmBackendGCS {
|
||||
}
|
||||
|
||||
async fn remove(&self, object: &str, rv: &str) -> Result<(), std::io::Error> {
|
||||
/*self.client
|
||||
.delete_object()
|
||||
.set_bucket(&self.bucket)
|
||||
.set_object(&self.get_dest(object))
|
||||
//.set_generation(object.generation)
|
||||
.send()
|
||||
.await?;*/
|
||||
// gRPC v2 DeleteObject requires the bucket in resource-name form. Without this the
|
||||
// deleted tiered object was never removed from GCS (empty impl returned Ok), leaking
|
||||
// remote data forever.
|
||||
self.control
|
||||
.delete_object()
|
||||
.set_bucket(format!("projects/_/buckets/{}", self.bucket))
|
||||
.set_object(self.get_dest(object))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn in_use(&self) -> Result<bool, std::io::Error> {
|
||||
/*let result = self.client
|
||||
.list_objects_v2(&self.bucket, &self.prefix, "", "", SLASH_SEPARATOR, 1)
|
||||
.await?;
|
||||
// Scope the listing to this tier's prefix (matching the other warm backends) and only
|
||||
// need to know whether a single object exists.
|
||||
let resp = self
|
||||
.control
|
||||
.list_objects()
|
||||
.set_parent(format!("projects/_/buckets/{}", self.bucket))
|
||||
.set_prefix(self.prefix.clone())
|
||||
.set_page_size(1)
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
|
||||
Ok(result.common_prefixes.len() > 0 || result.contents.len() > 0)*/
|
||||
Ok(false)
|
||||
Ok(!resp.objects.is_empty())
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -666,8 +666,10 @@ impl SetDisks {
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
result.before.drives = vec![HealDriveInfo::default(); disks.len()];
|
||||
result.after.drives = vec![HealDriveInfo::default(); disks.len()];
|
||||
// Filled below by pushing one entry per disk while zipping the (index-aligned) `errs`.
|
||||
// Pre-filling here would double the reported drive list once the push loop runs.
|
||||
result.before.drives = Vec::with_capacity(disks.len());
|
||||
result.after.drives = Vec::with_capacity(disks.len());
|
||||
|
||||
let errs = stat_all_dirs(&disks, bucket, object).await;
|
||||
let dangling_object = is_object_dir_dangling(&errs);
|
||||
|
||||
@@ -6862,13 +6862,20 @@ async fn get_storage_info(disks: &[Option<DiskStore>], eps: &[Endpoint]) -> rust
|
||||
}
|
||||
}
|
||||
pub async fn stat_all_dirs(disks: &[Option<DiskStore>], bucket: &str, prefix: &str) -> Vec<Option<DiskError>> {
|
||||
let mut errs = Vec::with_capacity(disks.len());
|
||||
let mut futures = Vec::with_capacity(disks.len());
|
||||
for disk in disks.iter().flatten() {
|
||||
// Spawn one future per disk slot so the returned vector stays index-aligned with `disks`
|
||||
// (and therefore with `set_endpoints`). Offline/None disks must yield DiskNotFound in-place
|
||||
// rather than being skipped, otherwise callers that zip `errs` against the full disks array
|
||||
// (heal_object_dir) would pair every error with the wrong disk/endpoint whenever any disk is
|
||||
// offline — and could `make_volume` on the wrong disk.
|
||||
for disk in disks.iter() {
|
||||
let disk = disk.clone();
|
||||
let bucket = bucket.to_string();
|
||||
let prefix = prefix.to_string();
|
||||
futures.push(tokio::spawn(async move {
|
||||
let Some(disk) = disk else {
|
||||
return Some(DiskError::DiskNotFound);
|
||||
};
|
||||
match disk.list_dir("", &bucket, &prefix, 1).await {
|
||||
Ok(entries) => {
|
||||
if !entries.is_empty() {
|
||||
@@ -6883,8 +6890,14 @@ pub async fn stat_all_dirs(disks: &[Option<DiskStore>], bucket: &str, prefix: &s
|
||||
|
||||
let results = join_all(futures).await;
|
||||
|
||||
for err in results.into_iter().flatten() {
|
||||
errs.push(err);
|
||||
// Preserve length/index alignment: a panicked probe becomes a corrupt-state error instead of
|
||||
// a silently-dropped slot that would re-shift every subsequent index.
|
||||
let mut errs = Vec::with_capacity(disks.len());
|
||||
for res in results.into_iter() {
|
||||
match res {
|
||||
Ok(err) => errs.push(err),
|
||||
Err(join_err) => errs.push(Some(DiskError::other(join_err.to_string()))),
|
||||
}
|
||||
}
|
||||
errs
|
||||
}
|
||||
@@ -10615,4 +10628,28 @@ mod tests {
|
||||
.expect_err("abandoned-parts check should stay in the upper reconciliation layer");
|
||||
assert!(matches!(abandoned_err, StorageError::NotImplemented));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stat_all_dirs_returns_index_aligned_vector_for_offline_disks() {
|
||||
// All-offline set: no real disk I/O needed. Isolates the length/index-alignment contract
|
||||
// that heal_object_dir depends on when it zips `errs` against the full `disks` array.
|
||||
let disks: Vec<Option<DiskStore>> = vec![None, None, None, None];
|
||||
|
||||
let errs = stat_all_dirs(&disks, "bucket", "object").await;
|
||||
|
||||
// Before the fix, offline disks contributed no future and the collected vector had length
|
||||
// 0, so any zip against `disks` paired errors with the wrong disk. After the fix each slot
|
||||
// is DiskNotFound, index-aligned with `disks`.
|
||||
assert_eq!(
|
||||
errs.len(),
|
||||
disks.len(),
|
||||
"stat_all_dirs must return one entry per disk slot to stay index-aligned"
|
||||
);
|
||||
for err in &errs {
|
||||
assert!(
|
||||
matches!(err, Some(DiskError::DiskNotFound)),
|
||||
"offline (None) disk slot must map to DiskNotFound in-place, got {err:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user