mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-27 15:37:02 +00:00
fix: 12 P1 reliability/security defects from the full-repo audit (backlog#806) (#4256)
* fix(rio): reject corrupted short compressed/encrypted blocks instead of panicking DecompressReader::poll_read and DecryptReader::poll_read sliced the block body with a fixed `[0..16]` index to read the length varint. The body length comes from an untrusted 24-bit header field, so a corrupted/truncated block shorter than 16 bytes made the slice panic and crash the request task — a read-path DoS on GET of tiered/corrupted data. Pass the whole (arbitrary-length-safe) slice to uvarint and reject a non-positive or out-of-range length prefix with InvalidData. Adds a repro test for each reader; all existing round-trip tests still pass. Refs rustfs/backlog#812 * fix(utils): close SSRF bypass via IPv4-mapped IPv6 addresses validate_outbound_ip branched on the IpAddr variant, and the V6 branch's is_loopback/is_unicast_link_local/is_unique_local checks never inspect the embedded IPv4 of an IPv4-mapped address (::ffff:a.b.c.d). The metadata guard also only matched the plain V4 169.254.169.254. So ::ffff:127.0.0.1, ::ffff:10.0.0.5 and ::ffff:169.254.169.254 all passed the outbound guard, letting an attacker reach loopback/private/metadata endpoints. Normalize IPv4-mapped IPv6 to its embedded IPv4 (via to_ipv4_mapped, which matches only the true mapped form) before classification. Adds reject tests for mapped loopback/private/metadata and an allow test for public IPv6. Refs rustfs/backlog#813 * fix(ecstore): streaming last-part loss, GCS tier Range/remove, stat_all_dirs alignment Four confirmed data-reliability defects: - put_object_multipart_stream: the CompleteMultipartUpload part-collection loop used exclusive `1..total_parts_count`, dropping the final part (and collecting zero parts for a single-part object) — silently truncating the completed object. Extracted collect_complete_parts (1..=total_parts_count) with unit tests. - GCS warm backend get() ignored the requested byte range, returning the whole object for a Range GET; now applies ReadRange::segment like the other backends. - GCS warm backend remove() was an empty stub, so deleting a tiered object left it on GCS forever; now deletes via StorageControl (added a control-plane client), and in_use() actually lists (prefix-scoped) instead of always returning false. - stat_all_dirs skipped None disk slots and dropped JoinErrors, returning a compressed, misaligned error vector; heal_object_dir then zipped it against the full disks array and could make_volume on the WRONG disk. Now returns one index-aligned entry per slot (None -> DiskNotFound), and heal no longer pre-fills the drive report (which would double it). Added an alignment test. Refs rustfs/backlog#807 * fix(kms): stop Vault backend from destroying/reviving keys on failure Two confirmed key-safety defects in the Vault KV2 backend: - get_key_material() 'self-healed' a decrypt or wrong-length failure by minting a fresh random master key and overwriting the stored value. That destroys the original key material, making every DEK ever wrapped by it permanently undecryptable. Decryption must never mutate the stored key: both branches now return a cryptographic_error instead. (The empty-material bootstrap path, which only fills a never-initialized key, is intentionally left intact.) - cancel_key_deletion() reset key_state to Enabled only in the returned response and never persisted it, so the key stayed PendingDeletion in storage and would still be reaped. It now writes the state back via update_key_metadata_in_storage and fails the request if the write fails. Adds ignored (Vault-requiring) integration tests documenting both behaviours. The third item (VaultTransit key state only in memory -> revived as Enabled after restart) is deferred: a fail-closed guard would break restart availability for all transit keys; the correct fix needs a persistent metadata store + Vault integration testing. Tracked in rustfs/backlog#808. Refs rustfs/backlog#808 * fix(admin): clamp STS AssumeRole duration; persist ImportBucketMetadata to disk Two confirmed admin-API defects: - Standard AssumeRole used the raw client-supplied DurationSeconds with no upper bound, so a caller could mint near-permanent temporary credentials. Clamp it to the AWS/MinIO STS window [900, 43200] (with 0 -> default 3600) via a shared clamp_assume_role_duration helper, and build the exp claim with saturating_add. This matches the existing AssumeRoleWithWebIdentity path. - ImportBucketMetadata only mutated an in-memory map and returned 200, silently dropping every imported config. It now persists each non-empty config via metadata_sys::update (which merges onto existing on-disk metadata) and returns InternalError if a write fails. Mapping extracted to imported_configs_to_persist with unit tests. Refs rustfs/backlog#809 * fix(heal): enqueue displacing request in release builds push_displacing_lower_priority folded the real enqueue call into debug_assert_eq!(self.push(request), Accepted). In release builds (debug_assertions off) the whole macro — including its argument — is compiled out, so after evicting a lower-priority queued item the new high-priority request was silently dropped and never healed. Hoist self.push(request) out of the assertion so the side effect runs in all builds. Adds a --release regression test. Refs rustfs/backlog#811 * fix(iam): propagate real delete_policy backend errors instead of swallowing them delete_policy's is_from_notify path had its error handling inverted: a real backend failure (disk IO / insufficient quorum) evicted the cache and returned Ok(()), reporting a phantom success while policy.json survived on disk (to be reloaded on the next full IAM reload); NoSuchPolicy — which should be idempotent success — returned Err. Propagate real errors and let NoSuchPolicy fall through to the idempotent cache-evict + Ok, matching delete_user / the notification handler in the same file. Adds a backend-error-injection regression test. Refs rustfs/backlog#810 * fix(utils): also normalize IPv4-compatible IPv6 in the SSRF guard The initial fix only unwrapped IPv4-mapped (::ffff:a.b.c.d) addresses; the deprecated IPv4-compatible form (::a.b.c.d, e.g. ::127.0.0.1 / ::169.254.169.254) still bypassed the guard. Reject pure-IPv6 specials (::, ::1, fe80::, fc00::) first, then normalize BOTH embedded-IPv4 forms before the IPv4 rules. Adds tests for compatible-form loopback/metadata and confirms ::1 / :: stay rejected. Found by adversarial review of the initial fix. Refs rustfs/backlog#813 * fix(ecstore): fix the same last-part loss in the parallel streaming path put_object_multipart_stream_parallel had the identical off-by-one (1..total_parts_count) that truncated the last part / produced zero parts for a single-part upload — reachable when concurrent stream parts are enabled. Reuse collect_complete_parts, which now returns an error instead of panicking on a gap in the parts map. Adds a missing-part error test. Found by adversarial review of the initial fix. Refs rustfs/backlog#807 * fix(kms): local backend must preserve key material on status change LocalKmsClient (the default KMS backend) regenerated the master key material on enable_key/disable_key/schedule_key_deletion/cancel_key_deletion — a pure status change. A single disable+enable cycle therefore destroyed the original key, making every DEK ever wrapped by it permanently undecryptable (silent data loss, no network needed). Preserve the existing material via get_key_material and re-save with only the status changed. Adds a hermetic regression test that wraps a DEK, cycles all four status methods, and asserts the DEK still decrypts. Found by adversarial review of the Vault fix. Refs rustfs/backlog#808 * test(rio): cover the length-prefix guard; correct its comment Add a DecompressReader test that feeds an unterminated length varint so uvarint returns 0 and the new guard (not the downstream codec) produces the InvalidData error, and reword the guard comment which overclaimed that the > len bound prevents a reachable panic (it is belt-and-suspenders). No behavior change. Found by adversarial review. Refs rustfs/backlog#812 * test(rio): build test block headers via vec! to satisfy clippy The new corrupted-block tests built the header with Vec::new() + repeated push, tripping clippy::vec_init_then_push (-D warnings in CI). Construct the fixed header bytes with vec![] instead. No behavior change. --------- Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Zhengchao An
@@ -524,9 +524,10 @@ impl KmsClient for LocalKmsClient {
|
||||
let mut master_key = self.load_master_key(key_id).await?;
|
||||
master_key.status = KeyStatus::Active;
|
||||
|
||||
// For simplicity, we'll regenerate key material
|
||||
// In a real implementation, we'd preserve the original key material
|
||||
let key_material = generate_key_material(&master_key.algorithm)?;
|
||||
// Preserve the existing key material. Regenerating it on a pure status change would
|
||||
// destroy the original master key and make every DEK ever wrapped by it permanently
|
||||
// undecryptable (silent data loss).
|
||||
let key_material = self.get_key_material(key_id).await?;
|
||||
self.save_master_key(&master_key, &key_material).await?;
|
||||
|
||||
// Update cache
|
||||
@@ -543,7 +544,9 @@ impl KmsClient for LocalKmsClient {
|
||||
let mut master_key = self.load_master_key(key_id).await?;
|
||||
master_key.status = KeyStatus::Disabled;
|
||||
|
||||
let key_material = generate_key_material(&master_key.algorithm)?;
|
||||
// Preserve the existing key material (see enable_key): a status change must never
|
||||
// regenerate the master key, or every DEK wrapped by it becomes undecryptable.
|
||||
let key_material = self.get_key_material(key_id).await?;
|
||||
self.save_master_key(&master_key, &key_material).await?;
|
||||
|
||||
// Update cache
|
||||
@@ -565,7 +568,10 @@ impl KmsClient for LocalKmsClient {
|
||||
let mut master_key = self.load_master_key(key_id).await?;
|
||||
master_key.status = KeyStatus::PendingDeletion;
|
||||
|
||||
let key_material = generate_key_material(&master_key.algorithm)?;
|
||||
// Preserve the existing key material (see enable_key): scheduling deletion must not
|
||||
// regenerate the master key, or cancelling the deletion later would recover a key that
|
||||
// can no longer decrypt existing data.
|
||||
let key_material = self.get_key_material(key_id).await?;
|
||||
self.save_master_key(&master_key, &key_material).await?;
|
||||
|
||||
// Update cache
|
||||
@@ -582,7 +588,9 @@ impl KmsClient for LocalKmsClient {
|
||||
let mut master_key = self.load_master_key(key_id).await?;
|
||||
master_key.status = KeyStatus::Active;
|
||||
|
||||
let key_material = generate_key_material(&master_key.algorithm)?;
|
||||
// Preserve the existing key material (see enable_key): cancelling deletion must recover
|
||||
// the ORIGINAL key, not mint a new one that cannot decrypt existing data.
|
||||
let key_material = self.get_key_material(key_id).await?;
|
||||
self.save_master_key(&master_key, &key_material).await?;
|
||||
|
||||
// Update cache
|
||||
@@ -1020,6 +1028,41 @@ mod tests {
|
||||
assert_eq!(decrypted, data_key.plaintext.clone().expect("No plaintext"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn key_state_transitions_preserve_master_key_material() {
|
||||
// Regression: enable/disable/schedule_deletion/cancel_deletion previously regenerated the
|
||||
// master key material on a pure status change, permanently destroying the ability to
|
||||
// decrypt any DEK wrapped by that key. A status cycle must preserve the material.
|
||||
let (client, _temp_dir) = create_test_client().await;
|
||||
|
||||
let key_id = "state-cycle-key";
|
||||
client.create_key(key_id, "AES_256", None).await.expect("create");
|
||||
|
||||
let request = GenerateKeyRequest::new(key_id.to_string(), "AES_256".to_string())
|
||||
.with_context("bucket".to_string(), "b".to_string());
|
||||
let data_key = client.generate_data_key(&request, None).await.expect("generate data key");
|
||||
let ciphertext = data_key.ciphertext.clone();
|
||||
let plaintext = data_key.plaintext.clone().expect("no plaintext");
|
||||
|
||||
// Cycle through every status-changing method the fix touches.
|
||||
client.disable_key(key_id, None).await.expect("disable");
|
||||
client.enable_key(key_id, None).await.expect("enable");
|
||||
client
|
||||
.schedule_key_deletion(key_id, 7, None)
|
||||
.await
|
||||
.expect("schedule deletion");
|
||||
client.cancel_key_deletion(key_id, None).await.expect("cancel deletion");
|
||||
|
||||
// Pre-fix, each of those regenerated the master key, so this unwrap fails with an AEAD
|
||||
// error. Post-fix, the original material is preserved and the DEK still decrypts.
|
||||
let decrypt_request = DecryptRequest::new(ciphertext).with_context("bucket".to_string(), "b".to_string());
|
||||
let decrypted = client
|
||||
.decrypt(&decrypt_request, None)
|
||||
.await
|
||||
.expect("DEK must still decrypt after status transitions");
|
||||
assert_eq!(decrypted, plaintext, "master key material must survive status transitions");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_encryption_operations() {
|
||||
let (client, _temp_dir) = create_test_client().await;
|
||||
|
||||
@@ -147,28 +147,29 @@ impl VaultKmsClient {
|
||||
let key_material = match self.decrypt_key_material(&key_data.encrypted_key_material).await {
|
||||
Ok(km) => km,
|
||||
Err(e) => {
|
||||
warn!(key_id, error = %e, "Vault KMS key material decrypt failed; regenerating");
|
||||
let new_key_material = generate_key_material(&key_data.algorithm)?;
|
||||
key_data.encrypted_key_material = self.encrypt_key_material(&new_key_material).await?;
|
||||
// Store the updated key data back to Vault
|
||||
self.store_key_data(key_id, &key_data).await?;
|
||||
return Ok(new_key_material);
|
||||
// Never regenerate/overwrite the master key on a decrypt failure: that would
|
||||
// destroy the original material and make every DEK wrapped by this key
|
||||
// permanently undecryptable. Surface the error so the read fails recoverably
|
||||
// instead of causing silent data loss.
|
||||
warn!(key_id, error = %e, "Vault KMS key material could not be decoded");
|
||||
return Err(KmsError::cryptographic_error(
|
||||
"decrypt",
|
||||
format!("Stored key material for {key_id} is corrupted: {e}"),
|
||||
));
|
||||
}
|
||||
};
|
||||
|
||||
// Validate key material length (should be 32 bytes for AES-256)
|
||||
// Validate key material length (should be 32 bytes for AES-256).
|
||||
if key_material.len() != 32 {
|
||||
// Try to fix: generate new key material if length is wrong
|
||||
warn!(
|
||||
"Key {} has invalid key material length ({} bytes), generating new key material",
|
||||
key_id,
|
||||
key_material.len()
|
||||
);
|
||||
let new_key_material = generate_key_material(&key_data.algorithm)?;
|
||||
key_data.encrypted_key_material = self.encrypt_key_material(&new_key_material).await?;
|
||||
// Store the updated key data back to Vault
|
||||
self.store_key_data(key_id, &key_data).await?;
|
||||
return Ok(new_key_material);
|
||||
// As above: do not overwrite the stored key. Report the fault instead.
|
||||
warn!(key_id, len = key_material.len(), "Vault KMS key material has invalid length");
|
||||
return Err(KmsError::cryptographic_error(
|
||||
"decrypt",
|
||||
format!(
|
||||
"Stored key material for {key_id} has invalid length ({} bytes, expected 32)",
|
||||
key_material.len()
|
||||
),
|
||||
));
|
||||
}
|
||||
|
||||
Ok(key_material)
|
||||
@@ -812,6 +813,11 @@ impl KmsBackend for VaultKmsBackend {
|
||||
key_metadata.key_state = KeyState::Enabled;
|
||||
key_metadata.deletion_date = None;
|
||||
|
||||
// Persist the reset state back to Vault. Without this the key stays PendingDeletion in
|
||||
// storage and would still be reaped, so we must fail the request if the write fails
|
||||
// rather than report a false success.
|
||||
self.update_key_metadata_in_storage(key_id, &key_metadata).await?;
|
||||
|
||||
Ok(CancelKeyDeletionResponse {
|
||||
key_id: key_id.clone(),
|
||||
key_metadata,
|
||||
@@ -877,4 +883,94 @@ mod tests {
|
||||
// Test health check
|
||||
client.health_check().await.expect("Health check failed");
|
||||
}
|
||||
|
||||
fn integration_vault_config() -> VaultConfig {
|
||||
VaultConfig {
|
||||
address: "http://127.0.0.1:8200".to_string(),
|
||||
auth_method: VaultAuthMethod::Token {
|
||||
token: "dev-only-token".to_string(),
|
||||
},
|
||||
kv_mount: "secret".to_string(),
|
||||
key_path_prefix: "rustfs/kms/keys".to_string(),
|
||||
mount_path: "transit".to_string(),
|
||||
namespace: None,
|
||||
tls: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore] // Requires a running Vault instance (dev mode)
|
||||
async fn test_corrupted_key_material_does_not_regenerate() {
|
||||
// Regression: get_key_material previously "self-healed" a decrypt/length failure by
|
||||
// minting a fresh random master key and overwriting the stored value — destroying the
|
||||
// original key and making every DEK wrapped by it permanently undecryptable.
|
||||
let client = VaultKmsClient::new(integration_vault_config()).await.expect("client");
|
||||
|
||||
let key_id = format!("corrupt-{}", uuid::Uuid::new_v4());
|
||||
client.create_key(&key_id, "AES_256", None).await.expect("create");
|
||||
|
||||
// Corrupt the stored material to an invalid base64 string.
|
||||
let mut key_data = client.get_key_data(&key_id).await.expect("read");
|
||||
key_data.encrypted_key_material = "!!!not-base64!!!".to_string();
|
||||
client.store_key_data(&key_id, &key_data).await.expect("store corrupt");
|
||||
|
||||
// Reading the material must now ERROR, not silently regenerate + overwrite.
|
||||
assert!(
|
||||
client.get_key_material(&key_id).await.is_err(),
|
||||
"corrupted key material must yield an error, not a fresh key"
|
||||
);
|
||||
|
||||
// And the stored (corrupted) material must be UNCHANGED.
|
||||
let after = client.get_key_data(&key_id).await.expect("reread");
|
||||
assert_eq!(
|
||||
after.encrypted_key_material, "!!!not-base64!!!",
|
||||
"get_key_material must not overwrite stored master key material on failure"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore] // Requires a running Vault instance (dev mode)
|
||||
async fn test_vault_cancel_key_deletion_persists_state() {
|
||||
use crate::config::{BackendConfig, KmsConfig};
|
||||
use crate::types::{CancelKeyDeletionRequest, CreateKeyRequest, DeleteKeyRequest, KeyStatus, KeyUsage};
|
||||
|
||||
let kms_config = KmsConfig {
|
||||
backend_config: BackendConfig::VaultKv2(Box::new(integration_vault_config())),
|
||||
..Default::default()
|
||||
};
|
||||
let backend = VaultKmsBackend::new(kms_config).await.expect("backend");
|
||||
|
||||
let key_id = format!("cancel-persist-{}", uuid::Uuid::new_v4());
|
||||
backend
|
||||
.create_key(CreateKeyRequest {
|
||||
key_name: Some(key_id.clone()),
|
||||
key_usage: KeyUsage::EncryptDecrypt,
|
||||
..Default::default()
|
||||
})
|
||||
.await
|
||||
.expect("create");
|
||||
|
||||
backend
|
||||
.delete_key(DeleteKeyRequest {
|
||||
key_id: key_id.clone(),
|
||||
pending_window_in_days: Some(7),
|
||||
force_immediate: Some(false),
|
||||
})
|
||||
.await
|
||||
.expect("schedule delete");
|
||||
|
||||
backend
|
||||
.cancel_key_deletion(CancelKeyDeletionRequest { key_id: key_id.clone() })
|
||||
.await
|
||||
.expect("cancel");
|
||||
|
||||
// Re-read the PERSISTED state from Vault. Before the fix, storage still held
|
||||
// PendingDeletion because cancel only mutated the response, never wrote back.
|
||||
let persisted = backend.client.get_key_data(&key_id).await.expect("reread");
|
||||
assert_eq!(
|
||||
persisted.status,
|
||||
KeyStatus::Active,
|
||||
"cancel_key_deletion must persist Active status to Vault, not only mutate the response"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user