Files
rustfs/crates/kms/tests/behavior_rotation.rs
T
唐小鸭 62cc19e937 fix(kms): repair unopenable ciphertext and cover the Vault backends (#5668)
* Add black-box behavior tests for KMS resilience and serialization

* fix(kms): repair unopenable ciphertext across backends

Black-box testing of the KMS crate surfaced several defects that make
encrypted data permanently unreadable.

Symmetric envelopes. The Local and Vault Transit backends returned raw
cipher output from `encrypt` while `decrypt` parsed a JSON envelope, so
anything sealed through the master-key path could never be opened again.
Local also discarded the AES-GCM nonce. Both now emit the same envelope
`decrypt` consumes, matching the Static backend.

Deterministic AAD. The object layer derived AEAD additional data by
serializing a `HashMap` directly. Iteration order differs per instance,
so a context rebuilt from storage produced different AAD bytes than the
one used to seal and the object stopped opening. Ordering by key removes
that dependency, matching the Static backend's existing `context_aad`.
Objects written with the default single-key context are unaffected,
since a one-entry map has only one serialization.

Cipher in the header projection. `metadata_to_headers` recorded the SSE
mode (`AES256` / `aws:kms`), which cannot represent ChaCha20-Poly1305,
so a ChaCha-sealed object came back claiming `aws:kms` and was opened
with the wrong cipher. The cipher now travels in
`x-rustfs-encryption-algorithm` — the header the storage layer already
reads but nothing ever wrote. Objects without it fall back as before.

Also: the Static backend ignored `key_spec` and always issued 256-bit
data keys; Local `list_keys` hardcoded `truncated: false`, ignored
`marker`, and paginated over unordered `read_dir`, so a paginating
client silently saw a partial key list; and Local and Vault KV2 reported
`key_id: "unknown"` from `decrypt` despite the envelope naming the
master key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(kms): cover both Vault backends and key rotation

The behavior suite ran only against Local and Static, and its own harness
documented the gap: the Vault backends had no business-capability
coverage at all. Setting `RUSTFS_KMS_VAULT_TOKEN` now adds Vault KV2 and
Vault Transit to every `for_each_backend` spec against a live server.
That lane is what surfaced the Transit envelope defect fixed in the
previous commit.

`rotate` and `versioning` are advertised only by the Vault backends, so
until now every capability-gated branch for them took the
`UnsupportedCapability` side and the working half was never asserted — a
rotation that dropped prior key versions would have gone green. The new
`behavior_rotation.rs` pins that half: material sealed before a rotation
still opens after it, repeated rotations accumulate versions rather than
overwriting a single spare, and the history survives a restart.

Two test defects fixed. `objects_round_trip_across_sizes_and_algorithms`
asserted a 1-byte object differs from its own ciphertext, which collides
once every 256 runs; the assertion now applies only where a collision is
not realistic, and small objects stay covered by the tag check and the
decrypt round-trip. `test_from_env_selects_token_file` depended on
`RUSTFS_KMS_VAULT_TOKEN` being absent from the caller's environment and
now clears it explicitly.

The snapshots directory was also removed from `.gitignore`: insta
snapshots are the assertions themselves, so leaving them untracked gives
CI nothing to compare against. Only `.snap.new` scratch files are
ignored now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(kms): adapt behavior suite to current key APIs

Rebasing onto main brought four API changes the suite predates.

`DeleteKeyRequest` gained `confirm_key_id`, and immediate deletion is now
gated on the server's `allow_immediate_deletion`. Scheduled deletions pass
`None`; the four specs that destroy a key outright echo the key id back
and opt the harness config in, which is what the gate asks of a real
caller.

`LocalBackupExportRequest` gained `sanitized_config`. These specs cover
the key-material path, so they seal no configuration and pass `None`.

`KmsCacheStats` became a named struct with real hit, miss, and eviction
counters. `cache_stats_returns_an_entry_count_and_no_hit_or_miss_data`
existed to pin the old placeholder behavior — that the second tuple
element was always zero — which main has since fixed, so it is now
`cache_stats_reports_hits_and_misses_separately` and asserts the counters
actually move.

Starting the service provisions the reserved probe key, so it shows up in
listings and backup bundles. Exact-set assertions filter it through a new
`without_probe_key` helper rather than naming it, keeping those specs
about the keys they seeded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(kms): bind the AAD to the stored context bytes

Review caught that canonicalizing the AAD on decrypt breaks objects sealed
before canonicalization existed, and it was right. The AAD is the
*serialization* of the encryption context, and `x-rustfs-encryption-context`
stores that exact byte sequence: `encrypt_object` fed one `HashMap` to the
AEAD and then moved the same map into the metadata the header is written
from, so the stored string is byte-identical to the AAD the object was
sealed under. Those objects are therefore recoverable — but only while
nothing round-trips the value through a `HashMap` and re-serializes it.

Recomputing sorted AAD on decrypt would have turned a readable object into
a permanently unreadable one. The previous behavior was worse than the
first analysis credited: it did not merely fail intermittently, it made
the failure deterministic.

`EncryptionMetadata` now carries `context_aad`, the bytes the object was
actually sealed with. Encryption records what it fed the AEAD, the header
projection stores those bytes verbatim (and preserves a legacy ordering
across a re-projection rather than rewriting it into sorted form), and
`headers_to_metadata` carries the stored string through untouched. Both
decrypt paths, SSE-KMS and SSE-C, prefer it and fall back to canonical
serialization only when no stored serialization exists. Canonicalization
still applies to everything newly sealed, so the original ordering bug
cannot recur.

Two tests pin this: a legacy record whose sealed bytes are non-canonical
must survive a full header round trip unchanged, and a context header
rewritten to an equivalent-but-reordered serialization must fail
authentication rather than silently re-deriving a working AAD. Both were
mutation-checked against the reinstated bug on each side.

Also from review: the lifecycle churn test asserted only that every
request was accounted for, which holds whether the state gate exists or
not, so both branches are now pinned deterministically after the churn
(asserting `refused > 0` on the concurrent phase would only trade the hole
for a scheduling flake). And the Local and Vault KV2 envelopes compare
`encryption_context` without authenticating it — `DekCrypto` seals only
the plaintext — which is now documented at both sites; closing it needs a
versioned envelope, since existing ciphertext was sealed without AAD.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-03 23:33:08 +08:00

307 lines
11 KiB
Rust

// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Black-box behavior: key rotation and the version history it must preserve.
//!
//! `BackendCapabilities::rotate` is documented as "rotation that retains prior
//! versions for decryption", and `versioning` as "multiple key versions
//! addressable after rotation". Those are the two claims this file exists to
//! hold, because breaking them destroys data silently: a rotation that dropped
//! the outgoing version would leave every object sealed before it permanently
//! unreadable, while every rotation itself still reported success.
//!
//! Only the Vault backends advertise these capabilities, so this file is the
//! working side of a contract the rest of the suite only ever sees refused.
//! Without the Vault lane on (`RUSTFS_KMS_VAULT_TOKEN`) these specs still run,
//! but they only assert the `UnsupportedCapability` half — see `common`.
mod common;
use common::{BackendCase, assert_unsupported_capability, ctx, for_each_backend, payload};
use rustfs_kms::{DecryptRequest, EncryptRequest, GenerateDataKeyRequest, KeySpec};
fn context() -> std::collections::HashMap<String, String> {
ctx(&[("bucket", "rotation-behavior"), ("object", "alpha.bin")])
}
fn generate_request(key_id: &str) -> GenerateDataKeyRequest {
GenerateDataKeyRequest {
key_id: key_id.to_string(),
key_spec: KeySpec::Aes256,
encryption_context: context(),
}
}
/// The core promise: material sealed before a rotation still opens after it.
///
/// This is the assertion that a "rotation" which merely overwrote the key
/// would fail. Everything else about rotation is recoverable; this is not.
#[tokio::test]
async fn ciphertext_from_before_a_rotation_still_decrypts_after_it() {
for_each_backend(|case: BackendCase| async move {
let manager = case.kms.kms().await;
let label = case.kind().name();
let caps = case.caps().await;
let key_id = case.key_id.clone();
let before = manager
.generate_data_key(generate_request(&key_id))
.await
.unwrap_or_else(|error| panic!("[{label}] generate before rotation should succeed: {error:?}"));
if !caps.rotate {
assert_unsupported_capability(manager.rotate_key(&key_id).await, "rotate_key");
return;
}
manager
.rotate_key(&key_id)
.await
.unwrap_or_else(|error| panic!("[{label}] a backend advertising rotate must rotate: {error:?}"));
let reopened = manager
.decrypt(DecryptRequest {
ciphertext: before.ciphertext_blob.clone(),
encryption_context: context(),
grant_tokens: Vec::new(),
})
.await
.unwrap_or_else(|error| {
panic!("[{label}] rotation must retain the prior version; pre-rotation ciphertext failed to open: {error:?}")
});
assert_eq!(
reopened.plaintext, before.plaintext_key,
"[{label}] the pre-rotation data key must come back byte-identical"
);
})
.await;
}
/// A rotation must not stop the key from being used going forward, and the
/// material it produces afterwards must be independent of the old version.
#[tokio::test]
async fn a_rotated_key_keeps_working_and_issues_fresh_material() {
for_each_backend(|case: BackendCase| async move {
let manager = case.kms.kms().await;
let label = case.kind().name();
let caps = case.caps().await;
let key_id = case.key_id.clone();
if !caps.rotate {
assert_unsupported_capability(manager.rotate_key(&key_id).await, "rotate_key");
return;
}
let before = manager
.generate_data_key(generate_request(&key_id))
.await
.expect("generate before rotation should succeed");
manager.rotate_key(&key_id).await.expect("rotate should succeed");
let after = manager
.generate_data_key(generate_request(&key_id))
.await
.unwrap_or_else(|error| panic!("[{label}] the key must still issue data keys after rotation: {error:?}"));
assert_ne!(
after.plaintext_key, before.plaintext_key,
"[{label}] a data key issued after rotation must not repeat the earlier one"
);
assert_ne!(
after.ciphertext_blob, before.ciphertext_blob,
"[{label}] the wrapped blob must differ across a rotation"
);
// Both generations must be openable at the same time — this is what
// `versioning` means in practice.
for (name, dek) in [("pre-rotation", &before), ("post-rotation", &after)] {
let opened = manager
.decrypt(DecryptRequest {
ciphertext: dek.ciphertext_blob.clone(),
encryption_context: context(),
grant_tokens: Vec::new(),
})
.await
.unwrap_or_else(|error| panic!("[{label}] the {name} data key must stay decryptable: {error:?}"));
assert_eq!(opened.plaintext, dek.plaintext_key, "[{label}] {name} round-trip");
}
})
.await;
}
/// Master-key encryption must survive a rotation on the same terms as data
/// keys: the ciphertext is what a caller stored, and it has to keep opening.
#[tokio::test]
async fn master_key_ciphertext_survives_a_rotation() {
for_each_backend(|case: BackendCase| async move {
let manager = case.kms.kms().await;
let label = case.kind().name();
let caps = case.caps().await;
let key_id = case.key_id.clone();
if !caps.rotate {
assert_unsupported_capability(manager.rotate_key(&key_id).await, "rotate_key");
return;
}
let plaintext = payload(512);
let sealed = manager
.encrypt(EncryptRequest {
key_id: key_id.clone(),
plaintext: plaintext.clone(),
encryption_context: context(),
grant_tokens: Vec::new(),
})
.await
.expect("encrypt before rotation should succeed");
manager.rotate_key(&key_id).await.expect("rotate should succeed");
let opened = manager
.decrypt(DecryptRequest {
ciphertext: sealed.ciphertext.clone(),
encryption_context: context(),
grant_tokens: Vec::new(),
})
.await
.unwrap_or_else(|error| panic!("[{label}] pre-rotation ciphertext must open after rotation: {error:?}"));
assert_eq!(opened.plaintext, plaintext, "[{label}] the plaintext must survive the rotation");
})
.await;
}
/// Repeated rotations must accumulate versions, not overwrite a single spare.
///
/// A backend that kept only "current and previous" would pass a single-rotation
/// test and still lose the oldest objects on the second rotation.
#[tokio::test]
async fn every_generation_survives_repeated_rotations() {
for_each_backend(|case: BackendCase| async move {
let manager = case.kms.kms().await;
let label = case.kind().name();
let caps = case.caps().await;
let key_id = case.key_id.clone();
if !caps.rotate || !caps.versioning {
assert_unsupported_capability(manager.rotate_key(&key_id).await, "rotate_key");
return;
}
let mut generations = Vec::new();
for round in 0..3 {
let dek = manager
.generate_data_key(generate_request(&key_id))
.await
.unwrap_or_else(|error| panic!("[{label}] generate in round {round} should succeed: {error:?}"));
generations.push(dek);
manager
.rotate_key(&key_id)
.await
.unwrap_or_else(|error| panic!("[{label}] rotation {round} should succeed: {error:?}"));
}
for (round, dek) in generations.iter().enumerate() {
let opened = manager
.decrypt(DecryptRequest {
ciphertext: dek.ciphertext_blob.clone(),
encryption_context: context(),
grant_tokens: Vec::new(),
})
.await
.unwrap_or_else(|error| {
panic!(
"[{label}] the data key from round {round} was lost after {} rotations: {error:?}",
generations.len()
)
});
assert_eq!(
opened.plaintext, dek.plaintext_key,
"[{label}] round {round} must round-trip after every later rotation"
);
}
})
.await;
}
/// Version history must live in the backend, not in process memory.
#[tokio::test]
async fn rotation_history_survives_a_restart() {
for_each_backend(|case: BackendCase| async move {
let mut case = case;
let label = case.kind().name();
let caps = case.caps().await;
let key_id = case.key_id.clone();
{
let manager = case.kms.kms().await;
if !caps.rotate {
assert_unsupported_capability(manager.rotate_key(&key_id).await, "rotate_key");
return;
}
}
let before = {
let manager = case.kms.kms().await;
let dek = manager
.generate_data_key(generate_request(&key_id))
.await
.expect("generate before rotation should succeed");
manager.rotate_key(&key_id).await.expect("rotate should succeed");
dek
};
case.kms.restart().await;
let manager = case.kms.kms().await;
let opened = manager
.decrypt(DecryptRequest {
ciphertext: before.ciphertext_blob.clone(),
encryption_context: context(),
grant_tokens: Vec::new(),
})
.await
.unwrap_or_else(|error| {
panic!("[{label}] a pre-rotation key must still open after a restart — the version history must be durable: {error:?}")
});
assert_eq!(
opened.plaintext, before.plaintext_key,
"[{label}] the retained version must survive a restart intact"
);
})
.await;
}
/// Rotating a key that does not exist must fail as a missing key, not be
/// silently treated as a no-op that a caller would read as success.
#[tokio::test]
async fn rotating_an_unknown_key_fails() {
for_each_backend(|case: BackendCase| async move {
let manager = case.kms.kms().await;
let label = case.kind().name();
let caps = case.caps().await;
let result = manager.rotate_key("rotation-no-such-key").await;
if !caps.rotate {
assert_unsupported_capability(result, "rotate_key");
return;
}
assert!(result.is_err(), "[{label}] rotating a key that does not exist must not report success");
})
.await;
}